Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

...

AUTHORS

  1. Pierre Le Sidaner
  2. DIO / OV, Observatoire de Paris
  3. Baptiste Cecconi
  4. CDPP / LESIA, Observatoire de Paris / CNRS
  5. Stéphane Erard
  6. LESIA, Observatoire de Paris / CNRS
  7. Renaud Savalle
  8. DIO / OV, Observatoire de Paris
  9. Deimleitner Markus
  10. University of Heidelberg
  11. Paul Sladen

...

Introduction

This document presents the full installation procedure to set up data distribution server with an EPN-TAP service. We propose here to set up three interfaces on the same physical server: a standard web server (Apache) to serve data files, a tool to have access statistics (using AWStats), and a DaCHS server to provide the virtual observatory interfaces (including a TAP service). The webserver front end will give access to data, usage statistics and a TAP interface. The various endpoints will be:
http://yourservername.domain.name/data/ Data repository
http://yourservername.domain.name/stat/ Access Statistic interface
http://yourservername.domain.name/vo/ DaCHS server interface
Section 2 describes the Debian distribution installation. Section 3 describes the installation of DaCHS and its configuration. Section 4 describes the installation and configuration of AWStat. Section 5 describes the configuration of the Apache server.

Debian Stable (Jessie) Installation

DaCHS (Data Center Helper Suite) must be installed on a fresh Debian Stable distribution. The current installation image for this distribution is available here:
https://www.debian.org/distrib/netinst#smallcd.
In this example, we used the following ISO image:
debian-8.2.0-amd64-netinst.iso
The installation demonstration is done on a Virtual Box guest computer. This virtual machine is emulating an AMD 64bits processor based system. On boot the Debian installation ISO image is mounted and the system boots from it. On the first screen (Fig. 2.1), select "Install". The next screen are used to configure the system:

  • Language used for the installation process, and for the system, once installed (Fig. 2.2): Select "English"
  • Location of the installation (Fig. 2.3): Select your country.
  • Locale used for the installation (Fig. 2.4): Select "en-US.UTF-8"
  • Keyboard map: Select your keyboard map.
  • Hostname for this computer (Fig. 2.5): Type the hostname (here "dachs")
  • Domain name for this computer (Fig. 2.6): Type the domain name (here "obspm.fr")
  • Root password (Fig. 2.7): Enter root password (pick a password, which is not easy to guess and don't forget it!)
  • Main user (Fig. 2.8): Enter Full Name, user name and password.
  • Disk configuration (Fig. 2.9): Select "Use entire disk"
  • Select Disk to partition (Fig. 2.10): Select the main disk (there should be only one).
  • Partition Scheme (Fig. 2.11): Select "All files in one partition"
  • Confirmation of partition set up (Fig. 2.12): Select "Finish partioning and write changes to disk"
  • Configuration of package manager (Fig. 2.13): Select a mirror close to your place.
  • Proxy configuration (Fig. 2.14): if needed enter the proxy information to go on internet.
  • Configuring popularity-contest (Fig. 2.15): This is up to you (here "no" is selected)
  • Software selection (Fig. 2.16): Select a minimal set of package, namely "web server", "ssh server", "standard system utilities". Note that no desktop environment is selected here. As a server, only shell access is necessary.
  • Installation of GRUB boot loader, part 1 (Fig. 2.17): Select "yes"
  • Installation of GRUB boot loader, part 2 (Fig. 2.18): Select the newly created partition.
  • Installation completion (Fig. 2.19): Select "continue" to finish installation and reboot.
  • Reboot (Fig. 2.20).

...

DaCHS Installation and Configuration

There are several ways to install DaCHS as presented by his author in [1]. For DaCHs installation you need an Debian Stable distribution, at this time Jessie (see Section 2).

Install DaCHS framework

On the server, log with the regular user created during installation (either on the computer itself, or remotely with ssh). The first thing is to install and set up the sudo command, which allows a user to issue commands with root rights. Use the su command to log as root, and type the root password.
user@yourservername:~$ su
Password:
Install the sudo package:
root@yourservername:/home/user# apt-get install sudo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
sudo
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 850 kB of archives.
After this operation, 2,694 kB of additional disk space will be used.
Get:1 http://ftp.fr.debian.org/debian/ jessie/main sudo amd64 1.8.10p3-1+deb8u2 [850 kB]
Fetched 850 kB in 0s (1,110 kB/s)
Selecting previously unselected package sudo.
(Reading database ... 33745 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.10p3-1+deb8u2_amd64.deb ...
Unpacking sudo (1.8.10p3-1+deb8u2) ...
Processing triggers for systemd (215-17+deb8u2) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up sudo (1.8.10p3-1+deb8u2) ...
Processing triggers for systemd (215-17+deb8u2) ...
Set up rights for your regular user to use the sudo command:
root@yourservername:/home/user# adduser "user" sudo
Adding user `user' to group `sudo' ...
Adding user user to group sudo
Done.
You have now the log out from your root shell and your regular user shell. Once you log back in with the regular user, you will be able to use the sudo command.
The repository source list must be updated to include DaCHS packages. Edit the file /etc/apt/sources.list with your preferred command line editor, for instance, you can use the pico editor (useful short cuts: ^O to save, ^X to quit):
user@yourservername:~$ sudo pico /etc/apt/sources.list
Add the following lines at the end of file (see Fig. 3.1):

...

Basic DaCHS Configuration

...

AWStat Installation and Configuration

...

Apache Configuration

...


Info
titleWork in Progress

Please use this Tutorial with caution, we are currently updating it.



Metadata list
hiddentrue
|| workpackage | WP6 |
|| task | 2 |
|| document number | 007 |
|| document version | 0.5 |
|| document title | Registering your VESPA EPN-TAP Server |
|| document type | TD |


Image Added


Image Added


EPN2020-RI


EUROPLANET2020 Research Infrastructure 

H2020-INFRAIA-2014-2015 

Grant agreement no: 654208


Document: VESPA-

Metadata from
1workpackage
-
Metadata from
1task
-
Metadata from
1document number
-
Metadata from
1document type
-v
Metadata from
1document version



Metadata from
1document title




Date: 

current-page-date

Start date of project: 01 September  2015

 Duration: 48 Months

Responsible WP Leader: OBSPARIS / Stéphane Erard


Project co-funded by the European Union's Horizon 2020 research and innovation programme

Dissemination level

PU

Public

  •   

PP

Restricted to other programme participants (including the Commission Service)

  •   

RE

Restricted to a group specified by the consortium (including the Commission Services)

  •   

CO

Confidential, only for members of the consortium (excluding the Commission Services)

  •   


Project Number

654208

Project Title

EPN2020 - RI

Project Duration

48 months: 01 September 2015 – 30 August 2019

Pagebreak

Document Number

VESPA-

Metadata from
1workpackage
-
Metadata from
1task
-
Metadata from
1document number
-
Metadata from
1document type
-v
Metadata from
1document version
(
Current page version
)

Delivery date

 

Title of Document

Metadata from
1document title

Contributing Work package (s)

Metadata from
1workpackage

Dissemination level

PU

Author (s)

Baptiste Cecconi, Pierre Le Sidaner, Stéphane Erard, Renaud Savalle, Markus Deimletner, Paul Sladen


Abstract: This tutorial describes first how to register your newly installed DaCHS server into the IVOA registry. It then introduces the various type of databases and links to corresponding tutorials.



Document history (to be deleted before submission to Commission)

Date

Version

Editor

Change

Status

21 Jan 2016

0.1

First issue

Status
colourYellow
titledraft

 

0.2

Added details

Status
colourYellow
titledraft

 
0.3Small modifications
Status
colourYellow
titledraft

 

0.4


Change name of document
Status
colourYellow
titledraft
31 Dec 2018

Metadata from
1document version

Refined procedure

Status
colourYellow
titledraft


Table of Contents

Table of Contents


Pagebreak

1. Introduction

This document presents the configuration procedure of the VESPA service installed during the first part of the tutorial. We propose here to set up the Registry configuration (Section 2), as well as a test TAP service (Section 3). 

Info

In all this document, you will have to replace all text between <<...>> with the value that you have selected for your installation.

In order to prepare the installation, here is a table to be filled with the parameters of you installation:

Parameter NameYour ValueExample Value for tutorial
my_servername................................................................................voparis-test
my_domain................................................................................obspm.fr
maintainer_email................................................................................vo.paris@obspm.fr
authority-ivoid (1)................................................................................ivo://vopdc.obspm
creator_name (2)
................................................................................Pierre Le Sidaner
logo_url (3)
................................................................................http://padc.obspm.fr/img/logo/padc.png
authority_creationDate (4)
................................................................................
2014-06-03T08:59:09.697573Z
authority_title
................................................................................Paris Astronomical Data Centre
authority_shortname
................................................................................PADC
authority_description
................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................

The Paris Astronomical Data Centre aims at providing VO access to its databases resources, at participating to international standards developments, at implementing VO compliant simulation codes, data visualization and analysis softwares. Paris Astronomical Data Centre offers a central support to the various projects through central storage and web servers computing facilities.

authority_url
................................................................................http://padc.obspm.fr
organization_title
................................................................................Observatoire de Paris
organization_description
................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ Founded in 1667, the Observatoire de Paris is the largest national research center for astronomy. About 30% of all French astronomers are working in its five laboratories and its institute. Situated on the Paris, Meudon and Nançay campuses, they are all associated with the CNRS and, in many cases, with the major scientific universities in the Paris area. The research done at the Paris observatory covers all the fields of contemporary astronomy and astrophysics.
organization_url
................................................................................ http://www.obspm.fr

(1) This ivo-id is that of your naming authority. If your institution is not registered you should put ivo://<<authority_name>>/org (see Tutorial Part 1).
(2) May be different from the maintainer_email. Contact Name is the Data Center contact person, whereas the Creator Name is the contact for this specific server.
(3) The logo must have the following maximum dimensions: width = 120px and height = 60px.  
(4) This date must be formatted in ISO-8901, see the example caution on format. 

2. DaCHS Registry Configuration

The Registry is the Information System of the Virtual Observatory (the yellow pages where all the services are registered). We describe a way to register your "authority ID" (description of the authority managing your resources) and the services provided by DaCHS. This operation shall be done only once. New services will be registered automatically.

First you need to create an identifier for your naming authority ID (ivo://authority_name)

Then open the configuration file:

Code Block
languagebash
sudo pico /var/gavo/etc/defaultmeta.txt

...and edit it with the following lines (see Tutorial Part 1 for some keywords):

Code Block
languagetext
linenumberstrue
publisher: <<authority_name>>
publisherID: <<authority-ivoid>>
contact.name: <<contact_name>> 
contact.address: <<contact_address>>
contact.email: <<contact_email>>
contact.telephone: <<contact_telephone>>
creator.name: <<creator_name>>
creator.logo: <<logo_url>>
authority.creationDate: <<authority_creationDate>>
authority.title: <<authority_title>>
authority.shortName: <<authority_shortname>>
authority.managingOrg: <<my_datacenter-ivoid>>
authority.referenceURL: <<authority_url>>
authority.description: <<authority_description>>
organization.title: <<organization_title>>
organization.referenceURL: <<organization_url>>
organization.description: <<organization_description>>
site.description:<<description of your site (data centre)>>


Then you have to update the userconfig.rd file in the same directory. This file does not exist yet on a fresh installation, so you have to create it. You first need to switch to the root user, and then create the file: 

Code Block
languagebash
cd /var/gavo/etc/
su #root password is asked
gavo admin dumpDF //userconfig > userconfig.rd
exit

Then open the registry-interfacerecords section of the userconfig.rd file for editing

Code Block
languagebash
sudo pico userconfig.rd

And in the file, remove the {UNCONFIGURED} text, so that the content is:

Code Block
languagexml
[...]
<!-- ================================ Registry Interface ============ -->
        <STREAM id="registry-interfacerecords">
                <doc>
                        These are services and registry records for the registry interface
                        of this service.
                        Even if together with defaultmeta, this will just work, keep
                        these elements in your etc/userconfig.rd.
                        The metaString macros in here generally point into defaultmeta.
                        Replace them with whatever actual text applies to your site;  we
                        will work to do away with defaultmeta.txt.
                </doc>
                <resRec id="authority"> <!-- ivo id of the authority is overridden in
                        nonservice.NonServiceResource -->
                        <meta>
                                creationDate: \\metaString{authority.creationDate}              
                                title: \\metaString{authority.title}              
                                shortName: \\metaString{authority.shortName}          
                                subject: Authority
                                managingOrg: \\metaString{authority.managingOrg}             
                                referenceURL: \\metaString{authority.referenceURL}                 
                                identifier: ivo://\getConfig{ivoa}{authority}
                                sets: ivo_managed
                        </meta>
                        <meta name="description">
                                \\metaString{authority.description}
                        </meta>
                </resRec>
                <resRec id="manager"> <!-- the organisation running this registry -->
                        <meta>
                                resType: organization
                                creationDate: \\metaString{authority.creationDate}              
                                title: \\metaString{organization.title}              
                                subject: Organization
                                referenceURL: \\metaString{organization.referenceURL}            
                                identifier: ivo://\getConfig{ivoa}{authority}/org
                                sets: ivo_managed
                        </meta>
                        <meta name="description">
                                \\metaString{organization.description}
                        </meta>
                </resRec>
[...]

In case your organisation is already registered, you need to remove the <resRec id="manager"> ... </resRec> element.

Once this is done, we can publish your registry interface, by issuing the following command:

Code Block
languagebash
dachs pub //services

This will declare your publishing registry, you can verify the content using  http://<<dachs-server-address>>/oai.xml

Read http://docs.g-vo.org/DaCHS/opguide.html#defining-basic-metadata for more details

3. Registering the DaCHS server in Registry of Registry

Go there: http://rofr.ivoa.net/regvalidate/

Enter the URL: http://<<dachs-server-address>>/oai.xml

First step on registry validation result on failed tests have to be None Found

Then a "Step 2: Register your Registry" will be proposed on the web interface with a button Register

Once your registry is inside the list of publishing registries, you have to wait at least one day for the information to be harvested and to see your service available in clients such as TOPCAT


4. Publish your TAP service data collections

In the IVOA registry the right way to declare TAP is to have a TAP service registered and all the data collections registered separately but liked to this TAP service. DaCHS will make the job for you.

To publish the TAP service

Code Block
languagebash
dachs pub //tap

The service TAP will be visible from http://<<dachs-server-address>>/oai.xml


Each of your Vespa service need to be published to be visible in Topcat, Aladin, Taphandle ....
For each service you may have in the q.rd file, into the <table> tag

Code Block
languagexml
<publish sets="local,ivo_managed"/>

from the directory where the q.rd is use the following command

Code Block
languagebash
dachs pub q
dachs serve restart

each collection will appear at http://<<dachs-server-address>>/oai.xml


5. Unregistering a service

In case you build a test service and you register it by mistake, we describe how to set this service as inactive so it will no longer be visible in the registry.

In the q.rd file defining your service, remove the line

Code Block
languagexml
<publish sets="local,ivo_managed"/>

make under user dachsroot the command  

Code Block
dachs pub q.rd
dachs serve restart

to be sure that the declaration have been removed go to

Code Block
http://<<dachs-server-address>>/oai.xml/verb=ListIdentifiers&metadataPrefix=ivo_vor

Your collection will appear in strike characters. 

6. Assessing your dataset

Depending on your dataset (just a series of files, number for data processing levels, presence of metadata, format of data, existing relational database...) various processing may be required. In order to prepare the sharing with EPN-TAP, here is a check list of what you need:

  • Public access to the data files you want to share: either on your server, or on a remote server with a publicly accessible URL (e.g., at NASA/PDS or ESA/PSA).
  • Metadata (relevant for EPNcore) for each file
  • Preview image for each file

Select a suitable data format if the current format is recommended by VESPA.


To be Continued...