You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Article under developpement, not already finished and reviewed do not take it into account for now


Setting-up an EPN-TAP service mean to publish a table in a DACHS server describing data we want to display. This table is referencing each file with a maximum amount of information by filling parameters defined by the standards of EPN TAP called epn_core view . The EPN-TAP standards has mandatory parameters (most of them could be left empty) and optional parameters, finally additional columns could be defined by users to provide specific information. VESPA is a way to interrogate registered services published in different DaCHS servers using keywords or ADQL queries.

This tutorial will provide a very simple example to help beginners to understand how they can set up an EPN-TAP service. In this purpose, we will set up a service called "Planets" providing informations on the solar system planets. It will only document the method using a CSV (Comma Separated Values) file to fill the table.

For this example, we want to display these data on our service :

namemean radius (km)

mean radius uncertainty

(km)

equatorial radius

(km)

equatorial radius

uncertainty

(km)

polar radius

(km)

polar

radius

uncertainty

(km)

rms

deviation

(km)

elevation

max

(km)

elevation

min

(km)

mass

(kg)

distance to

primary

(km)


sideral

rotation

period

(h)

Mercury2439.71.02439.71.02439.71.014.62.53.3014E2357909227.1407.504
Venus6051.81.06051.81.06051.81.011124.86732E24108209475.-5832.432
Earth6371.000.016378.140.016356.750.013.578.8511.525.97219E24149598262.23.93447232
Mars3389.50.23396.190.13376.0.13.022.647.556.41693E23227943824.24.624
Jupiter69911.671492.466854.1062.1311021.89813E27778340821.9.92496
Saturn58232.660268.454364.10102.982055.68319E261426666422.10.656
Uranus25362.725559.424973.2016.82808.68103E252870658186.-17.232
Neptune24622.1924764.1524341.3081401.0241E264498396441.16.104

Steps to follow:

First, a virtual machine hosting a DaCHS server must be setteled up. For simple services like this one, metadata must be referenced in a Comma Separated Values CSV file containing useful features, other methods are available. Then, a Ressource Descriptor must be written in XML. This resource descriptor will read the CSV file and fill a table on DaCHS server. Finally, the service must be registered on VESPA portal (not documented in this tutorial).


  • Install the server on a virtual machine:

First, it is necessary to have settled-up an EPN-TAP server hosting the DaCHS server, method described in this tutorial: EPN-TAP Server Installation for VESPA Data Provider Tutorial#TAPServerInstallationforVESPADataProviderTutorial-6.AWStatsInstallationandConfigurationincludingApache .


  • Define granules :

To publish your own service, you have to define granules. Each granule can link to a file (with the optional parameter access_url). Each granule must have one unique identifier which is a primary key  for the table (the mandatory parameter for this identifier is granule_uid). Each different type of granule must have an identifier (granule_gid parameter).

In our example, each granule have the same type and correspond to one planet. So, basically the granule_gid and granule_uid will be respectively "Planet" and the name of the planet. There is no need to define an access_url because there is no file linked for this example.


  • Define the epn_core parameters

Once granules are defined, you have to browse the EPN-TAP V2 parameters and their description to see what could be informed. In order to define clearly the organization of the final table, it is advised to fill a scheme with the EPN-TAP parameters on one side an what you decide to put into each in the other side (available here in xls format). A large part of the mandatory parameters could be left empty, other parameters you want to add could be defined as additional-columns.


  • Create a CSV file containing metadata

Metadatas which are varying from one granule to the other or cannot be post-treaded in the resource descriptor must be referenced in one column of a CSV file. The first row has to be the column's names, then each row will set the values for a granule. This CSV file will further be read by the resource descriptor to fill the tables in  DaCHS. You can use your favourite programming language to create this CSV file.

For this example, we give the link to a hand-written CSV,  you can downolad it and the associated Resource Descriptor on github:

https://github.com/epn-vespa/DaCHS-for-VESPA/tree/master/q.rd_examples/planets


  • Building a resource descriptor

The resource descriptor is an XML-written file named q.rd which gives DaCHS the method to read the CSV (in case of the CSV method) and fill the metadata table, it follows DACHS Standards for EPN TAP.

Structure of the q.rd file

See the article Building the resource descriptor for your EPN-TAP service in DaCHS (not yet finished) for more detailed information on the meta and table definitions.

For this example, the q.rd file has this structure :

<resource schema="...">

<meta .../>
...
<meta .../>

<table ...>

<mixin .../>

<column .../>
...
<column .../>

</table>

<data id="import">

<sources .../>

<csvGrammar> <rowfilter procDef="//products#define"> <bind name="table">"\schema.epn_core"</bind> </rowfilter> </csvGrammar>

<make table="epn_core">

<rowmaker idmaps="*">

<var key="...">...</var>
...
<var key="...">...</var>

<apply procDef="//epntap2#populate-2_0" name="fillepn">

<bind name="...">@...</bind>
...
<bind name="...">@...</bind>

</apply>
         </rowmaker>
      </make>
   </data>
</resource>

All the file is contained in the tag <resource>.

First, <meta> data are informed.

Then, <table> is defined containing <mixin> reference and <column> definition pointing on columns from the CSV file.

Then, into the tag <data> are defined data ingestion rules, the path of the sourcefile is defined into <sources>, the <csv grammar> is then written to follow EPN-TAP standards.

The <rowmaker> tag content defines how the mandatory and added optional parameters will fill the table.

<make> <rowmaker> elements aim to define varying and constant parameters for the resource.

<var> elements define constant and varying values for table's columns.

In the <apply> tag defined into, <bind> tags aim to parse table and fill constant rows previousyl defined in <var> elements.


Meta tags


The first part is a set of meta tags with different attributes which defines global characteristics of the table.

<resource schema="planets">
<meta name="title">Characteristics of Planets (demo)</meta>
<meta name="description" format="plain">
Main characteristics of planets. Data are included in the table, therefore most relevant parameters are non-standard in EPN-TAP. Data are retrieved from Archinal et al 2009 (IAU report, 2011CeMDA.109..101A) [radii] and Cox et al 2000 (Allen's astrophysical quantities, 2000asqu.book.....C) [masses, heliocentric distances, and rotation periods]. </meta>
<meta name="creationDate">2015-08-16T09:42:00Z</meta>
<meta name="subject">planet</meta>
<meta name="subject">mass</meta>
<meta name="subject">radius</meta>
<meta name="subject">period</meta>
<meta name="copyright">LESIA-Obs Paris</meta>
<meta name="creator.name">Stephane Erard</meta>
<meta name="publisher">Paris Astronomical Data Centre - LESIA</meta>
<meta name="contact.name">Stephane Erard</meta>
<meta name="contact.email">vo.paris@obspm.fr</meta>
<meta name="contact.address">Observatoire de Paris VOPDC, bat. Perrault, 77 av. Denfert Rochereau, 75014 Paris, FRANCE</meta>
<meta name="source">2000asqu.book.....C</meta>
<meta name="contentLevel">General</meta>
<meta name="contentLevel">University</meta>
<meta name="contentLevel">Research</meta>
<meta name="contentLevel">Amateur</meta>
<meta name="utype">ivo://vopdc.obspm/std/EpnCore#schema-2.0</meta>

Most of thes attributes are easy to understand, see Building the resource descriptor for your EPN-TAP service in DaCHS for detailed explanations and more meta elements.

Meta attribute "subject" is defined several times by different keywords defining data sample. "source" referers to the resource-related paper. "contentLevel" has multiple values, here it takes thes four values "General", "University", "Research", "Amateur" but it could take only a part of these.

Table definition

Then, the <table> definition starts, in every EPN-TAP services, the table <id> and <mixin> must respectively take the values  "epn_core" and " //epntap2#table-2.0".

spatial_frame_type element defines the type of coordinate system for the defined granules, it could take several values (see spatial_frame_type section in EPN TAP V2 parameter description) its value will impact the coordinates definition (parameters c1, c2 and c3).

You can list  predefined parameters chosen in the optional_columns attribute.

Here, the spatial_frame_type is defined as celestial which means coordiante defintion is ICRS one (but in this example, these coordinates are not relevant so it will not be informed below).

   <table id="epn_core" onDisk="true" adql="True">

      <mixin spatial_frame_type="celestial"
      optional_columns= "time_scale publisher bib_reference" >//epntap2#table-2_0</mixin>

After mixin definition,  you can start extra parameters definition with the tag <column>. To do that, you should define the attributes name, type, tablehead, unit (if relevant), description, ucd (defines the type of data, see ucd  IVOA documentation ), verblevel (a rate under 30 defining the columns importance). After columns definition, the table is complete.

      <column name="distance_to_primary" type="double precision"
tablehead="Distance_to_primary" unit="km"
description="Extra: Mean heliocentric distance (semi-major axis)"
ucd="pos.distance;stat.min"
verbLevel="2"/>
 <column name= ... />
...
<column name= ... />

</table>

here, data is import in case of a csv-filling

Then, the csv file path is in sources

Grammars will define

   <data id="import">
<sources>data/Masses2.csv</sources>
<csvGrammar>
<rowfilter procDef="//products#define">
<bind name="table">"\schema.epn_core"</bind>
</rowfilter>
</csvGrammar>

Still into the <data> tag, the element <make table="epn_core"> has to be defined with the <rowmaker idmaps="*">

The <var> elements ared defined, key refers to the columns name

constant valued columns could be defined like

<var key="{EPN TAP column name}">{value}</var>

varying valued columns could be defined by:

<var key="{EPN TAP parameter name}" source="{csv column name}"/>  or  <var key="{EPN TAP column name}" >{value}</var>

where value could refers to a column like @{column} or a simple operation like @{column}+1-@{other column}.

on columns during definition using python. To make more complex operations, is it possible  <code> tag could be added


      <make table="epn_core">
<rowmaker idmaps="*">
<var key="dataproduct_type">"ci"</var>
...
<var key="publisher">"LESIA" </var>

A


            <apply procDef="//epntap2#populate-2_0" name="fillepn">
<bind name="granule_gid">@granule_gid</bind>
...
<bind name="release_date">@release_date</bind>
</apply>

B

            </apply>
</rowmaker>
</make>
</data>
</resource>


  • Publish and check tables

You have to create a directory  containing your q.rd and your csv file in gavodachs directory : /var/gavo/inputs/{servicename}, where servie name is the same as resource schema value of the q.rd the resource descriptor in this directory must be named q.rd.

For our example, in your "planets" work directory,  in sudo mode type:

# mkdir /var/gavo/inputs/planets
# mkdir /var/gavo/inputs/planets/data
# cp planets_mixin.rd  /var/gavo/inputs/planets/
# cp Masses2.csv /var/gavo/inputs/planets/data/Masses2.csv

then open this directory

# cd  /var/gavo/inputs/planets/

and check the grammar of your csv and publish your service on your DaCHS server:

# gavo val q.rd

# gavo imp q.rd

# gavo serve restart

You will have to follow these 3 steps each time you change your q.rd or csv file. It is advised to work in another directory for security.

After that, on your browser, type the adress  http://<<my_servername>>.<<my_domain>>:8000/  previsously setteled-up during your server installation.

Click on the service name, you can now see your table informations. It is possible to send ADQL queries. If you want to select the whole database, you can type:

SELECT * FROM planets.epn_core

You can now check up how the table apprears and check if the rows are correctly filled.


  • Register the service

We will not follow this step for the service "Planets", because this service planets is already available in VESPA.

To register your own service in the RoR, you may follow this tutorial : Registering your VESPA EPN-TAP Server when you have finished the build of your RD and published.

Once your service has been published in your DaCHS server,  you may register in in the IVOA Registry of Registries , RoR to make it available in VESPA Query Interface.

To go further ...

An other example with iks service

Add a tag <code>

add a link to a thumnail

programming in python



  • No labels