EPN2020-RI


EUROPLANET2020 Research Infrastructure 

H2020-INFRAIA-2014-2015 

Grant agreement no: 654208


Document: VESPA-WP11-2-008-SS-v1.0(96)






VESPA EPN-TAP Library specification




Date: $action.dateFormatter.formatGivenString("yyyy-MM-dd",$content.getLastModificationDate())

Start date of project: 01 September  2015

 Duration: 48 Months

Responsible WP Leader: Stéphane Erard


Project co-funded by the European Union's Horizon 2020 research and innovation programme

Dissemination level

PU

Public

  •  

PP

Restricted to other programme participants (including the Commission Service)

  •  

RE

Restricted to a group specified by the consortium (including the Commission Services)

  •  

CO

Confidential, only for members of the consortium (excluding the Commission Services)

  •  

Project Number

654208

Project Title

EPN2020 - RI

Project Duration

48 months: 01 September 2015 – 30 August 2019

Document Number

WP11-task2--v

Delivery date

 

Title of Document

VESPA EPN-TAP Library specification

Contributing Work package (s)

WP11

Dissemination level

PU

Author (s)

Abstract:



Document history (to be deleted before submission to Commission)

Date

Version

Editor

Change

Status

 

First Release

ISSUED





Table of Contents

Introduction

The goal of this document is to specify the requirements for a library dedicated to the management of the EPN-TAP protocol. There will be several versions of the library. The first one, written in Java, will be used by CASSIS and 3DView at least. This specification will be used for the development of other versions, for example in PHP/Javascript for AMDA.

Reference documents

[RD1 ] The EPN-TAP protocol for the Planetary Science Virtual Observatory
https://hal.archives-ouvertes.fr/hal-01092953

[RD2 ]  TAP protocol  
http://ivoa.net/Documents/TAP

[RD3 ] VOTable format definition
http://www.ivoa.net/documents/VOTable/

[RD4 ] SIMPLE APPLICATION MESSAGING PROTOCOL
http://www.ivoa.net/documents/SAMP/

[RD5 ] IVOA Registry Relational Schema Version 1.0
http://www.ivoa.net/documents/RegTAP/

[RD6 ] Client Interfaces to the Virtual Observatory Registry
http://arxiv.org/pdf/1502.01186v1.pdf

General principles

Selection of services

Users must be able to select 
1/ all services compatible with the EPN-TAP protocol
2/ a subset of services depending of a set of keywords , for example the product type (time series, image, spectrum,etc)

Pre-selected parameters

Upon starting of the VESPA function, typically by opening a pop-up window, the context of the tool must be taken into account; e.g.  if a target name, area on body, spectral range, time range were previously selected, they must be pre-selected in the pop-up window.

Selection of parameters

After selection of services by the user, the list of all the parameters that can compose a query must be displayed. Mandatory and optional parameters must be displayed differently.
The EPN-TAP query is built with a selection of these parameters and sent to all the compliant services selected by the user. In some clients, experienced users may be able to compose directly an ADQL query.
 

Presentation of Query results 

Results of queries (lists of granules) must be classified by product type to facilitate the link to suitable visualization tools. A link to "preferred" tools (images to Aladin, spectra to Cassis, time series to AMDA or TopCat) will be proposed

List of methods

getEPNVOResources () returns a set of VOResource elements (one per EPN-TAP service)

getEPNVOResources (keywords) returns a set of VOREsource elements (one per EPN-TAP service corresponding to the keywords).The way keywords are defined is still to be defined

getEPNVOresource (ivoid) returns the VOResource element of the service identified by the ivoID

getEPNServices() returns a VOTable containing the list of EPN-TAP services and their attributes (from a predefined list)

getEPNServices(list of attributes) returns a VOTable containing the list of EPN-TAP services and their attributes (from the list of attributes)

getEPNServices(keywords, list of attributes) returns a VOTable containing the list of EPN-TAP services corresponding to the keywords and their attributes (from the list of attributes)

getEPNService(ivoID) returns a VOTable containing the attributes of the corresponding service (from a predefined list)

getEPNService(ivoID,list of attributes) returns a VOTable containing the attributes of the corresponding service (from the list of attributes)

getEPNCoreTableName(service_ivoid) returns the name of the EPNCore Table related to a service.

getTAPURL(service_ivoid) returns the Access URL of an EPN-TAP Service.

sendADQLQuery(TAPURL,ADQLQuery) returns the list of granules which are compliant with the ADQL Query, in VOTable format . TAPURL is build from elements taken in VOResource.
"ADQLQuery" is created by the Client. It is a full query containing the name of the EPNCore table, taken in VOResource. 

sendQuery(TAPURL,schema_name,Queryreturns the list of granules which are compliant with the Query, in VOTable format. "Query" is not an ADQL query. It is taken from a list of predefined queries. This list must be created. 

Query the IVOA Registries

      REST interface to the "Observatoire de Paris" Registry 

      It is possible to get the list of EPN-TAP services with the REST interface of the Registry maintained by  "Observatoire de Paris", using the following request:

             http://voparis-registry.obspm.fr/vo/ivoa/1/voresources/search?keywords=standardid:"ivo://ivoa.net/std/TAP"datamodel:"EpnCore"&max=100

      The Registry returns, in JSON format, a header followed by an array called "resources" containing the list of EPN-TAP services with their description

       Contents of the header:
                     total: number of VOResource elements compliant with the query
                     numberReturned: number of VOResource elements returned
                     From:  in case of pagination, starting VOresource, default value is 1

      The full specification of the RESTful interface is currently maintained at http://api.vo.obspm.fr/registry/

      RegTAP interface to IVOA Registries

       It is possible to get the list of EPN-TAP services with the RegTAP interface of several registries maintained by the IVOA. A RegTAP interface is actually an interface implementing the TAP interface, with RegTAP as model. Examples of queries to registries using TAP are given in RD5 (pages 31-36).

      List of IVOA Registries providing a RegTAP interface

          http://registry.euro-vo.org/regtap/tap

          http://gavo.aip.de/tap

          http://reg.g-vo.org/tap

          http://dc.zah.uni-heidelberg.de/tap

          http://gavo.aip.de/tap

Definition of terms

  • Library - The software library providing access to the VESPA services using TAP.
  • Client - The software hosting the Library, which is usually a visualization tool.
  • User - The real user accessing the Client to do some scientific analysis.
  • Scene - The configuration of the visualization panel (temporal range, spectral range, target...)

Scenario #1: Query all VESPA services known by the IVOA Registry (Query created by the client)

In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene, by selecting for example a time span and a target (target_name and target_class). He wants to discover data resources related to this Scene. The Client provides a "Find Data in VESPA" capability. When triggered, the Client internally executes the following steps:

  • Step 1. The Client calls getEPNServices(), to retrieve an up-to-date list of all EPN-TAP services in the IVOA Registry. getEPNServices() issues a RegTAP query to the IVOA Registry, searching for TAP services with EPNcore data model, and returns a VOTable containing metadata related to these services. The Client may display the list of services (in a simplified version, including the name of the service and its description, for instance) and may allow the User to select which services he wants to query.
  • Step 2. The Client calls sendQuery(TAPURL, schema_name,Query)to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices()  and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds  the ADQL query with the parameters given by the Client  and issues a TAP query to each service with this ADQL query. sendQuery(TAPURL, schema_name,Query) returns a  VOTable containing the selected rows for each EPN-TAP service.  
  • Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to either download the data, use them in the Client (if possible), or send them to SAMP-enabled software.

Scenario #1bis: Query all VESPA services known by the IVOA Registry (ADQL Query created by the client)

The difference between scenarios #1 and #1bis is the creation of the ADQL Query. In #1bis, the ADQL query is created by the Client whereas in #1, it is created by the Library 

In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene, by selecting for example a time span and a target (target_name and target_class). He wants to discover data resources related to this Scene. The Clientprovides a "Find Data in VESPA" capability. When triggered, the Client internally executes the following steps:

  • Step 1. The Client calls getEPNServices(), to retrieve an up-to-date list of all EPN-TAP services in the IVOA Registry. getEPNServices() issues a RegTAP query to the IVOA Registry, searching for TAP services with EPNcore data model, and returns a VOTable containing  metadata related to these services. The Client may display the list of services (in a simplified version, including the name of the service and its description, for instance) and may allow the User to select which services he wants to query.
  • Step 2. For each service previously selected, the Client creates an ADQL query and calls sendADQLQuery(TAPURL, schema_name,ADQLQuery)to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices()  and specifying the search parameters as set in  the ScenesendADQLQuery(TAPURL,schema_name,ADQLQuery) builds  the ADQL query with the parameters given by the Client  , issues a TAP query to each service with this ADQL query, and returns a  VOTable containing the selected rows for each EPN-TAP service.  
  • Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to either download the data, use them in the Client (if possible), or send them to SAMP-enabled software.

Scenario #2: Query a single VESPA service known by the client and the IVOA Registry

In this scenario, we assume that the client  knows the iovID of the service he wants to query; this service is already registered in the IVOA Registry. In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to access  remote data from a specific VESPA service, that the Client can ingest automatically. The Client provides an "Add remote Data from VESPA" capability. When triggered, the Client internally executes the following steps:

  • Step 1. The Client calls getEPNService(ivoID), to retrieve the IVOA registry record corresponding to the specific EPN-TAP service, using the known "ivoID" of this service. getEPNService(ivoID) issues a RegTAP query to the IVOA registry, searching for the service corresponding to this ivo-id, and returns a VOTable containing the description of this service. 
  • Step 2. The Client calls  sendQuery(TAPURL,schema_name,Query), to query the specific EPN-TAP service, using the access URL provided by getEPNService(ivoID)  and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query,and returns a VOTable containing the selected rows for the EPN-TAP service.  
  • Step 3. The Client displays the results in a form adapted with its scope and interface.


Scenario #2bis: Query a single VESPA service known by the IVOA Registry

In this scenario, the TAP service URL and the Schema name are obtained using specific methods, where as in Scenario #2, a single method provides the metadata, and the Client has to extract the TAP service URL and Schema name on its own.
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to access  remote data from a specific VESPA service, that the Client can ingest automatically. The Client provides an "Add remote Data from VESPA" capability. When triggered, the Client internally executes the following steps:

  • Step 1. The Client calls getTAPURL(service_ivoid) and getEPNCoreTableName(service_ivoid), to retrieve the AccessURL and schema_name of the service service, using its known "ivoID". 
  • Step 2. The Client calls sendQuery(TAPURL,schema_name,Query), to query the specific EPN-TAP service. sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query, and returns a VOTable containing the selected rows for the EPN-TAP service.  
  • Step 3. The Client displays the results in a form adapted with its scope and interface.


Scenario #3: Query a single VESPA service unknown by the IVOA Registry

In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to test a new VESPA service not yet  in the Registry, and knows its accessURL and schema_name. The Client provides a "Query Custom VESPA service " capability. When triggered, the Client internally executes the following steps:

  • Step 1. The Client calls sendQuery(TAPURL,schema_name,Query)  to query the specific EPN-TAP service, using the access URL and schema name provided by the User and specifying the search parameters as set in the Scene.  sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query,and returns a VOTable containing the selected rows for the EPN-TAP service.  
  • Step 2. The Client displays the results in a form adapted with its scope and interface.

This scenario is similar to the "Custom Resource" page of the generic VESPA client: http://vespa.obspm.fr/planetary/data/epn/query/resource/


Scenario #4: Query a subset of VESPA services known by the IVOA Registry

In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (as in scenario #1). The Client sets up a window in which users are able to select services they want to query, according to one or more keywords ( plain text). This is similar to the capability provided by TopCat.

  • Step 1. The Client calls getEPNServices(keywords), to query the Registry (RegTAP interface) to search for EPN-TAP services, according to one or more keywords. getEPNServices(keywords) returns a VOTable containing metadata related to these services. The Client may display the list of services (the name of the service, its ivoID and its description, for instance) and may allow the User to select which services he wants to query.   
  • Step 2. The Client calls sendQuery(TAPURL, schema_name,Query) to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices(keywords)  and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds  the ADQL query with the parameters given by the Client, issues a TAP query to each service with this ADQL query, and returns a  VOTable containing the selected rows for each EPN-TAP service.  
  • Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to select  the data, use them in the Client (if possible), or send them to SAMP-enabled software.

EPN-TAP Requests

The goal of an EPN-TAP query is to get the values of some parameters in a table. Therefore, the arguments of the query are the columns of this table. The arguments are used to filter the contents of the database. The response of the service contains only the rows of the table which correspond to these arguments.

An EPN-TAP Request is sent using HTTP GET or POST

The accessURL of the service can be found in the resources/capabilities/accessurl element ( see below )

for example, TAPURL = http://voparis-tap.obspm.fr/__system__/tap/run/tap

Add the part dedicated to the Query itself

Synchronous mode:

TAPURL/sync?QueryLanguage+query

Asynchronous mode:

TAPURL/async?QueryLanguage+query

Examples

       TAPURL = http://voparis-tap.obspm.fr/__system__/tap/run/tap

        HTTP GET Queries

           TAPURL/sync?request=doQuery&lang=ADQL&query=select top 100 * from apis.epn_core

          returns the first hundred granules from APIS

           TAPURL/sync/request=doquery&lang=adql&query=select * from apis.epn_core where time_min > '2455197.5' and time_max < '2455927.5'  

           returns from APIS the list of granules from 01/01/2010 (  2455197.5 ) to 01/01/2012 ( 2455927.5 )

        HTTP POST Queries

        this query is sent with the Unix curl command

         curl --data "REQUEST=doQuery&LANG=ADQL&QUERY=select * from amdadb.epn_core where instrument_host_name='Pioneer_10'"
                http://cdpp-epn-tap.cesr.fr/__system__/tap/run/tap/sync

         returns the list of granules which contain data from the Pioneer 10 Spacecraft

             

Example of VOResource element

<resources>
            <status>active</status>
            <publisher>IA2</publisher>
            <updated>2013-12-05T08:30:00Z</updated>
            <contentlevel>Research</contentlevel>
            <description>
                The Cosmic dust catalog is an internal resource of the SBDN, since we have internally developed original services to access this catalogs. NASA's Cosmic dust catalog 15 and 18 have                   been joined to obtain this service. 467 (from catalog 15) plus 957 (from catalog 18) dust grains with their main characteristics, images and X-ray spectra are listed. Not only cosmic dust                   particles are listed, but also terrestrial contamination (natural), terrestrial contamination (artificial) and aluminium oxide spheres.
            </description>
            <contributors>INAF-IAPS</contributors>
            <creator>IA2</creator>
            <provenance>ivo://esavo/registry</provenance>
            <referenceurl>
                 http://www.iasf-roma.inaf.it:8080/web/sbdn/cosmic-dust-catalog
            </referenceurl>
            <created>2013-02-20T08:00:49Z</created>
            <contactemail>ia2@oats.inaf.it</contactemail>
            <subjects>dust catalogue</subjects>
            <capabilities>
                  <accessurl>http://ia2-tap.oats.inaf.it:8080/epntap</accessurl>
                  <standardid>ivo://ivoa.net/std/TAP</standardid>
                  <datamodel>EpnCore-1.0</datamodel>
            </capabilities>
            <contactname>IA2</contactname>
            <title>INAF-IAPS RDB NASA dust catalogue TAP service</title>
            <shortname>nasadustcat</shortname>
            <identifier>
                ivo://ia2.inaf.it/hosted/iaps/epn/tap/nasadustcat
            </identifier>
            <type>CatalogService</type>
</resources>

  • No labels

10 Comments

  1. Registry queries should also be described here, as it started in the draft version (in French)

    1. This includes the new RegTAP, the current Registry Interface query API (and also the PADC (VOParis) registry interface ?)

  2. Examples of common queries to the Relational Registry (RegTAP) are given in :

    "IVOA Registry Relational Schema Version 1.0" page #31-36

     

    1. good, this reference should be in the document.

  3. Why do we need to query all services if we can query all resources straight forward ?

    1. why do you say we can query all resources straight forward ?

      1. With getEPNVOResources()

        1. For me the only possibility to get VOResources is to use the Registry REST API v1 of observatoire paris  but
          it is not a registry certified by the IVOA Institute. May be Pierre Le Sidaner can confirm ?

          The question is do we have to implement these functions getEPNVOResources ...

           

           

    2. I don't understand your question. VOResource is a set of metadata related to a service. It contains for example its accessURL. Query a service means "send an ADQL query to a service". It is up to the client to decide whether he wants to send a Query to all services or to a selection of services.

  4. getEPNVOResources() would be used by those who want to have a look at the full description of the services. It is an alternative to getEPNServices().

    I guess that most clients will never use it.

    We can keep the definition of this method in the document, but assign it a low priority (in term of requirement, assign it a MAY instead of MUST)