EPN2020-RI
EUROPLANET2020 Research Infrastructure
H2020-INFRAIA-2014-2015
Grant agreement no: 654208
Document: VESPA-WP11-2-008-SS-v1.0(96)
VESPA EPN-TAP Library specification
Date: $action.dateFormatter.formatGivenString("yyyy-MM-dd",$content.getLastModificationDate())
Start date of project: 01 September 2015
Duration: 48 Months
Responsible WP Leader: Stéphane Erard
Project co-funded by the European Union's Horizon 2020 research and innovation programme | ||
Dissemination level | ||
PU | Public | |
PP | Restricted to other programme participants (including the Commission Service) | |
RE | Restricted to a group specified by the consortium (including the Commission Services) | |
CO | Confidential, only for members of the consortium (excluding the Commission Services) |
Project Number | 654208 |
Project Title | EPN2020 - RI |
Project Duration | 48 months: 01 September 2015 – 30 August 2019 |
Document Number | WP11-task2--v |
Delivery date |
|
Title of Document | VESPA EPN-TAP Library specification |
Contributing Work package (s) | WP11 |
Dissemination level | PU |
Author (s) |
Abstract: |
Document history (to be deleted before submission to Commission) | ||||
Date | Version | Editor | Change | Status |
| First Release | ISSUED | ||
Table of Contents
Introduction
The goal of this document is to specify the requirements for a library dedicated to the management of the EPN-TAP protocol. There will be several versions of the library. The first one, written in Java, will be used by CASSIS and 3DView at least. This specification will be used for the development of other versions, for example in PHP/Javascript for AMDA.
Reference documents
[RD1 ] The EPN-TAP protocol for the Planetary Science Virtual Observatory
https://hal.archives-ouvertes.fr/hal-01092953
[RD2 ] TAP protocol
http://ivoa.net/Documents/TAP
[RD3 ] VOTable format definition
http://www.ivoa.net/documents/VOTable/
[RD4 ] SIMPLE APPLICATION MESSAGING PROTOCOL
http://www.ivoa.net/documents/SAMP/
[RD5 ] IVOA Registry Relational Schema Version 1.0
http://www.ivoa.net/documents/RegTAP/
[RD6 ] Client Interfaces to the Virtual Observatory Registry
http://arxiv.org/pdf/1502.01186v1.pdf
General principles
Selection of services
Users must be able to select
1/ all services compatible with the EPN-TAP protocol
2/ a subset of services depending of a set of keywords , for example the product type (time series, image, spectrum,etc)
Pre-selected parameters
Upon starting of the VESPA function, typically by opening a pop-up window, the context of the tool must be taken into account; e.g. if a target name, area on body, spectral range, time range were previously selected, they must be pre-selected in the pop-up window.
Selection of parameters
Presentation of Query results
Results of queries (lists of granules) must be classified by product type to facilitate the link to suitable visualization tools. A link to "preferred" tools (images to Aladin, spectra to Cassis, time series to AMDA or TopCat) will be proposed
List of methods
getEPNVOResources () returns a set of VOResource elements (one per EPN-TAP service)
getEPNVOResources (keywords) returns a set of VOREsource elements (one per EPN-TAP service corresponding to the keywords).The way keywords are defined is still to be defined
getEPNVOresource (ivoid) returns the VOResource element of the service identified by the ivoID
getEPNServices() returns a VOTable containing the list of EPN-TAP services and their attributes (from a predefined list)
getEPNServices(list of attributes) returns a VOTable containing the list of EPN-TAP services and their attributes (from the list of attributes)
getEPNServices(keywords, list of attributes) returns a VOTable containing the list of EPN-TAP services corresponding to the keywords and their attributes (from the list of attributes)
getEPNService(ivoID) returns a VOTable containing the attributes of the corresponding service (from a predefined list)
getEPNService(ivoID,list of attributes) returns a VOTable containing the attributes of the corresponding service (from the list of attributes)
getEPNCoreTableName(service_ivoid) returns the name of the EPNCore Table related to a service.
getTAPURL(service_ivoid) returns the Access URL of an EPN-TAP Service.
sendADQLQuery(TAPURL,ADQLQuery) returns the list of granules which are compliant with the ADQL Query, in VOTable format . TAPURL is build from elements taken in VOResource.
"ADQLQuery" is created by the Client. It is a full query containing the name of the EPNCore table, taken in VOResource.
sendQuery(TAPURL,schema_name,Query) returns the list of granules which are compliant with the Query, in VOTable format. "Query" is not an ADQL query. It is taken from a list of predefined queries. This list must be created.
Query the IVOA Registries
REST interface to the "Observatoire de Paris" Registry
It is possible to get the list of EPN-TAP services with the REST interface of the Registry maintained by "Observatoire de Paris", using the following request:
http://voparis-registry.obspm.fr/vo/ivoa/1/voresources/search?keywords=standardid:"ivo://ivoa.net/std/TAP"datamodel:"EpnCore"&max=100
The Registry returns, in JSON format, a header followed by an array called "resources" containing the list of EPN-TAP services with their description
Contents of the header:
total: number of VOResource elements compliant with the query
numberReturned: number of VOResource elements returned
From: in case of pagination, starting VOresource, default value is 1
The full specification of the RESTful interface is currently maintained at http://api.vo.obspm.fr/registry/
RegTAP interface to IVOA Registries
It is possible to get the list of EPN-TAP services with the RegTAP interface of several registries maintained by the IVOA. A RegTAP interface is actually an interface implementing the TAP interface, with RegTAP as model. Examples of queries to registries using TAP are given in RD5 (pages 31-36).
List of IVOA Registries providing a RegTAP interface
http://registry.euro-vo.org/regtap/tap
http://dc.zah.uni-heidelberg.de/tap
Definition of terms
- Library - The software library providing access to the VESPA services using TAP.
- Client - The software hosting the Library, which is usually a visualization tool.
- User - The real user accessing the Client to do some scientific analysis.
- Scene - The configuration of the visualization panel (temporal range, spectral range, target...)
Scenario #1: Query all VESPA services known by the IVOA Registry (Query created by the client)
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene, by selecting for example a time span and a target (target_name and target_class). He wants to discover data resources related to this Scene. The Client provides a "Find Data in VESPA" capability. When triggered, the Client internally executes the following steps:
- Step 1. The Client calls getEPNServices(), to retrieve an up-to-date list of all EPN-TAP services in the IVOA Registry. getEPNServices() issues a RegTAP query to the IVOA Registry, searching for TAP services with EPNcore data model, and returns a VOTable containing metadata related to these services. The Client may display the list of services (in a simplified version, including the name of the service and its description, for instance) and may allow the User to select which services he wants to query.
- Step 2. The Client calls sendQuery(TAPURL, schema_name,Query), to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices() and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds the ADQL query with the parameters given by the Client and issues a TAP query to each service with this ADQL query. sendQuery(TAPURL, schema_name,Query) returns a VOTable containing the selected rows for each EPN-TAP service.
- Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to either download the data, use them in the Client (if possible), or send them to SAMP-enabled software.
Scenario #1bis: Query all VESPA services known by the IVOA Registry (ADQL Query created by the client)
The difference between scenarios #1 and #1bis is the creation of the ADQL Query. In #1bis, the ADQL query is created by the Client whereas in #1, it is created by the Library
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene, by selecting for example a time span and a target (target_name and target_class). He wants to discover data resources related to this Scene. The Clientprovides a "Find Data in VESPA" capability. When triggered, the Client internally executes the following steps:
- Step 1. The Client calls getEPNServices(), to retrieve an up-to-date list of all EPN-TAP services in the IVOA Registry. getEPNServices() issues a RegTAP query to the IVOA Registry, searching for TAP services with EPNcore data model, and returns a VOTable containing metadata related to these services. The Client may display the list of services (in a simplified version, including the name of the service and its description, for instance) and may allow the User to select which services he wants to query.
- Step 2. For each service previously selected, the Client creates an ADQL query and calls sendADQLQuery(TAPURL, schema_name,ADQLQuery), to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices() and specifying the search parameters as set in the Scene. sendADQLQuery(TAPURL,schema_name,ADQLQuery) builds the ADQL query with the parameters given by the Client , issues a TAP query to each service with this ADQL query, and returns a VOTable containing the selected rows for each EPN-TAP service.
- Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to either download the data, use them in the Client (if possible), or send them to SAMP-enabled software.
Scenario #2: Query a single VESPA service known by the client and the IVOA Registry
In this scenario, we assume that the client knows the iovID of the service he wants to query; this service is already registered in the IVOA Registry. In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to access remote data from a specific VESPA service, that the Client can ingest automatically. The Client provides an "Add remote Data from VESPA" capability. When triggered, the Client internally executes the following steps:
- Step 1. The Client calls getEPNService(ivoID), to retrieve the IVOA registry record corresponding to the specific EPN-TAP service, using the known "ivoID" of this service. getEPNService(ivoID) issues a RegTAP query to the IVOA registry, searching for the service corresponding to this ivo-id, and returns a VOTable containing the description of this service.
- Step 2. The Client calls sendQuery(TAPURL,schema_name,Query), to query the specific EPN-TAP service, using the access URL provided by getEPNService(ivoID) and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query,and returns a VOTable containing the selected rows for the EPN-TAP service.
- Step 3. The Client displays the results in a form adapted with its scope and interface.
Scenario #2bis: Query a single VESPA service known by the IVOA Registry
In this scenario, the TAP service URL and the Schema name are obtained using specific methods, where as in Scenario #2, a single method provides the metadata, and the Client has to extract the TAP service URL and Schema name on its own.
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to access remote data from a specific VESPA service, that the Client can ingest automatically. The Client provides an "Add remote Data from VESPA" capability. When triggered, the Client internally executes the following steps:
- Step 1. The Client calls getTAPURL(service_ivoid) and getEPNCoreTableName(service_ivoid), to retrieve the AccessURL and schema_name of the service service, using its known "ivoID".
- Step 2. The Client calls sendQuery(TAPURL,schema_name,Query), to query the specific EPN-TAP service. sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query, and returns a VOTable containing the selected rows for the EPN-TAP service.
- Step 3. The Client displays the results in a form adapted with its scope and interface.
Scenario #3: Query a single VESPA service unknown by the IVOA Registry
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (idem scenario #1). The Client proposes to test a new VESPA service not yet in the Registry, and knows its accessURL and schema_name. The Client provides a "Query Custom VESPA service " capability. When triggered, the Client internally executes the following steps:
- Step 1. The Client calls sendQuery(TAPURL,schema_name,Query) to query the specific EPN-TAP service, using the access URL and schema name provided by the User and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds an ADQL query corresponding to the parameters of the scene , issues a TAP query to the service with this ADQL query,and returns a VOTable containing the selected rows for the EPN-TAP service.
- Step 2. The Client displays the results in a form adapted with its scope and interface.
This scenario is similar to the "Custom Resource" page of the generic VESPA client: http://vespa.obspm.fr/planetary/data/epn/query/resource/
Scenario #4: Query a subset of VESPA services known by the IVOA Registry
In the Client (AMDA, 3Dview, CASSIS, PlanetServer...), the User has set up a Scene (as in scenario #1). The Client sets up a window in which users are able to select services they want to query, according to one or more keywords ( plain text). This is similar to the capability provided by TopCat.
- Step 1. The Client calls getEPNServices(keywords), to query the Registry (RegTAP interface) to search for EPN-TAP services, according to one or more keywords. getEPNServices(keywords) returns a VOTable containing metadata related to these services. The Client may display the list of services (the name of the service, its ivoID and its description, for instance) and may allow the User to select which services he wants to query.
- Step 2. The Client calls sendQuery(TAPURL, schema_name,Query) to query each EPN-TAP service previously selected, using the access URL and schema_name provided by getEPNServices(keywords) and specifying the search parameters as set in the Scene. sendQuery(TAPURL,schema_name,Query) builds the ADQL query with the parameters given by the Client, issues a TAP query to each service with this ADQL query, and returns a VOTable containing the selected rows for each EPN-TAP service.
- Step 3. The Client displays the list of results in a form adapted with its scope and interface. The interface showing the results shall allow the User to select the data, use them in the Client (if possible), or send them to SAMP-enabled software.
EPN-TAP Requests
The goal of an EPN-TAP query is to get the values of some parameters in a table. Therefore, the arguments of the query are the columns of this table. The arguments are used to filter the contents of the database. The response of the service contains only the rows of the table which correspond to these arguments.
An EPN-TAP Request is sent using HTTP GET or POST
The accessURL of the service can be found in the resources/capabilities/accessurl element ( see below )
for example, TAPURL = http://voparis-tap.obspm.fr/__system__/tap/run/tap
Add the part dedicated to the Query itself
Synchronous mode:
TAPURL/sync?QueryLanguage+query
Asynchronous mode:
TAPURL/async?QueryLanguage+query
Examples
TAPURL = http://voparis-tap.obspm.fr/__system__/tap/run/tap
HTTP GET Queries
TAPURL/sync?request=doQuery&lang=ADQL&query=select top 100 * from apis.epn_core
returns the first hundred granules from APIS
TAPURL/sync/request=doquery&lang=adql&query=select * from apis.epn_core where time_min > '2455197.5' and time_max < '2455927.5'
returns from APIS the list of granules from 01/01/2010 ( 2455197.5 ) to 01/01/2012 ( 2455927.5 )
HTTP POST Queries
this query is sent with the Unix curl command
curl --data "REQUEST=doQuery&LANG=ADQL&QUERY=select * from amdadb.epn_core where instrument_host_name='Pioneer_10'"
http://cdpp-epn-tap.cesr.fr/__system__/tap/run/tap/sync
returns the list of granules which contain data from the Pioneer 10 Spacecraft
Example of VOResource element
<resources>
<status>active</status>
<publisher>IA2</publisher>
<updated>2013-12-05T08:30:00Z</updated>
<contentlevel>Research</contentlevel>
<description>
The Cosmic dust catalog is an internal resource of the SBDN, since we have internally developed original services to access this catalogs. NASA's Cosmic dust catalog 15 and 18 have been joined to obtain this service. 467 (from catalog 15) plus 957 (from catalog 18) dust grains with their main characteristics, images and X-ray spectra are listed. Not only cosmic dust particles are listed, but also terrestrial contamination (natural), terrestrial contamination (artificial) and aluminium oxide spheres.
</description>
<contributors>INAF-IAPS</contributors>
<creator>IA2</creator>
<provenance>ivo://esavo/registry</provenance>
<referenceurl>
http://www.iasf-roma.inaf.it:8080/web/sbdn/cosmic-dust-catalog
</referenceurl>
<created>2013-02-20T08:00:49Z</created>
<contactemail>ia2@oats.inaf.it</contactemail>
<subjects>dust catalogue</subjects>
<capabilities>
<accessurl>http://ia2-tap.oats.inaf.it:8080/epntap</accessurl>
<standardid>ivo://ivoa.net/std/TAP</standardid>
<datamodel>EpnCore-1.0</datamodel>
</capabilities>
<contactname>IA2</contactname>
<title>INAF-IAPS RDB NASA dust catalogue TAP service</title>
<shortname>nasadustcat</shortname>
<identifier>
ivo://ia2.inaf.it/hosted/iaps/epn/tap/nasadustcat
</identifier>
<type>CatalogService</type>
</resources>
10 Comments
Baptiste Cecconi
Registry queries should also be described here, as it started in the draft version (in French)
Baptiste Cecconi
This includes the new RegTAP, the current Registry Interface query API (and also the PADC (VOParis) registry interface ?)
Gangloff Michel
Examples of common queries to the Relational Registry (RegTAP) are given in :
"IVOA Registry Relational Schema Version 1.0" page #31-36
Baptiste Cecconi
good, this reference should be in the document.
Laurent Beigbeder
Why do we need to query all services if we can query all resources straight forward ?
Jean-Michel Glorian
why do you say we can query all resources straight forward ?
Laurent Beigbeder
With getEPNVOResources()
Jean-Michel Glorian
For me the only possibility to get VOResources is to use the Registry REST API v1 of observatoire paris but
it is not a registry certified by the IVOA Institute. May be Pierre Le Sidaner can confirm ?
The question is do we have to implement these functions getEPNVOResources ...
Gangloff Michel
I don't understand your question. VOResource is a set of metadata related to a service. It contains for example its accessURL. Query a service means "send an ADQL query to a service". It is up to the client to decide whether he wants to send a Query to all services or to a selection of services.
Gangloff Michel
getEPNVOResources() would be used by those who want to have a look at the full description of the services. It is an alternative to getEPNServices().
I guess that most clients will never use it.
We can keep the definition of this method in the document, but assign it a low priority (in term of requirement, assign it a MAY instead of MUST)