Introduction

The goal of this page is to identify a roadmap for the evolution of VESPA main search interface.

The VESPA portal is the main EPN-TAP client, which allows the user to discover and query EPN-TAP data services. New functions to be implemented during the course of the project will be discussed here. As a matter of principle, this interface is expected to be more versatile than generic TAP clients and embedded EPN-TAP clients/libraries, and more user friendly.

The older roadmap page from EPN2020 is available here.

Current status

The VESPA portal is intended to query all available EPN-TAP services in one go, or specific services individually in more details.

Two versions are used:

The code of the VESPA portal is also distributed for local implementation, e.g. on Virtual Machines. It is currently on a svn repo in Paris and only available on demand (general security policy with interfaces).

The main developer is Cyril Chauvin at ObsParis/DIO.

An helpdesk link on the first page is intended to send comments to the development team (support.epntap@obspm.fr)


At the onset of EPN-2024, the portal page is split in 3 areas:  

  • A query form on the left side, always visible and active, with several query modes
  • A query results area in the middle - this switches from a list of services to a table providing results in one selected service
  • A housekeeping area on the right side providing links to VO tools, links to examples and help page, and space to plot data thumbnails 

The portal uses a local registry of EPN-TAP services, rather than the IVOA registry. This is intended in particular to fulfill a requirement of the ANO5 service (French level), which is to certified data quality. The local registry only lists services which have been reviewed by the VESPA team. Only those are listed and queried together in default mode.

The current layout results from many interactions and user feedback during EPN-2020. 
• A general assessment of usability will be conducted by SpaceFrog Design, based on User Experience analysis (see VESPA UX/UI).

Main evolution to support future EPN-TAP versions

At the onset of EPN2024, all EPN-TAP services are in v2.0.

A need to implement (relatively) minor modifications will result in the release of EPN-TAP v2.1 in the mid-term. Both versions must be supported simultaneously.

Comments in blue below are requests from various review panels (EC, internal, ANO5 in France after labeling).


Additional functions to be implemented

Global layout / ergonomy

We'll study the evolution of the current layout to improve user experience.

• The most ergonomic model we've identified is the MAST interface, e.g.: https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html?searchQuery=M51

(query panel + results with footprints exposed on the same page, with real-time changes; tab system for successive requests). However, a major difference is that target or coordinates are always a parameter for Astronomy services, and all these data have spatial extension in the same reference frame.

• ESAsky is nicely done and quicker than MAST - again this is more visu on the sky, but there is a dedicated solar system object search based on ephemerides (not on catalogues): https://sky.esa.int

• Another interesting model is the Gaia search interface: https://gea.esac.esa.int/archive/

• TapHandle also has interesting functions to query individual TAP services: http://saada.unistra.fr/taphandle/

In particular, url and s_region can be SAMPED or displayed directly. The SAMP mechanism seems to be more flexible.

• Finally, some PDS interfaces have very effective interfaces, also to be studied (PILOT, OPUS, etc, as well as the PSA modern interface)

Fixes required on Query form

  • Reorganize query panels: add access_format to main panel (search with LIKE)
  • Reorganize query panels: add service_title to Data Reference panel (search with LIKE?) 
  • Also add file_name? (not always there, but same kind of request as granule_uid)
  • Implement all mandatory parameters in standard query mode panel
  • Update value lists for parameters: in particular "none" for spatial_frame_type 
  • Need to implement all common optional parameters in the "optional" panel - long standing issues, and a blocking one
  • The standard mode should detect EPN-TAP version of service, to handle both EPN-TAP 2.0 and 2.1 (when 2.1 is ready…)
  • Check with complete spoly in s_region (beware this is printed as STC-S after an ADQL query and in the output VOtable) see if INTERSECTS works - urgent (the new sandbox DaCHS supports it…). Check if there is an issue with longitude convention and s_region
  • If needed, check if sPoly can be generated on the fly from STC-S string in s_region (see simpleSTCSToPolygon function in module tapstc)
  • Implement spatial queries as in support.epntap ticket 31/5/2016 (with INTERSECTS, CONTAINS… + other modes, check that ticket).
    2019: the full query is more complicated - it requires to upload a table on a data server, so it has to go through TOPCAT. Basic INTERSECTS, etc (on a single region)… may go through the portal, but the Coord Syst should be checked in the same query.
  • Fix queries on string parameters: always use LIKE instead of =, use lower() on both arguments: where lower(instrument_name) like lower('%CIRS%') ; or provide options
  •  
  • Case support still to be implemented on processing_level, dataproduct_type, + target_class - ??

  • Add a list of measurement_type values present in connected services (but allow for other values in the query)

  • do this also for instrument_name & instrument_host_name?
  • On alt_target_name, bib_reference, it would be better to use a combination of hashlist and LIKE. Start with a simple LIKE for the time being.

  • Implement searches with LIKE (instead of =) for all other string parameters

  • On first and next pages: the list of tools is sometimes understood as the complete list of tools connected to VESPA - we have to fix this.
    Plus, some people want a short indication of what these tools are for… (maybe we can add a link to a special help page, or a help bubble, under the subtitle "Plotting tools"?)
    Plus-plus, links to VO support services would be nice (ssodnet, Miriade, Spanish filter service… )
  • We need a function to retrieve all parameters used in declared services (with indication of service where they appear). In some case, existing values may be also required, TBC (this is more a review tool than interface related). Test service in 2018, not finalized (unexpected issue with ?)
  • Need to reflect changes in EPN-TAP v2 PR version (21/7/2021) - in particular value lists

Additional functions on Query form

  • Add a button to convert W longitudes inputs to E longitude (EPNcore std) - this would save time is some situations (eg, query based on planetographic coordinates). This only stands when Spatial_Frame_Type = body
  • When Spatial_Frame_Type = celestial, use conversion system similar to Vizier (interpret all possible formats for RA-DEC, e.g., deg and hh:mm:ss)
  • Test coordinate conversion services, such as Treps or WebgeoCalc, or GDAL. To be used ~ like the name resolver on the interface? - This is more intended for workflows.
  • When ready, use the Observatory list as a resolver in the Instrument panel
  • In Custom/Direct query mode: add a possibility to query views which are not xxx.epn_core? That would provide a general TAP / ObsTAP interface for free. TBC
  • Add link page to other web-based VO clients? Would provide the possibility to get data from astronomy servers through other protocols and share them in the local Hub. This includes Taphandle, Vizier, MAST, etc… 
  • Don't do PDAP query in custom mode 

  • Support queries on multiple targets? Mars#Venus could be translated as target_name= lower('Venus') OR target_name= lower('Mars') - To be investigated.

  • Tag services with science theme (at least in the local registry), and add a query mode restricted to those. May include: surfaces / atmospheres / plasma-radio / exoplanets / solar / dynamics / various.
  • Interesting request: how do we retrieve all Jupiter satellites? Answer is to query quaero with:

    https://api.ssodnet.imcce.fr/quaero/1/sso/search?q=system:Jupiter&type:Satellite (: required after system, not =)


  •  

Re-organization of services Query results 

We currently have one box per service, which may become impractical when we have many services

Ideas:

  • add last modif date in box? Not the one from the registry though, it will never be updated (should be the most recent modification_date in service in v2 - where do we find this?)
  • Add some prefiltering function to only query/display services in a given area (eg: small bodies, surfaces…). This info has to be taken from the registry
  • Important: add provider institute logo in green boxes - can be found in registry declaration.
  • We also have to separate creator of service and creator of content - required by many data providers. See if something fits in RDA recommendations.

Individual service page (custom service)

  • It may be useful to be able to retrieve the existing values of all parameters, or min/max values for numerical fields, TBC

Evolution of  Results from service page (table)

We currently plot only the first 10-1000 results, but the VOtable transmitted from this page contains all results

Ideas:

  • "Download selection" button is implemented (now also on production version) - provides a zip file with results. Check if this requires protection from funny requests.
  • Fix round-off values after conversion - only keep a reasonable number of figures (e.g. in spectral range, not only)
  • Add time out or message if pb (depending on size of retrieved data set?) - currently returns an error
  • JSON output needed (for surfaces). Partially implemented in DaCHS (&FORMAT=json in request). Additional formatting may be required (see with Chiara Marmo). 
  • Need for a data merging function: add a "Merge and Send" function in menu "Data Selection" - would call a script to join tables based on first column, possibly with resampling (STILTS or CASSIS can resample). Most useful for spectra, requested by Grenada service.
  • add a button to show / hide duplicates - i.e., granules with identical values for the parameters displayed on this page (this would allow filtering multiple observations of the same target in the TNO service, for instance). default = hide (suggestion from Florence)
  • Result VOTable should not be systematically named sync, should at least include an extension (settable in DaCHS? - Json output is called sync.json - if not, change in client?)

  • Convert spectral resolution & sampling step from Hz to µm and back - involves 2 parameters, can be done with TAP request (see mail dated 1/8/2017) - TBC
  • Add conversion to cm-1 for spectral range (same conversion as in the query form)
  • Add either info to data service (URL + schema) or link to this in Taphandle, on option (button or info panel)
  • Check why ds9 does not receive fits images from VESPA (2019: OK from HST_planeto, but HTML error in ds9 - seems to be on their side)
  • Can we use a js9 based application to handle 2D spectra from APIS (instead of dying SpecView)?
  • Implement support for datalink, similar to TAP handle
  • Connect Planetary Cesium Viewer as well as Mizar to display bounding boxes - should even plot s_region in fact - this requires a CRS in the geojson output, TBD
  • Make subgranule_url module (in CRISM service) work like in TAPhandle
  • Support for multiple table resources: pass them one by one when SAMPing (required to support VizieR service)
  • SAMP buttons: preserve possibility to select the data type (e.g., to send spectra in TOPCAT: choose table instead of spectrum), but format should be handled automatically (TOPCAT makes a difference between fits and VOtable). 

Integrated service answers

At the onset of EPN2024, this is implemented in a common table merging answers from all services. This is an additional box in the Query results page (no table display):

  • a specific parameter is added to identify the service of origin for each granule - currently the service name, but this relies on services compliancy TBC (ivoID would be better)
  • Include TAP query in the result VOtable header
  • there is currently no associated table display (displaying Results from all services)? TBC if demanding or not
  • check whether coordinated observations require any special function. This must rely on v2 parameters target_time_min / max + target_name

This function may require to build a dynamic db merging all results - but workaround in 2019 test version
Another solution is to join the individual result tables with stilts, or even during the TAP query - done in the portal in 2019 test version (with astropy)


Use cases to consider for service interoperability:

1) Retrieve spectra of asteroids located 3-4 AU from the Sun

  • This assumes the existence of a service providing orbital parameters of asteroids (now OK with MPC)
  • Get distances from this, and use this parameter to filter results from the spectral service (e.g. M4ast)
  • The only common key is target_name - but this still has to be made consistent!

2) Compare observations and simulations

  • Send a query to SPICAM profiles service, grab profiles of interest
  • Get the exact observational parameters (season, local time, location)
  • Send another query using these parameters to a simulation service (MCD)

In both cases, this involves selecting granules of interest and 1 (or several) parameters in the result table, and sending a new query from VESPA to all services.
The second case is more complicated (several parameters) and may require a pipeline (TOPCAT, Jupyter nb?…).
We need to provide at least guidelines on how to do this with real examples.


Ideally, I'd like to click and select parameters in the SPICAM results page to retrieve their values and insert them in a new query.

1) Query all services with
target=Mars
dataproduct = profiles
location : as below

WHERE c1max >= 0.0 AND c1min <= 40.0 AND c2max >= 0.0 AND c2min <= 30.0 AND (dataproduct_type LIKE '%pr%') AND (lower(target_name)= lower('Mars') OR lower(target_name)= lower('4'))

+ add range in solar longitude and local time (not currently possible!)

2) display results from SPICAM and MCD
not currently possible: missing a button in All results to open a service results page in another window. Fixable in the interface ?

3) If selection is accurate enough:
send corresponding profiles to TOPCAT, plot together and compare

2b) Alternatively:

focus on SPICAM results, and enter magic mode.
Clicking Solar longitude would select both min /max values of current line and include them as range in a new query.
Idem for other parameters.

+ Select service #2 (here: MCD)
+ Issue query and plot results in another window

Alt: send new query to all services, plot all results together.


User feedback (through JCM)

- Facet search to be implemented, with number of results updated in real time. Avoid displaying situations with 0 answers

- OPUS is definitely a good model, especially search system & table vs thumbnails views

- Query tab appears too complicated: parameter names are no familiar, possible values are uncertain. Remove secondary parameters from the top panel (only processing level is left there)
In fact, parameters names could be replaced by more familiar ones (target, instrument, observatory/spacecraft…) and could appear only in the contextual help box (already done for Location tab).

- Processing level should be limited to 0-6 range, TBC (no negative values at least)




  • No labels