The goal of this page is to identify a roadmap for the evolution of VESPA main search interface.
The user interface is the main EPN-TAP client, which allows the user to discover and query EPN-TAP data services. New functions to be implemented during the course of the project will be discussed here. As a matter of principle, this interface is expected to be more versatile than embedded EPN-TAP clients/libraries, and more user friendly.
The upgraded interface is a deliverable from ObsParis at PM 40 (D11.9), with prototype at PM36.
The VESPA main user interface is intended to query all available EPN-TAP services in one go, or specific services individually in more details.
Since the program kick-off, two versions are available:
- Production version (standard user interface): http://vespa.obspm.fr
- Development version (with additional functions to be tested by the team): http://voparis-europlanet-dev.obspm.fr
The main developer is Cyril Chauvin at ObsParis/DIO.
An on-line help is available from the interface, describing all the functions implemented and the ergonomy: http://voparis-europlanet-dev.obspm.fr/planetary/data/epn/help
The helpdesk on the first page is intended to send comments to the development team (email@example.com)
The current system relies on a three-step process: Query form page; Query results page; Result from service page.
- The VESPA code is supposed to be distributed for local implementation, eg on Virtual Machines. It is currently on a svn repo in Paris - should we move it to the VESPA github? - no, this is a general security issue with interfaces.
Main evolution to support EPN-TAP v2
The client, initially designed for EPN-TAP v1, has been adapted to EPN-TAP v2 in spring 2016.
This involved an update of the search interface to query all v2 parameters. Older v1 services can be still handled provided they are slightly adapted (mostly the deprecated dataset & index concepts) - in practice, all services have been adapted and there should be no pure v1 service reachable on any TAP server, including sandboxes. In the mid-term, v1 is expected to disappear entirely and turned to v2 services.
Comments in blue below are requests from various review panels (EC, internal, ANO5 in France after labeling).
Additional functions to be implemented
Global layout / ergonomy
In the long term, we'll study the evolution of the current 3-page system.
• The most ergonomic model we've identified is the MAST interface, e.g.: https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html?searchQuery=M51
(query panel + results with footprints exposed on the same page, with real-time changes; tab system for successive requests). However, a major difference is that target or coordinates are always a parameter for Astronomy services, and all these data have spatial extension in the same reference frame.
• Another interesting model is the Gaia search interface: https://gea.esac.esa.int/archive/
• TapHandle also has interesting functions to query individual TAP services: http://saada.unistra.fr/taphandle/
- Cyril Chauvin (12/2016) the query and service results pages must be displayed together at all time (confirmed to be a strong expectation from ANO5 committee in France, ie a first priority!), e.g. as:
- Cyril Chauvin (4/2018) - this is becoming a priority, we need a basic working model and enough time to play with, so as to identify missing functions & develop iteratively. Don't care about buttons, pretty icons and so on for the time being. It has to be published by summer!
The left part (on one column) can be switched between various query modes, the right part can switch between services and results (either in individual services or overall)
An advantage is to see the number of results changing in all services as the query is modified (considered psychologically important!)
Rework layout of first (Query form) page
Cyril is currently working at a new layout based on more general html/css libraries (now on dev version). This is used in the framework of the current 3-page system.
Ergonomics needs to be optimized.
- Always print service name with proper case everywhere (upper case required when this is an acronym)
- (not on the interface) Support of v2 requires an update of all DaCHS servers to support case sensitive requests & list parsing (now done for √ Helio services + √ sandbox in Paris, + √ Toulouse, √ Rome)
Fixes required on (Query form) page
- Upgrade Custom resource mode to v2
- Implement v2 on "All VO" mode
- Finalize parameters cleaning in dev version (there are still some Index, etc: check with v2 list) ) — this is part of a review of all services
- In the mid term, the "All VO" mode on prod server should detect EPN-TAP version of service (i.e., with or without resource_type and other param mandatory in v1 but not in v2) - this is probably obsolete in 2017 since no EPN-TAP v1 service is expected to remain available in the registry
- The standard mode should detect EPN-TAP version of service, to handle both EPN-TAP 2.0 and 2.1 (when 2.1 is ready…; no hurry)
- Implement case support (requires all DaCHS servers in latest version)
- Remove trailing spaces from query values, and extra / in custom server URL
- In some cases the dev version is not reachable when public one is OK (ex: 16/5/2016 21:00) - Dev interface can crash after some queries, identify why and fix
- Fix mouse-over help: use generic v2 strings in std mode,
- to do also in Custom mode (identical), and advanced query page - or parse the service q.rd.
- Fix mouse-over help in location panel: change All to None, adapt names depending on selected service (see Baptiste's mail dated 18/5/2017) - take definitions on EPN-TAP parameters page, now checked
- Implement support for queries in longitude (when crossing prime meridian) - urgent (and required anyway for 1D searches)
- When only one coordinate (c1 or c2 min/max) is queried, compare with corresponding fields (min/max)
- Test CONTAINS and INTERSECTS with s_region. This is the optimal query mode on 2 spatial coordinates when a footprint is provided
- Check with complete spoly in s_region (beware this is printed as STC-S after an ADQL query and in the output VOtable) see if INTERSECTS works - urgent (the new sandbox DaCHS supports it…). Check if there is an issue with longitude convention and s_region
- If needed, check if sPoly can be generated on the fly from STC-S string in s_region (see simpleSTCSToPolygon function in module tapstc)
- Implement spatial queries as in support.epntap ticket 31/5/2016 (with INTERSECTS, CONTAINS… + other modes, check that ticket).
2019: the full query is more complicated - it requires to upload a table on a data server, so it has to go through TOPCAT. Basic INTERSECTS, etc (on a single region)… may go through the portal, but the Coord Syst should be checked in the same query.
- Define default behavior when searching for an interval on an axis - should return data overlapping this interval
- Is there a need for an option to return only data encompassing the whole interval (to get continuous coverage)? If yes, on option.
- Fix queries on string parameters: always use LIKE instead of =, use lower() on both arguments: where lower(instrument_name) like lower('%CIRS%') ; or provide options
- Test ivo_hashlist_has for multi-valued fields (lower function required on the tested parameter only):
Select top 10 * from titan.epn_core where 1 = ivo_hashlist_has(lower(instrument_name), 'CIRS' )
Pb: not compatible with 'like' function, TBC
(27/1/2017) Implemented on, target_name, instrument_host_name, instrument_name, measurement_type
- Complete dataproduct _type as per 2019 EPN-TAP doc
Case support still to be implemented on processing_level, dataproduct_type, + target_class (no problem expected) - ??
Add a list of measurement_type values present in connected services (but allow for other values in the query)
- Same for instrument_name & instrument_host_name?
On alt_target_name, bib_reference, it would be better to use a combination of hashlist and LIKE. Start with a simple LIKE for the time being.
Implement searches with LIKE (instead of =) for all other string parameters
- Reorganize query panels: add access_format to main panel (search with LIKE)
- Reorganize query panels: add service_title to Data Reference panel (search with LIKE?)
- Also add file_name? (not always there, but same kind of request as granule_uid)
- Implement all mandatory parameters in standard query mode panel
- Update value lists for parameters: in particular "none" for spatial_frame_type
- On first and next pages: the list of tools is sometimes understood are the complete list of tools connected to VESPA - we have to fix this. Plus, some people want a short indication of what these tools are for… (maybe we can add a link to a special help page, or a help bubble, under the subtitle "Plotting tools"?)
- Identify additional parameters to query SSHADE - OK, now done when querying individual services
- Registry - now that we can actually get the services from the registry (4/2018), we see a need to filter crap, tests, and things unwanted. Because VESPA is the only clients which queries all services, selecting the ones we put forward is an added scientific value in VESPA (the others are always accessible from the Custom mode; we don't want to display non-compliant services by default). It means that we need a table of acceptable/validated services, and get related info on those from the registry. OK, this is the local portal registry
- We need a function to retrieve all parameters used in declared services (with indication of service where they appear). In some case, existing values may be also required, TBC (this is more a review tool than interface related). Test service in 2018, not finalized (unexpected issue with ?)
Additional functions on first page (Query form)
- Systematically convert ] longitude range to [ ] range (terrestrial to planetary / EPNcore convention). This only stands when Spatial_Frame_Type = body
- Add a button to convert W longitudes inputs to E longitude (EPNcore std) - this would save time is some situations (eg, query based on planetographic coordinates). This only stands when Spatial_Frame_Type = body
- When Spatial_Frame_Type = celestial, use conversion system similar to Vizier (interpret all possible formats for RA-DEC, e.g., deg and hh:mm:ss)
- Add support for spectral queries in µm & cm-1 (at least) + use coeff and transforms from the IKS view (for µm)
- Do we need to enter queries through files, for mass processing? Not sure it makes sense (a "Direct query" field is already available)
- Direct Queries mode should be available on Custom resource (not going through registry) - urgent for tests
- Test coordinate conversion services, such as Treps or WebgeoCalc. To be used ~ like the name resolver on the interface? - This is more intended for workflows.
- When ready, use the Observatory list as a resolver in the Instrument panel
- If set, grab target type together with target name to query the name resolver and get main designation (in particular with asteroids/comets)
- Use name resolver to query all aliases of target name provided (with target_name like… OR target_name like… ?)
- In Custom/Direct query mode: add a possibility to query views which are not xxx.epn_core? That would provide a general TAP / ObsTAP interface for free. TBC
- Add link page to other web-based VO clients? Would provide the possibility to get data from astronomy servers through other protocols and share them in the local Hub. This includes Taphandle, Vizier, MAST, etc…
Don't do PDAP query in custom and advanced mode (not fixed on dev version)
- Fix hub registration also on dev version !
Support queries on multiple targets? Mars#Venus could be translated as target_name= lower('Venus') OR target_name= lower('Mars') - To be investigated.
Re-organization of second page (Query results)
We currently have one box per service, which will become impractical when we have many services
- restrain to one-line description of services, with collapsed info panel (instead of current 4-lines box)
- only plot services providing answers in std mode (full answer page still required to check service behavior in admin mode). Alternatively, plot services with answers first.
- write acronym first in the box, then full title.
- add last modif date in box? Not the one from the registry though, it will never be updated (should be the most recent modification_date in service in v2 - where do we find this?)
- query box says "Generated WHERE clause of ADQL statement" but the complete statement is written
- Important: add provider institute logo in green boxes - can be found in registry declaration.
- There is something to fix in the access to the registry: was going in error until recently (1/2016) when field referenceURL was empty (service did not appear on Query results page). In VESPA, this is only used to provide a link to a possible web site associated to the service. This may have to be fixed also in the EPN-TAP validator.
Individual service page (Advanced query form)
- Restore Advance Query mode on dev version! No longer displays specific parameters from a service, even in v1 (24/5)
It may be useful to be able to retrieve the existing values of all parameters, or min/max values for numerical fields, TBC
- it would be useful if the “Advanced Query Form” could be open with the same query already entered by the user, to be refined.
- Make Advance Query button available from service result page, in fact!
Evolution of third page (Results from service)
We currently plot only the first 10-1000 results, but the VOtable transmitted from this page contains all results
- "Download selection" button is implemented (now also on production version) - provides a zip file with results. Check if this requires protection from funny requests.
- Fix round-off values after conversion - only keep a reasonable number of figures (e.g. in spectral range, not only)
- Add time out or message if pb (depending on size of retrieved data set?) - currently returns an error
- Pagination issue to fix (offset fct available in DaCHS, requires last version)?
- Put color in "Processing" box! (OK: now grey)
- Convert JD times to ISO string for readibility (time_min and time_max)
- Try removing limitation to 1000 results - seems doable, but requires new read at each page change (4-5 s on local machine, may be faster on server) - check if OK
- Add units in table header for numerical parameters (may depend on print options, e.g., for spectral quantities, otherwise from q.rd) - OK, but some gremlins
- Check if base64 decoding is time-consuming or not. Check if getting a VOtable formatted in ascii is faster.
- JSON output needed (for surfaces). TBC, seems implemented in DaCHS (&FORMAT=json in request). Additional formatting may be required (see with Chiara Marmo). Done in 2017?
- Add support for STC-S footprints (s_region) => new fct SAMP as footprint (to Aladin at least)? see how TapHandle does it.
- Cyril is working on a new way to select the parameters to be plotted, TBC
- Callback mechanism to be implemented (to refine query from subset of first results)?
- Cyril proposed to insert a small Query form on this page. This is half-way towards the MAST interface principle, therefore only an alternative solution - TBC only if problem with the preferred one.
- Need for a data merging function: add a "Merge and Send" function in menu "Data Selection" - would call a script to join tables based on first column, possibly with resampling (STILTS or CASSIS can resample). Most useful for spectra, requested by Grenada service.
- add a button to show / hide duplicates - i.e., granules with identical values for the parameters displayed on this page (this would allow filtering multiple observations of the same target in the TNO service, for instance). default = hide (suggestion from Florence)
Result VOTable should not be systematically named sync, should at least include an extension (settable in DaCHS? - Json output is called sync.json - if not, change in client?)
Upgrade help page
- Open Help page in new window
- Add button to unselect everything
Update support links
- Convert spectral range from Hz to µm and back
- Convert spectral resolution & sampling step from Hz to µm and back - involves 2 parameters, can be done with TAP request (see mail dated 1/8/2017) - TBC
- Add conversion to cm-1 for spectral range
- Add either info to data service (URL + schema) or link to this in Taphandle, on option (button or info panel)
- Check what happens when no link is provided in access_url - seems to give a link to another VESPA query (DynAstVO and HFC1AR) - but this is a possible situation
- Check why ds9 does not receive fits images from VESPA (2019: OK from HST_planeto, but HTML error in ds9 - seems to be on their side)
- Can we use a js9 based application to handle 2D spectra from APIS (instead of dying SpecView)?
- Implement support for datalink, similar to TAP handle
- Connect Planetary Cesium Viewer as well as Mizar to display bounding boxes - should even plot s_region in fact - this requires a CRS in the geojson output, TBD
- Make subgranule_url module (in CRISM service) work like in TAPhandle
- Support for multiple table resources: pass them one by one when SAMPing (required to support VizieR service)
Link Miriade the same way as Mizar: new button saying get ephemeris (near Footprints). Will open Miriade in a new browser page (or use web service?) and send target name & time for selected granules. Get not only positional ephemeris, but also physical ones (ie, disk center and sub solar point coordinates), or propose options. Retrieve as a VOTable and send to TOPCAT directly. (see mail 13/4/2017) — 2018: done through datalink/dlmeta e.g. in HST; this is a bit more work on provider side, but this is more flexible.
Integrated service answers
- Cyril Chauvin This is a major addition to the interface, as it will provide the ability to cross-correlate answers from various services (the second priority according to ANO5 committee in France, and also requested by EC).
After some discussion, the best way seems to build a common table merging answers from all services. This will be an additional box in the Query results page:
- the table can be limited to std EPN-TAP parameters (mandatory & optional) - everything included in 2019 test version
- do not use multiple table resources, this won't pass via SAMP - concatenate directly instead.
- will not include additional parameters from individual services (undefined in EPNcore), so that the number of columns is limited - everything included in 2019 test version
- a specific parameter will be added to identify the service of origin for each granule - in principle it can simply be the service name, but this relies on services compliancy TBC (ivoID would be better)
- this box will contain a single button "Samp VOtable", typically to pass the description of answers to TOPCAT for further examination?
- there will be no associated third page (displaying Results from all services)? TBC if demanding or not
- we could add a button in the Query form page to SAMP a VOtable of all results directly, instead of printing the intermediate Query results page (same level as "Submit" and "Reset")
- check whether coordinated observations require any special function. This must rely on v2 parameters target_time_min / max + target_name
This function may require to build a dynamic db merging all results - but workaround in 2019 test version
Another solution is to join the individual result tables with stilts, or even during the TAP query - done in the portal in 2019 test version (with astropy)
Use cases to consider for service interoperability:
1) Retrieve spectra of asteroids located 3-4 AU from the Sun
- This assumes the existence of a service providing orbital parameters of asteroids (now OK with MPC)
- Get distances from this, and use this parameter to filter results from the spectral service (e.g. M4ast)
- The only common key is target_name - but this still has to be made consistent!
2) Compare observations and simulations
- Send a query to SPICAM profiles service, grab profiles of interest
- Get the exact observational parameters (season, local time, location)
- Send another query using these parameters to a simulation service (MCD)
In both cases, this involves selecting granules of interest and 1 (or several) parameters in the result table, and sending a new query from VESPA to all services.
The second case is more complicated (several parameters) and may require a pipeline / TOPCAT.
We need to provide at least guidelines on how to do this with real examples.
Ideally, I'd like to click and select parameters in the SPICAM results page to retrieve their values and insert them in a new query.
1) Query all services with
dataproduct = profiles
location : as below
WHERE c1max >= 0.0 AND c1min <= 40.0 AND c2max >= 0.0 AND c2min <= 30.0 AND (dataproduct_type LIKE '%pr%') AND (lower(target_name)= lower('Mars') OR lower(target_name)= lower('4'))
+ add range in solar longitude and local time (not currently possible!)
2) display results from SPICAM and MCD
not currently possible: missing a button in All results to open a service results page in another window. Fixable in the interface ?
3) If selection is accurate enough:
send corresponding profiles to TOPCAT, plot together and compare
focus on SPICAM results, and enter magic mode.
Clicking Solar longitude would select both min /max values of current line and include them as range in a new query.
Idem for other parameters.
+ Select service #2 (here: MCD)
+ Issue query and plot results in another window
Alt: send new query to all services, plot all results together.