This is intended as a list of survival tricks for users who are particularly reluctant to using VO tools, but need to retrieve data from VO services in VESPA.
1- Querying data services
It is easy to see the content of a data service in VESPA.
In the VESPA query interface, the central panel displays a list of connected services with the number of entries in each service (if a query has already been submitted, click the "Reset" and "Submit" buttons in the left panel).
If you know which service you want to look at, click on the service name in this table. This will display the content of the service, paginated. You can change the number of entries per page, or navigate through pages. You can then select your data visually, possibly with the help of thumbnails.
In the left panel, you'll find it easy to simply add a filter on target_name, target_class, or dataproduct_type parameters, or even on time_min / time_max.
• We stress that, independently from VO access, EPN-TAP provides a convenient way to describe personal datasets with a standard set of keywords (axes and physical quantities). You may want to use it to describe your own, private, datasets.
2- Accessing data
Most data services provide links to data files. In EPN-TAP services, such links are available under the access_url parameter. To retrieve several files at once, you only need to click on selected lines and use the "Data Selection > Download" item from the menu on the bottom left; this will download a zip file.
In other cases, the data are included in the service table directly, together with the descriptive parameters.
Some services propose several versions of the data files, e.g. an ascii file and a conversion to VOtable. Beware that in some cases the ascii version is the native format available in an external archive, while the VOtable version may have a different, updated or corrected content (this is the case of IKS or spectro_planets). In other cases, the conversion from ascii is performed on the fly and the content is identical (this is the case of M4ast or Titan).
2.1 Attached files
Many data services link to files in VOtable or fits format, which are two VO standards. These files can be downloaded easily on your disk through the VESPA portal or TAPhandle (the VESPA portal can also send them directly to the appropriate VO tools).
- Reading fits files in your environment should be obvious in most cases - fits libraries are widely available (however, TOPCAT will only open tables, while ds9 will only open images and cubes - other extensions may require specific processing).
- A VOtable can be read and used in TOPCAT. If you prefer to use a more familiar environment, you can save them from TOPCAT in various formats, including simple ascii, fits, or csv which can be read in many software - or even html or latex for use in documents.
- Several services provide a link to a web service under access_url, in particular for OGC/GIS services (e.g., CRISM, USGS_maps, M3…). Such links can be send from VESPA to QGIS through the EPN-TAP plug-in (jra-t4-tools-qgis-plugin).
2.2 Data included in the service table
The result of a query (as displayed in the VESPA portal) is also formatted as a VOTable, which can be read in TOPCAT and from there can be stored in ascii or other formats. The query can be issued directly from TOPCAT using its TAP interface (which is less user-friendly than the VESPA portal). Converting the complete result table will also provide access to the descriptive parameters.
VESPA's Google Sheet Add-on may also be handy in this case, see here: TAP Sheet (Google Sheets Add-On).
Finally, if you know on which TAP server the service is located, you can query a table directly from the command line and select the output format, e.g.:
wget "http://voparis-tap-planeto.obspm.fr/tap/sync?REQUEST=doQuery&LANG=ADQL&QUERY=SELECT * FROM exoplanet.epn_core WHERE "star_name" LIKE '%GJ 357%' &FORMAT=json"
(will retrieve description of exoplanets around star GJ 357. We're passing a TAP query via wget, which contains an ADQL query after the QUERY parameter. This mechanism applies to all EPN-TAP services. Available output formats depend on the server).
3- IDL
The most helpful IDL libraries in this context are NASA's Astron library, and possibly SSW.
The SSW (SolarSoftWare) library provides a complete VO interface, including queries to the registry and to individual services declared there (should work under Linux): http://www.lmsal.com/solarsoft/ssw_setup.html.
However, this library is not maintained and is partly obsolete, and may not work properly.
3.1 Fits files
Several fits routines are available in the Astron library: readfits is the simplest one to use, and handles many fits variations. In case of special formatting (e.g., multiple extensions), mrdfits may be required to access the data.
Corresponding routines are available to write fits files after modification: writefits and mwrfits (see their doc).
3.2 VOtable
Several routines or libraries handling VOtable exist, but they do not all comply with the standard.
The VESPA-provided read_vot routine reads all VOtable using the stilts library associated to TOPCAT (will work uner Linux and Mac, and with limitations under Windows): https://github.com/epn-vespa/IDL_VOtable
An equivalent write_vot routine exists but has limitations (e.g., it will not provide a fully informed FIELD description) - see routine headers for other uses (updates, etc).
The VOtable writer routine from the SSW library has been used with success to feed TOPCAT with data processed in IDL. This is particularly handy to write tables of spatially sparse observations, that TOPCAT can then integrate in Healpix maps much more efficiently than IDL (see here: 1- Various use cases on planetary surfaces with Aladin & TOPCAT).
3.3 PDS files
Some services may provide links to PDS3 files (e.g., VVEX). Those are read under IDL with the VESPA-provided library LecturePDS (https://github.com/epn-vespa/LecturePDS), or with the PDS Small Bodies Node library readpds: https://pdssbn.astro.umd.edu/tools/tools_readPDS.shtml
No multipurpose library exist in other environments, however ad-hoc routines may be available in other languages for specific datasets.
The SBN also provides a routine to read PDS4 data under IDL (no such files currently exist in EPN-TAP services).
3.4 GIS/OGC formats
IDL has WMS and WCS interfaces which allow it to browse the GIS servers underneath these EPN-TAP services (type ogc_wcs or ogc_wms in the IDL command line).
However, a direct call of the file url is OK and much simpler, especially if you've already selected files from the VESPA interface:
; from the EPN-TAP table displayed in VESPA
access_url= "http://access.planetserver.eu:8080/rasdaman/ows?&SERVICE=WCS&VERSION=2.0.1&REQUEST=GetCoverage&COVERAGEID=hrl000052ff_07_if155l_trr3&FORMAT=image/tiff"
; will write file in working directory (see options to IDLnetURL Get method for anything more elaborated)
oUrl = OBJ_NEW('IDLnetUrl')
pipo=oUrl->Get(URL=access_url)
help, pipo
PIPO STRING = '/YourPath/CRISM/idl.dat'
iopen, pipo, popo
help, popo
POPO FLOAT = Array[438, 60, 28]
tvscl, popo(200, *,*) >0.25<0.55
OBJ_DESTROY, oUrl
3.5 Locating files inside an EPN-TAP service
Similarly, if you already know the name of the files of interest, use VESPA's file grabbing web services with the IDLnetURL object. Usage from IDL is described here: File grabbing interface (fgi)
This type of access is intended for easy pipeline processing.
3.6 Access to the registry of services
The SSW library provides a complete VO interface under Linux, including queries to the registry and to individual services.
However, this library is partly obsolete and may not work properly.
4- Python
4.1 VOtable & fits files
Routines exist to handle these two formats in the astropy library: https://www.astropy.org
4.2 PDS files
The PDS Small Bodies Node also provides routines to read PDS4 data under Python (no such files currently exist in EPN-TAP services): https://pdssbn.astro.umd.edu/tools/tools_readPDS.shtml
4.3 Locating files inside an EPN-TAP service
The file grabbing web services from VESPA (and their use from Python) are described here: File grabbing interface (fgi)
This type of access is intended for easy pipeline processing.
4.4 Querying an EPN-TAP service via Python
Astropy + pyvo libraries cover some VO access: https://pyvo.readthedocs.io/
A short tutorial is available here (from Baptiste Cecconi):
https://github.com/epn-vespa/tutorials/blob/master/misc/pythonic-access-tutorial/pythonic-access-tutorial.md
An example of query from Jupyter notebook is provided here (from Baptiste Cecconi):
Direct query via pyvo (from Florence Henry):
In [1]: import pyvo
In [2]: service = pyvo.dal.TAPService("http://voparis-tap-planeto.obspm.fr/tap")
In [3]: query = "SELECT target_name, inclination, eccentricity FROM exoplanet.epn_core WHERE semi_major_axis < 5"
In [4]: results = service.search(query)
In [5]: results
Out[5]:
<Table masked=True length=2050>
...
In [6]: type(results)
Out[6]: pyvo.dal.tap.TAPResults
In [7]: len(results)
Out[7]: 2050
In [8]: results[0].get('target_name')
Out[8]: '11 Com b'
In [9]: results[2].get('target_name')
Out[9]: '14 And b'