List of EPN-TAP parameters
v2, 3/8/2015 (SE)
v3, 29/8/2015 (SE, integrating comments by PLS and BC)
v4, 7/9/2015 (SE, integrating comments by PLS)
v5, 7/10/2015 (SE, integrating comments by PLS: time_scale UCD)
v6, 7/11/2015 (SE, integrating comments by PLS/BC)
v7, 14/12/2015 (SE, added point 8, clarification of an older idea)
v8, 24/12/2015 (SE, review/integration of comments/corrections)
v9, 5-7/01/2016 (SE/BC, including comments from Chiara Marmo and Michel Gangloff)
v10, 21/02/2016 (SE, from model implementation of IKS)
v11, 11/5/2016 (SE, systematic comparison with ObsCore again & variations for spatial coordinates)
New in v2
- The "laundry list" format makes services easier to design and to query
- Allows grouping of results from several services at once
- Supports multispacecraft observations
- Speeds up mirroring of services (support for partial updates)
- Better support of footprints, and better interface with GIS
Main evolution relative to v1
1) The previous notion of dataset is deprecated
This was complex to handle in the database, and in general not relevant for the services.
2) Grouping of products is rationalized in version 2
A granule is still a record/line in the epn_core view database, and corresponds to the smallest data unit described by the service.
A "product" is typically a data file, or a service output, that can be reached through a URL
In version 2 both concepts coincide, while in v1 the same granule could be composed of several products related to the same initial data (an "observation")
In EPN-TAP v2, granules are referenced by 3 parameters:
- granule_uid provides a unique ID for the granule in the service, ie for each line in the epn_core view. It is equivalent to the previous index parameter (index is a reserved word in many database languages and should not have been used in the first place)
- granule_gid is related to a type of product: it is identical for all granules containing the same type of information for different observations (e.g., calibrated files). An explicit string is recommanded in this field, with suggested standard values (e.g., preview/native/calibrated/geometry/projection…).
- obs_id is related to an observation: it is identical for all granules related to the same observation, containing different type of data (e.g.: raw and calibrated data, associated geometry, etc). In many EPN-TAP v1 services, such products were described together on the same line with a unique index parameter.
These 3 parameters can be arbitrary alphanumeric strings — see example application to APIS service below.
In practice, different products related to the same observation are no longer described together on a single line of the epn_core view, but on successive lines associated by the same obs_id, each with a different granule_gid (and a specific granule_uid, see Table 2 below).
Each line in the epn_core view must describe only one product (plus a thumbnail wherever relevant). The notion of "main product" (which was more or less explicit in v1) is therefore deprecated, and the epn_core view in v2 includes more lines than in v1. Although less compact that the previous table presentation, this list presentation is much more efficient for machine-handling, and easier to design.
3) The notion of table/service parameters is deprecated
Such parameters were not directly accessible by TAP. In v2, constant parameters must be replicated in every line of the epn_core view.
Such parameters may be duplicated in the registry declaration though, so as to provide a fast description of services.
4) Footprints can be provided through s_region
s_region is a parameter in the ObsCore standard of IVOA, and ADQL allows for powerful query functions such as intersections. s_region introduces an STC-S string providing description of the observed area on a sphere.
We have to study a possible extension of its use from regions of the sky to body-related coordinates (either on ellipsoids or 3D shape models) .
The use in the context of GIS also has to be studied.
5) Better support of evolving services
creation_date and modification_date are now mandatory parameters for every granule. The latter is intended to optimize mirroring of services, by identifying the granules to be updated/copied.
These parameters must be provided as ISO 8601 strings with format "2013-11-17T10:41:00.00+01:00" (with no space) where the indication of time zone is mandatory. The associated data type must be "date". ADQL supports this format and filtering using time zone is possible.
6) Support of coordinated observations
The target_time parameter now provides observation time in the reference frame of the target. This is intended to facilitate the cross-correlation of observations from different locations, e.g., telescopic observations in support of space missions, or multi-spacecraft campaigns.
7) Axes ranges
All parameters defining a range are now introduced with a min and max value.
All floating point parameters are now in double precision to prevent errors.
8) Thematic extensions
Some science fields will require optional parameters, which need to be used consistently between services addressing the same field. Such extensions have to be designed by sub-groups involved in the corresponding data services, either as providers or users. This includes:
- Lab spectroscopy: parameters to describe mineralogical samples (and possibly other samples)
- Orbital/rotational parameters and physical properties of Solar System bodies (TBC, but requested by some)
- Results of planetary 3D modelling run
- Exoplanets / planetary systems properties
- Bibliographic entries? May be manageable otherwise, through bibcode / doi interpretation
9) Parameters which must be provided
are now clearly identified - those are not only mandatory, they also must provide a value.
They are mostly related to service description and granule identification.
+ See notes below the table.
Name, v2 | Must be filled? | Name, v1 | SQL type | Unit / Format | Description | UCD Please comment! | UCD in Obscore 1.1 (30/3/2016 version) | Comments |
|---|---|---|---|---|---|---|---|---|
EPNCore mandatory parameters |
|
|
|
| Current value | — ? : closest sense _ : N/A | ||
granule_uid | Y | index | Text |
| Internal table row index | meta.id meta.id;meta.dataset | meta.id | |
granule_gid | Y | Text |
| Common to granules of same type (e.g. same map projection, or geometry data products). Can be alphanum. | meta.id | meta.id | ||
obs_id | Y | Text |
| Associates granules derived from the same data (e.g. various representations / processing levels). Can be alphanum., may be the ID of original observation. | meta.id meta.id;obs | meta.id | ||
resource_type | Text |
| Can be dataset or granule | (meta.code.class) | _ | |||
dataset_id | Text |
| Dataset identification & granule reference | (meta.id) | _ | |||
dataproduct_type | Y | dataproduct_type | Text |
| Organization of the data product, from enumerated list | meta.code.class | meta.id | |
target_name | Y | target_name | Text |
| Standard IAU name of target (from a list related to target class), case sensitive | meta.id;src | meta.id;src | |
target_class | Y | target_class | Text |
| Type of target, from enumerated list | meta.code.class;src src.class | src.class | |
time_min | time_min | Double | d (date as JD) | Acquisition start time (in JD). UTC measured at time_origin location (default is observer's frame) | time.start time.start;obs | time.start;obs.exposure | ||
time_max | time_max | Double | d (date as JD) | Acquisition stop time (in JD). UTC measured at time_origin location (default is observer's frame) | time.end time.end;obs | time.end;obs.exposure | ||
time_sampling_step_min | time_sampling_step_min | Double | s | Min time sampling step | time.interval;stat.min | time.resolution — ? | ||
time_sampling_step_max | time_sampling_step_max | Double | s | Max time sampling step | time.interval;stat.max | |||
time_exp_min | time_exp_min | Double | s | Min integration time | time.duration;obs.exposure;stat.min | time.duration;obs.exposure | ||
time_exp_max | time_exp_max | Double | s | Max integration time | time.duration;obs.exposure;stat.max | |||
spectral_range_min | spectral_range_min | Double | Hz | Min spectral range (frequency) | em.freq;stat.min | em.wl;stat.min (always as wl) | Always as freq | |
spectral_range_max | spectral_range_max | Double | Hz | Max spectral range (frequency) | em.freq;stat.max | em.wl;stat.max | ||
spectral_sampling_step_min | spectral_sampling_step_min | Double | Hz | Min spectral sampling step | em.freq.step;stat.min | meta.number | ||
spectral_sampling_step_max | spectral_sampling_step_max | Double | Hz | Max spectral sampling step | em.freq.step;stat.max | meta.number | ||
spectral_resolution_min | spectral_resolution_min | Double | Hz | Min spectral resolution | spect.resolution;stat.min | spect.resolution | ||
spectral_resolution_max | spectral_resolution_max | Double | Hz | Max spectral resolution | spect.resolution;stat.max | |||
c1min | c1min | Double | (1) | Min of first coordinate | pos;stat.min pos.angDistance;stat.min pos.eq.ra for celestial pos.bodyrc.lon for body pos.cartesian.x for cartesian pos.healpix for healpix (with 2 coordinates ?) | pos.eq.ra | ||
c1max | c1max | Double | (1) | Max of first coordinate | pos;stat.max | |||
c2min | c2min | Double | (1) | Min of second coordinate | pos;stat.min pos.angDistance;stat.min pos.eq.dec for celestial pos.bodyrc.lat for body pos.cartesian.y for cartesian | pos.eq.dec | ||
c2max | c2max | Double | (1) | Max of second coordinate | pos;stat.max | |||
c3min | c3min | Double | (1) | Min of third coordinate | pos;stat.min pos.distance;pos.bodyrc pos.distance for celestial pos.bodyrc.alt for body ? (from surface only, implicitly from reference level) pos.cartesian.z for cartesian | |||
c3max | c3max | Double | (1) | Max of third coordinate | pos;stat.max | |||
s_region |
| Text | (STC-S String) | ObsCore-like footprint, assume spatial_coordinate_description | instr.fov phys.outline;obs.field | phys.outline;obs.field | Current UCD doubtful. Do we need another param for GIS interface? | |
c1_resol_min | c1_resol_min | Double | (1) | Min resolution in first coordinate | pos.resolution;stat.min pos.angResolution;stat.min | pos.angResolution;stat.min | (UCD is deprecated, but needed!) | |
c1_resol_max | c1_resol_max | Double | (1) | Max resolution in first coordinate | pos.resolution;stat.max | pos.angResolution;stat.max | (UCD is deprecated, but needed!) | |
c2_resol_min | c2_resol_min | Double | (1) | Min resolution in second coordinate | pos.resolution;stat.min | (UCD is deprecated, but needed!) | ||
c2_resol_max | c2_resol_max | Double | (1) | Max resolution in second coordinate | pos.resolution;stat.max | (UCD is deprecated, but needed!) | ||
c3_resol_min | c3_resol_min | Double | (1) | Min resolution in third coordinate | pos.resolution;stat.min | (UCD is deprecated, but needed!) | ||
c3_resol_max | c3_resol_max | Double | (1) | Max resolution in third coordinate | pos.resolution;stat.max | (UCD is deprecated, but needed!) | ||
spatial_frame_type | spatial_frame_type | Text | (1) | Flavor of coordinate system, defines the nature of coordinates. From enumerated list | meta.code.class;pos.frame | _ | ||
incidence_min | incidence_min | Double | deg | Min incidence angle (solar zenithal angle) | pos.posAng;stat.min pos.incidenceAng;stat.min | _ | ||
incidence_max | incidence_max | Double | deg | Max incidence angle (solar zenithal angle) | pos.posAng;stat.max pos.incidenceAng;stat.max | _ | ||
emergence_min | emergence_min | Double | deg | Min emergence angle | pos.posAng;stat.min pos.emergenceAng;stat.min | _ | ||
emergence_max | emergence_max | Double | deg | Max emergence angle | pos.posAng;stat.max pos.emergenceAng;stat.max | _ | ||
phase_min | phase_min | Double | deg | Min phase angle | pos.phaseAng;stat.min | |||
phase_max | phase_max | Double | deg | Max phase angle | pos.phaseAng;stat.max | |||
instrument_host_name | instrument_host_name | Text |
| Standard name of the observatory or spacecraft | meta.id;instr.obsty | meta.id;instr.tel | ||
instrument_name | instrument_name | Text |
| Standard name of instrument | meta.id;instr | meta.id;instr | ||
measurement_type | measurement_type | Text |
| UCD(s) defining the data | meta.ucd | meta.ucd | ||
processing_level | processing_level | Integer |
| CODMAC calibration level in v1 | meta.code;obs.calib meta.calibLevel | meta.code;obs.calib | To be replaced by PDS4 values in v2? | |
creation_date | Y | Date | (ISO-8601 String) | Date of first entry of this granule | time.creation | time;meta.dataset | ||
modification_date | Y | Date | (ISO-8601 String) | Date of last modification (used to handle mirroring) | time.update time.process | |||
release_date | Y | Date | (ISO-8601 String) | Start of public access period | time.release | time.release | ||
service_title | Y | service_title | Text |
| Title of resource (will be used to handle multiservice results) | meta.title | ||
Optional parameters |
|
|
|
|
| |||
access_url | access_url | Text |
| URL of the data file, case sensitive. If present, next 2 parameters must be present. | meta.ref.url;meta.file | meta.ref.url | Use this to link data! | |
access_format | access_format | Text | (mime type in lowercase) | File format type | meta.code.mime | meta.code.mime | ||
access_estsize | access_estsize | Integer | kbyte | Estimate file size in kbyte (with this spelling) | phys.size;meta.file | phys.size;meta.file | ||
| data_access_url | Text | If access_format indicates a detached label, this parameter is mandatory and points to the corresponding data file - both will be handled by the client before samping it to tools or downloading- TBC | meta.ref.url;meta.file | Must be filled? being questioned! | ||||
| access_md5 | Text | MD5 Hash for the file when available (real file) | meta.checksum;meta.file | |||||
| preview_url | Integer |
| URL of a preview image (std format with adequate resolution for user's purpose) | (meta.ref.url;meta.file) | replaced by another granule group | ||
| native_access_url | Text |
| URL of the data file in native form, case sensitive | (meta.ref.url;meta.file) | replaced by another granule group | ||
| native_access_format | Text |
| File format type in native form | (meta.id;class) | replaced by another granule group | ||
thumbnail_url | Text |
| URL of a thumbnail image with predefined size (png ~200 pix, for use in a client only) | meta.ref.url;meta.file meta.ref.url;meta.preview | ||||
file_name | file_name | Text |
| Name of the data file only, case sensitive | meta.id;meta.file | meta.title;obs — ? | ObsCore obs_title is for a short free text string describing the granule. Do we want this? | |
species | species | Text |
| Identifies a chemical species, case sensitive | meta.id;phys.atmol | |||
| alt_target_name | Text | Provides alternative target name if more common (e.g. comets) | meta.id;src | |||||
target_region | target_region | Text |
| Type of region of interest | meta.id;class obs.field | |||
feature_name | element_name | Text |
| Secondary name | meta.id;pos obs.field | |||
bib_reference | reference | Text |
| Bibcode, doi, or other biblio id, URL… | meta.bib meta.bib.bibcode (if bibcode) | meta.bib.bibcode (always as bibcode) | ||
ra | ra | Double | deg or h:m:s? | Right ascension (not hour angle…) | pos.eq.ra;meta.main | |||
dec | dec | Double | deg | Declination | pos.eq.dec;meta.main | |||
solar_longitude_min | solar_longitude | Double | deg | Min Solar longitude Ls (location on orbit / season) | pos.posangle pos.posAng;pos.heliocentric;stat.min | |||
solar_longitude_max | solar_longitude | Double | deg | Max Solar longitude Ls (location on orbit / season) | pos.posangle pos.posAng;pos.heliocentric;stat.max | |||
local_time_min | local_time_min | Double | h | Local time at observed region | time.phase;stat.min time.period.rotation;time.phase;stat.min | |||
local_time_max | local_time_max | Double | h | Local time at observed region | time.phase;stat.max time.period.rotation;time.phase;stat.max | |||
target_distance_min | target_distance | Double | km | Observer-target distance | pos.distance;stat.min | was target_distance without min/max in v1 | ||
| target_distance_max | Double | km | Observer-target distance | pos.distance;stat.max | ||||
target_time_min | Double | d | Observing time in target frame (simplest way to look for coordinated observations) | time.start | ?? | |||
target_time_max | Double | d | time.end | ?? | ||||
particle_spectral_type | particle_spectral_type | Text |
|
| Use phys.particle? | |||
particle_spectral_range_min | particle_spectral_range_min | Double |
|
| Use phys.particle? | |||
particle_spectral_range_max | particle_spectral_range_max | Double |
|
| Use phys.particle? | |||
particle_spectral_sampling_step_min | particle_spectral_sampling_step_min | Double |
|
| Use phys.particle? | |||
particle_spectral_sampling_step_max | particle_spectral_sampling_step_max | Double |
|
| Use phys.particle? | |||
particle_spectral_resolution_min | particle_spectral_resolution_min | Double |
|
| spect.resolution;stat.min Use phys.particle? | |||
particle_spectral_resolution_max | particle_spectral_resolution_max | Double |
|
| spect.resolution;stat.max Use phys.particle? | |||
More optional (TBC) | were "Relative to service" (Table header) |
|
|
|
| |||
publisher | publisher | Text |
| Resource publisher | meta.name meta.curation | meta.ref.url;meta.curation — ? | ||
spatial_coordinate_description | spatial_coordinate_description | Text |
| ID of specific coordinate system and version | meta.code.class;pos.frame | still TBD. Discussion in progress here: EPN-TAP v2: Current discussion topic | ||
spatial_origin | spatial_origin | Text |
| Defines the frame origin | meta.ref;pos.frame | |||
time_origin | time_origin | Text |
| Defines where the time is measured (e. g., ground vs spacecraft) [target_time is of course always on target]. | meta.ref;time.scale | TBC | ||
time_scale |
| Text |
| Always UTC in data services (may be relaxed in computational services such as ephemeris) - from enumerated list | time.scale |
(1): depending on context (as given by spatial_frame_type). Please comment here: EPN-TAP v2: Current discussion topic
Beware that datatypes apply to the epn_core view, not to the q.rd file where they can be different
Example table:
File name-type | granule_uid | granule_gid | obs_id |
A-Raw | 1 | native | A |
A-Calib | 2 | calibrated | A |
A-geom | 3 | geometry | A |
A-proj | 4 | projected | A |
B-Raw | 5 | native | B |
B-Calib | 6 | calibrated | B |
B-geom | 7 | geometry | B |
B-proj | 8 | projected | B |
Other modifications, to be reviewed
• string delimiter for enumerated lists = #first entry#second entry#…#last entry#
TBC with PDS, exoplanets, NSSDC… (requires a character which never appears in target names, etc) but seems to be the best solution
• Defaut values :
NULL/void: will never return an answer to a query using this parameter (TBC, seems ADQL-related. To be corrected if it is a limitation of the client)
For float / double : -inf for *_min +inf for *_max – still TBC (NaN won't do). To be tested on a real case.
For strings: not needed? (i.e. NULL/void is OK?)
• UCDs: to be reviewed against PDS4 and IPDA, and completed
• Processing levels: to be reviewed against PDS4 (again)
• min vs max :
if only one value available, it must appear in both fields
• Optional parameters: they come in sets that are logically related; if one is present, the related ones must be present also (e.g., 3 access_* parameters)
• Granule_gid : any general indication to providers? I.e.: preview, native, calibrated, geometry…
A client should be able to display the values from a service, TBC
• access_format: do we really use mime-types?
• Reshuffle previous "service parameters":
- Mandatory :
processing_level -mandatory
service_title –mandatory
publisher -mandatory??? - Optional - TBC
spatial_coordinate_description (default = body-fixed or J2000)
spatial_origin (default = body center or SS barycenter? Or observer location)
time_origin (default = observer)
time_scale (default = UTC – no other values allowed in data services? [only in computational services, e.g. ephemeris])
Same values to be used in registry declaration
• Support for PDS3 detached labels (proposal)
access_format = "PDS3label" (it has to be here: we need to know that there is a detached label)
then access_url points to the label, and data_access_url points to the file (param is mandatory in this case - although the data file name is in the label, it can be in another directory)
A script can then recover both files and do something with them. This mechanism can be extended to other formats with detached labels (ENVI…).
Example of V1 to V2 conversion with APIS database:
EPNcore Table v1
(was not really compliant…)
| index | resource_type | dataset_id | access_url | access_format | preview_url | native_access_url | native_access_format |
|---|---|---|---|---|---|---|---|
| 23801 | granule | original_data | o5g202x4q_x2d.jpg | jpg | o5g202x4q_x2d_small.jpg | fits | |
| 23802 | granule | processed_data | o5g202x4q_proc.jpg | jpg | o5g202x4q_proc_small.jpg | o5g202x4q_proc.fits | fits |
| 23803 | granule | cylindric_proj | o5g202x4q_cyl.jpg | jpg | o5g202x4q_cyl_small.jpg | ||
| 23804 | granule | polar_proj_north | o5g202x4q_pol_n.jpg | jpg | o5g202x4q_pol_n_small.jpg | ||
| 23805 | granule | polar_proj_south | o5g202x4q_pol_s.jpg | jpg | o5g202x4q_pol_s_small.jpg |
EPNcore Table v2
| granule_uid | granule_gid | obs_id | access_url | access_format | thumbnail_url |
|---|---|---|---|---|---|
| o5g202x4q_x2d | original_data | o5g202x4q | o5g202x4q_x2d.fits | image/fits | o5g202x4q_x2d_small.jpg |
| o5g202x4q_x2d_prev | original_data_preview | o5g202x4q | o5g202x4q_x2d.jpg | image/jpg | o5g202x4q_x2d_small.jpg |
| o5g202x4q_proc | processed_data | o5g202x4q | o5g202x4q_proc.fits | image/fits | o5g202x4q_proc_small.jpg |
| o5g202x4q_proc_prev | processed_data_preview | o5g202x4q | o5g202x4q_proc.jpg | image/jpg | o5g202x4q_proc_small.jpg |
| o5g202x4q_cyl | cylindrical_projection | o5g202x4q | o5g202x4q_cyl.jpg | image/jpg | o5g202x4q_cyl_small.jpg |
| o5g202x4q_pol_n | polar_projection_north | o5g202x4q | o5g202x4q_pol_n.jpg | image/jpg | o5g202x4q_pol_n_small.jpg |
| o5g202x4q_pol_s | polar_projection_south | o5g202x4q | o5g202x4q_pol_s.jpg | image/jpg | o5g202x4q_pol_s_small.jpg |