DAP4 Dataset Services Response: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 209: Line 209:
c.f. [[DAP4_Web_Services v3| DAP4 Web Services]]
c.f. [[DAP4_Web_Services v3| DAP4 Web Services]]


Here is an alternative response, made using a syntax of our own devising before looking into using Atom 1.0. The similarity between the two documents is impressive...
Here is an alternative response, made using a syntax of our own devising before looking into using Atom 1.0. The similarity between the two documents first suggested that we might find little cost in using Atom.


A DAP4 web service is described by:
A DAP4 web service is described by:

Revision as of 22:20, 6 July 2012


<-- back to OPULS Development

Author: ndp

Background

DAP2, while successful within several arenas in the scientific community, failed to provide a complete description of the web interface that a server should support. This came about for two reasons: DAP2 was developed at a time when 'web services' was an idea in considerable flux and, secondly, the DAP2 specification was written to focus on a data model (mostly) that supported interoperability. The current DAP4 draft specification document, like the DAP2 specification, does not address how a web-based interface to DAP4 should be built. However, we are going to provide a specification that will describe how such an interface for DAP4 should be built and this document contains information that should be part of that specification.

Problem addressed

Currently DAP servers provide a number of dataset level responses (services). These include DAP services such as the DDS, DAS, and DDX, plus other services that have been added over time such as the RDF and ISO responses. There is currently no way for a user (or software agent) to discover which of these many services might be available for a particular dataset. While THREDDS catalogs do provide a mechanism for determining which services are associated with which datasets, they don't actually provide a mechanism for discovering the details of the various service URLs. A THREDDS catalog might identify a DAP service, but the URL that can be assembled from the THREDDS catalog for that service is limited to just the base dataset URL. Typically dereferencing this URL will provide one of two things - either the underlying data file (for example a netcdf file) or an HTTP 403 error if direct access to the underlying files has been disabled in the server configuration.

Example:

This URL points to a dataset on a DAP server: 

http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc

Dereferencing the link will not connect you to a DAP service or DAP response. It will (likely) return the underlying netcdf file. The only way to know that DAP services exist for this is by reading the URL path and recognizing that it is probably an instance of a DAP server based on the position of the 'opendap' string in the URL. In fact, there is no guarantee that dereferencing this URL will even yield the underlying file, as that behavior is a configuration option for the server. If that access is not allowed in the configuration, then dereferencing the URL will simply return an HTTP 403 (forbidden) error.

Proposed solution

We intend to provide a 'Services' or 'Capabilities' response for DAP4. DAP dataset resource URLs should return an XML document describing the DAP services available for the dataset. This will be a required response and the only valid response for an 'unadorned' dataset resource URL. We will refer to this response as the 'Dataset Services Response' or 'services response' in the 'DAP4 Web Services' specification and the remainder of this document.

At minimum this document MUST provide, for each service available for the dataset:

  • A human readable title of that service (e.g., The DDX service for the dataset)
  • One or more links that can be dereferenced to get the various representations of the response for the dataset.
  • An unambiguous, unique to the service, xlink:role identifier that serves as a way to clearly identify what each service does.
  • A brief description of the service
  • A link to a complete, human readable, description of the service.

The response should also include:

  • A reference to an XSLT that a browser would use to render the description into HTML for a presentation view.
  • Descriptions and syntax of server side functions available for the dataset

Using Atom 1.0

The proposed encoding for this information is to use Atom 1.0. The advantages of this encoding are several and the detractions are close to nil. Atom 1.0 provides the DAP with a widely-used standard for representing information that is very general in nature (that data are accessible using a set of web services, each of which provides a sightly different kind of response and can be accessed using a different URL). Furthermore, we can use this same response in other responses a server might like to provide such as a data cast provides to subscribers or an OpenSearch result. Using Atom is important in those contexts because a response that it not specific to DAP is a requirement. At the same time, because the Atom specification does not impose needless or complex additional requirements, the cost of using it is low. In fact the only information in Atom 1.0 that is not listed above is a the name and email of an author for the entry. While this might be rationalization, including that information could help a number of users who send email to support lists asking about dataset X simply because they don't know where/who else to ask.

using Atom, A DAP4 web service is described by:

  • An 'entry' that serves to group a set of link elements that reference different services to a common set of information about the dataset. For the dataset, that information is:
    • A simple human readable name called 'title'.
    • An 'id' that uniquely identifies the 'entry'.
    • The last modified time for the dataset (updated).
    • A person responsible for the dataset (author).
    • An optional content element.
  • For each service, a link element is used where the four attributes title, rel, type, href are used as follows:
    • title A human-readable description of the service
    • rel See teh Atom spec
    • type A MIME-type that identifies the response as specific to OPeNDAP such as application/vnd.opendap.org.dap4.xdp+xml
    • href The full URL used to get the response

For the Atom-based response, we propose to use MIME types as Ethan suggested (application/vnd.opendap.org.dap4.data etc.) when describing services in the Dataset Services Response, and then, when the server provides those responses over HTTP it sets the content header to the lowest common mime type? For example, in the Dataset Services response we might describe the Dataset Metadata service like this: 

...
    <link title="DAP4 Dataset Metadata XML Document."
          rel="related"
          type="application/vnd.opendap.org.dap4.xdp+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp"/>
...

Yet, when a DAP4 server responds to a client asking for http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp the server would set the HTTP headers so that the Content-Type would be the lowest common mime type (text/xml) and then we could put the more expressive type in Content-Description: 

HTTP/1.1 200 OK
Date: Mon, 23 May 2005 22:38:34 GMT
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Content-Type: test/xml
Content-Description: application/vnd.opendap.org.dap4.xdp+xml; url=http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp
Content-Encoding: gzip
X-DAP: 4.0

For normative information on Atom, see http://tools.ietf.org/html/rfc5023 and http://tools.ietf.org/html/rfc4287.

Example: Validated Atom Document with one atom:entry for a dataset, and an atom:link for each service

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<entry xmlns="http://www.w3.org/2005/Atom">

    <title>DatasetServices: ECMWF_ERA-40_subset.ncml</title>

    <id>http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml</id>

    <updated>2003-12-13T18:30:02-05:00</updated>
    
    <author>
        <name>Nathan Potter</name>
        <email>johndoe@devnull.com</email>
    </author>
    <content type="xhtml" xml:lang="en" xml:base="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
        <div xmlns="http://www.w3.org/1999/xhtml">DAP Dataset Services for http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml</div>
    </content>

    <link title="Dataset Services Description"
          rel="self"
          type="application/atom+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml"/>
    
    <link title="DAP4 Dataset Metadata XML Document."
          rel="related"
          type="application/vnd.opendap.org.dap4.xdp+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp"/>
    
    <link title="DAP4 Data object."
          rel="enclosure"
          type="application/vnd.opendap.org.dap4.data" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap"/>

    <link title="OPeNDAP HTML Data Request Form for data sub-setting and access."
          rel="related"
          type="text/html" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.html" />

    <link title="DAP2 Data Object."
          rel="enclosure"
          type="application/vnd.opendap.org.dap2.data" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dods"/>
          
    <link title="DAP4 Data response in ASCII form."
          rel="enclosure"
          type="text/plain" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.asc"/>
          
    <link title="NetCDF-3 file-out response."
          rel="enclosure"
          type="application/x-netcdf" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.nc"/>
          
    <link title="XML Data Response"
          rel="enclosure"
          type="application/vnd.opendap.org.dap4.data+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdap"/>
          
    <link title="DAP2 DDX document"
          rel="related"
          type="application/vnd.opendap.org.dap2.ddx+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ddx"/>
          
    <link title="DAP2 DDS document"
          rel="related"
          type="application/vnd.opendap.org.dap2.dds" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dds"/>
          
    <link title="DAP2 DAS document"
          rel="related"
          type="application/vnd.opendap.org.dap2.dds" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.das"/>
          
    <link title="RDF representation of the Dataset Metadata response."
          rel="related"
          type="application/rdf+xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.rdf"/>
          
    <link title="DAP2 Dataset Information Page"
          rel="related"
          type="text/html" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.info"/>
          
    <link title="Server Version Document"
          rel="related"
          type="text/xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ver"/>
          
    <link title="ISO 19115 Metadata Representation of the Dataset (DDX) response."
          rel="related"
          type="text/xml" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.iso"/>
          
    <link title="ISO 19115 Metadata Representation conformance score for this dataset."
          rel="related"
          type="text/html" 
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.rubric"/>
          
    <link title="Dataset file access"
          rel="enclosure"
          type="text/xml"  
          href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.file"/>
          
</entry>

Rationale for the solution

The solution provides DAP4 servers with an interface that allows for both server-driven and agent driven content negotiation as described in [section 12 of the HTTP 1.1 specification]. Providing the Dataset Services Response in XML allows many downstream applications to ingest the content and leaves open the option for server implementers to add XSLT references to the document so that clients like browsers can easily build a presentation view of the information. Using the Atom 1.0 appears to involve no additional overhead and will enable other uses for the response.

Discussion

It is possible to embed in the above a reference to a XSLT that will transform the above XML into HTML similar to the following (with the caveat that this image is actually of a similar, but not identical, XML response):

ServiceDescriptionPrototype-01.png

Alternative encoding for the services response

c.f. DAP4 Web Services

Here is an alternative response, made using a syntax of our own devising before looking into using Atom 1.0. The similarity between the two documents first suggested that we might find little cost in using Atom.

A DAP4 web service is described by:

  • A simple human readable name called 'title'.
  • One or more access URLs and each with an associated media-type for the representation returned by that URL.
  • A unique namespace-like id that is used to define an xlink:role attribute. This is intended to allow people and software to unambiguously identify specific services, irrespective of their human readable title. The idea is that that the xlink:role tells you what is going to happen when you go to xlink:href.
  • An optional description.

A service is accessed by dereferencing its access URL, which is typically constructed by adding some type of suffix to the dataset's referent (aka base) URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.

A more extensive discussion of the various services that might appear in a DatasetServices document can be found on the DAP4 Web Services v3 page.

Example Response

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?xml-stylesheet type="text/xsl" href="/opendap/xsl/serviceDescription.xsl"?>
<DatasetServices 
    xmlns:xlink="http://www.w3.org/1999/xlink" 
    xml:base="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
    
    <Service title="Data Request Form" >
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.html" 
             xlink:role="http://services.opendap.org/dap4/data-request-form#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_HTML_DATA_Request_Form_Service">OPeNDAP HTML Data Request Form for data sub-setting and access.</Description>
    </Service>

    <Service title="Dataset">
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp" 
             xlink:role="http://services.opendap.org/dap4/dataset#">
        <link type="application/json" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdp.json" 
             xlink:role="http://services.opendap.org/dap4/dataset#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Service_-_The_metadata">DAP4 Dataset Description and Attribute XML Document.</Description>
    </Service>

    <Service title="DAP4 Data">
        <link type="multipart/mixed" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap" 
             xlink:role="http://services.opendap.org/dap4/data#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Data_Service">DAP4 Data object.</Description>
    </Service>

    <Service title="DAP2 Data">
        <link type="application/octet-stream" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dods"
             xlink:role="http://services.opendap.org/dap2/dods#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Data_Service">DAP2 Data Object.</Description>
    </Service>

    <Service title="ASCII Data">
        <link type="text/plain" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.asc"
             xlink:role="http://services.opendap.org/dap4/ascii#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_ASCII_Data_Service">The DAP4 Data response in ASCII form.</Description>
    </Service>

    <Service title="NetCDF-File">
        <link type="application/x-netcdf" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.nc"
             xlink:role="http://services.opendap.org/dap4/netcdf-3#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_NetCDF_File-out_Service">NetCDF file-out response.</Description>
    </Service>

    <Service title="XML Data Response">
         <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xdap"
             xlink:role="http://services.opendap.org/dap4/xml-data#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_XML_Data_Service">An XML document containing both the dataset's structural metadata along with data values.</Description>
    </Service>

    <Service title="DDX">
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ddx"
             xlink:role="http://services.opendap.org/dap2/ddx#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDX_Service">OPeNDAP Data Description and Attribute XML Document.</Description>
    </Service>

    <Service title="DDS">
        <link type="text/plain" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dds"
             xlink:role="http://services.opendap.org/dap2/dds#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDS_Service">OPeNDAP Data Description Structure.</Description>
    </Service>

    <Service title="DAS"
        <link type="text/plain" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.das"
             xlink:role="http://services.opendap.org/dap2/das#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DAS_Service">OPeNDAP Dataset Attribute Structure.</Description>
    </Service>

    <Service title="RDF">
        <link type="application/rdf+xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.rdf"
             xlink:role="http://services.opendap.org/dap4/rdf#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_RDF_Service">An RDF representation of the Dataset response (DDX) document.</Description>
    </Service>

    <Service title="INFO">
        <link type="text/html" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.info"
             xlink:role="http://services.opendap.org/dap2/info#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Info_Service">OPeNDAP Dataset Information Page.</Description>
    </Service>

    <Service title="Server Version">
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ver"
             xlink:role="http://services.opendap.org/dap4/version#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Server_Version_Service">An XML document containing information about the software version of the server..</Description>
    </Service>

    <Service title="ISO-19115">
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.iso"
             xlink:role="http://services.opendap.org/dap4/iso-19115-metadata#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_ISO_19115_Service">ISO 19115 Metadata Representation of the Dataset (DDX) response.</Description>
    </Service>

    <Service title="ISO-19115-Score">
        <link type="text/html" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.rubric"
             xlink:role="http://services.opendap.org/dap4/iso-19115-score#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_ISO_Conformance_Score_Service">ISO 19115 Metadata Representation conformance score for this dataset.</Description>
    </Service>

    <Service title="File Access"
        <link type="text/xml"  ## This value depends on the file being accessed.
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.file"
             xlink:role="http://services.opendap.org/dap4/file#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Native_File_Access_Service">Access to dataset file.</Description>
    </Service>

    <Service title="Dataset Services Description">
        <link type="text/xml" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml" 
             xlink:role="http://services.opendap.org/dap4/dataset#">
        <link type="application/json" 
             xlink:href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.json" 
             xlink:role="http://services.opendap.org/dap4/dataset#">
        <Description xlink:href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Services_Description_Service">An XML document itemizing the Services available for this dataset.</Description>
    </Service>

    <ServerSideFunctions>
        <Function name="geogrid"
          xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#geogrid">
            <Description>Allows a DAP Grid variable to be sub-sampled using georeferenced values.</Description>
        </Function>
        <Function name="grid"
          xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#grid">
            <Description>Allows a DAP Grid variable to be sub-sampled using the values of the coordinate axes.</Description>
        </Function>
        <Function name="linear_scale"
          xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#linear_scale">
            <Description>Applies a linear scale transform to the named variable.</Description>
        </Function>
        <Function name="version"
          xlink:href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#version">
            <Description>Returns version information for each server side function.</Description>
        </Function>
    </ServerSideFunctions>
</DatasetServices>


Retrieved from "https://docs.opendap.org/index.php?title=DAP4_Dataset_Services_Response&oldid=8795"