DAP4 Dataset Services Response

<-- back to OPULS Development

Author: ndp

Background

DAP2, while successful within several arenas in the scientific community, failed to provide a complete description of the web interface that a server should support. This came about for two reasons: DAP2 was developed at a time when 'web services' was an idea in considerable flux and, secondly, the DAP2 specification was written to focus on a data model (mostly) that supported interoperability. The current DAP4 draft specification document, like the DAP2 specification, does not address how a web-based interface to DAP4 should be built. However, we are going to provide a specification that will describe how such an interface for DAP4 should be built and this document contains information that should be part of that specification.

Problem addressed

Currently DAP servers provide a number of dataset level responses (services). These include DAP services such as the DDS, DAS, and DDX, plus other services that have been added over time such as the RDF and ISO responses. There is currently no way for a user (or software agent) to discover which of these many services might be available for a particular dataset. While THREDDS catalogs do provide a mechanism for determining which services are associated with which datasets, they don't actually provide a mechanism for discovering the details of the various service URLs. A THREDDS catalog might identify a DAP service, but the URL that can be assembled from the THREDDS catalog for that service is limited to just the base dataset URL. Typically dereferencing this URL will provide one of two things - either the underlying data file (for example a netcdf file) or an HTTP 403 error if direct access to the underlying files has been disabled in the server configuration.

Example:

This URL points to a dataset on a DAP server:

http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc

Dereferencing the link will not connect you to a DAP service or DAP response. It will (likely) return the underlying netcdf file. The only way to know that DAP services exist for this is by reading the URL path and recognizing that it is probably an instance of a DAP server based on the position of the 'opendap' string in the URL. In fact, there is no guarantee that dereferencing this URL will even yield the underlying file, as that behavior is a configuration option for the server. If that access is not allowed in the configuration, then dereferencing the URL will simply return an HTTP 403 (forbidden) error.

Proposed solution

We intend to provide a 'Services' or 'Capabilities' response for DAP4. Dereferencing an unadorned DAP4 dataset resource URL will return a document describing the DAP services available for the dataset. This will be a required response and the only valid response for an 'unadorned' dataset resource URL. We will refer to this response as the 'Dataset Services Response' or 'DSR' in the 'DAP4 Web Services' specification and the remainder of this document.

At minimum this document MUST provide, for each service available for the dataset:

• A human readable title of that service (e.g., The DDX service for the dataset)
• One or more links that can be dereferenced to get the various representations of the response for the dataset.
• An unambiguous, unique to the service type, resource role ID that serves as a way to clearly identify what each service does.
• A brief description of the service

The response should also include:

• A link to a complete, human readable, description of the service.
• A reference to an XSLT that a browser would use to render the description into HTML for a presentation view.
• Descriptions and syntax of server side functions available for the dataset

Note: If a client understands how to modify/augment the dataset resource URL ( as described in the DAP4 Web Services Specification ) such that it becomes a DAP response URL then it need never obtain the Dataset Services response.

Proposed encoding for the services response

c.f. DAP4 Web Services

A DAP4 web service is described by:

• Attribute title whose value is a simple human readable string called
• An attribute called role whose value is a unique to the service type, resource role ID (a URI) that serves as a way to clearly identify what each service does. This is intended to allow people and software to unambiguously identify specific services, irrespective of their human readable title. The idea is that that the role attribute tells you what is going to happen when you dereference the URL held in the link element.
• One or more link elements containing:
  • An attribute called href whose value is a URL that when dereferenced will return the service response in the media type described by the value of the type attribute.
  • An attribute called type whose value is the media-type for the representation returned by that URL held in the href attribute.
  • An optional attribute called description whose value is a human readable string that provides a brief description of the link elements semantics.
• An optional Description element that contains (if present):
  • An optional attribute called 'href that contains a URL that points to a human readable document describing the service.
  • The text content of the element MUST be a human readable description of the service.

A service is accessed by dereferencing one of the access URLs held in the href attribute of a link elements, which is typically constructed by adding some type of suffix to the dataset's referent (aka base) URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.

A more extensive discussion of the various services that might appear in a DatasetServices document can be found on the DAP4 Web Services v3 page.

Example Response

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?xml-stylesheet type="text/xsl" href="/opendap/xsl/serviceDescription.xsl"?>
<DatasetServices xmlns="http://xml.opendap.org/ns/DAP/4.0/dataset-services#"
    xml:base="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
    
    <Service title="DAP4 Dataset Services Response" role="http://services.opendap.org/dap4/dataset-services">
        <link description="Normative form of the DSR" 
              type="application/vnd.opendap.org.dataset-services+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
        <link description="HTML representation of the DSR"  
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.html">
        <link description="Normative DSR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xml">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Services_Description_Service">An index of the Services available for this data resource.</Description>
    </Service>


    <Service title="DAP4 Dataset Metadata Response"  role="http://services.opendap.org/dap4/dataset-metadata">
        <link description="Normative form of the DMR" 
              type="application/vnd.org.opendap.dap4.dataset-metadata+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr">
        <link description="Data Request Form" 
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.html">
        <link description="Normative DMR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.xml">
        <link description="RDF representation of DMR" 
              type="application/rdf+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.rdf">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Service_-_The_metadata">The DAP4 metadata content for this data resource..</Description>
    </Service>


    <Service title="DAP4 Data Response" role="http://services.opendap.org/dap4/data">
        <link type="application/vnd.org.opendap.dap4.data"  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap" >
        <link type="text/plain"  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.ascii" >
        <link type="text/xml"   
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.xml" >
        <link type="application/x-netcdf"   
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.nc" >
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Data_Service">DAP4 Data object for this data resource.</Description>
    </Service>
    
    
    <Service title="ISO-19115 Metadata Service" role="http://services.opendap.org/dap4/iso-19115">
        <link description="Dataset metadata as ISO-19115" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.iso">
        <link description="ISO-19115 score" 
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.rubric">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_ISO_Conformance_Score_Service">ISO-19115 Metadata Representation of the DMR.</Description>
    </Service>



    <Service title="DAP2 Data" role="http://services.opendap.org/dap2/dods">
        <link type="application/octet-stream" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dods">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Data_Service">DAP2 Data Object.</Description>
    </Service>

    <Service title="DDX" role="http://services.opendap.org/dap2/ddx">
        <link type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ddx">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDX_Service">OPeNDAP Data Description and Attribute XML Document.</Description>
    </Service>

    <Service title="DDS" role="http://services.opendap.org/dap2/dds">
        <link type="text/plain" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dds">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDS_Service">OPeNDAP Data Description Structure.</Description>
    </Service>

    <Service title="DAS" role="http://services.opendap.org/dap2/das" >
        <link type="text/plain" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.das">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DAS_Service">OPeNDAP Dataset Attribute Structure.</Description>
    </Service>


    <Service title="INFO" role="http://services.opendap.org/dap2/info">
        <link type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.info">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Info_Service">OPeNDAP Dataset Information Page.</Description>
    </Service>

    <Service title="Server Version" role="http://services.opendap.org/dap4/version">
        <link type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ver">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Server_Version_Service">An XML document containing information about the software version of the server..</Description>
    </Service>


    <Service title="File Access" role="http://services.opendap.org/dap4/file">
        <link type="text/xml ## This value depends on the file being accessed."  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.file">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Native_File_Access_Service">Access to dataset file.</Description>
    </Service>

    <ServerSideFunctions>
        <Function name="geogrid"
                  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#geogrid">
            <Description>Allows a DAP Grid variable to be sub-sampled using georeferenced values.</Description>
        </Function>
        <Function name="grid"
                  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#grid">
            <Description>Allows a DAP Grid variable to be sub-sampled using the values of the coordinate axes.</Description>
        </Function>
        <Function name="linear_scale"
                  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#linear_scale">
            <Description>Applies a linear scale transform to the named variable.</Description>
        </Function>
        <Function name="version"
                  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#version">
            <Description>Returns version information for each server side function.</Description>
        </Function>
    </ServerSideFunctions>
</DatasetServices>

Rationale for the solution

The solution provides DAP4 servers with an interface that allows for both server-driven and agent driven content negotiation as described in [section 12 of the HTTP 1.1 specification]. Providing the Dataset Services Response in XML allows many downstream applications to ingest the content and leaves open the option for server implementers to add XSLT references to the document so that clients like browsers can easily build a presentation view of the information. Using the Atom 1.0 appears to involve no additional overhead and will enable other uses for the response.

Discussion

It is possible to embed in the above a reference to a XSLT that will transform the above XML into HTML similar to the following (with the caveat that this image is actually of a similar, but not identical, XML response):