DAP4 Dataset Services Response

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽


<-- back to OPULS Development

Author: ndp

Background

DAP2, while successful within several arenas in the scientific community, failed to provide a complete description of the web interface that a server should support. This came about for two reasons: DAP2 was developed at a time when 'web services' was an idea in considerable flux and, secondly, the DAP2 specification was written to focus on a data model (mostly) that supported interoperability. The current DAP4 draft specification document, like the DAP2 specification, does not address how a web-based interface to DAP4 should be built. However, we are going to provide a specification that will describe how such an interface for DAP4 should be built and this document contains information that should be part of that specification.

Problem addressed

Currently DAP2 servers provide a number of dataset level responses (services). These include DAP2 services such as the DDS, DAS, and DDX, plus other services that have been added over time such as the RDF and ISO responses. There is currently no way for a user (or software agent) to discover which of these many services might be available for a particular dataset. While THREDDS catalogs do provide a mechanism for determining which services are associated with which datasets, they don't actually provide a mechanism for discovering the details of the various service URLs. A THREDDS catalog might identify a DAP2 service, but the URL that can be assembled from the THREDDS catalog for that service is limited to just the base dataset URL. Typically dereferencing this URL will provide one of two things - either the underlying data file (for example a netcdf file) or an HTTP 403 error if direct access to the underlying files has been disabled in the server configuration.

Example:

This URL points to a dataset on a DAP2 server:

http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc

Dereferencing the link will not connect you to a DAP2 service or DAP2 response. It will (likely) return the underlying netcdf file. The only way to know that DAP2 services exist for this is by reading the URL path and recognizing that it is probably an instance of a DAP2 server based on the composition of the URL. In fact, there is no guarantee that dereferencing this URL will even yield the underlying file, as that behavior is a configuration option for some server implementations. If that access is not allowed in the configuration, then dereferencing the URL will simply return an HTTP 403 (forbidden) error.

Proposed solution

We intend to provide a 'Services' or 'Capabilities' response for DAP4. Dereferencing an unadorned DAP4 dataset resource URL will return a document describing the DAP services available for the dataset. This will be a required response and the only valid response for an 'unadorned' dataset resource URL. We will refer to this response as the 'Dataset Services Response' or 'DSR' in the 'DAP4 Web Services' specification and the remainder of this document.

At minimum this document MUST provide, for each service available for the dataset:

  • A human readable title of that service (e.g., The DDX service for the dataset)
  • One or more links that can be dereferenced to get the various representations of the response for the dataset.
  • An unambiguous, unique to the service type, resource role ID that serves as a way to clearly identify what each service does.
  • A brief description of the service

The response should also include:

  • A link to a complete, human readable, description of the service.
  • A reference to an XSLT that a browser would use to render the description into HTML for a presentation view.
  • Descriptions and syntax of server side functions available for the dataset

Service Access

A service is accessed by dereferencing one of the access URLs held in the href attribute of a <dsr:link> elements, which is typically constructed by adding some type of suffix to the dataset's referent (aka base) URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.

A more extensive discussion of the various services that might appear in a DatasetServices document can be found on the DAP4 Web Services v3 page.

Note: If a client understands how to modify/augment the dataset resource URL ( as described in the DAP4 Web Services Specification ) such that it becomes a DAP response URL then it need never obtain the Dataset Services response.

Dataset Services Response Encoding

c.f. DAP4 Web Services

In this section the namespace prefix dsr is associated with the namespace http://xml.opendap.org/ns/DAP/4.0/dataset-services#.

dsr:DatasetServices Element

The Dataset Services Response MUST contain a top level <dsr:DatasetServices> element. This <dsr:DatasetServices> element MUST contain:

  • An xml:base attribute whose value is the resource URL that was dereferenced to request the DSR response.
  • A list of DAP versions supported by server
    These appear as one or more child <dsr:DapVersion> elements of the <dsr:DatasetServices> element.
  • The implementation version of the server that produced the DSR.
    This appears as a single <dsr:ServerSoftwareVersion> child element of the <dsr:DatasetServices> element.
  • A list of all available DAP4 services for the dataset
    This is represented as 3 or more <dsr:Service> elements. (There are 3 required services for a DAP4 server.)
  • A list of supported extensions
    • Resource type extensions
    • Media type extensions
    • Server-side function extensions

The <dsr:DatasetServices> element SHOULD contain:

  • A title attribute whose value is a human readable title for the dataset.

dsr:DapVersion Element

The <dsr:DapVersion> element contains a single DAP version value as a text string. The DAP version is represented as two integer values separated by a period, MM.mm where the first value (MM) is the major version of the DAP protocol, and the second value (mm) is the minor version. Multiple <dsr:DapVersion> elements are used to represent that server can support multiple versions of the DAP protocol, one instance for each supported version.

Example:

<DapVersion>4.0</DapVersion>
<DapVersion>3.2</DapVersion>
<DapVersion>2.0</DapVersion>

dsr:ServerSoftwareVersion Element

The <dsr:ServerSoftwareVersion> element has no attributes and may contain as little as a simple text string (e.g., "TDS 4.3.57" or "Hyrax 1.7.45"), or it may contain any well-formed XML content not in the dsr namespace that the server implementer sees fit to use to describe the software version of their server implementation.

dsr:Service Element

Each DAP4 web service is described by a <dsr:Service> element with:

  • An optional title attribute whose value is a simple human readable title for the service.
  • An required role attribute whose value is a unique to the service type, resource role ID (a URI) that serves as a way to clearly identify what each service does. This is intended to allow people and software to unambiguously identify specific services, irrespective of their human readable title. The idea is that that the role attribute tells you what is going to happen when you dereference the URL held in the <dsr:link> element. Each service has a single role and all of the links (alternate representations) must fulfill the same role.
  • An optional <dsr:Description> element .
  • One or more <dsr:link> elements.
Regarding the role attribute

With regards to multiple representations of a service response and the role attribute: While there may be many alternate representations of a response, not all will fufill the same role. Representations fulfilling the same role should be bundled together in their own Service with the appropriate role value. For example the DMR can be mapped to the ISO19115 metadata space, but IS0-19115 responses are clearly outside of the role of the regular DMR service, which returns a DAP4 metadata response. One could view it as the two roles described different domains.

dsr:Description Element

Each <dsr:Description> element MUST contain:

  • An optional href attribute whose value is a URL string that points to a human readable document describing the service.
  • The required text content of the dsr:Description element MUST be a human readable description of the service.

dsr:link Element

Each <dsr:link> element MUST contain:

  • A required attribute called href whose value is a URL that when dereferenced will return the service response in the media type described by the value of the type attribute.
  • A required type attribute whose value is the media-type for the normative representation returned by that URL held in the href attribute.
  • An optional attribute called description whose value is a human readable string that provides a brief description of the <dsr:link> elements semantics.
  • Zero or more <dsr:alt> elements, one for each and every alternate representation of the response that can be requested using HTTP server-driven content negotiation in conjunction with the href URL. Each <dsr:alt> element MUST contain:
    • A required type attribute whose value is the MIME type that would be used in the HTTP Accept header to request that representation from the server.

Extensions Elements (Place Holder: Needs Content!)

Details of the extensions (e.g. server side functions, etc.) should be inserted in this section.

Example: Minimum Required DSR

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?xml-stylesheet type="text/xsl" href="/opendap/xsl/serviceDescription.xsl"?>
<DatasetServices xmlns="http://xml.opendap.org/ns/DAP/4.0/dataset-services#"
    xml:base="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">

    <DapVersion>4.0</DapVersion>
    
    <!-- ServerSoftwareVersion element may contain text, or any XML element content as long as the XML is not in the document namespace -->
    <ServerSoftwareVersion>Hyrax-2.7.9</ServerSoftwareVersion>
        
    <Service title="DAP4 Dataset Services" role="http://services.opendap.org/dap4/dataset-services">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Services_Description_Service">An index of the Services available for this data resource.</Description>

        <link description="Normative form of the DSR" 
              type="application/vnd.opendap.org.dataset-services+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
              <alt type="text/xml"/>
        </link>
        
        <link description="Normative DSR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xml"/>
              
    </Service>

    <Service title="DAP4 Dataset Metadata"  role="http://services.opendap.org/dap4/dataset-metadata">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Service_-_The_metadata">The DAP4 metadata content for this data resource.</Description>

        <link description="Normative form of the DMR" 
              type="application/vnd.org.opendap.dap4.dataset-metadata+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr">
              <alt type="text/xml"/>
        </link>
                      
        <link description="Normative DMR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.xml"/>
              
    </Service>

    <Service title="DAP4 Data" role="http://services.opendap.org/dap4/data">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Data_Service">DAP4 Data object for this data resource.</Description>
        <link type="application/vnd.org.opendap.dap4.data"  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap" >
        </link>
        
    </Service>

    
</DatasetServices>

Example: something more complicated

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?xml-stylesheet type="text/xsl" href="/opendap/xsl/serviceDescription.xsl"?>
<DatasetServices xmlns="http://xml.opendap.org/ns/DAP/4.0/dataset-services#" 
                 xml:base="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">

    <DapVersion>4.0</DapVersion>
    <DapVersion>3.2</DapVersion>
    <DapVersion>2.0</DapVersion>
    
    <!-- ServerSoftwareVersion element may contain text, or any XML element content as long as the XML is not in the document namespace -->
    <ServerSoftwareVersion>Hyrax-2.7.9</ServerSoftwareVersion>
    
    <Description>This data resource description is optional and needs only to be human readable.</Description>
    
    <Service title="DAP4 Dataset Services" role="http://services.opendap.org/dap4/dataset-services">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Services_Description_Service">An index of the Services available for this data resource.</Description>

        <link description="Normative form of the DSR" 
              type="application/vnd.opendap.org.dataset-services+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml">
              <alt type="text/html"/>
              <alt type="text/xml"/>
        </link>
        
        <link description="HTML representation of the DSR"  
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.html"/>
              
        <link description="Normative DSR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.xml"/>
              
    </Service>

    <Service title="DAP4 Dataset Metadata"  role="http://services.opendap.org/dap4/dataset-metadata">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Dataset_Service_-_The_metadata">The DAP4 metadata content for this data resource..</Description>

        <link description="Normative form of the DMR" 
              type="application/vnd.org.opendap.dap4.dataset-metadata+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr">
              <alt type="text/html"/>
              <alt type="text/xml"/>
              <alt type="application/rdf+xml"/>
        </link>
        
        <link description="Data Request Form" 
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.html"/>
              
        <link description="Normative DMR with generic Content-Type" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.xml"/>
              
        <link description="RDF representation of DMR" 
              type="application/rdf+xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.rdf"/>
    </Service>

    <Service title="DAP4 Data" role="http://services.opendap.org/dap4/data">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Data_Service">DAP4 Data object for this data resource.</Description>
        <link type="application/vnd.org.opendap.dap4.data"  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap" >
              <alt type="text/plain"/>
              <alt type="text/xml"/>
              <alt type="application/x-netcdf"/>
        </link>
        
        <link type="text/plain"  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.ascii" />
        <link type="text/xml"   
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.xml" />
        <link type="application/x-netcdf"   
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dap.nc" />
    </Service>

    <!-- ##################################################################################### -->
    <!--   Optional Services                                                                   --> 
    <!-- ..................................................................................... -->
    <Service title="ISO-19115 Metadata Service" role="http://services.opendap.org/dap4/iso-19115">
        <link description="Dataset metadata as ISO-19115" 
              type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.iso">
              <alt type="text/html"/>
        </link>
        <link description="ISO-19115 conformance score" 
              type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dmr.rubric">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_ISO_Conformance_Score_Service">ISO-19115 conformance score for the DMR.</Description>
    </Service>


    <Service title="DAP2 Data" role="http://services.opendap.org/dap2/dods">
        <link type="application/octet-stream" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dods">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Data_Service">DAP2 Data Object.</Description>
    </Service>

    <Service title="DDX" role="http://services.opendap.org/dap2/ddx">
        <link type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ddx">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDX_Service">OPeNDAP Data Description and Attribute XML Document.</Description>
    </Service>

    <Service title="DDS" role="http://services.opendap.org/dap2/dds">
        <link type="text/plain" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.dds">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DDS_Service">OPeNDAP Data Description Structure.</Description>
    </Service>

    <Service title="DAS" role="http://services.opendap.org/dap2/das" >
        <link type="text/plain" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.das">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_DAS_Service">OPeNDAP Dataset Attribute Structure.</Description>
    </Service>


    <Service title="INFO" role="http://services.opendap.org/dap2/info">
        <link type="text/html" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.info">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP2:_Info_Service">OPeNDAP Dataset Information Page.</Description>
    </Service>

    <Service title="Server Version" role="http://services.opendap.org/dap4/version">
        <link type="text/xml" 
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.ver">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Server_Version_Service">An XML document containing information about the software version of the server..</Description>
    </Service>


    <Service title="File Access" role="http://services.opendap.org/dap4/file">
        <link type="text/xml ## This value depends on the file being accessed."  
              href="http://test.opendap.org:8090/opendap/hyrax/ECMWF_ERA-40_subset.ncml.file">
        <Description href="http://docs.opendap.org/index.php/DAP4_Web_Services#DAP4:_Native_File_Access_Service">Access to dataset file.</Description>
    </Service>


    <function name="geogrid" role="http://services.opendap.org/dap4/server-side-function/geogrid">
        <Description  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#geogrid">Allows a DAP Grid variable to be sub-sampled using georeferenced values.</Description>
    </function>

    <function name="grid" role="http://services.opendap.org/dap4/server-side-function/grid">
        <Description  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#grid">Allows a DAP Grid variable to be sub-sampled using the values of the coordinate axes.</Description>
    </function>

    <function name="linear_scale" role="http://services.opendap.org/dap4/server-side-function/linear-scale">
        <Description  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#linear_scale">Applies a linear scale transform to the named variable.</Description>
    </function>

    <function name="version" role="http://services.opendap.org/dap4/server-side-function/version">
        <Description  href="http://docs.opendap.org/index.php/Server_Side_Processing_Functions#version">Returns version information for each server side function.</Description>
    </function>
    
    <functionGroup name="ferret" role="http://pmel.noaa.gov/dap4extension/ferret" >
        <Description href="http://ferret.pmel.noaa.gov/Ferret/documentation">This server supports ferret functions.</Description>
    </functionGroup>

    <extension name="async" role="http://opendap.org/extension/async">
        <Description href="http://docs.opendap.org/index.php/DAP4:_Asynchronous_Request-Response_Proposal_v3">The server supports asynchronous transactions.</Description>
    </extension>
    
    
</DatasetServices>

Rationale for the solution

The solution provides DAP4 servers with an interface that allows for both server-driven and agent driven content negotiation as described in [section 12 of the HTTP 1.1 specification]. Providing the Dataset Services Response in XML allows many downstream applications to ingest the content and leaves open the option for server implementers to add XSLT references to the document so that clients like browsers can easily build a presentation view of the information.

Discussion

Here is one idea for the HTML version of the DSR; it could be generated by the server using XSLT on the XML or by the browser when the user explicitly requests the .xml variant of the response (by embedding a reference to the XSL transform in that response document):

ServiceDescriptionPrototype-01.png



Appendix: Dataset Services Response XML schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema 
    targetNamespace="http://xml.opendap.org/ns/DAP/4.0/dataset-services#"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:xml="http://www.w3.org/XML/1998/namespace"
    xmlns:dsr="http://xml.opendap.org/ns/DAP/4.0/dataset-services#"
    xmlns="http://xml.opendap.org/ns/DAP/4.0/dataset-services#"
    elementFormDefault="qualified" 
    attributeFormDefault="unqualified">
    
    <xs:import  namespace="http://www.w3.org/XML/1998/namespace" schemaLocation="http://www.w3.org/2001/03/xml.xsd"/>
    <!--
	
	-->
    <xs:element name="DatasetServices" type="DatasetServicesType"/>
    <xs:element name="DapVersion" type="DapVersionType"/>
    <xs:element name="SoftwareServerVersion" type="AnyXMLType"/>
    <xs:element name="Service" type="ServiceType"/>
    <xs:element name="Description" type="DescriptionType"/>
    <xs:element name="link" type="linkType"/>
    <xs:element name="alt" type="altType"/>
    <!--
	
	 -->
    <xs:complexType name="DatasetServicesType"> 
        <xs:annotation>
            <xs:documentation>DatasetServices root element type</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="DapVersion" minOccurs="1" maxOccurs="unbounded"/>
            <xs:element ref="SoftwareServerVersion" minOccurs="1" maxOccurs="1"/>
            <xs:element ref="Description" minOccurs="0" maxOccurs="1"/>
            <xs:element ref="Service" minOccurs="3" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute ref="xml:base"  use="required"/>
        <xs:attribute name="role" type="xs:anyURI" use="required"/>
    </xs:complexType>
    <!--
	
	 -->
    <xs:complexType name="ServiceType"> 
        <xs:annotation>
            <xs:documentation>DatasetServices Service type</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="Description" minOccurs="0" maxOccurs="1"/>
            <xs:element ref="link" minOccurs="1" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="title" type="xs:string" use="optional"/>
        <xs:attribute name="role" type="xs:anyURI" use="required"/>
    </xs:complexType>
    <!--
        
	--> 
    <xs:simpleType name="DapVersionType">
        <xs:restriction base="xs:string">
            <xs:pattern value="\d{1,}.\d{1,}"></xs:pattern>
        </xs:restriction>
    </xs:simpleType>
    <!--
        
	-->
    <xs:complexType name="DescriptionType">
        <xs:simpleContent>
            <xs:extension base="xs:string">
                <xs:attribute name="href" type="xs:anyURI"/>
            </xs:extension>
        </xs:simpleContent>
     </xs:complexType>
    <!--
        
	-->
    <xs:complexType name="altType">
        <xs:attribute name="type" type="xs:string"/>
    </xs:complexType>
    <!--	
        
	-->
    <xs:complexType name="linkType"> 
        <xs:annotation>
            <xs:documentation>DatasetServices link type</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="alt" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="href" type="xs:string" use="required"/>
        <xs:attribute name="type" type="xs:string" use="required"/>
        <xs:attribute name="description" type="xs:string" use="optional"/>         
    </xs:complexType>
    <!--
	
	 -->
    <xs:complexType name="AnyXMLType">
        <xs:annotation>
            <xs:documentation>
                Use this to embed arbitrary XML in an element. The resulting elements 
                contents are ignored by DAP software. Other software
                might find the information useful. The XML elements must
                satisfy the requirements for 'lax' processing under schema 1.0.
                In practice, that means just about anything.
                
                name: A name to associate with this chunk of XML
                *: This element can contain any other attributes that conform 
                to the schema 1.0 definition of 'lax' processing
                
                ( <xs:any/>+ ) 
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:any namespace="##any" minOccurs="0" processContents="lax"/>
        </xs:sequence>
        <xs:anyAttribute processContents="lax" namespace="##any"/>
    </xs:complexType>
    
    
</xs:schema>

fini

Contains cruft in comments...