DAP4: Specification Volume 2

From OPeNDAP Documentation

<< Back to OPULS Development

The Data Access Protocol: DAP Version 4.0


Volume 2: DAP4 Web Services
Date:November 6, 2013
Last Revised:04 July 2014
Status:Draft
Authors:John Caron (Unidata)
Ethan Davis (Unidata)
David Fulker (OPeNDAP)
James Gallagher (OPeNDAP)
Dennis Heimbigner (Unidata)
Nathan Potter (OPeNDAP)
Copyright:2014 University Corporation for Atmospheric Research and Opendap.org


Abstract

While Volume 1 of the DAP4 specification addresses the detailed semantics and structure of the DAP4 data and metadata content, this volume (Volume 2) details the manner in which clients can request things from a DAP4 server and what things the clients can expect to receive from a DAP4 server in response to their requests. The description of this interaction focuses on DAP4 clients and servers that are communicating via HTTP, and while this is perceived by the authors as the most common transport for DAP4 client/server interaction the specification is written in a broad enough manner to allow for other transport mechanisms.

Distribution of this document is unlimited.


Change List

2013.09.06: Initial Draft
2014.07.04 ASCII/.ascii changed to text/.txt; Removed most of the async text since it is duplicate of the material in the extension; Created a volume 3 containing the extensions


Contents

1 Introduction

This document defines the DAP4 Web Service protocol which DAP4-compliant software MUST support when utilizing the HTTP protocol to transmit DAP4 requests and responses, along with additional optional services that DAP4-compliant software SHOULD support.

The DAP4 protocol uses three basic responses to represent a data resource. One response, the Dataset Metadata Response (DMR) contains metadata information describing the structure of the data resource. That is, it characterizes the variables, their datatypes, names and attributes. The second response, the Data Response (DataDMR), contains both the metadata about the requested data and the actual data that was requested. The third basic response is the Dataset Services Response (DSR) that provides a listing of services, any alternate media representations if available, and all of the associated access URI's for a particular data resource.

The DAP4 protocol uses a fourth basic resource, the Constraint/Query Expression (CE), to represent subsetting of and, possibly, transformations of the dataset requested.

The DAP4 protocol uses a fifth basic resource, the Error Response (ER), to allow servers to communicate error information with clients. When a request for any of the three basic responses cannot be completed, an Error response is returned. If an error occurs before the standard response is initiated, an error response is returned in place of the standard response. If an error occurs after a data response has been initiated, an Error Response is returned as the final chunked record as described in Section 7 "DAP4 Chunked Data Representation" of the DAP4 Data Model document.[DAP4 Vol.1]

The DAP4 Data Model, constraint/query language, dataset metadata XML encoding, and binary data encoding are all defined in the DAP4 Volume 1: "Data Model, Persistent Representations, and Constraints" document.[DAP4 Vol.1]

Each of the three responses (Dataset Metadata, Data, and Dataset Services) are complete in and of themselves so that, for example, the Data response can be used by a client without ever requesting either of the two other responses. In many cases, client programs will request the Dataset Metadata response before requesting the Data response but there is no requirement they do so and servers SHALL NOT require that behavior on the part of clients.

These three standard dataset responses can each be accessed in two different ways. First, similar to DAP2 URL construction and to support client-driven content negotiation (see the section titled "HTTP - Content Negotiation and Media Types" below), each dataset resource has a standard URL suffix that can be added to the DAP4 dataset URL to retrieve the resource in its standard (normative) encoding. This allows clients, given a base DAP4 dataset URL, to construct DAP4 URLs in a simple and standard way for each of the three standard dataset resources.

Second, to support server-driven content negotiation (see the "HTTP - Content Negotiation and Media Types" section below) and hypermedia-driven style, a DAP4 resource role is defined for each standard dataset resource and a DAP4 media type is defined for each dataset resource encoding scheme.

Any particular instance of a DAP4 server MAY have one or more additional services, alternate media representations of service responses, or unique (to the server instance) server side functions. All of these things, including the core services and their default representations MUST be included in the Dataset Services Response.[DAP4 DSR]

The DAP4 web service is currently limited to HTTP GET requests though it is expected that extensions (e.g., asynchronous access) will use other HTTP methods (e.g., POST). This makes, for now, the DAP4 Constraint Expression something of a pseudo-resource type given that they are encoded as part of the URL query string rather than as an independent document.


2 Terms and Definitions

Dataset Service Response (DSR)
The DAP4 response type that contains a list of DAP (and other) services available for the dataset including any alternate media representations and all the associated access URIs.
Dataset Metadata Response (DMR)
The DAP4 response type that contains metadata information describing the structure of the requested data. The metadata information characterizes the requested data variables including their names, data types, shapes, and attributes.
Dataset Data Response (Data)
The DAP4 response type that contains both the dataset metadata and the binary data for the requested data.
Resource role ID
A URI that identifies the role of a resource, generally provided with a link to allow clients to identify the type of resource the link references. (For instance, an "atom:link" element has an optional "atom:rel" attribute.)
Media Type
A internet media type is a two-part identifier for resource encoding schemes, e.g. "text/html", "text/plain", "application/octet-stream". (See section 1.1 DAP4 Resource Roles and Media Types of this document)
DAP4 Constraint Expression (CE)
The constraint expression that encapsulates various sub-setting of, and possibly the application of server side functions to variables in a DAP4 dataset.
Query String (QS)
Everything after the "?" character in a URL.

2.1 DAP4 Resource Roles and Media Types

The standard DAP4 dataset resource roles and encodings (plus a few alternate encodings) that are defined in the core DAP4 documents are:

Dataset Services Response (DSR)

Resource Role
http://services.opendap.org/dap4/dataset-service
URL Suffix Media Type URL Example
none or ".dsr"
application/vnd.opendap.dap4.dataset-services+xml
Normative DSR encoding
http://server/path/dataset.nc

http://server/path/dataset.nc.dsr

".xml" or ".dsr.xml"
text/xml
Normative DSR encoding with generic media type
http://server/path/dataset.nc.xml

http://server/path/dataset.nc.dsr.xml

".html" or ".dsr.html"
text/html
HTML DSR encoding
http://server/path/dataset.nc.html

http://server/path/dataset.nc.dsr.html

Dataset Metadata Response (DMR)

Resource Role
http://services.opendap.org/dap4/dataset-metadata
URL Suffix Media Type URL Example
".dmr"
application/vnd.opendap.dap4.dataset-metadata+xml
Normative DMR encoding
http://server/path/dataset.nc.dmr
".dmr.xml"
text/xml
Normative DMR encoding with generic media type
http://server/path/dataset.nc.dmr.xml
".dmr.html"
text/html
HTML DMR encoding
http://server/path/dataset.nc.dmr.html

Dataset Data Response (Data)

Resource Role
http://services.opendap.org/dap4/data
URL Suffix Media Type URL Example
".dap"
application/vnd.opendap.dap4.data
Normative Data encoding
http://server/path/dataset.nc.dap
".dap.txt"
text/plain
Text (UTF-8) Data encoding
http://server/path/dataset.nc.dap.txt
".dap.xml"
text/xml
XML Data encoding
http://server/path/dataset.nc.dap.xml

Error Response (ER)

Resource Role
http://services.opendap.org/dap4/error
URL Suffix Media Type URL Example
N/A
application/vnd.opendap.dap4.error+xml
Normative Error encoding
N/A
N/A
text/xml
Normative Error encoding with generic media type
N/A
N/A
text/html
HTML Error encoding
N/A

2.2 DAP4 Core Specification Documents

  • DAP4 Volume 1: "Data Model, Persistence Representations, and Constraints"
  • DAP4 Web Service (this document)
  • DAP4 Dataset Services
  • DAP4 Requirements for Server-side Functions

2.3 Extensions to DAP4 Core Specifications

Several types of extensions can be made to the DAP4 core including:

  • New encodings for the core DAP4 response types
  • New response types
  • New server-side functions.

3 DAP4 Web Service Responses

The core of the DAP4 Web Service protocol consists of the four standard response types: Dataset Services Response (DSR), Dataset Metadata Response (DMR), Dataset Data Response (Data), DAP4 Error Response (Error) Each dataset served by a DAP4 compliant server MUST provide the DSR, DMR, and Data responses and MUST return errors documents as DAP4 Error Responses.

All of the example requests described below are based on the DAP4 dataset URL:

http://server.org:8080/dap/path/data.nc

3.1 DAP4 Dataset Services Response

The DAP4 Dataset Services Response (DSR)[DAP4 DSR] provides clients with a listing of all available DAP4 services and all the available encodings for those services as well as all available DAP4 extensions.

Each service (or response type) has a unique resource role (defined in the appropriate specification), each link (alternate representation) for a given service MUST fulfill that same role. This is not always a clear distinction to make. For example, the DAP4 Dataset Metadata Response can be mapped into ISO 19115 metadata. However, IS0 19115 is clearly a different domain.

The DAP4 Dataset Services Response MUST contain the following information:

  • List of DAP versions supported by server
  • The implementation version (e.g., "TDS 4.3.57" or "Hyrax 1.7.45")
  • List of all available DAP4 services for the dataset
  • For each DAP4 services listed, a list of all available links each with its corresponding media type
  • List of supported extensions
    • Resource type extensions
    • Media type extensions
    • Server-side function extensions

If SHOULD contain the following information:

  • A human readable title for the dataset
  • A human readable title for each service

To take advantage of web caching, servers should try to keep DSRs light weight (i.e., quick creation) and as stable as possible.

3.1.1 DSR Resource Role

DSRs are identified by the resource role:

http://services.opendap.org/dap4/dataset-services

3.1.2 Normative Encoding of the DSR

The normative XML representation for the Dataset Services Response is defined in the "Normative XML Encoding of the DSR" appendix. The media type for the normative XML representation is

application/vnd.opendap.dataset-services+xml

3.1.3 Service Behavior

When an HTTP GET request is made on a base DAP4 dataset URL, all DAP4 servers MUST return the normative XML encoding of the DSR given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type,
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DSR.

For example, the request:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.dataset-services+xml
Date: ...
3.1.3.1 Downgrading the Normative XML Encoding (Required)

When an HTTP GET request is made on a base DAP dataset URL with the suffix .xml
added to it:

request url = dataset_url.xml

the response MUST be the normative representation of the DSR along with the HTTP Content-Type header set to text/xml . For example:

GET /dap/path/data.nc.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

The normative XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: text/xml

3.1.4 Other Encodings of the DSR

3.1.4.1 HTML Encoding (Optional)

When an HTTP GET request is made on a base DAP dataset URL with the suffix .html
added to it:

request url = dataset_url.html

the server MUST reply with an HTML representation of the DSR, or return an HTTP status of 404 to indicate that an HTML representation of the DSR is not available. For example:

GET /dap/path/data.nc.html HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: ...

If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example this request:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: text/html

Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.2 Dataset Metadata Response

The Dataset Metadata Service returns the Dataset Metadata Response (DMR) which is a metadata description of the dataset. The normative representation of the DMR is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset, persisted as a DAP4 data model representation of the dataset held at the server.[DAP4_Vol1] The DMR service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied. If a constraint expression has been successfully applied, the service will returned the constrained view of the dap:Dataset object. The constrained view may contain different data structures than the unconstrained view as the constraint may alter the reasonable representation of the data set. Note that all dap:Attribute objects have been removed from constrained dap:Dataset objects. More information on the syntax of DAP4 constraint expressions can be found in Volume 1 of the DAP4 specification. [DAP4_Vol1]

3.2.1 DMR Resource Role

DMRs are identified by the resource role:

http://services.opendap.org/dap4/dataset-metadata

3.2.2 Normative Encoding of the DMR

The normative XML representation for the Dataset Metadata Response is defined in Volume 1 of the DAP4 specification.[DAP4_Vol1] The media type for the normative XML representation is:

application/vnd.opendap.dap4.dataset-metadata+xml

3.2.3 Service Behavior

Every DAP4 compliant server MUST return the normative representation of the Dataset Metadata Response (an XML document described Volume 1 of the DAP4 specification [DAP4_Vol1]) when a client attempts to access a dataset URL with the suffix ".dmr" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.

When an HTTP GET request is made on a base DAP dataset URL with the suffix .dmr added to it:

request url = dataset_url + '.dmr'+ [?dap_constraint]

the server MUST reply with an normative representation of the DMR for the (possibly constrained) dataset given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type (application/vnd.opendap.dap4.dataset-metadata+xml),
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DMR.

For example, the request:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.dataset-metadata+xml
Date: ...
3.2.3.1 Downgrading the Normative XML Encoding (Required)

While the normative representation of the the Dataset Metadata response is already an XML document, the normative media type is application/vnd.opendap.dataset-metadata+xml which may be unfamiliar to many generic clients (such as web browsers) and it is quite conceivable that such a client might ask for the more generic text/xml media type.

When an HTTP GET request is made on a DAP DMR URL with the suffix .xml
added to it:

request url = dataset_url.dmr.xml

the response MUST be the normative representation of the DMR along with the HTTP Content-Type header set to text/xml . For example:

GET /dap/path/data.nc.dmr.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

The normative XML representation of the DMR MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL ( and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: text/xml

3.2.4 Other Encodings of the DMR

3.2.4.1 HTML Encoding (Optional)

When an HTTP GET request is made on a DAP DMR URL with the suffix .html
added to it:

request url = dataset_url.dmr.html

the server MUST reply with an HTML representation of the DMR, or return an HTTP status of 404 (or 415) to indicate that an HTML representation of the DMR is not available. For example:

GET /dap/path/data.nc.dmr.html HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: ...

If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DMR. For example this request:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: text/html

Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3 DAP4: Data Response

The Data Service provides DAP4 data access to a dataset, and is the (primary) way that DAP4 returns data to a client. The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions. When the service is invoked it returns the DAP4 data object. On the wire this is a binary document with MIME type application/vnd.opendap.dap4.data. The payload is broken into two logical parts: A DMR-type xml document that describes the data and a BLOB that contains the actual data. For more on the information on the Data response and the internal structure of its payload along with a complete discussion of the DAP4 constraint expression syntax see Volume 1 of the DAP4 specification.[DAP4_Vol1]

3.3.1 Data Response Resource Role

DAP4 Data Responses are identified by the resource role:

http://services.opendap.org/dap4/data

3.3.2 Normative Encoding of the Data Response

The normative XML representation for the Data Response is defined in Volume 1 of the DAP4 specification.[DAP4_Vol1] The media type for the normative XML representation is:

application/vnd.opendap.dap4.data


3.3.3 Service Behavior

Every DAP4 compliant server MUST return the normative representation of the Data Response when a client attempts to access a dataset URL with the suffix ".dap" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them. The normative representation of the Data Response is described in Volume 1 of the DAP4 specification.[DAP4_Vol1]

When an HTTP GET request is made on a base DAP dataset URL with the suffix .dap added to it:

request url = dataset_url + '.dap'+ [?dap_constraint]

the server MUST reply with an normative representation of the (possibly constrained) data response for the dataset given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type (application/vnd.opendap.dap4.data),
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the Data Response.

For example, the request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.data
Date: ...

3.3.4 Other Encodings of the Data Response

3.3.4.1 Text Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .txt
added to it:

request url = dataset_url.dap.txt

the server MUST reply with the text representation of the Data Response using the utf-8 character set, or return an HTTP status of 404 (or 415) to indicate that a text representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.txt HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: ...

If available, the text representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an text encoded Data Response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: text/plain

Must return the text representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3.4.2 XML Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .xml
added to it:

request url = dataset_url.dap.xml

the server MUST reply with the XML representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that an XML representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

If available, the XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an XML encoded data response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: text/xml

Must return the XML representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3.4.3 NetCDF-3 Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .nc
added to it:

request url = dataset_url.dap.nc

the server MUST reply with a NetCDF-3 representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that a NetCDF-3 representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/x-netcdf
Date: ...

If available, the NetCDF-3 representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an NetCDF-3 encoded data response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: application/x-netcdf

Must return the NetCDF-3 representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3.4.4 NetCDF-4 Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .nc4
added to it:

request url = dataset_url.dap.nc4

the server MUST reply with a NetCDF-4 representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that a NetCDF-4 representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/x-netcdf;ver=4
Date: ...

If available, the NetCDF-4 representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an NetCDF-4 encoded data response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: application/x-netcdf;ver=4

Must return the NetCDF-4 representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.


3.4 DAP4 Error Response

The DAP4 protocol returns error information using an Error response. If a request for any of the three basic responses cannot be completed then an Error response is returned in its place.

The normative XML representation for the Error Response is defined by the following RELAX-NG schema.

<grammar xmlns="http://relaxng.org/ns/structure/1.0"
                xmlns:doc="http://www.example.com/annotation"
                datatypeLibrary="http://xml.opendap.org/datatypes/dap4"
                ns="http://xml.opendap.org/ns/DAP/4.0#"
                >
<start>
  <ref name="errorresponse"/>
</start>
<define name="errorresponse">
  <element name="Error">
    <optional>
      <attribute name="httpcode"><data type="dap4_integer"/></attribute>
    </optional>
    <optional>
      <interleave>
        <element name = "Message"><text/></Message>
        <element name = "Context"><text/></Message>
        <element name = "OtherInformation"><text/></Message>
      </interleave>
    </optional>
  </element>
</define>

The Error element has one optional attribute: the httpcode which is a standard HTTP protocol return code indicating the general class of error. When possible, this code should match the return code in the HTTP headers for the response.

The body of the <Error> element may contain any or all of the following inner elements each containing arbitrary text.

  1. <Message> — A short informative message describing the error.
  2. <Context> — Information describing the context in which the error occurred: position of a parse error in a constraint expression, for example.
  3. <OtherInformation> — Arbitrary additional text information: a Java stack trace, for example.

3.4.1 Error Response Resource Role

DAP4 Error Responses are identified by the resource role:

http://services.opendap.org/dap4/error

3.4.2 Normative Encoding of the Error Response

The normative XML representation for the Error Response is defined in Appendix x "Normative XML Encoding of the Error Response". The media type for the normative XML representation is:

application/vnd.opendap.dap4.error.xml

4 HTTP

The DAP4 Web Services specification describes the features of HTTP that are required to be a compliant DAP4 client or server. It does not attempt to describe all aspects of HTTP that DAP4 servers might implement or that DAP4 clients may see in response to DAP4 requests. Similarly, it does not cover all issues related to the performance and scalability of HTTP.

However, the following sections include both DAP4 requirements as well as some suggestions of HTTP features that servers and clients are encouraged to use.

4.1 Content Negotiation and Media Types

Though the DAP4 core specifications only describe one encoding for each type of resource, DAP4 web servers MAY have the ability to provide a given resource in a number of different media types. All media types available for a resource MUST be listed in the DAP4 Dataset Services response document.

DAP4 responses MUST use the "Content-Type" header field to identify the media type of the DAP4 response body. For example, the normative value for the XML encoded DMR response is application/vnd.opendap.dap4.dataset-metadata+xml.

The DAP4 Dataset Services response describes the available services and their media types, and through this description provides DAP4 software (client and/or server) with two different mechanisms to negotiate for different kinds of media representations. The first mechanism is server-driven content negotiation as described in the HTTP 1.1 specification, section 12, "Content Negotiation".[HTTP RFC2616 - Section 12] The second mechanism is similar to the agent-driven negotiation also described in section 12 of the HTTP 1.1 specification. The difference being that the "list of available representations ... [each with] its own URI" is provided by the DAP4 Dataset Services response rather than in response to an initial request.

Clients need not retrieve the Dataset Services response in order to access the normative representations of either the Dataset Metadata or Data responses, as these responses are required for every DAP4 server and are mapped to well known URL patterns. If clients attempt to access other representations or other services using agent-driven negotiation without first checking the Dataset Services response, they should be prepared to receive a 404 Not Found response.

When using server-driven negotiation, DAP4 clients are encouraged to, at a minimum, include "Accept"[HTTP RFC2616 - Section 14.1]and "User-Agent"[HTTP RFC2616 - Section 14.43] headers in their requests and to provide accurate and detailed information in the values of those headers. However, when server-driven negotiation is used, DAP4 servers are not limited to those headers for determining the media type that is returned. DAP4 clients should also be prepared to handle 415 "Unsupported Media type" response codes.


4.2 Redirects

While HTTP redirects are not directly part of the DAP4 web protocol it is strongly suggested that DAP4 client implementations be capable of processing HTTP redirects as nominally described in the HTTP-1.1 specification sections on redirection status codes and redirection response headers.[HTTP RFC2616][HTTP RFC2616 - Section 10.3][HTTP RFC2616 - Section 14.30] (It is suggested that implementers of DAP4 clients utilize an existing robust HTTP client library that will manage this for them.)

4.3 Caching of Responses and Conditional Requests

While, HTTP caching and conditional requests are not explicitly part of the DAP4 web protocol, they do provide important tools for improving the performance of both sides client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how HTTP caching [HTTP RFC2616 - Section 13] works, and utilize it for working with DAP servers.

4.4 Authentication/Authorization

Authentication is the process by which a user agent establishes the identity of the user to a server, and the server establishes it's identity with the user agent. While, HTTP authentication is not explicitly part of the DAP4 web protocol, it does provide important tools for securing the client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how HTTP authentication works, and utilize it for working with DAP servers.[HTTP RFC2617] Server implementers should pay particular attention to HTTP security considerations.[HTTP RFC2616 - Section 15]

Authorization is the process through which a server determines who/what has access to its various holdings and services. The HTTP protocol does not directly address the issue of authorization (even though HTTP defines the 401 response status using the word "authorization" it does not provide separate semantics for authentication and authorization which we see as an important distinction for data access), but any DAP4 server implementer should be aware that some kind of mechanism for controlling access to holdings and services will likely be desired by the people that install and operate their software.

4.5 Headers

4.5.1 Request Headers

These are the HTTP request headers that DAP clients using HTTP MAY utilize. DAP4 servers MUST accept these headers and act on them in a manner consistent with their descriptions below.

4.5.1.1 General
Accept
The HTTP Accept header MAY be used by clients that wish to engage in server-dirven content negotiation by requesting a particular representation of a resource in the initial request. The server MUST utilize this header, if present, in a manner consistent with the HTTP content negotiation]] specification.[HTTP RFC2616 - Section 12]
User-Agent
The HTTP User-Agent header MAY be used by clients to indicate their "software identity" to the server.[HTTP RFC2616 - Section 14.43] The server MAY utilize this header, if present, to alter the Content-Type of the response to something that is more likely to be digestible by the requesting client software.[HTTP RFC2616 - Section 14.17]
4.5.1.2 DAP Specific
There are no DAP specific headers required to make a general DAP request.


4.5.2 Response Headers

These are the HTTP response headers that DAP servers using HTTP MUST and SHOULD (as indicated) return.

4.5.2.1 General
Date
DAP4 servers MUST return an HTTP Date header whose value is the time of the response and which MUST be in RFC 1123 date/time format.[RFC1123]


Last-Modified
DAP4 servers SHOULD return an HTTP Last-Modified header whose value is the last modified time of the request resource and which MUST be in RFC 1123 date/time format.[RFC1123]


Content-Type
DAP4 servers MUST return an HTTP Content-Type header, the value of which SHOULD be set in accordance with the Dap4 Resource Roles and Media Types discussion in section 1.1 of this document.
Content-Description
DAP4 servers SHOULD return an HTTP Content-Description header.
Content-Disposition
DAP4 servers SHOULD return an HTTP Content-Disposition header when transmitting file typed responses.
Content-Encoding
DAP4 servers MUST return an HTTP Content-Encoding header when the content-coding of an entity is not "identity".


4.5.2.2 DAP Specific
X-DAP-Server
DAP4 servers SHOULD return the X-DAP-Server HTTP header. This HTTP header is used in a response to communicate the software version of the server. This may be a simple string with the server name and version number, or multiple software component versions may be represented. The value of this header is a string determined by the implementations author(s).
Example
X-DAP-Server: bes/3.10.0, libdap/3.11.2, dap-server/ascii/4.1.2, csv_handler/1.0.2, freeform_handler/3.8.4, fits_handler/1.0.7, fileout_netcdf/1.1.2, gateway_module/1.1.0, hdf4_handler/3.9.4, hdf5_handler/1.5.0, netcdf_handler/3.10.0, ncml_module/1.2.1, dap-server/usage/4.1.2, dap-server/www/4.1.2, xml_data_handler/1.0.1
Example
X-DAP-Server: TDS-4.19.3
X-DAP
DAP4 servers MUST return the X-DAP HTTP header. This HTTP header is used in a response to indicate the DAP protocol version used to encode the content of the response. This value is constrained to a format of "major_version" dot "minor version", where both major_version and minor_version are represented by an integer value.
Example
X-DAP: 4.0
Example
X-DAP: 2.17


4.6 Status Codes

DAP servers that provide an HTTP interface are expected to utilize the HTTP response codes in a manner consistent with the HTTP 1.1 specification.[HTTP RFC2616]

The ones that are detailed here are used by the DAP in a manner consistent with the specifications definition, but in support of specific DAP server behavior.

4.6.1 200 OK

A server MUST return an HTTP status of 200 when the request has been successful and that the returned document contains the requested resource. However since DAP responses can be quite large and since it is also possible for the server to encounter any number of problems during the marshaling, serialization, and subsequent transmission of the response it is possible that the server may have committed/transmitted the HTTP headers (in which the status value is transmitted) before a subsequent error is encountered. These types of errors are transmitted in the DAP4 over-the-wire protocol and all DAP4 clients MUST be able to read and process these errors.

4.6.2 400 Bad Request

The HTTP specification defines this status code as:

The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.[HTTP RFC2616 - Section 10.4.1]

DAP4 servers utilizes this code to mean the following.

4.6.2.1 400 Bad DAP4 Request Syntax
The 400 Bad DAP4 Request Syntax HTTP response code MUST be returned by the server when there is a problem with the syntax of the OPeNDAP URL. The problem could be in the formulation of the constraint expression, or it may be that the URL extension did not match any that are recognized by this server.

4.6.3 401 Unauthorized

The 401 Unauthorized HTTP response code MUST be returned by the server when access to the requested resource requires (not previously acquired) user authentication. See the HTTP specification-1.1 for usage and behavior.[HTTP RFC2616 - Section 10.4.2]

4.6.4 403 Forbidden

The 403 Forbidden HTTP response code MUST be returned when the server has understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. This is appropriate to return if, for example, the server software does not have read permission for the requested resource.[HTTP RFC2616 - Section 10.4.4]

4.6.5 404 Not Found

The 404 Not Found HTTP response code MUST be returned when the server has not found anything matching the Request-URI.[HTTP RFC2616 - Section 10.4.5]

4.6.6 415 Unsupported Media Type

The 415 Unsupported Media Type HTTP response code MUST be returned when the client requests a representation of the requested resource that the server cannot provide.[HTTP RFC2616 - Section 10.4.16]

4.6.7 500 Internal Server Error

The 500 Internal Server Error status code SHOULD be returned when the DAP server encounters internal problems when attempting to fulfill a request.[HTTP RFC2616 - Section 10.5.1]

4.7 Verbs (aka Methods)

4.7.1 GET

A DAP4 request may be made using the HTTP GET request method utilizing a Uniform Resource Identifier (URI) that encodes information specific to the DAP4.

Each GET request MUST conform to the HTTP specification (which basically says that a GET request MUST contain an HTTP protocol version number followed by a MIME-like message containing various headers that further describe the request.). While there are some optional DAP4 HTTP request headers that may be used, DAP4 requests do not require specific HTTP headers beyond those necessary for HTTP (see section 4.5.1 Request Headers of this document for more). DAP4 servers SHOULD support the use of the HTTP Accept request header to allow clients to engage in HTTP content negotiation for specific representations of a requested DAP4 response.[HTTP RFC2616 - Section 12]

The DAP server responds to the GET request with an HTTP compliant response (one that includes a status line containing the HTTP protocol version and an error or success code, followed by HTTP response headers and then response itself). There are two DAP specific HTTP headers that are always included in a DAP response over HTTP: X-DAP-Server and X-DAP, as described in section 4.5.2 of this document. The DAP response is the payload of the HTTP response. Unless otherwise negotiated, the data response payload is encoded using the chunked format as described in [DAP4 Vol.1], while the other responses are simple HTTP responses.

4.7.1.1 Examples
HTTP GET request.
GET /dap/path/data.nc.dap?/u,/v[0:8:1024] HTTP/1.1
Host: server.org
Accept: application/vnd.opendap.dap4.data


5 DAP4 URLs

In DAP4 we divide a dataset URL into two sections, the identifier and the query string. The identifier section is everything up to "?" character. The query string is the "?" character and everything after it.

For example in the URL:

http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.dmr?dap4.ce=/u;/lat;/lon

We have:

identifier = http://test.opendap.org:8080/opendap/data/nc/fnoc1.nc.dmr
query string = ?dap4.ce=/u;/lat;/lon

Additionally DAP4 URLs conform to the web convention in which the query string is decomposed as a set of key-value pairs (KVP) separated from each other by "&" characters:

?key=value&key=value&key=value ...

Many web services utilize this pattern, including OGC. The DAP2 constraint expression subsumed the entire query string, so it did not fit into the standard KVP model. Tomcat (and other web server frameworks) provide specific API methods for collecting the KVP from the query string, but again DAP2 doesn't play well with this. The DAP4 constraint is designed to operate in a KVP environment.

5.1 Query String Parameters

  • DAP4 query string parameters will always begin with the 5 character string "dap4." In this way query string parameters associated with the DAP4 protocol can be easily identified by both people and software.
  • DAP4 query string keys are case sensitive.
  • DAP4 servers MUST treat ALL values of query string KVPs as case sensitive.
  • Each DAP4 key may appear once in each query string (request URL).
  • The order of the keys does not matter, and unrecognized keys are ignored, along with their values.

The DAP4 protocol reserves the exclusive use of all future query string keys that begin with the 4 character token "dap4." This way future DAP4 service features may be added and invoked through the query string section of the request URL without interfering with other features and behaviors added to service implementations outside of the DAP4 development process.

The following keys are reserved.

dap4.ce
The DAP4 constraint expression is contained in a single query string parameter named "dap4.ce" This constraint expression contains all of the subsetting information for the dataset include the projection (which variables are to be returned), slicing (how the various arrays are to be decimated), and filtering (conditional retrieval of values). The fill discussion of the syntax of the constraint expression can be found in Section 8 of Volume 1 of the DAP4 specification. [DAP4 Vol.1]
dap4.async
The asynchronous response behavior is described in detail in Section 10 of Volume 1 of the DAP4 Specification.[DAP4 Vol.1] While the a DAP4 server's asynchronous response behavior can be controlled by a savvy client using HTTP headers it can also be managed using the DAP4 query string parameter "dap4.async"
dap4.func
While server-side functions are not addressed in the initial DAP4 specification or constraint expression syntax we do anticipate them being defined (in short order) in an extension to the DAP4 specification. For now we are working with a proposed server side function syntax in which a server side function is invoked as a key value pair, something like: dap4.func=ugr5(0,v,z,”10<lat<30”)

6 Normative References

[DAP4 Vol1] DAP4 Volume 1: Data Model, Persistent Representations, and Constraints.

[HTTP RFC2616] Hypertext Transfer Protocol -- HTTP/1.1 (as text).

[HTTP RFC2616 - Section 10.2.3] HTTP/1.1 Section 10.2.3 - Accepted

[HTTP RFC2616 - Section 10.3] HTTP/1.1 Section 10.3 - Redirection Status Codes

[HTTP RFC2616 - Section 10.4.1] HTTP/1.1 Section 10.4.1 - Bad Request

[HTTP RFC2616 - Section 10.4.2] HTTP/1.1 Section 10.4.2 - Unauthorized

[HTTP RFC2616 - Section 10.4.4] HTTP/1.1 Section 10.4.4 - Forbidden

[HTTP RFC2616 - Section 10.4.5] HTTP/1.1 Section 10.4.5 - Not Found

[HTTP RFC2616 - Section 10.4.10] HTTP/1.1 Section 10.4.10 - Conflict

[HTTP RFC2616 - Section 10.4.13] HTTP/1.1 Section 10.4.13 - Precondition Failed

[HTTP RFC2616 - Section 10.4.16] HTTP/1.1 Section 10.4.16 - Unsupported Media Type

[HTTP RFC2616 - Section 10.5.1] HTTP/1.1 Section 10.5.1 - Internal Server Error

[HTTP RFC2616 - Section 12] HTTP/1.1 Section 12 - Content Negotiation

[HTTP RFC2616 - Section 13] HTTP/1.1 Section 13 - Caching

[HTTP RFC2616 - Section 14.1] HTTP/1.1 Section 14.1 - Accept Header

[HTTP RFC2616 - Section 14.17] HTTP/1.1 Section 14.17 - Content-Type Header

[HTTP RFC2616 - Section 14.30] HTTP/1.1 Section 14.30 - Redirection Response Headers

[HTTP RFC2616 - Section 14.43] HTTP/1.1 Section 14.43 - User-Agent Header

[HTTP RFC2616 - Section 15] HTTP/1.1 Section 15 - Security Considerations

[HTTP RFC2617] - HTTP Authentication

[RFC1123] - Requirements for Internet Hosts

7 Non-Normative References

[REST] R. Fielding, UC Irvine Doctoral Thesis: "Architectural Styles and the Design of Network-based Software Architectures"

[MediaType] Wikipedia: Internet Media Type



8 Appendix - Ancillary Web Services (Beyond DAP4)

DAP4 servers MAY offer a number of other services that, while not exactly DAP4 services per say, are commonly available. This section lists some of the alternate services a DAP4 server might provide. By enumerating the available services in the Dataset Services Response servers can easily make software clients and the people that use them aware of the server's ancillary capabilities.

8.1 DAP4: HTML DATA Request Form Service

The HTML DATA Request Form Service provides browser based access to the Dataset. When invoked it returns a web-browser renderable document (in html) that provides a form (or other UI) that can be used to constrain and request data in accordance with the DAP4 specification as applied to the dataset .


suffix = .html
service url = dataset_url + .html
role id = http://services.opendap.org/dap4/data-request-form#

Default/primary media type: text/html | text/xhtml

8.2 DAP4: RDF Service

The RDF service provides an RDF representation of the Dataset document (DDX). The RDF response is an XML document containing an RDF version of the DAP4: Dataset Response.


suffix = .rdf
service url = dataset_url + .rdf
role id = http://services.opendap.org/dap4/rdf#

Default/primary media type: application/rdf+xml

8.3 DAP4: ISO 19115 Service

This service provides ISO 19115 metadata for the Dataset, if any can be found. When invoked it returns an XML document containing ISO 19115 metadata located in the DAP4: Dataset Response.


suffix = .iso
service url = dataset_url + .iso
role id = http://services.opendap.org/dap4/iso-19115-metadata#

Default/primary media type: text/xml

8.4 DAP4: ISO Conformance Score Service

This service provides a browser renderable document that describes how well the metadata held in the Dataset conforms to ISO 19115. When invoked this service returns a browser renderable document that scores how well the metadata held in the Dataset Response conforms to ISO 19115.


suffix = .rubric
service url = dataset_url + .rubric
role id = http://services.opendap.org/dap4/iso-19115-score#

Default/primary media type: text/html

8.5 DAP4: NetCDF File-out Service

This service provides data responses in NetCDF-4 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 4 file.

suffix = .nc4
service url = dataset_url + '.nc4' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf-4

8.6 DAP4: ASCII Data Service

This service provides data responses in ASCII format. When invoked the regular DAP data response will be repackaged as an ASCII representation of the data values.


suffix = .ascii
service url = dataset_url + '.ascii' + [?dap_constraint]
role id = http://services.opendap.org/dap4/ascii#

Default/primary media type: text/plain

8.7 DAP4: XML Data Service

This service provides data responses in XML format. When invoked the constrained Dataset response document (DDX) will be marked up with the data values of the request and returned. Large requests may be denied.


suffix = .xdap
service url = dataset_url + '.xdap' + [?dap_constraint]
role id = http://services.opendap.org/dap4/xml-data#

Default/primary media type: text/xml

8.8 DAP4: Native File Access Service

This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in. Servers MAY elect to not support this response, for example, for generated data, very large data, et cetera.


suffix = .file
service url = dataset_url + .file
role id = http://services.opendap.org/dap4/file#

Default/primary media type: Type varies with file type.

8.9 DAP4: Server Version Service

This service provides software versioning information. When invoked the service returns an XML file containing a description of the version of the server and it's components.


suffix = .ver
service url = dataset_url + .ver
role id = http://services.opendap.org/dap4/version#

Default/primary media type: text/xml

8.10 DAP2 Services

In order to support legacy client applications DAP4 server implementations MAY support the DAP2 services stack. If they do so the DAP2 services MUST be organized as described in this section.


8.10.1 DAP2: Data Service

The DAP2 data service provides DAP2 data access to the data resource.


suffix = .dods
service url = dataset_url + '.dods' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dods#

Default/primary media type: application/octet-stream

8.10.2 DAP2: DDX Service

The DAP2 DDX service provides DAP2 access to the data resource metadata. When invoked the service returns an XML document containing both syntactic and semantic dataset metadata in DAP2 XML format.

suffix = .ddx
service url = dataset_url + .ddx
role id = http://services.opendap.org/dap2/ddx#

Default/primary media type: text/xml

8.10.3 DAP2: DDS Service

The DAP2 DDS service provides access to the 'syntactic' metadata (aka use or structural metadata) for the data resource. When invoked returns a DAP2 DDS response document conforming to the DDS part of the DAP2 specification.


suffix = .dds
service url = dataset_url + '.dds' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dds#

Default/primary media type: text/plain

8.10.4 DAP2: DAS Service

The DAP2 DAS service provides access to the 'semantic' metadata (aka domain metadata) for the data resource. When invoked returns a DAP2 DAS response document conforming to the DAS part of the DAP2 specification.


suffix = .das
service url = dataset_url + .das
role id = http://services.opendap.org/dap2/das#

Default/primary media type: text/plain


8.10.5 DAP2: ASCII Data Service

This service provides DAP2 data responses in ASCII format. When invoked the regular DAP2 data response will be repackaged as an ASCII representation of the data values.


suffix = .ascii
service url = dataset_url + '.ascii' + [?dap_constraint]
role id = http://services.opendap.org/dap2/ascii#

Default/primary media type: text/plain

8.10.6 DAP2: JSON Data Service

This service provides DAP2 data responses in w10n JSON format. When invoked the regular DAP2 data response will be repackaged as an w10n JSON representation of the data values.


suffix = .json
service url = dataset_url + '.json' + [?dap_constraint]
role id = http://services.opendap.org/dap2/json#

Default/primary media type: text/plain

8.10.7 DAP2: Info Service

The DAP2 INFO service provides a browser renderable page that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way. When invoked this service returns a web browser renderable document that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way.


suffix = .info
service url = dataset_url + .info
role id = http://services.opendap.org/dap2/Info#

Default/primary media type: text/html

8.10.8 DAP2: NetCDF Service

This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.

suffix = .nc
service url = dataset_url + '.nc' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf


Template: ServiceTemplate