DAP4 Web Services v3

From OPeNDAP Documentation
Revision as of 22:28, 23 July 2012 by EthanDavis (talk | contribs) (Change to prefer DAP4 over more general media types.)
⧼opendap2-jumptonavigation⧽

<< Back to OPULS Development

Abstract

The OPeNDAP Data Access Protocol 4.0 (DAP4) Web Service specifies how DAP4 clients and servers communicate over HTTP. The core DAP4 resource types and their representations are documented in the following DAP4 documents:

  • DAP4 Data Model
  • DAP4 Constraint Expressions
  • DAP4 DDX and XML Encoding
  • DAP4 DataDDX and binary Encoding

Several types of extensions can be made to the DAP4 core including:

  • development of new encodings for the core DAP4 resource types, and
  • development of new server-side functions.

Overview

DAP4 is a stateless protocol that governs clients making requests from servers, and servers issuing responses to those requests. These messages are used to request information about a server and data made accessible by that server, as well as requesting data values themselves.

This document provides a detailed discussion of the web services (i.e. the structure of the request and response messages) which DAP-compliant software MUST support when utilizing the HTTP protocol to transmit DAP requests and responses, along with additional optional services that it SHOULD support.

The DAP uses three basic responses to represent a data resource. One response, the DDX returns metadata information describing the structure of the data resource. That is, it characterizes the variables, their datatypes, names and attributes. The second response, the DataDDX, returns both the metadata about the request, but also the data that was requested. The DDX and the metadata part of the DataDDX are represented using a specific XML[] representation. The syntax of that representation is defined in Section ?. The third basic response is a DatasetServices response which is a document that provides a listing of services, any alternate media representations if available, and all of the associated access URI's for a particular data resource.

The DAP returns error information using an Error response. If a request for any of the three basic responses cannot be completed then an Error response is returned in its place. The two responses (DDX and DataDDX) are complete in and of themselves so that, for example, the data response can be used by a client without ever requesting either of the two other responses. In many cases, client programs will request the DDX response first before requesting the DataDDX response but there is no requirement they do so and no server SHALL require that behavior on the part of clients. Operationally, communication between a DAP client and a DAP server utilizes some underlying already existing protocol. This as previously mentioned, document defines the way in which the DAP protocol utilizes the HTTP protocol..

These services are, for the most part, RESTful services (Fielding) and as such are stateless unless noted otherwise.

HTTP

The DAP4 Web Services design is intended to make easily available all of the Web Services and their various media-type representations to a wide range of clients.

Media Types

DAP4 web services MAY provide responses in a number of different media-types. DAP4 responses SHOULD use the "Content-Type" header field to define the media type of the DAP4 response body.

The value of the "Content-Type" header field SHOULD be the value specified in the media types specification. For example, the value for an XML encoded DDX response would be "application/vnd.org.opendap.ddx+xml". However, servers MAY decide to return a more general media type (e.g., "text/xml") if something in the request (e.g., the "Accept" or "User-Agent" header values) indicates the client might be better able to handle a more general media type. Servers should use caution when downgrading the response media type:

It is impossible for the server to accurately determine what might be "best" for any given user, since that would require complete knowledge of both the capabilities of the user agent and the intended use for the response (e.g., does the user want to view it on screen or print it on paper?).

Content Negotiation

DAP4 servers are expected to engage in server-driven content negotiation as described in the HTTP 1.1 specification, which means that the client can request a particular media-type representation of a resource (such as application/json) using the HTTP Accept header, and the server will either comply, or it will provide the default media-type representation for the requested resource.

Agent-driven negotiation is also supported in that the DAP4 Web Services Terminus will provide links (URIs) for all of the various services and alternate media representations for a particular data resource. This is in keeping with the HTTP 1.1 specification for agent-driven content negotiation which states:

Selection is based on a list of the available representations of the response included within the header fields or entity-body of the initial response, with each representation identified by its own URI.

Where we see the initial response as being the DAP Service Terminus response.

Headers

Request Headers

General
Accept
The HTTP Accept header MAY be used by clients that wish to engage in server-dirven content negotiation by requesting a particular representation of a resource in the initial request.
DAP Specific
There are no DAP specific headers required to make a general DAP request.
Asynchronous Transaction
X-DAP-Async-Accept
A client MAY indicate it's willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header. Clients MAY make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the X-DAP-Async-Accept HTTP header with a value given in seconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.

Response Headers

These are the HTTP response headers that DAP servers using HTTP MUST and SHOULD (as indicated) return.

General
Date
DAP4 servers MUST return an HTTP Date header whose value is the time of the response and which MUST be in RFC 1123 format.
Last-Modified
DAP4 servers SHOULD return an HTTP Last-Modified header whose value is the last modified time of the request resource and which MUST be in RFC 1123 format.
Content-Type
DAP4 servers MUST return an HTTP Content-Type header, the value of which SHOULD be set in accordance with the Media Types discussion above.
Content-Description
DAP4 servers SHOULD return an HTTP Content-Description header.
Content-Disposition
DAP4 servers SHOULD return an HTTP Content-Disposition header when transmitting file typed responses.
Content-Encoding
DAP4 servers MUST return an HTTP Content-Encoding header when the content-coding of an entity is not "identity".


DAP Specific
X-DAP-Server
DAP4 servers SHOULD return the X-DAP-Server HTTP header. This HTTP header is used in a response to communicate the software version of the server. This may be a simple string with the server name and version number, or multiple software component versions may be represented. The value of this header is a string determined by the implementations author(s).
Example
X-DAP-Server: bes/3.10.0, libdap/3.11.2, dap-server/ascii/4.1.2, csv_handler/1.0.2, freeform_handler/3.8.4, fits_handler/1.0.7, fileout_netcdf/1.1.2, gateway_module/1.1.0, hdf4_handler/3.9.4, hdf5_handler/1.5.0, netcdf_handler/3.10.0, ncml_module/1.2.1, dap-server/usage/4.1.2, dap-server/www/4.1.2, xml_data_handler/1.0.1
Example
X-DAP-Server: TDS-4.19.3
X-DAP
DAP4 servers MUST return the X-DAP HTTP header. This HTTP header is used in a response to indicate the DAP protocol version used to encode the content of the response. This value is constrained to a format of "major_version" dot "minor version", where both major_version and minor_version are represented by an integer value.
Example
X-DAP: 4.0
Example
X-DAP: 2.17
Asynchronous Transaction
X-DAP-Async-Required
DAP4 servers MUST return the X-DAP-Async-Required header if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the 400 DAP Asynchronous Response Required HTTP response code.

Status Codes

DAP servers that provide an HTTP interface are expected to utilize the HTTP response codes in a manner consistent with the HTTP 1.1 specification.

The ones that are detailed here are used by the DAP in a manner consistent with the specifications definition, but in support of specific DAP server behavior.

200 OK

A server indicates that the request has been successful and that the returned document contains the requested resource. However since DAP responses can be quite large and since it is also possible for the server to encounter any number of problems during the marshaling, serialization, and subsequent transmission of the response it is possible that the server may have committed/transmitted the HTTP headers (in which the status value is transmitted) before a subsequent error is encountered. These types of errors are transmitted in the DAP4 over-the-wire protocol and all DAP4 clients MUST be able to read and process these errors. See Section?

202 Accepted

A server indicates that a request has been accepted and will be handled asynchronously by returning a 202 Accepted HTTP response code. The response body MUST contain a DAP asynchronous response document.

400 Bad Request

The HTTP specification defines this status code as:

The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

DAP servers utilizes this code to mean one of two things.

400 Bad DAP Request Syntax

The 400 Bad DAP Request Syntax HTTP response code is used to indicate that is a problem with the syntax of the OPeNDAP URL. The problem could be in the formulation of the constraint expression, or it may be that the URL extension did not match any that are known by this server

400 DAP Asynchronous Response Required

The 400 DAP Asynchronous Response Required HTTP response code is used to indicate that the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response.

401 Unauthorized

The 401 Unauthorized HTTP response code indicates that the request requires user authentication. See the HTTP specification for usage and behavior.

403 Forbidden

The 403 Forbidden HTTP response code indicates that the server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. This is appropriate to return if, for example, the server software does not have read permission for the requested resource.

404 Not Found

The 404 Not Found HTTP response code indicates that the server has not found anything matching the Request-URI.

409 Conflict (DAP Async Response Not Available)

The 409 Conflict HTTP response code is used to indicate that the DAP request has been rejected because a previous asynchronous request has not been completed and the result is not ready for access.

412 Precondition Failed

The 412 Precondition Failed HTTP response code is used to indicate that the DAP request has been rejected because it did not meet the X-DAP-Async-Accept-If-Expected-Delay-Less-Than condition that was specified in the request.

500 Internal Server Error

DAP (and many other servers) may encounter problems during their efforts to

501 Not Implemented

Hmmm.... I added this because I had hooks for it in Hyrax, although further investigation leads me to believe that it's cruft.

502 Bad Gateway

Ibid.

Verbs (aka Methods)

GET

A DAP request may be made using the HTTP GET request method utilizing a Uniform Resource Identifier (URI) that encodes information specific to the DAP (see Section ?).

Each GET request MUST conform to the HTTP specification (which basically says that a GET request MUST contain an HTTP protocol version number followed by a MIME-like message containing various headers that further describe the request.). While there are some optional DAP HTTP request headers that may be used, DAP requests do not require specific HTTP headers beyond those necessary for HTTP. DAP servers SHOULD support the use of the HTTP Accept request header to allow clients to engage in HTTP content negotiation for specific representations of a DAP response.

Implementation Note
In practice, DAP clients typically use a third-party library implementation of HTTP/1.1 so the GET request, URI and HTTP version information are hidden from the client; it sees only the DAP Uniform Resource Locator (URL) and the request headers.

The DAP server responds to the GET request with an HTTP compliant response (one that includes a status line containing the HTTP protocol version and an error or success code, followed by HTTP response headers and then response itself). There are two DAP specific HTTP headers that are always included in a DAP response over HTTP: X-DAP-Server and X-DAP, as described above. The DAP response is the payload of the HTTP response. Unless otherwise negotiated, the data response payload is encoded in multpart-MIME format, while the other responses are simple HTTP responses This is further described in Section ?.

Examples
HTTP GET request.
GET /dap/path/data.nc.dap?u,v[0:8:1024] HTTP/1.1
Host: server.org
Accept: application/x-dap4


Web Services

The DAP web services are all defined as extensions to the data resource (aka base) URL. This data resource URL essentially becomes a prefix (and could even be seen as a namespace) for all of the services available for that data resource.

A Service is represented by:

  • A unique "namespace" like identifier that is used to define an xlink:role attribute.
  • A simple human readable name called 'title'.
  • An access URL and a media-type describing the response.
  • An optional description.

A service is accessed by dereferencing its access URL, which is typically constructed by adding some type of suffix to the data resource URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.


Primary DAP4 Services

All of the example requests described below are based on the data resource URL:

http://server.org:8080/dap/path/data.nc
Note regarding JSON responses

In the following we describe two media types in most cases: the primary media type (e.g., test/xml) and application/json. We have not defined how those JSON, BJSON, Universal binary JSON, ..., responses will be formatted.

For the JSON media type for the data response, we might look at http://ubjson.org/. However, I think JSON support is one place where we could get some solid input from the community.

DAP4: Dataset Web Services

The DAP Dataset Web Services response provides a listing of the various web services and their associated media-type representations available for a dataset. Dereferencing the base URL for a dataset accessed through a DAP4 web server will return a DatasetServices document that describes the various responses possible for that data set, including URLs that can be used to access those responses.

More information on the Service Description response can be found on the DAP4 Service Terminus v2 page.


service suffix =
service url = dataset_url
role id = http://services.opendap.org/dap4/dataset-services#

Default/primary media type:

  • text/xml

Optional media type(s):

  • application/json
url = dataset_url.json
HTTP Request Examples

Simple

This simple request will return the XML representation of the Dataset Web Services response.
GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Server-driven content negotiation

Client requests the Dataset Web Services response and indicates that it wants the JSON representation. There server SHOULD reply with the JSON representation if it is available, otherwise it may return it's default XML representation.
GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: application/json

Agent-driven content negotiation

Client requests the JSON representation of the Dataset Web Services response. The server MUST reply with the JSON representation if available, or with a status of 404 if not..
GET /dap/path/data.nc.json HTTP/1.1
Host: server.org:8080
Accept: */*

DAP4: Dataset Metadata Service

The Dataset Metadata Service returns a dataset description document (DDX) which is a metadata description of the dataset. The DDX response is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset, persisted as a DAP4: Data Model representation of the dataset held at the server. The Dataset metadata service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied. If a constraint expression has been successfully applied, the service will returned the constrained view of the dap:Dataset object. The constrained view may contain different data structures than the unconstrained view as the constraint may alter the reasonable representation of the data set. Note that All dap:Attribute objects have been removed from constrained dap:Dataset objects.

  • More information on the syntax of DAP4 constraint expressions can be found on DAP4: Data Model page.


service suffix = .xdp
service url = dataset_url + '.xdp' + [?dap_constraint]
role id = http://services.opendap.org/dap4/dataset#

Default/primary media type:

  • text/xml

Optional media type(s):

  • application/json
url = dataset_url.xdp.json[?dap_constraint]


HTTP Request Examples

Simple

This simple request will return the XML representation of the dataset metadata response.
GET /dap/path/data.nc.xdp HTTP/1.1
Host: server.org:8080
Accept: */*


Server-driven content negotiation

Client requests the dataset metadata response and indicates that it wants the JSON representation. There server SHOULD reply with the JSON representation if it is available, otherwise it may return it's default XML representation.
GET /dap/path/data.nc.xdp HTTP/1.1
Host: server.org:8080
Accept: application/json

Agent-driven content negotiation

Client requests the JSON representation of the dataset metadata response. The server MUST reply with the JSON representation if available, or with a status of 404 if not..
GET /dap/path/data.nc.xdp.json HTTP/1.1
Host: server.org:8080
Accept: */*

DAP4: Data Service

The Data Service provides DAP4 data access to a dataset, and is the way that DAP4 returns data to a client. The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions. When the service is invoked it returns the DAP4 data object. On the wire this is a multipart MIME document where the first MIME part contains the constrained Dataset response describing the data requested and the following MIME parts contain the data values, encoded using XDR, followed by a checksum, for each variable in the dataset.

More information on the Data response can be found on the DAP4: Responses page.

suffix = .dap
service url = dataset_url + '.dap' + [?dap_constraint]
role id = http://services.opendap.org/dap4/data#

Default/primary media type:

  • multipart/mixed
    • First Part: text/xml
    • Second Part: application/octet-stream

Optional media type(s):

  • application/json This Needs Elaboration - what do we mean here? A single JSON document? Replacing the DDX in part 1 with JSON?
url = dataset_url.dap.json[?dap_constraint]


HTTP Request Examples

Simple

This simple request will return the full contents of the dataset resource as a DAP4 data object.
GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: */*


Server-driven content negotiation

Client requests the DAP4 data response and indicates that it wants the JSON representation. There server SHOULD reply with the JSON representation if it is available, otherwise it may return it's default XML representation.
GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: application/json

Agent-driven content negotiation

Client requests the JSON representation of the DAP4 data response. The server MUST reply with the JSON representation if available, or with a status of 404 if not..
GET /dap/path/data.nc.dap.json HTTP/1.1
Host: server.org:8080
Accept: */*

Other DAP4 Services

DAP4: HTML DATA Request Form Service

The HTML DATA Request Form Service provides browser based access to the Dataset. When invoked it returns a web-browser renderable document (in html) that provides a form (or other UI) that can be used to constrain and request data in accordance with the DAP4 specification as applied to the dataset .


suffix = .html
service url = dataset_url + .html
role id = http://services.opendap.org/dap4/data-request-form#

Default/primary media type: text/html | text/xhtml

DAP4: RDF Service

The RDF service provides an RDF representation of the Dataset document (DDX). The RDF response is an XML document containing an RDF version of the DAP4: Dataset Response.


suffix = .rdf
service url = dataset_url + .rdf
role id = http://services.opendap.org/dap4/rdf#

Default/primary media type: application/rdf+xml

DAP4: ISO 19115 Service

This service provides ISO 19115 metadata for the Dataset, if any can be found. When invoked it returns an XML document containing ISO 19115 metadata located in the DAP4: Dataset Response.


suffix = .iso
service url = dataset_url + .iso
role id = http://services.opendap.org/dap4/iso-19115-metadata#

Default/primary media type: text/xml

DAP4: ISO Conformance Score Service

This service provides a browser renderable document that describes how well the metadata held in the Dataset conforms to ISO 19115. When invoked this service returns a browser renderable document that scores how well the metadata held in the Dataset Response conforms to ISO 19115.


suffix = .rubric
service url = dataset_url + .rubric
role id = http://services.opendap.org/dap4/iso-19115-score#

Default/primary media type: text/html

DAP4: NetCDF File-out Service

This service provides data responses in NetCDF-4 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 4 file.

suffix = .nc4
service url = dataset_url + '.nc4' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf-4

DAP4: ASCII Data Service

This service provides data responses in ASCII format. When invoked the regular DAP data response will be repackaged as an ASCII representation of the data values.


suffix = .ascii
service url = dataset_url + '.ascii' + [?dap_constraint]
role id = http://services.opendap.org/dap4/ascii#

Default/primary media type: text/plain

DAP4: XML Data Service

This service provides data responses in XML format. When invoked the constrained Dataset response document (DDX) will be marked up with the data values of the request and returned. Large requests may be denied.


suffix = .xdap
service url = dataset_url + '.xdap' + [?dap_constraint]
role id = http://services.opendap.org/dap4/xml-data#

Default/primary media type: text/xml

DAP4: Native File Access Service

This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in.


suffix = .file
service url = dataset_url + .file
role id = http://services.opendap.org/dap4/file#

Default/primary media type: Type varies with file type.

DAP4: Server Version Service

This service provides software versioning information. When invoked the services returns an XML file containing a description of the version of the server and it's components.


suffix = .ver
service url = dataset_url + .ver
role id = http://services.opendap.org/dap4/version#

Default/primary media type: text/xml

DAP2 Services

DAP2: Data Service

The DAP2 data service provides DAP2 data access to the data resource.


suffix = .dods
service url = dataset_url + '.dods' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dods#

Default/primary media type: application/octet-stream

DAP2: DDX Service

The DAP2 DDX service provides DAP2 access to the data resource metadata. When invoked the service returns an XML document containing both syntactic and semantic dataset metadata in DAP2 XML format.

suffix = .ddx
service url = dataset_url + .ddx
role id = http://services.opendap.org/dap2/ddx#

Default/primary media type: text/xml

DAP2: DDS Service

The DAP2 DDS service provides access to the 'syntactic' metadata (aka use or structural metadata) for the data resource. When invoked returns a DAP2 DDS response document conforming to the DDS part of the DAP2 specification.


suffix = .dds
service url = dataset_url + '.dds' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dds#

Default/primary media type: text/plain

DAP2: DAS Service

The DAP2 DAS service provides access to the 'semantic' metadata (aka domain metadata) for the data resource. When invoked returns a DAP2 DAS response document conforming to the DAS part of the DAP2 specification.


suffix = .das
service url = dataset_url + .das
role id = http://services.opendap.org/dap2/das#

Default/primary media type: text/plain

DAP2: Info Service

The DAP2 INFO service provides a browser renderable page that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way. When invoked this service returns a web browser renderable document that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way.


suffix = .info
service url = dataset_url + .info
role id = http://services.opendap.org/dap2/Info#

Default/primary media type: text/html

DAP2: NetCDF Service

This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.

suffix = .nc
service url = dataset_url + '.nc' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf


Template: ServiceTemplate

Retrieved from "https://docs.opendap.org/index.php?title=DAP4_Web_Services_v3&oldid=8821"