Difference between revisions of "DAP4 Web Services v3"

From OPeNDAP Documentation
(Created page with " <-- back to OPULS Development <<author>> == Background == == Problem addressed == == Proposed solution == == Rationale for the solution =...")
 
 
(476 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[Development#OPULS_Development | <-- back to OPULS Development]]
+
[[Category:Development|Development]][[Category:DAP4|DAP4]]
 +
[[OPULS_Development| << Back to OPULS Development]]
  
<<author>>
+
<font size="+2" color="red"> This page is out of date and will not be updated.  This is an old document that captures an intermediate point of the OPULS design work. This page should be referenced only as part of a baseline for the work. It has been superseded by the  ''[[DAP4: Specification Volume 2 | DAP4 Specification Volume 2: Web Services Specification]]'' </font>
  
== Background ==
+
== Introduction ==
  
== Problem addressed ==
+
This document defines the DAP4 Web Service protocol which DAP4-compliant software MUST support when utilizing the HTTP protocol to transmit DAP4 requests and responses, along with additional optional services that DAP4-compliant software SHOULD support.
  
== Proposed solution ==
+
The DAP4 protocol uses three basic responses to represent a data resource.  One response, the Dataset Metadata Response (DMR) contains metadata information describing the structure of the data resource. That is, it characterizes the variables, their datatypes, names and attributes. The second response, the Data Response (DataDMR), contains both the metadata about the requested data and the actual data that was requested. The third basic response is the Dataset Services Response (DSR) that provides a listing of services, any alternate media representations if available, and all of the associated access URI's for a particular data resource.
  
== Rationale for the solution ==
+
The DAP4 protocol uses a fourth basic resource, the Constraint/Query Expression (CE), to represent subsetting of and, possibly, transformations of the dataset requested.
  
== Discussion ==
+
The DAP4 protocol uses a fifth basic resource, the Error Response (ER), to allow servers to communicate error information with clients. When a request for any of the three basic responses cannot be completed, an Error response is returned. If an error occurs before the standard response is initiated, an error response is returned in place of the standard response. If an error occurs after a data response has been initiated, an Error Response is returned as the final chunked record as described in Section 7 "DAP4 Chunked Data Representation" of the DAP4 Data Model document <nowiki>[</nowiki>[[#DAP4_Vol1|DAP4_Vol1]]<nowiki>]</nowiki>.
  
 +
The DAP4 Data Model, constraint/query language, dataset metadata XML encoding, and binary data encoding are all defined in the DAP4 Volume 1: "Data Model, Persistent Representations, and Constraints" document <nowiki>[</nowiki>[[#DAP4_Vol1|DAP4_Vol1]]<nowiki>]</nowiki>.
  
 +
Each of the three responses (Dataset Metadata, Data, and Dataset Services) are complete in and of themselves so that, for example, the Data response can be used by a client without ever requesting either of the two other responses. In many cases, client programs will request the Dataset Metadata response before requesting the Data response but there is no requirement they do so and servers SHALL NOT require that behavior on the part of clients.
  
== Protocol Independence ==
+
These three standard dataset responses can each be accessed in two different ways. First, similar to DAP2 URL construction and to support client-driven content negotiation (see the 3.? "Content Negotiation" section below), each dataset resource has a standard URL suffix that can be added to the DAP4 dataset URL to retrieve the resource in its standard (normative) encoding. This allows clients, given a base DAP4 dataset URL, to construct DAP4 URLs in a simple and standard way for each of the three standard dataset resources.
  
'''Proposal''': The  DAP4 data model and it's persistent (over-the-wire) representations are transport protocol agnostic. All of the information required by the DAP must be present in the content of the DAP requests and responses. The DAP won't uniquely embed information required by the DAP into HTTP Headers or AMQP thingys, or other whatnot.  
+
Second, to support server-driven content negotiation (see the 3.? "Content Negotiation" section below) and a more hypermedia-driven style, a DAP4 resource role is defined for each standard dataset resource and a DAP4 media type is defined for each dataset resource encoding scheme.
 +
 
 +
Any particular instance of a DAP4 server MAY have one or more additional services, alternate media representations of service responses, or unique (to the server instance) server side functions. All of these things, including the core services and their default representations MUST be included in the [[DAP4_Dataset_Services_Response | Dataset Services Response]].
 +
 
 +
The DAP4 web service is currently limited to HTTP GET requests though it is expected that extensions (e.g., asynchronous access) will use other HTTP methods (e.g., POST). This makes, for now, the DAP4 Constraint Expression something of a pseudo-resource type given that they are encoded as part of the URL query string rather than as an independent document.
 +
 
 +
=== DAP4 Resource Roles and Media Types ===
 +
 
 +
The standard DAP4 dataset resource roles and encodings (plus a few alternate encodings) that are defined in the core DAP4 documents are:
 +
 
 +
'''Dataset Services Response (DSR)'''
 +
 
 +
<blockquote>
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="text-align: left;"| Resource Role
 +
|-
 +
|
 +
: '''<nowiki>http://services.opendap.org/dap4/dataset-service</nowiki>'''
 +
|}
 +
 
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="width: 15%" | URL Suffix
 +
! style="width: 55%" | Media Type
 +
! style="width: 30%" | URL Example
 +
|-
 +
| "" or "'''.dsr'''"
 +
|
 +
; application/vnd.opendap.org.dap4.dataset-services+xml
 +
: Normative DSR encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>
 +
<nowiki>http://server/path/dataset.nc</nowiki>'''.dsr'''
 +
|-
 +
| "'''.xml'''" or "'''.dsr.xml'''"
 +
|
 +
; text/xml
 +
: Normative DSR encoding with generic media type
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.xml'''
 +
<nowiki>http://server/path/dataset.nc</nowiki>'''.dsr.xml'''
 +
|-
 +
| "'''.html'''" or "'''.dsr.html'''"
 +
|
 +
; text/html
 +
: HTML DSR encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.html'''
 +
<nowiki>http://server/path/dataset.nc</nowiki>'''.dsr.html'''
 +
|}
 +
</blockquote>
 +
 
 +
'''Dataset Metadata Response (DMR)'''
 +
 
 +
<blockquote>
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="text-align: left;"| Resource Role
 +
|-
 +
|
 +
: '''<nowiki>http://services.opendap.org/dap4/dataset-metadata</nowiki>'''
 +
|}
 +
 
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="width: 15%" | URL Suffix
 +
! style="width: 55%" | Media Type
 +
! style="width: 30%" | URL Example
 +
|-
 +
| "'''.dmr'''"
 +
|
 +
; application/vnd.opendap.org.dap4.dataset-metadata+xml
 +
: Normative DMR encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dmr'''
 +
|-
 +
| "'''.dmr.xml'''"
 +
|
 +
; text/xml
 +
: Normative DMR encoding with generic media type
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dmr.xml'''
 +
|-
 +
| "'''.dmr.html'''"
 +
|
 +
; text/html
 +
: HTML DMR encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dsr.html'''
 +
|}
 +
</blockquote>
 +
 
 +
'''Dataset Data Response (Data)'''
 +
 
 +
<blockquote>
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="text-align: left;"| Resource Role
 +
|-
 +
|
 +
: '''<nowiki>http://services.opendap.org/dap4/data</nowiki>'''
 +
|}
 +
 
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="width: 15%" | URL Suffix
 +
! style="width: 55%" | Media Type
 +
! style="width: 30%" | URL Example
 +
|-
 +
| "'''.dap'''"
 +
|
 +
; application/vnd.opendap.org.dap4.data
 +
: Normative Data encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dap'''
 +
|-
 +
| "'''.dap.txt'''"
 +
|
 +
; text/plain
 +
: Text (UTF-8) Data encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dap.txt'''
 +
|-
 +
| "'''.dap.xml'''"
 +
|
 +
; text/xml
 +
: XML Data encoding
 +
| <nowiki>http://server/path/dataset.nc</nowiki>'''.dap.xml'''
 +
|}
 +
</blockquote>
 +
 
 +
'''Error Response (ER)'''
 +
 
 +
<blockquote>
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="text-align: left;"| Resource Role
 +
|-
 +
|
 +
: '''<nowiki>http://services.opendap.org/dap4/error</nowiki>'''
 +
|}
 +
 
 +
{| class="wikitable" style="font-size: 95%;" width="90%"
 +
! style="width: 15%" | URL Suffix
 +
! style="width: 55%" | Media Type
 +
! style="width: 30%" | URL Example
 +
|-
 +
| N/A
 +
|
 +
; application/vnd.opendap.org.dap4.error+xml
 +
: Normative Error encoding
 +
| N/A
 +
|-
 +
| N/A
 +
|
 +
; text/xml
 +
: Normative Error encoding with generic media type
 +
| N/A
 +
|-
 +
| N/A
 +
|
 +
; text/html
 +
: HTML Error encoding
 +
| N/A
 +
|}
 +
</blockquote>
 +
 
 +
 
 +
=== DAP4 Core Specification Documents ===
 +
 
 +
* DAP4 Volume 1: "Data Model, Persistence Representations, and Constraints"
 +
* DAP4 Web Service (this document)
 +
* DAP4 Dataset Services
 +
* DAP4 Requirements for Server-side Functions
 +
 
 +
=== Extensions to DAP4 Core Specifications ===
 +
 
 +
Several types of extensions can be made to the DAP4 core including:
 +
 
 +
* New encodings for the core DAP4 response types
 +
* New response types
 +
* New server-side functions.
 +
 
 +
== Terms and Definitions ==
 +
 
 +
; Dataset Service Response (DSR)
 +
: The DAP4 response type that contains a list of DAP (and other) services available for the dataset including any alternate media representations and all the associated access URIs.
 +
; Dataset Metadata Response (DMR)
 +
: The DAP4 response type that contains metadata information describing the structure of the requested data. The metadata information characterizes the requested data variables including their names, data types, shapes, and attributes.
 +
; Dataset Data Response (Data)
 +
: The DAP4 response type that contains both the dataset metadata and the binary data for the requested data.
 +
; Resource role ID
 +
: A URI that identifies the role of a resource, generally provided with a link to allow clients to identify the type of resource the link references. (For instance, an "atom:link" element has an optional "atom:rel" attribute.)
 +
; Media Type
 +
: A internet media type is a two-part identifier for resource encoding schemes [http://en.wikipedia.org/wiki/Internet_media_type]. (E.g., "text/html", "text/plain", "application/octet-stream".)
 +
 
 +
==  DAP4 Web Service Responses  ==
 +
 
 +
The core of the DAP4 Web Service protocol consists of the three standard response types: Dataset Services Response (DSR), Dataset Metadata Response (DMR), and Dataset Data Response (Data). Each dataset served by a DAP4 compliant server MUST provide these three response types.
 +
 
 +
All of the example requests described below are based on the DAP4 dataset URL:
 +
: http://server.org:8080/dap/path/data.nc
 +
 
 +
=== [[DAP4 Dataset Services Response]] ===
 +
 
 +
The DAP4 Dataset Services Response (DSR) provides clients with a listing of all available DAP4 services and all the available encodings for those services as well as all available DAP4 extensions.
 +
 
 +
Each service (or response type) has a unique resource role (defined in the appropriate specification), each link (alternate representation) for a given service MUST fulfill that same role. This is not always a clear distinction to make. For example, the DMR can be mapped into ISO 19115 metadata. However, IS0 19115 is clearly a different domain.
 +
 
 +
The Dataset Services Response MUST contain the following information:
 +
 
 +
* List of DAP versions supported by server
 +
* The implementation version (e.g., "TDS 4.3.57" or "Hyrax 1.7.45")
 +
* List of all available DAP4 services for the dataset
 +
* For each DAP4 services listed, a list of all available links each with its corresponding media type
 +
* List of supported extensions
 +
** Resource type extensions
 +
** Media type extensions
 +
** Server-side function extensions
 +
 
 +
If SHOULD contain the following information:
 +
 
 +
* A human readable title for the dataset
 +
* Human readable titles for each service
 +
 
 +
To take advantage of web caching, servers should try to keep DSRs light weight (i.e., quick creation) and as stable as possible.
 +
 
 +
==== DSR Resource Role ====
 +
 
 +
DSRs are identified by the resource role:
 +
 
 +
: '''<font size="2"><code><nowiki>http://services.opendap.org/dap4/dataset-services</nowiki></code></font>'''
 +
 
 +
==== Normative Encoding of the DSR ====
 +
 
 +
The normative XML representation for the Dataset Services Response is defined in the "Normative XML Encoding of the DSR" appendix. The media type for the normative XML representation is
 +
 
 +
: <font size="2"><code>'''application/vnd.opendap.org.dataset-services+xml'''</code></font>
 +
 
 +
==== Service Behavior ====
 +
 
 +
When an HTTP GET request is made on a base DAP4 dataset URL, all DAP4 servers MUST return the normative XML encoding of the DSR given these conditions:
 +
 
 +
* the request "Accept" header contains only the normative XML encoding media type,
 +
* the request "Accept" header equals "*/*", or
 +
* the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DSR.
 +
 
 +
For example, the request:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 
 +
Might result in the following response:
 +
 
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: application/vnd.opendap.org.dataset-services+xml
 +
Date: ...
 +
</source>
 +
 
 +
===== Downgrading the Normative XML Encoding (Required) =====
 +
 
 +
When an HTTP GET request is made on a base DAP dataset URL with the suffix <font size="2"><code>'''.xml'''</code></font><br/> added to it:
 +
 
 +
: request url =  <font size="2"><code>''dataset_url'''''.xml'''</code></font><br/>
 +
 
 +
the response MUST be the normative representation of the DSR along with the HTTP <font size="2"><code>Content-Type</code></font> header set to <font size="2"><code>text/xml</code></font> . For example:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.xml HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 
 +
Might result in the following response:
 +
 
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/xml; charset=utf-8
 +
Date: ...
 +
</source>
 +
 
 +
The normative XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/xml
 +
</source>
 +
 
 +
==== Other Encodings of the DSR ====
 +
 
 +
 
 +
===== HTML Encoding  (Optional) =====
 +
 
 +
When an HTTP GET request is made on a base DAP dataset URL with the suffix <font size="2"><code>'''.html'''</code></font><br/> added to it:
  
'''Discussion''':
+
: request url =  <font size="2"><code>''dataset_url'''''.html'''</code></font><br/>
  
# Does this matter?
+
the server MUST reply with an HTML representation of the DSR, '''or''' return an HTTP status of 404 to indicate that an HTML representation of the DSR is not available. For example:
# Would we need to add new syntax server-side functions etc. to the protocol to enable this?
 
  
== Overview ==
+
<source lang="httpd">
+
GET /dap/path/data.nc.html HTTP/1.1
This page describes the various web services that a DAP4 server must provide. These are for the most part REST services and as such are stateless unless noted otherwise.
+
Host: server.org:8080
 +
Accept: */*
 +
</source>
  
The services are all defined as a modification of the service (aka resource or base) URL. This base URL essentially becomes a prefix (and could even be seen as a namespace) for all of the services available for that data resource.
+
Might result in the following response:
  
In practice these have traditionally been implemented over HTTP. However they could just as easily be pushed over a different protocol, as long as the usage of the URL components remain consistent.  
+
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/html; charset=utf-8
 +
Date: ...
 +
</source>
  
==  Services  ==
+
If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example this request:
  
A Service is represented by:
+
<source lang="httpd">
* A unique "namespace" like identifier that is used to define an xlink:role attribute.
+
GET /dap/path/data.nc HTTP/1.1
* A simple human readable name called 'title'.
+
Host: server.org:8080
* An access URL.
+
Accept: text/html
* An optional description.
+
</source>
  
A service is accessed by dereferencing its access URL, which is typically constructed by adding some type of suffix to  the dataset's referent (aka base) URL. The way in which the query string (aka constraint expression) is used is defined by each service, and there is no requirement for inter-service query string API conformity.
+
Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.
 +
<!--
 +
=== [[DAP4 Dataset Services Response]] (Original) ===
  
=== Primary DAP4 Services ===
+
The DAP4 Dataset Services Response (DSR) provides a listing of the various DAP4 resources and their associated media-type representations available for a dataset. Dereferencing the base URL for a dataset accessed through a DAP4 web server will return a ''DatasetServices'' document that describes the various responses possible for that data set, including URLs that can be used to access those responses. In other words, the Dataset Services response is acquired by dereferencing the data resource (aka base) URL for the dataset and there is no suffix added to the data resource URL prior to that dereferencing.
  
 +
Every DAP4 compliant server MUST return the normative representation (an XML document [[DAP4 Dataset Services Response | as described here]]) of the Dataset Services response when a client attempts to access an unmodified dataset URL. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.
  
==== [[DAP Service Terminus  | DAP4: Dataset Services Description Service]] ====
 
The Dataset Services Service provides a listing of the various services available for a dataset. Dereferencing the base URL for a data set accessed through a DAP4 web server will return a ''service document'' that describes the various responses possible for that data set, including URLs that can be used to access those responses.
 
  
More information on the Service Description response can be found on the [[DAP Service Terminus]] page.
 
  
 +
More information on the normative XML representation for the Dataset Services Response can be found on the [[DAP4 Dataset Services Response]] page.
  
 +
service suffix = <br />
 
service url =  <font size="2"><code>'''dataset_url'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/dataset-services#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/dataset-services#</nowiki>'''</code></font><br/><br/>
  
====  '''[[DAP4: Dataset Service | DAP4: Dataset Service - The metadata]]''' ====
+
DAP4 media type:
 +
*<font size="2"><code>'''application/vnd.opendap.org.dataset-services+xml'''</code></font><br/>
  
 +
Alternate media type(s):
 +
*<font size="2"><code>'''text/html'''</code></font><br/>
 +
:media type suffix = <font size="2"><code>'''.html'''</code></font><br/>
 +
:url =  <font size="2"><code>''dataset_url'''''.html'''</code></font><br/>
 +
*<font size="2"><code>'''text/xml'''</code></font><br/>
 +
:media type suffix = <font size="2"><code>'''.xml'''</code></font><br/>
 +
:url =  <font size="2"><code>''dataset_url'''''.xml'''</code></font><br/>
  
 +
==== HTTP Request Examples ====
  
The Dataset Service provides a metadata description of the dataset. The Dataset response is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset, persisted as a [[DAP4: Data Model]] representation of the dataset held at the server. The Dataset service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied. If a constraint expression has been successfully applied, the service will returned the constrained view of the dap:Dataset object. The constrained view may contain different data structures than the unconstrained view as the constraint may alter the reasonable representation of the data set.
+
''Simple''
Note that [[DAP4:_Responses#Transmitting_Attributes_in_constrained_Dataset_documents | All dap:Attribute objects have been removed from constrained dap:Dataset objects.]]
+
: This simple request will return the normative XML representation of the Dataset Web Services response with the <font size="2"><code>'''Content-Type'''</code></font> header of the response set to <font size="2"><code>'''application/vnd.opendap.org.dataset-services+xml'''</code></font>.
 +
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source></font>
  
* More information on the Dataset response can be found on the [[DAP4: Responses#Dataset Response | DAP4: Responses]] page.
+
''Server-driven content negotiation for the HTML representation.''
 +
: This request for the Dataset Services response indicates that the client wants the HTML representation. The server SHOULD reply with the HTML representation if it is available, otherwise it MAY return it's normative XML representation.
 +
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/html
 +
</source></font>
  
* More information on the syntax of DAP4 constraint expressions can be found on [[DAP4:_Data_Model#Constraint_Expressions | DAP4: Data Model]] page.
+
''Server-driven content negotiation for the XML representation.''
 +
: This request for the the Dataset Services response indicates that the client wants the XML representation. The server MUST reply with the XML representation, and the server should set the <font size="2"><code>'''Content-Type'''</code></font> header to <font size="2"><code>'''text/xml'''</code></font>. While the normative representation of the the Dataset Services response is already an XML document, the normative media type is <font size="2"><code>'''application/vnd.opendap.org.dataset-services+xml'''</code></font> which may be unfamiliar to many generic clients (such as web browsers) and it is quite conceivable that such a client might ask for the more generic  <font size="2"><code>'''text/xml'''</code></font> media type.
 +
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/xml
 +
</source></font>
  
 +
''Agent-driven content negotiation for the HTML representation.''
 +
: This request is for the HTML version of Dataset Services response. The server MUST reply with the HTML representation (with the <font size="2"><code>'''Content-Type'''</code></font> header of the response set to <font size="2"><code>'''text/html'''</code></font>) if available, or with a status of 404 if not.
 +
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc.html HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source></font>
  
  
suffix = <font size="2"><code>'''.xml'''</code></font><br/>
+
''Agent-driven content negotiation for the XML representation.''
service url =  <font size="2"><code>'''dataset_url + '.xml' + [?dap_constraint]'''</code></font><br/>
+
: This request is for the XML version of Dataset Services response. The server MUST response with the XML representation (with the <font size="2"><code>'''Content-Type'''</code></font> header of the response set to <font size="2"><code>'''text/xml'''</code></font>) if available, or with a status of 404 if not.
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/dataset#</nowiki>'''</code></font><br/><br/>
+
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc.xml HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source></font>
  
==== '''[[DAP4: Data Service ]]''' ====
+
-->
The Data Service provides DAP4 data access to a dataset, and is the way that DAP4 returns data to a client.  The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions. When the service is invoked it returns over the wire as a multipart MIME document where the first MIME part contains the ''constrained'' Dataset response describing the data requested and the following MIME parts contain the data values, encoded using XDR,  followed by a checksum, for each variable in the dataset.
+
 
 +
=== Dataset Metadata Response <!--'''[[DAP4:_Responses#Dataset_Metadata_Response | DAP4: Dataset Metadata Response]]'''-->  ===
 +
The Dataset Metadata Service returns the Dataset Metadata Response (DMR) which is a metadata description of the dataset. The normative representation of the DMR  is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset, persisted as a [[DAP4: Data Model]] representation of the dataset held at the server. The DMR service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied. If a constraint expression has been successfully applied, the service will returned the constrained view of the dap:Dataset object. The constrained view may contain different data structures than the unconstrained view as the constraint may alter the reasonable representation of the data set.
 +
Note that [[DAP4:_Responses#Transmitting_Attributes_in_constrained_Dataset_documents | All dap:Attribute objects have been removed from constrained dap:Dataset objects.]] More information on the syntax of DAP4 constraint expressions can be found on [[DAP4:_Data_Model#Constraint_Expressions | DAP4: Data Model]] page.
 +
 
 +
==== DMR Resource Role ====
 +
 
 +
DMRs are identified by the resource role:
 +
 
 +
: '''<font size="2"><code><nowiki>http://services.opendap.org/dap4/dataset-metadata</nowiki></code></font>'''
 +
 
 +
==== Normative Encoding of the DSR ====
 +
 
 +
The normative XML representation for the Dataset Metadata Response is defined in the  [[DAP4: Responses#Dataset Response | DAP4: Responses]] page.  The media type for the normative XML representation is:
 +
 
 +
: <font size="2"><code>'''application/vnd.org.opendap.dap4.dataset-metadata+xml'''</code></font><br/>
 +
 
 +
(''More information on the syntax of DAP4 constraint expressions can be found on [[DAP4:_Data_Model#Constraint_Expressions | DAP4: Data Model]] page.'' )
 +
 
 +
==== Service Behavior ====
 +
 
 +
Every DAP4 compliant server MUST return the normative representation (an XML document  [[DAP4: Data Model | as described here ]]) of the Dataset Metadata Response when a client attempts to access a dataset URL with the suffix "<font size="2"><code>'''.dmr'''</code></font>" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.
 +
 
 +
When an HTTP GET request is made on a base DAP dataset URL with the suffix <font size="2"><code>'''.dmr'''</code></font> added to it:
 +
 
 +
: request url = <font size="2"><code>'''dataset_url + '.dmr''''+ [?dap_constraint]</code></font><br/>
 +
 
 +
the server MUST reply with an normative representation of the DMR for the (possibly constrained) dataset given these conditions:
 +
 
 +
* the request "Accept" header contains only the normative XML encoding media type (<font size="2"><code>'''application/vnd.org.opendap.dap4.dataset-metadata+xml'''</code></font>),
 +
* the request "Accept" header equals "*/*", or
 +
* the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DMR.
 +
 
 +
For example, the request:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dmr HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 
 +
Might result in the following response:
 +
 
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: application/vnd.opendap.org.dataset-metadata+xml
 +
Date: ...
 +
</source>
 +
 
 +
===== Downgrading the Normative XML Encoding (Required) =====
 +
While the normative representation of the the Dataset Metadata response is already an XML document, the normative media type is <font size="2"><code>'''application/vnd.opendap.org.dataset-metadata+xml'''</code></font> which may be unfamiliar to many generic clients (such as web browsers) and it is quite conceivable that such a client might ask for the more generic  <font size="2"><code>'''text/xml'''</code></font> media type.
 +
 
 +
When an HTTP GET request is made on a DAP DMR URL with the suffix <font size="2"><code>'''.xml'''</code></font><br/> added to it:
 +
 
 +
: request url =  <font size="2"><code>''dataset_url.dmr'''''.xml'''</code></font><br/>
 +
 
 +
the response MUST be the normative representation of the DMR along with the HTTP <font size="2"><code>Content-Type</code></font> header set to <font size="2"><code>text/xml</code></font> . For example:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dmr.xml HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 
 +
Might result in the following response:
 +
 
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/xml; charset=utf-8
 +
Date: ...
 +
</source>
 +
 
 +
The normative XML representation of the DMR MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL ( and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dmr HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/xml
 +
</source>
 +
 
 +
==== Other Encodings of the DMR ====
 +
 
 +
===== HTML Encoding  (Optional) =====
 +
 
 +
When an HTTP GET request is made on a DAP DMR URL with the suffix <font size="2"><code>'''.html'''</code></font><br/> added to it:
 +
 
 +
: request url =  <font size="2"><code>''dataset_url.dmr'''''.html'''</code></font><br/>
 +
 
 +
the server MUST reply with an HTML representation of the DMR, '''or''' return an HTTP status of 404 (or 415)  to indicate that an HTML representation of the DMR is not available. For example:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dmr.html HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 
 +
Might result in the following response:
 +
 
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/html; charset=utf-8
 +
Date: ...
 +
</source>
 +
 
 +
If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DMR. For example this request:
 +
 
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dmr HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/html
 +
</source>
 +
 
 +
Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.
 +
 
 +
=== DAP4: Data Response <!--'''[[DAP4:_Responses#Data_Response | DAP4: Data Response ]]'''--> ===
 +
The Data Service provides DAP4 data access to a dataset, and is the (primary) way that DAP4 returns data to a client.  The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions. When the service is invoked it returns the DAP4 data object. On the wire this is a binary document with MIME type ''application/vnd.opendap.org.dap4.data''. The payload is broken into two logical parts: A DMR-type xml document that describes the data and a BLOB that contains the actual data. For more on the internal structure of the Data response's payload, see [[DAP4: Encoding for the Data Response]] and [[DAP4: Chunked encoding]].
  
 
More information on the Data response can be found on the [[DAP4:_Responses#Data_Response | DAP4: Responses]] page.
 
More information on the Data response can be found on the [[DAP4:_Responses#Data_Response | DAP4: Responses]] page.
  
 +
==== Data Response Resource Role ====
 +
 +
DAP4 Data Responses are identified by the resource role:
 +
 +
: '''<font size="2"><code><nowiki>http://services.opendap.org/dap4/data</nowiki></code></font>'''
 +
 +
==== Normative Encoding of the Data Response ====
 +
 +
The normative XML representation for the Data Response is defined in the  [[DAP4:_Responses#Data_Response | DAP4: Responses]] page.  The media type for the normative XML representation is:
 +
 +
: <font size="2"><code>'''application/vnd.org.opendap.dap4.data'''</code></font><br/>
 +
 +
(''More information on the syntax of DAP4 constraint expressions can be found on [[DAP4:_Data_Model#Constraint_Expressions | DAP4: Data Model]] page.'' )
 +
 +
==== Service Behavior ====
 +
Every DAP4 compliant server MUST return the normative representation ([[DAP4: Data Model | as described here ]]) of the Data Response when a client attempts to access a dataset URL with the suffix "<font size="2"><code>'''.dap'''</code></font>" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.
 +
 +
When an HTTP GET request is made on a base DAP dataset URL with the suffix <font size="2"><code>'''.dap'''</code></font> added to it:
 +
 +
: request url = <font size="2"><code>'''dataset_url + '.dap''''+ [?dap_constraint]</code></font><br/>
 +
 +
the server MUST reply with an normative representation of the (possibly constrained) data response for the dataset given these conditions:
 +
 +
* the request "Accept" header contains only the normative XML encoding media type (<font size="2"><code>'''application/vnd.org.opendap.dap4.data'''</code></font>),
 +
* the request "Accept" header equals "*/*", or
 +
* the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the Data Response.
 +
 +
For example, the request:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
 +
Might result in the following response:
 +
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: application/vnd.opendap.org.data
 +
Date: ...
 +
</source>
 +
 +
==== Other Encodings of the Data Response ====
 +
 +
===== ASCII Encoding  (Optional) =====
 +
When an HTTP GET request is made on a DAP Data Response URL with the suffix <font size="2"><code>'''.ascii'''</code></font><br/> added to it:
 +
 +
: request url =  <font size="2"><code>''dataset_url.dap'''''.ascii'''</code></font><br/>
 +
 +
the server MUST reply with the ASCII representation of the Data Response, '''or''' return an HTTP status of 404 (or 415) to indicate that an ASCII representation of the Data Response is not available. For example:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap.ascii HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
Might result in the following response:
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/plain; charset=utf-8
 +
Date: ...
 +
</source>
 +
 +
If available, the ASCII representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an ASCII encoded Data Response. For example this request:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/plain
 +
</source>
 +
 +
Must return the ASCII representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.
 +
 +
===== XML Encoding  (Optional) =====
 +
 +
When an HTTP GET request is made on a DAP Data Response URL with the suffix <font size="2"><code>'''.xml'''</code></font><br/> added to it:
 +
 +
: request url =  <font size="2"><code>''dataset_url.dap'''''.xml'''</code></font><br/>
 +
 +
the server MUST reply with the XML representation of the Data Response, '''or''' return an HTTP status of 404 (or 415) to indicate that an XML representation of the Data Response is not available. For example:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap.xml HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
Might result in the following response:
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: text/xml; charset=utf-8
 +
Date: ...
 +
</source>
 +
 +
If available, the XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an XML encoded data response. For example this request:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap HTTP/1.1
 +
Host: server.org:8080
 +
Accept: text/xml
 +
</source>
 +
 +
Must return the XML representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.
 +
 +
===== NetCDF Encoding  (Optional) =====
 +
 +
When an HTTP GET request is made on a DAP Data Response URL with the suffix <font size="2"><code>'''.nc'''</code></font><br/> added to it:
 +
 +
: request url =  <font size="2"><code>''dataset_url.dap'''''.nc'''</code></font><br/>
 +
 +
the server MUST reply with the XML representation of the Data Response, '''or''' return an HTTP status of 404 (or 415) to indicate that an XML representation of the Data Response is not available. For example:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap.nc HTTP/1.1
 +
Host: server.org:8080
 +
Accept: */*
 +
</source>
 +
Might result in the following response:
 +
<source lang="httpd">
 +
HTTP/1.1 200 OK
 +
Content-Type: application/x-netcdf
 +
Date: ...
 +
</source>
 +
 +
If available, the NetCDF representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an NetCDF encoded data response. For example this request:
 +
 +
<source lang="httpd">
 +
GET /dap/path/data.nc.dap HTTP/1.1
 +
Host: server.org:8080
 +
Accept: application/x-netcdf
 +
</source>
 +
 +
Must return the NetCDF representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.
 +
 +
=== [[DAP4:_Asynchronous_Request-Response_Proposal_v3  | DAP4 Asynchronous Response]] ===
 +
 +
Some servers may not be able to fulfill DAP4 service requests in a timely manner.
 +
* A  DAP4 server that is retrieving data content from a near-line tape storage subsystem might take several minutes to access a particular data holding.
 +
* A DAP4 server that is providing access to data held in an Amazon Web Services Glacier Vault will have to wait ~4 hours before it can retrieve a particular holding.
 +
In these circumstances the server may return the [[DAP4:_Asynchronous_Request-Response_Proposal_v3 | DAP4 Asynchronous Response]].
 +
 +
==== Asynchronous Response Resource Role ====
 +
 +
DAP4 Asynchronous Responses are identified by the resource role:
 +
 +
: '''<font size="2"><code><nowiki>http://services.opendap.org/dap4/async</nowiki></code></font>'''
 +
 +
==== Normative Encoding of the Asynchronous Response ====
 +
 +
The normative XML representation for the Asynchronous Response is defined in Appendix x "Normative XML Encoding of the Asynchronous Response". The media type for the normative XML representation is:
  
 +
: <font size="2"><code>'''application/vnd.org.opendap.dap4.async.xml'''</code></font>
  
suffix = <font size="2"><code>'''.dap'''</code></font><br/>
+
=== DAP4 Error Response ===
service url = <font size="2"><code>'''dataset_url + '.dap' + [?dap_constraint]'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/data#</nowiki>'''</code></font><br/><br/>
 
  
=== Other DAP4 Services ===
+
The DAP4 protocol returns error information using an Error response. If a request for any of the three basic responses cannot be completed then an Error response is returned in its place.
  
 +
The normative XML representation for the Error Response is defined by the following RELAX-NG schema.
 +
<pre>
 +
<grammar xmlns="http://relaxng.org/ns/structure/1.0"
 +
                xmlns:doc="http://www.example.com/annotation"
 +
                datatypeLibrary="http://xml.opendap.org/datatypes/dap4"
 +
                ns="http://xml.opendap.org/ns/DAP/4.0#"
 +
                >
 +
&lt;start&gt;
 +
  &lt;ref name="errorresponse"/&gt;
 +
&lt;/start&gt;
 +
&lt;define name="errorresponse"&gt;
 +
  &lt;element name="Error"&gt;
 +
    &lt;optional&gt;
 +
      &lt;attribute name="httpcode"&gt;&lt;data type="dap4_integer"/&gt;&lt;/attribute&gt;
 +
    &lt;/optional&gt;
 +
    &lt;optional&gt;
 +
      &lt;interleave&gt;
 +
        &lt;element name = "Message"&gt;&lt;text/&gt;&lt;/Message&gt;
 +
        &lt;element name = "Context"&gt;&lt;text/&gt;&lt;/Message&gt;
 +
        &lt;element name = "OtherInformation"&gt;&lt;text/&gt;&lt;/Message&gt;
 +
      &lt;/interleave&gt;
 +
    &lt;/optional&gt;
 +
  &lt;/element&gt;
 +
&lt;/define&gt;
 +
</pre>
 +
 +
The Error element has one optional attribute: the ''httpcode'' which is a standard HTTP protocol return code indicating the general class of error. When possible, this code should match the return code in the HTTP headers for the response.
 +
 +
The body of the &lt;Error&gt; element may contain any or all of the following
 +
inner elements each containing arbitrary text.
 +
# &lt;Message&gt; &mdash; A short informative message describing the error.
 +
# &lt;Context&gt; &mdash; Information describing the context in which the error occurred: position of a parse error in a constraint expression, for example.
 +
# &lt;OtherInformation&gt; &mdash; Arbitrary additional text information: a Java stack trace, for example.
 +
 +
==== Error Response Resource Role ====
 +
 +
DAP4 Error Responses are identified by the resource role:
 +
 +
: '''<font size="2"><code><nowiki>http://services.opendap.org/dap4/error</nowiki></code></font>'''
 +
 +
==== Normative Encoding of the Error Response ====
 +
 +
The normative XML representation for the Error Response is defined in Appendix x "Normative XML Encoding of the Error Response". The media type for the normative XML representation is:
 +
 +
: <font size="2"><code>'''application/vnd.org.opendap.dap4.error.xml'''</code></font>
 +
 +
== HTTP ==
 +
 +
The DAP4 Web Services specification describes the features of HTTP that are required to be a compliant DAP4 client or server. It does not attempt to describe all aspects of HTTP that DAP4 servers might implement or that DAP4 clients may see in response to DAP4 requests. Similarly, it does not cover all issues related to the performance and scalability of HTTP.
 +
 +
However, the following sections include both DAP4 requirements as well as some suggestions of HTTP features that servers and clients are encouraged to use.
 +
 +
=== Content Negotiation and Media Types ===
 +
 +
Though the DAP4 core specifications only describe one encoding for each type of resource, DAP4 web servers MAY have the ability to provide a given resource in a number of different media types. All media types available for a resource MUST be listed in the DAP4 Dataset Services response document.
 +
 +
DAP4 responses MUST use the "Content-Type" header field to identify the media type of the DAP4 response body. For example, the normative value for the XML encoded DDX response is ''application/vnd.org.opendap.ddx+xml''.
 +
 +
The DAP4 Dataset Services response describes the available services and their media types and provides DAP4 software (client and/or server) with two different mechanisms to negotiate for different kinds of media representations. The first of these is server-driven content negotiation as described in the [http://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html HTTP 1.1 specification, section 12 "Content Negotiation"]. The second is similar to the agent-driven negotiation also described in section 12 of the HTTP 1.1 specification. The difference being that the "list of available representations ... [each with] its own URI" is provided by the DAP4 Dataset Services response rather than in response to an initial request.
 +
 +
Clients need not retrieve the Dataset Services response in order to access the normative representations of either the Dataset Metadata or Data responses, as these responses are required  for every DAP4 server and are mapped to well known URL patterns. If clients attempt to access other representations or other services using agent-driven negotiation without first checking the Dataset Services response, they should be prepared to receive a 404 Not Found response.
 +
 +
When using server-driven negotiation, DAP4 clients are encouraged to, at a minimum, include "Accept" and "User-Agent" headers in their requests and to provide accurate and detailed information in the values of those headers. However, when server-driven negotiation is used, DAP4 servers are not limited to those headers for determining the media type that is returned. DAP4 clients should also be prepared to handle 415 "Unsupported Media type" response codes.
 +
 +
<!--
 +
When using server-driven negotiation DAP4 servers MAY return a more general media type (e.g., "text/xml") if something in the request (e.g., the "Accept" or "User-Agent" header values) indicates the client might be better able to handle a more general media type. Servers should use caution when downgrading the response media type:
 +
 +
<blockquote>
 +
  It is impossible for the server to accurately determine what
 +
  might be "best" for any given user, since that would require
 +
  complete knowledge of both the capabilities of the user agent
 +
  and the intended use for the response (e.g., does the user want
 +
  to view it on screen or print it on paper?).
 +
</blockquote>
 +
 +
 +
<font color="red">
 +
When DAP4 servers engage in server-driven content negotiation they SHOULD only return DAP4 media types to clients that explicitly request them, otherwise they should return the most general media type that correctly represents the response content. In particular if a DAP4 server detects that a general purpose web browser (such as Internet Explorer, Firefox, Chrome, Safari, Opera, etc.) is requesting one of the three DAP4 response types then the Content-Type header in the servers response SHOULD be the most widely understood media type for the particular response. For the Dataset Services response and Dataset Metadata response this would be a media type of ''text/xml'' and for the Data response it would be ''multipart/mixed''. This is particularly important for the Dataset Services response and Dataset Metadata response as these XML documents may come with XSLT style sheet references that would allow the browser to transform the XML content into an HTML presentation suitable for the browsers. This presentation may even include a complete data request form that will allow the user to subset and select data in their browser. If the DAP4 specific media types are returned the browsers will simply write the response to disk and they will make no effort to render the response in the browser using the stylesheet reference.
 +
</font>
 +
 +
{| class="wikitable"  width="85%"
 +
|+ Server Driven Negotiation Debate (NEEDS RESOLVED)
 +
! Ethan's View
 +
! James' View
 +
|-
 +
|When using server-driven negotiation DAP4 servers MAY return a more general media type (e.g., "text/xml") if something in the request (e.g., the "Accept" or "User-Agent" header values) indicates the client might be better able to handle a more general media type. Servers should use caution when downgrading the response media type: <br/><br/>"''It is impossible for the server to accurately determine what might be "best" for any given user, since that would require complete knowledge of both the capabilities of the user agent and the intended use for the response (e.g., does the user want to view it on screen or print it on paper?).''"
 +
| style="width: 50%;" | When DAP4 servers engage in server-driven content negotiation they SHOULD only return DAP4 media types to clients that explicitly request them, otherwise they should return the most general media type that correctly represents the response content. In particular if a DAP4 server detects that a general purpose web browser (such as Internet Explorer, Firefox, Chrome, Safari, Opera, etc.) is requesting one of the three DAP4 response types then the Content-Type header in the servers response SHOULD be the most widely understood media type for the particular response. For the Dataset Services response and Dataset Metadata response this would be a media type of ''text/xml'' and for the Data response it would be ''multipart/mixed''. This is particularly important for the Dataset Services response and Dataset Metadata response as these XML documents may come with XSLT style sheet references that would allow the browser to transform the XML content into an HTML presentation suitable for the browsers. This presentation may even include a complete data request form that will allow the user to subset and select data in their browser. If the DAP4 specific media types are returned the browsers will simply write the response to disk and they will make no effort to render the response in the browser using the stylesheet reference.
 +
 +
|}
 +
==== A brief digression on ''vnd'' MIME types ====
 +
Looking at how web browsers use these types, the set of vendor MIME types appears to be driven by the browsers, and not the servers. For example, Chrome will look for a MIME type of ''application/vnd.google.drive.ext-type.png'' and do something special when it sees it. Unless we expect mass-market HTTP client software to recognize ''vnd.opendap...'' MIME types, there's little value in them. This conclusion matches what the developers in NASA came to conclude with respect to the vendor MIME types when used as values for the Content-Type header. It's certainly the case that these MIME types provide no value to DAP-aware clients. Providing responses with Content-Type of ''text/xml'' will result in a browser rendering the response, something we might make actually useful with some savvy coding (for example, the (in)famous web form interface could be encoded in it using an XSL transform).
 +
 +
-->
 +
 +
=== Redirects ===
 +
 +
While HTTP redirects are not directly part of the DAP4 web protocol it is strongly suggested that DAP4 client implementations be capable of processing HTTP redirects as nominally described in the [http://www.w3.org/Protocols/rfc2616/rfc2616.html HTTP specification] sections on [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3 redirection status codes] and [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.30 response headers] (It is suggested that implementers of DAP4 clients utilize an existing robust HTTP client library that will manage this for them.)
 +
 +
=== Caching of Responses and Conditional Requests ===
 +
 +
While, HTTP caching and conditional requests are not explicitly part of the DAP4 web protocol, they do provide important tools for improving the performance of both sides client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how [http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13 HTTP caching] works, and utilize it for working with DAP servers.
 +
 +
=== Authentication/Authorization ===
 +
 +
Authentication is the process by which a user agent establishes the identity of the user to a server, and the server establishes it's identity with the user agent. While, HTTP authentication is not explicitly part of the DAP4 web protocol, it does provide important tools for securing the client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how [http://www.ietf.org/rfc/rfc2617.txt HTTP authentication] works, and utilize it for working with DAP servers. Server implementers should pay particular attention to [http://www.w3.org/Protocols/rfc2616/rfc2616-sec15.html#sec15 HTTP security considerations].
 +
 +
Authorization is the process through which a server determines who/what has access to its various holdings and services. The HTTP protocol does not directly address the issue of authorization (even though HTTP defines the 401 response status using the word "authorization" it does not provide separate semantics for authentication and authorization which we see as an important distinction for data access), but any DAP4 server implementer should be aware that some kind of mechanism for controlling access to holdings and services will likely be desired by the people that install and operate their software.
 +
 +
=== Headers ===
 +
 +
==== Request Headers ====
 +
 +
These are the HTTP request headers that DAP clients using HTTP MAY utilize. DAP4 servers MUST accept these headers and act on them in a manner consistent with their descriptions below.
 +
 +
===== General =====
 +
:; Accept
 +
:: The HTTP Accept header MAY be used by clients that wish to engage in server-dirven content negotiation by requesting a particular representation of a resource in the initial request. The server MUST utilize this header, if present, in a manner consistent with the [http://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html#sec12 HTTP content negotiation] specification.
 +
 +
:; User-Agent
 +
:: The [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.43 HTTP User-Agent header] MAY be used by clients to indicate their "software identity" to the server. The server MAY utilize this header, if present, to alter the [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17 Content-Type] of the response to something that is more likely to be digestible by the requesting client software.
 +
 +
===== DAP Specific =====
 +
 +
:There are no DAP specific headers required to make a general DAP request.
 +
 +
===== Asynchronous Transaction =====
 +
; [[DAP4:_Asynchronous_Request-Response_Proposal_v3#Accept_DAP_Asynchronous_Response | X-DAP-Async-Accept]]
 +
: A client MAY indicate it's willingness to accept asynchronous responses by including the '''X-DAP-Async-Accept''' HTTP header. Clients MAY make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the '''X-DAP-Async-Accept''' HTTP header with a value given in seconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.
 +
 +
==== Response Headers ====
 +
 +
These are the HTTP response headers that DAP servers using HTTP MUST and SHOULD (as indicated) return.
 +
 +
===== General =====
 +
 +
:; Date
 +
:: DAP4 servers MUST return an HTTP '''Date''' header whose value is the time of the response and which MUST be in [http://www.ietf.org/rfc/rfc1123.txt  RFC 1123] format.
 +
 +
:; Last-Modified
 +
:: DAP4 servers SHOULD return an HTTP '''Last-Modified''' header whose value is the last modified time of the request resource and which MUST be in [http://www.ietf.org/rfc/rfc1123.txt  RFC 1123] format.
 +
 +
:; Content-Type
 +
:: DAP4 servers MUST return an HTTP '''Content-Type''' header, the value of which SHOULD be set in accordance with the [[#Media_Types | Media Types discussion]] above.
 +
 +
:; Content-Description
 +
:: DAP4 servers SHOULD return an HTTP '''Content-Description''' header.
 +
 +
:; Content-Disposition
 +
:: DAP4 servers SHOULD return an HTTP '''Content-Disposition''' header when transmitting file typed responses.
 +
 +
:; Content-Encoding
 +
:: DAP4 servers MUST return an HTTP '''Content-Encoding''' header when the content-coding of an entity is not "identity".
 +
 +
 +
<!--
 +
 +
<font color="Red">Update this to include the following headers....</font>
 +
<font size="2"><pre>
 +
HTTP/1.1 200 OK
 +
Date: Mon, 23 May 2005 22:38:34 GMT
 +
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
 +
Content-Type: multipart/related; type="text/xml"; start="<<start id>>";  boundary="<<boundary>>"
 +
Content-Description: data-ddx; url=...
 +
Content-Encoding: gzip
 +
XDAP: <<DAP version>>
 +
</pre></font>
 +
 +
-->
 +
 +
===== DAP Specific =====
 +
 +
:; X-DAP-Server
 +
:: DAP4 servers SHOULD return the '''X-DAP-Server''' HTTP header. This HTTP header is used in a response to communicate the software version of the server. This may be a simple string with the server name and version number, or multiple software component versions may be represented. The value of this header is a string determined by the implementations author(s).
 +
:: '''Example'''
 +
::: <font size="2"><tt>X-DAP-Server: bes/3.10.0, libdap/3.11.2, dap-server/ascii/4.1.2, csv_handler/1.0.2, freeform_handler/3.8.4, fits_handler/1.0.7, fileout_netcdf/1.1.2, gateway_module/1.1.0, hdf4_handler/3.9.4, hdf5_handler/1.5.0, netcdf_handler/3.10.0, ncml_module/1.2.1, dap-server/usage/4.1.2, dap-server/www/4.1.2, xml_data_handler/1.0.1</tt></font>
 +
 +
:: '''Example'''
 +
::: <font size="2"><tt>X-DAP-Server: TDS-4.19.3</tt></font>
 +
 +
:; X-DAP
 +
:: DAP4 servers MUST return the '''X-DAP''' HTTP header. This HTTP header is used in a response to indicate the DAP protocol version used to encode the content of the response. This value is constrained to a format of ''"major_version" dot "minor version"'', where both major_version and minor_version are represented by an integer value.
 +
:: '''Example'''
 +
::: <font size="2"><tt>X-DAP: 4.0</tt></font>
 +
:: '''Example'''
 +
::: <font size="2"><tt>X-DAP: 2.17</tt></font>
 +
 +
===== Asynchronous Transaction =====
 +
 +
:; [[DAP4:_Asynchronous_Request-Response_Proposal_v3#DAP_Asynchronous_Response_Required | X-DAP-Async-Required]]
 +
:: DAP4 servers MUST return the '''X-DAP-Async-Required''' header if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the [[DAP4:_Asynchronous_Request-Response_Proposal_v3#400_DAP_Asynchronous_Response_Required  | 400 DAP Asynchronous Response Required]] HTTP response code.
 +
 +
=== Status Codes ===
 +
 +
DAP servers that provide an HTTP interface are expected to utilize the HTTP response codes in a manner consistent with the [http://www.w3.org/Protocols/rfc2616/rfc2616.html  HTTP 1.1 specification].
 +
 +
The ones that are detailed here are used by the DAP in a manner consistent with the specifications definition, but in support of specific DAP server behavior.
 +
 +
==== 200 OK ====
 +
 +
A server MUST return an HTTP status of 200 when the request has been successful and that the returned document contains the requested resource. However since DAP responses can be quite large and since it is also possible for the server to encounter any number of problems during the marshaling, serialization, and subsequent transmission of the response it is possible that the server may have committed/transmitted the HTTP headers (in which the status value is transmitted) before a subsequent error is encountered. These types of errors are transmitted in the DAP4 over-the-wire protocol and all DAP4 clients MUST be able to read and process these errors. <font color="red">See Section?</font>
 +
 +
==== [[DAP4:_Asynchronous_Request-Response_Proposal_v3#202_Accepted | 202 Accepted]] ====
 +
 +
A server MUST return a '''202 Accepted''' HTTP response code when a request has been accepted and will be handled asynchronously. The response body MUST contain a [[DAP4:_Asynchronous_Request-Response_Proposal_v3#DAP4_Asynchronous_Request_Accepted  | DAP asynchronous response accepted]] document.
 +
 +
==== [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1 400 Bad Request] ====
 +
 +
The HTTP specification defines this status code as:
 +
:''The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.''
 +
DAP servers utilizes this code to mean one of two things.
 +
 +
===== 400 Bad DAP4 Request Syntax =====
 +
 +
:The '''400 Bad DAP4 Request Syntax''' HTTP response code MUST be returned by the server when there is a problem with the syntax of the OPeNDAP URL.  The problem could be in the formulation of the constraint expression, or it may be that the URL extension did not match any that are recognized by this server.
 +
 +
===== [[DAP4:_Asynchronous_Request-Response_Proposal_v3#400_DAP4_Asynchronous_Response_Required  | 400 DAP4 Asynchronous Response Required]] =====
 +
 +
:The '''400 DAP4 Asynchronous Response Required''' HTTP response code MUST be returned by the server when the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response. The server MUST include the '''[[DAP4:_Asynchronous_Request-Response_Proposal_v3#X-DAP-Async-Required  | X-DAP-Async-Required]]''' HTTP header and the [[DAP4:_Asynchronous_Request-Response_Proposal_v3#DAP4_Asynchronous_Response_Required | DAP4 Asynchronous Response Required]] document as the response body.
 +
 +
==== 401 Unauthorized ====
 +
 +
The '''401 Unauthorized''' HTTP response code MUST be returned by the server when access to the requested resource requires (not previously acquired) user authentication. See the [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html HTTP specification] for usage and behavior.
 +
 +
==== 403 Forbidden ====
 +
 +
The '''403 Forbidden''' HTTP response code MUST be returned when the server has understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. This is appropriate to return if, for example, the server software does not have read permission for the requested resource.
 +
 +
==== 404 Not Found ====
 +
The '''404 Not Found''' HTTP response code MUST be returned when the server has not found anything matching the Request-URI.
 +
 +
==== [[DAP4:_Asynchronous_Request-Response_Proposal_v3#409_Conflict_-_DAP4_Response_Not_Ready  | 409 Conflict (DAP4 Async Response Not Available)]] ====
 +
The '''409 Conflict''' HTTP response code SHOULD be returned by the server when the DAP request has been rejected because a previous asynchronous request has not been completed and the result is not ready for access. The response body MUST contain the [[DAP4:_Asynchronous_Request-Response_Proposal_v3#DAP4_Asynchronous_Response_Not_.28Yet.29_Available | DAP4 Asynchronous Response Not Yet Available]] document.
 +
 +
==== [[DAP4:_Asynchronous_Request-Response_Proposal_v3#412_Precondition_Failed  | 412 Precondition Failed]] ====
 +
 +
The '''412 Precondition Failed''' HTTP response code MUST be returned by the server if the DAP request has been rejected because it did not meet the '''[[DAP4:_Asynchronous_Request-Response_Proposal_v3#X-DAP-Async-Accept  | X-DAP-Async-Accept]]''' condition that was specified in the request. The response body MUST contain the [[DAP4:_Asynchronous_Request-Response_Proposal_v3#DAP4_Asynchronous_Request_Rejected | DAP4 Asynchronous Request Rejected]] document.
 +
 +
==== [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16  415 Unsupported Media Type] ====
 +
 +
The '''415 Unsupported Media Type''' HTTP response code MUST be returned when the client requests a representation of the requested resource that the server cannot provide.
 +
 +
==== [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1 500 Internal Server Error] ====
 +
 +
The [http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1 500 Internal Server Error] SHOULD be returned when the DAP server encounters internal problems when attempting to fulfill a request.
 +
 +
=== Verbs (aka Methods) ===
 +
 +
==== GET ====
 +
 +
A DAP request may be made using the HTTP GET request method utilizing a Uniform Resource Identifier (URI) that encodes information specific to the DAP (<font color="red">see Section ?</font>).
 +
 +
Each GET request MUST conform to the HTTP specification (which basically says that a GET request MUST contain an HTTP protocol version number followed by a MIME-like message containing various headers that further describe the request.). While there are some optional [[#Request_Headers | DAP HTTP request headers]] that may be used, DAP requests do not require specific HTTP headers beyond those necessary for HTTP. DAP servers SHOULD support the use of the HTTP Accept request header to allow clients to engage in [http://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html HTTP content negotiation] for specific representations of a DAP response.
 +
 +
; Implementation Note
 +
:''In practice, DAP clients typically use a third-party library implementation of HTTP/1.1 so the GET request, URI and HTTP version information are hidden from the client; it sees only the DAP Uniform Resource Locator (URL) and the request headers.''
 +
 +
The DAP server responds to the GET request with an HTTP compliant response (one that includes a status line containing the HTTP protocol version and an error or success code, followed by HTTP response headers and then response itself). There are two DAP specific HTTP headers that are always included in a DAP response over HTTP: <font size="3"><tt>X-DAP-Server</tt></font> and <font size="3"><tt>X-DAP</tt></font>,  [[#Response_Headers | as described above]]. The DAP response is the payload of the HTTP response.  Unless otherwise negotiated, the data response payload is encoded in multpart-MIME format, while the other responses are simple HTTP responses <font color="red">This is further described in Section ?.</font>
 +
 +
===== Examples =====
 +
; HTTP GET request.
 +
<font size="2"><source lang="httpd">
 +
GET /dap/path/data.nc.dap?u,v[0:8:1024] HTTP/1.1
 +
Host: server.org
 +
Accept: application/vnd.org.opendap.dap4.data
 +
</source></font>
 +
 +
<!--
 +
 +
==== POST ====
 +
 +
DAP4 supplies (via the DAP Service Terminus) a URI for each DAP4 service and each of its various media-type representations. How do we leverage this into a non-HTTP usage? I think we need to define a message document that is sent to the server (say over a socket) that carries everything we need in the request. Suggestion:
 +
 +
<font size="2"><source lang="xml">
 +
<request xmlns="http://opendap.org/ns/dap4/request">
 +
  <service>dap4-data</service>
 +
  <media-type>...</media-type>
 +
  <dataset id="/local/id/of/data/resource">
 +
    <ce>
 +
      <projection>x,y[0:3:99]</projection>
 +
      <selection>k<33 & i>17</selection>
 +
    </ce>
 +
  </dataset>
 +
</request>
 +
</source></font>
 +
 +
Obviously this needs more thought and discussion...
 +
 +
 +
 +
'''Proposal''': The  DAP4 data model and it's persistent (over-the-wire) representations are transport protocol agnostic. All of the information required by the DAP must be present in the content of the DAP requests and responses.  The DAP won't uniquely embed information required by the DAP into HTTP Headers or AMQP thingys, or other whatnot.
 +
 +
'''Discussion''':
 +
 +
# Does this matter?
 +
# Would we need to add new syntax server-side functions etc. to the protocol to enable this?
 +
-->
 +
 +
 +
 +
== Normative References ==
 +
 +
<div id="DAP4_Vol1"></div>
 +
<nowiki>[DAP4_Vol1]</nowiki> [[DAP4 Volume 1|DAP4 Volume 1: "Data Model, Persistent Representations, and Constraints"]].
 +
 +
<div id="HTTP_RFC2616"></div>
 +
<nowiki>[HTTP_RFC2616]</nowiki> "Hypertext Transfer Protocol -- HTTP/1.1" (as [http://www.ietf.org/rfc/rfc2616.txt text] or [http://www.w3.org/Protocols/rfc2616/rfc2616.html HTML]).
 +
 +
== Non-Normative References ==
 +
 +
<div id="REST_Fielding"></div>
 +
[REST] R. Fielding, UC Irvine Doctoral Thesis: [http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm|"Architectural Styles and the Design of Network-based Software Architectures"]
 +
 +
== Extra Web Service Stuff ==
 +
=== Other Common Services ===
 +
 +
DAPx servers typically offer a number of other services that, while not exactly DAP services, are commonly available. This section lists just some of the alternate services a DAP4 server might provide. By enumerating the available services in the  [[DAP4_Dataset_Services_Response | Dataset Services Response ]] servers can easily make clients (either software and monkey) aware of their ancillary capabilities.
  
 
==== [[DAP4: HTML DATA Request Form Service]] ====
 
==== [[DAP4: HTML DATA Request Form Service]] ====
Line 91: Line 1,002:
 
suffix = <font size="2"><code>'''.html'''</code></font><br/>
 
suffix = <font size="2"><code>'''.html'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .html'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .html'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/data-request-form#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/data-request-form#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/html'''</code></font> | <font size="2"><code>'''text/xhtml'''</code></font><br/>
  
 
==== [[DAP4: RDF Service  | DAP4: RDF Service]] ====
 
==== [[DAP4: RDF Service  | DAP4: RDF Service]] ====
Line 99: Line 1,012:
 
suffix = <font size="2"><code>'''.rdf'''</code></font><br/>
 
suffix = <font size="2"><code>'''.rdf'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .rdf'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .rdf'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/rdf#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/rdf#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''application/rdf+xml'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP4: ISO 19115 Service ]] ====
 
==== [[DAP4: ISO 19115 Service ]] ====
Line 107: Line 1,023:
 
suffix = <font size="2"><code>'''.iso'''</code></font><br/>
 
suffix = <font size="2"><code>'''.iso'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .iso'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .iso'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/iso-19115-metadata#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/iso-19115-metadata#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/xml'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP4: ISO Conformance Score Service ]] ====
 
==== [[DAP4: ISO Conformance Score Service ]] ====
Line 115: Line 1,034:
 
suffix = <font size="2"><code>'''.rubric'''</code></font><br/>
 
suffix = <font size="2"><code>'''.rubric'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .rubric'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .rubric'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/iso-19115-score#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/iso-19115-score#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/html'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP4: NetCDF File-out Service ]] ====
 
==== [[DAP4: NetCDF File-out Service ]] ====
This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.  
+
This service provides data responses in NetCDF-4 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 4 file.  
 +
 
 +
suffix = <font size="2"><code>'''.nc4'''</code></font><br/>
 +
service url =  <font size="2"><code>'''dataset_url + '.nc4' + [?dap_constraint]'''</code></font><br/>
 +
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/netcdf-3#</nowiki>'''</code></font><br/>
  
suffix = <font size="2"><code>'''.nc'''</code></font><br/>
+
Default/primary media type: <font size="2"><code>'''application/x-netcdf-4'''</code></font><br/>
service url =  <font size="2"><code>'''dataset_url + '.nc' + [?dap_constraint]'''</code></font><br/>
+
<br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/netcdf-3#</nowiki>'''</code></font><br/><br/>
 
  
 
==== [[DAP4: ASCII Data Service]] ====
 
==== [[DAP4: ASCII Data Service]] ====
Line 131: Line 1,056:
 
service url =  <font size="2"><code>'''dataset_url + '.ascii' + [?dap_constraint]'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + '.ascii' + [?dap_constraint]'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/ascii#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/ascii#</nowiki>'''</code></font><br/><br/>
 +
 +
Default/primary media type: <font size="2"><code>'''text/plain'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP4: XML Data Service]] ====
 
==== [[DAP4: XML Data Service]] ====
Line 139: Line 1,067:
 
service url =  <font size="2"><code>'''dataset_url + '.xdap' + [?dap_constraint]'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + '.xdap' + [?dap_constraint]'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/xml-data#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/xml-data#</nowiki>'''</code></font><br/><br/>
 +
 +
Default/primary media type: <font size="2"><code>'''text/xml'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP4: Native File Access Service ]] ====
 
==== [[DAP4: Native File Access Service ]] ====
This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in.
+
This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in. Servers MAY elect to not support this response, for example, for generated data, very large data, et cetera.
  
  
Line 147: Line 1,078:
 
service url =  <font size="2"><code>'''dataset_url + .file'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .file'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/file#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/file#</nowiki>'''</code></font><br/><br/>
 +
 +
Default/primary media type: Type varies with file type.
 +
<br/>
  
 
==== [[DAP4: Server Version Service  ]]====
 
==== [[DAP4: Server Version Service  ]]====
This service provides software versioning information. When invoked the services returns an XML file containing a description of the version of the server and it's components.
+
This service provides software versioning information. When invoked the service returns an XML file containing a description of the version of the server and it's components.
  
  
Line 155: Line 1,089:
 
service url =  <font size="2"><code>'''dataset_url + .ver'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .ver'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/version#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/version#</nowiki>'''</code></font><br/><br/>
 +
 +
Default/primary media type: <font size="2"><code>'''text/xml'''</code></font><br/>
 +
<br/>
  
 
=== DAP2 Services ===
 
=== DAP2 Services ===
 +
 +
In order to support legacy client applications DAP4 server implementations MAY support the DAP2 services stack. If they do so the DAP2 services MUST be organized as described in this section.
 +
 +
  
 
==== [[DAP2: Data Service]]====
 
==== [[DAP2: Data Service]]====
Line 165: Line 1,106:
 
service url =  <font size="2"><code>'''dataset_url + '.dods' + [?dap_constraint]'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + '.dods' + [?dap_constraint]'''</code></font><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/dods#</nowiki>'''</code></font><br/><br/>
 
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/dods#</nowiki>'''</code></font><br/><br/>
 +
 +
Default/primary media type: <font size="2"><code>'''application/octet-stream'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP2: DDX Service ]]====
 
==== [[DAP2: DDX Service ]]====
Line 171: Line 1,115:
 
suffix = <font size="2"><code>'''.ddx'''</code></font><br/>
 
suffix = <font size="2"><code>'''.ddx'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .ddx'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .ddx'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/ddx#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/ddx#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/xml'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP2: DDS Service ]]====
 
==== [[DAP2: DDS Service ]]====
Line 179: Line 1,126:
 
suffix = <font size="2"><code>'''.dds'''</code></font><br/>
 
suffix = <font size="2"><code>'''.dds'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + '.dds' + [?dap_constraint]'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + '.dds' + [?dap_constraint]'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/dds#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/dds#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/plain'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP2: DAS Service ]] ====
 
==== [[DAP2: DAS Service ]] ====
Line 187: Line 1,137:
 
suffix = <font size="2"><code>'''.das'''</code></font><br/>
 
suffix = <font size="2"><code>'''.das'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .das'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .das'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/das#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/das#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/plain'''</code></font><br/>
 +
<br/>
  
 
==== [[DAP2: Info Service ]] ====
 
==== [[DAP2: Info Service ]] ====
Line 195: Line 1,148:
 
suffix = <font size="2"><code>'''.info'''</code></font><br/>
 
suffix = <font size="2"><code>'''.info'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .info'''</code></font><br/>
 
service url =  <font size="2"><code>'''dataset_url + .info'''</code></font><br/>
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/Info#</nowiki>'''</code></font><br/><br/>
+
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap2/Info#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''text/html'''</code></font><br/>
 +
<br/>
 +
 
 +
==== [[DAP2: NetCDF Service ]] ====
 +
This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.
 +
 
 +
suffix = <font size="2"><code>'''.nc'''</code></font><br/>
 +
service url =  <font size="2"><code>'''dataset_url + '.nc' + [?dap_constraint]'''</code></font><br/>
 +
role id = <font size="2"><code>'''<nowiki>http://services.opendap.org/dap4/netcdf-3#</nowiki>'''</code></font><br/>
 +
 
 +
Default/primary media type: <font size="2"><code>'''application/x-netcdf'''</code></font><br/>
 +
<br/>
  
  
 
[[Template: ServiceTemplate]]
 
[[Template: ServiceTemplate]]

Latest revision as of 21:47, 13 November 2013

<< Back to OPULS Development

This page is out of date and will not be updated. This is an old document that captures an intermediate point of the OPULS design work. This page should be referenced only as part of a baseline for the work. It has been superseded by the DAP4 Specification Volume 2: Web Services Specification

Contents

1 Introduction

This document defines the DAP4 Web Service protocol which DAP4-compliant software MUST support when utilizing the HTTP protocol to transmit DAP4 requests and responses, along with additional optional services that DAP4-compliant software SHOULD support.

The DAP4 protocol uses three basic responses to represent a data resource. One response, the Dataset Metadata Response (DMR) contains metadata information describing the structure of the data resource. That is, it characterizes the variables, their datatypes, names and attributes. The second response, the Data Response (DataDMR), contains both the metadata about the requested data and the actual data that was requested. The third basic response is the Dataset Services Response (DSR) that provides a listing of services, any alternate media representations if available, and all of the associated access URI's for a particular data resource.

The DAP4 protocol uses a fourth basic resource, the Constraint/Query Expression (CE), to represent subsetting of and, possibly, transformations of the dataset requested.

The DAP4 protocol uses a fifth basic resource, the Error Response (ER), to allow servers to communicate error information with clients. When a request for any of the three basic responses cannot be completed, an Error response is returned. If an error occurs before the standard response is initiated, an error response is returned in place of the standard response. If an error occurs after a data response has been initiated, an Error Response is returned as the final chunked record as described in Section 7 "DAP4 Chunked Data Representation" of the DAP4 Data Model document [DAP4_Vol1].

The DAP4 Data Model, constraint/query language, dataset metadata XML encoding, and binary data encoding are all defined in the DAP4 Volume 1: "Data Model, Persistent Representations, and Constraints" document [DAP4_Vol1].

Each of the three responses (Dataset Metadata, Data, and Dataset Services) are complete in and of themselves so that, for example, the Data response can be used by a client without ever requesting either of the two other responses. In many cases, client programs will request the Dataset Metadata response before requesting the Data response but there is no requirement they do so and servers SHALL NOT require that behavior on the part of clients.

These three standard dataset responses can each be accessed in two different ways. First, similar to DAP2 URL construction and to support client-driven content negotiation (see the 3.? "Content Negotiation" section below), each dataset resource has a standard URL suffix that can be added to the DAP4 dataset URL to retrieve the resource in its standard (normative) encoding. This allows clients, given a base DAP4 dataset URL, to construct DAP4 URLs in a simple and standard way for each of the three standard dataset resources.

Second, to support server-driven content negotiation (see the 3.? "Content Negotiation" section below) and a more hypermedia-driven style, a DAP4 resource role is defined for each standard dataset resource and a DAP4 media type is defined for each dataset resource encoding scheme.

Any particular instance of a DAP4 server MAY have one or more additional services, alternate media representations of service responses, or unique (to the server instance) server side functions. All of these things, including the core services and their default representations MUST be included in the Dataset Services Response.

The DAP4 web service is currently limited to HTTP GET requests though it is expected that extensions (e.g., asynchronous access) will use other HTTP methods (e.g., POST). This makes, for now, the DAP4 Constraint Expression something of a pseudo-resource type given that they are encoded as part of the URL query string rather than as an independent document.

1.1 DAP4 Resource Roles and Media Types

The standard DAP4 dataset resource roles and encodings (plus a few alternate encodings) that are defined in the core DAP4 documents are:

Dataset Services Response (DSR)

Resource Role
http://services.opendap.org/dap4/dataset-service
URL Suffix Media Type URL Example
"" or ".dsr"
application/vnd.opendap.org.dap4.dataset-services+xml
Normative DSR encoding
http://server/path/dataset.nc

http://server/path/dataset.nc.dsr

".xml" or ".dsr.xml"
text/xml
Normative DSR encoding with generic media type
http://server/path/dataset.nc.xml

http://server/path/dataset.nc.dsr.xml

".html" or ".dsr.html"
text/html
HTML DSR encoding
http://server/path/dataset.nc.html

http://server/path/dataset.nc.dsr.html

Dataset Metadata Response (DMR)

Resource Role
http://services.opendap.org/dap4/dataset-metadata
URL Suffix Media Type URL Example
".dmr"
application/vnd.opendap.org.dap4.dataset-metadata+xml
Normative DMR encoding
http://server/path/dataset.nc.dmr
".dmr.xml"
text/xml
Normative DMR encoding with generic media type
http://server/path/dataset.nc.dmr.xml
".dmr.html"
text/html
HTML DMR encoding
http://server/path/dataset.nc.dsr.html

Dataset Data Response (Data)

Resource Role
http://services.opendap.org/dap4/data
URL Suffix Media Type URL Example
".dap"
application/vnd.opendap.org.dap4.data
Normative Data encoding
http://server/path/dataset.nc.dap
".dap.txt"
text/plain
Text (UTF-8) Data encoding
http://server/path/dataset.nc.dap.txt
".dap.xml"
text/xml
XML Data encoding
http://server/path/dataset.nc.dap.xml

Error Response (ER)

Resource Role
http://services.opendap.org/dap4/error
URL Suffix Media Type URL Example
N/A
application/vnd.opendap.org.dap4.error+xml
Normative Error encoding
N/A
N/A
text/xml
Normative Error encoding with generic media type
N/A
N/A
text/html
HTML Error encoding
N/A


1.2 DAP4 Core Specification Documents

  • DAP4 Volume 1: "Data Model, Persistence Representations, and Constraints"
  • DAP4 Web Service (this document)
  • DAP4 Dataset Services
  • DAP4 Requirements for Server-side Functions

1.3 Extensions to DAP4 Core Specifications

Several types of extensions can be made to the DAP4 core including:

  • New encodings for the core DAP4 response types
  • New response types
  • New server-side functions.

2 Terms and Definitions

Dataset Service Response (DSR)
The DAP4 response type that contains a list of DAP (and other) services available for the dataset including any alternate media representations and all the associated access URIs.
Dataset Metadata Response (DMR)
The DAP4 response type that contains metadata information describing the structure of the requested data. The metadata information characterizes the requested data variables including their names, data types, shapes, and attributes.
Dataset Data Response (Data)
The DAP4 response type that contains both the dataset metadata and the binary data for the requested data.
Resource role ID
A URI that identifies the role of a resource, generally provided with a link to allow clients to identify the type of resource the link references. (For instance, an "atom:link" element has an optional "atom:rel" attribute.)
Media Type
A internet media type is a two-part identifier for resource encoding schemes [1]. (E.g., "text/html", "text/plain", "application/octet-stream".)

3 DAP4 Web Service Responses

The core of the DAP4 Web Service protocol consists of the three standard response types: Dataset Services Response (DSR), Dataset Metadata Response (DMR), and Dataset Data Response (Data). Each dataset served by a DAP4 compliant server MUST provide these three response types.

All of the example requests described below are based on the DAP4 dataset URL:

http://server.org:8080/dap/path/data.nc

3.1 DAP4 Dataset Services Response

The DAP4 Dataset Services Response (DSR) provides clients with a listing of all available DAP4 services and all the available encodings for those services as well as all available DAP4 extensions.

Each service (or response type) has a unique resource role (defined in the appropriate specification), each link (alternate representation) for a given service MUST fulfill that same role. This is not always a clear distinction to make. For example, the DMR can be mapped into ISO 19115 metadata. However, IS0 19115 is clearly a different domain.

The Dataset Services Response MUST contain the following information:

  • List of DAP versions supported by server
  • The implementation version (e.g., "TDS 4.3.57" or "Hyrax 1.7.45")
  • List of all available DAP4 services for the dataset
  • For each DAP4 services listed, a list of all available links each with its corresponding media type
  • List of supported extensions
    • Resource type extensions
    • Media type extensions
    • Server-side function extensions

If SHOULD contain the following information:

  • A human readable title for the dataset
  • Human readable titles for each service

To take advantage of web caching, servers should try to keep DSRs light weight (i.e., quick creation) and as stable as possible.

3.1.1 DSR Resource Role

DSRs are identified by the resource role:

http://services.opendap.org/dap4/dataset-services

3.1.2 Normative Encoding of the DSR

The normative XML representation for the Dataset Services Response is defined in the "Normative XML Encoding of the DSR" appendix. The media type for the normative XML representation is

application/vnd.opendap.org.dataset-services+xml

3.1.3 Service Behavior

When an HTTP GET request is made on a base DAP4 dataset URL, all DAP4 servers MUST return the normative XML encoding of the DSR given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type,
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DSR.

For example, the request:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.org.dataset-services+xml
Date: ...
3.1.3.1 Downgrading the Normative XML Encoding (Required)

When an HTTP GET request is made on a base DAP dataset URL with the suffix .xml
added to it:

request url = dataset_url.xml

the response MUST be the normative representation of the DSR along with the HTTP Content-Type header set to text/xml . For example:

GET /dap/path/data.nc.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

The normative XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: text/xml

3.1.4 Other Encodings of the DSR

3.1.4.1 HTML Encoding (Optional)

When an HTTP GET request is made on a base DAP dataset URL with the suffix .html
added to it:

request url = dataset_url.html

the server MUST reply with an HTML representation of the DSR, or return an HTTP status of 404 to indicate that an HTML representation of the DSR is not available. For example:

GET /dap/path/data.nc.html HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: ...

If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 dataset URL (without a suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example this request:

GET /dap/path/data.nc HTTP/1.1
Host: server.org:8080
Accept: text/html

Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.2 Dataset Metadata Response

The Dataset Metadata Service returns the Dataset Metadata Response (DMR) which is a metadata description of the dataset. The normative representation of the DMR is an XML document that contains both the 'syntactic' (structural) and 'semantic' metadata for the dataset, persisted as a DAP4: Data Model representation of the dataset held at the server. The DMR service accepts a query string (constraint expression) that allows you to inspect the effects on the data structures when sub-setting and/or server side functions are applied. If a constraint expression has been successfully applied, the service will returned the constrained view of the dap:Dataset object. The constrained view may contain different data structures than the unconstrained view as the constraint may alter the reasonable representation of the data set. Note that All dap:Attribute objects have been removed from constrained dap:Dataset objects. More information on the syntax of DAP4 constraint expressions can be found on DAP4: Data Model page.

3.2.1 DMR Resource Role

DMRs are identified by the resource role:

http://services.opendap.org/dap4/dataset-metadata

3.2.2 Normative Encoding of the DSR

The normative XML representation for the Dataset Metadata Response is defined in the DAP4: Responses page. The media type for the normative XML representation is:

application/vnd.org.opendap.dap4.dataset-metadata+xml

(More information on the syntax of DAP4 constraint expressions can be found on DAP4: Data Model page. )

3.2.3 Service Behavior

Every DAP4 compliant server MUST return the normative representation (an XML document as described here ) of the Dataset Metadata Response when a client attempts to access a dataset URL with the suffix ".dmr" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.

When an HTTP GET request is made on a base DAP dataset URL with the suffix .dmr added to it:

request url = dataset_url + '.dmr'+ [?dap_constraint]

the server MUST reply with an normative representation of the DMR for the (possibly constrained) dataset given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type (application/vnd.org.opendap.dap4.dataset-metadata+xml),
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the DMR.

For example, the request:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.org.dataset-metadata+xml
Date: ...
3.2.3.1 Downgrading the Normative XML Encoding (Required)

While the normative representation of the the Dataset Metadata response is already an XML document, the normative media type is application/vnd.opendap.org.dataset-metadata+xml which may be unfamiliar to many generic clients (such as web browsers) and it is quite conceivable that such a client might ask for the more generic text/xml media type.

When an HTTP GET request is made on a DAP DMR URL with the suffix .xml
added to it:

request url = dataset_url.dmr.xml

the response MUST be the normative representation of the DMR along with the HTTP Content-Type header set to text/xml . For example:

GET /dap/path/data.nc.dmr.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

The normative XML representation of the DMR MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL ( and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DSR. For example:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: text/xml

3.2.4 Other Encodings of the DMR

3.2.4.1 HTML Encoding (Optional)

When an HTTP GET request is made on a DAP DMR URL with the suffix .html
added to it:

request url = dataset_url.dmr.html

the server MUST reply with an HTML representation of the DMR, or return an HTTP status of 404 (or 415) to indicate that an HTML representation of the DMR is not available. For example:

GET /dap/path/data.nc.dmr.html HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: ...

If available, the HTML representation MUST also be returned when an HTTP GET request is made on a base DAP4 DMR URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an HTML encoded DMR. For example this request:

GET /dap/path/data.nc.dmr HTTP/1.1
Host: server.org:8080
Accept: text/html

Must return the HTML representation of the DMR, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3 DAP4: Data Response

The Data Service provides DAP4 data access to a dataset, and is the (primary) way that DAP4 returns data to a client. The Data service accepts a query string (constraint expression) which allows you to subset the data and invoke server side functions. When the service is invoked it returns the DAP4 data object. On the wire this is a binary document with MIME type application/vnd.opendap.org.dap4.data. The payload is broken into two logical parts: A DMR-type xml document that describes the data and a BLOB that contains the actual data. For more on the internal structure of the Data response's payload, see DAP4: Encoding for the Data Response and DAP4: Chunked encoding.

More information on the Data response can be found on the DAP4: Responses page.

3.3.1 Data Response Resource Role

DAP4 Data Responses are identified by the resource role:

http://services.opendap.org/dap4/data

3.3.2 Normative Encoding of the Data Response

The normative XML representation for the Data Response is defined in the DAP4: Responses page. The media type for the normative XML representation is:

application/vnd.org.opendap.dap4.data

(More information on the syntax of DAP4 constraint expressions can be found on DAP4: Data Model page. )

3.3.3 Service Behavior

Every DAP4 compliant server MUST return the normative representation ( as described here ) of the Data Response when a client attempts to access a dataset URL with the suffix ".dap" appended to it. The DAP4 server MAY return alternate representations if the client indicates that it can accept them and the server can provide them.

When an HTTP GET request is made on a base DAP dataset URL with the suffix .dap added to it:

request url = dataset_url + '.dap'+ [?dap_constraint]

the server MUST reply with an normative representation of the (possibly constrained) data response for the dataset given these conditions:

  • the request "Accept" header contains only the normative XML encoding media type (application/vnd.org.opendap.dap4.data),
  • the request "Accept" header equals "*/*", or
  • the request "Accept" header does not indicate a preference for another media type in which the server knows how to encode the Data Response.

For example, the request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/vnd.opendap.org.data
Date: ...

3.3.4 Other Encodings of the Data Response

3.3.4.1 ASCII Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .ascii
added to it:

request url =  dataset_url.dap.ascii

the server MUST reply with the ASCII representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that an ASCII representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.ascii HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: ...

If available, the ASCII representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an ASCII encoded Data Response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: text/plain

Must return the ASCII representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3.4.2 XML Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .xml
added to it:

request url =  dataset_url.dap.xml

the server MUST reply with the XML representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that an XML representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.xml HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Date: ...

If available, the XML representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an XML encoded data response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: text/xml

Must return the XML representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.3.4.3 NetCDF Encoding (Optional)

When an HTTP GET request is made on a DAP Data Response URL with the suffix .nc
added to it:

request url =  dataset_url.dap.nc

the server MUST reply with the XML representation of the Data Response, or return an HTTP status of 404 (or 415) to indicate that an XML representation of the Data Response is not available. For example:

GET /dap/path/data.nc.dap.nc HTTP/1.1
Host: server.org:8080
Accept: */*

Might result in the following response:

HTTP/1.1 200 OK
Content-Type: application/x-netcdf
Date: ...

If available, the NetCDF representation MUST also be returned when an HTTP GET request is made on a base DAP4 Data Response URL (without an additional suffix) and the server uses server-driven content negotiation to decide that the best response for the client would be an NetCDF encoded data response. For example this request:

GET /dap/path/data.nc.dap HTTP/1.1
Host: server.org:8080
Accept: application/x-netcdf

Must return the NetCDF representation of the Data Response, if available. If no such representation is available then the server MAY return an HTTP status of 404 or even 415.

3.4 DAP4 Asynchronous Response

Some servers may not be able to fulfill DAP4 service requests in a timely manner.

  • A DAP4 server that is retrieving data content from a near-line tape storage subsystem might take several minutes to access a particular data holding.
  • A DAP4 server that is providing access to data held in an Amazon Web Services Glacier Vault will have to wait ~4 hours before it can retrieve a particular holding.

In these circumstances the server may return the DAP4 Asynchronous Response.

3.4.1 Asynchronous Response Resource Role

DAP4 Asynchronous Responses are identified by the resource role:

http://services.opendap.org/dap4/async

3.4.2 Normative Encoding of the Asynchronous Response

The normative XML representation for the Asynchronous Response is defined in Appendix x "Normative XML Encoding of the Asynchronous Response". The media type for the normative XML representation is:

application/vnd.org.opendap.dap4.async.xml

3.5 DAP4 Error Response

The DAP4 protocol returns error information using an Error response. If a request for any of the three basic responses cannot be completed then an Error response is returned in its place.

The normative XML representation for the Error Response is defined by the following RELAX-NG schema.

<grammar xmlns="http://relaxng.org/ns/structure/1.0"
                xmlns:doc="http://www.example.com/annotation"
                datatypeLibrary="http://xml.opendap.org/datatypes/dap4"
                ns="http://xml.opendap.org/ns/DAP/4.0#"
                >
<start>
  <ref name="errorresponse"/>
</start>
<define name="errorresponse">
  <element name="Error">
    <optional>
      <attribute name="httpcode"><data type="dap4_integer"/></attribute>
    </optional>
    <optional>
      <interleave>
        <element name = "Message"><text/></Message>
        <element name = "Context"><text/></Message>
        <element name = "OtherInformation"><text/></Message>
      </interleave>
    </optional>
  </element>
</define>

The Error element has one optional attribute: the httpcode which is a standard HTTP protocol return code indicating the general class of error. When possible, this code should match the return code in the HTTP headers for the response.

The body of the <Error> element may contain any or all of the following inner elements each containing arbitrary text.

  1. <Message> — A short informative message describing the error.
  2. <Context> — Information describing the context in which the error occurred: position of a parse error in a constraint expression, for example.
  3. <OtherInformation> — Arbitrary additional text information: a Java stack trace, for example.

3.5.1 Error Response Resource Role

DAP4 Error Responses are identified by the resource role:

http://services.opendap.org/dap4/error

3.5.2 Normative Encoding of the Error Response

The normative XML representation for the Error Response is defined in Appendix x "Normative XML Encoding of the Error Response". The media type for the normative XML representation is:

application/vnd.org.opendap.dap4.error.xml

4 HTTP

The DAP4 Web Services specification describes the features of HTTP that are required to be a compliant DAP4 client or server. It does not attempt to describe all aspects of HTTP that DAP4 servers might implement or that DAP4 clients may see in response to DAP4 requests. Similarly, it does not cover all issues related to the performance and scalability of HTTP.

However, the following sections include both DAP4 requirements as well as some suggestions of HTTP features that servers and clients are encouraged to use.

4.1 Content Negotiation and Media Types

Though the DAP4 core specifications only describe one encoding for each type of resource, DAP4 web servers MAY have the ability to provide a given resource in a number of different media types. All media types available for a resource MUST be listed in the DAP4 Dataset Services response document.

DAP4 responses MUST use the "Content-Type" header field to identify the media type of the DAP4 response body. For example, the normative value for the XML encoded DDX response is application/vnd.org.opendap.ddx+xml.

The DAP4 Dataset Services response describes the available services and their media types and provides DAP4 software (client and/or server) with two different mechanisms to negotiate for different kinds of media representations. The first of these is server-driven content negotiation as described in the HTTP 1.1 specification, section 12 "Content Negotiation". The second is similar to the agent-driven negotiation also described in section 12 of the HTTP 1.1 specification. The difference being that the "list of available representations ... [each with] its own URI" is provided by the DAP4 Dataset Services response rather than in response to an initial request.

Clients need not retrieve the Dataset Services response in order to access the normative representations of either the Dataset Metadata or Data responses, as these responses are required for every DAP4 server and are mapped to well known URL patterns. If clients attempt to access other representations or other services using agent-driven negotiation without first checking the Dataset Services response, they should be prepared to receive a 404 Not Found response.

When using server-driven negotiation, DAP4 clients are encouraged to, at a minimum, include "Accept" and "User-Agent" headers in their requests and to provide accurate and detailed information in the values of those headers. However, when server-driven negotiation is used, DAP4 servers are not limited to those headers for determining the media type that is returned. DAP4 clients should also be prepared to handle 415 "Unsupported Media type" response codes.


4.2 Redirects

While HTTP redirects are not directly part of the DAP4 web protocol it is strongly suggested that DAP4 client implementations be capable of processing HTTP redirects as nominally described in the HTTP specification sections on redirection status codes and response headers (It is suggested that implementers of DAP4 clients utilize an existing robust HTTP client library that will manage this for them.)

4.3 Caching of Responses and Conditional Requests

While, HTTP caching and conditional requests are not explicitly part of the DAP4 web protocol, they do provide important tools for improving the performance of both sides client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how HTTP caching works, and utilize it for working with DAP servers.

4.4 Authentication/Authorization

Authentication is the process by which a user agent establishes the identity of the user to a server, and the server establishes it's identity with the user agent. While, HTTP authentication is not explicitly part of the DAP4 web protocol, it does provide important tools for securing the client server interaction. Therefore, it is strongly suggested that DAP4 servers and client implementers be aware of how HTTP authentication works, and utilize it for working with DAP servers. Server implementers should pay particular attention to HTTP security considerations.

Authorization is the process through which a server determines who/what has access to its various holdings and services. The HTTP protocol does not directly address the issue of authorization (even though HTTP defines the 401 response status using the word "authorization" it does not provide separate semantics for authentication and authorization which we see as an important distinction for data access), but any DAP4 server implementer should be aware that some kind of mechanism for controlling access to holdings and services will likely be desired by the people that install and operate their software.

4.5 Headers

4.5.1 Request Headers

These are the HTTP request headers that DAP clients using HTTP MAY utilize. DAP4 servers MUST accept these headers and act on them in a manner consistent with their descriptions below.

4.5.1.1 General
Accept
The HTTP Accept header MAY be used by clients that wish to engage in server-dirven content negotiation by requesting a particular representation of a resource in the initial request. The server MUST utilize this header, if present, in a manner consistent with the HTTP content negotiation specification.
User-Agent
The HTTP User-Agent header MAY be used by clients to indicate their "software identity" to the server. The server MAY utilize this header, if present, to alter the Content-Type of the response to something that is more likely to be digestible by the requesting client software.
4.5.1.2 DAP Specific
There are no DAP specific headers required to make a general DAP request.
4.5.1.3 Asynchronous Transaction
X-DAP-Async-Accept
A client MAY indicate it's willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header. Clients MAY make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the X-DAP-Async-Accept HTTP header with a value given in seconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.

4.5.2 Response Headers

These are the HTTP response headers that DAP servers using HTTP MUST and SHOULD (as indicated) return.

4.5.2.1 General
Date
DAP4 servers MUST return an HTTP Date header whose value is the time of the response and which MUST be in RFC 1123 format.
Last-Modified
DAP4 servers SHOULD return an HTTP Last-Modified header whose value is the last modified time of the request resource and which MUST be in RFC 1123 format.
Content-Type
DAP4 servers MUST return an HTTP Content-Type header, the value of which SHOULD be set in accordance with the Media Types discussion above.
Content-Description
DAP4 servers SHOULD return an HTTP Content-Description header.
Content-Disposition
DAP4 servers SHOULD return an HTTP Content-Disposition header when transmitting file typed responses.
Content-Encoding
DAP4 servers MUST return an HTTP Content-Encoding header when the content-coding of an entity is not "identity".


4.5.2.2 DAP Specific
X-DAP-Server
DAP4 servers SHOULD return the X-DAP-Server HTTP header. This HTTP header is used in a response to communicate the software version of the server. This may be a simple string with the server name and version number, or multiple software component versions may be represented. The value of this header is a string determined by the implementations author(s).
Example
X-DAP-Server: bes/3.10.0, libdap/3.11.2, dap-server/ascii/4.1.2, csv_handler/1.0.2, freeform_handler/3.8.4, fits_handler/1.0.7, fileout_netcdf/1.1.2, gateway_module/1.1.0, hdf4_handler/3.9.4, hdf5_handler/1.5.0, netcdf_handler/3.10.0, ncml_module/1.2.1, dap-server/usage/4.1.2, dap-server/www/4.1.2, xml_data_handler/1.0.1
Example
X-DAP-Server: TDS-4.19.3
X-DAP
DAP4 servers MUST return the X-DAP HTTP header. This HTTP header is used in a response to indicate the DAP protocol version used to encode the content of the response. This value is constrained to a format of "major_version" dot "minor version", where both major_version and minor_version are represented by an integer value.
Example
X-DAP: 4.0
Example
X-DAP: 2.17
4.5.2.3 Asynchronous Transaction
X-DAP-Async-Required
DAP4 servers MUST return the X-DAP-Async-Required header if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the 400 DAP Asynchronous Response Required HTTP response code.

4.6 Status Codes

DAP servers that provide an HTTP interface are expected to utilize the HTTP response codes in a manner consistent with the HTTP 1.1 specification.

The ones that are detailed here are used by the DAP in a manner consistent with the specifications definition, but in support of specific DAP server behavior.

4.6.1 200 OK

A server MUST return an HTTP status of 200 when the request has been successful and that the returned document contains the requested resource. However since DAP responses can be quite large and since it is also possible for the server to encounter any number of problems during the marshaling, serialization, and subsequent transmission of the response it is possible that the server may have committed/transmitted the HTTP headers (in which the status value is transmitted) before a subsequent error is encountered. These types of errors are transmitted in the DAP4 over-the-wire protocol and all DAP4 clients MUST be able to read and process these errors. See Section?

4.6.2 202 Accepted

A server MUST return a 202 Accepted HTTP response code when a request has been accepted and will be handled asynchronously. The response body MUST contain a DAP asynchronous response accepted document.

4.6.3 400 Bad Request

The HTTP specification defines this status code as:

The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

DAP servers utilizes this code to mean one of two things.

4.6.3.1 400 Bad DAP4 Request Syntax
The 400 Bad DAP4 Request Syntax HTTP response code MUST be returned by the server when there is a problem with the syntax of the OPeNDAP URL. The problem could be in the formulation of the constraint expression, or it may be that the URL extension did not match any that are recognized by this server.
4.6.3.2 400 DAP4 Asynchronous Response Required
The 400 DAP4 Asynchronous Response Required HTTP response code MUST be returned by the server when the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response. The server MUST include the X-DAP-Async-Required HTTP header and the DAP4 Asynchronous Response Required document as the response body.

4.6.4 401 Unauthorized

The 401 Unauthorized HTTP response code MUST be returned by the server when access to the requested resource requires (not previously acquired) user authentication. See the HTTP specification for usage and behavior.

4.6.5 403 Forbidden

The 403 Forbidden HTTP response code MUST be returned when the server has understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. This is appropriate to return if, for example, the server software does not have read permission for the requested resource.

4.6.6 404 Not Found

The 404 Not Found HTTP response code MUST be returned when the server has not found anything matching the Request-URI.

4.6.7 409 Conflict (DAP4 Async Response Not Available)

The 409 Conflict HTTP response code SHOULD be returned by the server when the DAP request has been rejected because a previous asynchronous request has not been completed and the result is not ready for access. The response body MUST contain the DAP4 Asynchronous Response Not Yet Available document.

4.6.8 412 Precondition Failed

The 412 Precondition Failed HTTP response code MUST be returned by the server if the DAP request has been rejected because it did not meet the X-DAP-Async-Accept condition that was specified in the request. The response body MUST contain the DAP4 Asynchronous Request Rejected document.

4.6.9 415 Unsupported Media Type

The 415 Unsupported Media Type HTTP response code MUST be returned when the client requests a representation of the requested resource that the server cannot provide.

4.6.10 500 Internal Server Error

The 500 Internal Server Error SHOULD be returned when the DAP server encounters internal problems when attempting to fulfill a request.

4.7 Verbs (aka Methods)

4.7.1 GET

A DAP request may be made using the HTTP GET request method utilizing a Uniform Resource Identifier (URI) that encodes information specific to the DAP (see Section ?).

Each GET request MUST conform to the HTTP specification (which basically says that a GET request MUST contain an HTTP protocol version number followed by a MIME-like message containing various headers that further describe the request.). While there are some optional DAP HTTP request headers that may be used, DAP requests do not require specific HTTP headers beyond those necessary for HTTP. DAP servers SHOULD support the use of the HTTP Accept request header to allow clients to engage in HTTP content negotiation for specific representations of a DAP response.

Implementation Note
In practice, DAP clients typically use a third-party library implementation of HTTP/1.1 so the GET request, URI and HTTP version information are hidden from the client; it sees only the DAP Uniform Resource Locator (URL) and the request headers.

The DAP server responds to the GET request with an HTTP compliant response (one that includes a status line containing the HTTP protocol version and an error or success code, followed by HTTP response headers and then response itself). There are two DAP specific HTTP headers that are always included in a DAP response over HTTP: X-DAP-Server and X-DAP, as described above. The DAP response is the payload of the HTTP response. Unless otherwise negotiated, the data response payload is encoded in multpart-MIME format, while the other responses are simple HTTP responses This is further described in Section ?.

4.7.1.1 Examples
HTTP GET request.
GET /dap/path/data.nc.dap?u,v[0:8:1024] HTTP/1.1
Host: server.org
Accept: application/vnd.org.opendap.dap4.data



5 Normative References

[DAP4_Vol1] DAP4 Volume 1: "Data Model, Persistent Representations, and Constraints".

[HTTP_RFC2616] "Hypertext Transfer Protocol -- HTTP/1.1" (as text or HTML).

6 Non-Normative References

[REST] R. Fielding, UC Irvine Doctoral Thesis: "Architectural Styles and the Design of Network-based Software Architectures"

7 Extra Web Service Stuff

7.1 Other Common Services

DAPx servers typically offer a number of other services that, while not exactly DAP services, are commonly available. This section lists just some of the alternate services a DAP4 server might provide. By enumerating the available services in the Dataset Services Response servers can easily make clients (either software and monkey) aware of their ancillary capabilities.

7.1.1 DAP4: HTML DATA Request Form Service

The HTML DATA Request Form Service provides browser based access to the Dataset. When invoked it returns a web-browser renderable document (in html) that provides a form (or other UI) that can be used to constrain and request data in accordance with the DAP4 specification as applied to the dataset .


suffix = .html
service url = dataset_url + .html
role id = http://services.opendap.org/dap4/data-request-form#

Default/primary media type: text/html | text/xhtml

7.1.2 DAP4: RDF Service

The RDF service provides an RDF representation of the Dataset document (DDX). The RDF response is an XML document containing an RDF version of the DAP4: Dataset Response.


suffix = .rdf
service url = dataset_url + .rdf
role id = http://services.opendap.org/dap4/rdf#

Default/primary media type: application/rdf+xml

7.1.3 DAP4: ISO 19115 Service

This service provides ISO 19115 metadata for the Dataset, if any can be found. When invoked it returns an XML document containing ISO 19115 metadata located in the DAP4: Dataset Response.


suffix = .iso
service url = dataset_url + .iso
role id = http://services.opendap.org/dap4/iso-19115-metadata#

Default/primary media type: text/xml

7.1.4 DAP4: ISO Conformance Score Service

This service provides a browser renderable document that describes how well the metadata held in the Dataset conforms to ISO 19115. When invoked this service returns a browser renderable document that scores how well the metadata held in the Dataset Response conforms to ISO 19115.


suffix = .rubric
service url = dataset_url + .rubric
role id = http://services.opendap.org/dap4/iso-19115-score#

Default/primary media type: text/html

7.1.5 DAP4: NetCDF File-out Service

This service provides data responses in NetCDF-4 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 4 file.

suffix = .nc4
service url = dataset_url + '.nc4' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf-4

7.1.6 DAP4: ASCII Data Service

This service provides data responses in ASCII format. When invoked the regular DAP data response will be repackaged as an ASCII representation of the data values.


suffix = .ascii
service url = dataset_url + '.ascii' + [?dap_constraint]
role id = http://services.opendap.org/dap4/ascii#

Default/primary media type: text/plain

7.1.7 DAP4: XML Data Service

This service provides data responses in XML format. When invoked the constrained Dataset response document (DDX) will be marked up with the data values of the request and returned. Large requests may be denied.


suffix = .xdap
service url = dataset_url + '.xdap' + [?dap_constraint]
role id = http://services.opendap.org/dap4/xml-data#

Default/primary media type: text/xml

7.1.8 DAP4: Native File Access Service

This service provides direct access to the data source file (or whatever else) that is creating the DAP dataset resource. When invoked it returns the "native" data from whatever store (filesystem, etc.) it may be in. Servers MAY elect to not support this response, for example, for generated data, very large data, et cetera.


suffix = .file
service url = dataset_url + .file
role id = http://services.opendap.org/dap4/file#

Default/primary media type: Type varies with file type.

7.1.9 DAP4: Server Version Service

This service provides software versioning information. When invoked the service returns an XML file containing a description of the version of the server and it's components.


suffix = .ver
service url = dataset_url + .ver
role id = http://services.opendap.org/dap4/version#

Default/primary media type: text/xml

7.2 DAP2 Services

In order to support legacy client applications DAP4 server implementations MAY support the DAP2 services stack. If they do so the DAP2 services MUST be organized as described in this section.


7.2.1 DAP2: Data Service

The DAP2 data service provides DAP2 data access to the data resource.


suffix = .dods
service url = dataset_url + '.dods' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dods#

Default/primary media type: application/octet-stream

7.2.2 DAP2: DDX Service

The DAP2 DDX service provides DAP2 access to the data resource metadata. When invoked the service returns an XML document containing both syntactic and semantic dataset metadata in DAP2 XML format.

suffix = .ddx
service url = dataset_url + .ddx
role id = http://services.opendap.org/dap2/ddx#

Default/primary media type: text/xml

7.2.3 DAP2: DDS Service

The DAP2 DDS service provides access to the 'syntactic' metadata (aka use or structural metadata) for the data resource. When invoked returns a DAP2 DDS response document conforming to the DDS part of the DAP2 specification.


suffix = .dds
service url = dataset_url + '.dds' + [?dap_constraint]
role id = http://services.opendap.org/dap2/dds#

Default/primary media type: text/plain

7.2.4 DAP2: DAS Service

The DAP2 DAS service provides access to the 'semantic' metadata (aka domain metadata) for the data resource. When invoked returns a DAP2 DAS response document conforming to the DAS part of the DAP2 specification.


suffix = .das
service url = dataset_url + .das
role id = http://services.opendap.org/dap2/das#

Default/primary media type: text/plain

7.2.5 DAP2: Info Service

The DAP2 INFO service provides a browser renderable page that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way. When invoked this service returns a web browser renderable document that combines both the DAP2 'syntactic' and 'semantic' metadata for the data resource in a human readable way.


suffix = .info
service url = dataset_url + .info
role id = http://services.opendap.org/dap2/Info#

Default/primary media type: text/html

7.2.6 DAP2: NetCDF Service

This service provides data responses in NetCDF-3 file format. When invoked the regular DAP data response will be repackaged as a NetCDF 3 file.

suffix = .nc
service url = dataset_url + '.nc' + [?dap_constraint]
role id = http://services.opendap.org/dap4/netcdf-3#

Default/primary media type: application/x-netcdf


Template: ServiceTemplate