DAP4: Asynchronous Request-Response Proposal v3

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽

<-- back to OPULS Development

-- EthanDavis ndp 15:55, 15 June 2012 (PDT)

Attribution

This document represents my attempt at synthesizing/resolving the key aspects of Ethan's and Nathan's (that's me) proposals for DAP Async. I will begin with EThan's proposal as the template and I will inject the concepts that I think are important from my perspective.

Background

Over the years, a number of users have requested support for asynchronous request/response in OPeNDAP. Most of the requests have been from large data centers dealing with near-line data archives (e.g., tape archives) where access to the data may take tens of minutes. Another use case that has been discussed is server-side processing which, depending on the processing, may take some time to calculate.

At least one ad hoc attempt to implement this functionality has been made by an external group.

Problem addressed

This proposal aims to support requests for data that are not readily available. The proposal builds on some of the ideas in the DAP4: Capabilities and Versioning.

This proposal tries to support the following use-cases/requirements:

  • When a client that does not understand asynchronous DAP makes a request that requires an asynchronous response, it should be able to fail gracefully.
  • When a client does not know an asynchronous response will be required on a data request, it should be able to recover and use information in the response to decide to
    • not continue with a request
    • make request explicitly stating that it is willing to accept an asynchronous response.
  • Allow for the following notification schemes:
    • Client polling (default)
    • Email notification
    • Maintain connection and wait for response (see COMET, long poll, connection stayalive, etc.)
    • Client application provide HTTP URL on which it can receive notification (see Facebook Graph API's [notification] for how they support this).

Proposed solution

Using links contained in a dedicated asynchronous response document, media types described in the DAP4 Service Terminus v2 Proposal, along with specific service parameters that allow for the operation of the Async interface to be encapsulated in the URL.

Servers must reject requests that require an asynchronous response if the client has not indicated willingness to accept such a response. Rejection of such requests is indicated with a 400 DAP Asynchronous Response Required HTTP response code, inclusion of the X-DAP-Async-Required HTTP response header. The response body must contain the DAP asynchronous response XML documet. A client can indicate willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header.

Clients can make conditional requests for asynchronous responses including providing an indication the maximum time they are willing to wait with a X-DAP-Async-Accept HTTP header with a value given in milliseconds.

HTTP Response Headers

DAP Asynchronous Response Required

X-DAP-Async-Required

The X-DAP-Async-Required HTTP response header is included in the response if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the 400 DAP Asynchronous Response Required HTTP response code.

HTTP Request Headers

Accept DAP Asynchronous Response

X-DAP-Async-Accept

A client indicates willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header. Clients can make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the X-DAP-Async-Accept HTTP header with a value given in milliseconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.

HTTP Response Codes

202 Accepted

A server indicates that a request has been accepted and will be handled asynchronously by returning a 202 Accepted HTTP response code. The response body must contain a document in one of the asynchronous information media types listed below.

400 DAP Asynchronous Response Required

The 400 DAP Asynchronous Response Required HTTP response code is used to indicate that the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response.

The response code text is used to indicate the reason for the rejection. However, since the 400 HTTP response code is not specific to asynchronous DAP (the standard text for the 400 code is "Bad Request"), the DAP-Async-Required HTTP response header is also included in the response (see above).

Note that a standard 400 HTTP response code is returned. In this way, a client that does not understand asynchronous DAP can fail gracefully. The response code text message has been changed to be more informative of the reason for the failure. For clients that are aware of asynchronous DAP, the "DAP-Async-Required" header is set to "true". The body of the response also returns some information the client can use to decide on how it will continue.

An alternative would be to use a non-standard 4xx HTTP response code (e.g., we could choose 473). Clients should interpret any 4xx code that they do not recognize as a 400. However, this may not be handled well by all clients.

412 Precondition Failed

The 412 Precondition Failed HTTP response code is used to indicate that the DAP request has been rejected because it did not meet the DAP-Async-Accept-If-Expected-Delay-Less-Than condition (see above) that was specified in the request.

Asynchronous Response Documents

The two main uses of documents in these media types are:

  • to inform clients that a request will result in an asynchronous response and
  • to provide clients with the status of an an accepted asynchronous request.

More formal definitions of these media types can be found:

  • The XML representation is a document that follows the DAP Asynchronous XML schema and is declared in the namespace http://opendap.org/ns/dap/asynchronous.

DAP Asynchronous Response Required

<asynchronous>
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</asynchronous>

DAP Asynchronous Request Accepted

DAP Asynchronous Request Rejected

Rationale for the solution

Discussion

Examples

Constrained Data Request-Response using GET

Request:

GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
Accept: application/vnd.opendap.org.dap4.data

If the server decides it needs to handle this request in an asynchronous manner, it will refuse the request because it did not say it would accept an asynchronous response.

Response:

400 DAP Asynchronous Response Required
DAP-Async-Required: true
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</asynchronous>

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?x,y,temp HTTP/1.1
Host: server.org

Constrained Data Request-Response with DAP-Async-Accept Request Header

Request:

GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
DAP-Async-Accept: true
Accept: application/vnd.opendap.org.dap4.data

Response:

202 Accepted
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <status value="accepted"
          rel="http://opendap.org/rel/dap/asynch/status"
          type="application/vnd.opendap.org.dap.asynchronous+xml"
          href="http://server.org/dap/path/data.nc-ABCD.status" />
  ...
</asynchronous>

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync(),x,y,temp HTTP/1.1
Host: server.org

Constrained Data Request-Response with conditional DAP-Async-Accept Request Headers

Request:

GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
DAP-Async-Accept: true
DAP-Async-Accept-If-Expected-Delay-Less-Than: 60000
Accept: application/vnd.opendap.org.dap4.data

Response:

412 Precondition Failed
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" /> <!-- Greater than that specified in DAP-Async-Accept-If-Expected-Delay-Less-Than header. -->
  ...
</asynchronous>

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync(60000),x,y,temp HTTP/1.1
Host: server.org


Constrained Data Request-Response using POST

Request:

POST /dap/path/data.nc HTTP/1.1
Host: server.org
DAP-Async-Accept: true
Content-Type: application/vnd.opendap.org.dap4.ce+xml;charset=UTF-8
  
<constraintExp>...</constraintExp>

Response:

202 Accepted
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <status value="accepted"
          rel="self
               http://opendap.org/rel/dap/asynchronous"
          type="application/vnd.opendap.org.dap.asynchronous+xml"
          href="http://server.org/dap/path/data.nc-ABCD.status" />
  ...
</asynchronous>

Alternately, this request would produce the same result using only the URL:

POST /dap/path/data.nc.dap HTTP/1.1
Host: server.org

<constraintExp>acceptAsync(60000),x,y,temp</constraintExp>


Support notification of completion

Request:

Response: