DAP4: Asynchronous Request-Response Proposal v3: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 166: Line 166:
Accept: multipart/mixed
Accept: multipart/mixed
</pre></font>
</pre></font>
Alternately, this request would produce the same result using only the URL:
<font size="2"><pre>
GET /dap/path/data.nc.dap?acceptAsync(),x,y,temp HTTP/1.1
Host: server.org
</pre></font>


Response:
Response:
Line 185: Line 192:


</source></font>
</source></font>
Alternately, this request would produce the same result using only the URL:
<font size="2"><pre>
GET /dap/path/data.nc.dap?acceptAsync(),x,y,temp HTTP/1.1
Host: server.org
</pre></font>


=== Constrained Data Request-Response with conditional DAP-Async-Accept Request Headers ===
=== Constrained Data Request-Response with conditional DAP-Async-Accept Request Headers ===

Revision as of 20:11, 29 June 2012

<-- back to OPULS Development

-- EthanDavis ndp 15:55, 15 June 2012 (PDT)

Attribution

This document represents my attempt at synthesizing/resolving the key aspects of Ethan's and Nathan's (that's me) proposals for DAP Async. I will begin with EThan's proposal as the template and I will inject the concepts that I think are important from my perspective.

Background

Over the years, a number of users have requested support for asynchronous request/response in OPeNDAP. Most of the requests have been from large data centers dealing with near-line data archives (e.g., tape archives) where access to the data may take tens of minutes. Another use case that has been discussed is server-side processing which, depending on the processing, may take some time to calculate.

At least one ad hoc attempt to implement this functionality has been made by an external group.

Problem addressed

This proposal aims to support requests for data that are not readily available. The proposal builds on some of the ideas in the DAP4: Capabilities and Versioning.

This proposal tries to support the following use-cases/requirements:

  • When a client that does not understand asynchronous DAP makes a request that requires an asynchronous response, it should be able to fail gracefully.
  • When a client does not know an asynchronous response will be required on a data request, it should be able to recover and use information in the response to decide to
    • not continue with a request
    • make request explicitly stating that it is willing to accept an asynchronous response.
  • Allow for the following notification schemes:
    • Client polling (default)
    • Email notification
    • Maintain connection and wait for response (see COMET, long poll, connection stayalive, etc.)
    • Client application provide HTTP URL on which it can receive notification (see Facebook Graph API's [notification] for how they support this).

Proposed solution

Using links contained in a dedicated asynchronous response document, DAP specific HTTP headers, along with specific service parameters that allow for the operation of the Async interface to be encapsulated in the URL.

Servers must reject requests that require an asynchronous response if the client has not indicated willingness to accept such a response. Rejection of such requests is indicated with a 400 DAP Asynchronous Response Required HTTP response code, inclusion of the X-DAP-Async-Required HTTP response header. The response body must contain the DAP asynchronous response XML documet. A client can indicate willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header.

Clients can make conditional requests for asynchronous responses including providing an indication the maximum time they are willing to wait with a X-DAP-Async-Accept HTTP header with a value given in milliseconds.

HTTP Response Headers

DAP Asynchronous Response Required

X-DAP-Async-Required

The X-DAP-Async-Required HTTP response header is included in the response if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the 400 DAP Asynchronous Response Required HTTP response code.

HTTP Request Headers

Accept DAP Asynchronous Response

X-DAP-Async-Accept

A client indicates willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header. Clients can make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the X-DAP-Async-Accept HTTP header with a value given in milliseconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.

HTTP Response Codes

202 Accepted

A server indicates that a request has been accepted and will be handled asynchronously by returning a 202 Accepted HTTP response code. The response body must contain a document in one of the asynchronous information media types listed below.

400 DAP Asynchronous Response Required

The 400 DAP Asynchronous Response Required HTTP response code is used to indicate that the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response.

The response code text is used to indicate the reason for the rejection. However, since the 400 HTTP response code is not specific to asynchronous DAP (the standard text for the 400 code is "Bad Request"), the DAP-Async-Required HTTP response header is also included in the response (see above).

Note that a standard 400 HTTP response code is returned. In this way, a client that does not understand asynchronous DAP can fail gracefully. The response code text message has been changed to be more informative of the reason for the failure. For clients that are aware of asynchronous DAP, the "DAP-Async-Required" header is set to "true". The body of the response also returns some information the client can use to decide on how it will continue.

An alternative would be to use a non-standard 4xx HTTP response code (e.g., we could choose 473). Clients should interpret any 4xx code that they do not recognize as a 400. However, this may not be handled well by all clients.

412 Precondition Failed

The 412 Precondition Failed HTTP response code is used to indicate that the DAP request has been rejected because it did not meet the DAP-Async-Accept-If-Expected-Delay-Less-Than condition (see above) that was specified in the request.

Asynchronous Response Documents

The two uses of these documents are:

  • to inform clients that a request will result in an asynchronous response and
  • to provide clients with the status of an an accepted asynchronous request.
  • to inform clients that a request for and asynchronous response has been rejected.

These documents are by default rendered as XML that follows the DAP Asynchronous XML schema and are declared in the namespace http://opendap.org/ns/dap/asynchronous. Any particular server implementation MAY offer other representations of these documents (such as JSON, or ProtoBuf) and if so these alternate representations SHOULD be included in the Dataset Services response, and HTTP content negotiation should be used by clients to access these alternate representations. (Does this make sense? How do we include Asynchronous links etc. in the Dataset Services response? It seems like it would be so much simpler (K.I.S.S.) to just say - Asynchronous response documents are XML. Period.)

DAP Asynchronous Response Required

<asynchronous status="required">
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</asynchronous>

DAP Asynchronous Request Accepted

<asynchronous status="accepted">
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          rel="http://opendap.org/rel/dap/asynch/status"
          type="application/vnd.opendap.org.dap.asynchronous+xml"
          href="http://server.org/dap/path/data.nc-ABCD.status" />
  ...
</asynchronous>

DAP Asynchronous Request Rejected

<asynchronous status="rejected">
    <expectedDelay millisec="600000" /> <!-- Greater than that specified in DAP-Async-Accept-If-Expected-Delay-Less-Than header. -->
    <description>Acceptable access delay was less than estimated delay.</description>
</asynchronous>

Rationale for the solution

Discussion

Examples

Constrained Data Request-Response using GET

Simple Request
GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
Accept: */*

If the server decides it needs to handle this request in an asynchronous manner, it will refuse the request because it did not say it would accept an asynchronous response.

Response
400 DAP Asynchronous Response Required
DAP-Async-Required: true
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</asynchronous>


Constrained Data Request-Response with DAP-Async-Accept Request Header

Request:

GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
DAP-Async-Accept: 0
Accept: multipart/mixed

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync(),x,y,temp HTTP/1.1
Host: server.org


Response:

202 Accepted
Content-Type: text/xml;charset=UTF-8

<asynchronous status="accepted">
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          rel="http://opendap.org/rel/dap/asynch/status"
          type="application/vnd.opendap.org.dap.asynchronous+xml"
          href="http://server.org/dap/path/data.nc-ABCD.result" />
  ...
</asynchronous>

Constrained Data Request-Response with conditional DAP-Async-Accept Request Headers

Request:

GET /dap/path/data.nc?x,y,temp HTTP/1.1
Host: server.org
DAP-Async-Accept: 60000
Accept: application/vnd.opendap.org.dap4.data

Response:

412 Precondition Failed
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous status="rejected">
    <expectedDelay millisec="600000" /> <!-- Greater than that specified in DAP-Async-Accept-If-Expected-Delay-Less-Than header. -->
    <description>Acceptable access delay was less than estimated delay.</description>
</asynchronous>

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync(60000),x,y,temp HTTP/1.1
Host: server.org

Constrained Data Request-Response using POST

Request:

POST /dap/path/data.nc HTTP/1.1
Host: server.org
DAP-Async-Accept: true
Content-Type: application/vnd.opendap.org.dap4.ce+xml;charset=UTF-8
  
<constraintExp>...</constraintExp>

Response:

202 Accepted
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<asynchronous>
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <status value="accepted"
          rel="self
               http://opendap.org/rel/dap/asynchronous"
          type="application/vnd.opendap.org.dap.asynchronous+xml"
          href="http://server.org/dap/path/data.nc-ABCD.status" />
  ...
</asynchronous>

Alternately, this request would produce the same result using only the URL:

POST /dap/path/data.nc.dap HTTP/1.1
Host: server.org

<constraintExp>acceptAsync(60000),x,y,temp</constraintExp>


Support notification of completion

Request:

Response: