Difference between revisions of "DAP4: Asynchronous Request-Response Proposal v3"

From OPeNDAP Documentation
(foo)
Line 365: Line 365:
 
== Discussion ==
 
== Discussion ==
  
== Discussion ==
+
I think this is better than sliced bread [[User:Ndp|ndp]]

Revision as of 17:55, 9 July 2012

<-- back to OPULS Development

-- EthanDavis ndp 15:55, 15 June 2012 (PDT)

1 Attribution

This document represents my attempt at synthesizing/resolving the key aspects of Ethan's and Nathan's (that's me) proposals for DAP Async. I will begin with EThan's proposal as the template and I will inject the concepts that I think are important from my perspective.

2 Background

Over the years, a number of users have requested support for asynchronous request/response in OPeNDAP. Most of the requests have been from large data centers dealing with near-line data archives (e.g., tape archives) where access to the data may take tens of minutes. Another use case that has been discussed is server-side processing which, depending on the processing, may take some time to calculate.

At least one ad hoc attempt to implement this functionality has been made by an external group.

3 Problem addressed

This proposal aims to support requests for data that are not readily available. The proposal builds on some of the ideas in the DAP4: Capabilities and Versioning.

This proposal tries to support the following use-cases/requirements:

  • When a client that does not understand asynchronous DAP makes a request that requires an asynchronous response, it should be able to fail gracefully.
  • When a client does not know an asynchronous response will be required on a data request, it should be able to recover and use information in the response to decide to
    • not continue with a request
    • make request explicitly stating that it is willing to accept an asynchronous response.
  • Allow for the following notification schemes:
    • Client polling (default)
    • Email notification
    • Maintain connection and wait for response (see COMET, long poll, connection stayalive, etc.)
    • Client application provide HTTP URL on which it can receive notification (see Facebook Graph API's [notification] for how they support this).

4 Proposed solution

We propose utilizing two related pathways for achieving asynchronous response capabilities in DAP4: One utilizes a (soon to be ) well known key-value-pair as part of the DAP4 constraint expression syntax to allow the client to inform the server of it's willingness to accept an asynchronous repines. The second leverages a set DAP HTTP headers to achieve the same goal. In both cases the the server will return dedicated asynchronous response documents.


4.1 Client/Server Interaction

Servers must reject requests that require an asynchronous response if the client has not indicated willingness to accept such a response. Rejection of such requests is indicated with:

A client can indicate willingness to accept asynchronous responses in one of two ways:

If the client indicates that it must have access to the asynchronous response content within a certain time (utilizing either the X-DAP-Async-Accept HTTP header and/or the acceptAsync keyword in the constraint expression) and the server ascertains that the response will not be available in the clients time frame then the server MUST reject the request and returning an HTTP status of 412 and the DAP Asynchronous Request Rejected XML document.

When an asynchronous request is accept by the server the server will return the DAP Asynchronous Request Accepted XML document. This document contains a link to the pending result of the request.

If the client attempts to access the asynchronous result prior to it's availability the server MUST return an HTTP response status of 409 (DAP Response Not Ready) along with the DAP Asynchronous Response Not Available XML document.

If the client attempts to access the asynchronous result after it's availability the server SHOULD return an HTTP response status of 410 (Gone). Or I suppose if someone objects to 410 we could use 404 (Not Found).

4.2 DAP4 Constraint Expression extension for Async

By adding a keyword and value pair to the DAP4 constraint expression we can allow a client to encode it's willingness to accept an asynchronous response, along with the a maximum amount of time the client can wait before it can access the response.

acceptAsync
A value of zero indicates the client is willing to unconditionally accept an asynchronous response. A positive integer value will be interpreted as the number of milliseconds that the client will wait for access to the response. If the value is negative the serve MUST return an error.
Examples
Client is willing to unconditionally accept an asynchronous response
?acceptAsync=0&projection=x,y,temp
Client is willing to wait for 60000 milliseconds (1 minute) for access to the asynchronous response
?acceptAsync=60000&projection=x,y,temp

4.3 HTTP

In many (most?) cases the DAP4 protocol is going to be transported using HTTP. It is important for any DAP server/client interaction over HTTP to correctly utilize the various components of the HTTP protocol when possible. In order to support asynchronous responses DAP4 will utilize two dedicated DAP HTTP headers in addition to various HTTP response/status codes described below.

4.3.1 HTTP Response Headers

4.3.1.1 X-DAP-Async-Required

The X-DAP-Async-Required HTTP response header is included in the response if the request requires an asynchronous response and the client has not indicated willingness to accept such a response. Rejection of the request should also be indicated by the 400 DAP Asynchronous Response Required HTTP response code.

4.3.2 HTTP Request Headers

4.3.2.1 X-DAP-Async-Accept

A client indicates willingness to accept asynchronous responses by including the X-DAP-Async-Accept HTTP header. Clients can make conditional requests for asynchronous responses by indicating the maximum time they are willing to wait by using the X-DAP-Async-Accept HTTP header with a value given in milliseconds. A value of zero indicates that the client is willing to accept whatever delay the server may encounter.

4.3.3 HTTP Response Codes

4.3.3.1 202 Accepted

A server indicates that a request has been accepted and will be handled asynchronously by returning a '202 Accepted' HTTP response code. The response body must contain a document in one of the asynchronous information media types listed below.

4.3.3.2 400 DAP Asynchronous Response Required

The '400 DAP Asynchronous Response Required' HTTP response code is used to indicate that the DAP request has been rejected because an asynchronous response is required and the client did not indicate willingness to accept an asynchronous response.

The response code text is used to indicate the reason for the rejection. However, since the '400' HTTP response code is not specific to asynchronous DAP (the standard text for the '400' code is "Bad Request"), the X-DAP-Async-Required HTTP response header is also included in the response (see above).

Note that a standard 400 HTTP response code is returned. In this way, a client that does not understand asynchronous DAP can fail gracefully. The response code text message has been changed to be more informative of the reason for the failure. For clients that are aware of asynchronous DAP, the "DAP-Async-Required" header is set to "true". The body of the response also returns some information the client can use to decide on how it will continue.

An alternative would be to use a non-standard 4xx HTTP response code (e.g., we could choose 473). Clients should interpret any 4xx code that they do not recognize as a 400. However, this may not be handled well by all clients.

4.3.3.3 409 Conflict - DAP Response Not Ready

The '409 Conflict' HTTP response code is used to indicate that the DAP request has been rejected because a previous asynchronous request has not been completed and the result is not ready for access.

4.3.3.4 412 Precondition Failed

The '412 Precondition Failed' HTTP response code is used to indicate that the DAP request has been rejected because it did not meet the DAP-Async-Accept-If-Expected-Delay-Less-Than condition (see above) that was specified in the request.

4.4 Asynchronous Response Documents

The two uses of these documents are:

  • to inform clients that a request will result in an asynchronous response and
  • to provide clients with the status of an an accepted asynchronous request.
  • to inform clients that a request for and asynchronous response has been rejected.

These documents are by default rendered as XML that follows the DAP Asynchronous XML schema and are declared in the namespace http://opendap.org/ns/dap/asynchronous. Any particular server implementation MAY offer other representations of these documents (such as JSON, or ProtoBuf) and if so these alternate representations SHOULD be included in the Dataset Services response, and HTTP content negotiation should be used by clients to access these alternate representations. (Does this make sense? How do we include Asynchronous links etc. in the Dataset Services response? It seems like it would be so much simpler (K.I.S.S.) to just say - Asynchronous response documents are XML. Period.)

4.4.1 DAP Asynchronous Response Required

This document informs clients that a request will result in an asynchronous response, and that the client has not yet indicated it's willingness to accept an asynchronous response.

<AsynchronousResponse status="required">
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</AsynchronousResponse>

4.4.2 DAP Asynchronous Request Accepted

This document informs clients that a request for an asynchronous response has been accepted, along with operational information about retrieving the asynchronous response result.

<AsynchronousResponse status="accepted">
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          role="http://services.opendap.org/dap4/data#"
          type="multipart/mixed"
          href="http://server.org/async/path/data.nc?projection=x,y,temp" />
  ...
</AsynchronousResponse>


4.4.3 DAP Asynchronous Response Not Available

This document informs clients that a while a previous request for an asynchronous response has been accepted the result is not available.

<AsynchronousResponse status="pending">
  <expectedDelay millisec="599979" /> <!-- Estimated delay. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          role="http://services.opendap.org/dap4/data#"
          type="multipart/mixed"
          href="http://server.org/async/path/data.nc?projection=x,y,temp" />
  ...
</AsynchronousResponse>

4.4.4 DAP Asynchronous Request Rejected

This document informs clients that a request for an asynchronous response has been rejected because the client indicated that it wanted the result sooner than could be accomplished.

<AsynchronousResponse status="requestRejected">
    <expectedDelay millisec="600000" /> <!-- Greater than that specified in DAP-Async-Accept-If-Expected-Delay-Less-Than header. -->
    <description>Acceptable access delay was less than estimated delay.</description>
</asynchronous>
</AsynchronousResponse>

4.5 Examples

4.5.1 Constrained Data Request-Response using GET

Simple Request
GET /dap/path/data.nc?projection=x,y,temp HTTP/1.1
Host: server.org
Accept: */*

If the server decides it needs to handle this request in an asynchronous manner, it will refuse the request because it did not say it would accept an asynchronous response.

Response
400 DAP Asynchronous Response Required
X-DAP-Async-Required: true
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<AsynchronousResponse status="required">
  <expectedDelay millisec="600000" />
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" />
    <email />
    <longConnect />
    <http />
  </notificationSupport>
  ...
</AsynchronousResponse>

4.5.2 Constrained Data Request-Response with DAP-Async-Accept Request Header

Request:

GET /dap/path/data.nc?projection=x,y,temp HTTP/1.1
Host: server.org
X-DAP-Async-Accept: 0
Accept: multipart/mixed

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync=0&projection=x,y,temp HTTP/1.1
Host: server.org


Response:

202 Accepted
Content-Type: text/xml;charset=UTF-8

<AsynchronousResponse status="accepted">
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          role="http://services.opendap.org/dap4/data#"
          type="multipart/mixed"
          href="http://server.org/async/path/data.nc?projection=x,y,temp" />
  ...
</AsynchronousResponse>


4.5.3 Constrained Data Request-Response with conditional DAP-Async-Accept Request Headers

Request:

GET /dap/path/data.nc?projection=x,y,temp HTTP/1.1
Host: server.org
X-DAP-Async-Accept: 60000
Accept: application/vnd.opendap.org.dap4.data

Alternately, this request would produce the same result using only the URL:

GET /dap/path/data.nc.dap?acceptAsync=60000&projection=x,y,temp HTTP/1.1
Host: server.org

Response:

412 Precondition Failed
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<AsynchronousResponse status="requestRejected">
    <expectedDelay millisec="600000" /> <!-- Greater than that specified in DAP-Async-Accept-If-Expected-Delay-Less-Than header. -->
    <description>Acceptable access delay was less than estimated delay.</description>
</AsynchronousResponse>


4.5.4 Premature Request For Asynchronous Result

Request:

GET /async/path/data.nc?projection=x,y,temp HTTP/1.1
Host: server.org
Accept: multipart/mixed

Alternately, this request would produce the same result using only the URL:

GET /async/path/data.nc?projection=x,y,temp HTTP/1.1
Host: server.org


Response:

409 Conflict
Content-Type: text/xml;charset=UTF-8

<AsynchronousResponse status="pending">
  <expectedDelay millisec="599979" /> <!-- Estimated delay. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          role="http://services.opendap.org/dap4/data#"
          type="multipart/mixed"
          href="http://server.org/async/path/data.nc?projection=x,y,temp" />
  ...
</AsynchronousResponse>

4.5.5 Constrained Data Request-Response using POST

Request:

POST /dap/path/data.nc HTTP/1.1
Host: server.org
X-DAP-Async-Accept: true
Content-Type: application/vnd.opendap.org.dap4.ce+xml;charset=UTF-8
  
<constraintExp>...</constraintExp>

Alternately, this request would produce the same result using only the URL:

POST /dap/path/data.nc.dap HTTP/1.1
Host: server.org

<constraintExp>acceptAsync=60000&projection=x,y,temp</constraintExp>

Response:

202 Accepted
Content-Type: application/vnd.opendap.org.dap.asynchronous+xml;charset=UTF-8
 
<AsynchronousResponse status="accepted">
  <expectedDelay millisec="600000" /> <!-- Estimated delay of 10 minutes. -->
  <notificationSupport>
    <polling frequencyLimitInMillisecs="60000" /> <!-- Don't poll more often than every minute. -->
  </notificationSupport>
  <access
          role="http://services.opendap.org/dap4/data#"
          type="multipart/mixed"
          href="http://server.org/async/path/data.nc?projection=x,y,temp" />
  ...
</AsynchronousResponse>

4.5.6 Support notification of completion

Request:

thing

Response:

thing

5 Rationale for the solution

Um... Cause it's gonna work!


6 Discussion

I think this is better than sliced bread ndp