DAP 4.0 Essential Features

Version

We need to be able to extend the protocol so that different versions of servers and clients can still interoperate. This is, of course, a going to have conditions where data that cannot be encoded using an older version of the protocol cannot be served to an older client (because there's no way for it to support the newer protocol and still be an 'older client'). But newer clients can understand the older protocol and should be have reasonably.

Error

We need a way to reliably report errors when they are detected in midstream of data transmission. This is the primary 'new feature' of the transmission part of the protocol.

NGrid

We need a way to represent more sophisticated kinds of data organization than the simple vector maps supported by Grid. This is a change to the data model of the protocol.

Shared Dimensions

Shared Dimensions may not be necessary if NGrid is defined correctly. Alternatively, they may provide a way to redefine how the concepts represented by Grid and NGrid are encoded in the data model. I think these two things are different approaches to the same basic concepts.

These three/four features are the critical changes to the DAP to move it from version 2 to version 4. Other types like 64-bit integers, Opaque, and Enum seem critical, but once version number processing is handled correctly, they become much simpler enhancements. The NGrid/Shred-Dimension data model extension is fundamentally one that will affect the flow of control in clients, so it has more profound consequnces.

Left off this list are two lurking issues:

• Groups, as defined in HDF5
• Asynchronous responses

Version

There are three techniques we are looking at and/or have implemented so far:

• Using HTTP MIME headers (which can be generalized to be Request MIME document headers; not limited to HTTP)
• Using a function or keyword in the query string part of the URL
• Making new URLs by adding to the path part of the URL

None are perfect...

HTTP/MIME headers

The existing code uses the XDAP HTTP/MIME header. This could be adapted for use with other protocols but that can be cumbersome. This does not work for clients like web browsers without fairly complex setup. Compounding this is that client libraries are not using it.

Here's a compatibility matrix:

Compatibility Matrix

Server kind	2.0 Client	Generic client	3.x Client
Old Server	Works	Works	Works*
New Server	Partial**	Works	Works

*Client Falls back to DAP 2.0 (required by protocol)
**New servers have to support the old protocol, but there may be some features/plugins that decide they cannot render all data sets. In this case 'supporting the protocol' might mean returning an error message, which is why this gets 'Partial'.

Function/keyword

Another idea: Put something in the query part of the URL. This might look like this:

http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?dap(3.2),u[0:1:15][0:1:16][0:1:20]

Although parens are not technically allowed in a URL so the result is often the fairly obtuse:

http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?dap%283.2%29,u[0:1:15][0:1:16][0:1:20]

Since this would require changing the CE syntax, we could address the parens issue by adding keywords to the set of allowable things passed in a CE. The resulting URL might look like:

http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?dap3.2,u[0:1:15][0:1:16][0:1:20]

Both of these URLs have the same basic problem: with a server that does not recognize the function or keyword, the response is an error (a 400 response). So this syntax breaks existing servers, which means that to use this syntax in an automated sense a client needs to check for a 400 response and, if one comes back, guess it might be the function/keyword and try the request without it.

Here's a compatibility matrix:

Compatibility Matrix

Server kind	2.0 Client	Generic client	3.x Client
Old Server	Works	Works	Partial*
New Server	Partial**	Works	Works

*Client will get an obtuse error message when it uses the keyword and will have to either guess that maybe this is an old server or look at the version response.
**As with the above option using a MIME header, the 2.0 client may get an error response from the server. In this case, it can be verbose and explain the problem/situation, which is an improvement over the 3.x client --> Old server condition.

URL Path

We could embed the protocol version in the path part of the URL. Doing this will likely complicate issues we already have with the dispatch software. It also does not fix the issue, from the client perspective, that a request from a 3.x client to an old server results in a somewhat inscrutable error message. It does not require a change to the CE parser or hack to the code that does initial processing of the request (DODSFilter in libdap).

Compatibility Matrix

Server kind	2.0 Client	Generic client	3.x Client
Old Server	Works	Works	Partial*
New Server	Partial**	Works	Works