From OPeNDAP Documentation
Revision as of 14:04, 26 January 2008 by Yuan (talk | contribs) (Constraint expressions)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

1 Overview of the OPeNDAP Server Architecture

This appendix describes in functional terms the operation of a server.

Note that descriptions of server outputs naturally and unavoidably refer to certain kinds of inputs and vice versa. This makes it difficult to create definitions without forward and backward cross-references. Please be prepared to read this specification with a \htmlonly{virtual} thumb between the pages as you flip back and forth from the output to the input sections.

1.1 Outputs

A server sends to a client one of three different sorts of messages:

  • HTML;
  • ASCII data; or
  • Binary data.

1.1.1 HTML Data

There are three different kinds of HTML data returned by a OPeNDAP server. Clients that request these data should be prepared to display the HTML to the user.

Server Information (usage
One kind of HTML data is returned in response to a request for server information (using the .info URL suffix), and contains usage information for the server, including a formatted version of the dataset DAS and DDS. This information is formatted by the usage service program, invoked by the server.
WWW Interface 
The forms-based OPeNDAP WWW interface returns HTML data to the client. This can either be a form a user can use to create a OPeNDAP contraint expression or a OPeNDAP directory listing, depending on whether the URL indicates a file with the .html suffix, or a directory containing other files.
Error messages 
A badly formed URL will result in a OPeNDAP error message, which is simply some HTML text describing the supported URL suffixes. (See Note that though an error message could in theory be returned to any client, whether or not they can display HTML, in practice, only web browser clients are prone to these kinds of errors. Aside from web browser clients (e.g. Netscape), the OPeNDAP clients issue their server requests through the OPeNDAP client core libraries, which format the URL according to the OPeNDAP conventions.

NOTE: There are two kinds of OPeNDAP error messages. The particular one

described above is issued in response to a badly formed URL. Other kinds of errors are returned as ASCII data within a OPeNDAP data document. See ( Section 6.1.3).

1.1.2 ASCII Data (Text)

These are the different kinds of ASCII data returned by a fully-equipped OPeNDAP server. ("Fully equipped" implies that the server has, on its execution path, all the supported OPeNDAP service programs.)

The OPeNDAP Data Descriptor Structure is a description of the data contained in the dataset. It contains information about the data types and names represented in the dataset. Programmers can think of the DDS as containing the type declarations for the data. The DDS is fully described in The OPeNDAP User Guide . The DDS is created with a service program called *_dds, where the * is the same two-letter abbreviation used in the server name (nph-*).
The OPeNDAP Data Attribute Structure contains information about the data in the dataset---the metadata. It is a hierarchical list of name-type-value triples, where the names of the containers correspond to entries in the DDS. The DAS is fully described in The OPeNDAP User Guide . The DAS is created with a service program called *_das, where the * is the same two-letter abbreviation used in the server name (nph-*).
If a OPeNDAP server is equipped with a service program called asciival, it can convert binary data to ascii data on the fly, allowing you to use a standard web browser (such as Netscape) to examine data. This feature can also be used to import data into a client that may not be able to process OPeNDAP data encoded in the standard binary format.

1.1.3 Binary Data

The OPeNDAP data document consists of two parts, a DDS describing the data returned, and the data itself. It may also consist of ASCII data describing an error condition.

The DDS returned with the data differs slightly from the DDS returned by a simple request using the .dds URL suffix. Whereas the DDS returned in response to that URL is a description of the dataset available, the DDS returned in a data document describes the data being returned. If you ask for an entire dataset, there will be no difference between these two DDS's. However, if you

request only a part of a dataset, the DDS included in the data document will only reflect the part of the dataset returned to you.

As an example, consider a dataset containing an array with one hundred elements. The DDS for this dataset will contain an array with one hundred elements. However, if you send a server a request for just the first fifty elements of the array, the OPeNDAP data document returned will contain a DDS with only fifty elements. This is illustrated in figure 6.1.3.
The data requested from a server is (unsurprisingly) also contained in the OPeNDAP server's response to a data request. The data is encoded using the XDR standard (eXternal Data Representation), and packed into the second part of the response document. For more information on XDR, see Internet RFC 1014. For further details, see (Section 6.1.3), below.
Error Messages 
A OPeNDAP data document may contain an error message. This is a OPeNDAP Error object (containing an ASCII error message and some other data) stuck into the HTTP document where the data would usually go. This is where error conditions on the server are noted, as well as badly formed constraint expressions and file names.


The OPeNDAP Data Document and the DDS

The OPeNDAP Data Document and the DDS. For the dataset containing the vector temp, with 100 elements, the top "Simple DDS Request" shows what the DDS might look like for that dataset. The bottom "OPeNDAP Data Document" shows what might be returned by a request for all the even-numbered elements of the temp array. Note that the DDS has been altered to allow for the reduced number of elements in the returned data array.

The HTTP response containing the OPeNDAP data document is formatted like this:

HTTP/1.0 200 OK
XOPeNDAP-Server: <server/dap version string>
Content-type: application/octet-stream
Content-Description: dods-data
Content-Encoding: deflate
\var{<blank line>}
\var{<binary data, each variable given in the order it is listed in the DDS>}

Note the following:

  • OPeNDAP servers use HTTP 1.0.
  • The version string (after the XOPeNDAP-Server: header) must be <text>/<version number> where version number is x.y or x.z.y. This version number is parsed on the client side to ensure that a client is communicating with a compatible server.
  • The Content-Encoding is used only when the document is compressed using `deflate'. (This is the same as the compression used by the gzip program, and is implemented in libz.a.
  • Content-Description is used by the OPeNDAP client to figure out if the object is an error instead of a data object. If it is, the Content-Description field should read dods-error. Encoding the DAP Data Types

The OPeNDAP transmission protocol spearates variables into three classes: scalars, arrays, and sequences. A scalar is included in the OPeNDAP data document simply by writing its XDR representation. An array is sent by writing the number of elements (as an XDR int) followed by the elements themselves, also in the XDR format. However, arrays of Byte, Int16, Int32, UInt16, UInt32, Float32, and Float64 actually have their size sent twice, once by the DAP software and once by the XDR software. Here's a hex dump of some Int32 array data that shows this behavior:

00000000: 4461 7461 7365 7420 7b0a 2020 2020 496e  Dataset {.    In
00000010: 7433 3220 755b 7469 6d65 5f61 203d 2031  t32 u[time_a = 1
00000020: 5d5b 6c61 7420 3d20 315d 5b6c 6f6e 203d  ][lat = 1][lon =
00000030: 2032 315d 3b0a 7d20 666e 6f63 313b 0a44   21];.} fnoc1;.D
00000040: 6174 613a 0a00 0000 1500 0000 15ff fff9  ata:............
00000050: 40ff fff6 6fff fff3 e5ff fff1 ffff fff3  @...o...........
00000060: 4aff fff6 9aff fffb 1c00 0002 9600 0009  J...............
00000070: b300 000b 5e00 000b 0300 000b 8200 000a  ....^...........
00000080: b900 000a ae00 000b 7300 000a 2900 0008  ........s...)...
00000090: 5b00 0007 3500 0006 da00 0007 6900 0007  [...5.......i...
000000a0: 3e                                       >

The length bytes for the section start at address 0x45 (The length is 0x00000015.) and is repeated at 0x49. Here's how you can see this:

geturl "[0:0][0:0][0:20]"

The DDS for this dataset is (use geturl -d):

 Dataset {
    Int32 u[time_a = 16][lat = 17][lon = 21];
    Int32 v[time_a = 16][lat = 17][lon = 21];
    Float64 lat[lat = 17];
    Float64 lon[lon = 21];
    Float64 time[time = 16];
} fnoc1;

Structures and Grids are sent by serializing their components, one by one, in the order of their declaration in the DDS.

Sequences have a much more complex encoding scheme because one sequence may contain another sequence and because the sequence type provides no information about the length of a given instance. To send sequences the DAP uses two different algorithms, one up to OPeNDAP version 2.14 servers and clients and a better (more compact one) after that. OPeNDAP clients at version 2.15 and later can all read from both servers but they assume that a server that does not announce its version is very old and will attempt to read Sequences using the old algorithm. (So if you're writing a new OPeNDAP server that serves Sequences, you must remember to address the version requirement.)

Here's how the new algorithm encodes a sequence:

  1. Serialize row of the sequence by:
    1. Writing the start of instance marker (0x5A), and
    2. Serializing (writing the XDR encoding of) each element in the row in the order of appearance, then
  2. Write the end of sequence marker (0x5A).

1.2 Inputs

The input to a OPeNDAP server is contained in an HTTP "GET" request. Unlike a POST, the information in this kind of request is all in the URL. Consequently, examining the parts of a OPeNDAP URL will illustrate all the different sorts of requests a OPeNDAP server can handle.

Figure 6.2 contains a description of the parts of a OPeNDAP URL, not including the "Constraint Expression." The constraint expression and the parts of the OPeNDAP URL are described in detail in The OPeNDAP User Guide[constraint] and The OPeNDAP User Guide[url-parts.html].


^     ^                        ^      ^    ^        ^

|     |                        |      |    |        |
Protocol |                        |      |    |        |
Machine Name                      |      |    |        |
Server-----------------------------      |    |        |
Directory---------------------------------    |        |
Filename---------------------------------------        |
URL Suffix----------------------------------------------


Parts of a OPeNDAP URL (without a constraint expression)

1.2.1 Request types

A OPeNDAP server is equipped to respond to several different request types. Each request type is signified by a different URL suffix. The server itself is just a dispatch script, that determines the type of a request, and dispatches the request to the appropriate service program, and relays its result back to the client.

In figure 6.2.1, a OPeNDAP client makes a GET request to a OPeNDAP server (which is just an httpd daemon equipped with a bunch of CGI programs). The daemon invokes the OPeNDAP Server, which is a simple-minded dispatch script which in turn invokes the DAS, DDS, Data, or other service program.

actual size

The Architecture of a OPeNDAP Data Server

The request types, their suffixes, and the service programs are listed in \tableref{server-spec,suffix}.

Suffix Service Program Description
.dds *_dds Returns the Data Descriptor Structure for the specified dataset.
.das *_das Returns the Data Attribute Structure for the specified dataset.
.dods *_dods Returns binary data in the form of a OPeNDAP data document. See ( pguide,server,binary).
.asc asciival Converts data requests to ASCII values before sending them back to the client. This service is useful for invoking from

simple web browsers.

.info usage Returns an HTML formatted version of the dataset DDS and DAS, and any other server and dataset information provided in *.ovr


.html none Returns a URL constraint expression builder form,

based on the dataset DDS and DAS. This is the OPeNDAP WWW interface.

.ver none Returns the server version information.

1.2.2 Constraint expressions

A OPeNDAP server can accept a "constraint expression" contained in the URL query string. The OPeNDAP constraint expression describes how a OPeNDAP server should subsample a OPeNDAP dataset before sending the data back to the client. The details of the constraint expresssion syntax are covered in The OPeNDAP User Guide[constraint]. What's important here is simply that the constraint expression is a logical expression with two clauses: projection and selection.

The "projection" clause of a constraint expression specifies the data variables requested by the client, and the "selection" clause specifies the condition under which the client wants them. That is, the projection clause might specify that the client wants to see oceanic temperature data, and the selection clause would specify that only records from below 1000 meters should be returned.

1.2.3 Server functions

Within the context of a constraint expression, a server can implement functions a client would use to specify data. Since the constraint expression has two kinds of clauses, there are two kinds of server functions: projection and selection.

To implement another server-side constraint function, see . Following is a list of the canonical server-side functions implemented in all OPeNDAP servers. Selection

yes Projection