How to build the DataDDX response in/with Hyrax
1 Division of Labor
The BES and OLFS work together to support DAP, however, it is the job of the OLFS to correctly implement the DAP. The BES understands certain aspects of the DAP, but it does not actually build correct responses from DAP requests. Instead the OLFS reads a request and asks the BES to build parts of the correct response that it then assembles to make the complete response document.
To build a DataDDX response, the OLFS and BES will work together as shown in Figure 1. When the OLFS receives and processes a DAP4 DataDDX request, it will first parse from that request the information the BES needs and send that along with both the Multipart MIME (MPM) start and boundary values. The BES will use a handler to build a C++ DDS object and return the MPM document that forms the payload of the DataDDX response, with a few wrinkles, such as why the start and boundary values need to be passed by the OLFS to the BES, to be described shortly.
When the OLFS begins to receive the MPM payload for the DataDDX response, it will first write out the MIME headers for the response. Following that, it will stream the response from the BES, and following that, it will append (send to the stream) the closing section of the document.
Note: In Figures 1 and 2 the OLFS, the sequence of interactions between theBES and a generic Handler are shown using a UML sequence diagram. That type of diagram is intended to be used for objects and their methods, but I thought it was a useful diagram even though the idea of the BES, OLFS or a handler as an object is a bit of a stretch.
1.1 Why this design?
This design implements the DataDDX response with as little modification to the libdap and BES software as possible (and minimal changes to the OLFS, too). However, we want it to be usable as a building block for two things besides DAP4: A SOAP interface for DAP servers, where several DataDDX responses can be held in a single SOAP envelope; and implementations of DAP4 that use transport protocols other than HTTP.
Like most designs intended for several uses, this one is a compromise. Figure 2 shows a design that would be better at addressing the issues associated with support for other transport protocols and which might offer advantages to SOAP as well. In fact, it also has a cleaner separation of duties because the OLFS takes over complete responsibility for building the headers of the MPM document. But the cost is that the BES must build two DDS objects - an expensive operation, especially for large datasets. So the compromise reached is that some effort to build MIME headers, those that are part of the payload of the response, is moved into the BES/Handler/libdap software. However, because support for a SOAP interface means that we should be able to stuff several of these MPM documents into one SOAP envelope, the trailing MPM separator information is not written by the BES/Handler/libdap. Instead, the trailing separator that closes the document needs to be written by the OLFS.
With the design presented, the BES/... software can behave as efficiently as the current DataDDS code but each response can be used as a building block in a SOAP response - one response concatenated after the other - that is closed by the OLFS so the BES/Handler software does need to know about the way the responses are being packaged. At the same time the response format is likely usable by other transport protocols because either that implementation can use MPM as its payload or take apart the response (since MIME is designed to make that operation fairly easy).
The Handler running inside the BES uses libdap's DDS class and its methods to make the DDX and XDR-encoded data blob. The DDS::print_xml() method will take the Content-Id to be used with the data blob section as a parameter. A new method will be added to build the data blob.
A new set of functions will be added to libdap's collection of MIME-header writers to build the MPM boundary sections. Two functions will be needed: One to write the boundary headers for the DDX and one to write the boundary headers for the data blob.
The libdap DODSFilter class will get a new method that will package all of these calls in a way that's similar to the existing DODSFilter::send_das(), ..., send_data() methods
The BES will need to be modified to respond to a get dataddx request and will need to accept the start and boundary values for the MPM document from the OLFS. Each handler will likely need some modification to support the new response type.
The XML request document will look like the following:
<?xml version="1.0" encoding="UTF-8"?> <request reqID ="####" > <get type="dataddx" definition="def_name" returnAs="name"> <contentStartId>SOME_ID_STRING</ContentStartId > <mimeBoundary>MIME_BOUNDARY_STRING</MimeBoundary > </get> </request>
And the bescmdln command string will be:
get dataddx for def_name contentStartId SOME_ID_STRING mimeBoundary MIME_BOUNDARY_STRING;
The OLFS will need to be modified to recognize the new request, generate the correct boundary and start values, and complete the response by building the correct response headers and closing 'separator' headers.
2 Example Response Document
Content-Type: Multipart/Related; boundary=example-2; start="<950118.AEBH@XIson.com>" type="Text/x-Okie" --example-2 Content-Type: Text/xml; charset=iso-8859-1; declaration="<950118.AEB0@host.domain>" Content-ID: <950118.AEBH@host.domain> Content-Description: dap-ddx [DDX with <blob href=cid:950118.AECB@host.domain/>] --example-2 Content-Type: application/octet-stream Content-ID: <950118.AFDH@host.domain> Content-Transfer-Encoding: gzip Content-Description: dap-blob [XDR Encoded data] --example-2--
Here are the parts, broken down into which parts of Hyrax are responsible for what:
Content-Type: Multipart/Related; boundary=example-2; start="<950118.AEBH@XIson.com>" type="Text/x-Okie"
The response headers are built by the OLFS. It must figure out the value of boundary and start and both pass those to the BES and use them to build the Content-Type header.
--example-2 Content-Type: Text/xml; charset=iso-8859-1; declaration="<950118.AEB0@host.domain>" Content-ID: <950118.AEBH@host.domain> Content-Description: dap-ddx [DDX with <blob href=cid:950118.AECB@host.domain/>]
The part boundary, headers and DDX. This will be built by the BES (likely in BESDapTransmit) using a method in DODSFilter that will build/send the part headers and then call a method in DDS to build/send the DDX.
--example-2 Content-Type: application/octet-stream Content-ID: <950118.AFDH@host.domain> Content-Transfer-Encoding: gzip Content-Description: dap-blob [XDR Encoded data]
Teh data blob will be written also using a method in DODSFilter that will build/send the part headers and then the data blob. As with the previous part, this DODSFilter method will be called by the BES.
The closing part boundary will be written by the OLFS.