Making the transition to DAP4

This How-To describes modifying existing modules written for the Hyrax Back End Server (BES) so that they support both DAP4 as well as the older DAP2 protocol. The information here is mostly anecdotal, based on experience working with five existing handlers that read data and return DAP2 metadata and data responses. The libdap and bes software have been modified to provide support for all but a very few of the features described in the DAP4 draft specification document. The versions of those packages that include this support are accessible from our SVN repository on a special branch that exists for DAP4 development and testing.

In the following, I assume you're going to modify a module so that it will support DAP4, in addition to supporting DAP2. The basic steps are pretty simple:

• Get the dap4 Shrew branch
• Move your working copy of the module to a dap4 branch
• Modify the module
• Write tests
• Check in the result

How long does this take? On average it take about a day to two days for any given module.

Note: I'm using $svn as if it were a shell variable defined as https://scm.opendap.org/svn.

Get the DAP4 branch

The short version of this is to svn co $svn/branch/shrew/dap4. See Hyrax - Build the Shrew Project for more information on shrew (caveat: that's a fairly old article and unlikely to be completely accurate). Build the code, run the tests, make sure the libdap and bes libraries are installed, etc., so that you are comfortable that the code you're working with is functioning.

Move your module working copy to a dap4 branch

In the shrew project there is a a file names externals.txt that lists the external projects that are checked out to make up shrew. You will see that libdap, bes, and at least 5 of the 14 modules are on a branch. The externals.txt file will look something like this:

^/trunk/hyrax-dependencies/ src/dependencies

^/branch/libdap/dap4 src/libdap
^/branch/bes/dap4 src/bes

^/trunk/olfs/ src/olfs

^/branch/csv_handler/dap4 src/modules/csv_handler
...
^/branch/gdal_handler/dap4 src/modules/gdal_handler

^/trunk/hdf4_handler/ src/modules/hdf4_handler
^/trunk/hdf5_handler/ src/modules/hdf5_handler

^/branch/ncml_module/dap4 src/modules/ncml_module
^/trunk/gateway_module/ src/modules/gateway_module
...

I'm assuming you're working on a module that does not support DAP4 and so probably does not have a dap4 branch. Beware the old DAP4 branches in our repository; they are evil and should be avoided.

What you need to do to move your module to a dap4 branch in shrew/dap4:

1. Go to the top-level shrew directory: cd ~/src/dap4 (for example)
2. Make a branch called dap4 for your module: svn cp $svn/trunk/module $svn/branch/module/dap4 It'll ask for a comment, mumble about dap4...
3. Edit externals.txt so that the reference to the external module is not ^/trunk/module but instead ^/branch/module/dap4
4. Run the set-externals.sh script: ./set-externals.sh
5. Check in the changes to the top-level shrew directory: svn ci -N Again, comment about dap4 and your module in the comment...
6. Verify the new property value: svn pg svn:externals This should look very similar to the stuff int he externals.txt file and should show the branch is to be checked out into src/modules/module.
7. Go to the directory that holds your module: cd src/modules/module
8. Switch to the newly made branch: svn switch $svn/branch/module/dap4 . <-- Note the trailing dot; it means do the switch to the current directory
9. Verify that the switch worked: svn info The repository URL should point toward the branch.

OK, now we are done with SVN and revision control monkey shines for a while...

Modify the module

The most important thing to do when modifying a module so that it will support DAP4 is to make sure and run it's tests before you start hacking away at the code. Make sure all of the tests pass or that you know why the tests that don't pass are failing and that those failures are not that big a deal. I made all of the modules I worked on pass all their tests before I started modifying them - so far.

There are two kinds of modules we have in Hyrax that I'll cover here: Modules that read a specific kind of data and modules that build a new (non-DAP) kind of response. I'll call these format and transmitter modules, respectively.

Format module

There is one class every format module contains that will require modification: RequestHandler. You will need to add two new static (class) methods to RequestHandler so that it can return the 'dmr and dap (metadata and data, resp.) responses when the BES framework processes a request that contains <get type="dmr" .../> or <get type="dap" .../>. Here's what it looks like for the csv_handler software:

CSVRequestHandler::CSVRequestHandler(string name) : BESRequestHandler(name)
{
	add_handler(DAS_RESPONSE, CSVRequestHandler::csv_build_das);
	add_handler(DDS_RESPONSE, CSVRequestHandler::csv_build_dds);
	add_handler(DATA_RESPONSE, CSVRequestHandler::csv_build_data);

	// We can use the same DMR object for both the metadata and data
	// responses. jhrg 8/13/14
	add_handler(DMR_RESPONSE, CSVRequestHandler::csv_build_dmr);
	add_handler(DAP4DATA_RESPONSE, CSVRequestHandler::csv_build_dmr);

	add_handler(VERS_RESPONSE, CSVRequestHandler::csv_build_vers);
	add_handler(HELP_RESPONSE, CSVRequestHandler::csv_build_help);
}

And in the CSVRequestHandler.h header file:

...
	static bool csv_build_data(BESDataHandlerInterface &dhi);

	static bool csv_build_dmr(BESDataHandlerInterface &dhi);

	static bool csv_build_vers(BESDataHandlerInterface &dhi);
...

Before we describe how to write the new build_dmr() class method, note that while DAP2 used two different C++ objects for metadata and data, DAP4 has just one, the DMR, so for both the metadata ("dmr") and data ("dap") response, we can use the same class method. Of course, your handler might need to do something special when building a data response, so having a separate class method for that is an option. So far, csv, fits, freeform, netcdf, and gdal have not needed that so I don't have any examples of it yet.

Writing build_dmr()

There are two ways you can go about this task. The first, and likely more time consuming, is to modify your handler to build a DMR object and just return it. The second way is to use the existing capability in the module to build a DDS object that also contains DAP2 attributes - most handlers do this by building the DDS, then the DAS and then merging the two using the DDS::transfer_attributes(DDS *) method - and then transform that DDS into a DMR using one of the DMR object's methods. Your handler can either use DMR::DMR(D4BaseTypeFactory *factory, DDS &dds) or void DMR::build_using_dds(DDS &dds).

Here's code that implements build_dmr() suing the void DMR::build_using_dds(DDS &dds) method:

bool CSVRequestHandler::csv_build_dmr(BESDataHandlerInterface &dhi)
{
	// First step, build the 'full DDS'
	string data_path = dhi.container->access();

	BaseTypeFactory factory;
	DDS dds(&factory, name_path(data_path), "3.2");
	dds.filename(data_path);

	try {
		csv_read_descriptors(dds, data_path);

		DAS das;
		csv_read_attributes(das, data_path);
		Ancillary::read_ancillary_das(das, data_path);
		dds.transfer_attributes(&das);
	}
	catch (InternalErr &e) {
		throw BESDapError(e.get_error_message(), true, e.get_error_code(), __FILE__, __LINE__);
	}
	catch (Error &e) {
		throw BESDapError(e.get_error_message(), false, e.get_error_code(), __FILE__, __LINE__);
	}
	catch (...) {
		throw BESDapError("Caught unknown error build CSV DMR response", true, unknown_error, __FILE__, __LINE__);
	}

	// Second step, make a DMR using the DDS

	// Extract the DMR Response object - this holds the DMR used by the
	// other parts of the framework.
	BESResponseObject *response = dhi.response_handler->get_response_object();
	BESDMRResponse &bdmr = dynamic_cast<BESDMRResponse &>(*response);

	// Extract the DMR Response object - this holds the DMR used by the
	// other parts of the framework.
	DMR *dmr = bdmr.get_dmr();
	dmr->set_factory(new D4BaseTypeFactory);
	dmr->build_using_dds(dds);

	bdmr.set_dap4_constraint(dhi);
	bdmr.set_dap4_function(dhi);

	return true;
}

Line 3-8 build a new, temporary DDS. Lines 11-16 load metadata into the DDS using whatever code your module has to do that. This code is building instances of values that know about DAP2. See the above section on 'libdap support for DAP4 and the common issues you might face as well as discussion below about specific issues found with the five modules ported so far. Line 32-38 extract the DMR from the BESDMRResponse object. Line 39 does the magic transformation, copying the variables and attributes from the DDS to the DMR. Line 41-42 sets the DAP4 constraints so that code in the BES (bes/dap/BESDapTransmitter, etc.) will work correctly.

That's it. That was all that was needed to add support for DAP4 to the csv_handler code. But...