BES Aggregation

From OPeNDAP Documentation
Revision as of 12:54, 9 March 2009 by PatrickWest (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

By default there are no aggregation engines installed with the BES. But there is a mechanism to register your own aggregation engine. This page will discuss how to go about doing that.

1 How does aggregation work in the BES?

A request comes into the BES that references some containers, creates a definition using those containers adding constraints and specifying an aggregation engine to use and an aggregation command, and then a get command to grab the containers, constrain them, and then aggregate. The command would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
    <setContainer name="c1" space="catalog">/data/nc/fnoc1.nc</setContainer>
    <setContainer name="c2" space="catalog">/data/nc/fnoc2.nc</setContainer>
    <setContainer name="c3" space="catalog">/data/nc/fnoc3.nc</setContainer>
    <setContainer name="c4" space="catalog">/data/nc/fnoc4.nc</setContainer>
    <define name="d">
	<container name="c1">
            <constraint>u</constraint>
        </container>
	<container name="c2">
            <constraint>u</constraint>
        </container>
	<container name="c3">
            <constraint>u</constraint>
        </container>
	<container name="c4">
            <constraint>u</constraint>
        </container>
        <aggregate handler="someHandler" cmd="someCommand" />
    </define>
    <get type="dods" definition="d" />
</request>

In order to the BES to perform the specified aggregation there must be an aggregation engine registered with the BES call someHandler. The command cmd is passed to the aggregation engine and is parsed by the engine. This command can be different from engine to engine, there are no requirements for the command.

2 Creating your Aggregation Handler

The first step is to create an aggregation class that inherits from BESAggregationServer. The minimum that must be defined in this new class is the following:

class MyAggregationServer : public BESAggregationServer
{
public:
                       MyAggregationServer( string handler_name )
                              : BESAggregationServer( handler_name ) {} ;
    virtual            ~MyAggregationServer() {} ;

    virtual void       aggregate( BESDataHandlerInterface &dhi ) ;

    static BESAggregationServer *AggBuilder( string handler_name );
};

The value of the handler_name parameter will be someHandler. When a request comes in to aggregate, the BES takes that name and looks up a function in the aggregation factory with that name (someHandler) and calls that function. That function will return a new instance of BESAggregationServer. The function called will be AggBuilder. The function AggBuilder would look something liket his:

BESAggregationServer *
MyAggregationServer::AggBuilder( string handler_name )
{
    return new MyAggregationServer( handler_name ) ;
}

The only real method that you have to implement is the aggregate method. More on that later.

3 Registering your Aggregation Handler

You register your new aggregation handler with the Aggregation Factory BESAggFactory giving it the name that you want it to be referenced by in the define command (someHandler). What you are actually registering with the factory is a function that knows how to instantiate the new aggregation server. In the example above, MyAggregationServer::AggBuilder function is registered with the BESAggFactory. This should be done in your Module class' initialize method.

For example:

    BESDEBUG( "mydebug", "    adding " << modname << " aggregation" << endl )
    BESAggFactory::TheFactory()->add_handler( modname, MyAggregationServer::AggBuilder ) ;

Typically, the name of the aggregation engine is the same as the name of your module. In this example, modname would be "someHandler".

4 Writing your aggregate method

The aggregate method is the only method that you need to implement. This is the method that gets called to aggregate the resulting DataDDS object. The DataDDS object is stored in the BESDataHandlerInterface instance passed to your method. Here is an example of getting the DataDDS out of the BESDataHandlerInterface.

void
NCESGAggregationServer::aggregate( BESDataHandlerInterface &dhi )
{
    if( dhi.action == "das" )
    {
        string err = "DAS is not a valid request type in aggregated datasets" ;
        throw BESInternalError( err, __FILE__, __LINE__ ) ;
    }

    BESResponseObject *resp = dhi.response_handler->get_response_object() ;
    BESDataDDSResponse *bdds = dynamic_cast<BESDataDDSResponse *>(resp) ;
    if( !bdds )
    {
        string err = "response object is not a DataDDS" ;
        throw BESInternalError( err, __FILE__, __LINE__ ) ;
    }
    DataDDS *dds = bdds->get_dds() ;
    if( !dds )
    {
        string err = "dap response object is not a DataDDS" ;
        throw BESInternalError( err, __FILE__, __LINE__ ) ;
    }

    ....
    your code here
    .....

Once you have the DataDDS you have all of the data that has been read in and you can perform your aggregation. The DataDDS will be organized in the following manner. For each of the containers defined in the BES request (in our example, c1, c2, c3, and c4) there will be a structure containing the data for that container. So, in our example, you would have:

Dataset {
    Structure {
        Int16 u[time_a = 16][lat = 17][lon = 21];
    } c1;
    Structure {
        Int16 u[time_a = 16][lat = 17][lon = 21];
    } c2;
    Structure {
        Int16 u[time_a = 16][lat = 17][lon = 21];
    } c3;
    Structure {
        Int16 u[time_a = 16][lat = 17][lon = 21];
    } c4;
} fnoc1.nc;

The result of your aggregation will be a new DataDDS object that will take the place of the one you got out. The end of your function might look something like this:

    ....
    your code here
    ...
    BESDataDDSResponse *my_bdata = new BESDataDDSResponse( my_data ) ;
    dhi.response_handler->set_response_object( my_bdata ) ;
    delete bdds ;
}

5 Summary

We recommend that your first step in writing your own aggregation server is to create the classes that you will need with an empty aggregate method, get it compiled and installed, load the module into the BES, and make sure that your aggregation method is being called. Once you have done this, then you can write your aggregate method.

Here's what you need to do:

  1. Create your class that inherits from BESAggregationServer using the code from above.
  2. Register your AggBuilder function with the BESAggFactory in your Module class.
  3. Build your new code, creating a shared object library (.so file) that can be loaded into the BES
  4. Add your module to the BES configuration file
  5. Run the BES with this new module
  6. Implement your aggregate method

For more information on the Module class and adding it to the BES configuration file, please refer to the Extending the BES page.