AMQP Support in Hyrax: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 17: Line 17:


One approach to adapting DAP to a messaging architecture has already been implemented in our [http://rsg.opendap.org/server-4/templates/soapAPI.html interface to Hyrax for SOAP] and this might be the best starting point for an adaptation of DAP/HTTP to DAP and AMQP. One difference, however, that is likely to play a role in DAP over AMQP that doesn't show up in the SOAP interface is that DAP4 is now much farther along than when that software was written. The feature of DAP4 most important to this project is that DAP4 over HTTP no longer relies on HTTP headers as the sole way to return certain information. Instead, all information about a response is contained in the body of the response and ''some'' information is also contained in HTTP response headers to simplify writing HTTP clients and/or working with DAP2 clients. So, for example, the information about the version of DAP used to build a particular response is now part of the response body (in the ''<Dataset>'' element) ''and'' in the HTTP response header ''XDAP''. This means that HTTP clients can figure out the version before the response document is parsed and other protocols (e.g., AMQP) can get it from the response itself.
One approach to adapting DAP to a messaging architecture has already been implemented in our [http://rsg.opendap.org/server-4/templates/soapAPI.html interface to Hyrax for SOAP] and this might be the best starting point for an adaptation of DAP/HTTP to DAP and AMQP. One difference, however, that is likely to play a role in DAP over AMQP that doesn't show up in the SOAP interface is that DAP4 is now much farther along than when that software was written. The feature of DAP4 most important to this project is that DAP4 over HTTP no longer relies on HTTP headers as the sole way to return certain information. Instead, all information about a response is contained in the body of the response and ''some'' information is also contained in HTTP response headers to simplify writing HTTP clients and/or working with DAP2 clients. So, for example, the information about the version of DAP used to build a particular response is now part of the response body (in the ''<Dataset>'' element) ''and'' in the HTTP response header ''XDAP''. This means that HTTP clients can figure out the version before the response document is parsed and other protocols (e.g., AMQP) can get it from the response itself.
== Use Cases ==
In order to move forward and define the most useful way to use DAP2 and/or DAP4 over AMQP, we need to make suer there's a clear understanding of how the server is supposed to interact with the AMQP broker and how Hyrax within the OOI system will be used.

Revision as of 23:59, 2 December 2009

Figure 1. Hyrax Architecture at a High-level

Support for AMQP in hyrax is best handled by adding a new front-end to the server that can act as an AMQP client, reading information from an AMQP queue. Hyrax has an overall architecture that already supports this. Figure one shows a high-level view of the Hyrax architecture. The BES is the part f Hyrax that builds the bodies of a DAP response. The front-end (the OLFS) contains a set of handlers which respond to requests made using HTTP. Based on the request, the OLFS sends commands over a stateful connection to the BES asking it to make the correct response. Generally, the OLFS will have to parse the request URL and pass information from that URL to the BES. Even though the OLFS is designed to support several different 'protocols' like DAP or THREDDS, it is capable of responding to HTTP only (it is a Java Servlet; see Server Dispatch Operations for information about the OLFS design, implementation and extension capabilities). Thus, it makes the most sense to build a new front-end dedicated to AMQP.

While the architecture chosen to add support for AMQP is very important, other considerations are also critical to the success of the overall effort to run DAP over AMQP. One is the mapping between different DAP versions to AMQP. Since DAP was designed with HTTP in mind, how DAP and AMQP can best be matched merits serious consideration. This will entail looking at the current DAP implementation along with the evolving DAP, version 4, specification and its implementation.

Proposed Architecture for AMQP Support in Hyrax

Hyrax with AMQP

Figure 2. shows how a AMQP module could be added to work with Hyrax. The diagram implies that an actual installation could support both DAP/HTTP and DAP/AMQP interactions using one BES. That might be true, or it might not, depending on how the connection pooling is handled by the OLFS and new AMQP front-end. The DAP/SOAP interface of the OLFS is really different than the other three interfaces shown in the diagram because the SOAP messaging software uses the request document body instead of the URL while the other three interfaces/protocols all use the URL and ignore the request document body. However, it's still part of the OLFS because Hyrax uses SOAP messaging over HTTP.

One option we should consider is supporting DAP over AMQP using SOAP.

Sharing one BES between two front-ends

Note: The issue here is that the OLFS maintains a connection pool so we will need to look at how the OLFS interacts with the BES master process (aka besdaemon) to make sure that the AMQP front-end can make connections and get new listeners. The beslistener process uses stateful IPC and a protocol that is effectively unique to Hyrax.

How Best to Combine AMQP and HTTP

One approach to adapting DAP to a messaging architecture has already been implemented in our interface to Hyrax for SOAP and this might be the best starting point for an adaptation of DAP/HTTP to DAP and AMQP. One difference, however, that is likely to play a role in DAP over AMQP that doesn't show up in the SOAP interface is that DAP4 is now much farther along than when that software was written. The feature of DAP4 most important to this project is that DAP4 over HTTP no longer relies on HTTP headers as the sole way to return certain information. Instead, all information about a response is contained in the body of the response and some information is also contained in HTTP response headers to simplify writing HTTP clients and/or working with DAP2 clients. So, for example, the information about the version of DAP used to build a particular response is now part of the response body (in the <Dataset> element) and in the HTTP response header XDAP. This means that HTTP clients can figure out the version before the response document is parsed and other protocols (e.g., AMQP) can get it from the response itself.

Use Cases

In order to move forward and define the most useful way to use DAP2 and/or DAP4 over AMQP, we need to make suer there's a clear understanding of how the server is supposed to interact with the AMQP broker and how Hyrax within the OOI system will be used.