WCS Data Access
WCS data access via DAP2 requests
We have implemented two approaches to providing this access.
1. THREDDS DispatchHandler
Using this handler the olfs can read and serve very large THREDDS catalogs.
Mechanism: Reads and parses all local catalogs. Can cache:
- Only the file name (minimum memory footprint)
- raw (unparsed) bytes from catalog files resident in memory. (medium memory footprint)
- All catalog files parsed into memory resident JDOM Documents. (large memory footprint)
Responses are generated by either returning the THREDDS catalog as the source XML, or by transforming the catalog into an HTML document to be rendered in a a browser and viewed by humans.
WCS data can be served by carefully constructing the THREDDS service definitions and the dataset elements urlPath attributes to point to the wcs_service (described below).
This handler relies on XSLT to generate the human readable HTML for the catalog pages. The XSL code does not yet support the full THEDDS schema.
2. wcs_service
This handler provides DAP access to WCS servers. The WCS request URL is converted to a hex/ASCII representation and used as the data set name. For example, this WCS request:
http://geobrain.laits.gmu.edu/cgi-bin/ACCESS/wcs627?service=WCS&version=1.0.0&request=GetCoverage&coverage=/Volumes/RAIDL2/GeoData/CEOP/MYD05/LIN/MYD05_L2.A2002274.1240.005.2007073060134.hdf:Swath:mod05:Water_Vapor_Infrared&crs=WGS84&bbox=11.745000,49.825000,16.495000,54.575000&format=netCDF&TIME=2002-10-01&resx=0.25&resy=0.25&interpolationMethod=Nearest%20neighbor
Gets encoded like this:
687474703a2f2f67656f627261696e2e6c616974732e676d752e6564752f6367692d62696e2f4143434553532f7763733632373f736572766963653d5743532676657273696f6e3d312e302e3026726571756573743d476574436f76657261676526636f7665726167653d2f566f6c756d65732f524149444c322f47656f446174612f43454f502f4d594430352f4c494e2f4d594430355f4c322e41323030323237342e313234302e3030352e323030373037333036303133342e6864663a53776174683a6d6f6430353a57617465725f5661706f725f496e667261726564266372733d57475338342662626f783d31312e3734353030302c34392e3832353030302c31362e3439353030302c35342e35373530303026666f726d61743d6e65744344462654494d453d323030322d31302d303126726573783d302e323526726573793d302e323526696e746572706f6c6174696f6e4d6574686f643d4e6561726573742532306e65696768626f72
And used as a dataset name. Thus, this URL:
http://localhost:8080/opendap/wcs_service/687474703a2f2f67656f627261696e2e6c616974732e676d752e6564752f6367692d62696e2f4143434553532f7763733632373f736572766963653d5743532676657273696f6e3d312e302e3026726571756573743d476574436f76657261676526636f7665726167653d2f566f6c756d65732f524149444c322f47656f446174612f43454f502f4d594430352f4c494e2f4d594430355f4c322e41323030323237342e313234302e3030352e323030373037333036303133342e6864663a53776174683a6d6f6430353a57617465725f5661706f725f496e667261726564266372733d57475338342662626f783d31312e3734353030302c34392e3832353030302c31362e3439353030302c35342e35373530303026666f726d61743d6e65744344462654494d453d323030322d31302d303126726573783d302e323526726573793d302e323526696e746572706f6c6174696f6e4d6574686f643d4e6561726573742532306e65696768626f72.dds
Will get you the DDS for the dataset returned by the example WCS request.
Although the URL is "opaque" it is transparent within the THREDDS catalog. The urlPath attribute of the dataset element contains only the encoded WCS request URL. Additional properties (metadata) can be added to the THREDDS catalog, but will not show up in a DAS or DDX response from the service (cause that starts looking like AIS).
3. WCS DispatchHandler for the OLFS.
This was an attempt to build a generic client for a WCS service. The result was one that works for the CEOP AIRS WCS service (http://g0dup05u.ecs.nasa.gov/cgi-bin/ceopAIRX2RET) but in no way represents a generalized solution. It is explained fully here: WCS Handler [ We evaluated a number of other WCS servers and discovered a surprisingly broad range of interpretations of the specification.
I think the major obstacles are:
- Coordinate systems: My reading of the WCS specification made me think that the idea was for the servers to be doing re-projection work for the clients. This turns out to be not the common case. Some servers can re-project, others can re-project for some CoverageOfferings, and others not at all. Additionally, coordinate system conversion would we required to map "SIte" definitions to the various CoverageOfferings. This means that a generalized client for WCS must carry the software to do the re-projection work. Having the DAP server handle reprojection and coordinate conversion is (IMHO):
- Beyond the scope of this task
- Fraught with difficulty
- Scaleability: The WCS specification demands that each CoverageOffering be described by one (or both) of it's spatial or temporal extents. Problems arise when WCS servers hold satellite imagery which is bound by both space and time and then don't actually use temporal domain information in their CovergeOffering elements. They may have a time that represents when the data was acquired, but not a range. Essentially pushing the the job of finding coverage data for a specific time to the client. The result is that the WCS service returns a fairly large (>100MB) XML document in response to GetCapabilities request. This is unwieldy for the client and creates peformance issues. Holding the parsed documents in memory is a problem causes excessive (i.e. un-scaleable) memory use. Caching the documents on local disk and doesn't really help much as it:
- Requires the same memory usage: To get any piece of of the cached document it you have to parse the whole thing anyway.
- Could use MORE memory if multiple connections require the large document to parsed simultaneously
- And disk caching will greatly slow performance - reading and parsing the document for each request will be slow and could lead to disk thrashing.
It is my thinking at this time that we could potentially build a point and click map interface (similar to that used by LAS) for this. Pushing the acquisition and caching of the GetCapabilities document onto the client (browser?). It would be navigated by the user to build the WCS request URL which could then be encoded and sent to the wcs_service described above.
THREDDS catalogs do not work for this handler at this time.
NEEDS
thredds handler:
- Mechanism for reloading the THREDDS catalogs - (Check !RootCatalogs? Check every catalog?, Trigger?)
- Extend the thredds.xsl file to fully support the THREDDS schema.
- Compound services.
- Inherited metadata.
- Work w/Ethan and John.
- Add config parameter to allow storing the parsed XML document.
- Examine the XSL for catalogRef and make sure the xlink:href works correctly.
- Inspect and repair the way that catalogRef elements are evaluated during the servlet init(). Only look at local files (don't go off site)
- Look at what happens if absolute path names are used in naming catalog files (in the config's threddsCatalogRoot elements) or in the xlink:href of a catalogRef
wcs_service:
- Trusted Hosts list for wcs_service !DispatchHandler.
wcs_handler:
- Are there THREDDS catalogs for the WCS !DispatchHandler? No? Fix it...
