Hyrax - BES Configuration

From OPeNDAP Documentation
Revision as of 00:20, 22 March 2007 by Ndp (talk | contribs) (New page: Building the BES from and it's data handlers source or installing from an RPM will provide the default installtion with data and a valid configuration. This is suitable for testing. The fo...)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
⧼opendap2-jumptonavigation⧽

Building the BES from and it's data handlers source or installing from an RPM will provide the default installtion with data and a valid configuration. This is suitable for testing. The following details how you go about customizing it for your data center.


Administration & Logging

The BES.ServerAdministrator parameter is the address used in various mail messages returned to clients. Set this so that whoever gets the email will be able to fix problems ad/or respond to user questions. Also set the log file and log level. If the BES.LogName is set to a relative path it will be treated as relative to the directory where the BES is started. That is, if the BES is installed in /usr/local/bin but you start it in your home directory using the parameter value below, the log file will be bes.log in your home directory.

BES.ServerAdministrator=jgallagehr@opendap.org
BES.LogName=./bes.log
BES.LogVerbose=no

Because the BES is a server in its own right, you will need to tell it which network port and interface to use. Assuming you are running the BES and OLFS (i.e., all of Hyrax) on one machine, here's what you should do:

Setting the networking parameters

BES.ServerPort=10002
BES.ServerUnixSocket=/tmp/opendap.socket

The BES.ServerPort tells the BES which TCP/IP port to use when listening for commands. Unless you need to use a different port, use the default. Ports with numbers less than 1024 are special, otherwise you can use any number undler 65536. That said, stick with the default unless you know you need to change it.

Similarly, the default for BES.ServerUnixSocket is almost always what you want. This tells the BES which unix socket to listen on for commands.

The BES listens on both the TCP port and the Unix Socket. Local clients, on the same machine as the BES, can use the unix socket for a faster connection. Otherwise, clients on other machines will connect to the BES using the BES.ServerPort value.

Debugging Tip

Use the BES.ProcessManagerMethod parameter to control whether the BES acts like a normal Unix server or not. The default value of multiple causes the BES to accept many connections at once, like a typical server. The value single causes it to accept a single connection, process the commands sent to it and exit, greatly simplifying trouble shooting.

BES.ProcessManagerMethod=multiple

Controlling how compressed files are treated

When the BES is asked to serve a file that has been compressed, it first must decompress it and hten pass it to the correct data handler (except for those formats which support 'internal' compression such as HDF4). The BES.CacheDir parameter tells the BES where to store the uncompressed file. Note that the default value of /tmp is probably less safe than a directory that's only used by the BES for just this purpose. You might set this to <prefix>/var/bes/cache, for example.

The BES.CacheDir.MaxSize determines the size of the cache. When the size of the files exceeds this value (the units are megabytes), then the oldest file(s) are deleted until the size is less than MaxSize.

BES.Compressed.Extensions is a list (elements separated by semicolons) of regular expressions used to identify compressed files. if this parameter is not set the BES assumes that files ending in .gz, .Z or .bz2 are compressed.

The BES.Compressed.Script parameter points to a script that will be used to decompress the files. This script must match the different file extensions to the correct program. The default handles gzip, compress and bzip2.


BES.CacheDir=/tmp
BES.CacheDir.MaxSize=500
# BES.Compressed.Extensions=
BES.Compressed.Script=/usr/local/bin/bes-uncompress.sh

Loading Handlers

Virtually all of the BESs functions are contained in modules that are loaded when the server starts up. Each module is shared-object library. If you're curious about how these are built, take a look at any of the handler (e.g., the dap-server software has several handlers and those are used by almost every BES).

There are two steps to the configuration:

  1. List the module in the order you want it loaded using the BES.modules parameter
  2. Use the BES.module.module name parameter to tie the name using in BES.modules with a specific shared-object module file. Use absolute paths.

Here's an example:

BES.modules=dap,cmd,ascii,usage,www,nc,h4,ff
BES.module.dap=/usr/local/lib/bes/libdap_module.so
BES.module.cmd=/usr/local/lib/bes/libdap_cmd_module.so
BES.module.ascii=/usr/local/lib/bes/libascii_module.so
BES.module.usage=/usr/local/lib/bes/libusage_module.so
BES.module.www=/usr/local/lib/bes/libwww_module.so
BES.module.nc=/usr/local/lib/bes/libnc_module.so
BES.module.h4=/usr/local/lib/bes/libhdf4_module.so
BES.module.ff=/usr/local/lib/bes/libff_module.so

In this example configuration the BES loads the following modules:

dap
The DAP implementation
cmd
The BES command set
ascii
A handler to build the ASCII data response using the data response from any one of the data handlers
usage
A handler which retrieves metadata from a data handler and formats it as a text/plain response
www
A handler which retrieves metadata from a data handler and builds the HTML form interface for that data source
nc
The data handler for NetCDF3 files
h4
The data handler for HDF4 files
ff
The data handler for files which can be read using FreeForm

Parameters for Specific Handlers

Currently there's only one handler that has a configuration parameter all its own.

Use HDF4.CacheDir to set the directory where the HDF4 handler will cache DAP metadata objects. The default value of /tmp is OK, but you might want to change it to a directory specific to just this purpose, such as <prefix>/var/bes/hdf4.

HDF4.CacheDir=/tmp

Pointing to data

There are two parameters that can be used to tell the BES where your data are stored. Which one you use depends on whether you are setting up the BES to work as part of Hyrax (and thus with THREDDS catalogs) or as a standalone server. In either case set the value of the ..RootDirectory parameter to point to the root directory of your data files (only one may be specified). Use BES.Catalog.catalog.RootDirectory if the BES is being used as part of Hyrax and BES.Data.RootDirectory if not. So, if you are setting up Hyrax, set the value of BES.Catalog.catalog.RootDirectory but be sure to set BES.Data.RootDirectory to some value or the BES will not start. (This will be fixed soon jimg 23:12, 22 February 2007 (PST))

BES.Catalog.catalog.RootDirectory=/full/path/data/root/directory
BES.Data.RootDirectory=/full/path/data/root/directory

Next configure the mapping between data source names and data handlers. When the BES is asked to perform some commands on a particular data source, it uses regular expressions to figure out which data handler should be used to carry out the commands. The value of the BES.Catalog.catalog.TypeMatch parameter is holds the set of regular expressions. The value of this parameter is a list of handlers and expressions. Note that these regular expressions are like those used by grep on Unix and it's somewhat cryptic, but once you see the pattern, it's not that bad. Below the TypeMatch parameter is being told that any data source with a name that ends in .nc should be handled by the nc (netcdf) handler (see BES.module.nc above), any file with a .hdf, .HDF or .eos suffix should be processed using the HDF4 handler (note that case matters) and that data sources ending in .dat should use the FreeForm handler.

BES.Catalog.catalog.TypeMatch=nc:.*\.nc;h4:.*\.(hdf|HDF|eos);ff:.*\.dat;

If you fail to configure this correctly, the BES will return error messages stating that the type information has to be provided. However, it won't tell you this when it starts, only when the OLFS (or some other software) actually makes a data request. This is because it's possible to use BES commands in place of these regular expressions, although the Hyrax won't.

Finally, you can configure the types of information that the BES send back when a client requests catalog information. The Include and Exclude parameters provide this mechanism, also using a list of regular expressions (with each element of the list separated by a semicolon). In the example below, files that begin with a dot are excluded.

The Include expressions are applied to the node first, followed by the Exclude expressions. For collections of nodes, only the Exclude expressions are applied.

BES.Catalog.catalog.Include=;
BES.Catalog.catalog.Exclude=^\..*;