Hyrax - BES Configuration - Revision history

Jimg: /* Setting the networking parameters */

2013-12-19T20:40:15Z

Setting the networking parameters

← Older revision		Revision as of 20:40, 19 December 2013
Line 54:		Line 54:
	In the bes.conf configuration file, we have settings for how the BES should listen for requests:		In the bes.conf configuration file, we have settings for how the BES should listen for requests:

	BES.ServerPort=~~10002~~		BES.ServerPort=10022
	# BES.ServerUnixSocket=/tmp/opendap.socket		# BES.ServerUnixSocket=/tmp/opendap.socket

Jimg: /* Controlling how compressed files are treated */

2012-04-09T23:18:22Z

Controlling how compressed files are treated

← Older revision		Revision as of 23:18, 9 April 2012
Line 84:		Line 84:
	times when the cache holds more data than this value. Ideally this value		times when the cache holds more data than this value. Ideally this value
	should be several times the size of the largest file you plan to serve.		should be several times the size of the largest file you plan to serve.

	<!-- REMOVED jhrg 4/9/12		<!-- REMOVED jhrg 4/9/12
	===Tuning the cache to your data===		===Tuning the cache to your data===

Jimg: /* Controlling how compressed files are treated */

2012-04-09T23:18:07Z

Controlling how compressed files are treated

← Older revision		Revision as of 23:18, 9 April 2012
Line 80:		Line 80:
	The ''BES.CacheSize'' parameter sets the size of the cache in megabytes. When the size of the cached		The ''BES.CacheSize'' parameter sets the size of the cache in megabytes. When the size of the cached
	files exceeds this value, the cache will be purged using a		files exceeds this value, the cache will be purged using a
	~~last~~-~~accessed~~-~~first-removed~~ approach. Because it's usually impossible to		least-recently-used (where the file's access time is the 'use time') approach. Because it's usually impossible to
	determine the sizes of data files before decompressing them, there may be		determine the sizes of data files before decompressing them, there may be
	times when the cache holds more data than this value. Ideally this value		times when the cache holds more data than this value. Ideally this value

Jimg: /* Controlling how compressed files are treated */

2012-04-09T23:15:35Z

Controlling how compressed files are treated

← Older revision		Revision as of 23:15, 9 April 2012
Line 70:		Line 70:

	==Controlling how compressed files are treated==		==Controlling how compressed files are treated==

	Compression parameters are configured in the bes.conf configuration file.		Compression parameters are configured in the bes.conf configuration file.

Line 86:		Line 85:
	should be several times the size of the largest file you plan to serve.		should be several times the size of the largest file you plan to serve.

	<!-- ===Tuning the cache to your data===		<!-- REMOVED jhrg 4/9/12
			===Tuning the cache to your data===

	The two parameters ''BES.Uncompress.Retry'' and ''BES.Uncompress.NumTries'' are used, along with ''BES.CacheSize'', to tune the performance of the cache. The BES cache directory is locked to prevent the different beslistener		The two parameters ''BES.Uncompress.Retry'' and ''BES.Uncompress.NumTries'' are used, along with ''BES.CacheSize'', to tune the performance of the cache. The BES cache directory is locked to prevent the different beslistener

Jimg: /* Tuning the cache to your data */

2012-04-09T23:14:53Z

Tuning the cache to your data

← Older revision		Revision as of 23:14, 9 April 2012
Line 86:		Line 86:
	should be several times the size of the largest file you plan to serve.		should be several times the size of the largest file you plan to serve.

	===Tuning the cache to your data===		<!-- ===Tuning the cache to your data===

	The two parameters ''BES.Uncompress.Retry'' and ''BES.Uncompress.NumTries'' are used, along with ''BES.CacheSize'', to tune the performance of the cache. The BES cache directory is locked to prevent the different beslistener		The two parameters ''BES.Uncompress.Retry'' and ''BES.Uncompress.NumTries'' are used, along with ''BES.CacheSize'', to tune the performance of the cache. The BES cache directory is locked to prevent the different beslistener
Line 105:		Line 105:
	BES.CacheSize=500		BES.CacheSize=500
	BES.Uncompress.Retry=2000		BES.Uncompress.Retry=2000
	BES.Uncompress.NumTries=10		BES.Uncompress.NumTries=10 -->

	==Loading Software Modules==		==Loading Software Modules==

Ndp at 17:05, 19 March 2012

2012-03-19T17:05:18Z

← Older revision		Revision as of 17:05, 19 March 2012
Line 1:		Line 1:
	Building the BES and it's data handlers from source, or installing from the Linux RPMs, will provide the default installation with data and a valid configuration. This is suitable for testing. The following details how you go about customizing it for your data.		Building the BES and it's data handlers from source, or installing from the Linux RPMs, will provide the default installation with data and a valid configuration. This is suitable for testing. The following details how you go about customizing it for your data.

	= Location of the BES Configuation File =		== Location of the BES Configuation File ==

	The BES configuration file is called ''bes.conf'' and can be found in ''$prefix/etc/bes/'' if you built the software from source or in ''/etc/bes/'' if you used our RPM packages. By default ''$prefix'' is ''/usr/local''.		The BES configuration file is called ''bes.conf'' and can be found in ''$prefix/etc/bes/'' if you built the software from source or in ''/etc/bes/'' if you used our RPM packages. By default ''$prefix'' is ''/usr/local''.

	= Basic format of parameters =		== Basic format of parameters ==

	Parameters set in the BES configuration file have the following format:		Parameters set in the BES configuration file have the following format:
Line 28:		Line 28:
	Note that regular expressions can be used in the Include parameter to match a set of files.		Note that regular expressions can be used in the Include parameter to match a set of files.

	=Administration & Logging=		==Administration & Logging==

	In the bes.conf configuration file, the ''BES.ServerAdministrator'' parameter is the address used in various mail messages returned to clients. Set this so that whoever gets the email will be able to fix problems ad/or respond to user questions. Also set the log file and log level. If the ''BES.LogName'' is set to a relative path it will be treated as relative to the directory where the BES is started. That is, if the BES is installed in /usr/local/bin but you start it in your home directory using the parameter value below, the log file will be <tt>bes.log</tt> in your home directory.		In the bes.conf configuration file, the ''BES.ServerAdministrator'' parameter is the address used in various mail messages returned to clients. Set this so that whoever gets the email will be able to fix problems ad/or respond to user questions. Also set the log file and log level. If the ''BES.LogName'' is set to a relative path it will be treated as relative to the directory where the BES is started. That is, if the BES is installed in /usr/local/bin but you start it in your home directory using the parameter value below, the log file will be <tt>bes.log</tt> in your home directory.
Line 38:		Line 38:
	Because the BES is a server in its own right, you will need to tell it which network port and interface to use. Assuming you are running the BES and OLFS (i.e., all of Hyrax) on one machine, here's what you should do:		Because the BES is a server in its own right, you will need to tell it which network port and interface to use. Assuming you are running the BES and OLFS (i.e., all of Hyrax) on one machine, here's what you should do:

	=User and Group parameters=		==User and Group parameters==

	In the bes.conf configuration file, the BES must be started as root. One of the things that the BES does first is to start up a listener to listen for requests to the BES. This listener is started as root, but then the user and group of the process is set using parameters in the BES configuration file.		In the bes.conf configuration file, the BES must be started as root. One of the things that the BES does first is to start up a listener to listen for requests to the BES. This listener is started as root, but then the user and group of the process is set using parameters in the BES configuration file.
Line 50:		Line 50:
	BES.Group=#14		BES.Group=#14

	=Setting the networking parameters=		==Setting the networking parameters==

	In the bes.conf configuration file, we have settings for how the BES should listen for requests:		In the bes.conf configuration file, we have settings for how the BES should listen for requests:
Line 63:		Line 63:
	If both ''ServerPort'' and ''ServerUnixSocket'' are defined, the BES listens on both the TCP port and the Unix Socket. Local clients, on the same machine as the BES, can use the unix socket for a faster connection. Otherwise, clients on other machines will connect to the BES using the ''BES.ServerPort'' value. Note that the OLFS always uses only the TCP socket, even if the UNIX socket is present.		If both ''ServerPort'' and ''ServerUnixSocket'' are defined, the BES listens on both the TCP port and the Unix Socket. Local clients, on the same machine as the BES, can use the unix socket for a faster connection. Otherwise, clients on other machines will connect to the BES using the ''BES.ServerPort'' value. Note that the OLFS always uses only the TCP socket, even if the UNIX socket is present.

	=Debugging Tip=		==Debugging Tip==

	In the bes.conf configuration file, use the ''BES.ProcessManagerMethod'' parameter to control whether the BES acts like a normal Unix server or not. The default value of <tt>multiple</tt> causes the BES to accept many connections at once, like a typical server. The value <tt>single</tt> causes it to accept a single connection, process the commands sent to it and exit, greatly simplifying trouble shooting.		In the bes.conf configuration file, use the ''BES.ProcessManagerMethod'' parameter to control whether the BES acts like a normal Unix server or not. The default value of <tt>multiple</tt> causes the BES to accept many connections at once, like a typical server. The value <tt>single</tt> causes it to accept a single connection, process the commands sent to it and exit, greatly simplifying trouble shooting.
Line 69:		Line 69:
	BES.ProcessManagerMethod=multiple		BES.ProcessManagerMethod=multiple

	=Controlling how compressed files are treated=		==Controlling how compressed files are treated==

	Compression parameters are configured in the bes.conf configuration file.		Compression parameters are configured in the bes.conf configuration file.
Line 107:		Line 107:
	BES.Uncompress.NumTries=10		BES.Uncompress.NumTries=10

	=Loading Modules=		==Loading Software Modules==

	Virtually all of the BESs functions are contained in modules that are loaded when the server starts up. Each module is a shared-object library. The configuration for each of these modules is contained in its own configuration file and is stored in a directory located in the same directory as the bes.conf file called modules. ''$prefix/etc/bes/modules/''		Virtually all of the BESs functions are contained in modules that are loaded when the server starts up. Each module is a shared-object library. The configuration for each of these modules is contained in its own configuration file and is stored in a directory located in the same directory as the bes.conf file called modules. ''$prefix/etc/bes/modules/''

Ndp: /* Pointing to data */

2012-03-19T17:04:01Z

Pointing to data

← Older revision		Revision as of 17:04, 19 March 2012
Line 126:		Line 126:
	Module specific parameters will be included in its own configuration file. For example, any parameters specific to the netcdf data handler would be included in the nc.conf file.		Module specific parameters will be included in its own configuration file. For example, any parameters specific to the netcdf data handler would be included in the nc.conf file.

	=Pointing to data=		==Pointing to data==

	There are two parameters that can be used to tell the BES where your data are stored. Which one you use depends on whether you are setting up the BES to work as part of Hyrax (and thus with THREDDS catalogs) or as a standalone server. In either case set the value of the ''.RootDirectory'' parameter to point to the root directory of your data files (only one may be specified). Use ''BES.Catalog.catalog.RootDirectory'' in the dap.conf configuration file in the modules directory if the BES is being used as part of Hyrax, and ''BES.Data.RootDirectory'' in bes.conf itself if not. So, if you are setting up Hyrax, set the value of ''BES.Catalog.catalog.RootDirectory'' but be '''sure''' to set ''BES.Data.RootDirectory'' to some value or the BES will not start.		There are two parameters that can be used to tell the BES where your data are stored. Which one you use depends on whether you are setting up the BES to work as part of Hyrax (and thus with THREDDS catalogs) or as a standalone server. In either case set the value of the ''.RootDirectory'' parameter to point to the root directory of your data files (only one may be specified). Use ''BES.Catalog.catalog.RootDirectory'' in the dap.conf configuration file in the modules directory if the BES is being used as part of Hyrax, and ''BES.Data.RootDirectory'' in bes.conf itself if not. So, if you are setting up Hyrax, set the value of ''BES.Catalog.catalog.RootDirectory'' but be '''sure''' to set ''BES.Data.RootDirectory'' to some value or the BES will not start.
Line 157:		Line 157:

	If you fail to configure this correctly, the BES will return error messages stating that the type information has to be provided. However, it won't tell you this when it starts, only when the OLFS (or some other software) actually makes a data request. This is because it's possible to use BES commands in place of these regular expressions, although the Hyrax won't.		If you fail to configure this correctly, the BES will return error messages stating that the type information has to be provided. However, it won't tell you this when it starts, only when the OLFS (or some other software) actually makes a data request. This is because it's possible to use BES commands in place of these regular expressions, although the Hyrax won't.

			== Including and Excluding files and directories ==

	Finally, you can configure the types of information that the BES sends back when a client requests catalog information. The ''Include'' and ''Exclude'' parameters provide this mechanism, also using a list of regular expressions (with each element of the list separated by a semicolon). In the example below, files that begin with a dot are excluded. These parameters are set in the dap.conf configuration file.		Finally, you can configure the types of information that the BES sends back when a client requests catalog information. The ''Include'' and ''Exclude'' parameters provide this mechanism, also using a list of regular expressions (with each element of the list separated by a semicolon). In the example below, files that begin with a dot are excluded. These parameters are set in the dap.conf configuration file.
Line 165:		Line 167:
	BES.Catalog.catalog.Exclude=^\..*;		BES.Catalog.catalog.Exclude=^\..*;

			== Symbolic Links ==
	If you would like for symbolic links to be followed when retrieving data and for viewing catalog entries, then you need to set the following two parameters. The ''BES.FollowSymLinks'' parameter is for non-catalog containers and is used in conjunction with the ''BES.RootDirectory'' parameter above. It is NOT a general setting. The ''BES.Catalog.catalog.FollowSymLinks'' is for catalog requests and data containers in the catalog and is used in conjunction with the ''BES.Catalog.catalog.RootDirectory'' parameter above. The default is set to No in the installed configuration file. To allow for symbolic links to be followed you need to set this to Yes.		If you would like for symbolic links to be followed when retrieving data and for viewing catalog entries, then you need to set the following two parameters. The ''BES.FollowSymLinks'' parameter is for non-catalog containers and is used in conjunction with the ''BES.RootDirectory'' parameter above. It is NOT a general setting. The ''BES.Catalog.catalog.FollowSymLinks'' is for catalog requests and data containers in the catalog and is used in conjunction with the ''BES.Catalog.catalog.RootDirectory'' parameter above. The default is set to No in the installed configuration file. To allow for symbolic links to be followed you need to set this to Yes.

Jimg: /* Pointing to data */

2011-01-14T21:54:18Z

Pointing to data

← Older revision		Revision as of 21:54, 14 January 2011
Line 134:		Line 134:
	BES.Data.RootDirectory=/full/path/data/root/directory		BES.Data.RootDirectory=/full/path/data/root/directory

	~~And~~ in ~~dap~~.conf set the following if using Hyrax (usually the case)		Also in bes.conf set the following if using Hyrax (usually the case)

	BES.Catalog.catalog.RootDirectory=/full/path/data/root/directory		BES.Catalog.catalog.RootDirectory=/full/path/data/root/directory

PatrickWest at 17:41, 10 May 2010

2010-05-10T17:41:41Z

← Older revision		Revision as of 17:41, 10 May 2010
Line 178:		Line 178:

	Parameters for specific modules can be added to the BES configuration file for that specific module. No module-specific parameters should be added to bes.conf.		Parameters for specific modules can be added to the BES configuration file for that specific module. No module-specific parameters should be added to bes.conf.

			=Sample Installation and Configuration=
			[[Hyrax - Sample BES Installations\|Sample Installations Page]] shows how to download, build, install and configure for some sample installations.

PatrickWest: /* Loading Modules */

2010-04-25T15:53:41Z

Loading Modules

← Older revision		Revision as of 15:53, 25 April 2010
Line 110:		Line 110:

	Virtually all of the BESs functions are contained in modules that are loaded when the server starts up. Each module is a shared-object library. The configuration for each of these modules is contained in its own configuration file and is stored in a directory located in the same directory as the bes.conf file called modules. ''$prefix/etc/bes/modules/''		Virtually all of the BESs functions are contained in modules that are loaded when the server starts up. Each module is a shared-object library. The configuration for each of these modules is contained in its own configuration file and is stored in a directory located in the same directory as the bes.conf file called modules. ''$prefix/etc/bes/modules/''

			By default, all .conf files located in the modules are loaded by the BES per this parameter in the bes.conf configuration file:

			BES.Include=modules/.*\.conf$

			So if you don't want one of the modules to be loaded, simply change it's name to, say, nc.conf.sav and it won't be loaded.

	For example, if you are installing the general purpose server module (the dap-server module) then a dap-server.conf file will be installed in the modules directory. Also, most installations will include the dap module, allowing the BES to server OPeNDAP data. This configuration file is also included in the modules directory and is called dap.conf. For a data handler, say netcdf, there will be an nc.conf file located in the modules directory.		For example, if you are installing the general purpose server module (the dap-server module) then a dap-server.conf file will be installed in the modules directory. Also, most installations will include the dap module, allowing the BES to server OPeNDAP data. This configuration file is also included in the modules directory and is called dap.conf. For a data handler, say netcdf, there will be an nc.conf file located in the modules directory.