Wiki Testing/OPeNDAPUserGuideA: Difference between revisions
(22 intermediate revisions by 2 users not shown) | |||
Line 8: | Line 8: | ||
Full information about the latest versions of the OPeNDAP software is | Full information about the latest versions of the OPeNDAP software is | ||
available from the OPeNDAP web site: [http://www.opendap.org | available from the OPeNDAP web site: [http://www.opendap.org OPeNDAP Home Page]. | ||
This version of OPeNDAP uses CGI standard 1.1.1. | This version of OPeNDAP uses CGI standard 1.1.1. | ||
Line 18: | Line 18: | ||
====The OPeNDAP Web Site==== | ====The OPeNDAP Web Site==== | ||
We recommend that you use the [http://www.opendap.org | We recommend that you use the [http://www.opendap.org OPeNDAP Home Page] to obtain the most current version of the software. The "Software" page includes a form for | ||
selecting OPeNDAP components. Completing the form automatically creates | selecting OPeNDAP components. Completing the form automatically creates | ||
a custom compressed archive of all the software components you | a custom compressed archive of all the software components you | ||
Line 33: | Line 33: | ||
====By Anonymous FTP==== | ====By Anonymous FTP==== | ||
The OPeNDAP software may be down-loaded by anonymous <font color='green'>ftp</font> from the [ http://opendap.org/pub/dods | The OPeNDAP software may be down-loaded by anonymous <font color='green'>ftp</font> from the [http://opendap.org/pub/dods <cite>OPeNDAP ftp site</cite>]. | ||
Don't forget to down-load the core software <font color='green'>tar</font> file as well as the <font color='green'>tar</font> file corresponding to whatever data access API you need. | Don't forget to down-load the core software <font color='green'>tar</font> file as well as the <font color='green'>tar</font> file corresponding to whatever data access API you need. | ||
Line 91: | Line 91: | ||
If you downloaded binary files, you can skip the "building" steps. | If you downloaded binary files, you can skip the "building" steps. | ||
Line 114: | Line 114: | ||
Several pieces of software are required to run OPeNDAP. These are all | Several pieces of software are required to run OPeNDAP. These are all | ||
free software, and are descibed in | free software, and are descibed in [[Wiki_Testing/OPeNDAPUserGuideA| Appendix]]. | ||
===Installing the OPeNDAP Libraries=== | ===Installing the OPeNDAP Libraries=== | ||
In order to link and run client programs, the only OPeNDAP software that | In order to link and run client programs, the only OPeNDAP software that | ||
must be installed are the OPeNDAP libraries. The following libraries, | must be installed are the OPeNDAP libraries. The following libraries, | ||
described in ([http://www <cite> opd-client,link</cite>]) must be installed in the | described in ([http://www <cite> opd-client,link</cite>]) must be installed in the | ||
<font color='green'>/usr/lib</font> directory, or somewhere else where the linker will find | <font color='green'>/usr/lib</font> directory, or somewhere else where the linker will find them. Simply copy them into that directory from the OPeNDAP distribution. | ||
them. Simply copy them into that directory from the OPeNDAP distribution. | |||
*<font color='green'>libdap++.a</font> | *<font color='green'>libdap++.a</font> | ||
*The OPeNDAP version of the API your software uses. | *The OPeNDAP version of the API your software uses. | ||
==The OPeNDAP Client Initialization File (<font color='green'>.dodsrc</font>)== | ==The OPeNDAP Client Initialization File (<font color='green'>.dodsrc</font>)== | ||
Line 164: | Line 130: | ||
The following section refers only to OPeNDAP clients from release 3.2 and | The following section refers only to OPeNDAP clients from release 3.2 and | ||
after. | after. | ||
When an OPeNDAP client starts up, it checks to | When an OPeNDAP client starts up, it checks to | ||
Line 172: | Line 136: | ||
servers, and other http issues. If this file is not found, one is | servers, and other http issues. If this file is not found, one is | ||
created in the user's home directory, using default | created in the user's home directory, using default | ||
values. | values.[[Wiki_Testing/OPeNDAPUserGuideFootnotes|(24)]] | ||
The client initialization file is usually called <font color='green'>.dodsrc</font>, and is usually located in the user's home directory. You can change this by creating an environment variable called <font color='green'>DODS_CONF</font> and setting it to the full pathname of the configuration file. | |||
The client initialization file is usually called <font color='green'>.dodsrc</font>, and is | |||
usually located in the user's home directory. You can change this by | |||
creating an environment variable called <font color='green'> | |||
setting it to the full pathname of the configuration file | |||
Here is a sample configuration file: | Here is a sample configuration file: | ||
<pre> | |||
# OPeNDAP client configuration file. See the OPeNDAP | |||
# users guide for information. | |||
# | USE_CACHE=0 | ||
USE_CACHE= | # Cache and object size are given in megabytes (20 ==> 20Mb). | ||
MAX_CACHE_SIZE=20 | MAX_CACHE_SIZE=20 | ||
MAX_CACHED_OBJ=5 | MAX_CACHED_OBJ=5 | ||
IGNORE_EXPIRES=0 | IGNORE_EXPIRES=0 | ||
CACHE_ROOT=/home/ | CACHE_ROOT=/home/jimg/.dods_cache/ | ||
DEFAULT_EXPIRES=86400 | DEFAULT_EXPIRES=86400 | ||
ALWAYS_VALIDATE=0 | ALWAYS_VALIDATE=0 | ||
# Request servers compress responses if possible? | |||
# 1 (yes) or 0 (false). | |||
DEFLATE=0 | |||
# Should SSL certificates and hosts be validated? SSL | |||
# will only work with signed certificates. | |||
VALIDATE_SSL=1 | |||
# Proxy configuration (optional parts in []s): | |||
# You may also use the 'http_proxy' environment variable. | |||
# A value in this file will override that env variable. | |||
# PROXY_SERVER=<http://[username:password@]host[:port]> | |||
# NO_PROXY_FOR=<host|domain> | |||
PROXY_SERVER=http://username:password@squid.proxy.org:3128/ | |||
NO_PROXY_FOR=localhost | |||
# AIS_DATABASE=<file or url> | |||
</pre> | </pre> | ||
====Comments==== | ====Comments==== | ||
Starting a line with a <font color='green'> | Starting a line with a <font color='green'>#</font> makes that line a comment. | ||
====Caching==== | ====Caching==== | ||
The first seven parameters control caching. The OPeNDAP client | |||
The parameters | |||
can store data you've requested on your local computer. If you repeat | can store data you've requested on your local computer. If you repeat | ||
a request, the data can be retrieved from this local cache, saving the | a request, the data can be retrieved from this local cache, saving the | ||
Line 218: | Line 187: | ||
<blockquote> | <blockquote> | ||
; <font color='green'> | ; <font color='green'>USE_CACHE</font> : 1 turns on caching and 0 turns it off. | ||
; <font color='green'>MAX_CACHE_SIZE</font> : The value of <font color='green'>MAX_CACHE_SIZE</font> sets the maximum size of the cache in megabytes. Once the cache reaches this size, caching more objects will cause cache garbage collection. OPeNDAP will first purge the cache of any stale entries and then remove remaining entries starting with those that have the lowest hit count. Garbage collection stops when 90% purged. | |||
; <font color='green'>MAX_CACHED_OBJ</font> : This parameter sets the maximum size of any individual object in the cache in megabytes. Objects received from a server larger than this value will not be cached even if there's room for them without purging other objects. | |||
; <font color='green'>MAX_CACHE_SIZE</font> : The value of <font color='green'>MAX_CACHE_SIZE</font> sets | |||
the maximum size of the cache in megabytes. Once the cache reaches | |||
this size, caching more objects will cause cache garbage collection. | |||
OPeNDAP will first purge the cache of any stale entries and then remove | |||
remaining entries starting with those that have the lowest hit | |||
count. Garbage collection stops when 90 | |||
purged. | |||
; <font color='green'>MAX_CACHED_OBJ</font> : This parameter sets the maximum size of | |||
any individual object in the cache in megabytes. Objects received | |||
from a server larger than this value will not be cached even if | |||
there's room for them without purging other objects. | |||
</blockquote> | </blockquote> | ||
Many web documents, and OPeNDAP data, are delivered with an expiration | Many web documents, and OPeNDAP data, are delivered with an expiration | ||
Line 258: | Line 199: | ||
<blockquote> | <blockquote> | ||
; <font color='green'>IGNORE_EXPIRES</font> : If the value of this parameter is 1 | ; <font color='green'>IGNORE_EXPIRES</font> : If the value of this parameter is 1 (one), then expiration dates will be ignored, and the caching behavior will be ruled by the <font color='green'>DEFAULT_EXPIRES</font> parameter. | ||
; <font color='green'>CACHE_ROOT</font> : This parameter contains the pathname to the cache's top directory. If two or more users want to share a cache, then they must both have read and write permissions to the cache root. NB: Do not share caches unless you're using libdap version 3.8.1 or newer. | |||
(one), then expiration dates will be ignored, and the caching | ; <font color='green'>DEFAULT_EXPIRES</font> : Any data received without an expiration time will expire in the number of seconds given by this parameter. If <font color='green'>IGNORE_EXPIRES</font> is zero, this will apply to all data, whether or not it comes with an expiration date. The value is given in seconds. The configuration in the example is set for 24 hours. | ||
; <font color='green'>ALWAYS_VALIDATE</font> : Entries in a cache might be stale even though the default expiration time has not elapsed and some servers might not provide expiration dates for all of their responses. Setting this to '1' forces all cache entries to be validated before the client will use them. This makes for slower response times than just assuming the cached data is valid, but it is faster than fetching new data for every request. | |||
behavior will be ruled by the <font color='green'>DEFAULT_EXPIRES</font> parameter. | </blockquote> | ||
; <font color='green'>DEFAULT_EXPIRES</font> : Any data received without an expiration | |||
time will expire in the number of seconds given by this parameter. | |||
If <font color='green'>IGNORE_EXPIRES</font> is zero, this will apply to all data, | |||
====Proxy Servers==== | ====Proxy Servers==== | ||
An OPeNDAP client can negotiate proxy servers, with help from directions | An OPeNDAP client can negotiate proxy servers, with help from directions | ||
Line 284: | Line 213: | ||
<blockquote> | <blockquote> | ||
; <font color='green'>PROXY_SERVER</font> : This identifies a proxy server to use for | ; <font color='green'>PROXY_SERVER</font> : This identifies a proxy server to use for all OPeNDAP requests, except for requests specifically modified by the other two proxy behavior directives. The format is: | ||
all OPeNDAP requests, except for requests specifically modified by the | |||
other two proxy behavior directives. | |||
<pre> | <pre> | ||
PROXY_SERVER=http://[username:password@]<host>[:port] | |||
</pre> | </pre> | ||
The username, password and port are all optional elements. If no username and password is given the client will assume that none is needed. If the port is not given, the default value of 80 will be used. This syntax is new for clients using libdap 3.8.1 and newer; the old syntax is still supported but was error-prone and should not be used in new <font color='green'>.dodsrc</font> files. | |||
; <font color='green'>NO_PROXY_FOR</font> : Use this parameter to say that access to a certain host should never go through a proxy. | |||
; <font color='green'> | |||
</blockquote> | </blockquote> | ||
====Compression==== | ====Compression==== | ||
Many OPeNDAP servers support compression of the returned data. This can | Many OPeNDAP servers support compression of the returned data. This can save network bandwidth, and transmission time. You can tell your client to ask the server for compressed data by including a line in your <font color='green'>.dodsrc</font> file like this: | ||
save network bandwidth, and transmission time. You can tell your | |||
client to ask the server for compressed data by including a line in | |||
your <font color='green'>.dodsrc</font> file like this: | |||
<pre> | <pre> | ||
Line 408: | Line 229: | ||
</pre> | </pre> | ||
Using a value of zero will make the client request only uncompressed | Using a value of zero will make the client request only uncompressed data. If the directive is omitted, the default value is zero. | ||
data. | |||
====SSL Validation==== | |||
SSL Validation can be turned off using the <font color='green'>VALIDATE_SSL</font> parameter. if set to '1' all SLL hosts and certificates will be validated. The certificates must be signed for this to work. The default is on. Set to '0' to turn off. | |||
=Software you will need for OPeNDAP's Software= | |||
Most modern computers has all the basics in place. However, in some rare cases you might be stuck without some tools that you'll need to unpack our software. If that's the case here's what you'll need: | |||
<blockquote> | |||
; gzip : This is the GNU compression and de-compression program. You will need to install it before you can unpack any of the other software described here. | |||
; tar : This is used to bundle many files together into a single package and it is how we (and many others) package source code. | |||
</blockquote> | </blockquote> | ||
Most of the software you need for OPeNDAP is avaliable from the GNU | Most of the software you need for OPeNDAP is avaliable from the GNU | ||
archives. Refer to | archives. Refer to [http://www.gnu.org <cite>http://www.gnu.org</cite>] for instructions. Look at | ||
[http://www.gnu.org/order/ftp.html <cite>http://www.gnu.org/order/ftp.html</cite>]for a list of mirrors of that archive. Use the mirror closest to you, the transmission will be faster. | |||
to you, the transmission will be faster | |||
Follow the instructions to install each of the following software packages. Typically, you would install a package called <font color='green'>foo</font> as follows: | |||
Follow the instructions to install each of the following software | |||
packages. Typically, you would install a package called <font color='green'>foo</font> as | |||
follows: | |||
<pre> | <pre> | ||
Line 470: | Line 261: | ||
for each software package should be followed carefully. | for each software package should be followed carefully. | ||
==Building OPeNDAP's Software Using the Source Code== | |||
If you need to build the OPeNDAP software, or link it to existing libraries, you will need the following GNU software. Refer to [http://www.gnu.org <cite>http://www.gnu.org</cite>] forinstructions. Look at [http://www.gnu.org/order/ftp.html <cite>http://www.gnu.org/order/ftp.html </cite>]for a list of mirrors of that archive. Use the mirror closest to you, the transmission will be faster. | |||
If you | |||
the | |||
<blockquote> | <blockquote> | ||
; | ; GNU C/C++ Compiler : OPeNDAP needs <font color='green'>g++</font>, the GNU C++ compiler to compile. | ||
to compile. | |||
; binutils : The GNU linker is part of this package. | ; binutils : The GNU linker is part of this package. | ||
; libstdc++ : The standard | ; libstdc++ : The standard C++ library. Generally part of the C++ compiler so you rarely ever need to get this on its own. | ||
; GNU Make : GNU Make is not essential, but will make like easier. | ; GNU Make : GNU Make is not essential, but will make like easier. | ||
; flex : The GNU lexical-analyzer generator | ; flex : The GNU lexical-analyzer generator | ||
; bison : The GNU parser generator. | ; bison : The GNU parser generator. | ||
; CppUnit : Optional; You'll need this to run the unit tests included with our software | |||
; DejaGNU : Optional; Needed to run the regression tests included with our software | |||
; Doxygen : Optional; Needed to build the documentation | |||
</blockquote> | </blockquote> |
Latest revision as of 22:51, 21 April 2008
Installing the OPeNDAP Software
The current version of the Distributed Oceanographic Data System core software is 3.2. Note that this number applies only to the core software. The individual servers, clients, and client libraries have their own version numbers.
Full information about the latest versions of the OPeNDAP software is available from the OPeNDAP web site: OPeNDAP Home Page.
This version of OPeNDAP uses CGI standard 1.1.1.
Acquiring the OPeNDAP Software
The OPeNDAP Web Site
We recommend that you use the OPeNDAP Home Page to obtain the most current version of the software. The "Software" page includes a form for selecting OPeNDAP components. Completing the form automatically creates a custom compressed archive of all the software components you selected, which you can then download to your own machine.
Whenever possible, you should use the provided binary software rather than trying to compile and link OPeNDAP yourself. Compiling will work, but the OPeNDAP software is large, and it takes a long time to compile.
Don't forget to select the core software as well as the specific data access API or server you need. For an OPeNDAP client, you will also need Tcl/Tk and gzip.
By Anonymous FTP
The OPeNDAP software may be down-loaded by anonymous ftp from the OPeNDAP ftp site.
Don't forget to down-load the core software tar file as well as the tar file corresponding to whatever data access API you need.
The OPeNDAP project provides a small number of archive files containing
linked libraries and executables for some computing platforms. You
should use these files whenever possible, to avoid the hassle of
compiling the software yourself. The OPeNDAP software is a substantial
chunk of code, and it takes a long time to compile and link.
Installing the Software
To install the OPeNDAP software, choose a directory to be the OPeNDAP root directory. This directory must be identified with the DODS_ROOT environment variable for the OPeNDAP software to run.
First, set the working directory to the DODS_ROOT directory.For example,
export DODS_ROOT=/usr/local/DODS cd $DODS_ROOT
Next, expand the archives with gzip and unpack the expanded
files with tar:
gzip -d DODS-88772.tar.gz tar -xvf DODS-88772.tar
If you got the files via anonymous ftp, you would have to repeat the process for each down-loaded archive file, specifying the name of each component file. For example:
gzip -d DODS-core-2.19.tar.gz tar -xvf DODS-core-2.19.tar
Building the OPeNDAP Core
Unpacking the core software archive will create a configure script in the DODS_ROOT directory. The core software may then be built with:
cd $DODS_ROOT ./configure make
If you downloaded binary files, you can skip the "building" steps.
Building an OPeNDAP Client or Server
Unpacking the client library archives in the DODS_ROOT directory will produce directories such as jg-dods for the JGOFS software and nc-dods for the netCDF software. These will appear in the src subdirectory under DODS_ROOT. Each of these directories will also be equipped with a configure script to create a makefile, so the build procedure for them is the same as for the core software, for example:
cd $DODS_ROOT/src/nc-dods ./configure make
Getting the Software Used by DODS
Several pieces of software are required to run OPeNDAP. These are all free software, and are descibed in Appendix.
Installing the OPeNDAP Libraries
In order to link and run client programs, the only OPeNDAP software that must be installed are the OPeNDAP libraries. The following libraries, described in ( opd-client,link) must be installed in the /usr/lib directory, or somewhere else where the linker will find them. Simply copy them into that directory from the OPeNDAP distribution.
- libdap++.a
- The OPeNDAP version of the API your software uses.
The OPeNDAP Client Initialization File (.dodsrc)
The following section refers only to OPeNDAP clients from release 3.2 and after.
When an OPeNDAP client starts up, it checks to see whether the user has an initialization file available to control the setting of a number of parameters having to do with caching, proxy servers, and other http issues. If this file is not found, one is created in the user's home directory, using default values.(24)
The client initialization file is usually called .dodsrc, and is usually located in the user's home directory. You can change this by creating an environment variable called DODS_CONF and setting it to the full pathname of the configuration file.
Here is a sample configuration file:
# OPeNDAP client configuration file. See the OPeNDAP # users guide for information. USE_CACHE=0 # Cache and object size are given in megabytes (20 ==> 20Mb). MAX_CACHE_SIZE=20 MAX_CACHED_OBJ=5 IGNORE_EXPIRES=0 CACHE_ROOT=/home/jimg/.dods_cache/ DEFAULT_EXPIRES=86400 ALWAYS_VALIDATE=0 # Request servers compress responses if possible? # 1 (yes) or 0 (false). DEFLATE=0 # Should SSL certificates and hosts be validated? SSL # will only work with signed certificates. VALIDATE_SSL=1 # Proxy configuration (optional parts in []s): # You may also use the 'http_proxy' environment variable. # A value in this file will override that env variable. # PROXY_SERVER=<http://[username:password@]host[:port]> # NO_PROXY_FOR=<host|domain> PROXY_SERVER=http://username:password@squid.proxy.org:3128/ NO_PROXY_FOR=localhost # AIS_DATABASE=<file or url>
Comments
Starting a line with a # makes that line a comment.
Caching
The first seven parameters control caching. The OPeNDAP client can store data you've requested on your local computer. If you repeat a request, the data can be retrieved from this local cache, saving the expense of a network connection. Most web browsers operate the same way. You can control the caching behavior with the following configuration file parameters.
- USE_CACHE
- 1 turns on caching and 0 turns it off.
- MAX_CACHE_SIZE
- The value of MAX_CACHE_SIZE sets the maximum size of the cache in megabytes. Once the cache reaches this size, caching more objects will cause cache garbage collection. OPeNDAP will first purge the cache of any stale entries and then remove remaining entries starting with those that have the lowest hit count. Garbage collection stops when 90% purged.
- MAX_CACHED_OBJ
- This parameter sets the maximum size of any individual object in the cache in megabytes. Objects received from a server larger than this value will not be cached even if there's room for them without purging other objects.
Many web documents, and OPeNDAP data, are delivered with an expiration date in their header information. Generally, this is done for time-sensitive information that may not be valid after the expiration date. You can control the behavior of the OPeNDAP client with respect to expiration dates with two configuration file parameters.
- IGNORE_EXPIRES
- If the value of this parameter is 1 (one), then expiration dates will be ignored, and the caching behavior will be ruled by the DEFAULT_EXPIRES parameter.
- CACHE_ROOT
- This parameter contains the pathname to the cache's top directory. If two or more users want to share a cache, then they must both have read and write permissions to the cache root. NB: Do not share caches unless you're using libdap version 3.8.1 or newer.
- DEFAULT_EXPIRES
- Any data received without an expiration time will expire in the number of seconds given by this parameter. If IGNORE_EXPIRES is zero, this will apply to all data, whether or not it comes with an expiration date. The value is given in seconds. The configuration in the example is set for 24 hours.
- ALWAYS_VALIDATE
- Entries in a cache might be stale even though the default expiration time has not elapsed and some servers might not provide expiration dates for all of their responses. Setting this to '1' forces all cache entries to be validated before the client will use them. This makes for slower response times than just assuming the cached data is valid, but it is faster than fetching new data for every request.
Proxy Servers
An OPeNDAP client can negotiate proxy servers, with help from directions derived from its configuration file. There are three parameters that control proxy behavior. There can be more than one of each of these declarations.
- PROXY_SERVER
- This identifies a proxy server to use for all OPeNDAP requests, except for requests specifically modified by the other two proxy behavior directives. The format is:
PROXY_SERVER=http://[username:password@]<host>[:port]The username, password and port are all optional elements. If no username and password is given the client will assume that none is needed. If the port is not given, the default value of 80 will be used. This syntax is new for clients using libdap 3.8.1 and newer; the old syntax is still supported but was error-prone and should not be used in new .dodsrc files.
- NO_PROXY_FOR
- Use this parameter to say that access to a certain host should never go through a proxy.
Compression
Many OPeNDAP servers support compression of the returned data. This can save network bandwidth, and transmission time. You can tell your client to ask the server for compressed data by including a line in your .dodsrc file like this:
DEFLATE=1
Using a value of zero will make the client request only uncompressed data. If the directive is omitted, the default value is zero.
SSL Validation
SSL Validation can be turned off using the VALIDATE_SSL parameter. if set to '1' all SLL hosts and certificates will be validated. The certificates must be signed for this to work. The default is on. Set to '0' to turn off.
Software you will need for OPeNDAP's Software
Most modern computers has all the basics in place. However, in some rare cases you might be stuck without some tools that you'll need to unpack our software. If that's the case here's what you'll need:
- gzip
- This is the GNU compression and de-compression program. You will need to install it before you can unpack any of the other software described here.
- tar
- This is used to bundle many files together into a single package and it is how we (and many others) package source code.
Most of the software you need for OPeNDAP is avaliable from the GNU archives. Refer to http://www.gnu.org for instructions. Look at http://www.gnu.org/order/ftp.htmlfor a list of mirrors of that archive. Use the mirror closest to you, the transmission will be faster.
Follow the instructions to install each of the following software packages. Typically, you would install a package called foo as follows:
gzip -dc foo.tar.gz | tar xvf - cd foo ./configure make make install
This is simply a guide, of course, and the installation instructions for each software package should be followed carefully.
Building OPeNDAP's Software Using the Source Code
If you need to build the OPeNDAP software, or link it to existing libraries, you will need the following GNU software. Refer to http://www.gnu.org forinstructions. Look at http://www.gnu.org/order/ftp.html for a list of mirrors of that archive. Use the mirror closest to you, the transmission will be faster.
- GNU C/C++ Compiler
- OPeNDAP needs g++, the GNU C++ compiler to compile.
- binutils
- The GNU linker is part of this package.
- libstdc++
- The standard C++ library. Generally part of the C++ compiler so you rarely ever need to get this on its own.
- GNU Make
- GNU Make is not essential, but will make like easier.
- flex
- The GNU lexical-analyzer generator
- bison
- The GNU parser generator.
- CppUnit
- Optional; You'll need this to run the unit tests included with our software
- DejaGNU
- Optional; Needed to run the regression tests included with our software
- Doxygen
- Optional; Needed to build the documentation