Wiki Testing/OPeNDAPUserGuide3

The OPeNDAP Client

There are many different data analysis packages in use. Some packages, such as MATLAB and IDL, are commercially available, but many more are written for a specialized need or application. Many of these use one of the widely available sets of scientific data access functions (called an Application Program Interface, or API ) such as NetCDF, JGOFS, or HDF. There is great variety among all these programs, but one feature they share is that they all access data through files containing that data8. That is to say that each program begins by identifying a file containing the data the user wishes to examine or analyze.

An OPeNDAP client is simply a data analysis application linked with the OPeNDAP libraries instead of the standard data access API. Using this program, a user can look at files containing data in the same way as was possible without the OPeNDAP libraries. However, by using these libraries, a user can also use a URL (URL) , instead of a simple file name, to specify data located anywhere on the Internet. Figure 1.1.3 and figure 1.1.3 illustrate the operation of an application program linked with a standard data access API, and the same program linked with the OPeNDAP version of that API.

An OPeNDAP client is then a data analysis application program modified to become a web browser, somewhat like any other web browser (NCSA Mosaic, Netscape Navigator) with which you may be familiar. A web browser can only display the data it receives, however. What makes an OPeNDAP client different from another web browser is that, unlike Netscape, once the data has been received from an OPeNDAP server, the OPeNDAP client application can compute with it.

Like a web browser, an OPeNDAP client accepts a URL from a user, and parses it to come up with a protocol, an address, and a message. (See Section 2.1 for more information about URLs.) The browser then sends a message to the address, directed to the server who can service the desired protocol, asking for the information specified in the remainder of the URL. Unlike a typical web browser, an OPeNDAP client will not know what to do with data returned for a web page containing text and pictures, but an OPeNDAP server will return scientific data that an OPeNDAP client can understand and process.

Here is a simple example, using the ncview program. This program simply prints out the contents of a netCDF formatted data file, specified on the command line, like this:

> ncview fnocl.nc

Using OPeNDAP, this same function may be executed from any computer connected to the Internet by substituting a URL for the filename above:

> dncview http://dods.gso.uri.edu/cgi-bin/nc/data/fnocl.nc

(See figure 2.1 Aside from the fact that the data is remote, and must be specified with a URL, the program will seem to function in the same way it had with the simple netCDF library (albeit somewhat more slowly due to having to make network connections instead of local file operations). You can find dncview (the ncview program linked with the OPeNDAP library) in the

$DODS_ROOT/src/nc-dods/ncview

directory. Running the above command will produce the following output:

netcdf fnocl {
dimensions:
    time_a = 16
    lat = 17 ;
    lon = 21 ;
    time = 16 ;

variables:
    long u(time_a, lat, ion) ; 
        u:units = ``meter per second'' ; 
        u:long_name = ``Vector wind eastward component'' ; 
        u:missing_value = ``-32767'' ; 
        u:scale_factor = ``0.005'' ; 
    long v(time_a, lat, ion) ; 
        v:units = ``meter per second'' ;
        v:long_name = ``Vector wind northward component'' ;
        v:missing_value = ``-32767'' ;
        v:scale_factor = ``0.005'' ; 
    double lat(lat) ;
        lat:units = ``degree North'' ;
    double lon(lon) ;
        lon:units = ``degree East'' ; 
    double time(time) ;
        time:units = ``hours from base_time'' ;

// global attributes: 
        :base_time = ``88- 10-00:00:00'' ; 
        :title = ``FNOC UV wind components 
                           from 1988- 10 to 1988- 13.'' ;
data:
 u =
  -1728, -2449, -3099, -3585, -3254, -2406, -1252,
    662, 2483, 2910, 2819, 2946, 2745, 2734,
  2931, 2601, 2139, 1845, 1754, 1897, 1854, -1686,
...

Although there are packaged OPeNDAP browsing programs that a user can use to look at data, the user can also construct his or her own. Linking an OPeNDAP API with an already existing program allows a user to create a customized web browser that can access data available from any OPeNDAP server connected to the Internet.

The OPeNDAP APIs are designed to accurately mimic the behavior of several different commonly used scientific data APIs. As of this writing (August 25, 2004), the OPeNDAP API set includes:

API	Description	Components
netCDF	Support for gridded data, such as satellite data, interpolated ship station data, or current meter data.	Server and client.
JGOFS	Support for relational data, such as Sequences. Created by the Joint Globar Ocean Flux Study (JGOFS) project for use with oceanographic station data.	Server and client.
HDF	Support for gridded data. Commonly used for astronomical data and model data.	Server only.
DSP	Oceanographic and geophysical satellite data. Provides support for image processing. Developed at the University of Miami/RSMAS. Primarily used for AVHRR and CZCS data.	Server only.
GRIB	Support for gridded binary data. GRIB is the World Meteorological Organization (WMO) format for the storage of weather information and the exchange of weather product messages.	Server only, due in early 1999.
BUFR	The WMO's standard set of codes for the transmission and storage of meteorological data, using a compressed format with each data value occupying the least number of bits necessary to contain its range of values. Suitable for meteorological observations made from a single point or set of points.	Server only, due in early 1999.
FreeForm	On-the-fly conversion of arbitrarily formatted data, including relational data and gridded data. May be used for sequence data, satellite data, model data, or any other data format that can be described in the flexible FreeForm format definition language. This server can be used to serve data stored in almost all home-grown data formats.	Server only; no client required.
native OPeNDAP	The OPeNDAP class library may be used directly by a client program. It supports relational data, array data, gridded data, and a flexible assortment of data types that can be combined to c accommodate most data models.	Client.

The API set is extensible, meaning that developers can use the OPeNDAP software toolkit to write OPeNDAP-compliant versions of new APIs. See The DODS Toolkit Programmer's Guide for more information.

The most important result of this architecture is that, just as the use of the dncview program above is identical to the original ncview, a user can use remote OPeNDAP data and continue to use the same data analysis and display programs with which he or she is familiar. Any program that uses one of the OPeNDAP-supported APIs may be re-linked to use the OPeNDAP version of that API. This creates an OPeNDAP client. That and a connection to the Internet, are all that a researcher requires to gain access to the available OPeNDAP data.

Configuring Programs to Use OPeNDAP

Relinking an existing program with the OPeNDAP implementation of some data API is a simple procedure. Find the directory that contains the source/object code of the program you want to re-link and modify the makefile (typically called Makefile) for the program so that the OPeNDAP-compliant API library is used in place of the standard API library. (If you can't find the libraries on your system, see Appendix A, or ask the system administrator.) These libraries are:

libdap++.a Software common to all of the OPeNDAP-supported APIs. OPeNDAP also uses facilities from some standard libraries, and these must also be included in the link to resolve all the symbols.

libwww.a The World Wide Web library. This contains the functions used to communicate between the OPeNDAP client and server.

libexpect. Functions from the expect library are used to communicate between OPeNDAP client processes.

libtcl.a Contains definitions necessary for the expect library. The use of this library in the link is not related to the use of Tcl by OPeNDAP clients.

libstdc++.a The GNU C++ class library (This is not necessary if using g++ to re-link.)

You will also need to include the library containing the OPeNDAP-compliant version of the API. The name of this library of course depends on the API, but it is generally in the form

lib API-dods.a 

Where API is an abbreviation indicating the API emulated by the specified library. For example, the OPeNDAP-compliant netCDF library is called libnc-dods.a and the JGOFS version is libjg-dods.a.

An Example Using netCDF

The ncview program is a simple utility that prints the contents of a netCDF-format file to standard output. This section outlines the process used to modify the ncview makefile to link that program with the OPeNDAP netCDF API, thereby turning ncview into a network-ready OPeNDAP client. The process of linking any other program with the corresponding OPeNDAP library is entirely analogous to this one and only requires the substitution of the program name and the appropriate library.

First the link flags were modified so that the library search path would include the likely places to find the OPeNDAP libraries:

LDFLAGS = -g -L$(DODS_ROOT)/lib

DODS_ROOT is an environment variable that indicates the root directory of the OPeNDAP installation, and in this manual is used as shorthand for this directory. It is typically called something like /usr/local/DODS. If you cannot find these directories on your system, consult your system administrator, or refer to Appendix A for information about acquiring and installing the OPeNDAP software.

After the link flags were modified, the OPeNDAP libraries were added to the list of libraries used. The order in which the libraries are listed is important.

LIBS = -lnc-dods -ldap++ -lnc-dods -ldap++ -lwww -ltcl 
       -lexpect -lz -lrx

NOTE: Because OPeNDAP is implemented as a core set of classes contained in one library (libdap++.a) and a set of specializations of those classes in a second library (libnc-dods.a), and because there is a circular dependence between those two libraries, they must be included twice in the linker command.

Finally, g++ was substituted for the link command ( 9).

Potential Problems

When a user links an existing a program to the OPeNDAP libraries, there are several possible conditions that may cause problems.

• Some programs use more than one API.
• Some programs access data using both API and UNIX system calls.
• Some programs use undocumented features of the APIs.

If this is the case for a given program, there is generally no good solution beside rewriting the software to conform to a strict usage of the data reading parts of the given API. Of course if the problem is that the program uses more than one API, you can try linking the program with an OPeNDAP-compliant version of the second API as well.

• Re-linked programs can be very large.

The OPeNDAP libraries are large, and the g++, www, expect, and tcl libraries on which they are built are even larger. This means that the executable version of a re-linked OPeNDAP client can seem unreasonably obese. Much of the disk space is occupied by symbol tables, which can be removed from the executable file with the strip utility. In many cases, a user can recover a substantial amount of disk space this way.

CAUTION: Without familiarity with the OPeNDAP software, it is best only to strip the executable files. Stripping object files or libraries might leave them in a useless condition for the linker. Furthermore, stripping an executable file removes symbol names, which may make diagnosing problems more difficult.

The OPeNDAP libraries only affect the data reading functionality of the specified API. There are no OPeNDAP replacements for functions like netCDF's ncputrec(), that write data to a disk file. These functions are included in the OPeNDAP-compliant API library, but they operate in a manner identical to the original (non-OPeNDAP) versions, that is, they work on local files only, attempting to write "over the network" will result in an error.