The FreeForm Data Handler

From OPeNDAP Documentation
Revision as of 01:21, 11 December 2014 by Jimg (talk | contribs) (Serving data with timestamps in the file names)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

1 Preface

This document describes the OPeNDAP FreeForm ND Data Handler, which can be used with the OPeNDAP data server. It is not a complete description of the FreeForm ND software. For that, please refer to the ND manual.

This document contains much material originally written at the National Oceanic and Atmospheric Administration's National Environmental Satellite, Data, and Information Service, which is part of the National Geophysical Data Center in Boulder, Colorado.

We are interested in your comments about the OPeNDAP software, and the FreeForm ND software and this document. Send them to: support@opendap.org.

Using FreeForm ND with OPeNDAP, a researcher can easily make his or her data available to the wider community of OPeNDAP users without having to convert that data into another data file format. This document presents the FreeForm ND software, and shows how to use it with the OPeNDAP server.

1.1 Tasks Illustrated in this Guide

For a quick start to using the FreeForm ND software, see the list below of tasks described in this document.

  • Quick start. (here)
  • Serving tabular data. (here)
  • Array tabular data. (here)
  • Dealing with data file headers (here)
  • Setup of File servers (here)

1.1.1 Serving data with timestamps in the file names

This handler can read data stored in files that incorporate data strings in their names. This feature was added to support serving data produced and hosted by Remote Sensing Systems (RSS) and while the run-time parameters bear the name of that organization, they can be used for any data that fit the naming conventions they have developed. The naming convention is as follows:

The convention
+ '_' + <date_string> + <version> + [_d3d]
Daily data
When <date_string> includes YYYYMMDDVV, the file contains daily data.
Averaged data
When <date_string> only includes YYYYMMVV (no DD), or includes (DD) and optional _d3d then the file contains averaged data.

For daily data the format file should be named _daily.fmt while averaged data should be named _averaged.fmt.

To use this feature, set the run-time parameter FF.RSSFormatSupport to yes or true. If you store the format files (and optional ancillary DAS files) in a directory other than the data, use the parameter FF.RSSFormatFiles to name that other directory. Like all handler run-time configuration parameters, these can go in either the bes.conf or ff.conf file. Here's an example sniplet from ff.conf showing how these are used:

# 
# Data Handler Specific key/value parameters
#
FF.RSSFormatSupport = yes
FF.RSSFormatFiles = /usr/local/RSS

1.2 Who is this Guide for?

This guide is for people who wish to use FreeForm ND to serve scientific datasets using the OPeNDAP software. Scientists who wish to share their data with colleagues may also find this a useful system, since it is a relatively simple matter to set up a server that can allow remote access to your data.

This documentation assumes that the readers are familiar with computers and the internet, but are not necessarily programmers. More than a passing familiarity with different data file formats will be useful, as will an understanding of elementary internet concepts, such as URLs and http.

This manual also assumes some familiarity with the OPeNDAP software. If you are starting from scratch, knowing nothing at all about OPeNDAP, we strongly encourage you to browse the The OPeNDAP User Guide before reading too far here.

1.3 Organization of this Document

This book contains both introductory and reference material. There is also a description of the installation procedure.

Chapter 1 
contains an overview of the OPeNDAP FreeForm ND Data Handler software, including how to get it and install it.
Chapter 2 
provides a brief introduction to writing format descriptions and using the OPeNDAP FreeForm ND Data Handler.
Chapter 3 
provides detailed information about writing format descriptions to facilitate access to data in tabular formats.
Chapter 4 
provides detailed information about writing format descriptions to facilitate access to data in non-tabular (array) formats.
Chapter 5 
tells you how to work with header formats.
Chapter 6 
describes the operation of the OPeNDAP FreeForm ND Data Handler, with tips for writing format files.
Chapter 7 
describes the OPeNDAP file server.
Chapter 8 
presents FreeForm ND file name conventions, the search rules for locating format files, and standard command line arguments for FreeForm ND programs.
Chapter 9 
shows you how to use the FreeForm ND program newform to convert data from one format to another and also how to read the data in a binary file.
Chapter 10 
discusses the FreeForm ND program checkvar, which you can use to check data distribution and quality.
Appnedix A 
provides explanations for a small selection of tools that will be useful for programmers working with the HDF file format.
Appendix B 
presents a list of common FreeForm ND error messages. These are the error messages that may be issued by the FreeForm ND utilities, such as newform, not the OPeNDAP FreeForm ND Data Handler.


A position box is often used in this book to indicate column position of field values in data files. It is shown at the beginning of a data list in the documentation, but does not appear in the data file itself. It looks something like this:

1         2         3         4         5         6
012345678901234567890123456789012345678901234567890