The FreeForm Data Handler: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=Preface=
=Preface=
This document describes the OPeNDAP FreeForm ND Data Handler, which
This document describes the OPeNDAP FreeForm ND Data Handler, which
can be used with the OPeNDAP data server.  It is not a
can be used with the OPeNDAP data server.  It is not a
complete description of the \ffnd software.  For that, please refer
complete description of the FreeForm ND software.  For that, please refer
to the \ffnd manual.
to the ND manual.


This document contains much material originally written at the
This document contains much material originally written at the
Line 11: Line 9:
Environmental Satellite, Data, and Information Service, which is part
Environmental Satellite, Data, and Information Service, which is part
of the National Geophysical Data Center in Boulder, Colorado.
of the National Geophysical Data Center in Boulder, Colorado.
This document has been updated to include information on FreeForm ND,
the last release of FreeForm. FreeForm is now supported only for use
with the Distributed Oceanographic Data System; see the
[http://www.opendap.org/<cite>OPeNDAP</cite>] for more information.


We are interested in your comments about the OPeNDAP software, and the
We are interested in your comments about the OPeNDAP software, and the
FreeForm ND software and this document.  Send them to:
FreeForm ND software and this document.  Send them to:
[support@unidata.ucar.edu <cite>mailto:support@unidata.ucar.edu</cite>].
[mailto:support@opendap.org<cite>support@opendap.org</cite>].


Using FreeForm ND with OPeNDAP, a researcher can easily make his or her data
 
Using \ffnd with OPeNDAP, a researcher can easily make his or her data
available to the wider community of OPeNDAP users without having to
available to the wider community of OPeNDAP users without having to
convert that data into another data file format.  This document
convert that data into another data file format.  This document
presents the \ffnd software, and shows how to use it with the OPeNDAP server.
presents the FreeForm ND software, and shows how to use it with the OPeNDAP server.


==Tasks Illustrated in this Guide==


==Tasks Illustrated in this Guide==
For a quick start to using the FreeForm ND
software, see the list below of tasks described in this document.


*Quick start. ([http://docs.opendap.org/index.php/Wiki_Testing/dquick<cite>here</cite>])
*Serving tabular data. ([http://docs.opendap.org/index.php/Wiki_Testing/tblfmt<cite>here</cite>])
*Array tabular data. ([http://docs.opendap.org/index.php/Wiki_Testing/arrayfmt<cite>here</cite>])
*Dealing with data file headers ([http://docs.opendap.org/index.php/Wiki_Testing/hdrfmts<cite>here</cite>])
*Setup of File servers ([http://docs.opendap.org/index.php/Wiki_Testing/fileserv<cite>here</cite>])


For a quick start to getting, installing, and using the \ffnd\
=== Serving data with timestamps in the file names ===
software, see the list below of tasks described in this document.


This handler can read data stored in files that incorporate data strings in their names. This feature was added to support serving data produced and hosted by Remote Sensing Systems (RSS) and while the run-time parameters bear the name of that organization, they can be used for any data that fit the naming conventions they have developed. The naming convention is as follows:


*Quick start. ([http://www.opendap.org/<cite>ff,dquick</cite>])
;The convention: <data source> + '_' + <date_string> + <version> + [_d3d]  
*Getting and installing the \ffnd software.  ([http://www.opendap.org/<cite>ff,install-dods</cite>])
;Daily data: When <date_string> includes YYYYMMDDVV, the file contains ''daily'' data.  
*Serving tabular data. ([http://www.opendap.org/<cite>ff,tblfmt</cite>])
;Averaged data: When <date_string> only includes YYYYMMVV (no ''DD''), or includes (''DD'') and optional ''_d3d'' then the file contains ''averaged'' data.
*Array tabular data. ([http://www.opendap.org/<cite>ff,arrayfmt</cite>])
*Dealing with data file headers ([http://www.opendap.org/<cite>ff,hdrfmt</cite>])
*Setup of File servers ([http://www.opendap.org/<cite>fs</cite>])


For ''daily'' data the format file should be named ''<data source>_daily.fmt'' while averaged data should be named ''<data source>_averaged.fmt''.


To use this feature, set the run-time parameter ''FF.RSSFormatSupport'' to ''yes'' or ''true''. If you store the format files (and optional ancillary DAS files) in a directory other than the data, use the parameter ''FF.RSSFormatFiles'' to name that other directory. Like all handler run-time configuration parameters, these can go in either the ''bes.conf'' or ''ff.conf'' file. Here's an example sniplet from ff.conf showing how these are used:
<source lang=bash>
#
# Data Handler Specific key/value parameters
#
FF.RSSFormatSupport = yes
FF.RSSFormatFiles = /usr/local/RSS
</source>


==Who is this Guide for?==
==Who is this Guide for?==


 
This guide is for people who wish to use FreeForm ND to serve scientific
This guide is for people who wish to use \ffnd to serve scientific
datasets using the OPeNDAP software.  Scientists who wish to share their
datasets using the OPeNDAP software.  Scientists who wish to share their
data with colleagues may also find this a useful system, since it is a
data with colleagues may also find this a useful system, since it is a
relatively simple matter to set up a server that can allow remote
relatively simple matter to set up a server that can allow remote
access to your data.
access to your data.


This documentation assumes that the readers are familiar with
This documentation assumes that the readers are familiar with
Line 61: Line 62:
useful, as will an understanding of elementary internet concepts, such
useful, as will an understanding of elementary internet concepts, such
as URLs and http.
as URLs and http.


This manual also assumes some familiarity with the OPeNDAP software.  If
This manual also assumes some familiarity with the OPeNDAP software.  If
you are starting from scratch, knowing nothing at all about OPeNDAP, we
you are starting from scratch, knowing nothing at all about OPeNDAP, we
strongly encourage you to browse the \DODSuser before reading too far
strongly encourage you to browse the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>The OPeNDAP User Guide</cite>] before reading too far
here.
here.


==Organization of this Document==
==Organization of this Document==
Line 76: Line 75:
<blockquote>
<blockquote>


; [[Wiki_Testing/dintro | Chapter 1]] : contains an overview of the OPeNDAP FreeForm ND Data Handler
; [[Wiki_Testing/dintro | Chapter 1]] : contains an overview of the OPeNDAP FreeForm ND Data Handler software, including how to get it and install it.
software, including how to get it and install it.


; [[Wiki_Testing/dquick | Chapter 12]] : provides a brief introduction to writing
; [[Wiki_Testing/dquick | Chapter 2]] : provides a brief introduction to writing format descriptions and using the OPeNDAP FreeForm ND Data Handler.
format descriptions and using the OPeNDAP FreeForm ND Data Handler.


; [[Wiki_Testing/tblfmt | Chapter 3]] :provides detailed information about
; [[Wiki_Testing/tblfmt | Chapter 3]] :provides detailed information about writing format descriptions to facilitate access to data in tabular formats.
writing format descriptions to facilitate access to data in tabular
formats.


; [[Wiki_Testing/arrayfmt | Chapter 4]] : provides detailed information about
; [[Wiki_Testing/arrayfmt | Chapter 4]] : provides detailed information about writing format descriptions to facilitate access to data in non-tabular (array) formats.
writing format descriptions to facilitate access to data in
non-tabular (array) formats.


; [[Wiki_Testing/hdrfmts | Chapter 5]] : tells you how to work with header
; [[Wiki_Testing/hdrfmts | Chapter 5]] : tells you how to work with header formats.
formats.


; [[Wiki_Testing/ff-server | Chapter 6]] : describes the operation of the OPeNDAP FreeForm ND Data Handler, with tips for writing format files.
; [[Wiki_Testing/ff-server | Chapter 6]] : describes the operation of the OPeNDAP FreeForm ND Data Handler, with tips for writing format files.


; [[Wiki_Testing/fileserv | Chapter 7]] :
; [[Wiki_Testing/fileserv | Chapter 7]] : describes the OPeNDAP file server.


; [[Wiki_Testing/convs | Chapter 8]] : presents FreeForm ND file name
; [[Wiki_Testing/convs | Chapter 8]] : presents FreeForm ND file name conventions, the search rules for locating format files, and standard command line arguments for FreeForm ND programs.
conventions, the search rules for locating format files, and
standard command line arguments for FreeForm ND programs.


; [[Wiki_Testing/fmtconv | Chapter 9]] : shows you how to use the FreeForm ND
; [[Wiki_Testing/fmtconv | Chapter 9]] : shows you how to use the FreeForm ND program <font color='green'>newform</font> to convert data from one format to another and also how to read the data in a binary file.
program <font color='green'>newform</font> to convert data from one format to another and
also how to read the data in a binary file.


; [[Wiki_Testing/datachk | Chapter 10]] : discusses the FreeForm ND program
; [[Wiki_Testing/datachk | Chapter 10]] : discusses the FreeForm ND program <font color='green'>checkvar</font>, which you can use to check data distribution and quality.
<font color='green'>checkvar</font>, which you can use to check data distribution and
quality.


[[Wiki_Testing/hdfutils | Chapter 11]]
;[[Wiki_Testing/hdfutils | Appnedix A]] : provides explanations for a small selection of tools that will be useful for programmers working with the HDF file format.


; \appref{ff,hdf} : provides explanations for a small selection
; [[Wiki_Testing/errors | Appendix B]] : presents a list of common FreeForm ND error messages.  These are the error messages that may be issued by the FreeForm ND utilities, such as <font color='green'>newform</font>, not the OPeNDAP FreeForm ND Data Handler.
of tools that will be useful for programmers working with the HDF
file format.
 
; \appref{ff,errors} : presents a list of common FreeForm ND error
messages.  These are the error messages that may be issued by the
\ffnd utilities, such as <font color='green'>newform</font>, not the OPeNDAP FreeForm ND Data Handler.


</blockquote>
</blockquote>
Line 131: Line 111:
012345678901234567890123456789012345678901234567890
012345678901234567890123456789012345678901234567890
</pre>
</pre>
[[Category:BES Modules|Freeform]]

Latest revision as of 01:21, 11 December 2014

Preface

This document describes the OPeNDAP FreeForm ND Data Handler, which can be used with the OPeNDAP data server. It is not a complete description of the FreeForm ND software. For that, please refer to the ND manual.

This document contains much material originally written at the National Oceanic and Atmospheric Administration's National Environmental Satellite, Data, and Information Service, which is part of the National Geophysical Data Center in Boulder, Colorado.

We are interested in your comments about the OPeNDAP software, and the FreeForm ND software and this document. Send them to: support@opendap.org.

Using FreeForm ND with OPeNDAP, a researcher can easily make his or her data available to the wider community of OPeNDAP users without having to convert that data into another data file format. This document presents the FreeForm ND software, and shows how to use it with the OPeNDAP server.

Tasks Illustrated in this Guide

For a quick start to using the FreeForm ND software, see the list below of tasks described in this document.

  • Quick start. (here)
  • Serving tabular data. (here)
  • Array tabular data. (here)
  • Dealing with data file headers (here)
  • Setup of File servers (here)

Serving data with timestamps in the file names

This handler can read data stored in files that incorporate data strings in their names. This feature was added to support serving data produced and hosted by Remote Sensing Systems (RSS) and while the run-time parameters bear the name of that organization, they can be used for any data that fit the naming conventions they have developed. The naming convention is as follows:

The convention
+ '_' + <date_string> + <version> + [_d3d]
Daily data
When <date_string> includes YYYYMMDDVV, the file contains daily data.
Averaged data
When <date_string> only includes YYYYMMVV (no DD), or includes (DD) and optional _d3d then the file contains averaged data.

For daily data the format file should be named _daily.fmt while averaged data should be named _averaged.fmt.

To use this feature, set the run-time parameter FF.RSSFormatSupport to yes or true. If you store the format files (and optional ancillary DAS files) in a directory other than the data, use the parameter FF.RSSFormatFiles to name that other directory. Like all handler run-time configuration parameters, these can go in either the bes.conf or ff.conf file. Here's an example sniplet from ff.conf showing how these are used:

# 
# Data Handler Specific key/value parameters
#
FF.RSSFormatSupport = yes
FF.RSSFormatFiles = /usr/local/RSS

Who is this Guide for?

This guide is for people who wish to use FreeForm ND to serve scientific datasets using the OPeNDAP software. Scientists who wish to share their data with colleagues may also find this a useful system, since it is a relatively simple matter to set up a server that can allow remote access to your data.

This documentation assumes that the readers are familiar with computers and the internet, but are not necessarily programmers. More than a passing familiarity with different data file formats will be useful, as will an understanding of elementary internet concepts, such as URLs and http.

This manual also assumes some familiarity with the OPeNDAP software. If you are starting from scratch, knowing nothing at all about OPeNDAP, we strongly encourage you to browse the The OPeNDAP User Guide before reading too far here.

Organization of this Document

This book contains both introductory and reference material. There is also a description of the installation procedure.

Chapter 1
contains an overview of the OPeNDAP FreeForm ND Data Handler software, including how to get it and install it.
Chapter 2
provides a brief introduction to writing format descriptions and using the OPeNDAP FreeForm ND Data Handler.
Chapter 3
provides detailed information about writing format descriptions to facilitate access to data in tabular formats.
Chapter 4
provides detailed information about writing format descriptions to facilitate access to data in non-tabular (array) formats.
Chapter 5
tells you how to work with header formats.
Chapter 6
describes the operation of the OPeNDAP FreeForm ND Data Handler, with tips for writing format files.
Chapter 7
describes the OPeNDAP file server.
Chapter 8
presents FreeForm ND file name conventions, the search rules for locating format files, and standard command line arguments for FreeForm ND programs.
Chapter 9
shows you how to use the FreeForm ND program newform to convert data from one format to another and also how to read the data in a binary file.
Chapter 10
discusses the FreeForm ND program checkvar, which you can use to check data distribution and quality.
Appnedix A
provides explanations for a small selection of tools that will be useful for programmers working with the HDF file format.
Appendix B
presents a list of common FreeForm ND error messages. These are the error messages that may be issued by the FreeForm ND utilities, such as newform, not the OPeNDAP FreeForm ND Data Handler.


A position box is often used in this book to indicate column position of field values in data files. It is shown at the beginning of a data list in the documentation, but does not appear in the data file itself. It looks something like this:

1         2         3         4         5         6
012345678901234567890123456789012345678901234567890