https://docs.opendap.org/api.php?action=feedcontributions&feedformat=atom&user=YuanOPeNDAP Documentation - User contributions [en]2026-05-31T08:42:05ZUser contributionsMediaWiki 1.38.4https://docs.opendap.org/index.php?title=Wiki_Testing/adding-appa&diff=2083Wiki Testing/adding-appa2008-02-07T13:54:31Z<p>Yuan: /* DODS Names */</p>
<hr />
<div>=DODS Names=<br />
<br />
<br />
Here is a list of the currently used DODS variable names. The most<br />
up-to-date list is available at<br />
[http://unidata.ucar.edu/packages/dods/names.dat <cite>http://unidata.ucar.edu/packages/dods/names.dat</cite>]<br />
Where one exists, the GCMD parameter name follows the DODS variable<br />
name.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/adding-attr&diff=2077Wiki Testing/adding-attr2008-02-07T12:44:23Z<p>Yuan: /* A Worked Example */</p>
<hr />
<div>=Adding Data Attributes=<br />
<br />
<br />
One of the original design goals of DODS was to permit data providers<br />
to use DODS with as little work as possible. A corollary to this<br />
requirement is that if ancillary information about a dataset is<br />
required, it must be very easy to provide. The DODS architecture<br />
allows a data provider to add attribute information to a dataset<br />
without modifying the data files at all. All that is required is a<br />
separate file, stored in the same directory as the data.<br />
<br />
This chapter shows how you can easily add attribute information to an<br />
existing dataset to comply with the DODS data standard. We start by<br />
creating a small dataset and serving it with DODS FreeForm . A similar<br />
example is presented in [http://docs.opendap.org/index.php/Wiki_Testing/dpref<cite>FreeForm Guide</cite>] , highlighting different aspects.<br />
<br />
All the files used in this chapter are available on the [http://www.opendap.org/examples/ffsimple.dat<cite>DODS examples</cite>] .<br />
<br />
==A Worked Example==<br />
<br />
Consider the following table of numbers:<br />
<br />
<pre><br />
-47.303545 -176.161101 1.17125<br />
-25.928001 -0.777265 2.07288<br />
-28.286662 35.591879 2.36377<br />
<br />
12.588231 149.408117 -100.000<br />
-63.223548 55.319598 0.04503<br />
<br />
54.118314 -136.940570 1.04085<br />
-38.818812 91.411330 1.39978<br />
-34.577065 30.172129 2.09096<br />
<br />
27.331551 -155.233735 2.30917<br />
<br />
11.624981 -113.660611 2.75036<br />
</pre><br />
<br />
This table represents a series of temperature measurements at ten<br />
different geographical locations. The latitudes and longitudes are<br />
given in decimal degrees, and the temperatures are given in units of<br />
ten degrees. That is, the temperature as recorded, times ten, equals<br />
the temperature in Celsius. There is one temperature measurement<br />
missing, and its place is marked with a value of <font color='green'>-100.0</font>.<br />
<br />
(This dataset, and the ancillary files discussed in this chapter, are<br />
in the DODS examples directory, and is available at the DODS examples ,<br />
under <font color='green'>dasex.*</font>.)<br />
<br />
The dataset shown here can be served as is, with the FreeForm NDs , using<br />
the following format file (<font color='green'>dasex.fmt</font>):<br />
<br />
<pre><br />
ASCII_data "lat/lon"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
t 24 31 double 4<br />
</pre><br />
<br />
Put both of these files in your <font color='green'>htdocs</font> directory (refer to the<br />
documentation for your web server to locate this directory). After<br />
installing the FreeForm ND , you can issue data requests on this<br />
dataset. Use a web browser like netscape, and enter the following<br />
URL (Use your own server's name instead of <font color='green'>machine.edu</font>.):<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.dds<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Sequence {<br />
<br />
Float64 latitude;<br />
<br />
Float64 longitude;<br />
<br />
Float64 t;<br />
<br />
} lat/lon;<br />
} dasex;<br />
</pre><br />
<br />
The is the DDS (Data Description Structure) of the dasex dataset. (See<br />
[http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>DODS User Guide</cite>] for more information about the DDS and what it means, as<br />
well as a description of what a Sequence is.) Instead of <font color='green'>.dds</font> at<br />
the end of the URL, you can use <font color='green'>.asc</font> to see the data itself.<br />
Try that.<br />
<br />
To see data attributes, try the same URL, ending with <font color='green'>.das</font>:<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.das<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file " ...";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
The DAS is a list of the dataset's attributes. This dataset has two,<br />
and they are global ones. That is, they describe the entire dataset,<br />
and don't correspond to any of the data variables listed in the DDS.<br />
They are also in a container called <font color='green'>FF_GLOBAL</font>, which is another<br />
hint that they are global attributes. The <font color='green'>Server</font> string<br />
describes the actual DODS server that sent the data, and the<br />
<font color='green'>Native_file</font> string is a long one, containing information about<br />
the original file, and what it contained.<br />
<br />
This is all very well, but what if someone wanted to know what units<br />
the <font color='green'>t</font> variable was in, or where the data came from, or what that<br />
value of -100 is supposed to imply, or even whether the variable<br />
represents temperature or time. The data themselves are silent on<br />
those issues, so either we have to provide additional documentation,<br />
or we have to introduce attributes for the data variables, to make the<br />
dataset self-documenting. We do that with an ancillary file of data<br />
attributes. Call this file <font color='green'>dasex.das</font>:<br />
<br />
<pre><br />
Attributes {<br />
<br />
t {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now the DAS that is returned from the dataset looks like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file "...";<br />
<br />
}<br />
<br />
temp {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now you can see that <font color='green'>t</font> stands for "Temperature" and that the<br />
<font color='green'>-100</font> is missing data, and that all the measurements have to<br />
multiplied by 10 before they are in degrees Celsius.<br />
<br />
Now that the data have these additional attributes, the dataset is<br />
compliant with level 1 of the DODS data standard. This can also be<br />
noted in the DAS, with the addition of another global attribute<br />
container called <font color='green'>DODS</font>. A file called <font color='green'>dasex2.das</font> contains<br />
the following:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Conventions "DODS";<br />
<br />
String Acknowledge "Example dataset from the DODS documentation.";<br />
<br />
String history "Created for DODS standards book.";<br />
}<br />
</pre><br />
<br />
Add this information to the bottom of the <font color='green'>dasex.das</font> file<br />
(''before'' the last closing bracket), and now the dataset is<br />
compliant with level 2 of the DODS data standard.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/adding-attr&diff=2076Wiki Testing/adding-attr2008-02-07T12:23:18Z<p>Yuan: /* A Worked Example */</p>
<hr />
<div>=Adding Data Attributes=<br />
<br />
<br />
One of the original design goals of DODS was to permit data providers<br />
to use DODS with as little work as possible. A corollary to this<br />
requirement is that if ancillary information about a dataset is<br />
required, it must be very easy to provide. The DODS architecture<br />
allows a data provider to add attribute information to a dataset<br />
without modifying the data files at all. All that is required is a<br />
separate file, stored in the same directory as the data.<br />
<br />
This chapter shows how you can easily add attribute information to an<br />
existing dataset to comply with the DODS data standard. We start by<br />
creating a small dataset and serving it with DODS FreeForm . A similar<br />
example is presented in [http://docs.opendap.org/index.php/Wiki_Testing/dpref<cite>FreeForm Guide</cite>] , highlighting different aspects.<br />
<br />
All the files used in this chapter are available on the [http://www.opendap.org/examples/ffsimple.dat<cite>DODS examples</cite>] .<br />
<br />
==A Worked Example==<br />
<br />
Consider the following table of numbers:<br />
<br />
<pre><br />
-47.303545 -176.161101 1.17125<br />
-25.928001 -0.777265 2.07288<br />
-28.286662 35.591879 2.36377<br />
<br />
12.588231 149.408117 -100.000<br />
-63.223548 55.319598 0.04503<br />
<br />
54.118314 -136.940570 1.04085<br />
-38.818812 91.411330 1.39978<br />
-34.577065 30.172129 2.09096<br />
<br />
27.331551 -155.233735 2.30917<br />
<br />
11.624981 -113.660611 2.75036<br />
</pre><br />
<br />
This table represents a series of temperature measurements at ten<br />
different geographical locations. The latitudes and longitudes are<br />
given in decimal degrees, and the temperatures are given in units of<br />
ten degrees. That is, the temperature as recorded, times ten, equals<br />
the temperature in Celsius. There is one temperature measurement<br />
missing, and its place is marked with a value of <font color='green'>-100.0</font>.<br />
<br />
(This dataset, and the ancillary files discussed in this chapter, are<br />
in the DODS examples directory, and is available at the DODS examples ,<br />
under <font color='green'>dasex.*</font>.)<br />
<br />
The dataset shown here can be served as is, with the FreeForm NDs , using<br />
the following format file (<font color='green'>dasex.fmt</font>):\footnote{See the<br />
<br />
[http://docs.opendap.org/index.php/Wiki_Testing/dpref<cite>FreeForms Guide</cite>] for more information about that software.}<br />
<br />
<br />
<pre><br />
ASCII_data "lat/lon"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
t 24 31 double 4<br />
</pre><br />
<br />
Put both of these files in your <font color='green'>htdocs</font> directory (refer to the<br />
documentation for your web server to locate this directory). After<br />
installing the FreeForm ND , you can issue data requests on this<br />
dataset. Use a web browser like netscape, and enter the following<br />
URL (Use your own server's name instead of <font color='green'>machine.edu</font>.):<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.dds<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Sequence {<br />
<br />
Float64 latitude;<br />
<br />
Float64 longitude;<br />
<br />
Float64 t;<br />
<br />
} lat/lon;<br />
} dasex;<br />
</pre><br />
<br />
The is the DDS (Data Description Structure) of the dasex dataset. (See<br />
[http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>DODS User Guide</cite>] for more information about the DDS and what it means, as<br />
well as a description of what a Sequence is.) Instead of <font color='green'>.dds</font> at<br />
the end of the URL, you can use <font color='green'>.asc</font> to see the data itself.<br />
Try that.<br />
<br />
To see data attributes, try the same URL, ending with <font color='green'>.das</font>:<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.das<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file " ...";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
The DAS is a list of the dataset's attributes. This dataset has two,<br />
and they are global ones. That is, they describe the entire dataset,<br />
and don't correspond to any of the data variables listed in the DDS.<br />
They are also in a container called <font color='green'>FF_GLOBAL</font>, which is another<br />
hint that they are global attributes. The <font color='green'>Server</font> string<br />
describes the actual DODS server that sent the data, and the<br />
<font color='green'>Native_file</font> string is a long one, containing information about<br />
the original file, and what it contained.<br />
<br />
This is all very well, but what if someone wanted to know what units<br />
the <font color='green'>t</font> variable was in, or where the data came from, or what that<br />
value of -100 is supposed to imply, or even whether the variable<br />
represents temperature or time. The data themselves are silent on<br />
those issues, so either we have to provide additional documentation,<br />
or we have to introduce attributes for the data variables, to make the<br />
dataset self-documenting. We do that with an ancillary file of data<br />
attributes. Call this file <font color='green'>dasex.das</font>:<br />
<br />
<pre><br />
Attributes {<br />
<br />
t {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now the DAS that is returned from the dataset looks like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file "...";<br />
<br />
}<br />
<br />
temp {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now you can see that <font color='green'>t</font> stands for "Temperature" and that the<br />
<font color='green'>-100</font> is missing data, and that all the measurements have to<br />
multiplied by 10 before they are in degrees Celsius.<br />
<br />
Now that the data have these additional attributes, the dataset is<br />
compliant with level 1 of the DODS data standard. This can also be<br />
noted in the DAS, with the addition of another global attribute<br />
container called <font color='green'>DODS</font>. A file called <font color='green'>dasex2.das</font> contains<br />
the following:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Conventions "DODS";<br />
<br />
String Acknowledge "Example dataset from the DODS documentation.";<br />
<br />
String history "Created for DODS standards book.";<br />
}<br />
</pre><br />
<br />
Add this information to the bottom of the <font color='green'>dasex.das</font> file<br />
(''before'' the last closing bracket), and now the dataset is<br />
compliant with level 2 of the DODS data standard.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/adding-attr&diff=2075Wiki Testing/adding-attr2008-02-07T12:17:11Z<p>Yuan: /* Adding Data Attributes */</p>
<hr />
<div>=Adding Data Attributes=<br />
<br />
<br />
One of the original design goals of DODS was to permit data providers<br />
to use DODS with as little work as possible. A corollary to this<br />
requirement is that if ancillary information about a dataset is<br />
required, it must be very easy to provide. The DODS architecture<br />
allows a data provider to add attribute information to a dataset<br />
without modifying the data files at all. All that is required is a<br />
separate file, stored in the same directory as the data.<br />
<br />
This chapter shows how you can easily add attribute information to an<br />
existing dataset to comply with the DODS data standard. We start by<br />
creating a small dataset and serving it with DODS FreeForm . A similar<br />
example is presented in [http://docs.opendap.org/index.php/Wiki_Testing/dpref<cite>FreeForm Guide</cite>] , highlighting different aspects.<br />
<br />
All the files used in this chapter are available on the [http://www.opendap.org/examples/ffsimple.dat<cite>DODS examples</cite>] .<br />
<br />
==A Worked Example==<br />
<br />
Consider the following table of numbers:<br />
<br />
<pre><br />
-47.303545 -176.161101 1.17125<br />
-25.928001 -0.777265 2.07288<br />
-28.286662 35.591879 2.36377<br />
<br />
12.588231 149.408117 -100.000<br />
-63.223548 55.319598 0.04503<br />
<br />
54.118314 -136.940570 1.04085<br />
-38.818812 91.411330 1.39978<br />
-34.577065 30.172129 2.09096<br />
<br />
27.331551 -155.233735 2.30917<br />
<br />
11.624981 -113.660611 2.75036<br />
</pre><br />
<br />
This table represents a series of temperature measurements at ten<br />
different geographical locations. The latitudes and longitudes are<br />
given in decimal degrees, and the temperatures are given in units of<br />
ten degrees. That is, the temperature as recorded, times ten, equals<br />
the temperature in Celsius. There is one temperature measurement<br />
missing, and its place is marked with a value of <font color='green'>-100.0</font>.<br />
<br />
(This dataset, and the ancillary files discussed in this chapter, are<br />
in the DODS examples directory, and is available at the \DODSexamples ,<br />
under <font color='green'>dasex.*</font>.)<br />
<br />
The dataset shown here can be served as is, with the \DODSffs , using<br />
the following format file (<font color='green'>dasex.fmt</font>):\footnote{See the<br />
<br />
\DODSffbook for more information about that software.}<br />
<br />
<br />
<pre><br />
ASCII_data "lat/lon"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
t 24 31 double 4<br />
</pre><br />
<br />
Put both of these files in your <font color='green'>htdocs</font> directory (refer to the<br />
documentation for your web server to locate this directory). After<br />
installing the \DODSffs , you can issue data requests on this<br />
dataset. Use a web browser like netscape, and enter the following<br />
URL (Use your own server's name instead of <font color='green'>machine.edu</font>.):<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.dds<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Sequence {<br />
<br />
Float64 latitude;<br />
<br />
Float64 longitude;<br />
<br />
Float64 t;<br />
<br />
} lat/lon;<br />
} dasex;<br />
</pre><br />
<br />
The is the DDS (Data Description Structure) of the dasex dataset. (See<br />
\DODSuser for more information about the DDS and what it means, as<br />
well as a description of what a Sequence is.) Instead of <font color='green'>.dds</font> at<br />
the end of the URL, you can use <font color='green'>.asc</font> to see the data itself.<br />
Try that.<br />
<br />
To see data attributes, try the same URL, ending with <font color='green'>.das</font>:<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.das<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file " ...";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
The DAS is a list of the dataset's attributes. This dataset has two,<br />
and they are global ones. That is, they describe the entire dataset,<br />
and don't correspond to any of the data variables listed in the DDS.<br />
They are also in a container called <font color='green'>FF_GLOBAL</font>, which is another<br />
hint that they are global attributes. The <font color='green'>Server</font> string<br />
describes the actual DODS server that sent the data, and the<br />
<font color='green'>Native_file</font> string is a long one, containing information about<br />
the original file, and what it contained.<br />
<br />
This is all very well, but what if someone wanted to know what units<br />
the <font color='green'>t</font> variable was in, or where the data came from, or what that<br />
value of -100 is supposed to imply, or even whether the variable<br />
represents temperature or time. The data themselves are silent on<br />
those issues, so either we have to provide additional documentation,<br />
or we have to introduce attributes for the data variables, to make the<br />
dataset self-documenting. We do that with an ancillary file of data<br />
attributes. Call this file <font color='green'>dasex.das</font>\footnote{It is called<br />
<br />
dasex1.das in the \DODSexamples . You will have to rename this file<br />
<br />
to make it work.}:<br />
<br />
<pre><br />
Attributes {<br />
<br />
t {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now the DAS that is returned from the dataset looks like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file "...";<br />
<br />
}<br />
<br />
temp {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now you can see that <font color='green'>t</font> stands for "Temperature" and that the<br />
<font color='green'>-100</font> is missing data, and that all the measurements have to<br />
multiplied by 10 before they are in degrees Celsius.<br />
<br />
Now that the data have these additional attributes, the dataset is<br />
compliant with level 1 of the DODS data standard. This can also be<br />
noted in the DAS, with the addition of another global attribute<br />
container called <font color='green'>DODS</font>. A file called <font color='green'>dasex2.das</font> contains<br />
the following:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Conventions "DODS";<br />
<br />
String Acknowledge "Example dataset from the DODS documentation.";<br />
<br />
String history "Created for DODS standards book.";<br />
}<br />
</pre><br />
<br />
Add this information to the bottom of the <font color='green'>dasex.das</font> file<br />
(''before'' the last closing bracket), and now the dataset is<br />
compliant with level 2 of the DODS data standard.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/adding-attr&diff=2073Wiki Testing/adding-attr2008-02-07T12:13:08Z<p>Yuan: /* Adding Data Attributes */</p>
<hr />
<div>=Adding Data Attributes=<br />
<br />
<br />
One of the original design goals of DODS was to permit data providers<br />
to use DODS with as little work as possible. A corollary to this<br />
requirement is that if ancillary information about a dataset is<br />
required, it must be very easy to provide. The DODS architecture<br />
allows a data provider to add attribute information to a dataset<br />
without modifying the data files at all. All that is required is a<br />
separate file, stored in the same directory as the data.<br />
<br />
This chapter shows how you can easily add attribute information to an<br />
existing dataset to comply with the DODS data standard. We start by<br />
creating a small dataset and serving it with DODS FreeForm . A similar<br />
example is presented in [http://docs.opendap.org/index.php/Wiki_Testing/dpref<cite>FreeForm Guide</cite>] , highlighting different aspects.<br />
<br />
All the files used in this chapter are available on the \DODSexamples .<br />
<br />
==A Worked Example==<br />
<br />
Consider the following table of numbers:<br />
<br />
<pre><br />
-47.303545 -176.161101 1.17125<br />
-25.928001 -0.777265 2.07288<br />
-28.286662 35.591879 2.36377<br />
<br />
12.588231 149.408117 -100.000<br />
-63.223548 55.319598 0.04503<br />
<br />
54.118314 -136.940570 1.04085<br />
-38.818812 91.411330 1.39978<br />
-34.577065 30.172129 2.09096<br />
<br />
27.331551 -155.233735 2.30917<br />
<br />
11.624981 -113.660611 2.75036<br />
</pre><br />
<br />
This table represents a series of temperature measurements at ten<br />
different geographical locations. The latitudes and longitudes are<br />
given in decimal degrees, and the temperatures are given in units of<br />
ten degrees. That is, the temperature as recorded, times ten, equals<br />
the temperature in Celsius. There is one temperature measurement<br />
missing, and its place is marked with a value of <font color='green'>-100.0</font>.<br />
<br />
(This dataset, and the ancillary files discussed in this chapter, are<br />
in the DODS examples directory, and is available at the \DODSexamples ,<br />
under <font color='green'>dasex.*</font>.)<br />
<br />
The dataset shown here can be served as is, with the \DODSffs , using<br />
the following format file (<font color='green'>dasex.fmt</font>):\footnote{See the<br />
<br />
\DODSffbook for more information about that software.}<br />
<br />
<br />
<pre><br />
ASCII_data "lat/lon"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
t 24 31 double 4<br />
</pre><br />
<br />
Put both of these files in your <font color='green'>htdocs</font> directory (refer to the<br />
documentation for your web server to locate this directory). After<br />
installing the \DODSffs , you can issue data requests on this<br />
dataset. Use a web browser like netscape, and enter the following<br />
URL (Use your own server's name instead of <font color='green'>machine.edu</font>.):<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.dds<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Sequence {<br />
<br />
Float64 latitude;<br />
<br />
Float64 longitude;<br />
<br />
Float64 t;<br />
<br />
} lat/lon;<br />
} dasex;<br />
</pre><br />
<br />
The is the DDS (Data Description Structure) of the dasex dataset. (See<br />
\DODSuser for more information about the DDS and what it means, as<br />
well as a description of what a Sequence is.) Instead of <font color='green'>.dds</font> at<br />
the end of the URL, you can use <font color='green'>.asc</font> to see the data itself.<br />
Try that.<br />
<br />
To see data attributes, try the same URL, ending with <font color='green'>.das</font>:<br />
<br />
<pre><br />
http://machine.edu/cgi-bin/nph-ff/dasex.dat.das<br />
</pre><br />
<br />
You should see something like this:<br />
<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file " ...";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
The DAS is a list of the dataset's attributes. This dataset has two,<br />
and they are global ones. That is, they describe the entire dataset,<br />
and don't correspond to any of the data variables listed in the DDS.<br />
They are also in a container called <font color='green'>FF_GLOBAL</font>, which is another<br />
hint that they are global attributes. The <font color='green'>Server</font> string<br />
describes the actual DODS server that sent the data, and the<br />
<font color='green'>Native_file</font> string is a long one, containing information about<br />
the original file, and what it contained.<br />
<br />
This is all very well, but what if someone wanted to know what units<br />
the <font color='green'>t</font> variable was in, or where the data came from, or what that<br />
value of -100 is supposed to imply, or even whether the variable<br />
represents temperature or time. The data themselves are silent on<br />
those issues, so either we have to provide additional documentation,<br />
or we have to introduce attributes for the data variables, to make the<br />
dataset self-documenting. We do that with an ancillary file of data<br />
attributes. Call this file <font color='green'>dasex.das</font>\footnote{It is called<br />
<br />
dasex1.das in the \DODSexamples . You will have to rename this file<br />
<br />
to make it work.}:<br />
<br />
<pre><br />
Attributes {<br />
<br />
t {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now the DAS that is returned from the dataset looks like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
FF_GLOBAL {<br />
<br />
String Server "DODS FFND release 4.2.3";<br />
<br />
String Native_file "...";<br />
<br />
}<br />
<br />
temp {<br />
<br />
String short_name "Temperature";<br />
<br />
String units "DegreesC";<br />
<br />
Float64 missing_data -100.0;<br />
<br />
Float64 scale_factor 10.0;<br />
<br />
}<br />
<br />
latitude {<br />
<br />
String short_name "Latitude";<br />
<br />
String units "degree_north";<br />
<br />
}<br />
<br />
longitude {<br />
<br />
String short_name "Longitude";<br />
<br />
String units "degree_east";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
Now you can see that <font color='green'>t</font> stands for "Temperature" and that the<br />
<font color='green'>-100</font> is missing data, and that all the measurements have to<br />
multiplied by 10 before they are in degrees Celsius.<br />
<br />
Now that the data have these additional attributes, the dataset is<br />
compliant with level 1 of the DODS data standard. This can also be<br />
noted in the DAS, with the addition of another global attribute<br />
container called <font color='green'>DODS</font>. A file called <font color='green'>dasex2.das</font> contains<br />
the following:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Conventions "DODS";<br />
<br />
String Acknowledge "Example dataset from the DODS documentation.";<br />
<br />
String history "Created for DODS standards book.";<br />
}<br />
</pre><br />
<br />
Add this information to the bottom of the <font color='green'>dasex.das</font> file<br />
(''before'' the last closing bracket), and now the dataset is<br />
compliant with level 2 of the DODS data standard.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2061Wiki Testing/dods-standards2008-02-07T10:29:23Z<p>Yuan: /* Very Useful Attributes, But Not Essential (Class 2) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. <br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for the variable's data. The units attribute should be formatted as per the recommendations in the Unidata [http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. <br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to be multiplied by this factor after the data are read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to be added to the data after it is read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified. If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation function used to convert the raw data into the units specifed in the <font color='green'>units</font> attribute. <br />
<br />
<br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote> NOTE: A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</cite>] .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<br />
; <font color='green'>missing_value</font> : This is a conventional name for a missing value that will not be treated in any special way by the client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in that it describes data that isn't there, but shouldn't have been, either. That is, where a missing value might be used to fill in for a sensor malfunction, a null value is used to indicate that no data was taken. A dataset that contained random data interpolated onto a grid might use a null value on those grid points too distant from data values to make an accurate estimate.<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of missing data. In this case, data would never have been at those points. Land points in gridded sea-surface temperature data would be default values, as would the end of profile data vectors filled to uniform length. This differs some from the semantics of the COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the DODS project from adopting the COARDS name as a synonym.<br />
<br />
<br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from a list of DODS names (see \appref{std-names}). Including this attribute is a way to ensure that two datasets can be usefully compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to identify the dataset as conforming to the DODS data standard, identified here. The attribute should be a string with the value "DODS". Note that this is a global attribute, and should appear in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement paragraph that can appear in papers that use the data from this dataset. For example: "The principal investigators in the production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley [http://www.cdc.noaa.gov/coads/participants.html<cite>cite</cite>]." and so on. This is a global attribute, and should appear in the "DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to record the evolution of the data contained within a DODS data file. Applications which process this data can append their information to this attribute. This is a global attribute, and should appear in the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use of the data, they should be noted in the string with this name. This is a global attribute, and should appear in the "DODS" attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter valid names. See the [http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl<cite>GCMD Parameters" list maintained at GCMD</cite>]</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2060Wiki Testing/dods-standards2008-02-07T10:26:49Z<p>Yuan: /* Less Essential, But Still Useful Attributes (Class 1) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. <br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for the variable's data. The units attribute should be formatted as per the recommendations in the Unidata [http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. <br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to be multiplied by this factor after the data are read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to be added to the data after it is read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified. If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation function used to convert the raw data into the units specifed in the <font color='green'>units</font> attribute. <br />
<br />
<br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote> NOTE: A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</cite>] .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<br />
; <font color='green'>missing_value</font> : This is a conventional name for a missing value that will not be treated in any special way by the client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in that it describes data that isn't there, but shouldn't have been, either. That is, where a missing value might be used to fill in for a sensor malfunction, a null value is used to indicate that no data was taken. A dataset that contained random data interpolated onto a grid might use a null value on those grid points too distant from data values to make an accurate estimate.<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of missing data. In this case, data would never have been at those points. Land points in gridded sea-surface temperature data would be default values, as would the end of profile data vectors filled to uniform length. This differs some from the semantics of the COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the DODS project from adopting the COARDS name as a synonym.<br />
<br />
<br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from a list of DODS names (see \appref{std-names}). Including this attribute is a way to ensure that two datasets can be usefully compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to identify the dataset as conforming to the DODS data standard, identified here. The attribute should be a string with the value "DODS". Note that this is a global attribute, and should appear in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2058Wiki Testing/dods-standards2008-02-07T10:26:09Z<p>Yuan: /* Utterly Essential Attributes (Class 0) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. <br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for the variable's data. The units attribute should be formatted as per the recommendations in the Unidata [http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. <br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to be multiplied by this factor after the data are read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to be added to the data after it is read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified. If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation function used to convert the raw data into the units specifed in the <font color='green'>units</font> attribute. <br />
<br />
<br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote> NOTE: A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</cite>] .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<br />
; <font color='green'>missing_value</font> : This is a conventional name for a missing value that will not be treated in any special way by the client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in that it describes data that isn't there, but shouldn't have been, either. That is, where a missing value might be used to fill in for a sensor malfunction, a null value is used to indicate that no data was taken. A dataset that contained random data interpolated onto a grid might use a null value on those grid points too distant from data values to make an accurate estimate.<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of missing data. In this case, data would never have been at those points. Land points in gridded sea-surface temperature data would be default values, as would the end of profile data vectors filled to uniform length. This differs some from the semantics of the COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the DODS project from adopting the COARDS name as a synonym.<br />
<br />
<br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2057Wiki Testing/dods-standards2008-02-07T10:25:51Z<p>Yuan: /* Utterly Essential Attributes (Class 0) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. <br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for the variable's data. The units attribute should be formatted as per the recommendations in the Unidata [http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. <br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to be multiplied by this factor after the data are read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to be added to the data after it is read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified. If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation function used to convert the raw data into the units specifed in the <font color='green'>units</font> attribute. <br />
<br />
<br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote> NOTE: A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</ite>] .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<br />
; <font color='green'>missing_value</font> : This is a conventional name for a missing value that will not be treated in any special way by the client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in that it describes data that isn't there, but shouldn't have been, either. That is, where a missing value might be used to fill in for a sensor malfunction, a null value is used to indicate that no data was taken. A dataset that contained random data interpolated onto a grid might use a null value on those grid points too distant from data values to make an accurate estimate.<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of missing data. In this case, data would never have been at those points. Land points in gridded sea-surface temperature data would be default values, as would the end of profile data vectors filled to uniform length. This differs some from the semantics of the COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the DODS project from adopting the COARDS name as a synonym.<br />
<br />
<br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2056Wiki Testing/dods-standards2008-02-07T10:24:50Z<p>Yuan: /* Utterly Essential Attributes (Class 0) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. <br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for the variable's data. The units attribute should be formatted as per the recommendations in the Unidata [http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. <br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to be multiplied by this factor after the data are read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to be added to the data after it is read by the application that accesses the data. One or both of <font color='green'>scale_factor</font> or <font color='green'>add_offset</font> must be present if the data are not stored in the specified units, unless there is also a <font color='green'>transform</font> specified. If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation function used to convert the raw data into the units specifed in the <font color='green'>units</font> attribute. <br />
<br />
<br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote>NOTE: A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</ite>] .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<br />
; <font color='green'>missing_value</font> : This is a conventional name for a missing value that will not be treated in any special way by the client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in that it describes data that isn't there, but shouldn't have been, either. That is, where a missing value might be used to fill in for a sensor malfunction, a null value is used to indicate that no data was taken. A dataset that contained random data interpolated onto a grid might use a null value on those grid points too distant from data values to make an accurate estimate.<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of missing data. In this case, data would never have been at those points. Land points in gridded sea-surface temperature data would be default values, as would the end of profile data vectors filled to uniform length. This differs some from the semantics of the COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the DODS project from adopting the COARDS name as a synonym.<br />
<br />
<br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2055Wiki Testing/dods-standards2008-02-07T10:21:50Z<p>Yuan: /* Utterly Essential Attributes (Class 0) */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be used for labelling plots, for example. If a variable has no <font color='green'>long_name</font> attribute assigned, the variable name should be used as a default. This corresponds to the "Detailed Variable" field of the GCMD variable naming hierarchy. (\iso{Attribute_Label}{3507}{attrlabl})<br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for<br />
the variable's data. The units attribute should be formatted as per<br />
the recommendations in the Unidata<br />
[http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. (\iso{Attribute_Units_of_Measurement}{3522}{attrunit}).<br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to<br />
be multiplied by this factor after the data are read by the<br />
application that accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to<br />
be added to the data after it is read by the application that<br />
accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are<br />
present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation<br />
function used to convert the raw data into the units specifed in the<br />
<font color='green'>units</font> attribute.\footnote{The format of the function<br />
specification (i.e. Java, Perl, Tcl, Lisp, whatever) is not<br />
currently specified.}<br />
<br />
</blockquote><br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote>A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the \DODSuser .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing<br />
values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<blockquote><br />
; <font color='green'>missing_value</font> : This is a conventional name for a<br />
missing value that will not be treated in any special way by the<br />
client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in<br />
that it describes data that isn't there, but shouldn't have been,<br />
either. That is, where a missing value might be used to fill in for<br />
a sensor malfunction, a null value is used to indicate that no data<br />
was taken. A dataset that contained random data interpolated onto a<br />
grid might use a null value on those grid points too distant from<br />
data values to make an accurate estimate.\footnote{The COARDS<br />
standard does not make this distinction, and would use<br />
<font color='green'>missing_value</font> for these points.}<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of<br />
missing data. In this case, data would never have been at those<br />
points. Land points in gridded sea-surface temperature data would<br />
be default values, as would the end of profile data vectors filled<br />
to uniform length. This differs some from the semantics of the<br />
COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the<br />
DODS project from adopting the COARDS name as a synonym.<br />
<br />
</blockquote><br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2054Wiki Testing/dods-standards2008-02-07T10:21:21Z<p>Yuan: /* Essential and Optional Attributes */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs [[Wiki_Testing/dods-standards|(1)]]. To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be<br />
used for labelling plots, for example. If a variable has no<br />
<font color='green'>long_name</font> attribute assigned, the variable name should be<br />
used as a default. This corresponds to the "Detailed Variable"<br />
field of the GCMD variable naming hierarchy.<br />
(\iso{Attribute_Label}{3507}{attrlabl})<br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for<br />
the variable's data. The units attribute should be formatted as per<br />
the recommendations in the Unidata<br />
[http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. (\iso{Attribute_Units_of_Measurement}{3522}{attrunit}).<br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to<br />
be multiplied by this factor after the data are read by the<br />
application that accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to<br />
be added to the data after it is read by the application that<br />
accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are<br />
present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation<br />
function used to convert the raw data into the units specifed in the<br />
<font color='green'>units</font> attribute.\footnote{The format of the function<br />
specification (i.e. Java, Perl, Tcl, Lisp, whatever) is not<br />
currently specified.}<br />
<br />
</blockquote><br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote>A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the \DODSuser .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing<br />
values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<blockquote><br />
; <font color='green'>missing_value</font> : This is a conventional name for a<br />
missing value that will not be treated in any special way by the<br />
client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in<br />
that it describes data that isn't there, but shouldn't have been,<br />
either. That is, where a missing value might be used to fill in for<br />
a sensor malfunction, a null value is used to indicate that no data<br />
was taken. A dataset that contained random data interpolated onto a<br />
grid might use a null value on those grid points too distant from<br />
data values to make an accurate estimate.\footnote{The COARDS<br />
standard does not make this distinction, and would use<br />
<font color='green'>missing_value</font> for these points.}<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of<br />
missing data. In this case, data would never have been at those<br />
points. Land points in gridded sea-surface temperature data would<br />
be default values, as would the end of profile data vectors filled<br />
to uniform length. This differs some from the semantics of the<br />
COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the<br />
DODS project from adopting the COARDS name as a synonym.<br />
<br />
</blockquote><br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/dods-standards&diff=2053Wiki Testing/dods-standards2008-02-07T10:20:01Z<p>Yuan: /* The Emerging DODS Data Standards */</p>
<hr />
<div>=The Emerging DODS Data Standards=<br />
<br />
<br />
Using DODS, a scientist or data center can make their data readily<br />
available to anyone who knows about it. But there is a difference<br />
between "readily available" and "readily usable." Just because<br />
DODS can easily retrieve data doesn't mean that it is easy to use.<br />
<br />
The DODS project has found that the single biggest obstacle to ease of<br />
use of a dataset's data is incompatibility of metadata, or data<br />
attributes.\footnote{Please see the \DODSuser for a discussion of<br />
<br />
DODS treatment of attributes, or metadata.} This incompatibility<br />
can range from the seemingly simple, such as different data names, to<br />
the far less tractable, such as incompatible time representations.<br />
<br />
To address this problem, the DODS project has created the data<br />
attribute standard described in this chapter. This standard is<br />
''entirely optional'' . A server can still serve data from a DODS<br />
dataset that does not conform to this standard, and a DODS client can<br />
still read that data. However, a dataset that conforms to this<br />
standard will be more easily read and processed by another user.<br />
<br />
<br />
<blockquote>NOTE:The DODS project has an original emphasis on oceanography.<br />
Therefore, the DODS standard described here has been defined, where<br />
possible, to be compatible with existing geo-spatial metadata<br />
conventions, including the COARDS and the ISO Z39.50 GeoProfile<br />
conventions. It should be noted that the DODS standard is a small<br />
subset of these metadata definitions, and may well be applicable to<br />
data from other scientific (or other) disciplines. In addition, the<br />
architecture of the DODS software does not preclude the adoption of<br />
alternate data standards that may be more appropriate to those other<br />
fields.</blockquote><br />
<br />
<br />
In a similar effort, attempting to assert some order over incompatible<br />
data attributes among netCDF files, the COARDS project<br />
[http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html<cite>"Cooperative Ocean/Atmosphere Research Data Service"</cite>] has<br />
specified some standard attributes that shall be defined in a dataset.<br />
Note that this has little or nothing to do with netCDF itself. You<br />
can have a netCDF dataset that doesn't comply with the COARDS<br />
standard. But it will be more useful to others if it does.<br />
<br />
<br />
Note that the DODS architecture allows data attributes to be added to<br />
any dataset ''without'' modifying the data files themselves.<br />
This makes the task of conforming to the data standard fairly simple,<br />
in most cases consisting of adding a few files to a directory.<br />
<br />
<br />
Finally, this is not an exhaustive list of attributes possible.<br />
Beyond syntax-checking, the DODS software does no parsing of attribute<br />
structures for a dataset. You are free to add whatever attributes you<br />
choose to any dataset served by a DODS server. If you want to add<br />
enough data attributes to make your metadata 100\<br />
ISO Z39.50 GeoProfile, there is no reason not to do so.\footnote{We<br />
would suggest that if you do add GeoProfile attributes not mentioned<br />
in this document to your dataset, you should use the SGML attribute<br />
tags as attribute names. In the event that future DODS servers<br />
support a part of the GeoProfile protocol, this will allow your<br />
datasets to be merged seamlessly with that standard.}<br />
<br />
<br />
==Essential and Optional Attributes==<br />
<br />
One of the motivating principles of the DODS project is to keep the<br />
burden on the data provider as light as possible. To further decrease<br />
the burden of making data available, the DODS data standard attributes<br />
are separated into two groups:<br />
<br />
<br />
*Those that are essential to minimal data attribute<br />
interoperability, and<br />
*Those that are very useful, but not essential.<br />
<br />
<br />
The second group can be further split up among the more and less<br />
useful attributes.<br />
<br />
The DODS project recognizes these as different "classes" of<br />
compatibility. For example, a class 0 DODS dataset is one that is<br />
served by DODS, but does not comply with the DODS data standards,<br />
except in the most minimal way. A class 1 dataset conforms, but only<br />
contains the essential attributes. A class 2 datasets conforms and<br />
contains the essential and the class 2 attributes, and so on. The<br />
attributes described in the following sections are identified by their<br />
classes in this way.<br />
<br />
<br />
===Global and Variable Attributes===<br />
<br />
DODS attributes consist of sets of name-value<br />
pairs.\footnote{Actually, there is type information in there, too, so<br />
<br />
it's more accurate to say name-type-value triples, but "triples"<br />
<br />
is more awkward than "pair."} To say that a data variable has<br />
attributes is the same as saying that there is a set of attributes (in<br />
the dataset's DAS) with the same name as this variable. The set can<br />
contain one or more name-value pairs, such as "units" with<br />
"degreesC" and "scale_factor" with "2.35".<br />
<br />
Consider the dataset with a DDS like this:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Int16 temp[time = 16][lat = 17][lon = 21];<br />
<br />
Float32 lat[lat = 17];<br />
<br />
Float32 lon[lon = 21];<br />
<br />
Float32 time[time = 16];<br />
} test1;<br />
</pre><br />
<br />
The DAS for this DODS-compliant dataset might look like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
temp {<br />
<br />
String units "degreesC";<br />
<br />
String long_name "Surface Temperature";<br />
<br />
String missing_value "-32767";<br />
<br />
String scale_factor "0.005";<br />
<br />
String short_name "Temp";<br />
<br />
}<br />
<br />
lat {<br />
<br />
String units "degree North";<br />
<br />
String short_name "Latitude";<br />
<br />
}<br />
<br />
lon {<br />
<br />
String units "degree East";<br />
<br />
String short_name "Longitude";<br />
<br />
}<br />
<br />
time {<br />
<br />
String units "hours from base_time";<br />
<br />
String short_name "Time";<br />
<br />
}<br />
<br />
DODS {<br />
<br />
String conventions "DODS", "COARDS";<br />
<br />
}<br />
}<br />
</pre><br />
<br />
One of the attribute containers in this DAS does not correspond to any<br />
data variables. The attributes in this container are said to be<br />
"global" attributes, and modify the entire dataset. There is<br />
nothing special about the names; an attribute container can be called<br />
anything at all. However, the container called "DODS", if it<br />
exists, should contain the global attributes described in the DODS<br />
data standard, such as <font color='green'>conventions</font>, <font color='green'>Acknowledge</font>, and<br />
<font color='green'>history</font>.<br />
<br />
<br />
==Utterly Essential Attributes (Class 0)==<br />
<br />
The following attributes must appear in any DODS dataset for it to be<br />
considered class 0 compliant. These attributes must be defined for<br />
''each'' data variable in the dataset.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>long_name</font> : A long descriptive name (title). This could be<br />
used for labelling plots, for example. If a variable has no<br />
<font color='green'>long_name</font> attribute assigned, the variable name should be<br />
used as a default. This corresponds to the "Detailed Variable"<br />
field of the GCMD variable naming hierarchy.<br />
(\iso{Attribute_Label}{3507}{attrlabl})<br />
<br />
; <font color='green'>units</font> : A character array that specifies the units used for<br />
the variable's data. The units attribute should be formatted as per<br />
the recommendations in the Unidata<br />
[http://www.unidata.ucar.edu/packages/udunits/ <cite>udunits</cite>] package. (\iso{Attribute_Units_of_Measurement}{3522}{attrunit}).<br />
<br />
; <font color='green'>scale_factor</font> : If present for a variable, the data are to<br />
be multiplied by this factor after the data are read by the<br />
application that accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
<br />
; <font color='green'>add_offset</font> : If present for a variable, this number is to<br />
be added to the data after it is read by the application that<br />
accesses the data. One or both of <font color='green'>scale_factor</font> or<br />
<font color='green'>add_offset</font> must be present if the data are not stored in the<br />
specified units, unless there is also a <font color='green'>transform</font> specified.<br />
If both <font color='green'>scale_factor</font> and <font color='green'>add_offset</font> attributes are<br />
present, the data are first scaled before the offset is added.<br />
<br />
; <font color='green'>transform</font> : This is a string containing a transformation<br />
function used to convert the raw data into the units specifed in the<br />
<font color='green'>units</font> attribute.\footnote{The format of the function<br />
specification (i.e. Java, Perl, Tcl, Lisp, whatever) is not<br />
currently specified.}<br />
<br />
</blockquote><br />
<br />
There are some special cases, outlined in the next two sections.<br />
However, if your dataset contains no null values, and no data stored<br />
as DODS Arrays, the above list is complete.<br />
<br />
<blockquote>A DODS Array is a different data type than a Grid, and contains<br />
less information about its independent variables. For more<br />
information, see below, or refer to the \DODSuser .</blockquote><br />
<br />
<br />
===Missing Data===<br />
<br />
If a dataset contains missing data flagged with special values, those<br />
values must be specified in the attribute list of the variable. That<br />
is, if you have a data sequence called <font color='green'>ralph</font>, and it contains missing<br />
values flagged with -999, the DAS for the dataset should have an<br />
attribute container like this:<br />
<br />
<pre><br />
Attributes {<br />
<br />
...<br />
<br />
ralph {<br />
<br />
Float32 missing_value -999.0;<br />
<br />
...<br />
<br />
}<br />
<br />
...<br />
}<br />
</pre><br />
<br />
The DODS metadata standard provides three categories of missing data.<br />
These attribute values are essential only in the sense that they must<br />
be in the data variable's attribute container ''if you use them'' .<br />
If your data doesn't use these values---that is, if there are no<br />
missing values flagged with special numeric values--these attributes<br />
need not be specified.<br />
<br />
<blockquote><br />
; <font color='green'>missing_value</font> : This is a conventional name for a<br />
missing value that will not be treated in any special way by the<br />
client application. This attribute is part of the COARDS standard.<br />
<br />
; <font color='green'>null_value</font> : A null value differs from a missing value in<br />
that it describes data that isn't there, but shouldn't have been,<br />
either. That is, where a missing value might be used to fill in for<br />
a sensor malfunction, a null value is used to indicate that no data<br />
was taken. A dataset that contained random data interpolated onto a<br />
grid might use a null value on those grid points too distant from<br />
data values to make an accurate estimate.\footnote{The COARDS<br />
standard does not make this distinction, and would use<br />
<font color='green'>missing_value</font> for these points.}<br />
<br />
; <font color='green'>default_value</font> : A default value is yet another sort of<br />
missing data. In this case, data would never have been at those<br />
points. Land points in gridded sea-surface temperature data would<br />
be default values, as would the end of profile data vectors filled<br />
to uniform length. This differs some from the semantics of the<br />
COARDS <font color='green'>_FillValue</font>, but perhaps not so far as to prevent the<br />
DODS project from adopting the COARDS name as a synonym.<br />
<br />
</blockquote><br />
<br />
<br />
===Array Data===<br />
<br />
A dataset containing only an array may be missing some important<br />
information about the dataset's independent variables. To make the<br />
dataset conpliant with the Class 0 of the DODS standard, this<br />
information must be included in the attribute list.<br />
<br />
The information missing from an Array variable is the location of that<br />
array's corners---the <font color='green'>min</font> and <font color='green'>max</font> for each dimension---and<br />
other information about the array dimensions. The requirements,<br />
therefore, are that an Array's dimensions be named, and that attribute<br />
containers with those same names be contained in the dataset DAS.<br />
That is, for a dataset with the following array data:<br />
<br />
<pre><br />
Dataset {<br />
<br />
Array {<br />
<br />
Byte dsp_band_1[lat = 1024][lon = 1024];<br />
<br />
} dsp_band_1;<br />
</pre><br />
<br />
The DAS should look something like the following:<br />
<br />
<pre><br />
Attributes {<br />
<br />
dsp_band_1 {<br />
<br />
String long_name "AVHRR sea surface temperature data";<br />
<br />
String units "DegreesC";<br />
<br />
Float32 scale_factor 0.15625;<br />
<br />
Float32 add_offset 5.0;<br />
<br />
Float32 missing_value -999.0;<br />
<br />
}<br />
<br />
lat {<br />
<br />
String long_name "Latitude";<br />
<br />
String units "Degrees North";<br />
<br />
Float32 min 0.0;<br />
<br />
Float32 max 70.0;<br />
<br />
}<br />
<br />
lon {<br />
<br />
String long_name "Longitude";<br />
<br />
String units "Degrees East";<br />
<br />
Float32 min -100.0;<br />
<br />
Float32 max 0.0;<br />
<br />
}<br />
}<br />
</pre><br />
<br />
If using the name of the dimension would cause a name collision with<br />
some other variable in the dataset, you can use the name of the<br />
dimension prefixed by the name of the array. That is, in the above<br />
example, the attribute containers for the array dimensions would be<br />
<font color='green'>dsp_band_1.lat</font> and <font color='green'>dsp_band_1.lon</font>.<br />
<br />
<br />
==Less Essential, But Still Useful Attributes (Class 1)==<br />
<br />
The optional attributes are divided into two different classes. This<br />
again allows a data provider some latitude in the class of compliance<br />
with the standard.<br />
<br />
<blockquote><br />
; <font color='green'>short_name</font> : This is the name of the variable, taken from<br />
a list of DODS names (see \appref{std-names}). Including this<br />
attribute is a way to ensure that two datasets can be usefully<br />
compared with one another on a variable-by-variable basis.<br />
<br />
; <font color='green'>Convention</font> : This global attribute is recommended to<br />
identify the dataset as conforming to the DODS data standard,<br />
identified here. The attribute should be a string with the value<br />
"DODS". Note that this is a global attribute, and should appear<br />
in the "DODS" attribute container, like this:<br />
<br />
<pre><br />
DODS {<br />
<br />
String Convention "DODS"; }<br />
</pre><br />
<br />
Note that the presence of the "DODS" attribute container is itself<br />
a clue to whether the dataset is DODS-compliant or not.<br />
To be compliant with the<br />
GeoProfile, one element in the Convention vector should read<br />
"<font color='green'>FGDC Content Standards for Digital Geospatial Metadata</font>".<br />
<br />
</blockquote><br />
<br />
==Very Useful Attributes, But Not Essential (Class 2)==<br />
<br />
This is a set of attributes that has been found to be quite useful in<br />
the use of DODS datasets. A dataset can be considered to be class 2<br />
compliant if it contains more than one of these attributes.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>Acknowledge</font> : This string should contain an acknowledgement<br />
paragraph that can appear in papers that use the data from this<br />
dataset. For example: "The principal investigators in the<br />
production of these data are J.D. Elms, S.D. Woodruff, and<br />
S. Worley (http://www.cdc.noaa.gov/coads/participants.html)." and<br />
so on. This is a global attribute, and should appear in the<br />
"DODS" attribute container.<br />
<br />
; <font color='green'>history</font> : The <font color='green'>history</font> attribute is recommended to<br />
record the evolution of the data contained within a DODS data file.<br />
Applications which process this data can append their information to<br />
this attribute. This is a global attribute, and should appear in<br />
the "DODS" attribute container.<br />
<br />
; <font color='green'>Data_Use_Policy</font> : If there are any restrictions on the use<br />
of the data, they should be noted in the string with this name.<br />
This is a global attribute, and should appear in the "DODS"<br />
attribute container.<br />
<br />
; <font color='green'>Theme</font> : A string containing one or more of the GCMD parameter<br />
valid names. See \xlink{the "GCMD Parameters" list maintained at GCMD.} {http://gcmd.gsfc.nasa.gov/cgi-bin/md/valids_display.pl}<br />
<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/intro&diff=2052Wiki Testing/intro2008-02-07T10:17:15Z<p>Yuan: /* Server Interface */</p>
<hr />
<div>=DODS and Data Standards=<br />
<br />
<br />
Once upon a time there was DODS. And it was good. DODS solved many<br />
problems of dataset incompatibility by making remote datasets pretend<br />
to be in a format they were not in. This was tricky, and it was<br />
largely accomplished by hiding from the user the actual data storage<br />
format, and using many different converters to provide data in a<br />
standard transmission format.<br />
<br />
However, data storage format isn't everything. Though the DODS<br />
solution is quite a useful one, it has its limitations. Two datasets<br />
can be incompatible even though they share an identical data storage<br />
format. For example, consider two sea surface temperature datasets<br />
stored in netCDF format files. One can store its temperature in<br />
degrees Celsius, and another can store temperature as raw data, in<br />
unscaled satellite counts, that have to be converted into temperature<br />
values meaningful to a human.<br />
<br />
<br />
==Levels of Compatibility==<br />
<br />
<br />
In discussions about whether two different datasets are compatible, we<br />
must distinguish between different levels of compatibility. What<br />
level is important depends to some extent on how you intend to use the<br />
data. To a human, for example, two datasets could be considered<br />
compatible if they are about the same data variable. Two temperature<br />
datasets that cover the same area might be considered quite compatible,<br />
if all one is doing is looking at two different color contour maps<br />
lying on a table, or side-by-side on a computer screen.<br />
<br />
However if one is seeking datasets on which to execute various<br />
analyses and computations, a higher level of compatibility is<br />
required.<br />
<br />
The issues of data compatibility can be separated into four<br />
categories:<br />
<br />
<br />
*Data storage format<br />
*Data attributes<br />
*Data organization<br />
*Server interface<br />
<br />
<br />
===Data Storage Format===<br />
<br />
<br />
Two datasets whose data are stored and retrieved with different data<br />
format software are incompatible. A dataset stored with netCDF<br />
software cannot be read by a program to read HDF files, and vice<br />
versa.<br />
<br />
Data storage format was the first compatibility issue<br />
that DODS addressed, and it does so fairly successfully. Whatever<br />
storage format data are kept in, the DODS server translates them into<br />
a standard transmission format before sending them to the DODS client<br />
requesting that data. This translation happens file by file, however,<br />
so datasets that span multiple files may remain somewhat problematic,<br />
even though the file format itself is not an issue.<br />
<br />
<br />
===Data Attributes===<br />
<br />
Having solved the problem of data storage compatibility, DODS was<br />
faced with the problem of incompatible data attributes. That is,<br />
considering our two temperature datasets, one dataset could refer to<br />
its temperature data in an array called "T", while another could<br />
refer to an array called "Temp". To a human comparing the two<br />
files, the equation between the two might be obvious, but to a<br />
computer it might not. (Even to a human, the facts might not be as<br />
obvious as they seem. For example, "T" could also refer to<br />
"Time")<br />
<br />
Other data attributes that are important to be able to compute with a<br />
set of data include flags for missing data and unit and unit<br />
conversion information. The data standard outlined in<br />
([http://docs.opendap.org/index.php/Wiki_Testing/DataStandard<cite> dods-standard</cite>]) of this document is an attempt to address this issue.<br />
<br />
===Data Organization===<br />
<br />
The organization of a dataset can create another impediment to<br />
compatibility. Consider, for example, two datasets containing<br />
salinity measurements of the ocean at various different depths, and at<br />
different times. Assuming that there is some reason not to put all<br />
the information in a single file, each dataset must be divided among<br />
several files, and this is where certain incompatibilitles can sneak<br />
in.<br />
<br />
<br />
One dataset could store its data in several files, each of which<br />
contains a three-dimensional array---corresponding to latitude,<br />
longitude, and depth---of salinity values, corresponding to a<br />
particular time. But a similar dataset might be arranged as arrays of<br />
latitude, longitude, and time, with several different files<br />
corrsponding to different depths. Both arrangements are, in fact,<br />
common among on-line datasets.<br />
<br />
The DODS Catalog Server, under development, is intended<br />
to address the issue of data organization within a dataset, allowing a<br />
user to query differently organized datasets with the same syntax<br />
query.<br />
<br />
===Server Interface===<br />
<br />
The DODS software does create solutions to the compatibility issues<br />
outlined above. However, in doing so, it introduces a fourth<br />
compatibility issue, which is that of the server interface itself. A<br />
DODS server is programmed to respond to a set of server commands<br />
issued by a DODS client. These commands come in the form of URLs sent<br />
to the server.<br />
<br />
Because the DODS project is a widely distributed project, and because<br />
the datasets available through DODS are---by design---not under<br />
central control, it is not always certain that two given DODS servers<br />
will be running software from the same software release. Similarly,<br />
since there are provisions for making local modifications to server<br />
behavior, it is also not certain that server functions available on<br />
one server are available on another, even when they are both running<br />
at the same software release.<br />
<br />
Addressing these issues is an ongoing preoccupation of the DODS<br />
project and its (growing) user base. The DODS servers also contain<br />
provisions for making this information available to clients. See<br />
[http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>OPeNDAP User Guide</cite>] for more information.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/intro&diff=2051Wiki Testing/intro2008-02-07T10:16:31Z<p>Yuan: /* Data Organization */</p>
<hr />
<div>=DODS and Data Standards=<br />
<br />
<br />
Once upon a time there was DODS. And it was good. DODS solved many<br />
problems of dataset incompatibility by making remote datasets pretend<br />
to be in a format they were not in. This was tricky, and it was<br />
largely accomplished by hiding from the user the actual data storage<br />
format, and using many different converters to provide data in a<br />
standard transmission format.<br />
<br />
However, data storage format isn't everything. Though the DODS<br />
solution is quite a useful one, it has its limitations. Two datasets<br />
can be incompatible even though they share an identical data storage<br />
format. For example, consider two sea surface temperature datasets<br />
stored in netCDF format files. One can store its temperature in<br />
degrees Celsius, and another can store temperature as raw data, in<br />
unscaled satellite counts, that have to be converted into temperature<br />
values meaningful to a human.<br />
<br />
<br />
==Levels of Compatibility==<br />
<br />
<br />
In discussions about whether two different datasets are compatible, we<br />
must distinguish between different levels of compatibility. What<br />
level is important depends to some extent on how you intend to use the<br />
data. To a human, for example, two datasets could be considered<br />
compatible if they are about the same data variable. Two temperature<br />
datasets that cover the same area might be considered quite compatible,<br />
if all one is doing is looking at two different color contour maps<br />
lying on a table, or side-by-side on a computer screen.<br />
<br />
However if one is seeking datasets on which to execute various<br />
analyses and computations, a higher level of compatibility is<br />
required.<br />
<br />
The issues of data compatibility can be separated into four<br />
categories:<br />
<br />
<br />
*Data storage format<br />
*Data attributes<br />
*Data organization<br />
*Server interface<br />
<br />
<br />
===Data Storage Format===<br />
<br />
<br />
Two datasets whose data are stored and retrieved with different data<br />
format software are incompatible. A dataset stored with netCDF<br />
software cannot be read by a program to read HDF files, and vice<br />
versa.<br />
<br />
Data storage format was the first compatibility issue<br />
that DODS addressed, and it does so fairly successfully. Whatever<br />
storage format data are kept in, the DODS server translates them into<br />
a standard transmission format before sending them to the DODS client<br />
requesting that data. This translation happens file by file, however,<br />
so datasets that span multiple files may remain somewhat problematic,<br />
even though the file format itself is not an issue.<br />
<br />
<br />
===Data Attributes===<br />
<br />
Having solved the problem of data storage compatibility, DODS was<br />
faced with the problem of incompatible data attributes. That is,<br />
considering our two temperature datasets, one dataset could refer to<br />
its temperature data in an array called "T", while another could<br />
refer to an array called "Temp". To a human comparing the two<br />
files, the equation between the two might be obvious, but to a<br />
computer it might not. (Even to a human, the facts might not be as<br />
obvious as they seem. For example, "T" could also refer to<br />
"Time")<br />
<br />
Other data attributes that are important to be able to compute with a<br />
set of data include flags for missing data and unit and unit<br />
conversion information. The data standard outlined in<br />
([http://docs.opendap.org/index.php/Wiki_Testing/DataStandard<cite> dods-standard</cite>]) of this document is an attempt to address this issue.<br />
<br />
===Data Organization===<br />
<br />
The organization of a dataset can create another impediment to<br />
compatibility. Consider, for example, two datasets containing<br />
salinity measurements of the ocean at various different depths, and at<br />
different times. Assuming that there is some reason not to put all<br />
the information in a single file, each dataset must be divided among<br />
several files, and this is where certain incompatibilitles can sneak<br />
in.<br />
<br />
<br />
One dataset could store its data in several files, each of which<br />
contains a three-dimensional array---corresponding to latitude,<br />
longitude, and depth---of salinity values, corresponding to a<br />
particular time. But a similar dataset might be arranged as arrays of<br />
latitude, longitude, and time, with several different files<br />
corrsponding to different depths. Both arrangements are, in fact,<br />
common among on-line datasets.<br />
<br />
The DODS Catalog Server, under development, is intended<br />
to address the issue of data organization within a dataset, allowing a<br />
user to query differently organized datasets with the same syntax<br />
query.<br />
<br />
===Server Interface===<br />
<br />
The DODS software does create solutions to the compatibility issues<br />
outlined above. However, in doing so, it introduces a fourth<br />
compatibility issue, which is that of the server interface itself. A<br />
DODS server is programmed to respond to a set of server commands<br />
issued by a DODS client. These commands come in the form of URLs sent<br />
to the server.<br />
<br />
Because the DODS project is a widely distributed project, and because<br />
the datasets available through DODS are---by design---not under<br />
central control, it is not always certain that two given DODS servers<br />
will be running software from the same software release. Similarly,<br />
since there are provisions for making local modifications to server<br />
behavior, it is also not certain that server functions available on<br />
one server are available on another, even when they are both running<br />
at the same software release.<br />
<br />
Addressing these issues is an ongoing preoccupation of the DODS<br />
project and its (growing) user base. The DODS servers also contain<br />
provisions for making this information available to clients. See<br />
\DODSuser for more information.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/intro&diff=2050Wiki Testing/intro2008-02-07T10:16:10Z<p>Yuan: /* Data Attributes */</p>
<hr />
<div>=DODS and Data Standards=<br />
<br />
<br />
Once upon a time there was DODS. And it was good. DODS solved many<br />
problems of dataset incompatibility by making remote datasets pretend<br />
to be in a format they were not in. This was tricky, and it was<br />
largely accomplished by hiding from the user the actual data storage<br />
format, and using many different converters to provide data in a<br />
standard transmission format.<br />
<br />
However, data storage format isn't everything. Though the DODS<br />
solution is quite a useful one, it has its limitations. Two datasets<br />
can be incompatible even though they share an identical data storage<br />
format. For example, consider two sea surface temperature datasets<br />
stored in netCDF format files. One can store its temperature in<br />
degrees Celsius, and another can store temperature as raw data, in<br />
unscaled satellite counts, that have to be converted into temperature<br />
values meaningful to a human.<br />
<br />
<br />
==Levels of Compatibility==<br />
<br />
<br />
In discussions about whether two different datasets are compatible, we<br />
must distinguish between different levels of compatibility. What<br />
level is important depends to some extent on how you intend to use the<br />
data. To a human, for example, two datasets could be considered<br />
compatible if they are about the same data variable. Two temperature<br />
datasets that cover the same area might be considered quite compatible,<br />
if all one is doing is looking at two different color contour maps<br />
lying on a table, or side-by-side on a computer screen.<br />
<br />
However if one is seeking datasets on which to execute various<br />
analyses and computations, a higher level of compatibility is<br />
required.<br />
<br />
The issues of data compatibility can be separated into four<br />
categories:<br />
<br />
<br />
*Data storage format<br />
*Data attributes<br />
*Data organization<br />
*Server interface<br />
<br />
<br />
===Data Storage Format===<br />
<br />
<br />
Two datasets whose data are stored and retrieved with different data<br />
format software are incompatible. A dataset stored with netCDF<br />
software cannot be read by a program to read HDF files, and vice<br />
versa.<br />
<br />
Data storage format was the first compatibility issue<br />
that DODS addressed, and it does so fairly successfully. Whatever<br />
storage format data are kept in, the DODS server translates them into<br />
a standard transmission format before sending them to the DODS client<br />
requesting that data. This translation happens file by file, however,<br />
so datasets that span multiple files may remain somewhat problematic,<br />
even though the file format itself is not an issue.<br />
<br />
<br />
===Data Attributes===<br />
<br />
Having solved the problem of data storage compatibility, DODS was<br />
faced with the problem of incompatible data attributes. That is,<br />
considering our two temperature datasets, one dataset could refer to<br />
its temperature data in an array called "T", while another could<br />
refer to an array called "Temp". To a human comparing the two<br />
files, the equation between the two might be obvious, but to a<br />
computer it might not. (Even to a human, the facts might not be as<br />
obvious as they seem. For example, "T" could also refer to<br />
"Time")<br />
<br />
Other data attributes that are important to be able to compute with a<br />
set of data include flags for missing data and unit and unit<br />
conversion information. The data standard outlined in<br />
([http://docs.opendap.org/index.php/Wiki_Testing/DataStandard<cite> dods-standard</cite>]) of this document is an attempt to address this issue.<br />
<br />
===Data Organization===<br />
<br />
The organization of a dataset can create another impediment to<br />
compatibility. Consider, for example, two datasets containing<br />
salinity measurements of the ocean at various different depths, and at<br />
different times. Assuming that there is some reason not to put all<br />
the information in a single file, each dataset must be divided among<br />
several files, and this is where certain incompatibilitles can sneak<br />
in.<br />
<br />
<br />
One dataset could store its data in several files, each of which<br />
contains a three-dimensional array---corresponding to latitude,<br />
longitude, and depth---of salinity values, corresponding to a<br />
particular time. But a similar dataset might be arranged as arrays of<br />
latitude, longitude, and time, with several different files<br />
corrsponding to different depths. Both arrangements are, in fact,<br />
common among on-line datasets.<br />
<br />
The DODS Catalog Server, under development (as of \today), is intended<br />
to address the issue of data organization within a dataset, allowing a<br />
user to query differently organized datasets with the same syntax<br />
query.<br />
<br />
<br />
===Server Interface===<br />
<br />
The DODS software does create solutions to the compatibility issues<br />
outlined above. However, in doing so, it introduces a fourth<br />
compatibility issue, which is that of the server interface itself. A<br />
DODS server is programmed to respond to a set of server commands<br />
issued by a DODS client. These commands come in the form of URLs sent<br />
to the server.<br />
<br />
Because the DODS project is a widely distributed project, and because<br />
the datasets available through DODS are---by design---not under<br />
central control, it is not always certain that two given DODS servers<br />
will be running software from the same software release. Similarly,<br />
since there are provisions for making local modifications to server<br />
behavior, it is also not certain that server functions available on<br />
one server are available on another, even when they are both running<br />
at the same software release.<br />
<br />
Addressing these issues is an ongoing preoccupation of the DODS<br />
project and its (growing) user base. The DODS servers also contain<br />
provisions for making this information available to clients. See<br />
\DODSuser for more information.</div>Yuanhttps://docs.opendap.org/index.php?title=The_FreeForm_Data_Handler&diff=2047The FreeForm Data Handler2008-02-07T09:55:43Z<p>Yuan: /* Tasks Illustrated in this Guide */</p>
<hr />
<div>=Preface=<br />
<br />
<br />
This document describes the OPeNDAP FreeForm ND Data Handler, which<br />
can be used with the OPeNDAP data server. It is not a<br />
complete description of the FreeForm ND software. For that, please refer<br />
to the ND manual.<br />
<br />
This document contains much material originally written at the<br />
National Oceanic and Atmospheric Administration's National<br />
Environmental Satellite, Data, and Information Service, which is part<br />
of the National Geophysical Data Center in Boulder, Colorado.<br />
<br />
This document has been updated to include information on FreeForm ND,<br />
the last release of FreeForm. FreeForm is now supported only for use<br />
with the Distributed Oceanographic Data System; see the<br />
[http://www.opendap.org/<cite>OPeNDAP</cite>] for more information.<br />
<br />
We are interested in your comments about the OPeNDAP software, and the<br />
FreeForm ND software and this document. Send them to:<br />
[mailto:support@opendap.org<cite>support@opendap.org</cite>].<br />
<br />
<br />
Using FreeForm ND with OPeNDAP, a researcher can easily make his or her data<br />
available to the wider community of OPeNDAP users without having to<br />
convert that data into another data file format. This document<br />
presents the FreeForm ND software, and shows how to use it with the OPeNDAP server.<br />
<br />
<br />
==Tasks Illustrated in this Guide==<br />
<br />
<br />
For a quick start to getting, installing, and using the FreeForm ND<br />
software, see the list below of tasks described in this document.<br />
<br />
<br />
*Quick start. ([http://docs.opendap.org/index.php/Wiki_Testing/dquick<cite>here</cite>])<br />
*Getting and installing the FreeForm ND software. ([http://docs.opendap.org/index.php/Wiki_Testing/dintro#Installing_the_OPeNDAP_FreeForm_ND_Data_Handler<cite>here</cite>])<br />
*Serving tabular data. ([http://docs.opendap.org/index.php/Wiki_Testing/tblfmt<cite>here</cite>])<br />
*Array tabular data. ([http://docs.opendap.org/index.php/Wiki_Testing/arrayfmt<cite>here</cite>])<br />
*Dealing with data file headers ([http://docs.opendap.org/index.php/Wiki_Testing/hdrfmts<cite>here</cite>])<br />
*Setup of File servers ([http://docs.opendap.org/index.php/Wiki_Testing/fileserv<cite>here</cite>])<br />
<br />
==Who is this Guide for?==<br />
<br />
<br />
This guide is for people who wish to use FreeForm ND to serve scientific<br />
datasets using the OPeNDAP software. Scientists who wish to share their<br />
data with colleagues may also find this a useful system, since it is a<br />
relatively simple matter to set up a server that can allow remote<br />
access to your data.<br />
<br />
<br />
This documentation assumes that the readers are familiar with<br />
computers and the internet, but are not necessarily programmers. More<br />
than a passing familiarity with different data file formats will be<br />
useful, as will an understanding of elementary internet concepts, such<br />
as URLs and http.<br />
<br />
<br />
This manual also assumes some familiarity with the OPeNDAP software. If<br />
you are starting from scratch, knowing nothing at all about OPeNDAP, we<br />
strongly encourage you to browse the [http://docs.opendap.org/index.php/Wiki_Testing/OpeNDAP_User%27s_Guide<cite>The OPeNDAP User Guide</cite>] before reading too far<br />
here.<br />
<br />
<br />
==Organization of this Document==<br />
<br />
This book contains both introductory and reference material. There is<br />
also a description of the installation procedure.<br />
<br />
<blockquote><br />
<br />
; [[Wiki_Testing/dintro | Chapter 1]] : contains an overview of the OPeNDAP FreeForm ND Data Handler software, including how to get it and install it.<br />
<br />
; [[Wiki_Testing/dquick | Chapter 2]] : provides a brief introduction to writing format descriptions and using the OPeNDAP FreeForm ND Data Handler.<br />
<br />
; [[Wiki_Testing/tblfmt | Chapter 3]] :provides detailed information about writing format descriptions to facilitate access to data in tabular formats.<br />
<br />
; [[Wiki_Testing/arrayfmt | Chapter 4]] : provides detailed information about writing format descriptions to facilitate access to data in non-tabular (array) formats.<br />
<br />
; [[Wiki_Testing/hdrfmts | Chapter 5]] : tells you how to work with header formats.<br />
<br />
; [[Wiki_Testing/ff-server | Chapter 6]] : describes the operation of the OPeNDAP FreeForm ND Data Handler, with tips for writing format files.<br />
<br />
; [[Wiki_Testing/fileserv | Chapter 7]] : describes the OPeNDAP file server.<br />
<br />
; [[Wiki_Testing/convs | Chapter 8]] : presents FreeForm ND file name conventions, the search rules for locating format files, and standard command line arguments for FreeForm ND programs.<br />
<br />
; [[Wiki_Testing/fmtconv | Chapter 9]] : shows you how to use the FreeForm ND program <font color='green'>newform</font> to convert data from one format to another and also how to read the data in a binary file.<br />
<br />
; [[Wiki_Testing/datachk | Chapter 10]] : discusses the FreeForm ND program <font color='green'>checkvar</font>, which you can use to check data distribution and quality.<br />
<br />
;[[Wiki_Testing/hdfutils | Appnedix A]] : provides explanations for a small selection of tools that will be useful for programmers working with the HDF file format.<br />
<br />
; [[Wiki_Testing/errors | Appendix B]] : presents a list of common FreeForm ND error messages. These are the error messages that may be issued by the FreeForm ND utilities, such as <font color='green'>newform</font>, not the OPeNDAP FreeForm ND Data Handler.<br />
<br />
</blockquote><br />
<br />
<br />
A position box is often used in this book to indicate column position<br />
of field values in data files. It is shown at the beginning of a data<br />
list in the documentation, but does not appear in the data file<br />
itself. It looks something like this:<br />
<br />
<pre><br />
1 2 3 4 5 6<br />
012345678901234567890123456789012345678901234567890<br />
</pre></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/errors&diff=2045Wiki Testing/errors2008-02-06T17:33:16Z<p>Yuan: /* Error Messages */</p>
<hr />
<div>=Error Handling=<br />
<br />
The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error captured, error type and a short message are placed in the message queue. If a fatal error occurs, the program stops executing and displays all error messages in the queue.<br />
<br />
<br />
==Error Messages==<br />
<br />
The following is a list of some possible error messages with suggestions for corrections.<br />
<br />
; Problem opening, reading, or writing to file : Check that all file names and paths are correct. <br />
; Problem making format : Make sure there is a format file describing the data file formats. Check that input and output format descriptions in the format file accurately describe the data. <br />
; Problem making header format : If a header exists in the data file, it must be described in a format file. Check that the header description accurately describes the header in your data file. <br />
; Problem getting value :<br />
; Problem processing variable list : The data formats may not be described correctly or there may be some inconsistencies in the data. Check also for unprintable characters at the end of the data file. <br />
; File length / Record length mismatch :<br />
; Record Length or CR Problem : This usually happens because the input format description is not correct. Make sure the format description's last position is the last character before the end-of-line character. If you have a header, make sure it is described correctly. The header's length must include all characters up until the last end-of-line-character before the data begins. <br />
; Binary Overflow : Try using a larger output variable type such as a long instead of a short. Be sure you have given enough space for the values to be written. See Chapter for more information. <br />
; Variable not found : The variable names in your output format must match the variable names in the input format unless you are using conversion variables. <br />
; Data Overflow : Data overflow does not usually cause a fatal error and FreeForm ND functions try to anticipate them. If overflow occurs for a particular value, ***'s are written to that value's location. If you find these in your output, check your variable positions and precision. Increase field width or use a "larger" data type. Be sure the output format specifies space for the output variable. For instance, FreeForm ND adds a leading zero in front of decimal points. If the original data did not have a leading zero, the output will have one more digit than the input. <br />
; Insufficient memory allocation : The application has run out of memory. Try using the -b (local buffer size) option, or modify autoexec.bat and config.sys and comment out devices, TSR's, etc.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/errors&diff=2044Wiki Testing/errors2008-02-06T17:32:36Z<p>Yuan: /* Error Messages */</p>
<hr />
<div>=Error Handling=<br />
<br />
The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error captured, error type and a short message are placed in the message queue. If a fatal error occurs, the program stops executing and displays all error messages in the queue.<br />
<br />
<br />
==Error Messages==<br />
<br />
The following is a list of some possible error messages with suggestions for corrections.<br />
<br />
; Problem opening, reading, or writing to file :<br />
Check that all file names and paths are correct. <br />
; Problem making format :<br />
Make sure there is a format file describing the data file formats. Check that input and output format descriptions in the format file accurately describe the data. <br />
; Problem making header format :<br />
If a header exists in the data file, it must be described in a format file. Check that the header description accurately describes the header in your data file. <br />
; Problem getting value :<br />
; Problem processing variable list :<br />
The data formats may not be described correctly or there may be some inconsistencies in the data. Check also for unprintable characters at the end of the data file. <br />
; File length / Record length mismatch :<br />
; Record Length or CR Problem :<br />
This usually happens because the input format description is not correct. Make sure the format description's last position is the last character before the end-of-line character. If you have a header, make sure it is described correctly. The header's length must include all characters up until the last end-of-line-character before the data begins. <br />
; Binary Overflow :<br />
Try using a larger output variable type such as a long instead of a short. Be sure you have given enough space for the values to be written. See Chapter for more information. <br />
; Variable not found :<br />
The variable names in your output format must match the variable names in the input format unless you are using conversion variables. <br />
; Data Overflow : Data overflow does not usually cause a fatal error and FreeForm ND functions try to anticipate them. If overflow occurs for a particular value, ***'s are written to that value's location. If you find these in your output, check your variable positions and precision. Increase field width or use a "larger" data type. Be sure the output format specifies space for the output variable. For instance, FreeForm ND adds a leading zero in front of decimal points. If the original data did not have a leading zero, the output will have one more digit than the input. <br />
; Insufficient memory allocation :<br />
The application has run out of memory. Try using the -b (local buffer size) option, or modify autoexec.bat and config.sys and comment out devices, TSR's, etc.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/errors&diff=2043Wiki Testing/errors2008-02-06T17:32:15Z<p>Yuan: /* Error Messages */</p>
<hr />
<div>=Error Handling=<br />
<br />
The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error captured, error type and a short message are placed in the message queue. If a fatal error occurs, the program stops executing and displays all error messages in the queue.<br />
<br />
<br />
==Error Messages==<br />
<br />
The following is a list of some possible error messages with suggestions for corrections.<br />
<br />
; Problem opening, reading, or writing to file :<br />
Check that all file names and paths are correct. <br />
; Problem making format :<br />
Make sure there is a format file describing the data file formats. Check that input and output format descriptions in the format file accurately describe the data. <br />
; Problem making header format :<br />
If a header exists in the data file, it must be described in a format file. Check that the header description accurately describes the header in your data file. <br />
; Problem getting value :<br />
; Problem processing variable list :<br />
The data formats may not be described correctly or there may be some inconsistencies in the data. Check also for unprintable characters at the end of the data file. <br />
; File length / Record length mismatch :<br />
; Record Length or CR Problem :<br />
This usually happens because the input format description is not correct. Make sure the format description's last position is the last character before the end-of-line character. If you have a header, make sure it is described correctly. The header's length must include all characters up until the last end-of-line-character before the data begins. <br />
; Binary Overflow :<br />
Try using a larger output variable type such as a long instead of a short. Be sure you have given enough space for the values to be written. See Chapter for more information. <br />
; Variable not found :<br />
The variable names in your output format must match the variable names in the input format unless you are using conversion variables. <br />
; Data Overflow :<br />
Data overflow does not usually cause a fatal error and FreeForm ND functions try to anticipate them. If overflow occurs for a particular value, ***'s are written to that value's location. If you find these in your output, check your variable positions and precision. Increase field width or use a "larger" data type. Be sure the output format specifies space for the output variable. For instance, FreeForm ND adds a leading zero in front of decimal points. If the original data did not have a leading zero, the output will have one more digit than the input. <br />
; Insufficient memory allocation :<br />
The application has run out of memory. Try using the -b (local buffer size) option, or modify autoexec.bat and config.sys and comment out devices, TSR's, etc.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/errors&diff=2042Wiki Testing/errors2008-02-06T17:31:53Z<p>Yuan: /* Error Messages */</p>
<hr />
<div>=Error Handling=<br />
<br />
The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error captured, error type and a short message are placed in the message queue. If a fatal error occurs, the program stops executing and displays all error messages in the queue.<br />
<br />
<br />
==Error Messages==<br />
<br />
The following is a list of some possible error messages with suggestions for corrections.<br />
<br />
; Problem opening, reading, or writing to file :<br />
Check that all file names and paths are correct. <br />
; Problem making format :<br />
Make sure there is a format file describing the data file formats. Check that input and output format descriptions in the format file accurately describe the data. <br />
; Problem making header format :<br />
If a header exists in the data file, it must be described in a format file. Check that the header description accurately describes the header in your data file. <br />
; Problem getting value :<br />
; Problem processing variable list :<br />
The data formats may not be described correctly or there may be some inconsistencies in the data. Check also for unprintable characters at the end of the data file. <br />
; File length / Record length mismatch :<br />
; Record Length or CR Problem :<br />
This usually happens because the input format description is not correct. Make sure the format description's last position is the last character before the end-of-line character. If you have a header, make sure it is described correctly. The header's length must include all characters up until the last end-of-line-character before the data begins. <br />
; Binary Overflow :<br />
Try using a larger output variable type such as a long instead of a short. Be sure you have given enough space for the values to be written. See Chapter for more information. <br />
; Variable not found :<br />
The variable names in your output format must match the variable names in the input format unless you are using conversion variables. <br />
; Data Overflow :<br />
******<br />
Data overflow does not usually cause a fatal error and FreeForm ND functions try to anticipate them. If overflow occurs for a particular value, ***'s are written to that value's location. If you find these in your output, check your variable positions and precision. Increase field width or use a "larger" data type. Be sure the output format specifies space for the output variable. For instance, FreeForm ND adds a leading zero in front of decimal points. If the original data did not have a leading zero, the output will have one more digit than the input. <br />
; Insufficient memory allocation :<br />
The application has run out of memory. Try using the -b (local buffer size) option, or modify autoexec.bat and config.sys and comment out devices, TSR's, etc.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/errors&diff=2041Wiki Testing/errors2008-02-06T17:30:13Z<p>Yuan: New page: =Error Handling= The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error capture...</p>
<hr />
<div>=Error Handling=<br />
<br />
The FreeForm ND error handling system captures errors, such as improper usage, code problems, and system errors, and places them in an error queue. For each error captured, error type and a short message are placed in the message queue. If a fatal error occurs, the program stops executing and displays all error messages in the queue.<br />
<br />
<br />
==Error Messages==<br />
<br />
The following is a list of some possible error messages with suggestions for corrections.<br />
<br />
; Problem opening, reading, or writing to file :<br />
Check that all file names and paths are correct. <br />
; Problem making format :<br />
Make sure there is a format file describing the data file formats. Check that input and output format descriptions in the format file accurately describe the data. <br />
Problem making header format<br />
If a header exists in the data file, it must be described in a format file. Check that the header description accurately describes the header in your data file. <br />
Problem getting value<br />
Problem processing variable list<br />
The data formats may not be described correctly or there may be some inconsistencies in the data. Check also for unprintable characters at the end of the data file. <br />
File length / Record length mismatch<br />
Record Length or CR Problem<br />
This usually happens because the input format description is not correct. Make sure the format description's last position is the last character before the end-of-line character. If you have a header, make sure it is described correctly. The header's length must include all characters up until the last end-of-line-character before the data begins. <br />
Binary Overflow<br />
Try using a larger output variable type such as a long instead of a short. Be sure you have given enough space for the values to be written. See Chapter for more information. <br />
Variable not found<br />
The variable names in your output format must match the variable names in the input format unless you are using conversion variables. <br />
Data Overflow<br />
******<br />
Data overflow does not usually cause a fatal error and FreeForm ND functions try to anticipate them. If overflow occurs for a particular value, ***'s are written to that value's location. If you find these in your output, check your variable positions and precision. Increase field width or use a "larger" data type. Be sure the output format specifies space for the output variable. For instance, FreeForm ND adds a leading zero in front of decimal points. If the original data did not have a leading zero, the output will have one more digit than the input. <br />
Insufficient memory allocation<br />
The application has run out of memory. Try using the -b (local buffer size) option, or modify autoexec.bat and config.sys and comment out devices, TSR's, etc.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/hdfutils&diff=2040Wiki Testing/hdfutils2008-02-06T17:28:00Z<p>Yuan: /* <font color='green'>makehdf</font> */</p>
<hr />
<div>=HDF Utilities=<br />
<br />
<br />
FreeForm ND includes three utilities for use with HDF (hierarchical<br />
data format) files: <font color='green'>makehdf</font>, <font color='green'>splitdat</font>, and <font color='green'>pntshow</font>.<br />
These programs were built using both the FreeForm library and the HDF<br />
library, which was developed at the National Center for Supercomputer<br />
Applications (NCSA).<br />
<br />
The <font color='green'>makehdf</font> program converts binary and ASCII data files to HDF<br />
files and converts multiplexed band interleaved by pixel image files<br />
into a series of single parameter files. The <font color='green'>splitdat</font> program is<br />
used to separate and reformat data files containing headers and data<br />
into separate header and data files, or to translate them into HDF<br />
files. The <font color='green'>pntshow</font> program extracts point data from HDF files<br />
into binary or ASCII format.<br />
<br />
It is assumed in this chapter that you have a working familiarity with<br />
HDF terminology and conventions. See the HDF user documentation for<br />
detailed information.<br />
<br />
<blockquote>NOTE: Do not try the examples in this chapter. The example file set is<br />
incomplete. </blockquote><br />
<br />
<br />
==<font color='green'>makehdf</font>==<br />
<br />
<br />
Using <font color='green'>makehdf</font> you can convert data files with formats described<br />
in a FreeForm format file into HDF files. You should follow FreeForm<br />
naming conventions for the data and format files. For details about<br />
FreeForm conventions, see ([[Wiki_Testing/convs | Chapter 8]]).<br />
<br />
<blockquote>A dBASE input file must be converted to ASCII or binary using<br />
<font color='green'>newform</font> before you can run <font color='green'>makehdf</font> on it. </blockquote><br />
<br />
The HDF file resulting from a conversion consists either of a group of<br />
scientific datasets (SDS's), one for each variable in the input data<br />
file, or of a ''vgroup'' containing all the variables as one<br />
''vdata''. If you are working with grid data, you will want SDS's<br />
(the default) in the output HDF file. A vdata (<font color='green'>-vd</font> option) is the<br />
appropriate choice for point data.<br />
<br />
The <font color='green'>makehdf</font> command has the following form:<br />
<br />
<pre><br />
makehdf input_file [-r rows] [-c columns] [-v var_file] [-d HDF_description_file]<br />
<br />
[-xl x_label -yl y_label] [-xu x_units -yu y_units] [-xf x_format -yf y_format] [-id file_id] [-vd [vdata_file]] [-dmx [-sep]] [-df] [-md missing_data_file] [-dof HDF_file] <br />
</pre><br />
<br />
<br />
<br />
; ''input_file'' :<br />
Name of the input data file. Following FreeForm naming conventions,<br />
the standard extensions for data files are <font color='green'>.dat</font> for ASCII<br />
format and <font color='green'>.bin</font> for binary.<br />
<br />
; -r rows :<br />
Option flag followed by the number of rows in each resulting<br />
scientific dataset. The number of rows must be specified through<br />
this option on the command line, or in an equivalence table, or in a<br />
header (<font color='green'>.hdr</font>) file defined according to FreeForm standards.<br />
<br />
; -c columns :<br />
Option flag followed by the number of columns in each resulting<br />
scientific dataset. The number of columns must be specified through<br />
this option on the command line, or in an equivalence table, or in a<br />
header (<font color='green'>.hdr</font>) file defined according to FreeForm standards.<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of the variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by <font color='green'>makehdf</font>. Variable names in \var{var_file} can be<br />
separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
; -d HDF_description_file :<br />
Option flag followed by the name of the file containing a<br />
description of the input file. The description will be stored as a<br />
file annotation in the resulting HDF file.<br />
<br />
; -xl x_label -yl y_label :<br />
Option flags followed by strings (labels) describing the x and y<br />
axes; labels must be in quotes (<font color='green'>" "</font>) if more than one word.<br />
<br />
; -xu x_units -yu y_units :<br />
Option flags followed by strings indicating the measurement units<br />
for the x and y axes; strings must be in quotes (<font color='green'>" "</font>) if more than<br />
one word.<br />
<br />
; - xf x_format -yf y_format :<br />
Option flags followed by strings indicating the formats to be used<br />
in displaying scale for the x and y dimensions; strings must be in<br />
quotes (<font color='green'>" "</font>) if more than one word.<br />
<br />
; - id file_id :<br />
Option flag followed by a string that will be stored as the ID of<br />
the resulting HDF file.<br />
<br />
; - vd [vdata_file] :<br />
Option flag indicating that the output HDF file should contain a<br />
vdata. The optional file name specifies the name of the output HDF<br />
file; the default is <font color='green'>input_file.HDF</font>.<br />
<br />
; - dmx [-sep] :<br />
The option flag <font color='green'>-dmx</font> indicates that input data should be<br />
demultiplexed from band interleaved by pixel to band sequential form<br />
in <font color='green'>input_file.dmx</font>. If <font color='green'>-dmx</font> is followed by <font color='green'>-sep</font>,<br />
the input data are demultiplexed into separate variable files called<br />
<font color='green'>data_file.1</font> \ldots <font color='green'>data_file.n</font><br />
<br />
; - df :<br />
To use this option, the input file (<font color='green'>data_file.ext</font>) must be a<br />
binary demultiplexed (band sequential) file. For each input<br />
variable in the applicable FreeForm format description file, there<br />
is a corresponding demultiplexed section in the output HDF file.<br />
<br />
; - md missing_data_file :<br />
Option flag followed by the name of the file defining missing data<br />
(data you want to exclude). Use this option only along with the<br />
vdata (-vd) option. Each line in the missing data file has the form:<br />
<br />
<pre><br />
variable_name lower_limit upper_limit<br />
</pre><br />
<br />
The precision of the upper and lower limits matches the precision of<br />
the input data.<br />
<br />
; - dof HDF_file :<br />
Option flag followed by the name of the output HDF file. If you do<br />
not use the <font color='green'>-dof</font> option, the default output file name is<br />
<font color='green'>input_file.HDF</font>.<br />
<br />
<br />
<br />
===Example===<br />
<br />
You will use <font color='green'>makehdf</font> to store <font color='green'>latlon.dat</font> as an HDF file.<br />
The HDF file will consist of two SDS's, one each for the two variables<br />
latitude and longitude. Each SDS will have four rows and five columns.<br />
<br />
To convert <font color='green'>latlon.dat</font> to an HDF file, enter the following<br />
command:<br />
<br />
<pre><br />
makehdf latlon.dat -r 4 -c 5<br />
</pre><br />
<br />
As <font color='green'>makehdf</font> translates <font color='green'>latlon.dat</font> into HDF, processing<br />
information is displayed on the screen:<br />
<br />
<pre><br />
1 Caches (1150 bytes) Processed: 800 bytes written to latlon.dmx<br />
Writing latlon.HDF and calculating maxima and minima ...<br />
<br />
Variable latitude:<br />
Minimum: -86.432712 Maximum 89.170904<br />
Variable longitude:<br />
Minimum: -176.161101 Maximum 165.066193<br />
</pre><br />
<br />
The output from <font color='green'>makehdf</font> is an HDF file named <font color='green'>latlon.HDF</font><br />
(by default). It contains the minimum and maximum values for the two<br />
variables as well as the two SDS's.<br />
<br />
A temporary file named <font color='green'>latlon.dmx</font> was also created. It contains<br />
the data from latlon.dat in demultiplexed form . The data was<br />
converted from its original multiplexed form to enable <font color='green'>makehdf</font><br />
to write sections of data to SDS's.<br />
<br />
If you start with a demultiplexed file such as <font color='green'>latlon.dmx</font>, the<br />
translation process is much quicker, particularly for large data<br />
files. As an illustration, try this. Rename <font color='green'>latlon.dmx</font> to<br />
<font color='green'>latlon.bin</font> (renaming is necessary for <font color='green'>makehdf</font> to find the<br />
format description file <font color='green'>latlon.fmt</font> by default). Enter the<br />
following command:<br />
<br />
<pre><br />
makehdf latlon.bin -df -r 4 -c 5<br />
</pre><br />
<br />
The output file again is <font color='green'>latlon.HDF</font>, but notice that no<br />
demultiplexing was done.<br />
<br />
==<font color='green'>splitdat</font>==<br />
<br />
<br />
The <font color='green'>splitdat</font> program translates files with headers and data into<br />
separate header and data files or into HDF files. If the translation<br />
is to separate header and data files, the header file can include<br />
indexing information.<br />
<br />
The combination of header and data records in a file is often used for<br />
point data sets that include a number of observations made at one or<br />
more stations or locations in space. The header records contain<br />
information about the stations or locations of the measurements. The<br />
data records hold the observational data. A station record usually<br />
indicates how many data records follow it. The structure of such a<br />
file is similar to the following:<br />
<br />
<pre><br />
Header for Station 1<br />
Observation 1 for Station 1<br />
Observation 2 for Station 1<br />
<br />
.<br />
<br />
.<br />
Observation N for Station 1<br />
<br />
Header for Station 2<br />
Observation 1 for Station 2<br />
Observation 2 for Station 2<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
Observation N for Station 2<br />
<br />
Header for Station 3<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
</pre><br />
<br />
Many applications have difficulty reading this sort of heterogeneous<br />
data file. One solution is to split the data into two homogeneous<br />
files, one containing the headers, the other containing the data. With<br />
<font color='green'>splitdat</font>, you can easily create the separate data and header<br />
files. To use <font color='green'>splitdat</font> for this purpose, the input and output<br />
formats for the record headers and the data must be described in a<br />
FreeForm format description file. To use <font color='green'>splitdat</font> for<br />
translating files to HDF, the input format must be described in a<br />
FreeForm format description file. You should follow FreeForm naming<br />
conventions for the data and format files. For details about FreeForm<br />
conventions, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
The <font color='green'>splitdat</font> command has the following form:<br />
<br />
\proto{<font color='green'>splitdat</font> \var{input_file} [\var{output_data_file} > \var{output_header_file}]}<br />
<br />
<blockquote><br />
<br />
; \var{input_file} :<br />
Name of the file to be processed. Following FreeForm naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font><br />
for ASCII format and <font color='green'>.bin</font> for binary.<br />
<br />
; \var{output_data_file} :<br />
Name of the output file into which data are transferred with the<br />
format specified in the applicable FreeForm format description file.<br />
The standard extensions are the same as for input files. If an<br />
output file name is not specified, the default is standard output.<br />
<br />
; \var{output_header_file} :<br />
Name of the output file into which headers from the input file are<br />
transferred with the format specified in the applicable FreeForm<br />
format description file. If an output header file name is not<br />
specified, the default is standard output.<br />
</blockquote><br />
<br />
===Index Creation===<br />
<br />
You can use the two variables begin and extent (described below) in<br />
the format description for the output record headers to indicate the<br />
location and size of the data block associated with each record<br />
header. If you then use <font color='green'>splitdat</font>, the header file that results<br />
can be used as an index to the data file.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>begin</font> :<br />
Indicates the offset to the beginning of the data associated with a<br />
particular header. If the data is being translated to HDF, the units<br />
are records; if not, the units are bytes.<br />
<br />
; <font color='green'>extent</font> :<br />
Indicates the number of records (HDF) or bytes (non-HDF) associated<br />
with each header record.<br />
</blockquote><br />
<br />
<br />
====Example====<br />
<br />
You will use <font color='green'>splitdat</font> to extract the headers and data from a<br />
rawinsonde (a device for gathering meteorological data) ASCII data<br />
file named <font color='green'>hara.dat</font> (HARA = Historic Arctic Rawinsonde Archive)<br />
and create two output files-<font color='green'>23338.dat</font> containing the ASCII data<br />
and <font color='green'>23338hdr.dat</font> containing the ASCII headers. The format<br />
description file <font color='green'>hara.fmt</font> should contain the necessary format<br />
descriptions.<br />
<br />
Here is <font color='green'>hara.fmt</font>:<br />
<br />
<pre><br />
ASCII_input_record_header "ASCII Location Record input format"<br />
WMO_station_ID_number 1 5 char 0<br />
latitude 6 10 long 2<br />
longitude_east 11 15 long 2<br />
year 17 18 uchar 0<br />
month 19 20 uchar 0<br />
day 21 22 uchar 0<br />
hour 23 24 uchar 0<br />
flag_processing_1 28 28 char 0<br />
flag_processing_2 29 29 char 0<br />
flag_processing_3 30 30 char 0<br />
station_type 31 31 char 0<br />
sea_level_elev 32 36 long 0<br />
instrument_type 37 38 uchar 0<br />
number_of_observations 40 42 ushort 0<br />
identification_code 44 44 char 0<br />
<br />
ASCII_input_data "Historical Arctic Rawinsonde Archive input format"<br />
atmospheric_pressure 1 5 long 1<br />
geopotential_height 7 11 long 0<br />
temperature_deg 13 16 short 0<br />
dewpoint_depression 18 20 short 0<br />
wind_direction 22 24 short 0<br />
wind_speed_m/s 26 28 short 0<br />
flag_qg 30 30 char 0<br />
flag_qg1 31 31 char 0<br />
flag_qt 33 33 char 0<br />
flag_qt1 34 34 char 0<br />
flag_qd 36 36 char 0<br />
flag_qd1 37 37 char 0<br />
flag_qw 39 39 char 0<br />
flag_qw1 40 40 char 0<br />
flag_qp 42 42 char 0<br />
flag_levck 43 43 char 0<br />
<br />
ASCII_output_record_header "ASCII Location Record output format"<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
<br />
ASCII_output_data "Historical Arctic Rawinsonde Archive output format"<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
</pre><br />
<br />
To "split" <font color='green'>hara.dat</font>, enter the following command:<br />
<br />
<pre><br />
splitdat hara.dat 23338.dat > 23338hdr.dat<br />
</pre><br />
<br />
The data values from <font color='green'>hara.dat</font> are stored in <font color='green'>23338.dat</font> and<br />
the headers in <font color='green'>23338hdr.dat</font>.<br />
<br />
Because the variables begin and extent were used in the header output<br />
format in <font color='green'>hara.fmt</font> to indicate data offset and number of<br />
records, <font color='green'>23338hdr.dat</font> has two columns of data showing offset and<br />
extent. Thus, it can serve as an index into <font color='green'>23338.dat</font>.<br />
<br />
===HDF Translation===<br />
<br />
If output files are not specified on the <font color='green'>splitdat</font> command line,<br />
a file named <font color='green'>input_file.HDF</font> is created. It is hierarchically<br />
named and organized as follows:<br />
<br />
<pre><br />
<br />
vgroup<br />
<br />
input_file_name<br />
<br />
/ \<br />
<br />
/ \<br />
<br />
vdata1 vdata2<br />
"PointIndex" "input_file_name"<br />
</pre><br />
<br />
<br />
*<font color='green'>vdata1</font> contains the record headers<br />
*<font color='green'>vdata2</font> contains the data<br />
*If writing to a Vset (represented by a vgroup), both output<br />
formats are converted to binary, if not binary already.<br />
<br />
====Example====<br />
<br />
To create the file <font color='green'>hara.HDF</font> from <font color='green'>hara.dat</font>, enter the<br />
following abbreviated command:<br />
<br />
<pre><br />
splitdat hara.dat<br />
</pre><br />
<br />
The output formats in <font color='green'>hara.fmt</font> are automatically converted to<br />
binary, and subsequently the ASCII data in <font color='green'>hara.dat</font> are also<br />
converted to binary for HDF storage.<br />
<br />
==<font color='green'>pntshow</font>==<br />
<br />
<br />
The <font color='green'>pntshow</font> program is a versatile tool for extracting point<br />
data from HDF files containing scientific datasets and Vsets. The<br />
extraction can be done into any binary or ASCII format described in a<br />
FreeForm format description file. Before using <font color='green'>pntshow</font> on an HDF<br />
file, you should pack the file using the NCSA-developed HDF utility<br />
hdfpack.<br />
<br />
You can use <font color='green'>pntshow</font> to extract headers and data from an HDF file<br />
into separate files or to extract just the data. It's a good idea to<br />
define GeoVu keywords in an equivalence table to facilitate access to<br />
HDF objects. For information about equivalence tables, see the GeoVu<br />
Tools Reference Guide. The input and output formats must be described<br />
in a FreeForm format description file. You should follow FreeForm<br />
naming conventions for the data and format files. For details about<br />
FreeForm conventions, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
If a format description file is not specified on the command line, the<br />
output format is taken by default from the FreeForm output format<br />
annotation stored in the HDF file. If there is no annotation, a<br />
default ASCII output format is used.<br />
<br />
<blockquote>An equivalence table takes precedence over<br />
everything. (vdata=1963, SDS=702)</blockquote><br />
<br />
If you have not specified an HDF object in an equivalence table,<br />
<font color='green'>pntshow</font> uses the following sequence to determine the appropriate<br />
source for output:<br />
<br />
<br />
#Output the first vdata with class name Data.<br />
#Output the largest vdata.<br />
#Output the first SDS.<br />
<br />
If no vdatas exist in the file, but an SDS is found, it is extracted<br />
and a default ASCII output format is used.<br />
<br />
===Extracting Headers and Data===<br />
<br />
The <font color='green'>pntshow</font> command takes the following form when you want to<br />
extract headers and data from HDF files into separate files.<br />
<br />
<pre><br />
pntshow input_HDF_file [-h [output_header_file]] [-hof output_header_format_file]<br />
<br />
[-hof output_header_format_file] [-d [output_data_file]] [-dof output_data_format_file]<br />
</pre><br />
<br />
<blockquote><br />
<br />
; \var{input_HDF_file} :<br />
Name of the input HDF file, which has been packed using<br />
<font color='green'>hdfpack</font>.<br />
<br />
; \hdfh :<br />
Option flag followed optionally by the name of the file designated<br />
to contain the record headers currently stored in a vdata with a<br />
class name of Index. If an output header file name is not specified,<br />
the default is standard output.<br />
<br />
; \hdfhof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for the headers extracted to standard output or<br />
output_header_file.<br />
<br />
; \hdfd :<br />
Option flag followed optionally by the name of the file designated<br />
to contain the data currently stored in a vdata with a class name of<br />
Data. If an output file name is not specified, the default is<br />
standard output.<br />
<br />
; \hdfdof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for data extracted to standard output or<br />
\var{output_data_file}.<br />
<br />
</blockquote><br />
<br />
====Example====<br />
<br />
You will extract data and headers from <font color='green'>hara.HDF</font> (created by<br />
<font color='green'>splitdat</font> in a previous example). This file contains two vdatas:<br />
one has the class name Data and the other has the class name Index.<br />
Because this file is extremely small, no appending links were created<br />
in the file, so there is no need to pack the file before using<br />
<font color='green'>pntshow</font>, though you can if you wish.<br />
<br />
To extract data and headers from <font color='green'>hara.HDF</font>, enter the following<br />
command:<br />
<br />
<pre><br />
pntshow hara.HDF -d haradata.dat -h harahdrs.dat<br />
</pre><br />
<br />
The data from the vdata designated as Data in <font color='green'>hara.HDF</font> are now<br />
stored in <font color='green'>haradata.dat</font>. The data are in their original format<br />
because the original output format was stored by <font color='green'>splitdat</font> in the<br />
HDF file. The header data from the vdata designated as Index in<br />
<font color='green'>hara.HDF</font> are now stored in <font color='green'>harahdrs.dat</font>. In addition to<br />
the original header data, the variables begin and extent have also<br />
been extracted to <font color='green'>harahdrs.dat</font>.<br />
<br />
===Extracting Data Only===<br />
<br />
The <font color='green'>pntshow</font> command takes the following form when you want to<br />
extract just the data from an HDF file:<br />
<br />
<pre><br />
pntshow input_HDF_file [-of default_output_format_file]<br />
<br />
[> output_file]<br />
</pre><br />
<br />
<blockquote><br />
; \var{input_HDF_file} :<br />
Name of the input HDF file, which has been packed using <font color='green'>hdfpack</font>.<br />
<br />
; \hdfof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for data extracted to standard output or<br />
\var{output_file.}<br />
<br />
; \var{output_file} :<br />
Name of the output file into which data is transferred. If an output<br />
file name is not specified, the default is standard output.<br />
<br />
</blockquote><br />
<br />
====Examples====<br />
<br />
You can use <font color='green'>pntshow</font> to extract designated variables from an HDF<br />
file. In this example, you will extract temperature and pressure<br />
values from <font color='green'>hara.HDF</font> to an ASCII format. First, the following<br />
format description file must exist.<br />
<br />
Here is <font color='green'>haradata.fmt</font>:<br />
<br />
<pre><br />
ASCII_output_data "ASCII format for pressure, temp"<br />
atmospheric_pressure 1 10 long 1<br />
temperature_deg 15 25 float 1<br />
</pre><br />
<br />
To create a file named <font color='green'>temppres.dat</font> containing only the<br />
temperature and pressure variables, enter either of the following<br />
commands:<br />
<br />
<pre><br />
pntshow hara.HDF -of haradata.fmt > temppres.dat<br />
</pre><br />
<br />
or<br />
<br />
<pre><br />
pntshow hara.HDF -d temppres.dat -dof haradata.fmt<br />
</pre><br />
<br />
If you use the first command, <font color='green'>pntshow</font> searches <font color='green'>hara.HDF</font><br />
for a vdata named Data. Since <font color='green'>hara.HDF</font> contains only one vdata<br />
named <font color='green'>Data</font>, this vdata is extracted by default with the format<br />
specified in <font color='green'>haradata.fmt</font>.<br />
<br />
The results are the same if you use the second command. Now, try<br />
running <font color='green'>pntshow</font> on the previously created file <font color='green'>latlon.HDF</font>,<br />
which contains two SDS's. Use the following command:<br />
<br />
<pre><br />
pntshow latlon.HDF > latlon.SDS<br />
</pre><br />
<br />
The <font color='green'>latlon.SDS</font> file now contains the latitude and longitude<br />
values extracted from <font color='green'>latlon.HDF</font>. They have the default ASCII<br />
output format. You could have used the -of option to specify an output<br />
format included in a FreeForm format description file.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/hdfutils&diff=2039Wiki Testing/hdfutils2008-02-06T17:27:34Z<p>Yuan: /* <font color='green'>makehdf</font> */</p>
<hr />
<div>=HDF Utilities=<br />
<br />
<br />
FreeForm ND includes three utilities for use with HDF (hierarchical<br />
data format) files: <font color='green'>makehdf</font>, <font color='green'>splitdat</font>, and <font color='green'>pntshow</font>.<br />
These programs were built using both the FreeForm library and the HDF<br />
library, which was developed at the National Center for Supercomputer<br />
Applications (NCSA).<br />
<br />
The <font color='green'>makehdf</font> program converts binary and ASCII data files to HDF<br />
files and converts multiplexed band interleaved by pixel image files<br />
into a series of single parameter files. The <font color='green'>splitdat</font> program is<br />
used to separate and reformat data files containing headers and data<br />
into separate header and data files, or to translate them into HDF<br />
files. The <font color='green'>pntshow</font> program extracts point data from HDF files<br />
into binary or ASCII format.<br />
<br />
It is assumed in this chapter that you have a working familiarity with<br />
HDF terminology and conventions. See the HDF user documentation for<br />
detailed information.<br />
<br />
<blockquote>NOTE: Do not try the examples in this chapter. The example file set is<br />
incomplete. </blockquote><br />
<br />
<br />
==<font color='green'>makehdf</font>==<br />
<br />
<br />
Using <font color='green'>makehdf</font> you can convert data files with formats described<br />
in a FreeForm format file into HDF files. You should follow FreeForm<br />
naming conventions for the data and format files. For details about<br />
FreeForm conventions, see ([[Wiki_Testing/convs | Chapter 8]]).<br />
<br />
<blockquote>A dBASE input file must be converted to ASCII or binary using<br />
<font color='green'>newform</font> before you can run <font color='green'>makehdf</font> on it. </blockquote><br />
<br />
The HDF file resulting from a conversion consists either of a group of<br />
scientific datasets (SDS's), one for each variable in the input data<br />
file, or of a ''vgroup'' containing all the variables as one<br />
''vdata''. If you are working with grid data, you will want SDS's<br />
(the default) in the output HDF file. A vdata (<font color='green'>-vd</font> option) is the<br />
appropriate choice for point data.<br />
<br />
The <font color='green'>makehdf</font> command has the following form:<br />
<br />
<pre><br />
makehdf input_file [-r rows] [-c columns] [-v var_file] [-d HDF_description_file]<br />
<br />
[-xl x_label -yl y_label] [-xu x_units -yu y_units] [-xf x_format -yf y_format] [-id file_id] [-vd [vdata_file]] [-dmx [-sep]] [-df] [-md missing_data_file] [-dof HDF_file] <br />
</pre><br />
<br />
<blockquote><br />
<br />
; ''input_file'' :<br />
Name of the input data file. Following FreeForm naming conventions,<br />
the standard extensions for data files are <font color='green'>.dat</font> for ASCII<br />
format and <font color='green'>.bin</font> for binary.<br />
<br />
; -r rows :<br />
Option flag followed by the number of rows in each resulting<br />
scientific dataset. The number of rows must be specified through<br />
this option on the command line, or in an equivalence table, or in a<br />
header (<font color='green'>.hdr</font>) file defined according to FreeForm standards.<br />
<br />
; -c columns :<br />
Option flag followed by the number of columns in each resulting<br />
scientific dataset. The number of columns must be specified through<br />
this option on the command line, or in an equivalence table, or in a<br />
header (<font color='green'>.hdr</font>) file defined according to FreeForm standards.<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of the variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by <font color='green'>makehdf</font>. Variable names in \var{var_file} can be<br />
separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
; -d HDF_description_file :<br />
Option flag followed by the name of the file containing a<br />
description of the input file. The description will be stored as a<br />
file annotation in the resulting HDF file.<br />
<br />
; -xl x_label -yl y_label :<br />
Option flags followed by strings (labels) describing the x and y<br />
axes; labels must be in quotes (<font color='green'>" "</font>) if more than one word.<br />
<br />
; -xu x_units -yu y_units :<br />
Option flags followed by strings indicating the measurement units<br />
for the x and y axes; strings must be in quotes (<font color='green'>" "</font>) if more than<br />
one word.<br />
<br />
; - xf x_format -yf y_format :<br />
Option flags followed by strings indicating the formats to be used<br />
in displaying scale for the x and y dimensions; strings must be in<br />
quotes (<font color='green'>" "</font>) if more than one word.<br />
<br />
; - id file_id :<br />
Option flag followed by a string that will be stored as the ID of<br />
the resulting HDF file.<br />
<br />
; - vd [vdata_file] :<br />
Option flag indicating that the output HDF file should contain a<br />
vdata. The optional file name specifies the name of the output HDF<br />
file; the default is <font color='green'>input_file.HDF</font>.<br />
<br />
; - dmx [-sep] :<br />
The option flag <font color='green'>-dmx</font> indicates that input data should be<br />
demultiplexed from band interleaved by pixel to band sequential form<br />
in <font color='green'>input_file.dmx</font>. If <font color='green'>-dmx</font> is followed by <font color='green'>-sep</font>,<br />
the input data are demultiplexed into separate variable files called<br />
<font color='green'>data_file.1</font> \ldots <font color='green'>data_file.n</font><br />
<br />
; - df :<br />
To use this option, the input file (<font color='green'>data_file.ext</font>) must be a<br />
binary demultiplexed (band sequential) file. For each input<br />
variable in the applicable FreeForm format description file, there<br />
is a corresponding demultiplexed section in the output HDF file.<br />
<br />
; - md missing_data_file :<br />
Option flag followed by the name of the file defining missing data<br />
(data you want to exclude). Use this option only along with the<br />
vdata (-vd) option. Each line in the missing data file has the form:<br />
<br />
<pre><br />
variable_name lower_limit upper_limit<br />
</pre><br />
<br />
The precision of the upper and lower limits matches the precision of<br />
the input data.<br />
<br />
; - dof HDF_file :<br />
Option flag followed by the name of the output HDF file. If you do<br />
not use the <font color='green'>-dof</font> option, the default output file name is<br />
<font color='green'>input_file.HDF</font>.<br />
<br />
</blockquote><br />
<br />
===Example===<br />
<br />
You will use <font color='green'>makehdf</font> to store <font color='green'>latlon.dat</font> as an HDF file.<br />
The HDF file will consist of two SDS's, one each for the two variables<br />
latitude and longitude. Each SDS will have four rows and five columns.<br />
<br />
To convert <font color='green'>latlon.dat</font> to an HDF file, enter the following<br />
command:<br />
<br />
<pre><br />
makehdf latlon.dat -r 4 -c 5<br />
</pre><br />
<br />
As <font color='green'>makehdf</font> translates <font color='green'>latlon.dat</font> into HDF, processing<br />
information is displayed on the screen:<br />
<br />
<pre><br />
1 Caches (1150 bytes) Processed: 800 bytes written to latlon.dmx<br />
Writing latlon.HDF and calculating maxima and minima ...<br />
<br />
Variable latitude:<br />
Minimum: -86.432712 Maximum 89.170904<br />
Variable longitude:<br />
Minimum: -176.161101 Maximum 165.066193<br />
</pre><br />
<br />
The output from <font color='green'>makehdf</font> is an HDF file named <font color='green'>latlon.HDF</font><br />
(by default). It contains the minimum and maximum values for the two<br />
variables as well as the two SDS's.<br />
<br />
A temporary file named <font color='green'>latlon.dmx</font> was also created. It contains<br />
the data from latlon.dat in demultiplexed form . The data was<br />
converted from its original multiplexed form to enable <font color='green'>makehdf</font><br />
to write sections of data to SDS's.<br />
<br />
If you start with a demultiplexed file such as <font color='green'>latlon.dmx</font>, the<br />
translation process is much quicker, particularly for large data<br />
files. As an illustration, try this. Rename <font color='green'>latlon.dmx</font> to<br />
<font color='green'>latlon.bin</font> (renaming is necessary for <font color='green'>makehdf</font> to find the<br />
format description file <font color='green'>latlon.fmt</font> by default). Enter the<br />
following command:<br />
<br />
<pre><br />
makehdf latlon.bin -df -r 4 -c 5<br />
</pre><br />
<br />
The output file again is <font color='green'>latlon.HDF</font>, but notice that no<br />
demultiplexing was done.<br />
<br />
==<font color='green'>splitdat</font>==<br />
<br />
<br />
The <font color='green'>splitdat</font> program translates files with headers and data into<br />
separate header and data files or into HDF files. If the translation<br />
is to separate header and data files, the header file can include<br />
indexing information.<br />
<br />
The combination of header and data records in a file is often used for<br />
point data sets that include a number of observations made at one or<br />
more stations or locations in space. The header records contain<br />
information about the stations or locations of the measurements. The<br />
data records hold the observational data. A station record usually<br />
indicates how many data records follow it. The structure of such a<br />
file is similar to the following:<br />
<br />
<pre><br />
Header for Station 1<br />
Observation 1 for Station 1<br />
Observation 2 for Station 1<br />
<br />
.<br />
<br />
.<br />
Observation N for Station 1<br />
<br />
Header for Station 2<br />
Observation 1 for Station 2<br />
Observation 2 for Station 2<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
Observation N for Station 2<br />
<br />
Header for Station 3<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
</pre><br />
<br />
Many applications have difficulty reading this sort of heterogeneous<br />
data file. One solution is to split the data into two homogeneous<br />
files, one containing the headers, the other containing the data. With<br />
<font color='green'>splitdat</font>, you can easily create the separate data and header<br />
files. To use <font color='green'>splitdat</font> for this purpose, the input and output<br />
formats for the record headers and the data must be described in a<br />
FreeForm format description file. To use <font color='green'>splitdat</font> for<br />
translating files to HDF, the input format must be described in a<br />
FreeForm format description file. You should follow FreeForm naming<br />
conventions for the data and format files. For details about FreeForm<br />
conventions, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
The <font color='green'>splitdat</font> command has the following form:<br />
<br />
\proto{<font color='green'>splitdat</font> \var{input_file} [\var{output_data_file} > \var{output_header_file}]}<br />
<br />
<blockquote><br />
<br />
; \var{input_file} :<br />
Name of the file to be processed. Following FreeForm naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font><br />
for ASCII format and <font color='green'>.bin</font> for binary.<br />
<br />
; \var{output_data_file} :<br />
Name of the output file into which data are transferred with the<br />
format specified in the applicable FreeForm format description file.<br />
The standard extensions are the same as for input files. If an<br />
output file name is not specified, the default is standard output.<br />
<br />
; \var{output_header_file} :<br />
Name of the output file into which headers from the input file are<br />
transferred with the format specified in the applicable FreeForm<br />
format description file. If an output header file name is not<br />
specified, the default is standard output.<br />
</blockquote><br />
<br />
===Index Creation===<br />
<br />
You can use the two variables begin and extent (described below) in<br />
the format description for the output record headers to indicate the<br />
location and size of the data block associated with each record<br />
header. If you then use <font color='green'>splitdat</font>, the header file that results<br />
can be used as an index to the data file.<br />
<br />
<blockquote><br />
<br />
; <font color='green'>begin</font> :<br />
Indicates the offset to the beginning of the data associated with a<br />
particular header. If the data is being translated to HDF, the units<br />
are records; if not, the units are bytes.<br />
<br />
; <font color='green'>extent</font> :<br />
Indicates the number of records (HDF) or bytes (non-HDF) associated<br />
with each header record.<br />
</blockquote><br />
<br />
<br />
====Example====<br />
<br />
You will use <font color='green'>splitdat</font> to extract the headers and data from a<br />
rawinsonde (a device for gathering meteorological data) ASCII data<br />
file named <font color='green'>hara.dat</font> (HARA = Historic Arctic Rawinsonde Archive)<br />
and create two output files-<font color='green'>23338.dat</font> containing the ASCII data<br />
and <font color='green'>23338hdr.dat</font> containing the ASCII headers. The format<br />
description file <font color='green'>hara.fmt</font> should contain the necessary format<br />
descriptions.<br />
<br />
Here is <font color='green'>hara.fmt</font>:<br />
<br />
<pre><br />
ASCII_input_record_header "ASCII Location Record input format"<br />
WMO_station_ID_number 1 5 char 0<br />
latitude 6 10 long 2<br />
longitude_east 11 15 long 2<br />
year 17 18 uchar 0<br />
month 19 20 uchar 0<br />
day 21 22 uchar 0<br />
hour 23 24 uchar 0<br />
flag_processing_1 28 28 char 0<br />
flag_processing_2 29 29 char 0<br />
flag_processing_3 30 30 char 0<br />
station_type 31 31 char 0<br />
sea_level_elev 32 36 long 0<br />
instrument_type 37 38 uchar 0<br />
number_of_observations 40 42 ushort 0<br />
identification_code 44 44 char 0<br />
<br />
ASCII_input_data "Historical Arctic Rawinsonde Archive input format"<br />
atmospheric_pressure 1 5 long 1<br />
geopotential_height 7 11 long 0<br />
temperature_deg 13 16 short 0<br />
dewpoint_depression 18 20 short 0<br />
wind_direction 22 24 short 0<br />
wind_speed_m/s 26 28 short 0<br />
flag_qg 30 30 char 0<br />
flag_qg1 31 31 char 0<br />
flag_qt 33 33 char 0<br />
flag_qt1 34 34 char 0<br />
flag_qd 36 36 char 0<br />
flag_qd1 37 37 char 0<br />
flag_qw 39 39 char 0<br />
flag_qw1 40 40 char 0<br />
flag_qp 42 42 char 0<br />
flag_levck 43 43 char 0<br />
<br />
ASCII_output_record_header "ASCII Location Record output format"<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
<br />
ASCII_output_data "Historical Arctic Rawinsonde Archive output format"<br />
<br />
.<br />
<br />
.<br />
<br />
.<br />
</pre><br />
<br />
To "split" <font color='green'>hara.dat</font>, enter the following command:<br />
<br />
<pre><br />
splitdat hara.dat 23338.dat > 23338hdr.dat<br />
</pre><br />
<br />
The data values from <font color='green'>hara.dat</font> are stored in <font color='green'>23338.dat</font> and<br />
the headers in <font color='green'>23338hdr.dat</font>.<br />
<br />
Because the variables begin and extent were used in the header output<br />
format in <font color='green'>hara.fmt</font> to indicate data offset and number of<br />
records, <font color='green'>23338hdr.dat</font> has two columns of data showing offset and<br />
extent. Thus, it can serve as an index into <font color='green'>23338.dat</font>.<br />
<br />
===HDF Translation===<br />
<br />
If output files are not specified on the <font color='green'>splitdat</font> command line,<br />
a file named <font color='green'>input_file.HDF</font> is created. It is hierarchically<br />
named and organized as follows:<br />
<br />
<pre><br />
<br />
vgroup<br />
<br />
input_file_name<br />
<br />
/ \<br />
<br />
/ \<br />
<br />
vdata1 vdata2<br />
"PointIndex" "input_file_name"<br />
</pre><br />
<br />
<br />
*<font color='green'>vdata1</font> contains the record headers<br />
*<font color='green'>vdata2</font> contains the data<br />
*If writing to a Vset (represented by a vgroup), both output<br />
formats are converted to binary, if not binary already.<br />
<br />
====Example====<br />
<br />
To create the file <font color='green'>hara.HDF</font> from <font color='green'>hara.dat</font>, enter the<br />
following abbreviated command:<br />
<br />
<pre><br />
splitdat hara.dat<br />
</pre><br />
<br />
The output formats in <font color='green'>hara.fmt</font> are automatically converted to<br />
binary, and subsequently the ASCII data in <font color='green'>hara.dat</font> are also<br />
converted to binary for HDF storage.<br />
<br />
==<font color='green'>pntshow</font>==<br />
<br />
<br />
The <font color='green'>pntshow</font> program is a versatile tool for extracting point<br />
data from HDF files containing scientific datasets and Vsets. The<br />
extraction can be done into any binary or ASCII format described in a<br />
FreeForm format description file. Before using <font color='green'>pntshow</font> on an HDF<br />
file, you should pack the file using the NCSA-developed HDF utility<br />
hdfpack.<br />
<br />
You can use <font color='green'>pntshow</font> to extract headers and data from an HDF file<br />
into separate files or to extract just the data. It's a good idea to<br />
define GeoVu keywords in an equivalence table to facilitate access to<br />
HDF objects. For information about equivalence tables, see the GeoVu<br />
Tools Reference Guide. The input and output formats must be described<br />
in a FreeForm format description file. You should follow FreeForm<br />
naming conventions for the data and format files. For details about<br />
FreeForm conventions, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
If a format description file is not specified on the command line, the<br />
output format is taken by default from the FreeForm output format<br />
annotation stored in the HDF file. If there is no annotation, a<br />
default ASCII output format is used.<br />
<br />
<blockquote>An equivalence table takes precedence over<br />
everything. (vdata=1963, SDS=702)</blockquote><br />
<br />
If you have not specified an HDF object in an equivalence table,<br />
<font color='green'>pntshow</font> uses the following sequence to determine the appropriate<br />
source for output:<br />
<br />
<br />
#Output the first vdata with class name Data.<br />
#Output the largest vdata.<br />
#Output the first SDS.<br />
<br />
If no vdatas exist in the file, but an SDS is found, it is extracted<br />
and a default ASCII output format is used.<br />
<br />
===Extracting Headers and Data===<br />
<br />
The <font color='green'>pntshow</font> command takes the following form when you want to<br />
extract headers and data from HDF files into separate files.<br />
<br />
<pre><br />
pntshow input_HDF_file [-h [output_header_file]] [-hof output_header_format_file]<br />
<br />
[-hof output_header_format_file] [-d [output_data_file]] [-dof output_data_format_file]<br />
</pre><br />
<br />
<blockquote><br />
<br />
; \var{input_HDF_file} :<br />
Name of the input HDF file, which has been packed using<br />
<font color='green'>hdfpack</font>.<br />
<br />
; \hdfh :<br />
Option flag followed optionally by the name of the file designated<br />
to contain the record headers currently stored in a vdata with a<br />
class name of Index. If an output header file name is not specified,<br />
the default is standard output.<br />
<br />
; \hdfhof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for the headers extracted to standard output or<br />
output_header_file.<br />
<br />
; \hdfd :<br />
Option flag followed optionally by the name of the file designated<br />
to contain the data currently stored in a vdata with a class name of<br />
Data. If an output file name is not specified, the default is<br />
standard output.<br />
<br />
; \hdfdof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for data extracted to standard output or<br />
\var{output_data_file}.<br />
<br />
</blockquote><br />
<br />
====Example====<br />
<br />
You will extract data and headers from <font color='green'>hara.HDF</font> (created by<br />
<font color='green'>splitdat</font> in a previous example). This file contains two vdatas:<br />
one has the class name Data and the other has the class name Index.<br />
Because this file is extremely small, no appending links were created<br />
in the file, so there is no need to pack the file before using<br />
<font color='green'>pntshow</font>, though you can if you wish.<br />
<br />
To extract data and headers from <font color='green'>hara.HDF</font>, enter the following<br />
command:<br />
<br />
<pre><br />
pntshow hara.HDF -d haradata.dat -h harahdrs.dat<br />
</pre><br />
<br />
The data from the vdata designated as Data in <font color='green'>hara.HDF</font> are now<br />
stored in <font color='green'>haradata.dat</font>. The data are in their original format<br />
because the original output format was stored by <font color='green'>splitdat</font> in the<br />
HDF file. The header data from the vdata designated as Index in<br />
<font color='green'>hara.HDF</font> are now stored in <font color='green'>harahdrs.dat</font>. In addition to<br />
the original header data, the variables begin and extent have also<br />
been extracted to <font color='green'>harahdrs.dat</font>.<br />
<br />
===Extracting Data Only===<br />
<br />
The <font color='green'>pntshow</font> command takes the following form when you want to<br />
extract just the data from an HDF file:<br />
<br />
<pre><br />
pntshow input_HDF_file [-of default_output_format_file]<br />
<br />
[> output_file]<br />
</pre><br />
<br />
<blockquote><br />
; \var{input_HDF_file} :<br />
Name of the input HDF file, which has been packed using <font color='green'>hdfpack</font>.<br />
<br />
; \hdfof :<br />
Option flag followed by the name of the FreeForm format file that<br />
describes the format for data extracted to standard output or<br />
\var{output_file.}<br />
<br />
; \var{output_file} :<br />
Name of the output file into which data is transferred. If an output<br />
file name is not specified, the default is standard output.<br />
<br />
</blockquote><br />
<br />
====Examples====<br />
<br />
You can use <font color='green'>pntshow</font> to extract designated variables from an HDF<br />
file. In this example, you will extract temperature and pressure<br />
values from <font color='green'>hara.HDF</font> to an ASCII format. First, the following<br />
format description file must exist.<br />
<br />
Here is <font color='green'>haradata.fmt</font>:<br />
<br />
<pre><br />
ASCII_output_data "ASCII format for pressure, temp"<br />
atmospheric_pressure 1 10 long 1<br />
temperature_deg 15 25 float 1<br />
</pre><br />
<br />
To create a file named <font color='green'>temppres.dat</font> containing only the<br />
temperature and pressure variables, enter either of the following<br />
commands:<br />
<br />
<pre><br />
pntshow hara.HDF -of haradata.fmt > temppres.dat<br />
</pre><br />
<br />
or<br />
<br />
<pre><br />
pntshow hara.HDF -d temppres.dat -dof haradata.fmt<br />
</pre><br />
<br />
If you use the first command, <font color='green'>pntshow</font> searches <font color='green'>hara.HDF</font><br />
for a vdata named Data. Since <font color='green'>hara.HDF</font> contains only one vdata<br />
named <font color='green'>Data</font>, this vdata is extracted by default with the format<br />
specified in <font color='green'>haradata.fmt</font>.<br />
<br />
The results are the same if you use the second command. Now, try<br />
running <font color='green'>pntshow</font> on the previously created file <font color='green'>latlon.HDF</font>,<br />
which contains two SDS's. Use the following command:<br />
<br />
<pre><br />
pntshow latlon.HDF > latlon.SDS<br />
</pre><br />
<br />
The <font color='green'>latlon.SDS</font> file now contains the latitude and longitude<br />
values extracted from <font color='green'>latlon.HDF</font>. They have the default ASCII<br />
output format. You could have used the -of option to specify an output<br />
format included in a FreeForm format description file.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/datachk&diff=2038Wiki Testing/datachk2008-02-06T17:22:31Z<p>Yuan: /* Variable Summaries */</p>
<hr />
<div>=Data Checking=<br />
<br />
<br />
The FreeForm ND-based utility program <font color='green'>checkvar</font> creates variable<br />
summary files, lists of maximum and minimum values, and summaries of<br />
processing activity. You can use this information to check data<br />
quality and to examine the distribution of the data.<br />
<br />
<br />
<br />
==Generating the Summaries==<br />
<br />
<br />
<br />
A variable summary file (or list file), which contains histogram<br />
information showing the variable's distribution in the data file, is<br />
created for each variable (or designated variables) in the specified<br />
data file. You can optionally specify an output file in which a<br />
summary of processing activity is saved.<br />
<br />
Variable summaries (list files) can be helpful for performing quality<br />
control checks of data. For example, you could run <font color='green'>checkvar</font> on<br />
an ASCII file, convert the file to binary, and then run <font color='green'>checkvar</font><br />
on the binary file. The output from <font color='green'>checkvar</font> should be the same<br />
for both the ASCII and binary files. You can also use variable<br />
summaries to look at the data distribution in a data set before<br />
extracting data.<br />
<br />
The <font color='green'>checkvar</font> command has the following form:<br />
<br />
<pre><br />
<br />
checkvar input_file [-f format_file] [-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-p precision] [-m maxbins] [-md missing_data_flag] [-mm] [-o processing_summary]<br />
<br />
</pre><br />
<br />
<br />
The <font color='green'>checkvar</font> program needs to find only an input format<br />
description. Output format descriptions will be ignored. If conversion<br />
variables are included in input or output formats, no conversion is<br />
performed when you run <font color='green'>checkvar</font>, since it ignores output<br />
formats.<br />
<br />
For descriptions of the standard arguments (first eleven arguments<br />
above), see ([[Wiki_Testing/convs|Section 8.6]]).<br />
<br />
<blockquote><br />
<br />
; -p precision:<br />
Option flag followed by the number of decimal places. The number<br />
represents the power of 10 that data is multiplied by prior to<br />
binning. A value of 0 bins on one's, 1 on tenth's, and so on. This<br />
option allows an adjustment of the resolution of the <font color='green'>checkvar</font><br />
output.<br />
The default is 0; maximum is 5.<br />
<br />
<br />
<blockquote>NOTE:If you use the <font color='green'>-p</font> option on the command line, the precision set in the relevant format file is overridden. The precision in<br />
the format file serves as the default. </blockquote><br />
<br />
; -m maxbins:<br />
Option flag followed by the approximate maximum number of bins<br />
desired in <font color='green'>checkvar</font> output. The <font color='green'>checkvar</font> program keeps track of the number of bins filled as the data is processed. The<br />
smaller the number of bins, the faster <font color='green'>checkvar</font> runs. By<br />
keeping the number of bins small, you can check the gross aspects of<br />
data distribution rather than the details. The number of bins is adjusted dynamically as <font color='green'>checkvar</font> runs depending on the distribution of data in the input file. If the number of filled bins becomes > 1.5 * maxbins, the width of the bins is doubled to keep the total number near the desired maximum. The default is 100 bins; minimum is 6. Must be < 10,000.<br />
<br />
<br />
<blockquote>NOTE: The precision (-p) and maxbins (-m) options have no effect on<br />
character variables. </blockquote><br />
<br />
<br />
; -md missing_data_flag:<br />
Option flag followed by a flag value that <font color='green'>checkvar</font> should<br />
ignore across all variables in creating histogram data. Missing data flags are used in a data file to indicate missing or meaningless data. If you want <font color='green'>checkvar</font> to ignore more than one value, use the query (<font color='green'>-q</font>) option in conjunction with the variable file (<font color='green'>-v</font>) option.<br />
<br />
; -mm :<br />
Option flag indicating that only the maximum and minimum values of variables are calculated and displayed in the processing summary. Variable summary files are not created.<br />
<br />
; -o ''processing_summary'' :<br />
Option flag followed by the name of the file in which summary information displayed during processing is stored.<br />
<br />
</blockquote><br />
<br />
==Example==<br />
<br />
<br />
You will use <font color='green'>checkvar</font> with a precision of 3 to create a<br />
processing summary file and summary files for the two variables<br />
latitude and longitude in the file <font color='green'>latlon.dat</font>.<br />
<br />
Here is <font color='green'>latlon.dat</font>:<br />
<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
To create the summary files, enter the following command:<br />
<br />
<pre><br />
checkvar latlon.dat -p 3 -o latlon.sum<br />
</pre><br />
<br />
A summary of processing information and the maximum and minimum for<br />
each variable are displayed on the screen. The following three files<br />
are created:<br />
<br />
<br />
*<font color='green'>latlon.sum</font> recaps processing activity, maximums and minimums<br />
*<font color='green'>latitude.lst</font> shows distribution of the latitude values in <font color='green'>latlon.dat</font><br />
*<font color='green'>longitude.lst</font> shows distribution of the longitude values in <font color='green'>latlon.dat</font>.<br />
<br />
==Interpreting the Summaries==<br />
<br />
<br />
The processing and variable summary files output by <font color='green'>checkvar</font><br />
from the example in the previous section are shown and discussed<br />
below.<br />
<br />
===Processing Summary===<br />
<br />
<br />
If you specify an output file on the command line, it stores the<br />
information that is displayed on the screen during processing. The<br />
file <font color='green'>latlon.sum</font> was specified as the output file in the example<br />
above.<br />
<br />
Here is <font color='green'>latlon.sum</font>:<br />
<br />
<pre><br />
Input file : latlon.dat<br />
Requested precision = 3, Approximate number of sorting bins = 100<br />
<br />
Input data format (latlon.fmt)<br />
ASCII_input_data "ASCII format"<br />
The format contains 2 variables; length is 24.<br />
<br />
Output data format (latlon.fmt)<br />
binary_output_data "binary format"<br />
The format contains 2 variables; length is 16.<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
latitude: 20 values read<br />
minimum: -83.223548 found at record 5<br />
maximum: 77.883119 found at record 12<br />
Summary file: latitude.lst<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
longitude: 20 values read<br />
minimum: -176.161101 found at record 1<br />
maximum: 149.408117 found at record 4<br />
Summary file: longitude.lst.<br />
</pre><br />
<br />
<br />
The processing summary file <font color='green'>latlon.sum</font> first shows the name of<br />
the input data file (<font color='green'>latlon.dat</font>). If you specified precision and<br />
a maximum number of bins on the command line, those values are given<br />
as Requested precision, in this case 3, and Approximate number of<br />
sorting bins, in this case the default value of 100. If precision is<br />
not specified, No requested precision is shown.<br />
<br />
A summary of each format shows the type of format (in this case, Input<br />
data format and Output data format) and the name of the format file<br />
containing the format descriptions (<font color='green'>latlon.fmt</font>), whether<br />
specified on the command line or located through the default search<br />
sequence (as detailed in chapter 4). In this case, it was located by<br />
default. Since <font color='green'>checkvar</font> only needs an input format description,<br />
it ignores output format descriptions. Next, you see the format<br />
descriptor as resolved by FreeForm ND (e.g., <font color='green'>ASCII_input_data</font>)<br />
and the format title (e.g., "ASCII format"). Then the number of<br />
variables in a record and total record length are given; for ASCII,<br />
record length includes the end-of-line character (1 byte for Unix).<br />
<br />
A section for each variable processed by <font color='green'>checkvar</font> indicates the<br />
histogram precision and actual number of sorting bins. Under some<br />
circumstances, the precision of values in the histogram file may be<br />
different than the precision you specified on the command line. The<br />
default value for precision, if none is specified on the command line,<br />
is the precision specified in the relevant format description file or<br />
5, whichever is smaller. The second line shows the name of the<br />
variable (latitude, longitude) and the number of values in the data<br />
file for the variable (20 for both latitude and longitude).<br />
<br />
The minimum and maximum values for the variable are shown next<br />
(-83.223548 is the minimum and 77.883119 is the maximum value for<br />
latitude). The maximum and minimum values are given here with a<br />
precision of 6, which is the precision specified in the format<br />
description file. The locations of the maximum and minimum values in<br />
the input file are indicated. (-83.223548 is the fifth latitude value<br />
in <font color='green'>latlon.dat</font> and 77.883119 is the twelfth). Finally, the name<br />
of the histogram data (or variable summary) file generated for each<br />
variable is given (<font color='green'>latitude.lst</font> and <font color='green'>longitude.lst</font>).<br />
<br />
===Variable Summaries===<br />
<br />
<br />
The name of each variable summary file (list file) output by<br />
<font color='green'>checkvar</font> is of the form <font color='green'>variable.lst</font> for numeric variables and <font color='green'>variable.cst</font> for character variables. The data in *<font color='green'>.lst</font>, and *<font color='green'>.cst</font> files can be loaded into histogram plot programs for graphical representation. (You must be familiar enough<br />
with your program of choice to manipulate the data as necessary in order to achieve the desired result.) In Unix, there is no need to abbreviate the base file name.<br />
<br />
<blockquote>NOTE: If you use the -v option, the order of variables in var_file has<br />
no effect on the numbering of base file names of the variable summary files. </blockquote><br />
<br />
<br />
The two example variable summary files, <font color='green'>latitude.lst</font> and<br />
<font color='green'>longitude.lst</font>, are shown next.<br />
<br />
<center><br />
{| <br />
|<br />
{| border="2"<br />
| <font color='green'>latitude.lst</font> <br />
|-<br />
| -83.224 1 <br />
|-<br />
| -69.274 1<br />
|-<br />
| -65.865 1<br />
|-<br />
| -63.212 1<br />
|-<br />
| -63.003 1<br />
|-<br />
| -47.304 1<br />
|-<br />
| -34.578 1<br />
|-<br />
| -28.287 1<br />
|-<br />
| -12.967 1<br />
|-<br />
| -0.929 1<br />
|-<br />
| 11.624 1<br />
|-<br />
| 12.588 1<br />
|-<br />
| 27.331 1<br />
|-<br />
| 29.918 1<br />
|-<br />
| 35.086 1<br />
|-<br />
| 35.130 1<br />
|-<br />
| 38.818 1<br />
|-<br />
| 54.118 1<br />
|-<br />
| 77.652 1<br />
|-<br />
| 77.883 1<br />
<br />
|}<br />
||<br />
{| border="2"<br />
| <font color='green'>longitude.lst</font><br />
|-<br />
| -176.162 1<br />
|-<br />
| -155.234 1<br />
|-<br />
| -153.544 1<br />
|-<br />
| -136.941 1<br />
|-<br />
| -113.661 1<br />
|-<br />
| -79.178 1<br />
|-<br />
| -77.506 1<br />
|-<br />
| -55.442 1<br />
|-<br />
| -21.644 1<br />
|-<br />
| 0.777 1<br />
|-<br />
| 30.172 1<br />
|-<br />
| 35.591 1<br />
|-<br />
| 36.356 1<br />
|-<br />
| 38.875 1<br />
|-<br />
| 55.319 1<br />
|-<br />
| 62.152 1<br />
|-<br />
| 91.411 1<br />
|-<br />
| 134.124 1<br />
|-<br />
| 144.804 1<br />
|-<br />
| 149.408 1<br />
<br />
|}<br />
|}<br />
</center><br />
<br />
The variable summary files consist of two columns. The first indicates<br />
boundary values for data bins and the second gives the number of data<br />
points in each bin. Because a precision of 3 was specified in the<br />
example, each boundary value has three decimal places. The boundary<br />
values are determined dynamically by <font color='green'>checkvar</font> and often do not<br />
correspond to data values in the input file, even if the<br />
<font color='green'>checkvar</font> and data file precisions are the same.<br />
<br />
The first data bin in <font color='green'>latitude.lst</font> contains data points in the<br />
range -83.224 (inclusive) to -69.274 (exclusive); neither boundary<br />
number exists in <font color='green'>latlon.dat</font>. The first bin has one data point,<br />
-83.223548. The fourth data bin contains latitude values from -63.212<br />
(inclusive) to -63.003 (exclusive), again with neither boundary value<br />
occurring in the data file. The data point in the fourth bin is<br />
-63.211962.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/datachk&diff=2037Wiki Testing/datachk2008-02-06T17:20:33Z<p>Yuan: /* Example */</p>
<hr />
<div>=Data Checking=<br />
<br />
<br />
The FreeForm ND-based utility program <font color='green'>checkvar</font> creates variable<br />
summary files, lists of maximum and minimum values, and summaries of<br />
processing activity. You can use this information to check data<br />
quality and to examine the distribution of the data.<br />
<br />
<br />
<br />
==Generating the Summaries==<br />
<br />
<br />
<br />
A variable summary file (or list file), which contains histogram<br />
information showing the variable's distribution in the data file, is<br />
created for each variable (or designated variables) in the specified<br />
data file. You can optionally specify an output file in which a<br />
summary of processing activity is saved.<br />
<br />
Variable summaries (list files) can be helpful for performing quality<br />
control checks of data. For example, you could run <font color='green'>checkvar</font> on<br />
an ASCII file, convert the file to binary, and then run <font color='green'>checkvar</font><br />
on the binary file. The output from <font color='green'>checkvar</font> should be the same<br />
for both the ASCII and binary files. You can also use variable<br />
summaries to look at the data distribution in a data set before<br />
extracting data.<br />
<br />
The <font color='green'>checkvar</font> command has the following form:<br />
<br />
<pre><br />
<br />
checkvar input_file [-f format_file] [-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-p precision] [-m maxbins] [-md missing_data_flag] [-mm] [-o processing_summary]<br />
<br />
</pre><br />
<br />
<br />
The <font color='green'>checkvar</font> program needs to find only an input format<br />
description. Output format descriptions will be ignored. If conversion<br />
variables are included in input or output formats, no conversion is<br />
performed when you run <font color='green'>checkvar</font>, since it ignores output<br />
formats.<br />
<br />
For descriptions of the standard arguments (first eleven arguments<br />
above), see ([[Wiki_Testing/convs|Section 8.6]]).<br />
<br />
<blockquote><br />
<br />
; -p precision:<br />
Option flag followed by the number of decimal places. The number<br />
represents the power of 10 that data is multiplied by prior to<br />
binning. A value of 0 bins on one's, 1 on tenth's, and so on. This<br />
option allows an adjustment of the resolution of the <font color='green'>checkvar</font><br />
output.<br />
The default is 0; maximum is 5.<br />
<br />
<br />
<blockquote>NOTE:If you use the <font color='green'>-p</font> option on the command line, the precision set in the relevant format file is overridden. The precision in<br />
the format file serves as the default. </blockquote><br />
<br />
; -m maxbins:<br />
Option flag followed by the approximate maximum number of bins<br />
desired in <font color='green'>checkvar</font> output. The <font color='green'>checkvar</font> program keeps track of the number of bins filled as the data is processed. The<br />
smaller the number of bins, the faster <font color='green'>checkvar</font> runs. By<br />
keeping the number of bins small, you can check the gross aspects of<br />
data distribution rather than the details. The number of bins is adjusted dynamically as <font color='green'>checkvar</font> runs depending on the distribution of data in the input file. If the number of filled bins becomes > 1.5 * maxbins, the width of the bins is doubled to keep the total number near the desired maximum. The default is 100 bins; minimum is 6. Must be < 10,000.<br />
<br />
<br />
<blockquote>NOTE: The precision (-p) and maxbins (-m) options have no effect on<br />
character variables. </blockquote><br />
<br />
<br />
; -md missing_data_flag:<br />
Option flag followed by a flag value that <font color='green'>checkvar</font> should<br />
ignore across all variables in creating histogram data. Missing data flags are used in a data file to indicate missing or meaningless data. If you want <font color='green'>checkvar</font> to ignore more than one value, use the query (<font color='green'>-q</font>) option in conjunction with the variable file (<font color='green'>-v</font>) option.<br />
<br />
; -mm :<br />
Option flag indicating that only the maximum and minimum values of variables are calculated and displayed in the processing summary. Variable summary files are not created.<br />
<br />
; -o ''processing_summary'' :<br />
Option flag followed by the name of the file in which summary information displayed during processing is stored.<br />
<br />
</blockquote><br />
<br />
==Example==<br />
<br />
<br />
You will use <font color='green'>checkvar</font> with a precision of 3 to create a<br />
processing summary file and summary files for the two variables<br />
latitude and longitude in the file <font color='green'>latlon.dat</font>.<br />
<br />
Here is <font color='green'>latlon.dat</font>:<br />
<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
To create the summary files, enter the following command:<br />
<br />
<pre><br />
checkvar latlon.dat -p 3 -o latlon.sum<br />
</pre><br />
<br />
A summary of processing information and the maximum and minimum for<br />
each variable are displayed on the screen. The following three files<br />
are created:<br />
<br />
<br />
*<font color='green'>latlon.sum</font> recaps processing activity, maximums and minimums<br />
*<font color='green'>latitude.lst</font> shows distribution of the latitude values in <font color='green'>latlon.dat</font><br />
*<font color='green'>longitude.lst</font> shows distribution of the longitude values in <font color='green'>latlon.dat</font>.<br />
<br />
==Interpreting the Summaries==<br />
<br />
<br />
The processing and variable summary files output by <font color='green'>checkvar</font><br />
from the example in the previous section are shown and discussed<br />
below.<br />
<br />
===Processing Summary===<br />
<br />
<br />
If you specify an output file on the command line, it stores the<br />
information that is displayed on the screen during processing. The<br />
file <font color='green'>latlon.sum</font> was specified as the output file in the example<br />
above.<br />
<br />
Here is <font color='green'>latlon.sum</font>:<br />
<br />
<pre><br />
Input file : latlon.dat<br />
Requested precision = 3, Approximate number of sorting bins = 100<br />
<br />
Input data format (latlon.fmt)<br />
ASCII_input_data "ASCII format"<br />
The format contains 2 variables; length is 24.<br />
<br />
Output data format (latlon.fmt)<br />
binary_output_data "binary format"<br />
The format contains 2 variables; length is 16.<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
latitude: 20 values read<br />
minimum: -83.223548 found at record 5<br />
maximum: 77.883119 found at record 12<br />
Summary file: latitude.lst<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
longitude: 20 values read<br />
minimum: -176.161101 found at record 1<br />
maximum: 149.408117 found at record 4<br />
Summary file: longitude.lst.<br />
</pre><br />
<br />
<br />
The processing summary file <font color='green'>latlon.sum</font> first shows the name of<br />
the input data file (<font color='green'>latlon.dat</font>). If you specified precision and<br />
a maximum number of bins on the command line, those values are given<br />
as Requested precision, in this case 3, and Approximate number of<br />
sorting bins, in this case the default value of 100. If precision is<br />
not specified, No requested precision is shown.<br />
<br />
A summary of each format shows the type of format (in this case, Input<br />
data format and Output data format) and the name of the format file<br />
containing the format descriptions (<font color='green'>latlon.fmt</font>), whether<br />
specified on the command line or located through the default search<br />
sequence (as detailed in chapter 4). In this case, it was located by<br />
default. Since <font color='green'>checkvar</font> only needs an input format description,<br />
it ignores output format descriptions. Next, you see the format<br />
descriptor as resolved by FreeForm ND (e.g., <font color='green'>ASCII_input_data</font>)<br />
and the format title (e.g., "ASCII format"). Then the number of<br />
variables in a record and total record length are given; for ASCII,<br />
record length includes the end-of-line character (1 byte for Unix).<br />
<br />
A section for each variable processed by <font color='green'>checkvar</font> indicates the<br />
histogram precision and actual number of sorting bins. Under some<br />
circumstances, the precision of values in the histogram file may be<br />
different than the precision you specified on the command line. The<br />
default value for precision, if none is specified on the command line,<br />
is the precision specified in the relevant format description file or<br />
5, whichever is smaller. The second line shows the name of the<br />
variable (latitude, longitude) and the number of values in the data<br />
file for the variable (20 for both latitude and longitude).<br />
<br />
The minimum and maximum values for the variable are shown next<br />
(-83.223548 is the minimum and 77.883119 is the maximum value for<br />
latitude). The maximum and minimum values are given here with a<br />
precision of 6, which is the precision specified in the format<br />
description file. The locations of the maximum and minimum values in<br />
the input file are indicated. (-83.223548 is the fifth latitude value<br />
in <font color='green'>latlon.dat</font> and 77.883119 is the twelfth). Finally, the name<br />
of the histogram data (or variable summary) file generated for each<br />
variable is given (<font color='green'>latitude.lst</font> and <font color='green'>longitude.lst</font>).<br />
<br />
===Variable Summaries===<br />
<br />
<br />
The name of each variable summary file (list file) output by<br />
<font color='green'>checkvar</font> is of the form <font color='green'>variable.lst</font> for numeric variables<br />
and <font color='green'>variable.cst</font> for character variables. The data in<br />
*<font color='green'>.lst</font>, and *<font color='green'>.cst</font> files can be loaded into histogram plot<br />
programs for graphical representation. (You must be familiar enough<br />
with your program of choice to manipulate the data as necessary in<br />
order to achieve the desired result.) In Unix, there is no need to<br />
abbreviate the base file name.<br />
<br />
<blockquote>If you use the -v option, the order of variables in var_file has<br />
no effect on the numbering of base file names of the variable<br />
summary files. </blockquote><br />
<br />
The two example variable summary files, <font color='green'>latitude.lst</font> and<br />
<font color='green'>longitude.lst</font>, are shown next.<br />
<br />
<br />
{| <br />
|<br />
{| border="2"<br />
| <font color='green'>latitude.lst</font> <br />
|-<br />
| -83.224 1 <br />
|-<br />
| -69.274 1<br />
|-<br />
| -65.865 1<br />
|-<br />
| -63.212 1<br />
|-<br />
| -63.003 1<br />
|-<br />
| -47.304 1<br />
|-<br />
| -34.578 1<br />
|-<br />
| -28.287 1<br />
|-<br />
| -12.967 1<br />
|-<br />
| -0.929 1<br />
|-<br />
| 11.624 1<br />
|-<br />
| 12.588 1<br />
|-<br />
| 27.331 1<br />
|-<br />
| 29.918 1<br />
|-<br />
| 35.086 1<br />
|-<br />
| 35.130 1<br />
|-<br />
| 38.818 1<br />
|-<br />
| 54.118 1<br />
|-<br />
| 77.652 1<br />
|-<br />
| 77.883 1<br />
<br />
|}<br />
||<br />
{| border="2"<br />
| <font color='green'>longitude.lst</font><br />
|-<br />
| -176.162 1<br />
|-<br />
| -155.234 1<br />
|-<br />
| -153.544 1<br />
|-<br />
| -136.941 1<br />
|-<br />
| -113.661 1<br />
|-<br />
| -79.178 1<br />
|-<br />
| -77.506 1<br />
|-<br />
| -55.442 1<br />
|-<br />
| -21.644 1<br />
|-<br />
| 0.777 1<br />
|-<br />
| 30.172 1<br />
|-<br />
| 35.591 1<br />
|-<br />
| 36.356 1<br />
|-<br />
| 38.875 1<br />
|-<br />
| 55.319 1<br />
|-<br />
| 62.152 1<br />
|-<br />
| 91.411 1<br />
|-<br />
| 134.124 1<br />
|-<br />
| 144.804 1<br />
|-<br />
| 149.408 1<br />
<br />
|}<br />
|}<br />
<br />
The variable summary files consist of two columns. The first indicates<br />
boundary values for data bins and the second gives the number of data<br />
points in each bin. Because a precision of 3 was specified in the<br />
example, each boundary value has three decimal places. The boundary<br />
values are determined dynamically by <font color='green'>checkvar</font> and often do not<br />
correspond to data values in the input file, even if the<br />
<font color='green'>checkvar</font> and data file precisions are the same.<br />
<br />
The first data bin in <font color='green'>latitude.lst</font> contains data points in the<br />
range -83.224 (inclusive) to -69.274 (exclusive); neither boundary<br />
number exists in <font color='green'>latlon.dat</font>. The first bin has one data point,<br />
-83.223548. The fourth data bin contains latitude values from -63.212<br />
(inclusive) to -63.003 (exclusive), again with neither boundary value<br />
occurring in the data file. The data point in the fourth bin is<br />
-63.211962.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/datachk&diff=2036Wiki Testing/datachk2008-02-06T17:20:04Z<p>Yuan: /* Generating the Summaries */</p>
<hr />
<div>=Data Checking=<br />
<br />
<br />
The FreeForm ND-based utility program <font color='green'>checkvar</font> creates variable<br />
summary files, lists of maximum and minimum values, and summaries of<br />
processing activity. You can use this information to check data<br />
quality and to examine the distribution of the data.<br />
<br />
<br />
<br />
==Generating the Summaries==<br />
<br />
<br />
<br />
A variable summary file (or list file), which contains histogram<br />
information showing the variable's distribution in the data file, is<br />
created for each variable (or designated variables) in the specified<br />
data file. You can optionally specify an output file in which a<br />
summary of processing activity is saved.<br />
<br />
Variable summaries (list files) can be helpful for performing quality<br />
control checks of data. For example, you could run <font color='green'>checkvar</font> on<br />
an ASCII file, convert the file to binary, and then run <font color='green'>checkvar</font><br />
on the binary file. The output from <font color='green'>checkvar</font> should be the same<br />
for both the ASCII and binary files. You can also use variable<br />
summaries to look at the data distribution in a data set before<br />
extracting data.<br />
<br />
The <font color='green'>checkvar</font> command has the following form:<br />
<br />
<pre><br />
<br />
checkvar input_file [-f format_file] [-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-p precision] [-m maxbins] [-md missing_data_flag] [-mm] [-o processing_summary]<br />
<br />
</pre><br />
<br />
<br />
The <font color='green'>checkvar</font> program needs to find only an input format<br />
description. Output format descriptions will be ignored. If conversion<br />
variables are included in input or output formats, no conversion is<br />
performed when you run <font color='green'>checkvar</font>, since it ignores output<br />
formats.<br />
<br />
For descriptions of the standard arguments (first eleven arguments<br />
above), see ([[Wiki_Testing/convs|Section 8.6]]).<br />
<br />
<blockquote><br />
<br />
; -p precision:<br />
Option flag followed by the number of decimal places. The number<br />
represents the power of 10 that data is multiplied by prior to<br />
binning. A value of 0 bins on one's, 1 on tenth's, and so on. This<br />
option allows an adjustment of the resolution of the <font color='green'>checkvar</font><br />
output.<br />
The default is 0; maximum is 5.<br />
<br />
<br />
<blockquote>NOTE:If you use the <font color='green'>-p</font> option on the command line, the precision set in the relevant format file is overridden. The precision in<br />
the format file serves as the default. </blockquote><br />
<br />
; -m maxbins:<br />
Option flag followed by the approximate maximum number of bins<br />
desired in <font color='green'>checkvar</font> output. The <font color='green'>checkvar</font> program keeps track of the number of bins filled as the data is processed. The<br />
smaller the number of bins, the faster <font color='green'>checkvar</font> runs. By<br />
keeping the number of bins small, you can check the gross aspects of<br />
data distribution rather than the details. The number of bins is adjusted dynamically as <font color='green'>checkvar</font> runs depending on the distribution of data in the input file. If the number of filled bins becomes > 1.5 * maxbins, the width of the bins is doubled to keep the total number near the desired maximum. The default is 100 bins; minimum is 6. Must be < 10,000.<br />
<br />
<br />
<blockquote>NOTE: The precision (-p) and maxbins (-m) options have no effect on<br />
character variables. </blockquote><br />
<br />
<br />
; -md missing_data_flag:<br />
Option flag followed by a flag value that <font color='green'>checkvar</font> should<br />
ignore across all variables in creating histogram data. Missing data flags are used in a data file to indicate missing or meaningless data. If you want <font color='green'>checkvar</font> to ignore more than one value, use the query (<font color='green'>-q</font>) option in conjunction with the variable file (<font color='green'>-v</font>) option.<br />
<br />
; -mm :<br />
Option flag indicating that only the maximum and minimum values of variables are calculated and displayed in the processing summary. Variable summary files are not created.<br />
<br />
; -o ''processing_summary'' :<br />
Option flag followed by the name of the file in which summary information displayed during processing is stored.<br />
<br />
</blockquote><br />
<br />
==Example==<br />
<br />
<br />
You will use <font color='green'>checkvar</font> with a precision of 3 to create a<br />
processing summary file and summary files for the two variables<br />
latitude and longitude in the file <font color='green'>latlon.dat</font>.<br />
<br />
Here is <font color='green'>latlon.dat</font>:<br />
<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
To create the summary files, enter the following command:<br />
<br />
<pre><br />
checkvar latlon.dat -p 3 -o latlon.sum<br />
</pre><br />
<br />
A summary of processing information and the maximum and minimum for<br />
each variable are displayed on the screen. The following three files<br />
are created:<br />
<br />
<br />
*<font color='green'>latlon.sum</font> recaps processing activity, maximums and minimums<br />
*<font color='green'>latitude.lst</font> shows distribution of the latitude values in<br />
<font color='green'>latlon.dat</font><br />
*<font color='green'>longitude.lst</font> shows distribution of the longitude values<br />
in <font color='green'>latlon.dat</font>.<br />
<br />
==Interpreting the Summaries==<br />
<br />
<br />
The processing and variable summary files output by <font color='green'>checkvar</font><br />
from the example in the previous section are shown and discussed<br />
below.<br />
<br />
===Processing Summary===<br />
<br />
<br />
If you specify an output file on the command line, it stores the<br />
information that is displayed on the screen during processing. The<br />
file <font color='green'>latlon.sum</font> was specified as the output file in the example<br />
above.<br />
<br />
Here is <font color='green'>latlon.sum</font>:<br />
<br />
<pre><br />
Input file : latlon.dat<br />
Requested precision = 3, Approximate number of sorting bins = 100<br />
<br />
Input data format (latlon.fmt)<br />
ASCII_input_data "ASCII format"<br />
The format contains 2 variables; length is 24.<br />
<br />
Output data format (latlon.fmt)<br />
binary_output_data "binary format"<br />
The format contains 2 variables; length is 16.<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
latitude: 20 values read<br />
minimum: -83.223548 found at record 5<br />
maximum: 77.883119 found at record 12<br />
Summary file: latitude.lst<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
longitude: 20 values read<br />
minimum: -176.161101 found at record 1<br />
maximum: 149.408117 found at record 4<br />
Summary file: longitude.lst.<br />
</pre><br />
<br />
<br />
The processing summary file <font color='green'>latlon.sum</font> first shows the name of<br />
the input data file (<font color='green'>latlon.dat</font>). If you specified precision and<br />
a maximum number of bins on the command line, those values are given<br />
as Requested precision, in this case 3, and Approximate number of<br />
sorting bins, in this case the default value of 100. If precision is<br />
not specified, No requested precision is shown.<br />
<br />
A summary of each format shows the type of format (in this case, Input<br />
data format and Output data format) and the name of the format file<br />
containing the format descriptions (<font color='green'>latlon.fmt</font>), whether<br />
specified on the command line or located through the default search<br />
sequence (as detailed in chapter 4). In this case, it was located by<br />
default. Since <font color='green'>checkvar</font> only needs an input format description,<br />
it ignores output format descriptions. Next, you see the format<br />
descriptor as resolved by FreeForm ND (e.g., <font color='green'>ASCII_input_data</font>)<br />
and the format title (e.g., "ASCII format"). Then the number of<br />
variables in a record and total record length are given; for ASCII,<br />
record length includes the end-of-line character (1 byte for Unix).<br />
<br />
A section for each variable processed by <font color='green'>checkvar</font> indicates the<br />
histogram precision and actual number of sorting bins. Under some<br />
circumstances, the precision of values in the histogram file may be<br />
different than the precision you specified on the command line. The<br />
default value for precision, if none is specified on the command line,<br />
is the precision specified in the relevant format description file or<br />
5, whichever is smaller. The second line shows the name of the<br />
variable (latitude, longitude) and the number of values in the data<br />
file for the variable (20 for both latitude and longitude).<br />
<br />
The minimum and maximum values for the variable are shown next<br />
(-83.223548 is the minimum and 77.883119 is the maximum value for<br />
latitude). The maximum and minimum values are given here with a<br />
precision of 6, which is the precision specified in the format<br />
description file. The locations of the maximum and minimum values in<br />
the input file are indicated. (-83.223548 is the fifth latitude value<br />
in <font color='green'>latlon.dat</font> and 77.883119 is the twelfth). Finally, the name<br />
of the histogram data (or variable summary) file generated for each<br />
variable is given (<font color='green'>latitude.lst</font> and <font color='green'>longitude.lst</font>).<br />
<br />
===Variable Summaries===<br />
<br />
<br />
The name of each variable summary file (list file) output by<br />
<font color='green'>checkvar</font> is of the form <font color='green'>variable.lst</font> for numeric variables<br />
and <font color='green'>variable.cst</font> for character variables. The data in<br />
*<font color='green'>.lst</font>, and *<font color='green'>.cst</font> files can be loaded into histogram plot<br />
programs for graphical representation. (You must be familiar enough<br />
with your program of choice to manipulate the data as necessary in<br />
order to achieve the desired result.) In Unix, there is no need to<br />
abbreviate the base file name.<br />
<br />
<blockquote>If you use the -v option, the order of variables in var_file has<br />
no effect on the numbering of base file names of the variable<br />
summary files. </blockquote><br />
<br />
The two example variable summary files, <font color='green'>latitude.lst</font> and<br />
<font color='green'>longitude.lst</font>, are shown next.<br />
<br />
<br />
{| <br />
|<br />
{| border="2"<br />
| <font color='green'>latitude.lst</font> <br />
|-<br />
| -83.224 1 <br />
|-<br />
| -69.274 1<br />
|-<br />
| -65.865 1<br />
|-<br />
| -63.212 1<br />
|-<br />
| -63.003 1<br />
|-<br />
| -47.304 1<br />
|-<br />
| -34.578 1<br />
|-<br />
| -28.287 1<br />
|-<br />
| -12.967 1<br />
|-<br />
| -0.929 1<br />
|-<br />
| 11.624 1<br />
|-<br />
| 12.588 1<br />
|-<br />
| 27.331 1<br />
|-<br />
| 29.918 1<br />
|-<br />
| 35.086 1<br />
|-<br />
| 35.130 1<br />
|-<br />
| 38.818 1<br />
|-<br />
| 54.118 1<br />
|-<br />
| 77.652 1<br />
|-<br />
| 77.883 1<br />
<br />
|}<br />
||<br />
{| border="2"<br />
| <font color='green'>longitude.lst</font><br />
|-<br />
| -176.162 1<br />
|-<br />
| -155.234 1<br />
|-<br />
| -153.544 1<br />
|-<br />
| -136.941 1<br />
|-<br />
| -113.661 1<br />
|-<br />
| -79.178 1<br />
|-<br />
| -77.506 1<br />
|-<br />
| -55.442 1<br />
|-<br />
| -21.644 1<br />
|-<br />
| 0.777 1<br />
|-<br />
| 30.172 1<br />
|-<br />
| 35.591 1<br />
|-<br />
| 36.356 1<br />
|-<br />
| 38.875 1<br />
|-<br />
| 55.319 1<br />
|-<br />
| 62.152 1<br />
|-<br />
| 91.411 1<br />
|-<br />
| 134.124 1<br />
|-<br />
| 144.804 1<br />
|-<br />
| 149.408 1<br />
<br />
|}<br />
|}<br />
<br />
The variable summary files consist of two columns. The first indicates<br />
boundary values for data bins and the second gives the number of data<br />
points in each bin. Because a precision of 3 was specified in the<br />
example, each boundary value has three decimal places. The boundary<br />
values are determined dynamically by <font color='green'>checkvar</font> and often do not<br />
correspond to data values in the input file, even if the<br />
<font color='green'>checkvar</font> and data file precisions are the same.<br />
<br />
The first data bin in <font color='green'>latitude.lst</font> contains data points in the<br />
range -83.224 (inclusive) to -69.274 (exclusive); neither boundary<br />
number exists in <font color='green'>latlon.dat</font>. The first bin has one data point,<br />
-83.223548. The fourth data bin contains latitude values from -63.212<br />
(inclusive) to -63.003 (exclusive), again with neither boundary value<br />
occurring in the data file. The data point in the fourth bin is<br />
-63.211962.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/datachk&diff=2035Wiki Testing/datachk2008-02-06T17:17:58Z<p>Yuan: /* Generating the Summaries */</p>
<hr />
<div>=Data Checking=<br />
<br />
<br />
The FreeForm ND-based utility program <font color='green'>checkvar</font> creates variable<br />
summary files, lists of maximum and minimum values, and summaries of<br />
processing activity. You can use this information to check data<br />
quality and to examine the distribution of the data.<br />
<br />
<br />
<br />
==Generating the Summaries==<br />
<br />
<br />
<br />
A variable summary file (or list file), which contains histogram<br />
information showing the variable's distribution in the data file, is<br />
created for each variable (or designated variables) in the specified<br />
data file. You can optionally specify an output file in which a<br />
summary of processing activity is saved.<br />
<br />
Variable summaries (list files) can be helpful for performing quality<br />
control checks of data. For example, you could run <font color='green'>checkvar</font> on<br />
an ASCII file, convert the file to binary, and then run <font color='green'>checkvar</font><br />
on the binary file. The output from <font color='green'>checkvar</font> should be the same<br />
for both the ASCII and binary files. You can also use variable<br />
summaries to look at the data distribution in a data set before<br />
extracting data.<br />
<br />
The <font color='green'>checkvar</font> command has the following form:<br />
<br />
<pre><br />
<br />
checkvar input_file [-f format_file] [-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-p precision] [-m maxbins] [-md missing_data_flag] [-mm] [-o processing_summary]<br />
<br />
</pre><br />
<br />
<br />
The <font color='green'>checkvar</font> program needs to find only an input format<br />
description. Output format descriptions will be ignored. If conversion<br />
variables are included in input or output formats, no conversion is<br />
performed when you run <font color='green'>checkvar</font>, since it ignores output<br />
formats.<br />
<br />
For descriptions of the standard arguments (first eleven arguments<br />
above), see ([[Wiki_Testing/convs|Section 8.6]).<br />
<br />
<blockquote><br />
<br />
; -p precision:<br />
Option flag followed by the number of decimal places. The number<br />
represents the power of 10 that data is multiplied by prior to<br />
binning. A value of 0 bins on one's, 1 on tenth's, and so on. This<br />
option allows an adjustment of the resolution of the <font color='green'>checkvar</font><br />
output.<br />
The default is 0; maximum is 5.<br />
<br />
<br />
<blockquote>NOTE:If you use the <font color='green'>-p</font> option on the command line, the precision set in the relevant format file is overridden. The precision in<br />
the format file serves as the default. </blockquote><br />
<br />
; -m maxbins:<br />
Option flag followed by the approximate maximum number of bins<br />
desired in <font color='green'>checkvar</font> output. The <font color='green'>checkvar</font> program keeps track of the number of bins filled as the data is processed. The<br />
smaller the number of bins, the faster <font color='green'>checkvar</font> runs. By<br />
keeping the number of bins small, you can check the gross aspects of<br />
data distribution rather than the details. The number of bins is adjusted dynamically as <font color='green'>checkvar</font> runs depending on the distribution of data in the input file. If the number of filled bins becomes > 1.5 * maxbins, the width of the bins is doubled to keep the total number near the desired maximum. The default is 100 bins; minimum is 6. Must be < 10,000.<br />
<br />
<br />
<blockquote>NOTE: The precision (-p) and maxbins (-m) options have no effect on<br />
character variables. </blockquote><br />
<br />
<br />
; -md missing_data_flag:<br />
Option flag followed by a flag value that <font color='green'>checkvar</font> should<br />
ignore across all variables in creating histogram data. Missing data flags are used in a data file to indicate missing or meaningless data. If you want <font color='green'>checkvar</font> to ignore more than one value, use the query (<font color='green'>-q</font>) option in conjunction with the variable file (<font color='green'>-v</font>) option.<br />
<br />
; -mm :<br />
Option flag indicating that only the maximum and minimum values of variables are calculated and displayed in the processing summary. Variable summary files are not created.<br />
<br />
; -o :<br />
Option flag followed by the name of the file in which summary information displayed during processing is stored.<br />
<br />
</blockquote><br />
<br />
==Example==<br />
<br />
<br />
You will use <font color='green'>checkvar</font> with a precision of 3 to create a<br />
processing summary file and summary files for the two variables<br />
latitude and longitude in the file <font color='green'>latlon.dat</font>.<br />
<br />
Here is <font color='green'>latlon.dat</font>:<br />
<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
To create the summary files, enter the following command:<br />
<br />
<pre><br />
checkvar latlon.dat -p 3 -o latlon.sum<br />
</pre><br />
<br />
A summary of processing information and the maximum and minimum for<br />
each variable are displayed on the screen. The following three files<br />
are created:<br />
<br />
<br />
*<font color='green'>latlon.sum</font> recaps processing activity, maximums and minimums<br />
*<font color='green'>latitude.lst</font> shows distribution of the latitude values in<br />
<font color='green'>latlon.dat</font><br />
*<font color='green'>longitude.lst</font> shows distribution of the longitude values<br />
in <font color='green'>latlon.dat</font>.<br />
<br />
==Interpreting the Summaries==<br />
<br />
<br />
The processing and variable summary files output by <font color='green'>checkvar</font><br />
from the example in the previous section are shown and discussed<br />
below.<br />
<br />
===Processing Summary===<br />
<br />
<br />
If you specify an output file on the command line, it stores the<br />
information that is displayed on the screen during processing. The<br />
file <font color='green'>latlon.sum</font> was specified as the output file in the example<br />
above.<br />
<br />
Here is <font color='green'>latlon.sum</font>:<br />
<br />
<pre><br />
Input file : latlon.dat<br />
Requested precision = 3, Approximate number of sorting bins = 100<br />
<br />
Input data format (latlon.fmt)<br />
ASCII_input_data "ASCII format"<br />
The format contains 2 variables; length is 24.<br />
<br />
Output data format (latlon.fmt)<br />
binary_output_data "binary format"<br />
The format contains 2 variables; length is 16.<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
latitude: 20 values read<br />
minimum: -83.223548 found at record 5<br />
maximum: 77.883119 found at record 12<br />
Summary file: latitude.lst<br />
<br />
Histogram data precision: 3, Number of sorting bins: 20<br />
longitude: 20 values read<br />
minimum: -176.161101 found at record 1<br />
maximum: 149.408117 found at record 4<br />
Summary file: longitude.lst.<br />
</pre><br />
<br />
<br />
The processing summary file <font color='green'>latlon.sum</font> first shows the name of<br />
the input data file (<font color='green'>latlon.dat</font>). If you specified precision and<br />
a maximum number of bins on the command line, those values are given<br />
as Requested precision, in this case 3, and Approximate number of<br />
sorting bins, in this case the default value of 100. If precision is<br />
not specified, No requested precision is shown.<br />
<br />
A summary of each format shows the type of format (in this case, Input<br />
data format and Output data format) and the name of the format file<br />
containing the format descriptions (<font color='green'>latlon.fmt</font>), whether<br />
specified on the command line or located through the default search<br />
sequence (as detailed in chapter 4). In this case, it was located by<br />
default. Since <font color='green'>checkvar</font> only needs an input format description,<br />
it ignores output format descriptions. Next, you see the format<br />
descriptor as resolved by FreeForm ND (e.g., <font color='green'>ASCII_input_data</font>)<br />
and the format title (e.g., "ASCII format"). Then the number of<br />
variables in a record and total record length are given; for ASCII,<br />
record length includes the end-of-line character (1 byte for Unix).<br />
<br />
A section for each variable processed by <font color='green'>checkvar</font> indicates the<br />
histogram precision and actual number of sorting bins. Under some<br />
circumstances, the precision of values in the histogram file may be<br />
different than the precision you specified on the command line. The<br />
default value for precision, if none is specified on the command line,<br />
is the precision specified in the relevant format description file or<br />
5, whichever is smaller. The second line shows the name of the<br />
variable (latitude, longitude) and the number of values in the data<br />
file for the variable (20 for both latitude and longitude).<br />
<br />
The minimum and maximum values for the variable are shown next<br />
(-83.223548 is the minimum and 77.883119 is the maximum value for<br />
latitude). The maximum and minimum values are given here with a<br />
precision of 6, which is the precision specified in the format<br />
description file. The locations of the maximum and minimum values in<br />
the input file are indicated. (-83.223548 is the fifth latitude value<br />
in <font color='green'>latlon.dat</font> and 77.883119 is the twelfth). Finally, the name<br />
of the histogram data (or variable summary) file generated for each<br />
variable is given (<font color='green'>latitude.lst</font> and <font color='green'>longitude.lst</font>).<br />
<br />
===Variable Summaries===<br />
<br />
<br />
The name of each variable summary file (list file) output by<br />
<font color='green'>checkvar</font> is of the form <font color='green'>variable.lst</font> for numeric variables<br />
and <font color='green'>variable.cst</font> for character variables. The data in<br />
*<font color='green'>.lst</font>, and *<font color='green'>.cst</font> files can be loaded into histogram plot<br />
programs for graphical representation. (You must be familiar enough<br />
with your program of choice to manipulate the data as necessary in<br />
order to achieve the desired result.) In Unix, there is no need to<br />
abbreviate the base file name.<br />
<br />
<blockquote>If you use the -v option, the order of variables in var_file has<br />
no effect on the numbering of base file names of the variable<br />
summary files. </blockquote><br />
<br />
The two example variable summary files, <font color='green'>latitude.lst</font> and<br />
<font color='green'>longitude.lst</font>, are shown next.<br />
<br />
<br />
{| <br />
|<br />
{| border="2"<br />
| <font color='green'>latitude.lst</font> <br />
|-<br />
| -83.224 1 <br />
|-<br />
| -69.274 1<br />
|-<br />
| -65.865 1<br />
|-<br />
| -63.212 1<br />
|-<br />
| -63.003 1<br />
|-<br />
| -47.304 1<br />
|-<br />
| -34.578 1<br />
|-<br />
| -28.287 1<br />
|-<br />
| -12.967 1<br />
|-<br />
| -0.929 1<br />
|-<br />
| 11.624 1<br />
|-<br />
| 12.588 1<br />
|-<br />
| 27.331 1<br />
|-<br />
| 29.918 1<br />
|-<br />
| 35.086 1<br />
|-<br />
| 35.130 1<br />
|-<br />
| 38.818 1<br />
|-<br />
| 54.118 1<br />
|-<br />
| 77.652 1<br />
|-<br />
| 77.883 1<br />
<br />
|}<br />
||<br />
{| border="2"<br />
| <font color='green'>longitude.lst</font><br />
|-<br />
| -176.162 1<br />
|-<br />
| -155.234 1<br />
|-<br />
| -153.544 1<br />
|-<br />
| -136.941 1<br />
|-<br />
| -113.661 1<br />
|-<br />
| -79.178 1<br />
|-<br />
| -77.506 1<br />
|-<br />
| -55.442 1<br />
|-<br />
| -21.644 1<br />
|-<br />
| 0.777 1<br />
|-<br />
| 30.172 1<br />
|-<br />
| 35.591 1<br />
|-<br />
| 36.356 1<br />
|-<br />
| 38.875 1<br />
|-<br />
| 55.319 1<br />
|-<br />
| 62.152 1<br />
|-<br />
| 91.411 1<br />
|-<br />
| 134.124 1<br />
|-<br />
| 144.804 1<br />
|-<br />
| 149.408 1<br />
<br />
|}<br />
|}<br />
<br />
The variable summary files consist of two columns. The first indicates<br />
boundary values for data bins and the second gives the number of data<br />
points in each bin. Because a precision of 3 was specified in the<br />
example, each boundary value has three decimal places. The boundary<br />
values are determined dynamically by <font color='green'>checkvar</font> and often do not<br />
correspond to data values in the input file, even if the<br />
<font color='green'>checkvar</font> and data file precisions are the same.<br />
<br />
The first data bin in <font color='green'>latitude.lst</font> contains data points in the<br />
range -83.224 (inclusive) to -69.274 (exclusive); neither boundary<br />
number exists in <font color='green'>latlon.dat</font>. The first bin has one data point,<br />
-83.223548. The fourth data bin contains latitude values from -63.212<br />
(inclusive) to -63.003 (exclusive), again with neither boundary value<br />
occurring in the data file. The data point in the fourth bin is<br />
-63.211962.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2034Wiki Testing/fmtconv2008-02-06T17:05:32Z<p>Yuan: /* File Names and Context */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry -o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font> back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in <font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>: The following output appears on the screen: <br />
<pre> <br />
54.118314 -136.940570<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
35.130219 -153.543091<br />
35.086084 -21.643402<br />
</pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([[Wiki_Testing/convs| Chapter 8]] ).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude values in the ASCII format given in the format description file <font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>. This provides the context needed for FreeForm ND programs to determine which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the input format in <font color='green'>ll.fmt</font> as the output format. Use the following command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually used. Since <font color='green'>ASCII_input_data</font> was used as the output format, <font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file <font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are described in <font color='green'>latlon.fmt</font>. The data file names do not provide the context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs) which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in <font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2033Wiki Testing/fmtconv2008-02-06T17:02:31Z<p>Yuan: /* Including a Query */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry -o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font> back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in <font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>: The following output appears on the screen: <br />
<pre> <br />
54.118314 -136.940570<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
35.130219 -153.543091<br />
35.086084 -21.643402<br />
</pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2032Wiki Testing/fmtconv2008-02-06T17:02:16Z<p>Yuan: /* Including a Query */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry -o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font> back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in <font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <br />
<pre> <br />
54.118314 -136.940570<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
35.130219 -153.543091<br />
35.086084 -21.643402<br />
</pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2031Wiki Testing/fmtconv2008-02-06T17:01:38Z<p>Yuan: /* Including a Query */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry -o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font> back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2030Wiki Testing/fmtconv2008-02-06T17:01:21Z<p>Yuan: /* Including a Query */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry -o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2029Wiki Testing/fmtconv2008-02-06T17:00:19Z<p>Yuan: /* Reading the Binary File */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2028Wiki Testing/fmtconv2008-02-06T16:59:39Z<p>Yuan: /* Reading the Binary File */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each<br />
value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2027Wiki Testing/fmtconv2008-02-06T16:58:44Z<p>Yuan: /* Reading the Binary File */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter d to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
<br />
If desired, you can enter the commands to readfile from an input command file rather than directly from the command line. The example command file latlon.in is shown next.<br />
<br />
Here is latlon.in:<br />
<br />
<pre><br />
llllllp0 llPq <br />
</pre><br />
<br />
The 6 l's (l for long) cause the first 6 values in the file to be displayed. The sequence p0 causes a return to the top (position 0) of the file. A position number (0) must be followed by a blank space. The 2 l's display the first two values again. The P displays the current file position and length, and q closes readfile.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in <br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision of the long variables in latlon2.fmt.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each<br />
value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2026Wiki Testing/fmtconv2008-02-06T16:57:26Z<p>Yuan: /* Reading the Binary File */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter d to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter l to view each<br />
value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The P displays the<br />
current file position and length, and q closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2025Wiki Testing/fmtconv2008-02-06T16:56:09Z<p>Yuan: /* Reading the Binary File */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter d to view each value (or press Return to view each value after the first).<br />
#Enter q to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2024Wiki Testing/fmtconv2008-02-06T16:54:54Z<p>Yuan: /* <font color='green'>readfile</font> */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing b. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type s or l to swap short or long integers<br />
respectively, or type f or d to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type p. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type P to find out your current position in the file<br />
and total file length in bytes. Type q to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter \inp{d} to view each value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2023Wiki Testing/fmtconv2008-02-06T16:53:36Z<p>Yuan: /* <font color='green'>readfile</font> */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> ''binary_data_file''<br />
<br />
When the program starts, it shows the available options, shown in<br />
table 9.3. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<center><br />
The <font color='green'>readfile</font> program options<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
</center><br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing \inp{b}. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type \inp{s} or \inp{l} to swap short or long integers<br />
respectively, or type \inp{f} or \inp{d} to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type \inp{p}. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type \inp{P} to find out your current position in the file<br />
and total file length in bytes. Type \inp{q} to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter \inp{d} to view each value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2022Wiki Testing/fmtconv2008-02-06T16:52:33Z<p>Yuan: /* <font color='green'>chkform</font> */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([[Wiki_Testing/convs|Section 8.6]]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
; -ol log_file: Puts a log of processing information into the specified ''log_file''.<br />
<br />
; -el error_log_file: Creates an error log file that contains whatever error messages are issued by <font color='green'>chkform</font>.<br />
<br />
; -ep : In normal operation, <font color='green'>chkform</font> asks you to manually acknowledge each important error by typing something on the keyboard. If you use this option, <font color='green'>chkform</font> will not stop to prompt, but will continue processing until either the file is procesed, or there is an error preventing more processing.<br />
<br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> \var{binary_data_file}<br />
<br />
When the program starts, it shows the available options, shown in<br />
\tableref{ff,tab,readfileopts}. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<br />
\caption{The <font color='green'>readfile</font> program options}<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing \inp{b}. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type \inp{s} or \inp{l} to swap short or long integers<br />
respectively, or type \inp{f} or \inp{d} to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type \inp{p}. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type \inp{P} to find out your current position in the file<br />
and total file length in bytes. Type \inp{q} to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter \inp{d} to view each value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2021Wiki Testing/fmtconv2008-02-06T16:49:07Z<p>Yuan: /* <font color='green'>newform</font> */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> ''input_file'' [-f format_file] [-if-if input_format_file] [-of output_format_file]<br />
<br />
[-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
For descriptions of the arguments, see ([[Wiki_Testing/convs|Section 9.3]]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/fmtconv|Section 9.3]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([http://www <cite> ff,convs,argument</cite>]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
<blockquote><br />
; \dashol : Puts a log of processing information into the specified<br />
\var{log_file}.<br />
<br />
; \dashel : Creates an error log file that contains whatever error<br />
messages are issued by <font color='green'>chkform</font>.<br />
<br />
; \dashep : In normal operation, <font color='green'>chkform</font> asks you to manually<br />
acknowledge each important error by typing something on the<br />
keyboard. If you use this option, <font color='green'>chkform</font> will not stop to<br />
prompt, but will continue processing until either the file is<br />
procesed, or there is an error preventing more processing.<br />
</blockquote><br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> \var{binary_data_file}<br />
<br />
When the program starts, it shows the available options, shown in<br />
\tableref{ff,tab,readfileopts}. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<br />
\caption{The <font color='green'>readfile</font> program options}<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing \inp{b}. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type \inp{s} or \inp{l} to swap short or long integers<br />
respectively, or type \inp{f} or \inp{d} to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type \inp{p}. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type \inp{P} to find out your current position in the file<br />
and total file length in bytes. Type \inp{q} to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter \inp{d} to view each value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/fmtconv&diff=2020Wiki Testing/fmtconv2008-02-06T16:46:22Z<p>Yuan: /* <font color='green'>newform</font> */</p>
<hr />
<div>=Format Conversion=<br />
<br />
<br />
The FreeForm ND utility program newform lets you convert data from one<br />
format to another. This allows you to pass data to applications in the<br />
format they require. You may also want to create binary archives for<br />
efficient data storage and access. With newform, conversion of ASCII<br />
data to binary format is straightforward. If you wish to read the data<br />
in a binary file, you can convert it to ASCII with newform, or use the<br />
interactive program readfile. You can also convert data from one ASCII<br />
format to another ASCII format with newform.<br />
<br />
==<font color='green'>newform</font>==<br />
<br />
<br />
The FreeForm ND-based program <font color='green'>newform</font>[[Wiki_Testing/Footnotes|(4)]] is a general tool for changing the format ofa data file. The only required command line argument, if you use FreeForm ND naming conventions, is the name of the input data file.<br />
The reformatted data is written to standard output (the screen) unless<br />
you specify an output file. If you reformat to binary, you will<br />
generally want to store the output in a file.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved in a conversion before you<br />
can use <font color='green'>newform</font> to perform the conversion. The standard<br />
extension for format description files is <font color='green'>.fmt</font>. If you do not<br />
explicitly specify the format description file on the command line,<br />
which is unnecessary if you use FreeForm ND naming conventions,<br />
<font color='green'>newform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([[Wiki_Testing/convs|Chapter 8]]).<br />
<br />
The <font color='green'>newform</font> command has the following form:<br />
<br />
<font color='green'>newform</font> \var{input_file} [\dashf] [\dashif] [\dashof]<br />
[\dashft] [\dashift] [\dashoft] [\dashb] [\dashc] [\dashvv] [\dashq]<br />
[\dasho]<br />
<br />
For descriptions of the arguments, see ([http://www <cite> ff,convs,argument</cite>]).<br />
<br />
If you want to convert an ASCII file to a binary file, and you follow<br />
the FreeForm ND naming conventions, the command is simply:<br />
<br />
<pre><br />
newform datafile.dat -o datafile.bin<br />
</pre><br />
<br />
where datafile is the file name of your choosing.<br />
<br />
If data files and format files are not in the current directory or in<br />
the same directory, you can specify the appropriate path name. For<br />
example, if the input data file is not in the current directory, you<br />
can enter:<br />
<br />
<pre><br />
newform /path/datafile.dat -o datafile.bin<br />
</pre><br />
<br />
<br />
To read the data in the resulting binary file, you can reformat back<br />
to ASCII using the command:<br />
<br />
<pre><br />
newform datafile.bin -o datafile.ext<br />
</pre><br />
<br />
or you can use the <font color='green'>readfile</font> program, described in<br />
([[Wiki_Testing/convs|Section 8.6]]).<br />
<br />
==<font color='green'>chkform</font>==<br />
<br />
<br />
<br />
Though <font color='green'>newform</font> is useful for checking data formats, it is<br />
limited by requiring a format file to specify an output format. Since<br />
some OPeNDAP FreeForm ND applications (such as the OPeNDAP FreeForm handler) do not<br />
require an output format, this is extra work for the dataset<br />
administrator. For these occasions, OPeNDAP FreeForm ND provides a simpler<br />
format-checking program, called <font color='green'>chkform</font>.<br />
<br />
The <font color='green'>chkform</font> program attempts to read an ASCII file, using the<br />
specified input format. If the format allows the file to be read<br />
<br />
properly, <font color='green'>chkform</font> says so. However, if the input format<br />
contains errors, or does not accurately reflect the contents of the<br />
given data file, <font color='green'>chkform</font> delivers an error message, and attempts<br />
to provide a rudimentary diagnosis of the problem.<br />
<br />
You must create a format description file (or files) with format<br />
descriptions for the data files involved before you can use<br />
<font color='green'>chkform</font> to chack the format. As with <font color='green'>newform</font>, the<br />
standard extension for format description files is <font color='green'>.fmt</font>. If you<br />
do not explicitly specify the format description file on the command<br />
line (unnecessary if you use FreeForm ND naming conventions)<br />
<font color='green'>chkform</font> follows the FreeForm ND search sequence to find a format<br />
file.<br />
<br />
For details about FreeForm ND naming conventions and the search<br />
sequence, see ([http://www <cite> ff,convs</cite>]).<br />
<br />
The <font color='green'>chkform</font> command has the following form:<br />
<br />
<pre><br />
chkform input_file [-if input_format_file] [-ift "title"] [-b local_buffer_size]<br />
[-c count] [-q query_file] [-ol log_file] [-el error_log_file] [-ep]<br />
</pre><br />
<br />
Most of the arguments are described in<br />
([http://www <cite> ff,convs,argument</cite>]). The following are specific to<br />
<font color='green'>chkform</font>:<br />
<br />
<blockquote><br />
; \dashol : Puts a log of processing information into the specified<br />
\var{log_file}.<br />
<br />
; \dashel : Creates an error log file that contains whatever error<br />
messages are issued by <font color='green'>chkform</font>.<br />
<br />
; \dashep : In normal operation, <font color='green'>chkform</font> asks you to manually<br />
acknowledge each important error by typing something on the<br />
keyboard. If you use this option, <font color='green'>chkform</font> will not stop to<br />
prompt, but will continue processing until either the file is<br />
procesed, or there is an error preventing more processing.<br />
</blockquote><br />
<br />
As in the above examples, if you have an ASCII data file called<br />
<font color='green'>datafile.dat</font>, supposedly described in a format file called<br />
<font color='green'>datafile.fmt</font>, you can use <font color='green'>chkform</font> like this:<br />
<br />
<pre><br />
chkform datafile.dat<br />
</pre><br />
<br />
If processing is successful, you will see something like the<br />
following:<br />
<br />
<pre><br />
<br />
Welcome to Chkform release 4.2.3 -- an NGDC FreeForm ND application<br />
<br />
<br />
(llmaxmin.fmt) ASCII_input_file_header "Latitude/Longitude Limits"<br />
File llmaxmin.dat contains 1 header record (71 bytes)<br />
Each record contains 6 fields and is 71 characters long.<br />
<br />
(llmaxmin.fmt) ASCII_input_data "lat/lon"<br />
File llmaxmin.dat contains 10 data records (230 bytes)<br />
Each record contains 3 fields and is 23 characters long.<br />
<br />
100<br />
<br />
No errors found (11 lines checked)<br />
</pre><br />
<br />
<br />
==<font color='green'>readfile</font>==<br />
<br />
<br />
FreeForm ND includes <font color='green'>readfile</font>, a simple interactive binary file<br />
reader. The program has one required command line argument, the name<br />
of the file to be read. You do not have to write format descriptions<br />
to use <font color='green'>readfile</font>.<br />
<br />
The <font color='green'>readfile</font> command has the following form:<br />
<br />
<font color='green'>readfile</font> \var{binary_data_file}<br />
<br />
When the program starts, it shows the available options, shown in<br />
\tableref{ff,tab,readfileopts}. At the <font color='green'>readfile</font> prompt, type<br />
these option codes to view binary encoded values. (Pressing return<br />
repeats the last option.)<br />
<br />
<br />
\caption{The <font color='green'>readfile</font> program options}<br />
<br />
{| border="1" <br />
| <font color='green'>c</font> || char --- 1 byte character<br />
|-<br />
| <font color='green'>s</font> || short --- 2 byte signed integer<br />
|-<br />
| <font color='green'>l</font> || long --- 4 byte signed integer<br />
|-<br />
| <font color='green'>f</font> || float --- 4 byte single-precision floating point<br />
|-<br />
| <font color='green'>d</font> || double --- 8 byte double-precision floating point<br />
|-<br />
| <font color='green'>uc</font> || uchar --- 1 byte unsigned integer<br />
|-<br />
| <font color='green'>us</font> || ushort --- 2 byte unsigned integer<br />
|-<br />
| <font color='green'>ul</font> || ulong --- 4 byte unsigned integer<br />
|-<br />
| <font color='green'>b</font> || Toggle between "big-endian" and your machine's native byte<br />
order<br />
|-<br />
<font color='green'>p</font> || Set new file position<br />
|-<br />
| <font color='green'>P</font> ||Show present file position and length<br />
|-<br />
| <font color='green'>h</font> || Display this help screen<br />
|-<br />
| <font color='green'>q</font> || Quit<br />
<br />
|}<br />
<br />
<br />
The options let you interactively read your way through the specified<br />
binary file. The first position in the file is 0. You must type the<br />
character(s) indicating variable type (e.g., us for unsigned short) to<br />
view each value, so you need to know the data types of variables in<br />
the file and the order in which they occur. If successive variables<br />
are of the same type, you can press Return to view each value after<br />
the first of that type.<br />
<br />
You can toggle the byte-order switch on and off by typing \inp{b}. The<br />
byte-order option is used to read a binary data file that requires<br />
byte swapping. This is the case when you need cross-platform access to<br />
a file that is not byte-swapped, for example, if you are on a Unix<br />
machine reading data from a CD-ROM formatted for a PC. When the switch<br />
is on, type \inp{s} or \inp{l} to swap short or long integers<br />
respectively, or type \inp{f} or \inp{d} to swap floats or doubles.<br />
The <font color='green'>readfile</font> program does not byte swap the file itself (the<br />
file is unchanged) but byte swaps the data values internally for<br />
display purposes only.<br />
<br />
To go to another position in the file, type \inp{p}. You are prompted<br />
to enter the new file position in bytes. If, for example, each value<br />
in the file is 4 bytes long and you type 16, you will be positioned at<br />
the first byte of the fifth value. If you split fields (by not<br />
repositioning at the beginning of a field), the results will probably<br />
be garbage. Type \inp{P} to find out your current position in the file<br />
and total file length in bytes. Type \inp{q} to exit from<br />
<font color='green'>readfile</font>.<br />
<br />
You can also use an input command file rather than entering commands<br />
directly. In that case, the <font color='green'>readfile</font> command has the following<br />
form:<br />
<br />
<pre><br />
readfile binary_data_file < input_command_file<br />
</pre><br />
<br />
<br />
==Creating a Binary Archive==<br />
<br />
<br />
By storing data files in binary, you save disk space and make access<br />
by applications more efficient. An ASCII data file can take two to<br />
five times the disk space of a comparable binary data file. Not only<br />
is there less information in each byte, but extra bytes are needed for<br />
decimal points, delimiters, and end-of-line markers.<br />
<br />
It is very easy to create a binary archive using <font color='green'>newform</font> as the<br />
following examples show. The input data for these examples are in the<br />
ASCII file <font color='green'>latlon.dat</font> (shown below). They consist of 20 random<br />
latitude and longitude values. The size of the file on a Unix system<br />
is 460 bytes.<br />
<br />
Here is the <font color='green'>latlon.dat</font> file:<br />
<br />
<pre><br />
-47.303545 -176.161101<br />
-0.928001 0.777265<br />
-28.286662 35.591879<br />
12.588231 149.408117<br />
-83.223548 55.319598<br />
54.118314 -136.940570<br />
38.818812 91.411330<br />
-34.577065 30.172129<br />
27.331551 -155.233735<br />
11.624981 -113.660611<br />
77.652742 -79.177679<br />
77.883119 -77.505502<br />
-65.864879 -55.441896<br />
-63.211962 134.124014<br />
35.130219 -153.543091<br />
29.918847 144.804390<br />
-69.273601 38.875778<br />
-63.002874 36.356024<br />
35.086084 -21.643402<br />
-12.966961 62.152266<br />
</pre><br />
<br />
<br />
===Simple ASCII to Binary Conversion===<br />
<br />
In this example, you will use <font color='green'>newform</font> to convert the ASCII data<br />
file <font color='green'>latlon.dat</font> into the binary file <font color='green'>latlon.bin</font>. The input<br />
and output data formats are described in <font color='green'>latlon.fmt</font>.<br />
<br />
Here is the <font color='green'>latlon.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.bin<br />
/ and latlon.dat. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
binary_data "binary format"<br />
latitude 1 8 double 6<br />
longitude 9 16 double 6<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
</pre><br />
<br />
The binary and ASCII variables both have the same names. The binary<br />
variable latitude occupies positions 1 to 8 and longitude occupies<br />
positions 9-16. The corresponding ASCII variables occupy positions<br />
1-10 and 12-22. Both the binary and ASCII variables are stored as<br />
doubles and have a precision of 6.<br />
<br />
===Converting to Binary===<br />
<br />
To convert from an ASCII representation of the numbers in<br />
<font color='green'>latlon.dat</font> to a binary representation:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example files.<br />
#Enter the following command: <br />
<br />
<pre> newform latlon.dat -o latlon.bin </pre><br />
<br />
Because FreeForm ND filenaming conventions have been used,<br />
<font color='green'>newform</font> will locate and use <font color='green'>latlon.fmt</font> for the<br />
translation. The <font color='green'>newform</font> program creates a new data file<br />
(effectively a binary archive) called <font color='green'>latlon.bin</font>. The size of<br />
the archive file is 2/3 the size of <font color='green'>latlon.dat</font>. Additionally,<br />
the data do not have to be converted to machine-readable<br />
representation by applications.<br />
<br />
There are two methods for checking the data in <font color='green'>latlon.bin</font> to<br />
make sure they converted correctly. You can reformat back to ASCII and<br />
view the resulting file, or use <font color='green'>readfile</font> to read<br />
<font color='green'>latlon.bin</font>.<br />
<br />
===Reconverting to Native Format===<br />
<br />
<br />
Use the following <font color='green'>newform</font> command to reformat the binary data in<br />
<font color='green'>latlon.bin</font> to its native ASCII format:<br />
<br />
<pre><br />
newform latlon.bin -o latlon.rf<br />
</pre><br />
<br />
The ASCII file <font color='green'>latlon.rf</font> matches (but does not overwrite) the<br />
original input file <font color='green'>latlon.dat</font>. You can confirm this by using a<br />
file comparison utility. The <font color='green'>diff</font> command is generally available on<br />
Unix platforms.<br />
<br />
To use <font color='green'>diff</font> to compare the <font color='green'>latlon</font> ASCII files, enter the<br />
command:<br />
<br />
<pre><br />
diff latlon.dat latlon.rf<br />
</pre><br />
<br />
The output should be something along these lines:<br />
<br />
<pre><br />
Files are effectively identical.<br />
</pre><br />
<br />
Several implementations of the <font color='green'>diff</font> utility don't print anything<br />
if the two input files are identical.<br />
<br />
<br />
<blockquote>NOTE: The <font color='green'>diff</font> utility may detect a difference in other similar<br />
cases because FreeForm ND adds a leading zero in front of a decimal<br />
and interprets a blank as a zero if the field is described as a<br />
number. (A blank described as a character is interpreted as a<br />
blank.) </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
<br />
To use <font color='green'>readfile</font> to read the data in <font color='green'>latlon.bin</font>:<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon.bin </pre><br />
#The data are stored as doubles, so enter \inp{d} to view each value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
===Conversion to a More Portable Binary===<br />
<br />
In this example, you will use <font color='green'>newform</font> to reformat the latitude<br />
and longitude values in the ASCII data file <font color='green'>latlon.dat</font> into<br />
binary longs in the binary file <font color='green'>latlon2.bin</font>. The input and<br />
output data formats are described in <font color='green'>latlon2.fmt</font>.<br />
<br />
This is what's in <font color='green'>latlon2.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for data files latlon.dat<br />
/ and latlon2.bin. Each record in both files contains two fields,<br />
/ latitude and longitude.<br />
<br />
ASCII_data "ASCII format"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_data "binary format"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The ASCII and binary variables both have the same names. The ASCII<br />
variable latitude occupies positions 1-10 and longitude occupies<br />
positions 12-22. The ASCII variables are defined to be of type double.<br />
The binary variables occupy four bytes each (positions 1-4 and 5-8)<br />
and are of type long. The precision for all is 6.<br />
<br />
===Converting to Binary Long===<br />
<br />
In the previous example, both the ASCII and binary variables were<br />
defined to be doubles. Binary longs, which are 4-byte integers, may be<br />
more portable across different platforms than binary doubles or<br />
floats.<br />
<br />
To convert the ASCII data in <font color='green'>latlon.dat</font> to binary longs:<br />
<br />
<br />
#Change to the directory that contains the FreeForm ND example<br />
files.<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -o latlon2.bin </pre><br />
<br />
It creates the binary archive file <font color='green'>latlon2.bin</font> with the 20<br />
latitude and longitude values in <font color='green'>latlon.dat</font> stored as binary<br />
longs.<br />
<br />
This example duplicates one in chapter 2. If you completed that<br />
example, an error message will indicate that <font color='green'>latlon2.bin</font> exists.<br />
You can rename, move, or delete the existing file.<br />
<br />
The size of the archive file <font color='green'>latlon2.bin</font> is about 1/3 the size<br />
of <font color='green'>latlon.dat</font>. Also, the data do not have to be converted to<br />
machine representation by applications. The main tradeoff in achieving<br />
savings in space and access time is that although binary longs are<br />
more portable than binary doubles or floats, any binary representation<br />
is less portable than ASCII.<br />
<br />
<blockquote>CAUTION: There may be a loss of precision when input data of<br />
type double is converted to long. </blockquote><br />
<br />
<br />
===Reading the Binary File===<br />
<br />
Once again, you can use <font color='green'>readfile</font> to check the data in the binary<br />
archive you created.<br />
<br />
<br />
#Enter the following command: <pre> readfile latlon2.bin </pre><br />
#The data are stored as longs, so enter \inp{l} to view each<br />
value (or press Return to view each value after the first).<br />
#Enter \inp{q} to quit <font color='green'>readfile</font>.<br />
<br />
If desired, you can enter the commands to <font color='green'>readfile</font> from an input<br />
command file rather than directly from the command line. The example<br />
command file <font color='green'>latlon.in</font> is shown next.<br />
<br />
Here is <font color='green'>latlon.in</font>:<br />
<br />
<pre><br />
llllllp0 llPq<br />
</pre><br />
<br />
The 6 l's (l for <font color='green'>long</font>) cause the first 6 values in the file to be<br />
displayed. The sequence p0 causes a return to the top (position 0) of<br />
the file. A position number (0) must be followed by a blank space. The<br />
2 l's display the first two values again. The \inp{P} displays the<br />
current file position and length, and \inp{q} closes <font color='green'>readfile</font>.<br />
<br />
If you enter the following command:<br />
<br />
<pre><br />
readfile latlon2.bin < latlon.in<br />
</pre><br />
<br />
you should see the following output on the screen:<br />
<br />
<pre><br />
long: -47303545<br />
long: -176161101<br />
long: -928001<br />
long: 777265<br />
long: -28286662<br />
long: 35591879<br />
New File Position = 0<br />
long: -47303545<br />
long: -176161101<br />
File Position: 8 File Length: 160<br />
</pre><br />
<br />
The floating point numbers have been multiplied by 106, the precision<br />
of the long variables in <font color='green'>latlon2.fmt</font>.<br />
<br />
===Including a Query===<br />
<br />
You can use the query option (<font color='green'>-q</font> \var{query_file}) to specify<br />
exactly which records in the data file <font color='green'>newform</font> should process.<br />
The query file contains query criteria. Query syntax is summarized in<br />
Appendix C.<br />
<br />
In this example, you will specify a query so that <font color='green'>newform</font> will<br />
reformat only those value pairs in <font color='green'>latlon.dat</font> where latitude is<br />
positive and longitude is negative into the binary file<br />
<font color='green'>llposneg.bin</font>. The input and output data formats are described in<br />
<font color='green'>latlon2.fmt</font>.<br />
<br />
The query criteria are specified in the following file, called<br />
<font color='green'>llposneg.qry</font>:<br />
<br />
<pre><br />
[latitude] > 0 & [longitude] < 0<br />
</pre><br />
<br />
To convert the desired data in <font color='green'>latlon.dat</font> to binary and then<br />
view the results:<br />
<br />
<br />
#Enter the following command: <pre> newform latlon.dat -f latlon2.fmt -q llposneg.qry<br />
-o llposneg.bin </pre> The <font color='green'>llposneg.bin</font> file now contains the positive/negative latitude/longitude pairs in binary form.<br />
#To view the data, first convert the data in <font color='green'>llposneg.bin</font><br />
back to ASCII format: newform llposneg.bin -f latlon2.fmt -o llposneg.dat<br />
#Enter the appropriate command to display the data in<br />
<font color='green'>llposneg.dat</font>, e.g. <font color='green'>more</font>:<br />
The following output appears on the screen: <pre> 54.118314 -136.940570 27.331551 -155.233735 11.624981 -113.660611 77.652742 -79.177679 77.883119 -77.505502 35.130219 -153.543091 35.086084 -21.643402 </pre> <br />
<br />
<blockquote>NOTE: As demonstrated in the examples above, you can check the data in<br />
a binary file either by using <font color='green'>readfile</font> or by converting the<br />
data back to ASCII using <font color='green'>newform</font> and then viewing it. </blockquote><br />
<br />
<br />
==File Names and Context==<br />
<br />
<br />
In the preceding examples, the read/write type (input or output) was<br />
not included in the format descriptors (<font color='green'>ASCII_data</font> and<br />
<font color='green'>binary_data</font>). FreeForm ND naming conventions were used, so<br />
<font color='green'>newform</font> can determine from the context which format should be<br />
used for input and which for output. Consider the command:<br />
<br />
<pre><br />
newform latlon.dat -o latlon.bin<br />
</pre><br />
<br />
The input file extension is <font color='green'>.dat</font> and the output file extension<br />
is <font color='green'>.bin</font>. These extensions provide context indicating that ASCII<br />
should be used as the input format and binary should be used as the<br />
output format. The format description file that <font color='green'>newform</font> will<br />
look for is the file with the same name as the input file and the<br />
extension <font color='green'>.fmt</font>, i.e., <font color='green'>latlon.fmt</font>.<br />
<br />
If you use the following command:<br />
<br />
<pre><br />
newform latlon.bin<br />
</pre><br />
<br />
to translate the binary archive <font color='green'>latlon.bin</font> back to ASCII,<br />
<font color='green'>newform</font> identifies the input format as binary and uses the ASCII<br />
format for output. The ASCII data is written to the screen because an<br />
output file was not specified.<br />
<br />
For information about FreeForm ND file name conventions, see<br />
([http://www <cite> ff,convs</cite>]).<br />
<br />
==="Nonstandard" Data File Names===<br />
<br />
<br />
If you are working with data files that do not use FreeForm ND naming<br />
conventions, you need to more explicitly define the context. For<br />
example, the files <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, <font color='green'>lldat3.ll</font>,<br />
<font color='green'>lldat4.ll</font>, and <font color='green'>lldat5.ll</font> all have latitude and longitude<br />
values in the ASCII format given in the format description file<br />
<font color='green'>lldat.fmt</font>. If you wanted to archive these files in binary<br />
format, you could not use a command of the form used in the previous<br />
examples, i.e., <font color='green'>newform</font> <font color='green'>datafile.dat</font> -o <font color='green'>datafile.bin</font><br />
with <font color='green'>datafile.fmt</font> as the default format description file.<br />
<br />
First, the ASCII data files do not have the extension <font color='green'>.dat</font>,<br />
which identifies them as ASCII files. Second, you would need five<br />
separate format description files, all with the same content:<br />
<font color='green'>lldat1.fmt</font>, <font color='green'>lldat2.fmt</font>, <font color='green'>lldat3.fmt</font>,<br />
<font color='green'>lldat4.fmt</font>, and <font color='green'>lldat5.fmt</font>. Creating the format<br />
description file <font color='green'>ll.fmt</font> solves both problems.<br />
<br />
Here is the <font color='green'>ll.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file that describes latlon<br />
/ data in files with the extension .ll<br />
<br />
ASCII_input_data "ASCII format for .ll latlon data"<br />
latitude 1 10 double 6<br />
longitude 12 22 double 6<br />
<br />
binary_output_data "binary format for .ll latlon data"<br />
latitude 1 4 long 6<br />
longitude 5 8 long 6<br />
</pre><br />
<br />
The name used for the format description file, <font color='green'>ll.fmt</font>, follows<br />
the FreeForm ND convention that one format description file can be<br />
utilized for multiple data files, all with the same extension, if the<br />
format description file is named <font color='green'>ext.fmt</font>. Also, the read/write<br />
type (input or output) is made explicit by including it in the format<br />
descriptors <font color='green'>ASCII_input_data</font> and <font color='green'>binary_output_data</font>.<br />
This provides the context needed for FreeForm ND programs to determine<br />
which format to use for input and which for output.<br />
<br />
Use the following commands to produce binary versions of the ASCII<br />
input files:<br />
<br />
<pre><br />
newform lldat1.ll -o llbin1.ll<br />
newform lldat2.ll -o llbin2.ll<br />
newform lldat3.ll -o llbin3.ll<br />
newform lldat4.ll -o llbin4.ll<br />
newform lldat5.ll -o llbin5.ll<br />
</pre><br />
<br />
If you want to convert back to ASCII, you can switch the words input<br />
and output in the format description file <font color='green'>ll.fmt</font>. You could then<br />
use the following commands to convert back to native ASCII format with<br />
output written to the screen:<br />
<br />
<pre><br />
newform llbin1.ll<br />
newform llbin2.ll<br />
newform llbin3.ll<br />
newform llbin4.ll<br />
newform llbin5.ll<br />
</pre><br />
<br />
It is also possible to convert back to ASCII without switching the<br />
read/write types input and output in <font color='green'>ll.fmt</font>. You can specify<br />
input and output formats by title instead. In this case, you want to<br />
use the output format in <font color='green'>ll.fmt</font> as the input format and the<br />
input format in <font color='green'>ll.fmt</font> as the output format. Use the following<br />
command to convert <font color='green'>llbin1.ll</font> back to ASCII:<br />
<br />
<pre><br />
newform <font color='green'>llbin1.ll</font> -ift "binary format for .ll latlon data"<br />
<br />
-oft "ASCII format for .ll latlon data"<br />
</pre><br />
<br />
Notice that <font color='green'>newform</font> reports back the read/write type actually<br />
used. Since <font color='green'>ASCII_input_data</font> was used as the output format,<br />
<font color='green'>newform</font> reports it as <font color='green'>ASCII_output_data</font>.<br />
<br />
Now assume that you want to convert the ASCII data file<br />
<font color='green'>llvals.asc</font> (not included in the example file set) to the binary<br />
file <font color='green'>latlon3.bin</font>, and the input and output data formats are<br />
described in <font color='green'>latlon.fmt</font>. The data file names do not provide the<br />
context allowing <font color='green'>newform</font> to find <font color='green'>latlon.fmt</font> by default, so<br />
you must include all file names on the command line:<br />
<br />
<pre><br />
newform llvals.asc -f latlon.fmt -o latlon3.bin<br />
</pre><br />
<br />
==="Nonstandard" Format Description File Names===<br />
<br />
<br />
If you are using a format description file that does not follow<br />
FreeForm ND file naming conventions, you must include its name on the<br />
command line. Assume that you want to convert the ASCII data file<br />
<font color='green'>latlon.dat</font> to the binary file <font color='green'>latlon.bin</font>, and the input<br />
and output data formats are both described in <font color='green'>llvals.frm</font> (not<br />
included in the example file set). The data file names follow FreeForm<br />
ND conventions, but the name of the format description file does not,<br />
so it will not be located through the default search sequence. Use the<br />
following command to convert to binary:<br />
<br />
<pre><br />
newform latlon.dat -f llvals.frm -o latlon.bin<br />
</pre><br />
<br />
<br />
Suppose now that the input format is described in <font color='green'>latlon.fmt</font> and<br />
the output format in <font color='green'>llvals.frm</font>. You do not need to explicitly<br />
specify the input format description file because it will be located<br />
by default, but you must specify the output format description file<br />
name. In this case, the command would be:<br />
<br />
<pre><br />
newform latlon.dat -of llvals.frm -o latlon.bin<br />
</pre><br />
<br />
You can always unambiguously specify the names of format description<br />
files and data files, whether or not their names follow FreeForm ND<br />
conventions. Assume you want to look only at longitude values in<br />
<font color='green'>latlon.bin</font> and that you want them defined as integers (longs)<br />
which are right-justified at column 30. You will reformat the<br />
specified binary data in <font color='green'>latlon.bin</font> into ASCII data in<br />
<font color='green'>longonly.dat</font> and then view it. The input format is found in<br />
<font color='green'>latlon.fmt</font>, the output format in <font color='green'>longonly.fmt</font>.<br />
<br />
Here is <font color='green'>longonly.fmt</font>:<br />
<br />
<pre><br />
/ This is the format description file for viewing longitude as an<br />
/ integer value right-justified at column 30.<br />
<br />
ASCII_data "ASCII output format, right-justified at 30"<br />
longitude 20 30 long 6<br />
</pre><br />
<br />
In this case, you have decided to look at the first 5 longitude values. Use the following command to unambiguously designate all files involved:<br />
<br />
<pre><br />
newform latlon.bin -if latlon.fmt -of longonly.fmt -c 5<br />
-o longonly.dat<br />
</pre><br />
<br />
When you view <font color='green'>longonly.dat</font>, you should see the following 5 values:<br />
<br />
<pre><br />
<br />
1 2 3 4<br />
1234567890123456789012345678901234567890<br />
<br />
-176161101<br />
777265<br />
35591879<br />
149408117<br />
55319598<br />
<br />
</pre><br />
<br />
==Changing ASCII Formats==<br />
<br />
<br />
<br />
You may encounter situations where a specific ASCII format is<br />
required, and your data cannot be used in its native ASCII format.<br />
With <font color='green'>newform</font>, you can easily reformat one ASCII format to<br />
another. In this example, you will reformat California earthquake data<br />
from one ASCII format to three other ASCII formats commonly used for<br />
such data.The file <font color='green'>calif.tap</font> contains data about earthquakes in<br />
California with magnitudes > 5.0 since 1980. The data were initially<br />
distributed by NGDC on tape, hence the <font color='green'>.tap</font> extension. The data<br />
format is described in <font color='green'>eqtape.fmt</font>:<br />
<br />
Here is the <font color='green'>eqtape.fmt</font> file:<br />
<br />
<pre><br />
/ This is the format description file for the NGDC .tap format,<br />
/ which is used for data distributed on floppy disks or tapes.<br />
<br />
ASCII_data ".tap format"<br />
source_code 1 3 char 0<br />
century 4 6 short 0<br />
year 7 8 short 0<br />
month 9 10 short 0<br />
day 11 12 short 0<br />
hour 13 14 short 0<br />
minute 15 16 short 0<br />
second 17 19 short 1<br />
latitude_abs 20 24 long 3<br />
latitude_ns 25 25 char 0<br />
longitude_abs 26 31 long 3<br />
longitude_ew 32 32 char 0<br />
depth 33 35 short 0<br />
magnitude_mb 36 38 short 2<br />
MB 39 40 constant 0<br />
isoseismal 41 43 char 0<br />
intensity 44 44 char 0<br />
<br />
/ The NGDC record check format includes<br />
/ six flags in characters 45 to 50. These<br />
/ can be treated as one variable to allow<br />
/ multiple flags to be set in a single pass,<br />
/ or each can be set by itself.<br />
<br />
ngdc_flags 45 50 char 0<br />
diastrophic 45 45 char 0<br />
tsunami 46 46 char 0<br />
seiche 47 47 char 0<br />
volcanism 48 48 char 0<br />
non_tectonic 49 49 char 0<br />
infrasonic 50 50 char 0<br />
<br />
fe_region 51 53 short 0<br />
magnitude_ms 54 55 short 1<br />
MS 56 57 char 0<br />
z_h 58 58 char 0<br />
cultural 59 59 char 0<br />
other 60 60 char 0<br />
magnitude_other 61 63 short 2<br />
other_authority 64 66 char 0<br />
ide 67 67 char 0<br />
depth_control 68 68 char 0<br />
number_stations_qual 69 71 char 0<br />
time_authority 72 72 char 0<br />
magnitude_local 73 75 short 2<br />
local_scale 76 77 char 0<br />
local_authority 78 80 char 0<br />
</pre><br />
<br />
<br />
Three other formats used for California earthquake data are<br />
hypoellipse, hypoinverse, and hypo71. Subsets of these formats are<br />
described in the format description file <font color='green'>hypo.fmt</font>. The format<br />
descriptions include the parameters required by the AcroSpin program<br />
that is distributed as part of the IASPEI Software Library (Volume 2).<br />
AcroSpin shows 3D views of earthquake point data.<br />
<br />
Here is the <font color='green'>hypo.fmt</font> file:<br />
<br />
<pre><br />
/ This format description file describes subsets of the<br />
/ hypoellipse, hypoinverse, and hypo71 formats.<br />
<br />
ASCII_data "hypoellipse format"<br />
<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
<br />
ASCII_data "hypoinverse format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 7 8 uchar 0<br />
minute 9 10 uchar 0<br />
second 11 14 ushort 2<br />
latitude_deg_abs 15 16 uchar 0<br />
latitude_ns 17 17 char 0<br />
latitude_min 18 21 ushort 2<br />
longitude_deg_abs 22 24 uchar 0<br />
longitude_ew 25 25 char 0<br />
longitude_min 26 29 ushort 2<br />
depth 30 34 short 2<br />
magnitude_local 35 36 uchar 1<br />
number_of_times 37 39 short 0<br />
maximum_azimuthal_gap 40 42 short 0<br />
nearest_station 43 45 short 1<br />
rms_travel_time_residual 46 49 short 2<br />
<br />
ASCII_data "hypo71 format"<br />
year 1 2 uchar 0<br />
month 3 4 uchar 0<br />
day 5 6 uchar 0<br />
hour 8 9 uchar 0<br />
minute 10 11 uchar 0<br />
second 12 17 float 2<br />
latitude_deg_abs 18 20 uchar 0<br />
latitude_ns 21 21 char 0<br />
latitude_min 22 26 float 2<br />
longitude_deg_abs 27 30 uchar 0<br />
longitude_ew 31 31 char 0<br />
longitude_min 32 36 float 2<br />
depth 37 43 float 2<br />
magnitude_local 44 50 float 2<br />
number_of_times 51 53 short 0<br />
maximum_azimuthal_gap 54 57 float 0<br />
nearest_station 58 62 short 1<br />
rms_travel_time_residual 63 67 float 2<br />
error_horizontal 68 72 float 1<br />
error_vertical 73 77 float 1<br />
s_waves_used 79 79 char 0<br />
</pre><br />
<br />
<br />
The parameters from the California earthquake data in the NGDC format<br />
needed for use with the AcroSpin program can be extracted and<br />
converted using the following commands:<br />
<br />
<pre><br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoellipse format" -o calif.he<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypoinverse format" -o calif.hi<br />
newform calif.tap -if eqtape.fmt -of hypo.fmt<br />
<br />
-oft "hypo71 format" -o calif.h71<br />
</pre><br />
<br />
If you develop an application that accesses seismicity data in a<br />
particular ASCII format, you need only to write an appropriate format<br />
description file in order to convert NGDC data into the format used by<br />
the application. This lets you make use of the data that NGDC provides<br />
in a format that works for you.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2019Wiki Testing/convs2008-02-06T15:15:44Z<p>Yuan: /* Defining Filters */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<br />
<br />
; '''input_file''' : Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' : Option flag followed by the name of the output file. The standard extensions are the same as for input files.<br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<br />
; -f format_file : Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file : Option flag followed by the name of the format description file describing the input data. Also use this option for an input variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -of output_format_file : Option flag followed by the name of the format description file describing the output data. Also use this option for an output variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -ft title : Option flag followed by the title (enclosed in quotes) of the format to be used for both input and output data, in which case there is no reformatting. The title follows format type on the first line of a format description in a format description file.<br />
<br />
; -ift title : Option flag followed by the title (enclosed in quotes) of the desired input format.<br />
<br />
; -oft title : Option flag followed by the title (enclosed in quotes) of the desired output format.<br />
<br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<br />
<br />
; -b local_buffer_size: Option flag followed by the size of the memory buffer used to<br />
process the data and format files. Default buffer size is 32,768. You many want to decrease the buffer size if you are running with low memory. Keep in mind that too small a buffer may result in unexpected behavior.<br />
<br />
; -c count: Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process. If <math>count > 0</math>, \var{count} records at the beginning of the file are processed. If <math>count < 0</math>, \var{count} records at the tail or end of the file are processed.<br />
<br />
; -v var_file : Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be processed by the FreeForm ND program. Variable names in var_file can be separated by one or more spaces or each name can be on a separate line.<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([[Wiki_Testing/fileserv| Chapter 7]]) for a complete description of<br />
the query syntax.</div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2018Wiki Testing/convs2008-02-06T15:14:49Z<p>Yuan: /* Changing Run-time Parameters */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<br />
<br />
; '''input_file''' : Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' : Option flag followed by the name of the output file. The standard extensions are the same as for input files.<br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<br />
; -f format_file : Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file : Option flag followed by the name of the format description file describing the input data. Also use this option for an input variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -of output_format_file : Option flag followed by the name of the format description file describing the output data. Also use this option for an output variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -ft title : Option flag followed by the title (enclosed in quotes) of the format to be used for both input and output data, in which case there is no reformatting. The title follows format type on the first line of a format description in a format description file.<br />
<br />
; -ift title : Option flag followed by the title (enclosed in quotes) of the desired input format.<br />
<br />
; -oft title : Option flag followed by the title (enclosed in quotes) of the desired output format.<br />
<br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<br />
<br />
; -b local_buffer_size: Option flag followed by the size of the memory buffer used to<br />
process the data and format files. Default buffer size is 32,768. You many want to decrease the buffer size if you are running with low memory. Keep in mind that too small a buffer may result in unexpected behavior.<br />
<br />
; -c count: Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process. If <math>count > 0</math>, \var{count} records at the beginning of the file are processed. If <math>count < 0</math>, \var{count} records at the tail or end of the file are processed.<br />
<br />
; -v var_file : Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be processed by the FreeForm ND program. Variable names in var_file can be separated by one or more spaces or each name can be on a separate line.<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2017Wiki Testing/convs2008-02-06T15:13:53Z<p>Yuan: /* Specifying Format Description Source */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<br />
<br />
; '''input_file''' : Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' : Option flag followed by the name of the output file. The standard extensions are the same as for input files.<br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<br />
; -f format_file : Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file : Option flag followed by the name of the format description file describing the input data. Also use this option for an input variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -of output_format_file : Option flag followed by the name of the format description file describing the output data. Also use this option for an output variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -ft title : Option flag followed by the title (enclosed in quotes) of the format to be used for both input and output data, in which case there is no reformatting. The title follows format type on the first line of a format description in a format description file.<br />
<br />
; -ift title : Option flag followed by the title (enclosed in quotes) of the desired input format.<br />
<br />
; -oft title : Option flag followed by the title (enclosed in quotes) of the desired output format.<br />
<br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<blockquote><br />
<br />
; -b local_buffer_size:<br />
Option flag followed by the size of the memory buffer used to<br />
process the data and format files.<br />
Default buffer size is 32,768. You many want to decrease the buffer<br />
size if you are running with low memory. Keep in mind that too small<br />
a buffer may result in unexpected behavior.<br />
<br />
; -c count:<br />
Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process.<br />
If <math>count > 0</math>, \var{count} records at the beginning of the file are<br />
processed.<br />
If <math>count < 0</math>, \var{count} records at the tail or end of the file<br />
are processed.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by the FreeForm ND program. Variable names in var_file can<br />
be separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
</blockquote><br />
<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2016Wiki Testing/convs2008-02-06T15:13:19Z<p>Yuan: /* Specifying Format Description Source */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<br />
<br />
; '''input_file''' : Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' : Option flag followed by the name of the output file. The standard extensions are the same as for input files.<br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<br />
; -f format_file : Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file : Option flag followed by the name of the format description file describing the input data. Also use this option for an input variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -of output_format_file : Option flag followed by the name of the format description file describing the output data. Also use this option for an output variable description file written using earlier versions of FreeForm ND.<br />
<br />
; -ft title : Option flag followed by the title (enclosed in quotes) of the format<br />
to be used for both input and output data, in which case there is no reformatting. The title follows format type on the first line of a format description in a format description file.<br />
<br />
; -ift title : Option flag followed by the title (enclosed in quotes) of the<br />
desired input format.<br />
<br />
; -oft title : Option flag followed by the title (enclosed in quotes) of the<br />
desired output format.<br />
<br />
<br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<blockquote><br />
<br />
; -b local_buffer_size:<br />
Option flag followed by the size of the memory buffer used to<br />
process the data and format files.<br />
Default buffer size is 32,768. You many want to decrease the buffer<br />
size if you are running with low memory. Keep in mind that too small<br />
a buffer may result in unexpected behavior.<br />
<br />
; -c count:<br />
Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process.<br />
If <math>count > 0</math>, \var{count} records at the beginning of the file are<br />
processed.<br />
If <math>count < 0</math>, \var{count} records at the tail or end of the file<br />
are processed.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by the FreeForm ND program. Variable names in var_file can<br />
be separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
</blockquote><br />
<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2015Wiki Testing/convs2008-02-06T15:12:28Z<p>Yuan: /* Specifying Input and Output Files */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<br />
<br />
; '''input_file''' : Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' : Option flag followed by the name of the output file. The standard extensions are the same as for input files.<br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<blockquote><br />
<br />
; -f format_file :<br />
Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the input data. Also use this option for an input<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -of output_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the output data. Also use this option for an output<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -ft title :<br />
Option flag followed by the title (enclosed in quotes) of the format<br />
to be used for both input and output data, in which case there is no<br />
reformatting. The title follows format type on the first line of a<br />
format description in a format description file.<br />
<br />
; -ift title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired input format.<br />
<br />
; -oft title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired output format.<br />
<br />
</blockquote><br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<blockquote><br />
<br />
; -b local_buffer_size:<br />
Option flag followed by the size of the memory buffer used to<br />
process the data and format files.<br />
Default buffer size is 32,768. You many want to decrease the buffer<br />
size if you are running with low memory. Keep in mind that too small<br />
a buffer may result in unexpected behavior.<br />
<br />
; -c count:<br />
Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process.<br />
If <math>count > 0</math>, \var{count} records at the beginning of the file are<br />
processed.<br />
If <math>count < 0</math>, \var{count} records at the tail or end of the file<br />
are processed.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by the FreeForm ND program. Variable names in var_file can<br />
be separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
</blockquote><br />
<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2014Wiki Testing/convs2008-02-06T15:11:53Z<p>Yuan: /* Command Line Arguments */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>NOTE: To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<blockquote><br />
<br />
; '''input_file''' :<br />
Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for<br />
ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' :<br />
Option flag followed by the name of the output file. The standard<br />
extensions are the same as for input files.<br />
<br />
</blockquote><br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<blockquote><br />
<br />
; -f format_file :<br />
Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the input data. Also use this option for an input<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -of output_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the output data. Also use this option for an output<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -ft title :<br />
Option flag followed by the title (enclosed in quotes) of the format<br />
to be used for both input and output data, in which case there is no<br />
reformatting. The title follows format type on the first line of a<br />
format description in a format description file.<br />
<br />
; -ift title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired input format.<br />
<br />
; -oft title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired output format.<br />
<br />
</blockquote><br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<blockquote><br />
<br />
; -b local_buffer_size:<br />
Option flag followed by the size of the memory buffer used to<br />
process the data and format files.<br />
Default buffer size is 32,768. You many want to decrease the buffer<br />
size if you are running with low memory. Keep in mind that too small<br />
a buffer may result in unexpected behavior.<br />
<br />
; -c count:<br />
Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process.<br />
If <math>count > 0</math>, \var{count} records at the beginning of the file are<br />
processed.<br />
If <math>count < 0</math>, \var{count} records at the tail or end of the file<br />
are processed.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by the FreeForm ND program. Variable names in var_file can<br />
be separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
</blockquote><br />
<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuanhttps://docs.opendap.org/index.php?title=Wiki_Testing/convs&diff=2013Wiki Testing/convs2008-02-06T15:11:30Z<p>Yuan: /* Case Sensitivity */</p>
<hr />
<div>=FreeForm ND Conventions=<br />
<br />
<br />
<br />
File name conventions have been defined for FreeForm ND. If you follow<br />
these conventions, FreeForm ND can locate format files through a<br />
default search sequence. Using the file name conventions also lets you<br />
reduce the number of arguments on the command line. In addition to<br />
standard file names, FreeForm ND programs recognize various standard<br />
command line arguments.<br />
<br />
==File Name Conventions==<br />
<br />
<br />
Naming conventions have been established for files accessed by<br />
FreeForm ND. Although you are not required to follow these<br />
conventions, using them lets you enter abbreviated commands when you<br />
are using FreeForm ND-based programs. FreeForm ND can then<br />
automatically execute several operations:<br />
<br />
<br />
* Determination of input and output formats when they are not explicitly identified in the relevant format descriptions in format files<br />
* Location of format files when they are not specified on the command line<br />
<br />
==File Name Extensions==<br />
<br />
<br />
The expected extensions for data files are as follows:<br />
<br />
<blockquote><br />
<br />
; <font color='green'>.dat</font> : For ASCII, e.g., <font color='green'>latlon.dat</font><br />
<br />
; <font color='green'>.dab</font> : For dBASE, e.g., <font color='green'>latlon.dab</font><br />
<br />
; <font color='green'>.bin</font> : binary or anything that is not <font color='green'>.dat</font> or <font color='green'>.dab</font>, e.g., <font color='green'>latlon.bin</font><br />
</blockquote><br />
<br />
<br />
The expected extension for format description files is <font color='green'>.fmt</font>, e.g.,<br />
<font color='green'>latlon.fmt</font>. You should not use mixed case extensions for format<br />
description files if you want to take advantage of FreeForm ND's<br />
default search capabilities. If you explicitly specify the names of<br />
format description files on the command line, you can use mixed case<br />
extensions.<br />
<br />
<blockquote>Previous versions of FreeForm ND used variable description files<br />
(formerly called format specification files) each of which contained<br />
variable descriptions for one file. Expected extensions for these<br />
files were <font color='green'>.afm</font> (ASCII), <font color='green'>.bfm</font> (binary), and <font color='green'>.dfm</font> (dBASE). Variable descriptions for one or more files can now be<br />
incorporated into a single format description file. It is<br />
recommended that you convert and combine (as appropriate) existing<br />
variable description files into format description files. </blockquote><br />
<br />
<br />
==File Name Relationships==<br />
<br />
<br />
FreeForm ND-based programs expect certain relationships between data<br />
file and format description file names as outlined below.<br />
<br />
<br />
* The data file is named datafile.ext where datafile is the file name of your choosing and ext is the extension. Example: <font color='green'>latlon.dat</font><br />
* The corresponding format description file should be named datafile.fmt. Example: <font color='green'>latlon.fmt</font><br />
* If one format description file is used for multiple data files, all with the same extension, the format description file should be named ext.fmt. Example: <font color='green'>ll.fmt</font> is the format description file for <font color='green'>lldat1.ll</font>, <font color='green'>lldat2.ll</font>, and <font color='green'>lldat3.ll</font>.<br />
<br />
Again, although not required, it is to your advantage to use these<br />
conventions.<br />
<br />
==Determining Input and Output Formats==<br />
<br />
<br />
You can optionally include the read/write type ("<font color='green'>input</font>" or<br />
"<font color='green'>output</font>") in format descriptors, e.g.,<br />
<font color='green'>ASCII_input_data</font>. You may not want to specify the read/write<br />
type in some circumstances. For example, you may need to translate<br />
from native ASCII to binary, then back to ASCII. ASCII is the input<br />
format in the first translation and the output format in the second<br />
translation, vice versa for binary. You would need to edit the format<br />
description file before executing the second translation if you<br />
included read/write type in the format descriptors.<br />
<br />
<blockquote>If you use the -ft option, you do not need to edit the format<br />
description file. See ([[Wiki_Testing/convs|Section 8.6.2]]) later<br />
in this chapter.</blockquote><br />
<br />
If you do not specify read/write type, FreeForm ND can nevertheless<br />
determine which format in a format description file is input and which<br />
is output as long as you have adhered to FreeForm ND filenaming<br />
conventions.<br />
<br />
<br />
*If the input format is not specified, and<br />
**the input data filename extension is <font color='green'>.bin</font>, assume binary input.<br />
**the input data filename extension is <font color='green'>.dab</font>, assume dBASE input.<br />
**the input data filename extension is <font color='green'>.dat</font>, assume ASCII input.<br />
**the input data filename extension is anything else, assume binary input.<br />
*:<br />
*If the output format is not specified, and<br />
**the input format is dBASE, the output is ASCII or binary, whichever is found first.<br />
**the input format is ASCII, the output is binary or dBASE, whichever is found first.<br />
*:<br />
<br />
<blockquote>The appropriate format descriptions must be in the format<br />
description file(s) used by FreeForm ND for a translation. If, for<br />
example, FreeForm ND determines the input format is binary and the<br />
output format is ASCII, there must be a format description for each<br />
type. </blockquote><br />
<br />
The checkvar program needs only an input format.<br />
<br />
==Locating Format Files==<br />
<br />
<br />
FreeForm ND programs use the following search sequence to find a<br />
format file (format or variable description file) for the data file<br />
datafile.ext when the format file name is not explicitly specified on<br />
the command line. In summary, FreeForm ND searches the directory<br />
specified by the GeoVu keyword <font color='green'>format_dir</font> (defined in a equivalence<br />
table or in the environment), the current or working directory, and<br />
the data file's home directory. The rules are applied in the order<br />
given below until a format file is found or all rules have been<br />
exhausted. If the relevant format file does not follow FreeForm ND<br />
conventions for name or location, it should be explicitly specified on<br />
the command line.<br />
<br />
<blockquote>GeoVu is a FreeForm ND-based application for data access and<br />
visualization. FreeForm ND applications other than GeoVu use GeoVu<br />
keywords. </blockquote><br />
<br />
For information about equivalence tables, see the GeoVu Tools<br />
Reference Guide, available from the NGDC.<br />
<br />
===Search Sequence===<br />
<br />
<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named datafile.fmt.<br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. <blockquote>Step 2 is included to accommodate variable description files that were created using previous versions of FreeForm ND. It is recommended that you convert existing variable description files to format description files. </blockquote><br />
#Search the directory given by the GeoVu keyword <font color='green'>format_dir</font> for a format description file named ext.fmt. If the GeoVu keyword <font color='green'>format_dir</font> is not found, FreeForm ND continues the search for a format file as follows.<br />
#Search the current (default) directory for a format description file named datafile.fmt.<br />
#Search the current directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the current directory for a format description file named <font color='green'>ext.fmt</font>. If the data file's home directory is not the same as the current directory, FreeForm ND continues the search for a format file with steps 7-9. The data file's home directory is given by the directory path component of the data file name. If the data file name has no directory path component, the home directory search is not done.<br />
#Search the data file's home directory for a format description file named <font color='green'>datafile.fmt</font>.<br />
#Search the data file's home directory for variable description files named <font color='green'>datafile.afm</font>, <font color='green'>datafile.bfm</font>, and <font color='green'>datafile.dfm</font>. Use the criteria in step 2 for determining input and output format files.<br />
#Search the data file's home directory for a format description file named <font color='green'>ext.fmt</font>.<br />
<br />
===Case Sensitivity===<br />
<br />
<br />
FreeForm ND adheres to the following rules for case sensitivity (in<br />
applicable operating systems) when it searches for a format file for<br />
the data file datafile.ext.<br />
<br />
<br />
*FreeForm ND preserves the case of datafile, for example, the default format file for the data file <font color='green'>LATLON.BIN</font> is <font color='green'>LATLON.fmt</font> (or <font color='green'>LATLON.bfm</font>).<br />
*FreeForm ND searches for a format file with a lower case extension. That is, the format file must have its extension in lower case no matter what the case of datafile. For example, the default format file for the data file <font color='green'>LatLon.dat</font> is <font color='green'>LatLon.fmt</font> (or <font color='green'>LatLon.afm</font>), and <font color='green'>TIMEDATE.fmt</font> (or <font color='green'>TIMEDATE.bfm</font>) is the default format file for <font color='green'>TIMEDATE.bin</font>.<br />
*In searching for a format description file of type <font color='green'>ext.fmt</font>, FreeForm ND preserves the case of ext. For example, for data files named <font color='green'>lldat1.LL</font>, <font color='green'>lldat2.LL</font>, and <font color='green'>latlon3.LL</font>, the default format description file is <font color='green'>LL.fmt</font>.<br />
<br />
==Command Line Arguments==<br />
<br />
<br />
FreeForm ND programs can take various command line arguments. The most<br />
widely used or standard arguments are discussed in this section. They<br />
are used for several different purposes: identifying input and output<br />
files, identifying format files and titles, changing run-time<br />
operation parameters, and defining data filters.<br />
<br />
The only required argument for any FreeForm ND program is the name of<br />
the input file or file to be processed. All other arguments are<br />
optional and can be in any order following the input file name. The<br />
command line of a FreeForm ND program with the standard arguments has<br />
the following form:<br />
<br />
application_name input_file [-f format_file]<br />
<br />
[-if input_format_file] [-of output_format_file] [-ft "title"] [-ift "title"] [-oft "title"] [-b local_buffer_size] [-c count] [-v var_file] [-q query_file] [-o output_file]<br />
<br />
<blockquote>To see a summary of command line usage for a FreeForm ND<br />
program, enter the program's name on the command line without any<br />
arguments. </blockquote><br />
<br />
<br />
===Specifying Input and Output Files===<br />
<br />
<br />
<br />
<blockquote><br />
<br />
; '''input_file''' :<br />
Name of the file to be processed. Following FreeForm ND naming<br />
conventions, the standard extensions for data files are <font color='green'>.dat</font> for<br />
ASCII format, <font color='green'>.bin</font> for binary, and <font color='green'>.dab</font> for dBASE.<br />
<br />
; '''-o output_file''' :<br />
Option flag followed by the name of the output file. The standard<br />
extensions are the same as for input files.<br />
<br />
</blockquote><br />
<br />
===Specifying Format Description Source===<br />
<br />
<br />
FreeForm ND offers a number of command line options for specifying the<br />
source of the format descriptions that a program must find in order to<br />
process data. The proper option or combination of options to use<br />
depends on how you have constructed your format files.<br />
<br />
<blockquote><br />
<br />
; -f format_file :<br />
Option flag followed by the name of the format description file<br />
describing both input and output data.<br />
<br />
; -if input_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the input data. Also use this option for an input<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -of output_format_file :<br />
Option flag followed by the name of the format description file<br />
describing the output data. Also use this option for an output<br />
variable description file written using earlier versions of FreeForm<br />
ND.<br />
<br />
; -ft title :<br />
Option flag followed by the title (enclosed in quotes) of the format<br />
to be used for both input and output data, in which case there is no<br />
reformatting. The title follows format type on the first line of a<br />
format description in a format description file.<br />
<br />
; -ift title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired input format.<br />
<br />
; -oft title :<br />
Option flag followed by the title (enclosed in quotes) of the<br />
desired output format.<br />
<br />
</blockquote><br />
<br />
<blockquote>NOTE: Previous versions of FreeForm ND used variable description files<br />
(<font color='green'>.afm</font>, <font color='green'>.bfm</font>, <font color='green'>.dfm</font>). It is recommended that you<br />
convert and combine (as appropriate) existing variable description<br />
files into format description files. </blockquote><br />
<br />
The various options available for specifying the source of a format<br />
description offer you a great deal of flexibility-in naming files,<br />
setting up format description files, and on the command line. In using<br />
these options, you need to consider the content of your format<br />
description files and how FreeForm ND will interpret the arguments on<br />
the command line.<br />
<br />
<br />
===Changing Run-time Parameters===<br />
<br />
<br />
FreeForm ND includes three arguments that let you change run-time<br />
parameters according to your needs. One argument lets you specify<br />
local buffer size, another indicates the number of records to process,<br />
and the third indicates which variables to process.<br />
<br />
<blockquote><br />
<br />
; -b local_buffer_size:<br />
Option flag followed by the size of the memory buffer used to<br />
process the data and format files.<br />
Default buffer size is 32,768. You many want to decrease the buffer<br />
size if you are running with low memory. Keep in mind that too small<br />
a buffer may result in unexpected behavior.<br />
<br />
; -c count:<br />
Option flag followed by a number that specifies how many data<br />
records at the head or tail of the file to process.<br />
If <math>count > 0</math>, \var{count} records at the beginning of the file are<br />
processed.<br />
If <math>count < 0</math>, \var{count} records at the tail or end of the file<br />
are processed.<br />
<br />
; -v var_file :<br />
Option flag followed by the name of a variable file. The file<br />
contains names of the variables in the input data file to be<br />
processed by the FreeForm ND program. Variable names in var_file can<br />
be separated by one or more spaces or each name can be on a separate<br />
line.<br />
<br />
</blockquote><br />
<br />
<br />
===Defining Filters===<br />
<br />
<br />
The query option lets you define data filters via a query file so you<br />
can precisely specify which data to process. The FreeForm ND program<br />
will process only those records meeting the query criteria.<br />
<br />
<blockquote><br />
<br />
; -q query_file :<br />
Option flag followed by the name of the file containing query<br />
criteria. See ([http://www <cite> ff,query</cite>]) for a complete description of<br />
the query syntax.<br />
</blockquote></div>Yuan