OPeNDAP Documentation - User contributions [en]

NCML Module Aggregation JoinExisting

2010-12-16T19:31:53Z

Mjohnson: /* Explicit Listing of Granules */

[[Category:Aggregation]]
[[Category:NCML]]

= Join Existing Aggregation =

A ''joinExisting'' aggregation joins multiple granule datasets by concatenating the specified outer dimensional data from the granules into the output. This results in matrices of the same number of dimensions, but with larger outer dimension cardinality. The outer dimension sizes of the granules may vary across granule, but any inner dimensions for multi-dimensional data still are required to match.

The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinExisting. Note that version 1.1.0 of the module does not support all features of joinExisting! Future versions will add more features.

'''PLEASE NOTE''' that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.

== Introduction ==

This page describes the behavior of the initial implementation of joinExisting for version 1.1.0 of the NcML Module, bundled with Hyrax 1.6.2. It is a limited feature set described below. Please see the Limitations section for more information.

In version 1.1.0, a joinExisting aggregation may be specified in two ways currently:

* Using explicit lists of '''netcdf''' elements with the ''ncoords'' attribute correctly specified for all of them.
* Using a '''scan''' element with ''ncoords'' specified and all matching granule datasets having this dimension size

Our example below will clarify this.

Future versions of the module will implement more of the joinExisting feature set.

== Examples ==

Here we give an example that illustrates the functionality offered by the current version of the aggregation. This example may also be found on:

http://test.opendap.org:8090/opendap/ioos/mday_joinExist.ncml

with the data granules located in

http://test.opendap.org:8090/opendap/coverage/mday/

=== Granules ===

Assume we have some number of granule datasets with a DDS the same as the following (modulo the dataset name):

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 1][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 1];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
} PH2006001_2006031_ssta.nc;
</pre>

=== Explicit Listing of Granules ===

We see that here ''time'' is the outer dimension, which is the only dimension we may join along (it is an error to specify an inner). Given some number of granules with this same shape, consider the following explicit joinExisting aggregation:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules">

<aggregation type="joinExisting"
dimName="time" >


<netcdf location="/coverage/mday/PH2006001_2006031_ssta.nc"
ncoords="1"/>
<netcdf location="/coverage/mday/PH2006032_2006059_ssta.nc"
ncoords="1"/>
<netcdf location="/coverage/mday/PH2006060_2006090_ssta.nc"
ncoords="1"/>

</aggregation>

</netcdf>
</pre>

First, note that the ''ncoords'' attribute '''MUST''' be specified on the individual granules for this version of the module (future versions will relax this requirement and automatically and efficiently load the sizes). Note that we also specify the ''dimName''. Any data array whose outer dimension is called this will be subject to aggregation in the output.

Serving this from Hyrax will result in the following DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

We see that the time dimension is now of size 3 to match that we joined three granule datasets together.

Also notice that the map vector for the joined dimension, time, has been duplicated as a sibling of the dataset. This is done automatically by the aggregation and it is copied into the actual map of the Grid. This copy is to facilitate datasets which have multiple Grid's that are to be joined --- the top-level map vector is used as the canonical template map which is then copied into the maps for all the aggregated Grids. In the case of the joined data being of type Array, this vector would already exist as the coordinate variable for the data matrix. Since this is the source map for all aggregated Grid's, any attribute (metadata) changes should be made explicitly on this top-level coordinate variable so that the metadata is shared among all the aggregated Grid map vectors.

=== Using the scan element with ncoords extension ===

Since all of the granules are of uniform size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules using scan@ncoords"
>

<attribute name="Description" type="string"
value=" joinExisting test on netcdf Grid granules using scan@ncoords"/>

<aggregation type="joinExisting"
dimName="time" >


<scan location="/coverage/mday"
subdirs="false"
suffix=".nc"
ncoords="1"
/>

</aggregation>

</netcdf>

</pre>

which we see results in the same DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

== Limitations ==

The current version implements only basic functionality. If there is extended functionality that is needed for your use, please send <mailto:support@opendap.org> to let us know!

=== Join Dimension Sizes Must Be Explicitly Declared ===

As we have seen, the most important limitation is that the ncoords must be specified for efficiency reasons. Future versions will relax this requirement. The problem is that the size of the output join dimension is dependent on checking the DDS of ''every'' granule in the aggregation, which is computationally expensive for large aggregations unless it is computed once and cached. The next feature release of the module will include a caching system for solving this general problem and allowing the ncoords to be loaded automatically rather than being specified.

=== Source of Data for Agrgegated Coordinate Variable on Join Dimension ===

This version does not allow the join dimension's data to be declared explicitly in the NcML as the NcML tutorial page describes. This version automatically aggregates '''all''' variables with the outer dimension matching the ''dimName''. This includes the coordinate variable (map vector in the case of Grid's) for the join dimension. These data cannot be overridden from those pulled from the files. Currently the TDS lists about 5 ways this data can be specified in addition to pulling them from the granules --- we only can pull them from granules now, which seems the most common use.

=== Source of Join Dimension Metadata ===

The metadata for the coordinate variable is pulled from the ''first'' granule dataset. Modification of coordinate variable metadata is not fully supported yet.

== Future Work ==

The most important addition in the next release will be the ability to efficiently load and cache the granule dimension sizes offline to allow for a highly interactive user experience when data is requested. This means that the ''ncoords'' attribute will no longer be required on the granules, although doing so results in a more efficient aggregation speed.

We also plan to allow the data and metadata for the join dimension coordinate variable to be overridden and modified.

BES - Modules - NcML Module

2010-10-27T20:19:36Z

Mjohnson: /* Aggregation Tutorials */

= Introduction =
[[Category:NCML]]
[[Category:BES Modules|NCML Module]]

'''Welcome to the OPeNDAP NcML Data Handler Module v1.1.0 for [http://www.opendap.org/download/hyrax.html Hyrax 1.6.2]!'''

[Updated for NcML Module v1.1.0 for Hyrax 1.6.2 Release -- 15 September 2010]

This module may be added to a [http://www.opendap.org/download/hyrax.html Hyrax 1.6.2] server to extend its data
serving capability to NcML 2.2 files (see
http://www.unidata.ucar.edu/software/netcdf/ncml/). NcML provides support for modifying other datasets in various ways, such as adding metadata and data and aggregating multiple datasets in several ways.

We refer the reader to the Unidata NcML tutorial: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html which will give the basics of using NcML. We then give a reference manual for the various elements and their attributes. Additionally we have provided a more extensive tutorial on NcML aggregation. Please see [[#Aggregation Tutorials]].

Since the DAP Grid is a common case, we will also give a simple example for adding metadata to the various parts of a Grid dataset using NcML. Please see [[Grid_Metadata_Tutorial]].

== Current Release ==

The most recent release is 1.1.0, which is bundled with (and requires) Hyrax server 1.6.2. To download, visit the [http://www.opendap.org/download/ncml_handler.html NcML Module Page].

New features in this release:

* joinExisting aggregation initial implementation ([[NCML_Module_Aggregation_JoinExisting]])

==Features==

This version currently implements a subset of NcML 2.2
functionality, along with some OPeNDAP extensions:

* Metadata Manipulation
** Addition, Removal, and Modification of attributes to other datasets (NetCDF, HDF4, HD5, etc.) served by the same Hyrax 1.6 server
** Extends NcML 2.2 to allow for common nested "attribute containers"
** Attributes can be DAP2 types as well as the NcML types
** Attributes can be of the special "OtherXML" type for injecting arbitrary XML into a DDX response
* Data Manipulation
** Addition of new data variables (scalars or arrays of basic types as well as structures)
** Variables may be removed from the wrapped dataset
** Allows the creation of "pure virtual" datasets which do not wrap another dataset
* Aggregations: JoinNew, JoinExisting Union [[#Aggregation Tutorials]]
** JoinNew Aggregation ([[NCML_Module_Aggregation_JoinNew]])
*** Allows multiple datasets to be "joined" by creating a new outer dimension for the aggregated variable
*** Aggregation member datasets can be listed explicitly with explicit coordinates for the new dimension for each member
*** Scan: Aggregations can be specified "automatically" by scanning a directory for files matching certain criteria, such as a suffix or regular expression.
*** Metadata may be added to the new coordinate variable for the new dimension
** JoinExisting Aggregation ([[NCML_Module_Aggregation_JoinExisting]])
*** Currently requires the author to specify the ''ncoords'' attribute.
*** Scan may also be used with ''ncoords'' attribute for uniform sized granules
*** Only allows join dimension to be aggregated from granules and not overridden in NcML
** Union Aggregation ([[NCML_Module_Aggregation_Union]])
*** Merges all member datasets into one by taking the first named instance of variables and metadata from the members
*** Useful for combining two or more datasets with different variables into a single set

== Installation from Source ==
For information on how to build and install the NcML Data Module, please see the INSTALL file that came with the source distribution.

= Installation Overview =

The NcML Module requires a working Hyrax 1.6 installation. It is a module
that is dynamically loaded into the Hyrax BES (Back End Server) to
allow it to handle NcML files.

Please see the file INSTALL for full build and install instructions as
well as requirements.

'''NOTE:''' After installation, you MUST restart Hyrax by restarting the BES and
OLFS so the NcML Module is loaded!

== Requirement: International Components for Unicode (ICU) Library ==

The most important external requirement is an installation of the International Components for Unicode (ICU) version 3.6 or higher (tested up to 4.2.1). The source distributions (as well as some binaries) may be found at the site: http://site.icu-project.org/download

If you are using Linux RPM's to run Hyrax, you can get an RPM for ICU as well. Search for the RPM named "libicu" using a package manager or yum, e.g. If you are compiling the module from source, you will also need the RPM "libicu-devel" to get the headers installed.

If you install in the default locations, the ncml_module should find the libraries and headers. Otherwise, please consult the INSTALL file for more information about installing ICU to a non-standard location.

== Testing Installation ==

Test data is provided to see if the installation was successful. The file sample_virtual_dataset.ncml is a dataset purely created in NcML and doesn't contain an underlying dataset.
You may also view fnoc1_improved.ncml to test adding attributes to an existing netCDF dataset (fnoc1.nc), but this requires the netCDF data handler to be
installed first! Several other examples installed also use the HDF4 and HDF5 handlers.

= Functionality =

This version of the NcML Module implements a subset of NcML 2.2
functionality. The reader is directed to http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/ for more information on NcML.

Our module can currently:

*Refer only to files being served locally (not remotely)
*Add, modify, and remove attribute metadata to a dataset
*Create a purely virtual dataset using just NcML and no underlying dataset
*Create new scalar variables of any simple NcML type or simple DAP type
*Create new Structure variables (which can contain new child variables)
*Create new N-dimensional arrays of simple types (NcML or DAP)
*Remove existing variables from a wrapped dataset
*Rename existing variables in a wrapped dataset
*Name dimensions as a mnemonic for specifying Array shapes
*Perform union aggregations on multiple datasets, virtual or wrapped or both
*Perform joinNew aggregations to merge a variable across multiple datasets by creating a new outer dimension
*Specify aggregation member datasets by scanning directories for files matching certain criteria

We describe each supported NcML element in detail below.

== <netcdf> Element ==

The <netcdf> element is used to define a dataset, either a wrapped dataset that is to be modified, a pure virtual dataset, or a member dataset of an aggregation. The <netcdf> element is assumed to be the topmost node, or as a child of an aggregation element.

=== Local vs. Remote Datasets ===

We assume that the location attribute (''netcdf@location'') refers to the full path (with respect to the BES data root directory) of a '''''local''''' dataset (served by the same Hyrax server). The current version of the module cannot be used to modify remote datasets.

If ''netcdf@location'' is the empty string (or unspecified, as empty is the default), the dataset is a pure virtual dataset, fully specified within the NcML file itself. Attributes and variables may be fully described and accessed with constraints just as normal datasets in this manner. The installed sample datafile "sample_virtual_dataset.ncml" is an example test case for this functionality.

=== Unsupported Attributes ===

The current version does not support the following attributes of <netcdf>:

*enhance
*addRecords
*fmrcDefinition (will be supported when FMRC aggregation is added)
*ncoords (will be supported when joinExisting is added)

== <readMetadata> Element ==

The <readMetadata/> element is the default, so is effectively not needed.

== <explicit> element ==

The <explicit/> element simply clears all attribute tables in the referred to netcdf@location before applying the rest of the
NcML transformations to the metadata.

== <dimension> Element ==

The <dimension> element has limited functionality in this release since the DAP2 doesn't support dimensions as more than mnemonics at this time. The limitations are:

* We only parse the ''dimension@name'' and ''dimension@length'' attributes.
* Dimensions can only be specified as a direct child of a <netcdf> element prior to any reference to them

For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="samples" length="5"/>

</netcdf>
</pre>

The dimension element sets up a mapping from the ''name'' to the unsigned integer ''length'' and can be used in a ''variable@shape'' to specify a length for an array dimension (see the section on <variable> below). The dimension map is cleared when </netcdf> is encountered (though this doesn't matter currently since we allow only one right now, but it will matter for aggregation, potentially). We also do not support <group>, which is the only other legal place in NcML 2.2 for a dimension element.

'''Parse Errors''':
* If the name and length are not both specified.
* If the dimension name already exists in the current scope
* If the length is not an unsigned integer
* If any of the other attributes specified in NcML 2.2 are used. We do not handle them, so we consider them errors now.

== <variable> Element ==

The <variable> element is used to:

* Provide lexical scope for a contained <attribute> or <variable> element
* Rename existing variables
* Add new scalar variables of simple types
* Add new Structure variables
* Add new N-dimensional Array's of simple types
* Specify the coordinate variable for the new dimension in a joinNew aggregation

We describe each in turn in more detail.

=== Specifying Lexical Scope with <variable type=""> ===

Consider the following example:

<variable name="u">
<attribute name="Metadata" type="string">This is metadata!</attribute>
</variable>

This code assumes that a variable named "u" exists (of any type since we do not specify) and provides the lexical scope for the
attribute "Metadata" which will be added or modified within the attribute table for the variable "u" (it's qualified name would be "u.Metadata").

==== Nested DAP Structure and Grid Scopes ====

Scoping variable elements may be nested if the containing variable is a Structure (this includes the special case of Grid)

<pre>
<variable name="DATA_GRANULE" type="Structure">
<variable name="PlanetaryGrid" type="Structure">
<variable name="percipitate">
<attribute name="units" type="String" value="inches"/>
</variable>
</variable>
</variable>
</pre>

This adds a "unit" attribute to the variable "percipitate" within the nested Structure's
("DATA_GRANULE.PlanetaryGrid.percipitate" as fully qualified name).
Note that we '''must''' refer to the type explicitly as a "Structure" so the parser knows
to traverse the tree.

'''Note''' the variable might be of type Grid, but the type "Structure" must be used in the NcML to traverse it.

==== Adding Multiple Attributes to the Same Variable ====

Once the variable's scope is set by the opening <variable> element, more than one attribute can be specified within it. This will make the NcML more readable and also will make the parsing more efficient since the variable will only need to be looked up once.

For example,

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

should be preferred over:

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
</variable>

<variable name="Foo">
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

although they produce the same result. Any number of attributes can be specified before the variable is closed.

=== Renaming Existing Variables ===

The attribute ''variable@orgName'' is used to rename an existing variable.

For example:

<pre>
<variable name="NewName" orgName="OldName"/>
</pre>

will rename an existing variable at the current scope named "OldName" to "NewName". After this point in the NcML file (such as in constraints specified for the DAP request), the variable is known by "NewName".

Note that the type is not required here --- the variable is assumed to exist and its existing type is used. It is not possible to change the type of an existing variable at this time!

'''Parse Errors''':
* If a variable with ''variable@orgName'' doesn't exist in the current scope
* If the new name ''variable@name'' is already taken in the current scope
* If a new variable is created but does not have exactly one values element

=== Adding a New Scalar Variable ===

The <variable> element can be used to create a new scalar variable of a simple type (i.e. an atomic NcML type such as "int" or "float", or any DAP atomic type, such as "UInt32" or "URL") by specifying an empty ''variable@shape'' (which is the default), a simple type for ''variable@type'', and a contained <values> element with the one value of correct type.

For example:

<pre>
<variable name="TheAnswerToLifeTheUniverseAndEverything" type="double">
<attribute name="SolvedBy" type="String" value="Deep Thought"/>
<values>42.000</values>
</variable>
</pre>

will create a new variable named "TheAnswerToLifeTheUniverseAndEverything" at the current scope. It has no shape so will be a scalar of type "double" and will have the value 42.0.

'''Parse Errors''':
* It is a parse error to not specify a <values> element with exactly one proper value of the variable type.
* It is a parse error to specify a malformed or out of bounds value for the data type

=== Adding a New Structure Variable ===

A new Structure variable can be specified at the global scope or within another Structure. It is illegal for an array to have type structure, so the shape must be empty.

For example:

<pre>
<variable name="MyNewStructure" type="Structure">
<attribute name="MetaData" type="String" value="This is metadata!"/>
<variable name="ContainedScalar1" type="String"><values>I live in a new structure!</values></variable>
<variable name="ContainedInt1" type="int"><values>42</values></variable>
</variable>
</pre>

specifies a new structure called "MyNewStructure" which contains two scalar variable fields "ContainedScalar1" and "ContainedInt1".

Nested structures are allowed as well.

'''Parse Error''':
* If another variable or attribute exists at the current scope with the new name.
* If a <values> element is specified as a direct child of a new Structure --- structures cannot contain values, only attributes and other variables.

=== Adding a New N-dimensional Array ===

An N-dimensional array of a simple type may be created virtually as well by specifying a non-empty ''variable@shape''. The shape contains the array dimensions in left-to-right order of slowest varying dimension first. For example:

<pre>
<variable name="FloatArray" type="float" shape="2 5">

<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will specify a 2x5 dimension array of float values called "FloatArray". The <values> element must contain 2x5=10 values in row major order (slowest varying dimension first). Since whitespace is the default separator, we use a newline to show the dimension boundary for the values, which is easy to see for a 2D matrix such as this.

A dimension name may also be used to refer mnemonically to a length. The DAP response will use this mnemonic in its output, but it is not currently used for shared dimensions, only as a mnemonic. See the section on the <dimension> element for more information. For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="sample" length="5"/>
<variable name="FloatArray" type="float" shape="station sample">
<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will produce the same 2x5 array, but will incorporate the dimension mnemonics into the response. For example, here's the DDS response:

<pre>
Dataset {
Float32 FloatArray[station = 2][samples = 5];
} sample_virtual_dataset.ncml;
</pre>

Note that the <values> element respects the ''values@separator'' attribute if whitespace isn't correct. This is very useful for arrays of strings with whitespace, for example.

<pre>
<variable name="StringArray" type="string" shape="3">
<values separator="*">String 1*String 2*String 3</values>
</variable>
</pre>

creates a length 3 array of string StringArray = {"String 1", "String 2", "String 3"}.

'''Parse Errors''':
* It is an error to specify the incorrect number of values
* It is an error if any value is malformed or out of range for the data type.
* It is an error to specify a named dimension which does not exist in the current <netcdf> scope.
* It is an error to specify an Array whose flattened size (product of dimensions) is > 2^31-1.

=== Specifying the new coordinate variable for a joinNew aggregation ===

In the special case of a joinNew aggregation, the new coordinate variable may be specified with the <variable> element. The new coordinate variable is ''defined'' to have the same name as the new dimension. This allows for several things:

*Explicit specification of the variable type and coordinates for the new dimension
*Specification of the metadata for the new coordinate variable

In the first case, the author can specify explicitly the type of the new coordinate variable and the actual values for each dataset. In this case, the variable ''must'' be specified ''after'' the aggregation element in the file so the new dimension's size (number of member datasets) may be known and error checking performed. Metadata can also be added to the variable here.

In the second case, the author may just specify the variable name, which allows one to specify the metadata for a coordinate variable that is automatically generated by the aggregation itself. This is the only allowable case for a variable element to ''not'' contain a values element! Coordinate variables are generated automatically in two cases:

*The author has specified an explicit list of member datasets, with or without explicit coordVal attributes.
*The author has used a <scan> element to specify the member datasets via a directory scan

In this case, the <variable> element may come before or after the <aggregation>.

'''Parse Errors:'''

*If an explicit variable is declared for the new coordinate variable:
** And it contains explicit values, the number of values must be equal to the number of member datasets in the aggregation.
** It must be specifed ''after'' the <aggregation> element

* If a numeric coordVal is used to specify the first member dataset's coordinate, then ''all'' datasets must contain a numerical coordinate.

* An error is thrown if the specified aggregation variable (variableAgg) is not found in ''all'' member datasets.

* An error is thrown if the specified aggregation variable is not of the same type in ''all'' member datasets. Coercion is ''not'' performed!

* An error is thrown if the specified aggregation variables in all member datasets do not have the same shape

* An error is thrown if an explicit coordinate variable is specified with a shape that is ''not'' the same as the new dimension name (and the variable name itself).

== <values> Element ==

The <values> element can only be used in the context of a '''new''' variable of scalar or array type. We cannot change the values for existing variables in this version of the handler.
The characters content of a <values> element is considered to be a separated list of value tokens valid for the type of the variable of the parent element. The number of specified tokens in the content ''must'' equal the product of the dimensions of the enclosing ''variable@shape'', or be one value for a scalar. It is an error to ''not'' specify a <values> element for a declared new variable as well.

=== Changing the Separator Tokens ===

The author may specify values@separator to change the default value token separator from the default whitespace. This is very useful for specifying arrays of strings with whitespace in them, or if data in CSV form is being pasted in.

=== Autogeneration of Uniform Arrays ===

We also can parse ''values@start'' and ''values@increment'' INSTEAD OF tokens in the content. This will "autogenerate" a uniform array of values of the given product of dimensions length for the containing variable. For example:

<pre>
<variable name="Evens" type="int" shape="100">
<values start="0" increment="2"/>
</variable>
</pre>

will specify an array of the first 100 even numbers (including 0).

'''Parse Errors''':
* If the incorrect number of tokens are specified for the containing variable's shape
* If any value token cannot be parsed as a valid value for the containing variable's type
* If content is specified in addition to start and increment
* If only one of start or increment is specified
* If the values element is placed anywhere except within a NEW variable.

== <attribute> Element ==

As an overview, whenever the parser encounters an <attribute> with a non-existing name (at the current scope),
it creates a new one, whether a container or atomic attribute (see below). If the attribute exists,
its value and/or type is modified to those specified in the <attribute> element. If an attribute
structure (container) exists, it is used to define a nested lexical scope for child attributes.

Attributes may be scalar (one value) or one dimensional arrays. Arrays are specified by using whitespace (default) to separate the different values. The attribute@separator may also be set in order to specify a different separator, such as CSV format or to specify a non-whitespace separator so strings with whitespace are not tokenized. We will give examples of creating array attributes below.

=== Adding New Attributes or Modifying an Existing Attribute ===

If a specified attribute with the attribute@name does not exist at the current lexical scope,
a new one is created with the given type and value. For example, assume "new_metadata" doesn't
exist at the current parse scope. Then:

<attribute name="new_metadata" type="string" value="This is a new entry!"/>

will create the attribute at that scope. Note that value can be specified in the content of the
element as well. This is identical to the above:

<attribute name="new_metadata" type="string">This is a new entry!</attribute>

If the attribute@name already exists at the scope, it is modified to contain the specified type and value.

==== Arrays ====

As in NcML, for numerical types an array can be specified by separating the tokens by whitespace (default) or be
specifying the token separator with attribute@separator. For example,

<attribute name="myArray" type="int">1 2 3</attribute>

and

<attribute name="myArray" type="int" separator=",">1,2,3</attribute>

both specify the same array of three integers named "myArray".

TODO Add more information on splitting with a separator!

==== Structures (Containers) ====

We use attribute@type="Structure" to define a new (or existing) attribute container. So if we
wanted to add a new attribute structure, we'd use something like this:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

Assuming "MySamples" doesn't already exist, an attribute container will be created at the current scope
and the "Location" and "Samples" attributes will be added to it.

Note that we can create nested attribute structures to arbitrary depth this way as well.

If the attribute container with the given name already exists at the current scope,
then the attribute@type="Structure" form is used to define
the lexical scope for the container. In other words, child <attribute> elements will
be processed within the scope of the container. For example, in the above example, if
"MySamples" already exists, then the "Location" and "Samples" will be processed within the
existing container (they may or may not already exist as well).

==== Renaming an Existing Attribute or Attribute Container ====

We also support the attribute@orgName attribute for renaming attributes.

For example,

<attribute name="NewName" orgName="OldName" type="string"/>

will rename an existing attribute "OldName" to "NewName" while leaving its value alone.
If attribute@value is also specified, then the attribute is renamed ''and'' has its value modified.

This works for renaming attribute containers as well:

<attribute name="MyNewContainer" orgName="MyOldContainer" type="Structure"/>

will rename an existing "MyOldContainer" to "MyNewContainer". Note that any children
of this container will remain in it.

=== DAP ''OtherXML'' Extension ===

The module now allows specification of attributes of the new DAP type "OtherXML". This allows the NCML file author to inject arbitrary well-formed XML into an attribute for clients that want XML metadata rather than just string or url. Internally, the attribute is still a string (and in a DAP DAS response will be quoted inside one string). However, since it is XML, the NCMLParser still parses it and checks it for well-formedness (but NOT against schemas). This extension allows the NCMLParser to parse the arbitrary XML within the given attribute without causing errors, since it can be any XML.

The injected XML is most useful in the DDX response, where it shows up directly in the response as XML. XSLT and other clients can then parse it.

==== Errors ====

*The XML '''must''' be in the content of the <attribute type="OtherXML"> element. It is a parser error for ''attribute@value'' to be set if ''attribute@type'' is "OtherXML".
*The XML must also be well-formed since it is parsed. A parse error will be thrown if the OtherXML is malformed.

==== Example ====

Here's an example of the use of this special case.

<pre>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/coverage/200803061600_HFRadar_USEGC_6km_rtv_SIO.nc">

<attribute name="someName" type="OtherXML">
<Domain xmlns="http://www.opengis.net/wcs/1.1"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
>
<SpatialDomain>
<ows:BoundingBox crs="urn:ogc:def:crs:EPSG::4326">
<ows:LowerCorner>-97.8839 21.736</ows:LowerCorner>
<ows:UpperCorner>-57.2312 46.4944</ows:UpperCorner>
</ows:BoundingBox>
</SpatialDomain>
<TemporalDomain>
<gml:timePosition>2008-03-27T16:00:00.000Z</gml:timePosition>
</TemporalDomain>
</Domain>
<SupportedCRS xmlns="http://www.opengis.net/wcs/1.1">urn:ogc:def:crs:EPSG::4326</SupportedCRS>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">netcdf-cf1.0</SupportedFormat>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">dap2.0</SupportedFormat>
</attribute>

</netcdf>
</pre>

'''TODO'' Put the DDX response for the above in here!

====Namespace Closure====

Furthermore, the parser will make the chunk of OtherXML "namespace closed". This means any namespaces specified in parent NCML elements of the OtherXML tree will be "brought down" and added to the ''root'' OtherXML elements so that the subtree may be pulled out and added to the DDX and still have its namespaces. The algorithm doesn't just bring used prefixes, but brings ''all'' of the lexically scoped closest namespaces in all ancestors. In other words, it adds unique namespaces (as determined by prefix) in order from the root of the OtherXML tree as it traverses to the root of the NCML document.

Namespace closure is a syntactic sugar that simplifies the author's task since they can specify the namespaces just once at the top of the NCML file and expect that when the subtree of XML is added to the DDX that these namespaces will come along with that subtree of XML. Otherwise they have to explicitly add the namespaces to each attributes.

'''TODO''' Add an example!

== <remove> Element ==

The <remove> element can remove attributes and variables. For example:

<pre>
<attribute name="NC_GLOBAL" type="Structure">
<remove name="base_time" type="attribute"/>
</attribute>
</pre>

will remove the attribute named "base_time" in the attribute structure named "NC_GLOBAL".

Note that this works for attribute containers as well! We could recursively remove the ''entire''
attribute container (i.e. it and all its children) with:

<pre>
<remove name="NC_GLOBAL" type="attribute"/>
</pre>

It also can be used to remove variables from existing datasets:

</pre>
<remove name="SomeExistingVariable" type="variable"/>
</pre>

This also recurses on variables of type Structure --- the entire structure including all of its children are removed from the dataset's response.

'''Parse Errors''':
* It is a parse error if the given attribute or variable doesn't exist in the current scope

== <aggregation> Element ==

Aggregation involves combining multiple datasets (<netcdf>) into a virtual "single" dataset in various ways. For a tutorial on aggregation in NcML 2.2, the reader is referred to the Unidata page: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

NcML 2.2 supports multiple types of aggregation: union, joinNew, joinExisting, and fmrc (forecast model run collection).

The current version of the NcML module supports two of these aggregations:

*Union [[NCML_Module_Aggregation_Union]]
*JoinNew [[NCML_Module_Aggregation_JoinNew]]

A ''union'' aggregation specifies that the first instance of a variable or attribute (by name) that is found in the ordered list of datasets will be the one in the output aggregation. This is useful for combining two dataset files, each which may contain a single variable, into a composite dataset with both variables.

A JoinNew aggregation joins a variable which exists in multiple datasets (usually samples of a datum over time) into a new variable containing the data from ''all'' member datasets by creating a new outer dimension. The ''i''th component in the new outer dimension is the variable's data from the ''i''th member dataset. It also adds a new coordinate variable of whose name is the new dimension's name and whose shape (length) is the new dimension as well. This new coordinate variable may be explicitly given by the author or may be autogenerated in one of several ways.

== <scan> Element ==

The scan element can be used within an aggregation context to allow a directory to be searched in various ways in order to specify the members of an aggregation. This allows a static NcML file to refer to an aggregation which may change over time, such as where a new data file is generated each day.

We describe usage of the <scan> element in detail in the joinNew aggregation tutorial at [[NCML_Module_Aggregation_JoinNew]].

= Errors =

There are three types of error messages that may be returned:

* Internal Error
* Resource Not Found Error
* Parse Error

== Internal Errors ==
'''Internal errors''' should be reported to support@opendap.org as they are likely bugs.

== Resource Not Found Errors ==

If the netcdf@location specifies a non-existent local dataset (one that is not being served by
the same Hyrax server), it will specify the resource was not found. This may also be returned
if a handler for the specified dataset is not currently loaded in the BES. Users should test that the
dataset to be wrapped already exists and can be viewed on the running server before writing NcML
to add metadata. It's also an error to refer to remote datasets (at this time).

== Parse Errors ==

'''Parse errors''' are user errors in the NcML file. These could be malformed XML, malformed NcML,
unimplemented features of NcML, or could be errors in referring to the wrapped dataset.

The error message should specify
the error condition as well as the "current scope" as a fully qualified DAP name within the
loaded dataset. This should be enough information to correct the parse error as new NcML
files are created.

The parser will generate parse errors in various
situations where it expects to find certain structure in the underlying dataset. Some examples:

* A variable of the given name was not found at the current scope
* attribute@orgName was specified, but the attribute cannot be found at current scope.
* attribute@orgName was specified, but the new name is already used at current scope.
* remove specified a non-existing attribute name

= Grid Metadata Tutorial =

Please see the page [[Grid_Metadata_Tutorial]] for an example of adding metadata to the various parts of a DAP Grid variable.

= Aggregation Tutorials =

The NcML module may also be used to aggregate multiple datasets into one virtual dataset. We currently support two of the NcML aggregtions:

*Union [[NCML_Module_Aggregation_Union]]: combine multiple datasets into one by merging variables together, selecting the first of each unique name.
*JoinNew [[NCML_Module_Aggregation_JoinNew]]: combine variables across multiple datasets by creating a new outer dimension and coordinate variable for each of the sample datasets.
*JoinExisting [[NCML_Module_Aggregation_JoinExisting]]: combine variables with a common named outer dimension along that dimension by concatenating data for that dimension
Please see the sections for a tutorial on the various uses of these aggregations.

= Additions/Changes to NcML 2.2 =

This section will keep track of changes to the NcML 2.2 schema. Eventually these
will be rolled into a new schema.

== Attribute Structures (Containers) ==
This module also adds functionality beyond the current NcML 2.2 schema
--- it can handle nested <attribute> elements in order to make
attribute structures. This is done by using the <attribute
type="Structure"> form, for example:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

"MyContainer" describes an attribute structure with two attribute fields,
a string "Location" and an array of int's called "Samples".
Note that an attribute structure of this form can
only contain other <attribute> elements and NOT a value.

If the container does not already exist, it will be created at the scope it is declared, which could
be:

* Global (top of dataset)
* Within a variable's attribute table
* Within another attribute container

If an attribute container of the given name already exists at the lexical scope, it is traversed
in order to define the scope for the nested (children) attributes it contains.

== Unspecified Variable Type Matching for Lexical Scope ==

We also allow the type attribute of a variable element (variable@type) to be the empty string
(or unspecified) when using existing variables to define the lexical scope of an <attribute>
transformation. In the schema, variable@type is (normally) required.

== DAP 2 Types ==

Additionally, we allow DAP2 atomic types (such as UInt32, URL) in addition to the NcML types.
The NcML types are mapped onto the closest DAP2 type internally.

== DAP OtherXML Attribute Type ==

We also allow attributes to be of the new DAP type "OtherXML" for injecting arbitrary XML into an attribute as content rather than trying to form a string. This allows the parser to check well-formedness.

== Forward Declaration of Dimensions ==

Since we use a SAX parser for efficiency, we require the <dimension> elements to come ''before'' their use in a ''variable@shape''. One way to change the schema to allow this is to force the dimension elements to be specified in a sequence after explicit and metadata choice and before all other elements.

== Aggregation Element Location and Processing Order Differences ==

NcML specifies that if a dataset (<netcdf> element) specifies an aggregation element, the aggregation element is always processed first, regardless of its ordering within the <netcdf> element. Our parser, since it is SAX and not DOM, modifies this behavior in that order matters in some cases:

* Metadata (<attribute>) elements specified ''prior'' to an aggregation "shadow" the aggregation versions. This is be useful for "overriding" an attribute or variable in a union aggregation, where the first found will take precedence.
* JoinNew: If the new coordinate variable's data is to be set explicitly by specifying the new dimension's shape (either with explicit data or the autogenerated data using values@start and values@increment attributes), the <variable> ''must'' come after the aggregation since the size of the dimension is unknown until the aggregation element is processed.

= Backward Compatibility Issues =

Due to the way shared dimensions were implemented in the NetCDF, HDF4, and HDF5 handlers, the DAS responses did not follow the DAP2 specification. The NcML module, on the other hand, generates DAP2 compliant DAS for these datasets, which means that wrapping some datasets in NcML will generate a DAS with a different structure. This is important for the NcML author since it changes the names of attributes and variables. In order for the module to find the correct scope for adding metadata, for example, the DAP2 DAS must be used.

In general, what this means is that an empty "passthrough" NcML file should be the starting point for authoring an NcML file. This file would just specify a dataset and nothing else:

<pre>
<netcdf location="/data/ncml/myNetcdf.nc"/>
</pre>

The author would then request the DAS response for the NCML file and use that as the starting point for modifications to the original dataset.

More explicit examples are given below.

== NetCDF ==

The NetCDF handler represents some NC datasets as a DAP 2 Grid, but the returned DAS is not consistent with the DAP 2 spec for the attribute hierarchy for such a Grid. The map vector attributes are placed as siblings of the grid attributes rather than within the grid lexical scope. For example, here's the NetCDF Handler DDS for a given file:

<pre>
Dataset {
Grid {
Array:
Int16 cldc[time = 456][lat = 21][lon = 360];
Maps:
Float64 time[time = 456];
Float32 lat[lat = 21];
Float32 lon[lon = 360];
} cldc;
} cldc.mean.nc;
</pre>

showing the Grid. Here's the DAS the NetCDF handler generates:

<pre>
Attributes {
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
}
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
}
</pre>

Note the map vector attributes are in the "dataset" scope.

Here's the DAS that the NcML Module produces from the correctly formed DDX:

<pre>
Attributes {
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
cldc {
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
}
}
</pre>

Here the Grid Structure "cldc" and its contained data array (of the same name "cldc") and map vectors have their own attribute containers as DAP 2 specifies.

What this means for the author of an NcML file adding metadata to a NetCDF dataset that returns a Grid is that they should generate a "passthrough" file and get the DAS and then specify modifications based on that structure.

Here's an example passthrough:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">
</netcdf>
</pre>

For example, to add an attribute to the map vector "lat" in the above, we'd need the following NcML:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">

<variable name="cldc" type="Structure">

<variable name="lat">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lat!</attribute>
</variable>
<variable name="lon">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lon!</attribute>
</variable>
</variable>
</netcdf>
</pre>

This clearly shows that the structure of the Grid must be used in the NcML: the attribute being added is technically "cldc.lat.Description" in a fully qualified name. The parser would return an error if it was attempted as "lat.Description" as the NetCDF DAS for the original file would have led one to believe.

== HDF4/HDF5 ==

Similarly to the NetCDF case, the Hyrax HDF4 Module produces DAS responses that do not respect the DAP2 specification. If an NcML file is used to "wrap" an HDF4 dataset, the
correct DAP2 DAS response will be generated, however.

This is important for those writing NcML for HDF4 data since the
lexical scope for attributes relies on the correct DAS form --- to
handle this, the user should start with a "passthrough" NcML file (see the above NetCDF example) and
use the DAS from that as the starting point for knowing the structure
the NcML handler expects to see in the NcML file. Alternatively, the DDX has the
proper attribute structure as well (the DAS is generated from it).

= Known Bugs =

There are no known bugs currently.

= Planned Future Enhancements =

Planned enhancements for future versions of the module include:

* New NcML Aggregations
** JoinExisting
*** Joins a variable across multiple datasets by appending the data for a given dimension from each dataset
*** Will also allow directory scans for specifying the aggregation
** Forecast Model Run Collection (FMRC)
*** Special case of JoinNew for forecast data with two time variables
*** See: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html

= Copyright =

This software is copyrighted under the GNU Lesser GPL. Please see the
files COPYING and COPYRIGHT that came with this distribution.

OPeNDAP Developer's Workshop 2010

2010-10-18T16:35:28Z

Mjohnson: /* The NcML handler will have to do its own caching */

== Where & When ==
* Location: '''Troy, NY'''
* Dates: '''6-8 Oct'''

== Meeting modus operandi ==
* mockups
* whiteboard designs
* use cases
* best to exploit face-to-face for what it affords.

== Agenda ==
----
----
===Wednesday===

;0830 - Coffee

==== Server Side Functions ====

* '''Registration'''
** modularized
** each function -> class
** version
*** can you handle this
*** service
*** how do we package this
* '''Discovery'''
** Ask the OLFS/BES what functions for dataset
*** what functions
*** capabilities
* '''Format'''
** function(p1,p2,p3,p4...)
* '''Usage'''
** Run by handler during loading - slection
** Run during constraint parsing
** Run during transmission/serialization (constraint evaluation)
** What about functions like "version"??

void
function_version(int, BaseType *[], DDS &, BaseType **btpp)

typedef void(*bool_func)(int argc, BaseType *argv[], DDS &dds, bool *result);
typedef void(*btp_func)(int argc, BaseType *argv[], DDS &dds, BaseType **btpp);
typedef void(*proj_func)(int argc, BaseType *argv[], DDS &dds, ConstraintEvaluator &ce);

===== Use Cases =====
* Scientist has a dataset and wants to discover what functions are available to apply to that dataset
* Data provider has a set of server-side functions that they wish to provide for data users. They want to add these modules to the OPeNDAP via dynamically loaded modules
* Scientist has a dataset and wants to apply geogrid function to that dataset
* Scientist wants to discover the the shape and size (metadata) of a particular function against a dataset

===== Plan: Define a class/hierarchy =====
* Define a set of classes for server-saide functions
* Have the BES instantiate those from .so files
* Those instances are then passed into a ConstraintEvaluator instance which is then passed to a handler
* The BES conf file(s) can include information so that some functions are always loaded and some are loaded only for specific handlers (e.g., FreeForm).

* Take the server side functions out of libdap and add to BES modules
* Add the server side functions to server side class
* These classes will be able to handle things like get version, given a ddx(?) can this function be run against it (return true/false)
* At load time the classes are created within the BES, the functions are registered with libdap. This maintains binary compatability.

----

==== Server Administration ====
===== Admin interface =====
* admin connects to a listener
* can restart a listener
* can restart the BES with ADMIN
* turn on/off debugging through admin interface
* keep track of how many connections are allowed/active
* re-write besdaemon/beslistener (2-3 hrs)
* allowed size of response from BES (1 hr)

:: http://docs.opendap.org/index.php/Hyrax_Admin_Interface

===== BES design changes =====
* drop the besdaemon from the architecture
* listener is started instead of daemon
* listener keeps track of connections made and socket connected to
* listener keeps track of connections dropped, removes from list
* limit the number of open connections
* besdaemon will be able to have both hard and graceful restart
* Out of band comm will enable access to logs and conf files in addition to the restarts
* capability to send back configuration information
* capability to receive new configuration information and write to disk and reload
* we don't sweat the fact that the logs are shared - we can build a fancy interface out at the servlet level to filter by specific client using the PID info in the log.
* Changes in the conf only are used when a process is started.
* Soft shutdown, listener sends certain signal to child, when done handling current request, go down
* Hard shutdown, listener sends certain signal to child, go down now
* capability to stream back the log file

===== Authentication and Authorization =====
* Will people be authenticating through ESG to get to Hyrax? Or will Hyrax need to have an authentication piece? Authorization to ESG AuthZ.
* encrypting the data? Is it through the BES or through the middle tier? If middle tier then no need to encrypt the information through the BES.
* client application to need to allow login to get data? Scientist gets a URL back to access data through OPeNDAP. To get that data, have to be logged in? Authorization needs to happen.
* Certificate is on the client disk, when request made within matlab client, grabs certificate and sends along with data URL

===== Throttling the response =====
* Function added to DDS and base types to be able to pre-compute the response size based on projections and selections
* Should also work for sequences, at least be able to say that you can have x number of rows returned.

----

==== Lunch ====
{|
|James
| :
|Pizza Margarita
|-
|Dan
| :
|Corned Beef
|-
|Patrick
| :
|Veggy Pizza
|-
|Michael
| :
|Chikin Sandwich
|-
|Nathan
| :
|Turkey Club Sandwich
|}

----

==== Easier install for Hyrax ====

# Modify nightly build to use ''shrew''
# Build RPMs from the NB shrew
## Add dependencies to all RPM spec files
# Modify all Makefiles to support pkg/dmg binaries
# Integrate into NB on OS/X Server VMs
# build meta package for OS/X using above
# Run RPM/Linux NB on VMs too
# Test both PRMs and pkg/dmgs on ''completely'' clean VMs

==== SQL Handler ====

# Add to README and INSTALL; fill these in and make them standard WRT the other README/INSTALL files.
# Add to the README so that there is a simple 'How To' for the server's configutation and then write stuff up for the docs wiki.
# Look at SQL Handler requirements in docs wiki and match up to current developed functionality
# Getting attributes

==== JGOFS ====
* Still a lot of organizations using jgofs and wanting opendap access
* Current library is very difficult to work with and still fork/exec the methods
* Could take the library and convert to use the autotools
** Also change from fork and exec to dynamically loaded module
* Or be a pass-thru/gateway/proxy module, like WCS, and just pass the request back to the proper JGOFS server that is already running.
** Figure out how to return catalog information from the jgofs servers
** How do we get attributes/metadata...

==== Active file system ====
''A filesystem that creates a "signal" whenever something changes.''

'''In many cases this boils down to caching binary objects in/around the BES'''

* One way to do this is to use ODB (http://www.codesynthesis.com/products/odb/)
* Another way is to use the boost library (http://www.boost.org/)

----

==== 1800 - Dinner ====

Mmmmmmm Irish/Mexican
----

----

----
===Thursday===

==== 0830 - Coffee ====
: Mmmmmm that was tasty...

==== NcML Ingestion ====

* Sending in NcML from a client
* Use a proxy server to shield clients from the mechanics
** The aggregations need to persist and at least some of this must be at the origin server
** The NcML is held at a proxy server and sent to the origin server where it's validated, etc. and the origin server (must be a Hyrax) returns an id that can be used to reference the aggregation
** The proxy retains that id and takes requests against it.
* Can we simplfy this and remove the proxy given that the aggregation has to persist?
* How long does the aggregation persist? That is, what criteria are used to 'flush' an 'uploaded' aggregation.

===== Why we want this? =====
* EML --> Building aggreagations at a client
* TDS migration

===== Design mechanics for this feature =====
* Errors: How to get information about bad ncml back to a client
* Assme that the server (Hyrax) is modified to use POST for this feature
* Handle errors by returning text as built by the NCML handler
* OLFS currently has no POST termini and we could make an explict terminus for this ''but'' we could make a special servlet for this feature

===== Alternatives =====
* Two things: We add support for HTTP acccess to the ncml handler - this is how we handle aggreagtions of remote entities
* For the NcML migration we modify the BES/NcML commands to support uploading NCML from the OLFS to a specific BES/NCML-handler instance
* Advantages: This solves the first problem w/o any changes to exisitng origin servers at the cost of poor accesses to real-world aggreations. Actually useful aggregations would be sent to the 'origin server' by hand between people.
* There seem to be no real increased costs for NCML migration.

===== To support exteranal access =====
* NcML handler becomes a HTTP client
* Might require remote URLs to be listed explicitly? Performance implemented

==== BES Internal Caching ====

===== Caching compressed data =====

* The BES caching issue is only regarding the compressed files, their decompression and maintaining a restricted 'cache' size.
* This caching scheme needs to include a two stage cache where a single space with no size limit is used for at most one item. This is used for the actual decompression phase. Only one writer may access this at any time.
* A second space is used to store N-bytes of decompressed files. This space must lock all accesses during a write (and must block on both reads and writes). Call this the 'main cache.'
* The main cache must store spin locks for read access to all of the cache items.
* Modify access to items in the cache so that they are accessed using an object such that it's dtor updates the spin lock (so that the spin lock is manged even when exceptions are thrown).
* The object (above) also provides access to a path to the cached item; this path is accessed and passed to the handlers.
* See libdap's HTTPCache and Response classes for an example implmentation

===== Caching binary objects =====

* While it's desireable it's also really hard to retro-fit
* To serialize the DDS we would have to serialize BaseType and all of the concrete classes that are derived from it (NCByte, ...)
* It's not clear that caching these kinds o objects will provide a huge benefit for users/clients.

===== The NcML handler will have to do its own caching and add information to HTTP caching GETs=====
* It will need to support the Last Modified Time of aggregations if we're going to get caching systems like Squid working correctly. Actually, for ANY ncml file, not just aggregations, we need to give the BES a proxy for the modification time. The wrapped dataset could change, but not the NcML, which implies the cache is invalidated but not reflected by the ncml file time.

* This will need to be a feature added to the BES for the showInfo command. Basically, it will need to allow the modules to register a delegate function to fill in the info response for a given file. This include the last modified time as well as the size of the file, since the NcML does not reflect the size of the actual wrapped dataset. The default response (for no delegate registered) will be the current BES behaviour of filling in the file mod time and size.
----

==== Lunch ====
{|
|James
| :
|Veggie Burger
|-
|Dan
| :
|Something tasty
|-
|Patrick
| :
|Veggy Burger
|-
|Michael
| :
|Corned Beef on Rye
|-
|Nathan
| :
|Chikin Salad
|}
----

=== BES module architecture ===

* Show Version: This is a built-in BES command
* <showVersion .../>
* show version --> BESVersionResponseHandler (if this mapping exists, then we can build a response for this command)
* a ResponseHandler knows hw to make the response object and how to get it filled in (and in this case it does fill it in a little bit)
* So you do get a repnse from this command from this command with no modules loaded.
* Sequence:
** BESVersionResponseHandler
*** Makes response object
*** knows how to get it filled in
**** Go to each response handler and see if they know how to fill in version
* Now... Load a module like the Dap Module or NCRequestHandler knows how to fill in a version response.
* It (NCResponseHandler) registers this capability

* Response handlers are assoicated with commands (like 'show version' or 'get')
* Request handlers are associated with modules (like things that read files or process DAP object)

* <get type="dds" defintion="d" /> (BES command)
* The XML command that reads the above turns this into: getDDS
* Ths calls the DapDDSResponseHandler.
** Thsi knows how to make the response object (whcih is a DDS)
** It goes to eac container in the definition (d in this ex) and goes to te matching Request Handler and asks for the dds
** The Response handler knows how to send the response

Now, for the caching....

* < UpdateCache/>
* NcML_Module
** Init
*** updateCache --> XML___
*** updateCache --> UCResponseHandler

----

==== 1800 - Dinner ====

----
----
----

=== Friday ===

==== 0830 - Coffee ====
==== Module check ====
:: Making sure a loaded module matches the version of BES and any required modules

==== SSL authentication/authorization issues ====
:: Currently problem with SSL authentication and keeping SSL channel open for secure communication

==== Multiple catalogs ====
:: BES and OLFS integration

==== 1230 - Lunch ====

==== Strategy Breakout (cont) ====

==== 1800 - Dinner and Departure ====

BES - Modules - NcML Module

2010-09-17T18:35:45Z

Mjohnson: /* Introduction */

= Introduction =
[[Category:NCML]]
[[Category:BES Modules|NCML Module]]

'''Welcome to the OPeNDAP NcML Data Handler Module v1.1.0 for [http://www.opendap.org/download/hyrax.html Hyrax 1.6.2]!'''

[Updated for NcML Module v1.1.0 for Hyrax 1.6.2 Release -- 15 September 2010]

This module may be added to a [http://www.opendap.org/download/hyrax.html Hyrax 1.6.2] server to extend its data
serving capability to NcML 2.2 files (see
http://www.unidata.ucar.edu/software/netcdf/ncml/). NcML provides support for modifying other datasets in various ways, such as adding metadata and data and aggregating multiple datasets in several ways.

We refer the reader to the Unidata NcML tutorial: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html which will give the basics of using NcML. We then give a reference manual for the various elements and their attributes. Additionally we have provided a more extensive tutorial on NcML aggregation. Please see [[#Aggregation Tutorials]].

Since the DAP Grid is a common case, we will also give a simple example for adding metadata to the various parts of a Grid dataset using NcML. Please see [[Grid_Metadata_Tutorial]].

== Current Release ==

The most recent release is 1.1.0, which is bundled with (and requires) Hyrax server 1.6.2. To download, visit the [http://www.opendap.org/download/ncml_handler.html NcML Module Page].

New features in this release:

* joinExisting aggregation initial implementation ([[NCML_Module_Aggregation_JoinExisting]])

==Features==

This version currently implements a subset of NcML 2.2
functionality, along with some OPeNDAP extensions:

* Metadata Manipulation
** Addition, Removal, and Modification of attributes to other datasets (NetCDF, HDF4, HD5, etc.) served by the same Hyrax 1.6 server
** Extends NcML 2.2 to allow for common nested "attribute containers"
** Attributes can be DAP2 types as well as the NcML types
** Attributes can be of the special "OtherXML" type for injecting arbitrary XML into a DDX response
* Data Manipulation
** Addition of new data variables (scalars or arrays of basic types as well as structures)
** Variables may be removed from the wrapped dataset
** Allows the creation of "pure virtual" datasets which do not wrap another dataset
* Aggregations: JoinNew, JoinExisting Union [[#Aggregation Tutorials]]
** JoinNew Aggregation ([[NCML_Module_Aggregation_JoinNew]])
*** Allows multiple datasets to be "joined" by creating a new outer dimension for the aggregated variable
*** Aggregation member datasets can be listed explicitly with explicit coordinates for the new dimension for each member
*** Scan: Aggregations can be specified "automatically" by scanning a directory for files matching certain criteria, such as a suffix or regular expression.
*** Metadata may be added to the new coordinate variable for the new dimension
** JoinExisting Aggregation ([[NCML_Module_Aggregation_JoinExisting]])
*** Currently requires the author to specify the ''ncoords'' attribute.
*** Scan may also be used with ''ncoords'' attribute for uniform sized granules
*** Only allows join dimension to be aggregated from granules and not overridden in NcML
** Union Aggregation ([[NCML_Module_Aggregation_Union]])
*** Merges all member datasets into one by taking the first named instance of variables and metadata from the members
*** Useful for combining two or more datasets with different variables into a single set

== Installation from Source ==
For information on how to build and install the NcML Data Module, please see the INSTALL file that came with the source distribution.

= Installation Overview =

The NcML Module requires a working Hyrax 1.6 installation. It is a module
that is dynamically loaded into the Hyrax BES (Back End Server) to
allow it to handle NcML files.

Please see the file INSTALL for full build and install instructions as
well as requirements.

'''NOTE:''' After installation, you MUST restart Hyrax by restarting the BES and
OLFS so the NcML Module is loaded!

== Requirement: International Components for Unicode (ICU) Library ==

The most important external requirement is an installation of the International Components for Unicode (ICU) version 3.6 or higher (tested up to 4.2.1). The source distributions (as well as some binaries) may be found at the site: http://site.icu-project.org/download

If you are using Linux RPM's to run Hyrax, you can get an RPM for ICU as well. Search for the RPM named "libicu" using a package manager or yum, e.g. If you are compiling the module from source, you will also need the RPM "libicu-devel" to get the headers installed.

If you install in the default locations, the ncml_module should find the libraries and headers. Otherwise, please consult the INSTALL file for more information about installing ICU to a non-standard location.

== Testing Installation ==

Test data is provided to see if the installation was successful. The file sample_virtual_dataset.ncml is a dataset purely created in NcML and doesn't contain an underlying dataset.
You may also view fnoc1_improved.ncml to test adding attributes to an existing netCDF dataset (fnoc1.nc), but this requires the netCDF data handler to be
installed first! Several other examples installed also use the HDF4 and HDF5 handlers.

= Functionality =

This version of the NcML Module implements a subset of NcML 2.2
functionality. The reader is directed to http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/ for more information on NcML.

Our module can currently:

*Refer only to files being served locally (not remotely)
*Add, modify, and remove attribute metadata to a dataset
*Create a purely virtual dataset using just NcML and no underlying dataset
*Create new scalar variables of any simple NcML type or simple DAP type
*Create new Structure variables (which can contain new child variables)
*Create new N-dimensional arrays of simple types (NcML or DAP)
*Remove existing variables from a wrapped dataset
*Rename existing variables in a wrapped dataset
*Name dimensions as a mnemonic for specifying Array shapes
*Perform union aggregations on multiple datasets, virtual or wrapped or both
*Perform joinNew aggregations to merge a variable across multiple datasets by creating a new outer dimension
*Specify aggregation member datasets by scanning directories for files matching certain criteria

We describe each supported NcML element in detail below.

== <netcdf> Element ==

The <netcdf> element is used to define a dataset, either a wrapped dataset that is to be modified, a pure virtual dataset, or a member dataset of an aggregation. The <netcdf> element is assumed to be the topmost node, or as a child of an aggregation element.

=== Local vs. Remote Datasets ===

We assume that the location attribute (''netcdf@location'') refers to the full path (with respect to the BES data root directory) of a '''''local''''' dataset (served by the same Hyrax server). The current version of the module cannot be used to modify remote datasets.

If ''netcdf@location'' is the empty string (or unspecified, as empty is the default), the dataset is a pure virtual dataset, fully specified within the NcML file itself. Attributes and variables may be fully described and accessed with constraints just as normal datasets in this manner. The installed sample datafile "sample_virtual_dataset.ncml" is an example test case for this functionality.

=== Unsupported Attributes ===

The current version does not support the following attributes of <netcdf>:

*enhance
*addRecords
*fmrcDefinition (will be supported when FMRC aggregation is added)
*ncoords (will be supported when joinExisting is added)

== <readMetadata> Element ==

The <readMetadata/> element is the default, so is effectively not needed.

== <explicit> element ==

The <explicit/> element simply clears all attribute tables in the referred to netcdf@location before applying the rest of the
NcML transformations to the metadata.

== <dimension> Element ==

The <dimension> element has limited functionality in this release since the DAP2 doesn't support dimensions as more than mnemonics at this time. The limitations are:

* We only parse the ''dimension@name'' and ''dimension@length'' attributes.
* Dimensions can only be specified as a direct child of a <netcdf> element prior to any reference to them

For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="samples" length="5"/>

</netcdf>
</pre>

The dimension element sets up a mapping from the ''name'' to the unsigned integer ''length'' and can be used in a ''variable@shape'' to specify a length for an array dimension (see the section on <variable> below). The dimension map is cleared when </netcdf> is encountered (though this doesn't matter currently since we allow only one right now, but it will matter for aggregation, potentially). We also do not support <group>, which is the only other legal place in NcML 2.2 for a dimension element.

'''Parse Errors''':
* If the name and length are not both specified.
* If the dimension name already exists in the current scope
* If the length is not an unsigned integer
* If any of the other attributes specified in NcML 2.2 are used. We do not handle them, so we consider them errors now.

== <variable> Element ==

The <variable> element is used to:

* Provide lexical scope for a contained <attribute> or <variable> element
* Rename existing variables
* Add new scalar variables of simple types
* Add new Structure variables
* Add new N-dimensional Array's of simple types
* Specify the coordinate variable for the new dimension in a joinNew aggregation

We describe each in turn in more detail.

=== Specifying Lexical Scope with <variable type=""> ===

Consider the following example:

<variable name="u">
<attribute name="Metadata" type="string">This is metadata!</attribute>
</variable>

This code assumes that a variable named "u" exists (of any type since we do not specify) and provides the lexical scope for the
attribute "Metadata" which will be added or modified within the attribute table for the variable "u" (it's qualified name would be "u.Metadata").

==== Nested DAP Structure and Grid Scopes ====

Scoping variable elements may be nested if the containing variable is a Structure (this includes the special case of Grid)

<pre>
<variable name="DATA_GRANULE" type="Structure">
<variable name="PlanetaryGrid" type="Structure">
<variable name="percipitate">
<attribute name="units" type="String" value="inches"/>
</variable>
</variable>
</variable>
</pre>

This adds a "unit" attribute to the variable "percipitate" within the nested Structure's
("DATA_GRANULE.PlanetaryGrid.percipitate" as fully qualified name).
Note that we '''must''' refer to the type explicitly as a "Structure" so the parser knows
to traverse the tree.

'''Note''' the variable might be of type Grid, but the type "Structure" must be used in the NcML to traverse it.

==== Adding Multiple Attributes to the Same Variable ====

Once the variable's scope is set by the opening <variable> element, more than one attribute can be specified within it. This will make the NcML more readable and also will make the parsing more efficient since the variable will only need to be looked up once.

For example,

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

should be preferred over:

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
</variable>

<variable name="Foo">
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

although they produce the same result. Any number of attributes can be specified before the variable is closed.

=== Renaming Existing Variables ===

The attribute ''variable@orgName'' is used to rename an existing variable.

For example:

<pre>
<variable name="NewName" orgName="OldName"/>
</pre>

will rename an existing variable at the current scope named "OldName" to "NewName". After this point in the NcML file (such as in constraints specified for the DAP request), the variable is known by "NewName".

Note that the type is not required here --- the variable is assumed to exist and its existing type is used. It is not possible to change the type of an existing variable at this time!

'''Parse Errors''':
* If a variable with ''variable@orgName'' doesn't exist in the current scope
* If the new name ''variable@name'' is already taken in the current scope
* If a new variable is created but does not have exactly one values element

=== Adding a New Scalar Variable ===

The <variable> element can be used to create a new scalar variable of a simple type (i.e. an atomic NcML type such as "int" or "float", or any DAP atomic type, such as "UInt32" or "URL") by specifying an empty ''variable@shape'' (which is the default), a simple type for ''variable@type'', and a contained <values> element with the one value of correct type.

For example:

<pre>
<variable name="TheAnswerToLifeTheUniverseAndEverything" type="double">
<attribute name="SolvedBy" type="String" value="Deep Thought"/>
<values>42.000</values>
</variable>
</pre>

will create a new variable named "TheAnswerToLifeTheUniverseAndEverything" at the current scope. It has no shape so will be a scalar of type "double" and will have the value 42.0.

'''Parse Errors''':
* It is a parse error to not specify a <values> element with exactly one proper value of the variable type.
* It is a parse error to specify a malformed or out of bounds value for the data type

=== Adding a New Structure Variable ===

A new Structure variable can be specified at the global scope or within another Structure. It is illegal for an array to have type structure, so the shape must be empty.

For example:

<pre>
<variable name="MyNewStructure" type="Structure">
<attribute name="MetaData" type="String" value="This is metadata!"/>
<variable name="ContainedScalar1" type="String"><values>I live in a new structure!</values></variable>
<variable name="ContainedInt1" type="int"><values>42</values></variable>
</variable>
</pre>

specifies a new structure called "MyNewStructure" which contains two scalar variable fields "ContainedScalar1" and "ContainedInt1".

Nested structures are allowed as well.

'''Parse Error''':
* If another variable or attribute exists at the current scope with the new name.
* If a <values> element is specified as a direct child of a new Structure --- structures cannot contain values, only attributes and other variables.

=== Adding a New N-dimensional Array ===

An N-dimensional array of a simple type may be created virtually as well by specifying a non-empty ''variable@shape''. The shape contains the array dimensions in left-to-right order of slowest varying dimension first. For example:

<pre>
<variable name="FloatArray" type="float" shape="2 5">

<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will specify a 2x5 dimension array of float values called "FloatArray". The <values> element must contain 2x5=10 values in row major order (slowest varying dimension first). Since whitespace is the default separator, we use a newline to show the dimension boundary for the values, which is easy to see for a 2D matrix such as this.

A dimension name may also be used to refer mnemonically to a length. The DAP response will use this mnemonic in its output, but it is not currently used for shared dimensions, only as a mnemonic. See the section on the <dimension> element for more information. For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="sample" length="5"/>
<variable name="FloatArray" type="float" shape="station sample">
<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will produce the same 2x5 array, but will incorporate the dimension mnemonics into the response. For example, here's the DDS response:

<pre>
Dataset {
Float32 FloatArray[station = 2][samples = 5];
} sample_virtual_dataset.ncml;
</pre>

Note that the <values> element respects the ''values@separator'' attribute if whitespace isn't correct. This is very useful for arrays of strings with whitespace, for example.

<pre>
<variable name="StringArray" type="string" shape="3">
<values separator="*">String 1*String 2*String 3</values>
</variable>
</pre>

creates a length 3 array of string StringArray = {"String 1", "String 2", "String 3"}.

'''Parse Errors''':
* It is an error to specify the incorrect number of values
* It is an error if any value is malformed or out of range for the data type.
* It is an error to specify a named dimension which does not exist in the current <netcdf> scope.
* It is an error to specify an Array whose flattened size (product of dimensions) is > 2^31-1.

=== Specifying the new coordinate variable for a joinNew aggregation ===

In the special case of a joinNew aggregation, the new coordinate variable may be specified with the <variable> element. The new coordinate variable is ''defined'' to have the same name as the new dimension. This allows for several things:

*Explicit specification of the variable type and coordinates for the new dimension
*Specification of the metadata for the new coordinate variable

In the first case, the author can specify explicitly the type of the new coordinate variable and the actual values for each dataset. In this case, the variable ''must'' be specified ''after'' the aggregation element in the file so the new dimension's size (number of member datasets) may be known and error checking performed. Metadata can also be added to the variable here.

In the second case, the author may just specify the variable name, which allows one to specify the metadata for a coordinate variable that is automatically generated by the aggregation itself. This is the only allowable case for a variable element to ''not'' contain a values element! Coordinate variables are generated automatically in two cases:

*The author has specified an explicit list of member datasets, with or without explicit coordVal attributes.
*The author has used a <scan> element to specify the member datasets via a directory scan

In this case, the <variable> element may come before or after the <aggregation>.

'''Parse Errors:'''

*If an explicit variable is declared for the new coordinate variable:
** And it contains explicit values, the number of values must be equal to the number of member datasets in the aggregation.
** It must be specifed ''after'' the <aggregation> element

* If a numeric coordVal is used to specify the first member dataset's coordinate, then ''all'' datasets must contain a numerical coordinate.

* An error is thrown if the specified aggregation variable (variableAgg) is not found in ''all'' member datasets.

* An error is thrown if the specified aggregation variable is not of the same type in ''all'' member datasets. Coercion is ''not'' performed!

* An error is thrown if the specified aggregation variables in all member datasets do not have the same shape

* An error is thrown if an explicit coordinate variable is specified with a shape that is ''not'' the same as the new dimension name (and the variable name itself).

== <values> Element ==

The <values> element can only be used in the context of a '''new''' variable of scalar or array type. We cannot change the values for existing variables in this version of the handler.
The characters content of a <values> element is considered to be a separated list of value tokens valid for the type of the variable of the parent element. The number of specified tokens in the content ''must'' equal the product of the dimensions of the enclosing ''variable@shape'', or be one value for a scalar. It is an error to ''not'' specify a <values> element for a declared new variable as well.

=== Changing the Separator Tokens ===

The author may specify values@separator to change the default value token separator from the default whitespace. This is very useful for specifying arrays of strings with whitespace in them, or if data in CSV form is being pasted in.

=== Autogeneration of Uniform Arrays ===

We also can parse ''values@start'' and ''values@increment'' INSTEAD OF tokens in the content. This will "autogenerate" a uniform array of values of the given product of dimensions length for the containing variable. For example:

<pre>
<variable name="Evens" type="int" shape="100">
<values start="0" increment="2"/>
</variable>
</pre>

will specify an array of the first 100 even numbers (including 0).

'''Parse Errors''':
* If the incorrect number of tokens are specified for the containing variable's shape
* If any value token cannot be parsed as a valid value for the containing variable's type
* If content is specified in addition to start and increment
* If only one of start or increment is specified
* If the values element is placed anywhere except within a NEW variable.

== <attribute> Element ==

As an overview, whenever the parser encounters an <attribute> with a non-existing name (at the current scope),
it creates a new one, whether a container or atomic attribute (see below). If the attribute exists,
its value and/or type is modified to those specified in the <attribute> element. If an attribute
structure (container) exists, it is used to define a nested lexical scope for child attributes.

Attributes may be scalar (one value) or one dimensional arrays. Arrays are specified by using whitespace (default) to separate the different values. The attribute@separator may also be set in order to specify a different separator, such as CSV format or to specify a non-whitespace separator so strings with whitespace are not tokenized. We will give examples of creating array attributes below.

=== Adding New Attributes or Modifying an Existing Attribute ===

If a specified attribute with the attribute@name does not exist at the current lexical scope,
a new one is created with the given type and value. For example, assume "new_metadata" doesn't
exist at the current parse scope. Then:

<attribute name="new_metadata" type="string" value="This is a new entry!"/>

will create the attribute at that scope. Note that value can be specified in the content of the
element as well. This is identical to the above:

<attribute name="new_metadata" type="string">This is a new entry!</attribute>

If the attribute@name already exists at the scope, it is modified to contain the specified type and value.

==== Arrays ====

As in NcML, for numerical types an array can be specified by separating the tokens by whitespace (default) or be
specifying the token separator with attribute@separator. For example,

<attribute name="myArray" type="int">1 2 3</attribute>

and

<attribute name="myArray" type="int" separator=",">1,2,3</attribute>

both specify the same array of three integers named "myArray".

TODO Add more information on splitting with a separator!

==== Structures (Containers) ====

We use attribute@type="Structure" to define a new (or existing) attribute container. So if we
wanted to add a new attribute structure, we'd use something like this:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

Assuming "MySamples" doesn't already exist, an attribute container will be created at the current scope
and the "Location" and "Samples" attributes will be added to it.

Note that we can create nested attribute structures to arbitrary depth this way as well.

If the attribute container with the given name already exists at the current scope,
then the attribute@type="Structure" form is used to define
the lexical scope for the container. In other words, child <attribute> elements will
be processed within the scope of the container. For example, in the above example, if
"MySamples" already exists, then the "Location" and "Samples" will be processed within the
existing container (they may or may not already exist as well).

==== Renaming an Existing Attribute or Attribute Container ====

We also support the attribute@orgName attribute for renaming attributes.

For example,

<attribute name="NewName" orgName="OldName" type="string"/>

will rename an existing attribute "OldName" to "NewName" while leaving its value alone.
If attribute@value is also specified, then the attribute is renamed ''and'' has its value modified.

This works for renaming attribute containers as well:

<attribute name="MyNewContainer" orgName="MyOldContainer" type="Structure"/>

will rename an existing "MyOldContainer" to "MyNewContainer". Note that any children
of this container will remain in it.

=== DAP ''OtherXML'' Extension ===

The module now allows specification of attributes of the new DAP type "OtherXML". This allows the NCML file author to inject arbitrary well-formed XML into an attribute for clients that want XML metadata rather than just string or url. Internally, the attribute is still a string (and in a DAP DAS response will be quoted inside one string). However, since it is XML, the NCMLParser still parses it and checks it for well-formedness (but NOT against schemas). This extension allows the NCMLParser to parse the arbitrary XML within the given attribute without causing errors, since it can be any XML.

The injected XML is most useful in the DDX response, where it shows up directly in the response as XML. XSLT and other clients can then parse it.

==== Errors ====

*The XML '''must''' be in the content of the <attribute type="OtherXML"> element. It is a parser error for ''attribute@value'' to be set if ''attribute@type'' is "OtherXML".
*The XML must also be well-formed since it is parsed. A parse error will be thrown if the OtherXML is malformed.

==== Example ====

Here's an example of the use of this special case.

<pre>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/coverage/200803061600_HFRadar_USEGC_6km_rtv_SIO.nc">

<attribute name="someName" type="OtherXML">
<Domain xmlns="http://www.opengis.net/wcs/1.1"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
>
<SpatialDomain>
<ows:BoundingBox crs="urn:ogc:def:crs:EPSG::4326">
<ows:LowerCorner>-97.8839 21.736</ows:LowerCorner>
<ows:UpperCorner>-57.2312 46.4944</ows:UpperCorner>
</ows:BoundingBox>
</SpatialDomain>
<TemporalDomain>
<gml:timePosition>2008-03-27T16:00:00.000Z</gml:timePosition>
</TemporalDomain>
</Domain>
<SupportedCRS xmlns="http://www.opengis.net/wcs/1.1">urn:ogc:def:crs:EPSG::4326</SupportedCRS>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">netcdf-cf1.0</SupportedFormat>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">dap2.0</SupportedFormat>
</attribute>

</netcdf>
</pre>

'''TODO'' Put the DDX response for the above in here!

====Namespace Closure====

Furthermore, the parser will make the chunk of OtherXML "namespace closed". This means any namespaces specified in parent NCML elements of the OtherXML tree will be "brought down" and added to the ''root'' OtherXML elements so that the subtree may be pulled out and added to the DDX and still have its namespaces. The algorithm doesn't just bring used prefixes, but brings ''all'' of the lexically scoped closest namespaces in all ancestors. In other words, it adds unique namespaces (as determined by prefix) in order from the root of the OtherXML tree as it traverses to the root of the NCML document.

Namespace closure is a syntactic sugar that simplifies the author's task since they can specify the namespaces just once at the top of the NCML file and expect that when the subtree of XML is added to the DDX that these namespaces will come along with that subtree of XML. Otherwise they have to explicitly add the namespaces to each attributes.

'''TODO''' Add an example!

== <remove> Element ==

The <remove> element can remove attributes and variables. For example:

<pre>
<attribute name="NC_GLOBAL" type="Structure">
<remove name="base_time" type="attribute"/>
</attribute>
</pre>

will remove the attribute named "base_time" in the attribute structure named "NC_GLOBAL".

Note that this works for attribute containers as well! We could recursively remove the ''entire''
attribute container (i.e. it and all its children) with:

<pre>
<remove name="NC_GLOBAL" type="attribute"/>
</pre>

It also can be used to remove variables from existing datasets:

</pre>
<remove name="SomeExistingVariable" type="variable"/>
</pre>

This also recurses on variables of type Structure --- the entire structure including all of its children are removed from the dataset's response.

'''Parse Errors''':
* It is a parse error if the given attribute or variable doesn't exist in the current scope

== <aggregation> Element ==

Aggregation involves combining multiple datasets (<netcdf>) into a virtual "single" dataset in various ways. For a tutorial on aggregation in NcML 2.2, the reader is referred to the Unidata page: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

NcML 2.2 supports multiple types of aggregation: union, joinNew, joinExisting, and fmrc (forecast model run collection).

The current version of the NcML module supports two of these aggregations:

*Union [[NCML_Module_Aggregation_Union]]
*JoinNew [[NCML_Module_Aggregation_JoinNew]]

A ''union'' aggregation specifies that the first instance of a variable or attribute (by name) that is found in the ordered list of datasets will be the one in the output aggregation. This is useful for combining two dataset files, each which may contain a single variable, into a composite dataset with both variables.

A JoinNew aggregation joins a variable which exists in multiple datasets (usually samples of a datum over time) into a new variable containing the data from ''all'' member datasets by creating a new outer dimension. The ''i''th component in the new outer dimension is the variable's data from the ''i''th member dataset. It also adds a new coordinate variable of whose name is the new dimension's name and whose shape (length) is the new dimension as well. This new coordinate variable may be explicitly given by the author or may be autogenerated in one of several ways.

== <scan> Element ==

The scan element can be used within an aggregation context to allow a directory to be searched in various ways in order to specify the members of an aggregation. This allows a static NcML file to refer to an aggregation which may change over time, such as where a new data file is generated each day.

We describe usage of the <scan> element in detail in the joinNew aggregation tutorial at [[NCML_Module_Aggregation_JoinNew]].

= Errors =

There are three types of error messages that may be returned:

* Internal Error
* Resource Not Found Error
* Parse Error

== Internal Errors ==
'''Internal errors''' should be reported to support@opendap.org as they are likely bugs.

== Resource Not Found Errors ==

If the netcdf@location specifies a non-existent local dataset (one that is not being served by
the same Hyrax server), it will specify the resource was not found. This may also be returned
if a handler for the specified dataset is not currently loaded in the BES. Users should test that the
dataset to be wrapped already exists and can be viewed on the running server before writing NcML
to add metadata. It's also an error to refer to remote datasets (at this time).

== Parse Errors ==

'''Parse errors''' are user errors in the NcML file. These could be malformed XML, malformed NcML,
unimplemented features of NcML, or could be errors in referring to the wrapped dataset.

The error message should specify
the error condition as well as the "current scope" as a fully qualified DAP name within the
loaded dataset. This should be enough information to correct the parse error as new NcML
files are created.

The parser will generate parse errors in various
situations where it expects to find certain structure in the underlying dataset. Some examples:

* A variable of the given name was not found at the current scope
* attribute@orgName was specified, but the attribute cannot be found at current scope.
* attribute@orgName was specified, but the new name is already used at current scope.
* remove specified a non-existing attribute name

= Grid Metadata Tutorial =

Please see the page [[Grid_Metadata_Tutorial]] for an example of adding metadata to the various parts of a DAP Grid variable.

= Aggregation Tutorials =

The NcML module may also be used to aggregate multiple datasets into one virtual dataset. We currently support two of the NcML aggregtions:

*Union [[NCML_Module_Aggregation_Union]]: combine multiple datasets into one by merging variables together, selecting the first of each unique name.
*JoinNew [[NCML_Module_Aggregation_JoinNew]]: combine variables across multiple datasets by creating a new outer dimension and coordinate variable for each of the sample datasets.

Please see the sections for a tutorial on the various uses of these aggregations.

= Additions/Changes to NcML 2.2 =

This section will keep track of changes to the NcML 2.2 schema. Eventually these
will be rolled into a new schema.

== Attribute Structures (Containers) ==
This module also adds functionality beyond the current NcML 2.2 schema
--- it can handle nested <attribute> elements in order to make
attribute structures. This is done by using the <attribute
type="Structure"> form, for example:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

"MyContainer" describes an attribute structure with two attribute fields,
a string "Location" and an array of int's called "Samples".
Note that an attribute structure of this form can
only contain other <attribute> elements and NOT a value.

If the container does not already exist, it will be created at the scope it is declared, which could
be:

* Global (top of dataset)
* Within a variable's attribute table
* Within another attribute container

If an attribute container of the given name already exists at the lexical scope, it is traversed
in order to define the scope for the nested (children) attributes it contains.

== Unspecified Variable Type Matching for Lexical Scope ==

We also allow the type attribute of a variable element (variable@type) to be the empty string
(or unspecified) when using existing variables to define the lexical scope of an <attribute>
transformation. In the schema, variable@type is (normally) required.

== DAP 2 Types ==

Additionally, we allow DAP2 atomic types (such as UInt32, URL) in addition to the NcML types.
The NcML types are mapped onto the closest DAP2 type internally.

== DAP OtherXML Attribute Type ==

We also allow attributes to be of the new DAP type "OtherXML" for injecting arbitrary XML into an attribute as content rather than trying to form a string. This allows the parser to check well-formedness.

== Forward Declaration of Dimensions ==

Since we use a SAX parser for efficiency, we require the <dimension> elements to come ''before'' their use in a ''variable@shape''. One way to change the schema to allow this is to force the dimension elements to be specified in a sequence after explicit and metadata choice and before all other elements.

== Aggregation Element Location and Processing Order Differences ==

NcML specifies that if a dataset (<netcdf> element) specifies an aggregation element, the aggregation element is always processed first, regardless of its ordering within the <netcdf> element. Our parser, since it is SAX and not DOM, modifies this behavior in that order matters in some cases:

* Metadata (<attribute>) elements specified ''prior'' to an aggregation "shadow" the aggregation versions. This is be useful for "overriding" an attribute or variable in a union aggregation, where the first found will take precedence.
* JoinNew: If the new coordinate variable's data is to be set explicitly by specifying the new dimension's shape (either with explicit data or the autogenerated data using values@start and values@increment attributes), the <variable> ''must'' come after the aggregation since the size of the dimension is unknown until the aggregation element is processed.

= Backward Compatibility Issues =

Due to the way shared dimensions were implemented in the NetCDF, HDF4, and HDF5 handlers, the DAS responses did not follow the DAP2 specification. The NcML module, on the other hand, generates DAP2 compliant DAS for these datasets, which means that wrapping some datasets in NcML will generate a DAS with a different structure. This is important for the NcML author since it changes the names of attributes and variables. In order for the module to find the correct scope for adding metadata, for example, the DAP2 DAS must be used.

In general, what this means is that an empty "passthrough" NcML file should be the starting point for authoring an NcML file. This file would just specify a dataset and nothing else:

<pre>
<netcdf location="/data/ncml/myNetcdf.nc"/>
</pre>

The author would then request the DAS response for the NCML file and use that as the starting point for modifications to the original dataset.

More explicit examples are given below.

== NetCDF ==

The NetCDF handler represents some NC datasets as a DAP 2 Grid, but the returned DAS is not consistent with the DAP 2 spec for the attribute hierarchy for such a Grid. The map vector attributes are placed as siblings of the grid attributes rather than within the grid lexical scope. For example, here's the NetCDF Handler DDS for a given file:

<pre>
Dataset {
Grid {
Array:
Int16 cldc[time = 456][lat = 21][lon = 360];
Maps:
Float64 time[time = 456];
Float32 lat[lat = 21];
Float32 lon[lon = 360];
} cldc;
} cldc.mean.nc;
</pre>

showing the Grid. Here's the DAS the NetCDF handler generates:

<pre>
Attributes {
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
}
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
}
</pre>

Note the map vector attributes are in the "dataset" scope.

Here's the DAS that the NcML Module produces from the correctly formed DDX:

<pre>
Attributes {
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
cldc {
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
}
}
</pre>

Here the Grid Structure "cldc" and its contained data array (of the same name "cldc") and map vectors have their own attribute containers as DAP 2 specifies.

What this means for the author of an NcML file adding metadata to a NetCDF dataset that returns a Grid is that they should generate a "passthrough" file and get the DAS and then specify modifications based on that structure.

Here's an example passthrough:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">
</netcdf>
</pre>

For example, to add an attribute to the map vector "lat" in the above, we'd need the following NcML:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">

<variable name="cldc" type="Structure">

<variable name="lat">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lat!</attribute>
</variable>
<variable name="lon">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lon!</attribute>
</variable>
</variable>
</netcdf>
</pre>

This clearly shows that the structure of the Grid must be used in the NcML: the attribute being added is technically "cldc.lat.Description" in a fully qualified name. The parser would return an error if it was attempted as "lat.Description" as the NetCDF DAS for the original file would have led one to believe.

== HDF4/HDF5 ==

Similarly to the NetCDF case, the Hyrax HDF4 Module produces DAS responses that do not respect the DAP2 specification. If an NcML file is used to "wrap" an HDF4 dataset, the
correct DAP2 DAS response will be generated, however.

This is important for those writing NcML for HDF4 data since the
lexical scope for attributes relies on the correct DAS form --- to
handle this, the user should start with a "passthrough" NcML file (see the above NetCDF example) and
use the DAS from that as the starting point for knowing the structure
the NcML handler expects to see in the NcML file. Alternatively, the DDX has the
proper attribute structure as well (the DAS is generated from it).

= Known Bugs =

There are no known bugs currently.

= Planned Future Enhancements =

Planned enhancements for future versions of the module include:

* New NcML Aggregations
** JoinExisting
*** Joins a variable across multiple datasets by appending the data for a given dimension from each dataset
*** Will also allow directory scans for specifying the aggregation
** Forecast Model Run Collection (FMRC)
*** Special case of JoinNew for forecast data with two time variables
*** See: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html

= Copyright =

This software is copyrighted under the GNU Lesser GPL. Please see the
files COPYING and COPYRIGHT that came with this distribution.

NCML Module Aggregation JoinExisting

2010-09-17T18:29:49Z

Mjohnson: /* Future Work */

[[Category:Aggregation]]
[[Category:NCML]]

= Join Existing Aggregation =

A ''joinExisting'' aggregation joins multiple granule datasets by concatenating the specified outer dimensional data from the granules into the output. This results in matrices of the same number of dimensions, but with larger outer dimension cardinality. The outer dimension sizes of the granules may vary across granule, but any inner dimensions for multi-dimensional data still are required to match.

The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinExisting. Note that version 1.1.0 of the module does not support all features of joinExisting! Future versions will add more features.

'''PLEASE NOTE''' that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.

== Introduction ==

This page describes the behavior of the initial implementation of joinExisting for version 1.1.0 of the NcML Module, bundled with Hyrax 1.6.2. It is a limited feature set described below. Please see the Limitations section for more information.

In version 1.1.0, a joinExisting aggregation may be specified in two ways currently:

* Using explicit lists of '''netcdf''' elements with the ''ncoords'' attribute correctly specified for all of them.
* Using a '''scan''' element with ''ncoords'' specified and all matching granule datasets having this dimension size

Our example below will clarify this.

Future versions of the module will implement more of the joinExisting feature set.

== Examples ==

Here we give an example that illustrates the functionality offered by the current version of the aggregation. This example may also be found on:

http://test.opendap.org:8090/opendap/ioos/mday_joinExist.ncml

with the data granules located in

http://test.opendap.org:8090/opendap/coverage/mday/

=== Granules ===

Assume we have some number of granule datasets with a DDS the same as the following (modulo the dataset name):

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 1][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 1];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
} PH2006001_2006031_ssta.nc;
</pre>

=== Explicit Listing of Granules ===

We see that here ''time'' is the outer dimension, which is the only dimension we may join along (it is an error to specify an inner). Given some number of granules with this same shape, consider the following explicit joinExisting aggregation:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules">

<aggregation type="joinExisting"
dimName="time" >


<netcdf location="/coverage/mday/PH2006001_2006031_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006032_2006059_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006060_2006090_ssta.nc"
ncoords="1">

</aggregation>

</netcdf>
</pre>

First, note that the ''ncoords'' attribute '''MUST''' be specified on the individual granules for this version of the module (future versions will relax this requirement and automatically and efficiently load the sizes). Note that we also specify the ''dimName''. Any data array whose outer dimension is called this will be subject to aggregation in the output.

Serving this from Hyrax will result in the following DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

We see that the time dimension is now of size 3 to match that we joined three granule datasets together.

Also notice that the map vector for the joined dimension, time, has been duplicated as a sibling of the dataset. This is done automatically by the aggregation and it is copied into the actual map of the Grid. This copy is to facilitate datasets which have multiple Grid's that are to be joined --- the top-level map vector is used as the canonical template map which is then copied into the maps for all the aggregated Grids. In the case of the joined data being of type Array, this vector would already exist as the coordinate variable for the data matrix. Since this is the source map for all aggregated Grid's, any attribute (metadata) changes should be made explicitly on this top-level coordinate variable so that the metadata is shared among all the aggregated Grid map vectors.

=== Using the scan element with ncoords extension ===

Since all of the granules are of uniform size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules using scan@ncoords"
>

<attribute name="Description" type="string"
value=" joinExisting test on netcdf Grid granules using scan@ncoords"/>

<aggregation type="joinExisting"
dimName="time" >


<scan location="/coverage/mday"
subdirs="false"
suffix=".nc"
ncoords="1"
/>

</aggregation>

</netcdf>

</pre>

which we see results in the same DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

== Limitations ==

The current version implements only basic functionality. If there is extended functionality that is needed for your use, please send <mailto:support@opendap.org> to let us know!

=== Join Dimension Sizes Must Be Explicitly Declared ===

As we have seen, the most important limitation is that the ncoords must be specified for efficiency reasons. Future versions will relax this requirement. The problem is that the size of the output join dimension is dependent on checking the DDS of ''every'' granule in the aggregation, which is computationally expensive for large aggregations unless it is computed once and cached. The next feature release of the module will include a caching system for solving this general problem and allowing the ncoords to be loaded automatically rather than being specified.

=== Source of Data for Agrgegated Coordinate Variable on Join Dimension ===

This version does not allow the join dimension's data to be declared explicitly in the NcML as the NcML tutorial page describes. This version automatically aggregates '''all''' variables with the outer dimension matching the ''dimName''. This includes the coordinate variable (map vector in the case of Grid's) for the join dimension. These data cannot be overridden from those pulled from the files. Currently the TDS lists about 5 ways this data can be specified in addition to pulling them from the granules --- we only can pull them from granules now, which seems the most common use.

=== Source of Join Dimension Metadata ===

The metadata for the coordinate variable is pulled from the ''first'' granule dataset. Modification of coordinate variable metadata is not fully supported yet.

== Future Work ==

The most important addition in the next release will be the ability to efficiently load and cache the granule dimension sizes offline to allow for a highly interactive user experience when data is requested. This means that the ''ncoords'' attribute will no longer be required on the granules, although doing so results in a more efficient aggregation speed.

We also plan to allow the data and metadata for the join dimension coordinate variable to be overridden and modified.

NCML Module Aggregation JoinExisting

2010-09-17T18:26:42Z

Mjohnson: /* Limitations */

[[Category:Aggregation]]
[[Category:NCML]]

= Join Existing Aggregation =

A ''joinExisting'' aggregation joins multiple granule datasets by concatenating the specified outer dimensional data from the granules into the output. This results in matrices of the same number of dimensions, but with larger outer dimension cardinality. The outer dimension sizes of the granules may vary across granule, but any inner dimensions for multi-dimensional data still are required to match.

The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinExisting. Note that version 1.1.0 of the module does not support all features of joinExisting! Future versions will add more features.

'''PLEASE NOTE''' that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.

== Introduction ==

This page describes the behavior of the initial implementation of joinExisting for version 1.1.0 of the NcML Module, bundled with Hyrax 1.6.2. It is a limited feature set described below. Please see the Limitations section for more information.

In version 1.1.0, a joinExisting aggregation may be specified in two ways currently:

* Using explicit lists of '''netcdf''' elements with the ''ncoords'' attribute correctly specified for all of them.
* Using a '''scan''' element with ''ncoords'' specified and all matching granule datasets having this dimension size

Our example below will clarify this.

Future versions of the module will implement more of the joinExisting feature set.

== Examples ==

Here we give an example that illustrates the functionality offered by the current version of the aggregation. This example may also be found on:

http://test.opendap.org:8090/opendap/ioos/mday_joinExist.ncml

with the data granules located in

http://test.opendap.org:8090/opendap/coverage/mday/

=== Granules ===

Assume we have some number of granule datasets with a DDS the same as the following (modulo the dataset name):

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 1][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 1];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
} PH2006001_2006031_ssta.nc;
</pre>

=== Explicit Listing of Granules ===

We see that here ''time'' is the outer dimension, which is the only dimension we may join along (it is an error to specify an inner). Given some number of granules with this same shape, consider the following explicit joinExisting aggregation:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules">

<aggregation type="joinExisting"
dimName="time" >


<netcdf location="/coverage/mday/PH2006001_2006031_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006032_2006059_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006060_2006090_ssta.nc"
ncoords="1">

</aggregation>

</netcdf>
</pre>

First, note that the ''ncoords'' attribute '''MUST''' be specified on the individual granules for this version of the module (future versions will relax this requirement and automatically and efficiently load the sizes). Note that we also specify the ''dimName''. Any data array whose outer dimension is called this will be subject to aggregation in the output.

Serving this from Hyrax will result in the following DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

We see that the time dimension is now of size 3 to match that we joined three granule datasets together.

Also notice that the map vector for the joined dimension, time, has been duplicated as a sibling of the dataset. This is done automatically by the aggregation and it is copied into the actual map of the Grid. This copy is to facilitate datasets which have multiple Grid's that are to be joined --- the top-level map vector is used as the canonical template map which is then copied into the maps for all the aggregated Grids. In the case of the joined data being of type Array, this vector would already exist as the coordinate variable for the data matrix. Since this is the source map for all aggregated Grid's, any attribute (metadata) changes should be made explicitly on this top-level coordinate variable so that the metadata is shared among all the aggregated Grid map vectors.

=== Using the scan element with ncoords extension ===

Since all of the granules are of uniform size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules using scan@ncoords"
>

<attribute name="Description" type="string"
value=" joinExisting test on netcdf Grid granules using scan@ncoords"/>

<aggregation type="joinExisting"
dimName="time" >


<scan location="/coverage/mday"
subdirs="false"
suffix=".nc"
ncoords="1"
/>

</aggregation>

</netcdf>

</pre>

which we see results in the same DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

== Limitations ==

The current version implements only basic functionality. If there is extended functionality that is needed for your use, please send <mailto:support@opendap.org> to let us know!

=== Join Dimension Sizes Must Be Explicitly Declared ===

As we have seen, the most important limitation is that the ncoords must be specified for efficiency reasons. Future versions will relax this requirement. The problem is that the size of the output join dimension is dependent on checking the DDS of ''every'' granule in the aggregation, which is computationally expensive for large aggregations unless it is computed once and cached. The next feature release of the module will include a caching system for solving this general problem and allowing the ncoords to be loaded automatically rather than being specified.

=== Source of Data for Agrgegated Coordinate Variable on Join Dimension ===

This version does not allow the join dimension's data to be declared explicitly in the NcML as the NcML tutorial page describes. This version automatically aggregates '''all''' variables with the outer dimension matching the ''dimName''. This includes the coordinate variable (map vector in the case of Grid's) for the join dimension. These data cannot be overridden from those pulled from the files. Currently the TDS lists about 5 ways this data can be specified in addition to pulling them from the granules --- we only can pull them from granules now, which seems the most common use.

=== Source of Join Dimension Metadata ===

The metadata for the coordinate variable is pulled from the ''first'' granule dataset. Modification of coordinate variable metadata is not fully supported yet.

== Future Work ==

NCML Module Aggregation JoinExisting

2010-09-17T18:16:07Z

Mjohnson:

[[Category:Aggregation]]
[[Category:NCML]]

= Join Existing Aggregation =

A ''joinExisting'' aggregation joins multiple granule datasets by concatenating the specified outer dimensional data from the granules into the output. This results in matrices of the same number of dimensions, but with larger outer dimension cardinality. The outer dimension sizes of the granules may vary across granule, but any inner dimensions for multi-dimensional data still are required to match.

The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinExisting. Note that version 1.1.0 of the module does not support all features of joinExisting! Future versions will add more features.

'''PLEASE NOTE''' that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.

== Introduction ==

This page describes the behavior of the initial implementation of joinExisting for version 1.1.0 of the NcML Module, bundled with Hyrax 1.6.2. It is a limited feature set described below. Please see the Limitations section for more information.

In version 1.1.0, a joinExisting aggregation may be specified in two ways currently:

* Using explicit lists of '''netcdf''' elements with the ''ncoords'' attribute correctly specified for all of them.
* Using a '''scan''' element with ''ncoords'' specified and all matching granule datasets having this dimension size

Our example below will clarify this.

Future versions of the module will implement more of the joinExisting feature set.

== Examples ==

Here we give an example that illustrates the functionality offered by the current version of the aggregation. This example may also be found on:

http://test.opendap.org:8090/opendap/ioos/mday_joinExist.ncml

with the data granules located in

http://test.opendap.org:8090/opendap/coverage/mday/

=== Granules ===

Assume we have some number of granule datasets with a DDS the same as the following (modulo the dataset name):

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 1][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 1];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
} PH2006001_2006031_ssta.nc;
</pre>

=== Explicit Listing of Granules ===

We see that here ''time'' is the outer dimension, which is the only dimension we may join along (it is an error to specify an inner). Given some number of granules with this same shape, consider the following explicit joinExisting aggregation:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules">

<aggregation type="joinExisting"
dimName="time" >


<netcdf location="/coverage/mday/PH2006001_2006031_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006032_2006059_ssta.nc"
ncoords="1">
<netcdf location="/coverage/mday/PH2006060_2006090_ssta.nc"
ncoords="1">

</aggregation>

</netcdf>
</pre>

First, note that the ''ncoords'' attribute '''MUST''' be specified on the individual granules for this version of the module (future versions will relax this requirement and automatically and efficiently load the sizes). Note that we also specify the ''dimName''. Any data array whose outer dimension is called this will be subject to aggregation in the output.

Serving this from Hyrax will result in the following DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

We see that the time dimension is now of size 3 to match that we joined three granule datasets together.

Also notice that the map vector for the joined dimension, time, has been duplicated as a sibling of the dataset. This is done automatically by the aggregation and it is copied into the actual map of the Grid. This copy is to facilitate datasets which have multiple Grid's that are to be joined --- the top-level map vector is used as the canonical template map which is then copied into the maps for all the aggregated Grids. In the case of the joined data being of type Array, this vector would already exist as the coordinate variable for the data matrix. Since this is the source map for all aggregated Grid's, any attribute (metadata) changes should be made explicitly on this top-level coordinate variable so that the metadata is shared among all the aggregated Grid map vectors.

=== Using the scan element with ncoords extension ===

Since all of the granules are of uniform size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:

<pre>
<?xml version="1.0" encoding="UTF-8"?>

<netcdf title="joinExisting test on netcdf Grid granules using scan@ncoords"
>

<attribute name="Description" type="string"
value=" joinExisting test on netcdf Grid granules using scan@ncoords"/>

<aggregation type="joinExisting"
dimName="time" >


<scan location="/coverage/mday"
subdirs="false"
suffix=".nc"
ncoords="1"
/>

</aggregation>

</netcdf>

</pre>

which we see results in the same DDS:

<pre>
Dataset {
Grid {
Array:
Float32 PHssta[time = 3][altitude = 1][lat = 4096][lon = 8192];
Maps:
Float64 time[time = 3];
Float64 altitude[altitude = 1];
Float64 lat[lat = 4096];
Float64 lon[lon = 8192];
} PHssta;
Float64 time[time = 3];
} mday_joinExist.ncml;
</pre>

== Limitations ==

== Future Work ==

NCML Module Aggregation JoinExisting

2010-09-17T17:34:39Z

Mjohnson: New page: Category:Aggregation Category:NCML = Join Existing Aggregation = A ''joinExisting'' aggregation joins existing TODO .... The reader is also directed to a basic tutorial of this...

[[Category:Aggregation]]
[[Category:NCML]]

= Join Existing Aggregation =

A ''joinExisting'' aggregation joins existing TODO ....

The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinExisting.

'''PLEASE NOTE''' that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.

== Introduction ==

== Examples ==

== Limitations ==

== Future Work ==

BES - Modules - NcML Module

2010-09-17T17:32:13Z

Mjohnson: /* Introduction */

= Introduction =
[[Category:NCML]]
[[Category:BES Modules|NCML Module]]

'''Welcome to the OPeNDAP NcML Data Handler Module v1.1.0 for Hyrax 1.6.2!'''

[Updated for NcML Module v1.1.0 for Hyrax 1.6.2 Release -- 15 September 2010]

This module may be added to a Hyrax 1.6.2 server to extend its data
serving capability to NcML 2.2 files (see
http://www.unidata.ucar.edu/software/netcdf/ncml/). NcML provides support for modifying other datasets in various ways, such as adding metadata and data and aggregating multiple datasets in several ways.

We refer the reader to the Unidata NcML tutorial: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html which will give the basics of using NcML. We then give a reference manual for the various elements and their attributes. Additionally we have provided a more extensive tutorial on NcML aggregation. Please see [[#Aggregation Tutorials]].

Since the DAP Grid is a common case, we will also give a simple example for adding metadata to the various parts of a Grid dataset using NcML. Please see [[Grid_Metadata_Tutorial]].

==Features==

This version currently implements a subset of NcML 2.2
functionality, along with some OPeNDAP extensions:

* Metadata Manipulation
** Addition, Removal, and Modification of attributes to other datasets (NetCDF, HDF4, HD5, etc.) served by the same Hyrax 1.6 server
** Extends NcML 2.2 to allow for common nested "attribute containers"
** Attributes can be DAP2 types as well as the NcML types
** Attributes can be of the special "OtherXML" type for injecting arbitrary XML into a DDX response
* Data Manipulation
** Addition of new data variables (scalars or arrays of basic types as well as structures)
** Variables may be removed from the wrapped dataset
** Allows the creation of "pure virtual" datasets which do not wrap another dataset
* Aggregations: JoinNew, JoinExisting Union [[#Aggregation Tutorials]]
** JoinNew Aggregation ([[NCML_Module_Aggregation_JoinNew]])
*** Allows multiple datasets to be "joined" by creating a new outer dimension for the aggregated variable
*** Aggregation member datasets can be listed explicitly with explicit coordinates for the new dimension for each member
*** Scan: Aggregations can be specified "automatically" by scanning a directory for files matching certain criteria, such as a suffix or regular expression.
*** Metadata may be added to the new coordinate variable for the new dimension
** JoinExisting Aggregation ([[NCML_Module_Aggregation_JoinExisting]])
*** Currently requires the author to specify the ''ncoords'' attribute.
*** Scan may also be used with ''ncoords'' attribute for uniform sized granules
*** Only allows join dimension to be aggregated from granules and not overridden in NcML
** Union Aggregation ([[NCML_Module_Aggregation_Union]])
*** Merges all member datasets into one by taking the first named instance of variables and metadata from the members
*** Useful for combining two or more datasets with different variables into a single set

== Installation from Source ==
For information on how to build and install the NcML Data Module, please see the INSTALL file that came with the source distribution.

= Installation Overview =

The NcML Module requires a working Hyrax 1.6 installation. It is a module
that is dynamically loaded into the Hyrax BES (Back End Server) to
allow it to handle NcML files.

Please see the file INSTALL for full build and install instructions as
well as requirements.

'''NOTE:''' After installation, you MUST restart Hyrax by restarting the BES and
OLFS so the NcML Module is loaded!

== Requirement: International Components for Unicode (ICU) Library ==

The most important external requirement is an installation of the International Components for Unicode (ICU) version 3.6 or higher (tested up to 4.2.1). The source distributions (as well as some binaries) may be found at the site: http://site.icu-project.org/download

If you are using Linux RPM's to run Hyrax, you can get an RPM for ICU as well. Search for the RPM named "libicu" using a package manager or yum, e.g. If you are compiling the module from source, you will also need the RPM "libicu-devel" to get the headers installed.

If you install in the default locations, the ncml_module should find the libraries and headers. Otherwise, please consult the INSTALL file for more information about installing ICU to a non-standard location.

== Testing Installation ==

Test data is provided to see if the installation was successful. The file sample_virtual_dataset.ncml is a dataset purely created in NcML and doesn't contain an underlying dataset.
You may also view fnoc1_improved.ncml to test adding attributes to an existing netCDF dataset (fnoc1.nc), but this requires the netCDF data handler to be
installed first! Several other examples installed also use the HDF4 and HDF5 handlers.

= Functionality =

This version of the NcML Module implements a subset of NcML 2.2
functionality. The reader is directed to http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/ for more information on NcML.

Our module can currently:

*Refer only to files being served locally (not remotely)
*Add, modify, and remove attribute metadata to a dataset
*Create a purely virtual dataset using just NcML and no underlying dataset
*Create new scalar variables of any simple NcML type or simple DAP type
*Create new Structure variables (which can contain new child variables)
*Create new N-dimensional arrays of simple types (NcML or DAP)
*Remove existing variables from a wrapped dataset
*Rename existing variables in a wrapped dataset
*Name dimensions as a mnemonic for specifying Array shapes
*Perform union aggregations on multiple datasets, virtual or wrapped or both
*Perform joinNew aggregations to merge a variable across multiple datasets by creating a new outer dimension
*Specify aggregation member datasets by scanning directories for files matching certain criteria

We describe each supported NcML element in detail below.

== <netcdf> Element ==

The <netcdf> element is used to define a dataset, either a wrapped dataset that is to be modified, a pure virtual dataset, or a member dataset of an aggregation. The <netcdf> element is assumed to be the topmost node, or as a child of an aggregation element.

=== Local vs. Remote Datasets ===

We assume that the location attribute (''netcdf@location'') refers to the full path (with respect to the BES data root directory) of a '''''local''''' dataset (served by the same Hyrax server). The current version of the module cannot be used to modify remote datasets.

If ''netcdf@location'' is the empty string (or unspecified, as empty is the default), the dataset is a pure virtual dataset, fully specified within the NcML file itself. Attributes and variables may be fully described and accessed with constraints just as normal datasets in this manner. The installed sample datafile "sample_virtual_dataset.ncml" is an example test case for this functionality.

=== Unsupported Attributes ===

The current version does not support the following attributes of <netcdf>:

*enhance
*addRecords
*fmrcDefinition (will be supported when FMRC aggregation is added)
*ncoords (will be supported when joinExisting is added)

== <readMetadata> Element ==

The <readMetadata/> element is the default, so is effectively not needed.

== <explicit> element ==

The <explicit/> element simply clears all attribute tables in the referred to netcdf@location before applying the rest of the
NcML transformations to the metadata.

== <dimension> Element ==

The <dimension> element has limited functionality in this release since the DAP2 doesn't support dimensions as more than mnemonics at this time. The limitations are:

* We only parse the ''dimension@name'' and ''dimension@length'' attributes.
* Dimensions can only be specified as a direct child of a <netcdf> element prior to any reference to them

For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="samples" length="5"/>

</netcdf>
</pre>

The dimension element sets up a mapping from the ''name'' to the unsigned integer ''length'' and can be used in a ''variable@shape'' to specify a length for an array dimension (see the section on <variable> below). The dimension map is cleared when </netcdf> is encountered (though this doesn't matter currently since we allow only one right now, but it will matter for aggregation, potentially). We also do not support <group>, which is the only other legal place in NcML 2.2 for a dimension element.

'''Parse Errors''':
* If the name and length are not both specified.
* If the dimension name already exists in the current scope
* If the length is not an unsigned integer
* If any of the other attributes specified in NcML 2.2 are used. We do not handle them, so we consider them errors now.

== <variable> Element ==

The <variable> element is used to:

* Provide lexical scope for a contained <attribute> or <variable> element
* Rename existing variables
* Add new scalar variables of simple types
* Add new Structure variables
* Add new N-dimensional Array's of simple types
* Specify the coordinate variable for the new dimension in a joinNew aggregation

We describe each in turn in more detail.

=== Specifying Lexical Scope with <variable type=""> ===

Consider the following example:

<variable name="u">
<attribute name="Metadata" type="string">This is metadata!</attribute>
</variable>

This code assumes that a variable named "u" exists (of any type since we do not specify) and provides the lexical scope for the
attribute "Metadata" which will be added or modified within the attribute table for the variable "u" (it's qualified name would be "u.Metadata").

==== Nested DAP Structure and Grid Scopes ====

Scoping variable elements may be nested if the containing variable is a Structure (this includes the special case of Grid)

<pre>
<variable name="DATA_GRANULE" type="Structure">
<variable name="PlanetaryGrid" type="Structure">
<variable name="percipitate">
<attribute name="units" type="String" value="inches"/>
</variable>
</variable>
</variable>
</pre>

This adds a "unit" attribute to the variable "percipitate" within the nested Structure's
("DATA_GRANULE.PlanetaryGrid.percipitate" as fully qualified name).
Note that we '''must''' refer to the type explicitly as a "Structure" so the parser knows
to traverse the tree.

'''Note''' the variable might be of type Grid, but the type "Structure" must be used in the NcML to traverse it.

==== Adding Multiple Attributes to the Same Variable ====

Once the variable's scope is set by the opening <variable> element, more than one attribute can be specified within it. This will make the NcML more readable and also will make the parsing more efficient since the variable will only need to be looked up once.

For example,

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

should be preferred over:

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
</variable>

<variable name="Foo">
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

although they produce the same result. Any number of attributes can be specified before the variable is closed.

=== Renaming Existing Variables ===

The attribute ''variable@orgName'' is used to rename an existing variable.

For example:

<pre>
<variable name="NewName" orgName="OldName"/>
</pre>

will rename an existing variable at the current scope named "OldName" to "NewName". After this point in the NcML file (such as in constraints specified for the DAP request), the variable is known by "NewName".

Note that the type is not required here --- the variable is assumed to exist and its existing type is used. It is not possible to change the type of an existing variable at this time!

'''Parse Errors''':
* If a variable with ''variable@orgName'' doesn't exist in the current scope
* If the new name ''variable@name'' is already taken in the current scope
* If a new variable is created but does not have exactly one values element

=== Adding a New Scalar Variable ===

The <variable> element can be used to create a new scalar variable of a simple type (i.e. an atomic NcML type such as "int" or "float", or any DAP atomic type, such as "UInt32" or "URL") by specifying an empty ''variable@shape'' (which is the default), a simple type for ''variable@type'', and a contained <values> element with the one value of correct type.

For example:

<pre>
<variable name="TheAnswerToLifeTheUniverseAndEverything" type="double">
<attribute name="SolvedBy" type="String" value="Deep Thought"/>
<values>42.000</values>
</variable>
</pre>

will create a new variable named "TheAnswerToLifeTheUniverseAndEverything" at the current scope. It has no shape so will be a scalar of type "double" and will have the value 42.0.

'''Parse Errors''':
* It is a parse error to not specify a <values> element with exactly one proper value of the variable type.
* It is a parse error to specify a malformed or out of bounds value for the data type

=== Adding a New Structure Variable ===

A new Structure variable can be specified at the global scope or within another Structure. It is illegal for an array to have type structure, so the shape must be empty.

For example:

<pre>
<variable name="MyNewStructure" type="Structure">
<attribute name="MetaData" type="String" value="This is metadata!"/>
<variable name="ContainedScalar1" type="String"><values>I live in a new structure!</values></variable>
<variable name="ContainedInt1" type="int"><values>42</values></variable>
</variable>
</pre>

specifies a new structure called "MyNewStructure" which contains two scalar variable fields "ContainedScalar1" and "ContainedInt1".

Nested structures are allowed as well.

'''Parse Error''':
* If another variable or attribute exists at the current scope with the new name.
* If a <values> element is specified as a direct child of a new Structure --- structures cannot contain values, only attributes and other variables.

=== Adding a New N-dimensional Array ===

An N-dimensional array of a simple type may be created virtually as well by specifying a non-empty ''variable@shape''. The shape contains the array dimensions in left-to-right order of slowest varying dimension first. For example:

<pre>
<variable name="FloatArray" type="float" shape="2 5">

<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will specify a 2x5 dimension array of float values called "FloatArray". The <values> element must contain 2x5=10 values in row major order (slowest varying dimension first). Since whitespace is the default separator, we use a newline to show the dimension boundary for the values, which is easy to see for a 2D matrix such as this.

A dimension name may also be used to refer mnemonically to a length. The DAP response will use this mnemonic in its output, but it is not currently used for shared dimensions, only as a mnemonic. See the section on the <dimension> element for more information. For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="sample" length="5"/>
<variable name="FloatArray" type="float" shape="station sample">
<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will produce the same 2x5 array, but will incorporate the dimension mnemonics into the response. For example, here's the DDS response:

<pre>
Dataset {
Float32 FloatArray[station = 2][samples = 5];
} sample_virtual_dataset.ncml;
</pre>

Note that the <values> element respects the ''values@separator'' attribute if whitespace isn't correct. This is very useful for arrays of strings with whitespace, for example.

<pre>
<variable name="StringArray" type="string" shape="3">
<values separator="*">String 1*String 2*String 3</values>
</variable>
</pre>

creates a length 3 array of string StringArray = {"String 1", "String 2", "String 3"}.

'''Parse Errors''':
* It is an error to specify the incorrect number of values
* It is an error if any value is malformed or out of range for the data type.
* It is an error to specify a named dimension which does not exist in the current <netcdf> scope.
* It is an error to specify an Array whose flattened size (product of dimensions) is > 2^31-1.

=== Specifying the new coordinate variable for a joinNew aggregation ===

In the special case of a joinNew aggregation, the new coordinate variable may be specified with the <variable> element. The new coordinate variable is ''defined'' to have the same name as the new dimension. This allows for several things:

*Explicit specification of the variable type and coordinates for the new dimension
*Specification of the metadata for the new coordinate variable

In the first case, the author can specify explicitly the type of the new coordinate variable and the actual values for each dataset. In this case, the variable ''must'' be specified ''after'' the aggregation element in the file so the new dimension's size (number of member datasets) may be known and error checking performed. Metadata can also be added to the variable here.

In the second case, the author may just specify the variable name, which allows one to specify the metadata for a coordinate variable that is automatically generated by the aggregation itself. This is the only allowable case for a variable element to ''not'' contain a values element! Coordinate variables are generated automatically in two cases:

*The author has specified an explicit list of member datasets, with or without explicit coordVal attributes.
*The author has used a <scan> element to specify the member datasets via a directory scan

In this case, the <variable> element may come before or after the <aggregation>.

'''Parse Errors:'''

*If an explicit variable is declared for the new coordinate variable:
** And it contains explicit values, the number of values must be equal to the number of member datasets in the aggregation.
** It must be specifed ''after'' the <aggregation> element

* If a numeric coordVal is used to specify the first member dataset's coordinate, then ''all'' datasets must contain a numerical coordinate.

* An error is thrown if the specified aggregation variable (variableAgg) is not found in ''all'' member datasets.

* An error is thrown if the specified aggregation variable is not of the same type in ''all'' member datasets. Coercion is ''not'' performed!

* An error is thrown if the specified aggregation variables in all member datasets do not have the same shape

* An error is thrown if an explicit coordinate variable is specified with a shape that is ''not'' the same as the new dimension name (and the variable name itself).

== <values> Element ==

The <values> element can only be used in the context of a '''new''' variable of scalar or array type. We cannot change the values for existing variables in this version of the handler.
The characters content of a <values> element is considered to be a separated list of value tokens valid for the type of the variable of the parent element. The number of specified tokens in the content ''must'' equal the product of the dimensions of the enclosing ''variable@shape'', or be one value for a scalar. It is an error to ''not'' specify a <values> element for a declared new variable as well.

=== Changing the Separator Tokens ===

The author may specify values@separator to change the default value token separator from the default whitespace. This is very useful for specifying arrays of strings with whitespace in them, or if data in CSV form is being pasted in.

=== Autogeneration of Uniform Arrays ===

We also can parse ''values@start'' and ''values@increment'' INSTEAD OF tokens in the content. This will "autogenerate" a uniform array of values of the given product of dimensions length for the containing variable. For example:

<pre>
<variable name="Evens" type="int" shape="100">
<values start="0" increment="2"/>
</variable>
</pre>

will specify an array of the first 100 even numbers (including 0).

'''Parse Errors''':
* If the incorrect number of tokens are specified for the containing variable's shape
* If any value token cannot be parsed as a valid value for the containing variable's type
* If content is specified in addition to start and increment
* If only one of start or increment is specified
* If the values element is placed anywhere except within a NEW variable.

== <attribute> Element ==

As an overview, whenever the parser encounters an <attribute> with a non-existing name (at the current scope),
it creates a new one, whether a container or atomic attribute (see below). If the attribute exists,
its value and/or type is modified to those specified in the <attribute> element. If an attribute
structure (container) exists, it is used to define a nested lexical scope for child attributes.

Attributes may be scalar (one value) or one dimensional arrays. Arrays are specified by using whitespace (default) to separate the different values. The attribute@separator may also be set in order to specify a different separator, such as CSV format or to specify a non-whitespace separator so strings with whitespace are not tokenized. We will give examples of creating array attributes below.

=== Adding New Attributes or Modifying an Existing Attribute ===

If a specified attribute with the attribute@name does not exist at the current lexical scope,
a new one is created with the given type and value. For example, assume "new_metadata" doesn't
exist at the current parse scope. Then:

<attribute name="new_metadata" type="string" value="This is a new entry!"/>

will create the attribute at that scope. Note that value can be specified in the content of the
element as well. This is identical to the above:

<attribute name="new_metadata" type="string">This is a new entry!</attribute>

If the attribute@name already exists at the scope, it is modified to contain the specified type and value.

==== Arrays ====

As in NcML, for numerical types an array can be specified by separating the tokens by whitespace (default) or be
specifying the token separator with attribute@separator. For example,

<attribute name="myArray" type="int">1 2 3</attribute>

and

<attribute name="myArray" type="int" separator=",">1,2,3</attribute>

both specify the same array of three integers named "myArray".

TODO Add more information on splitting with a separator!

==== Structures (Containers) ====

We use attribute@type="Structure" to define a new (or existing) attribute container. So if we
wanted to add a new attribute structure, we'd use something like this:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

Assuming "MySamples" doesn't already exist, an attribute container will be created at the current scope
and the "Location" and "Samples" attributes will be added to it.

Note that we can create nested attribute structures to arbitrary depth this way as well.

If the attribute container with the given name already exists at the current scope,
then the attribute@type="Structure" form is used to define
the lexical scope for the container. In other words, child <attribute> elements will
be processed within the scope of the container. For example, in the above example, if
"MySamples" already exists, then the "Location" and "Samples" will be processed within the
existing container (they may or may not already exist as well).

==== Renaming an Existing Attribute or Attribute Container ====

We also support the attribute@orgName attribute for renaming attributes.

For example,

<attribute name="NewName" orgName="OldName" type="string"/>

will rename an existing attribute "OldName" to "NewName" while leaving its value alone.
If attribute@value is also specified, then the attribute is renamed ''and'' has its value modified.

This works for renaming attribute containers as well:

<attribute name="MyNewContainer" orgName="MyOldContainer" type="Structure"/>

will rename an existing "MyOldContainer" to "MyNewContainer". Note that any children
of this container will remain in it.

=== DAP ''OtherXML'' Extension ===

The module now allows specification of attributes of the new DAP type "OtherXML". This allows the NCML file author to inject arbitrary well-formed XML into an attribute for clients that want XML metadata rather than just string or url. Internally, the attribute is still a string (and in a DAP DAS response will be quoted inside one string). However, since it is XML, the NCMLParser still parses it and checks it for well-formedness (but NOT against schemas). This extension allows the NCMLParser to parse the arbitrary XML within the given attribute without causing errors, since it can be any XML.

The injected XML is most useful in the DDX response, where it shows up directly in the response as XML. XSLT and other clients can then parse it.

==== Errors ====

*The XML '''must''' be in the content of the <attribute type="OtherXML"> element. It is a parser error for ''attribute@value'' to be set if ''attribute@type'' is "OtherXML".
*The XML must also be well-formed since it is parsed. A parse error will be thrown if the OtherXML is malformed.

==== Example ====

Here's an example of the use of this special case.

<pre>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/coverage/200803061600_HFRadar_USEGC_6km_rtv_SIO.nc">

<attribute name="someName" type="OtherXML">
<Domain xmlns="http://www.opengis.net/wcs/1.1"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
>
<SpatialDomain>
<ows:BoundingBox crs="urn:ogc:def:crs:EPSG::4326">
<ows:LowerCorner>-97.8839 21.736</ows:LowerCorner>
<ows:UpperCorner>-57.2312 46.4944</ows:UpperCorner>
</ows:BoundingBox>
</SpatialDomain>
<TemporalDomain>
<gml:timePosition>2008-03-27T16:00:00.000Z</gml:timePosition>
</TemporalDomain>
</Domain>
<SupportedCRS xmlns="http://www.opengis.net/wcs/1.1">urn:ogc:def:crs:EPSG::4326</SupportedCRS>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">netcdf-cf1.0</SupportedFormat>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">dap2.0</SupportedFormat>
</attribute>

</netcdf>
</pre>

'''TODO'' Put the DDX response for the above in here!

====Namespace Closure====

Furthermore, the parser will make the chunk of OtherXML "namespace closed". This means any namespaces specified in parent NCML elements of the OtherXML tree will be "brought down" and added to the ''root'' OtherXML elements so that the subtree may be pulled out and added to the DDX and still have its namespaces. The algorithm doesn't just bring used prefixes, but brings ''all'' of the lexically scoped closest namespaces in all ancestors. In other words, it adds unique namespaces (as determined by prefix) in order from the root of the OtherXML tree as it traverses to the root of the NCML document.

Namespace closure is a syntactic sugar that simplifies the author's task since they can specify the namespaces just once at the top of the NCML file and expect that when the subtree of XML is added to the DDX that these namespaces will come along with that subtree of XML. Otherwise they have to explicitly add the namespaces to each attributes.

'''TODO''' Add an example!

== <remove> Element ==

The <remove> element can remove attributes and variables. For example:

<pre>
<attribute name="NC_GLOBAL" type="Structure">
<remove name="base_time" type="attribute"/>
</attribute>
</pre>

will remove the attribute named "base_time" in the attribute structure named "NC_GLOBAL".

Note that this works for attribute containers as well! We could recursively remove the ''entire''
attribute container (i.e. it and all its children) with:

<pre>
<remove name="NC_GLOBAL" type="attribute"/>
</pre>

It also can be used to remove variables from existing datasets:

</pre>
<remove name="SomeExistingVariable" type="variable"/>
</pre>

This also recurses on variables of type Structure --- the entire structure including all of its children are removed from the dataset's response.

'''Parse Errors''':
* It is a parse error if the given attribute or variable doesn't exist in the current scope

== <aggregation> Element ==

Aggregation involves combining multiple datasets (<netcdf>) into a virtual "single" dataset in various ways. For a tutorial on aggregation in NcML 2.2, the reader is referred to the Unidata page: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

NcML 2.2 supports multiple types of aggregation: union, joinNew, joinExisting, and fmrc (forecast model run collection).

The current version of the NcML module supports two of these aggregations:

*Union [[NCML_Module_Aggregation_Union]]
*JoinNew [[NCML_Module_Aggregation_JoinNew]]

A ''union'' aggregation specifies that the first instance of a variable or attribute (by name) that is found in the ordered list of datasets will be the one in the output aggregation. This is useful for combining two dataset files, each which may contain a single variable, into a composite dataset with both variables.

A JoinNew aggregation joins a variable which exists in multiple datasets (usually samples of a datum over time) into a new variable containing the data from ''all'' member datasets by creating a new outer dimension. The ''i''th component in the new outer dimension is the variable's data from the ''i''th member dataset. It also adds a new coordinate variable of whose name is the new dimension's name and whose shape (length) is the new dimension as well. This new coordinate variable may be explicitly given by the author or may be autogenerated in one of several ways.

== <scan> Element ==

The scan element can be used within an aggregation context to allow a directory to be searched in various ways in order to specify the members of an aggregation. This allows a static NcML file to refer to an aggregation which may change over time, such as where a new data file is generated each day.

We describe usage of the <scan> element in detail in the joinNew aggregation tutorial at [[NCML_Module_Aggregation_JoinNew]].

= Errors =

There are three types of error messages that may be returned:

* Internal Error
* Resource Not Found Error
* Parse Error

== Internal Errors ==
'''Internal errors''' should be reported to support@opendap.org as they are likely bugs.

== Resource Not Found Errors ==

If the netcdf@location specifies a non-existent local dataset (one that is not being served by
the same Hyrax server), it will specify the resource was not found. This may also be returned
if a handler for the specified dataset is not currently loaded in the BES. Users should test that the
dataset to be wrapped already exists and can be viewed on the running server before writing NcML
to add metadata. It's also an error to refer to remote datasets (at this time).

== Parse Errors ==

'''Parse errors''' are user errors in the NcML file. These could be malformed XML, malformed NcML,
unimplemented features of NcML, or could be errors in referring to the wrapped dataset.

The error message should specify
the error condition as well as the "current scope" as a fully qualified DAP name within the
loaded dataset. This should be enough information to correct the parse error as new NcML
files are created.

The parser will generate parse errors in various
situations where it expects to find certain structure in the underlying dataset. Some examples:

* A variable of the given name was not found at the current scope
* attribute@orgName was specified, but the attribute cannot be found at current scope.
* attribute@orgName was specified, but the new name is already used at current scope.
* remove specified a non-existing attribute name

= Grid Metadata Tutorial =

Please see the page [[Grid_Metadata_Tutorial]] for an example of adding metadata to the various parts of a DAP Grid variable.

= Aggregation Tutorials =

The NcML module may also be used to aggregate multiple datasets into one virtual dataset. We currently support two of the NcML aggregtions:

*Union [[NCML_Module_Aggregation_Union]]: combine multiple datasets into one by merging variables together, selecting the first of each unique name.
*JoinNew [[NCML_Module_Aggregation_JoinNew]]: combine variables across multiple datasets by creating a new outer dimension and coordinate variable for each of the sample datasets.

Please see the sections for a tutorial on the various uses of these aggregations.

= Additions/Changes to NcML 2.2 =

This section will keep track of changes to the NcML 2.2 schema. Eventually these
will be rolled into a new schema.

== Attribute Structures (Containers) ==
This module also adds functionality beyond the current NcML 2.2 schema
--- it can handle nested <attribute> elements in order to make
attribute structures. This is done by using the <attribute
type="Structure"> form, for example:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

"MyContainer" describes an attribute structure with two attribute fields,
a string "Location" and an array of int's called "Samples".
Note that an attribute structure of this form can
only contain other <attribute> elements and NOT a value.

If the container does not already exist, it will be created at the scope it is declared, which could
be:

* Global (top of dataset)
* Within a variable's attribute table
* Within another attribute container

If an attribute container of the given name already exists at the lexical scope, it is traversed
in order to define the scope for the nested (children) attributes it contains.

== Unspecified Variable Type Matching for Lexical Scope ==

We also allow the type attribute of a variable element (variable@type) to be the empty string
(or unspecified) when using existing variables to define the lexical scope of an <attribute>
transformation. In the schema, variable@type is (normally) required.

== DAP 2 Types ==

Additionally, we allow DAP2 atomic types (such as UInt32, URL) in addition to the NcML types.
The NcML types are mapped onto the closest DAP2 type internally.

== DAP OtherXML Attribute Type ==

We also allow attributes to be of the new DAP type "OtherXML" for injecting arbitrary XML into an attribute as content rather than trying to form a string. This allows the parser to check well-formedness.

== Forward Declaration of Dimensions ==

Since we use a SAX parser for efficiency, we require the <dimension> elements to come ''before'' their use in a ''variable@shape''. One way to change the schema to allow this is to force the dimension elements to be specified in a sequence after explicit and metadata choice and before all other elements.

== Aggregation Element Location and Processing Order Differences ==

NcML specifies that if a dataset (<netcdf> element) specifies an aggregation element, the aggregation element is always processed first, regardless of its ordering within the <netcdf> element. Our parser, since it is SAX and not DOM, modifies this behavior in that order matters in some cases:

* Metadata (<attribute>) elements specified ''prior'' to an aggregation "shadow" the aggregation versions. This is be useful for "overriding" an attribute or variable in a union aggregation, where the first found will take precedence.
* JoinNew: If the new coordinate variable's data is to be set explicitly by specifying the new dimension's shape (either with explicit data or the autogenerated data using values@start and values@increment attributes), the <variable> ''must'' come after the aggregation since the size of the dimension is unknown until the aggregation element is processed.

= Backward Compatibility Issues =

Due to the way shared dimensions were implemented in the NetCDF, HDF4, and HDF5 handlers, the DAS responses did not follow the DAP2 specification. The NcML module, on the other hand, generates DAP2 compliant DAS for these datasets, which means that wrapping some datasets in NcML will generate a DAS with a different structure. This is important for the NcML author since it changes the names of attributes and variables. In order for the module to find the correct scope for adding metadata, for example, the DAP2 DAS must be used.

In general, what this means is that an empty "passthrough" NcML file should be the starting point for authoring an NcML file. This file would just specify a dataset and nothing else:

<pre>
<netcdf location="/data/ncml/myNetcdf.nc"/>
</pre>

The author would then request the DAS response for the NCML file and use that as the starting point for modifications to the original dataset.

More explicit examples are given below.

== NetCDF ==

The NetCDF handler represents some NC datasets as a DAP 2 Grid, but the returned DAS is not consistent with the DAP 2 spec for the attribute hierarchy for such a Grid. The map vector attributes are placed as siblings of the grid attributes rather than within the grid lexical scope. For example, here's the NetCDF Handler DDS for a given file:

<pre>
Dataset {
Grid {
Array:
Int16 cldc[time = 456][lat = 21][lon = 360];
Maps:
Float64 time[time = 456];
Float32 lat[lat = 21];
Float32 lon[lon = 360];
} cldc;
} cldc.mean.nc;
</pre>

showing the Grid. Here's the DAS the NetCDF handler generates:

<pre>
Attributes {
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
}
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
}
</pre>

Note the map vector attributes are in the "dataset" scope.

Here's the DAS that the NcML Module produces from the correctly formed DDX:

<pre>
Attributes {
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
cldc {
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
}
}
</pre>

Here the Grid Structure "cldc" and its contained data array (of the same name "cldc") and map vectors have their own attribute containers as DAP 2 specifies.

What this means for the author of an NcML file adding metadata to a NetCDF dataset that returns a Grid is that they should generate a "passthrough" file and get the DAS and then specify modifications based on that structure.

Here's an example passthrough:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">
</netcdf>
</pre>

For example, to add an attribute to the map vector "lat" in the above, we'd need the following NcML:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">

<variable name="cldc" type="Structure">

<variable name="lat">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lat!</attribute>
</variable>
<variable name="lon">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lon!</attribute>
</variable>
</variable>
</netcdf>
</pre>

This clearly shows that the structure of the Grid must be used in the NcML: the attribute being added is technically "cldc.lat.Description" in a fully qualified name. The parser would return an error if it was attempted as "lat.Description" as the NetCDF DAS for the original file would have led one to believe.

== HDF4/HDF5 ==

Similarly to the NetCDF case, the Hyrax HDF4 Module produces DAS responses that do not respect the DAP2 specification. If an NcML file is used to "wrap" an HDF4 dataset, the
correct DAP2 DAS response will be generated, however.

This is important for those writing NcML for HDF4 data since the
lexical scope for attributes relies on the correct DAS form --- to
handle this, the user should start with a "passthrough" NcML file (see the above NetCDF example) and
use the DAS from that as the starting point for knowing the structure
the NcML handler expects to see in the NcML file. Alternatively, the DDX has the
proper attribute structure as well (the DAS is generated from it).

= Known Bugs =

There are no known bugs currently.

= Planned Future Enhancements =

Planned enhancements for future versions of the module include:

* New NcML Aggregations
** JoinExisting
*** Joins a variable across multiple datasets by appending the data for a given dimension from each dataset
*** Will also allow directory scans for specifying the aggregation
** Forecast Model Run Collection (FMRC)
*** Special case of JoinNew for forecast data with two time variables
*** See: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html

= Copyright =

This software is copyrighted under the GNU Lesser GPL. Please see the
files COPYING and COPYRIGHT that came with this distribution.

BES - Modules - NcML Module

2010-07-09T17:17:00Z

Mjohnson: /* Introduction */

= Introduction =
[[Category:NCML]]
[[Category:BES Modules|NCML Module]]

'''Welcome to the OPeNDAP NcML Data Handler Module v1.0.3 for Hyrax 1.6!'''

[Updated for NcML Module v1.0.3 for Hyrax 1.6.1 Release -- 9 July 2010]

This module may be added to a Hyrax 1.6 server to extend its data
serving capability to NcML 2.2 files (see
http://www.unidata.ucar.edu/software/netcdf/ncml/). NcML provides support for modifying other datasets in various ways, such as adding metadata and data and aggregating multiple datasets in several ways.

We refer the reader to the Unidata NcML tutorial: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html which will give the basics of using NcML. We then give a reference manual for the various elements and their attributes. Additionally we have provided a more extensive tutorial on NcML aggregation. Please see [[#Aggregation Tutorials]].

Since the DAP Grid is a common case, we will also give a simple example for adding metadata to the various parts of a Grid dataset using NcML. Please see [[Grid_Metadata_Tutorial]].

==Features==

This version currently implements a subset of NcML 2.2
functionality, along with some OPeNDAP extensions:

* Metadata Manipulation
** Addition, Removal, and Modification of attributes to other datasets (NetCDF, HDF4, HD5, etc.) served by the same Hyrax 1.6 server
** Extends NcML 2.2 to allow for common nested "attribute containers"
** Attributes can be DAP2 types as well as the NcML types
** Attributes can be of the special "OtherXML" type for injecting arbitrary XML into a DDX response
* Data Manipulation
** Addition of new data variables (scalars or arrays of basic types as well as structures)
** Variables may be removed from the wrapped dataset
** Allows the creation of "pure virtual" datasets which do not wrap another dataset
* Aggregations: JoinNew and Union [[#Aggregation Tutorials]]
** JoinNew Aggregation ([[NCML_Module_Aggregation_JoinNew]])
*** Allows multiple datasets to be "joined" by creating a new outer dimension for the aggregated variable
*** Aggregation member datasets can be listed explicitly with explicit coordinates for the new dimension for each member
*** Scan: Aggregations can be specified "automatically" by scanning a directory for files matching certain criteria, such as a suffix or regular expression.
*** Metadata may be added to the new coordinate variable for the new dimension
** Union Aggregation ([[NCML_Module_Aggregation_Union]])
*** Merges all member datasets into one by taking the first named instance of variables and metadata from the members
*** Useful for combining two or more datasets with different variables into a single set

== Installation from Source ==
For information on how to build and install the NcML Data Module, please see the INSTALL file that came with the source distribution.

= Installation Overview =

The NcML Module requires a working Hyrax 1.6 installation. It is a module
that is dynamically loaded into the Hyrax BES (Back End Server) to
allow it to handle NcML files.

Please see the file INSTALL for full build and install instructions as
well as requirements.

'''NOTE:''' After installation, you MUST restart Hyrax by restarting the BES and
OLFS so the NcML Module is loaded!

== Requirement: International Components for Unicode (ICU) Library ==

The most important external requirement is an installation of the International Components for Unicode (ICU) version 3.6 or higher (tested up to 4.2.1). The source distributions (as well as some binaries) may be found at the site: http://site.icu-project.org/download

If you are using Linux RPM's to run Hyrax, you can get an RPM for ICU as well. Search for the RPM named "libicu" using a package manager or yum, e.g. If you are compiling the module from source, you will also need the RPM "libicu-devel" to get the headers installed.

If you install in the default locations, the ncml_module should find the libraries and headers. Otherwise, please consult the INSTALL file for more information about installing ICU to a non-standard location.

== Testing Installation ==

Test data is provided to see if the installation was successful. The file sample_virtual_dataset.ncml is a dataset purely created in NcML and doesn't contain an underlying dataset.
You may also view fnoc1_improved.ncml to test adding attributes to an existing netCDF dataset (fnoc1.nc), but this requires the netCDF data handler to be
installed first! Several other examples installed also use the HDF4 and HDF5 handlers.

= Functionality =

This version of the NcML Module implements a subset of NcML 2.2
functionality. The reader is directed to http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/ for more information on NcML.

Our module can currently:

*Refer only to files being served locally (not remotely)
*Add, modify, and remove attribute metadata to a dataset
*Create a purely virtual dataset using just NcML and no underlying dataset
*Create new scalar variables of any simple NcML type or simple DAP type
*Create new Structure variables (which can contain new child variables)
*Create new N-dimensional arrays of simple types (NcML or DAP)
*Remove existing variables from a wrapped dataset
*Rename existing variables in a wrapped dataset
*Name dimensions as a mnemonic for specifying Array shapes
*Perform union aggregations on multiple datasets, virtual or wrapped or both
*Perform joinNew aggregations to merge a variable across multiple datasets by creating a new outer dimension
*Specify aggregation member datasets by scanning directories for files matching certain criteria

We describe each supported NcML element in detail below.

== <netcdf> Element ==

The <netcdf> element is used to define a dataset, either a wrapped dataset that is to be modified, a pure virtual dataset, or a member dataset of an aggregation. The <netcdf> element is assumed to be the topmost node, or as a child of an aggregation element.

=== Local vs. Remote Datasets ===

We assume that the location attribute (''netcdf@location'') refers to the full path (with respect to the BES data root directory) of a '''''local''''' dataset (served by the same Hyrax server). The current version of the module cannot be used to modify remote datasets.

If ''netcdf@location'' is the empty string (or unspecified, as empty is the default), the dataset is a pure virtual dataset, fully specified within the NcML file itself. Attributes and variables may be fully described and accessed with constraints just as normal datasets in this manner. The installed sample datafile "sample_virtual_dataset.ncml" is an example test case for this functionality.

=== Unsupported Attributes ===

The current version does not support the following attributes of <netcdf>:

*enhance
*addRecords
*fmrcDefinition (will be supported when FMRC aggregation is added)
*ncoords (will be supported when joinExisting is added)

== <readMetadata> Element ==

The <readMetadata/> element is the default, so is effectively not needed.

== <explicit> element ==

The <explicit/> element simply clears all attribute tables in the referred to netcdf@location before applying the rest of the
NcML transformations to the metadata.

== <dimension> Element ==

The <dimension> element has limited functionality in this release since the DAP2 doesn't support dimensions as more than mnemonics at this time. The limitations are:

* We only parse the ''dimension@name'' and ''dimension@length'' attributes.
* Dimensions can only be specified as a direct child of a <netcdf> element prior to any reference to them

For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="samples" length="5"/>

</netcdf>
</pre>

The dimension element sets up a mapping from the ''name'' to the unsigned integer ''length'' and can be used in a ''variable@shape'' to specify a length for an array dimension (see the section on <variable> below). The dimension map is cleared when </netcdf> is encountered (though this doesn't matter currently since we allow only one right now, but it will matter for aggregation, potentially). We also do not support <group>, which is the only other legal place in NcML 2.2 for a dimension element.

'''Parse Errors''':
* If the name and length are not both specified.
* If the dimension name already exists in the current scope
* If the length is not an unsigned integer
* If any of the other attributes specified in NcML 2.2 are used. We do not handle them, so we consider them errors now.

== <variable> Element ==

The <variable> element is used to:

* Provide lexical scope for a contained <attribute> or <variable> element
* Rename existing variables
* Add new scalar variables of simple types
* Add new Structure variables
* Add new N-dimensional Array's of simple types
* Specify the coordinate variable for the new dimension in a joinNew aggregation

We describe each in turn in more detail.

=== Specifying Lexical Scope with <variable type=""> ===

Consider the following example:

<variable name="u">
<attribute name="Metadata" type="string">This is metadata!</attribute>
</variable>

This code assumes that a variable named "u" exists (of any type since we do not specify) and provides the lexical scope for the
attribute "Metadata" which will be added or modified within the attribute table for the variable "u" (it's qualified name would be "u.Metadata").

==== Nested DAP Structure and Grid Scopes ====

Scoping variable elements may be nested if the containing variable is a Structure (this includes the special case of Grid)

<pre>
<variable name="DATA_GRANULE" type="Structure">
<variable name="PlanetaryGrid" type="Structure">
<variable name="percipitate">
<attribute name="units" type="String" value="inches"/>
</variable>
</variable>
</variable>
</pre>

This adds a "unit" attribute to the variable "percipitate" within the nested Structure's
("DATA_GRANULE.PlanetaryGrid.percipitate" as fully qualified name).
Note that we '''must''' refer to the type explicitly as a "Structure" so the parser knows
to traverse the tree.

'''Note''' the variable might be of type Grid, but the type "Structure" must be used in the NcML to traverse it.

==== Adding Multiple Attributes to the Same Variable ====

Once the variable's scope is set by the opening <variable> element, more than one attribute can be specified within it. This will make the NcML more readable and also will make the parsing more efficient since the variable will only need to be looked up once.

For example,

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

should be preferred over:

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
</variable>

<variable name="Foo">
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

although they produce the same result. Any number of attributes can be specified before the variable is closed.

=== Renaming Existing Variables ===

The attribute ''variable@orgName'' is used to rename an existing variable.

For example:

<pre>
<variable name="NewName" orgName="OldName"/>
</pre>

will rename an existing variable at the current scope named "OldName" to "NewName". After this point in the NcML file (such as in constraints specified for the DAP request), the variable is known by "NewName".

Note that the type is not required here --- the variable is assumed to exist and its existing type is used. It is not possible to change the type of an existing variable at this time!

'''Parse Errors''':
* If a variable with ''variable@orgName'' doesn't exist in the current scope
* If the new name ''variable@name'' is already taken in the current scope
* If a new variable is created but does not have exactly one values element

=== Adding a New Scalar Variable ===

The <variable> element can be used to create a new scalar variable of a simple type (i.e. an atomic NcML type such as "int" or "float", or any DAP atomic type, such as "UInt32" or "URL") by specifying an empty ''variable@shape'' (which is the default), a simple type for ''variable@type'', and a contained <values> element with the one value of correct type.

For example:

<pre>
<variable name="TheAnswerToLifeTheUniverseAndEverything" type="double">
<attribute name="SolvedBy" type="String" value="Deep Thought"/>
<values>42.000</values>
</variable>
</pre>

will create a new variable named "TheAnswerToLifeTheUniverseAndEverything" at the current scope. It has no shape so will be a scalar of type "double" and will have the value 42.0.

'''Parse Errors''':
* It is a parse error to not specify a <values> element with exactly one proper value of the variable type.
* It is a parse error to specify a malformed or out of bounds value for the data type

=== Adding a New Structure Variable ===

A new Structure variable can be specified at the global scope or within another Structure. It is illegal for an array to have type structure, so the shape must be empty.

For example:

<pre>
<variable name="MyNewStructure" type="Structure">
<attribute name="MetaData" type="String" value="This is metadata!"/>
<variable name="ContainedScalar1" type="String"><values>I live in a new structure!</values></variable>
<variable name="ContainedInt1" type="int"><values>42</values></variable>
</variable>
</pre>

specifies a new structure called "MyNewStructure" which contains two scalar variable fields "ContainedScalar1" and "ContainedInt1".

Nested structures are allowed as well.

'''Parse Error''':
* If another variable or attribute exists at the current scope with the new name.
* If a <values> element is specified as a direct child of a new Structure --- structures cannot contain values, only attributes and other variables.

=== Adding a New N-dimensional Array ===

An N-dimensional array of a simple type may be created virtually as well by specifying a non-empty ''variable@shape''. The shape contains the array dimensions in left-to-right order of slowest varying dimension first. For example:

<pre>
<variable name="FloatArray" type="float" shape="2 5">

<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will specify a 2x5 dimension array of float values called "FloatArray". The <values> element must contain 2x5=10 values in row major order (slowest varying dimension first). Since whitespace is the default separator, we use a newline to show the dimension boundary for the values, which is easy to see for a 2D matrix such as this.

A dimension name may also be used to refer mnemonically to a length. The DAP response will use this mnemonic in its output, but it is not currently used for shared dimensions, only as a mnemonic. See the section on the <dimension> element for more information. For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="sample" length="5"/>
<variable name="FloatArray" type="float" shape="station sample">
<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will produce the same 2x5 array, but will incorporate the dimension mnemonics into the response. For example, here's the DDS response:

<pre>
Dataset {
Float32 FloatArray[station = 2][samples = 5];
} sample_virtual_dataset.ncml;
</pre>

Note that the <values> element respects the ''values@separator'' attribute if whitespace isn't correct. This is very useful for arrays of strings with whitespace, for example.

<pre>
<variable name="StringArray" type="string" shape="3">
<values separator="*">String 1*String 2*String 3</values>
</variable>
</pre>

creates a length 3 array of string StringArray = {"String 1", "String 2", "String 3"}.

'''Parse Errors''':
* It is an error to specify the incorrect number of values
* It is an error if any value is malformed or out of range for the data type.
* It is an error to specify a named dimension which does not exist in the current <netcdf> scope.
* It is an error to specify an Array whose flattened size (product of dimensions) is > 2^31-1.

=== Specifying the new coordinate variable for a joinNew aggregation ===

In the special case of a joinNew aggregation, the new coordinate variable may be specified with the <variable> element. The new coordinate variable is ''defined'' to have the same name as the new dimension. This allows for several things:

*Explicit specification of the variable type and coordinates for the new dimension
*Specification of the metadata for the new coordinate variable

In the first case, the author can specify explicitly the type of the new coordinate variable and the actual values for each dataset. In this case, the variable ''must'' be specified ''after'' the aggregation element in the file so the new dimension's size (number of member datasets) may be known and error checking performed. Metadata can also be added to the variable here.

In the second case, the author may just specify the variable name, which allows one to specify the metadata for a coordinate variable that is automatically generated by the aggregation itself. This is the only allowable case for a variable element to ''not'' contain a values element! Coordinate variables are generated automatically in two cases:

*The author has specified an explicit list of member datasets, with or without explicit coordVal attributes.
*The author has used a <scan> element to specify the member datasets via a directory scan

In this case, the <variable> element may come before or after the <aggregation>.

'''Parse Errors:'''

*If an explicit variable is declared for the new coordinate variable:
** And it contains explicit values, the number of values must be equal to the number of member datasets in the aggregation.
** It must be specifed ''after'' the <aggregation> element

* If a numeric coordVal is used to specify the first member dataset's coordinate, then ''all'' datasets must contain a numerical coordinate.

* An error is thrown if the specified aggregation variable (variableAgg) is not found in ''all'' member datasets.

* An error is thrown if the specified aggregation variable is not of the same type in ''all'' member datasets. Coercion is ''not'' performed!

* An error is thrown if the specified aggregation variables in all member datasets do not have the same shape

* An error is thrown if an explicit coordinate variable is specified with a shape that is ''not'' the same as the new dimension name (and the variable name itself).

== <values> Element ==

The <values> element can only be used in the context of a '''new''' variable of scalar or array type. We cannot change the values for existing variables in this version of the handler.
The characters content of a <values> element is considered to be a separated list of value tokens valid for the type of the variable of the parent element. The number of specified tokens in the content ''must'' equal the product of the dimensions of the enclosing ''variable@shape'', or be one value for a scalar. It is an error to ''not'' specify a <values> element for a declared new variable as well.

=== Changing the Separator Tokens ===

The author may specify values@separator to change the default value token separator from the default whitespace. This is very useful for specifying arrays of strings with whitespace in them, or if data in CSV form is being pasted in.

=== Autogeneration of Uniform Arrays ===

We also can parse ''values@start'' and ''values@increment'' INSTEAD OF tokens in the content. This will "autogenerate" a uniform array of values of the given product of dimensions length for the containing variable. For example:

<pre>
<variable name="Evens" type="int" shape="100">
<values start="0" increment="2"/>
</variable>
</pre>

will specify an array of the first 100 even numbers (including 0).

'''Parse Errors''':
* If the incorrect number of tokens are specified for the containing variable's shape
* If any value token cannot be parsed as a valid value for the containing variable's type
* If content is specified in addition to start and increment
* If only one of start or increment is specified
* If the values element is placed anywhere except within a NEW variable.

== <attribute> Element ==

As an overview, whenever the parser encounters an <attribute> with a non-existing name (at the current scope),
it creates a new one, whether a container or atomic attribute (see below). If the attribute exists,
its value and/or type is modified to those specified in the <attribute> element. If an attribute
structure (container) exists, it is used to define a nested lexical scope for child attributes.

Attributes may be scalar (one value) or one dimensional arrays. Arrays are specified by using whitespace (default) to separate the different values. The attribute@separator may also be set in order to specify a different separator, such as CSV format or to specify a non-whitespace separator so strings with whitespace are not tokenized. We will give examples of creating array attributes below.

=== Adding New Attributes or Modifying an Existing Attribute ===

If a specified attribute with the attribute@name does not exist at the current lexical scope,
a new one is created with the given type and value. For example, assume "new_metadata" doesn't
exist at the current parse scope. Then:

<attribute name="new_metadata" type="string" value="This is a new entry!"/>

will create the attribute at that scope. Note that value can be specified in the content of the
element as well. This is identical to the above:

<attribute name="new_metadata" type="string">This is a new entry!</attribute>

If the attribute@name already exists at the scope, it is modified to contain the specified type and value.

==== Arrays ====

As in NcML, for numerical types an array can be specified by separating the tokens by whitespace (default) or be
specifying the token separator with attribute@separator. For example,

<attribute name="myArray" type="int">1 2 3</attribute>

and

<attribute name="myArray" type="int" separator=",">1,2,3</attribute>

both specify the same array of three integers named "myArray".

TODO Add more information on splitting with a separator!

==== Structures (Containers) ====

We use attribute@type="Structure" to define a new (or existing) attribute container. So if we
wanted to add a new attribute structure, we'd use something like this:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

Assuming "MySamples" doesn't already exist, an attribute container will be created at the current scope
and the "Location" and "Samples" attributes will be added to it.

Note that we can create nested attribute structures to arbitrary depth this way as well.

If the attribute container with the given name already exists at the current scope,
then the attribute@type="Structure" form is used to define
the lexical scope for the container. In other words, child <attribute> elements will
be processed within the scope of the container. For example, in the above example, if
"MySamples" already exists, then the "Location" and "Samples" will be processed within the
existing container (they may or may not already exist as well).

==== Renaming an Existing Attribute or Attribute Container ====

We also support the attribute@orgName attribute for renaming attributes.

For example,

<attribute name="NewName" orgName="OldName" type="string"/>

will rename an existing attribute "OldName" to "NewName" while leaving its value alone.
If attribute@value is also specified, then the attribute is renamed ''and'' has its value modified.

This works for renaming attribute containers as well:

<attribute name="MyNewContainer" orgName="MyOldContainer" type="Structure"/>

will rename an existing "MyOldContainer" to "MyNewContainer". Note that any children
of this container will remain in it.

=== DAP ''OtherXML'' Extension ===

The module now allows specification of attributes of the new DAP type "OtherXML". This allows the NCML file author to inject arbitrary well-formed XML into an attribute for clients that want XML metadata rather than just string or url. Internally, the attribute is still a string (and in a DAP DAS response will be quoted inside one string). However, since it is XML, the NCMLParser still parses it and checks it for well-formedness (but NOT against schemas). This extension allows the NCMLParser to parse the arbitrary XML within the given attribute without causing errors, since it can be any XML.

The injected XML is most useful in the DDX response, where it shows up directly in the response as XML. XSLT and other clients can then parse it.

==== Errors ====

*The XML '''must''' be in the content of the <attribute type="OtherXML"> element. It is a parser error for ''attribute@value'' to be set if ''attribute@type'' is "OtherXML".
*The XML must also be well-formed since it is parsed. A parse error will be thrown if the OtherXML is malformed.

==== Example ====

Here's an example of the use of this special case.

<pre>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/coverage/200803061600_HFRadar_USEGC_6km_rtv_SIO.nc">

<attribute name="someName" type="OtherXML">
<Domain xmlns="http://www.opengis.net/wcs/1.1"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
>
<SpatialDomain>
<ows:BoundingBox crs="urn:ogc:def:crs:EPSG::4326">
<ows:LowerCorner>-97.8839 21.736</ows:LowerCorner>
<ows:UpperCorner>-57.2312 46.4944</ows:UpperCorner>
</ows:BoundingBox>
</SpatialDomain>
<TemporalDomain>
<gml:timePosition>2008-03-27T16:00:00.000Z</gml:timePosition>
</TemporalDomain>
</Domain>
<SupportedCRS xmlns="http://www.opengis.net/wcs/1.1">urn:ogc:def:crs:EPSG::4326</SupportedCRS>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">netcdf-cf1.0</SupportedFormat>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">dap2.0</SupportedFormat>
</attribute>

</netcdf>
</pre>

'''TODO'' Put the DDX response for the above in here!

====Namespace Closure====

Furthermore, the parser will make the chunk of OtherXML "namespace closed". This means any namespaces specified in parent NCML elements of the OtherXML tree will be "brought down" and added to the ''root'' OtherXML elements so that the subtree may be pulled out and added to the DDX and still have its namespaces. The algorithm doesn't just bring used prefixes, but brings ''all'' of the lexically scoped closest namespaces in all ancestors. In other words, it adds unique namespaces (as determined by prefix) in order from the root of the OtherXML tree as it traverses to the root of the NCML document.

Namespace closure is a syntactic sugar that simplifies the author's task since they can specify the namespaces just once at the top of the NCML file and expect that when the subtree of XML is added to the DDX that these namespaces will come along with that subtree of XML. Otherwise they have to explicitly add the namespaces to each attributes.

'''TODO''' Add an example!

== <remove> Element ==

The <remove> element can remove attributes and variables. For example:

<pre>
<attribute name="NC_GLOBAL" type="Structure">
<remove name="base_time" type="attribute"/>
</attribute>
</pre>

will remove the attribute named "base_time" in the attribute structure named "NC_GLOBAL".

Note that this works for attribute containers as well! We could recursively remove the ''entire''
attribute container (i.e. it and all its children) with:

<pre>
<remove name="NC_GLOBAL" type="attribute"/>
</pre>

It also can be used to remove variables from existing datasets:

</pre>
<remove name="SomeExistingVariable" type="variable"/>
</pre>

This also recurses on variables of type Structure --- the entire structure including all of its children are removed from the dataset's response.

'''Parse Errors''':
* It is a parse error if the given attribute or variable doesn't exist in the current scope

== <aggregation> Element ==

Aggregation involves combining multiple datasets (<netcdf>) into a virtual "single" dataset in various ways. For a tutorial on aggregation in NcML 2.2, the reader is referred to the Unidata page: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

NcML 2.2 supports multiple types of aggregation: union, joinNew, joinExisting, and fmrc (forecast model run collection).

The current version of the NcML module supports two of these aggregations:

*Union [[NCML_Module_Aggregation_Union]]
*JoinNew [[NCML_Module_Aggregation_JoinNew]]

A ''union'' aggregation specifies that the first instance of a variable or attribute (by name) that is found in the ordered list of datasets will be the one in the output aggregation. This is useful for combining two dataset files, each which may contain a single variable, into a composite dataset with both variables.

A JoinNew aggregation joins a variable which exists in multiple datasets (usually samples of a datum over time) into a new variable containing the data from ''all'' member datasets by creating a new outer dimension. The ''i''th component in the new outer dimension is the variable's data from the ''i''th member dataset. It also adds a new coordinate variable of whose name is the new dimension's name and whose shape (length) is the new dimension as well. This new coordinate variable may be explicitly given by the author or may be autogenerated in one of several ways.

== <scan> Element ==

The scan element can be used within an aggregation context to allow a directory to be searched in various ways in order to specify the members of an aggregation. This allows a static NcML file to refer to an aggregation which may change over time, such as where a new data file is generated each day.

We describe usage of the <scan> element in detail in the joinNew aggregation tutorial at [[NCML_Module_Aggregation_JoinNew]].

= Errors =

There are three types of error messages that may be returned:

* Internal Error
* Resource Not Found Error
* Parse Error

== Internal Errors ==
'''Internal errors''' should be reported to support@opendap.org as they are likely bugs.

== Resource Not Found Errors ==

If the netcdf@location specifies a non-existent local dataset (one that is not being served by
the same Hyrax server), it will specify the resource was not found. This may also be returned
if a handler for the specified dataset is not currently loaded in the BES. Users should test that the
dataset to be wrapped already exists and can be viewed on the running server before writing NcML
to add metadata. It's also an error to refer to remote datasets (at this time).

== Parse Errors ==

'''Parse errors''' are user errors in the NcML file. These could be malformed XML, malformed NcML,
unimplemented features of NcML, or could be errors in referring to the wrapped dataset.

The error message should specify
the error condition as well as the "current scope" as a fully qualified DAP name within the
loaded dataset. This should be enough information to correct the parse error as new NcML
files are created.

The parser will generate parse errors in various
situations where it expects to find certain structure in the underlying dataset. Some examples:

* A variable of the given name was not found at the current scope
* attribute@orgName was specified, but the attribute cannot be found at current scope.
* attribute@orgName was specified, but the new name is already used at current scope.
* remove specified a non-existing attribute name

= Grid Metadata Tutorial =

Please see the page [[Grid_Metadata_Tutorial]] for an example of adding metadata to the various parts of a DAP Grid variable.

= Aggregation Tutorials =

The NcML module may also be used to aggregate multiple datasets into one virtual dataset. We currently support two of the NcML aggregtions:

*Union [[NCML_Module_Aggregation_Union]]: combine multiple datasets into one by merging variables together, selecting the first of each unique name.
*JoinNew [[NCML_Module_Aggregation_JoinNew]]: combine variables across multiple datasets by creating a new outer dimension and coordinate variable for each of the sample datasets.

Please see the sections for a tutorial on the various uses of these aggregations.

= Additions/Changes to NcML 2.2 =

This section will keep track of changes to the NcML 2.2 schema. Eventually these
will be rolled into a new schema.

== Attribute Structures (Containers) ==
This module also adds functionality beyond the current NcML 2.2 schema
--- it can handle nested <attribute> elements in order to make
attribute structures. This is done by using the <attribute
type="Structure"> form, for example:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

"MyContainer" describes an attribute structure with two attribute fields,
a string "Location" and an array of int's called "Samples".
Note that an attribute structure of this form can
only contain other <attribute> elements and NOT a value.

If the container does not already exist, it will be created at the scope it is declared, which could
be:

* Global (top of dataset)
* Within a variable's attribute table
* Within another attribute container

If an attribute container of the given name already exists at the lexical scope, it is traversed
in order to define the scope for the nested (children) attributes it contains.

== Unspecified Variable Type Matching for Lexical Scope ==

We also allow the type attribute of a variable element (variable@type) to be the empty string
(or unspecified) when using existing variables to define the lexical scope of an <attribute>
transformation. In the schema, variable@type is (normally) required.

== DAP 2 Types ==

Additionally, we allow DAP2 atomic types (such as UInt32, URL) in addition to the NcML types.
The NcML types are mapped onto the closest DAP2 type internally.

== DAP OtherXML Attribute Type ==

We also allow attributes to be of the new DAP type "OtherXML" for injecting arbitrary XML into an attribute as content rather than trying to form a string. This allows the parser to check well-formedness.

== Forward Declaration of Dimensions ==

Since we use a SAX parser for efficiency, we require the <dimension> elements to come ''before'' their use in a ''variable@shape''. One way to change the schema to allow this is to force the dimension elements to be specified in a sequence after explicit and metadata choice and before all other elements.

== Aggregation Element Location and Processing Order Differences ==

NcML specifies that if a dataset (<netcdf> element) specifies an aggregation element, the aggregation element is always processed first, regardless of its ordering within the <netcdf> element. Our parser, since it is SAX and not DOM, modifies this behavior in that order matters in some cases:

* Metadata (<attribute>) elements specified ''prior'' to an aggregation "shadow" the aggregation versions. This is be useful for "overriding" an attribute or variable in a union aggregation, where the first found will take precedence.
* JoinNew: If the new coordinate variable's data is to be set explicitly by specifying the new dimension's shape (either with explicit data or the autogenerated data using values@start and values@increment attributes), the <variable> ''must'' come after the aggregation since the size of the dimension is unknown until the aggregation element is processed.

= Backward Compatibility Issues =

Due to the way shared dimensions were implemented in the NetCDF, HDF4, and HDF5 handlers, the DAS responses did not follow the DAP2 specification. The NcML module, on the other hand, generates DAP2 compliant DAS for these datasets, which means that wrapping some datasets in NcML will generate a DAS with a different structure. This is important for the NcML author since it changes the names of attributes and variables. In order for the module to find the correct scope for adding metadata, for example, the DAP2 DAS must be used.

In general, what this means is that an empty "passthrough" NcML file should be the starting point for authoring an NcML file. This file would just specify a dataset and nothing else:

<pre>
<netcdf location="/data/ncml/myNetcdf.nc"/>
</pre>

The author would then request the DAS response for the NCML file and use that as the starting point for modifications to the original dataset.

More explicit examples are given below.

== NetCDF ==

The NetCDF handler represents some NC datasets as a DAP 2 Grid, but the returned DAS is not consistent with the DAP 2 spec for the attribute hierarchy for such a Grid. The map vector attributes are placed as siblings of the grid attributes rather than within the grid lexical scope. For example, here's the NetCDF Handler DDS for a given file:

<pre>
Dataset {
Grid {
Array:
Int16 cldc[time = 456][lat = 21][lon = 360];
Maps:
Float64 time[time = 456];
Float32 lat[lat = 21];
Float32 lon[lon = 360];
} cldc;
} cldc.mean.nc;
</pre>

showing the Grid. Here's the DAS the NetCDF handler generates:

<pre>
Attributes {
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
}
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
}
</pre>

Note the map vector attributes are in the "dataset" scope.

Here's the DAS that the NcML Module produces from the correctly formed DDX:

<pre>
Attributes {
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
cldc {
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
}
}
</pre>

Here the Grid Structure "cldc" and its contained data array (of the same name "cldc") and map vectors have their own attribute containers as DAP 2 specifies.

What this means for the author of an NcML file adding metadata to a NetCDF dataset that returns a Grid is that they should generate a "passthrough" file and get the DAS and then specify modifications based on that structure.

Here's an example passthrough:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">
</netcdf>
</pre>

For example, to add an attribute to the map vector "lat" in the above, we'd need the following NcML:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">

<variable name="cldc" type="Structure">

<variable name="lat">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lat!</attribute>
</variable>
<variable name="lon">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lon!</attribute>
</variable>
</variable>
</netcdf>
</pre>

This clearly shows that the structure of the Grid must be used in the NcML: the attribute being added is technically "cldc.lat.Description" in a fully qualified name. The parser would return an error if it was attempted as "lat.Description" as the NetCDF DAS for the original file would have led one to believe.

== HDF4/HDF5 ==

Similarly to the NetCDF case, the Hyrax HDF4 Module produces DAS responses that do not respect the DAP2 specification. If an NcML file is used to "wrap" an HDF4 dataset, the
correct DAP2 DAS response will be generated, however.

This is important for those writing NcML for HDF4 data since the
lexical scope for attributes relies on the correct DAS form --- to
handle this, the user should start with a "passthrough" NcML file (see the above NetCDF example) and
use the DAS from that as the starting point for knowing the structure
the NcML handler expects to see in the NcML file. Alternatively, the DDX has the
proper attribute structure as well (the DAS is generated from it).

= Known Bugs =

There are no known bugs currently.

= Planned Future Enhancements =

Planned enhancements for future versions of the module include:

* New NcML Aggregations
** JoinExisting
*** Joins a variable across multiple datasets by appending the data for a given dimension from each dataset
*** Will also allow directory scans for specifying the aggregation
** Forecast Model Run Collection (FMRC)
*** Special case of JoinNew for forecast data with two time variables
*** See: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html

= Copyright =

This software is copyrighted under the GNU Lesser GPL. Please see the
files COPYING and COPYRIGHT that came with this distribution.

BES - Modules - NcML Module

2010-06-29T18:44:21Z

Mjohnson: /* Introduction */

= Introduction =
[[Category:NCML]]
[[Category:BES Modules|NCML Module]]

'''Welcome to the OPeNDAP NcML Data Handler Module v1.0.2 for Hyrax 1.6!'''

[Updated for NcML Module v1.0.2 for Hyrax 1.6.1 Release -- 29 June 2010]

This module may be added to a Hyrax 1.6 server to extend its data
serving capability to NcML 2.2 files (see
http://www.unidata.ucar.edu/software/netcdf/ncml/). NcML provides support for modifying other datasets in various ways, such as adding metadata and data and aggregating multiple datasets in several ways.

We refer the reader to the Unidata NcML tutorial: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html which will give the basics of using NcML. We then give a reference manual for the various elements and their attributes. Additionally we have provided a more extensive tutorial on NcML aggregation. Please see [[#Aggregation Tutorials]].

Since the DAP Grid is a common case, we will also give a simple example for adding metadata to the various parts of a Grid dataset using NcML. Please see [[Grid_Metadata_Tutorial]].

==Features==

This current version (v1.0.2) currently implements a subset of NcML 2.2
functionality, along with some OPeNDAP extensions:

* Metadata Manipulation
** Addition, Removal, and Modification of attributes to other datasets (NetCDF, HDF4, HD5, etc.) served by the same Hyrax 1.6 server
** Extends NcML 2.2 to allow for common nested "attribute containers"
** Attributes can be DAP2 types as well as the NcML types
** Attributes can be of the special "OtherXML" type for injecting arbitrary XML into a DDX response
* Data Manipulation
** Addition of new data variables (scalars or arrays of basic types as well as structures)
** Variables may be removed from the wrapped dataset
** Allows the creation of "pure virtual" datasets which do not wrap another dataset
* Aggregations: JoinNew and Union [[#Aggregation Tutorials]]
** JoinNew Aggregation ([[NCML_Module_Aggregation_JoinNew]])
*** Allows multiple datasets to be "joined" by creating a new outer dimension for the aggregated variable
*** Aggregation member datasets can be listed explicitly with explicit coordinates for the new dimension for each member
*** Scan: Aggregations can be specified "automatically" by scanning a directory for files matching certain criteria, such as a suffix or regular expression.
*** Metadata may be added to the new coordinate variable for the new dimension
** Union Aggregation ([[NCML_Module_Aggregation_Union]])
*** Merges all member datasets into one by taking the first named instance of variables and metadata from the members
*** Useful for combining two or more datasets with different variables into a single set

== Installation from Source ==
For information on how to build and install the NcML Data Module, please see the INSTALL file that came with the source distribution.

= Installation Overview =

The NcML Module requires a working Hyrax 1.6 installation. It is a module
that is dynamically loaded into the Hyrax BES (Back End Server) to
allow it to handle NcML files.

Please see the file INSTALL for full build and install instructions as
well as requirements.

'''NOTE:''' After installation, you MUST restart Hyrax by restarting the BES and
OLFS so the NcML Module is loaded!

== Requirement: International Components for Unicode (ICU) Library ==

The most important external requirement is an installation of the International Components for Unicode (ICU) version 3.6 or higher (tested up to 4.2.1). The source distributions (as well as some binaries) may be found at the site: http://site.icu-project.org/download

If you are using Linux RPM's to run Hyrax, you can get an RPM for ICU as well. Search for the RPM named "libicu" using a package manager or yum, e.g. If you are compiling the module from source, you will also need the RPM "libicu-devel" to get the headers installed.

If you install in the default locations, the ncml_module should find the libraries and headers. Otherwise, please consult the INSTALL file for more information about installing ICU to a non-standard location.

== Testing Installation ==

Test data is provided to see if the installation was successful. The file sample_virtual_dataset.ncml is a dataset purely created in NcML and doesn't contain an underlying dataset.
You may also view fnoc1_improved.ncml to test adding attributes to an existing netCDF dataset (fnoc1.nc), but this requires the netCDF data handler to be
installed first! Several other examples installed also use the HDF4 and HDF5 handlers.

= Functionality =

This version of the NcML Module implements a subset of NcML 2.2
functionality. The reader is directed to http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/ for more information on NcML.

Our module can currently:

*Refer only to files being served locally (not remotely)
*Add, modify, and remove attribute metadata to a dataset
*Create a purely virtual dataset using just NcML and no underlying dataset
*Create new scalar variables of any simple NcML type or simple DAP type
*Create new Structure variables (which can contain new child variables)
*Create new N-dimensional arrays of simple types (NcML or DAP)
*Remove existing variables from a wrapped dataset
*Rename existing variables in a wrapped dataset
*Name dimensions as a mnemonic for specifying Array shapes
*Perform union aggregations on multiple datasets, virtual or wrapped or both
*Perform joinNew aggregations to merge a variable across multiple datasets by creating a new outer dimension
*Specify aggregation member datasets by scanning directories for files matching certain criteria

We describe each supported NcML element in detail below.

== <netcdf> Element ==

The <netcdf> element is used to define a dataset, either a wrapped dataset that is to be modified, a pure virtual dataset, or a member dataset of an aggregation. The <netcdf> element is assumed to be the topmost node, or as a child of an aggregation element.

=== Local vs. Remote Datasets ===

We assume that the location attribute (''netcdf@location'') refers to the full path (with respect to the BES data root directory) of a '''''local''''' dataset (served by the same Hyrax server). The current version of the module cannot be used to modify remote datasets.

If ''netcdf@location'' is the empty string (or unspecified, as empty is the default), the dataset is a pure virtual dataset, fully specified within the NcML file itself. Attributes and variables may be fully described and accessed with constraints just as normal datasets in this manner. The installed sample datafile "sample_virtual_dataset.ncml" is an example test case for this functionality.

=== Unsupported Attributes ===

The current version does not support the following attributes of <netcdf>:

*enhance
*addRecords
*fmrcDefinition (will be supported when FMRC aggregation is added)
*ncoords (will be supported when joinExisting is added)

== <readMetadata> Element ==

The <readMetadata/> element is the default, so is effectively not needed.

== <explicit> element ==

The <explicit/> element simply clears all attribute tables in the referred to netcdf@location before applying the rest of the
NcML transformations to the metadata.

== <dimension> Element ==

The <dimension> element has limited functionality in this release since the DAP2 doesn't support dimensions as more than mnemonics at this time. The limitations are:

* We only parse the ''dimension@name'' and ''dimension@length'' attributes.
* Dimensions can only be specified as a direct child of a <netcdf> element prior to any reference to them

For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="samples" length="5"/>

</netcdf>
</pre>

The dimension element sets up a mapping from the ''name'' to the unsigned integer ''length'' and can be used in a ''variable@shape'' to specify a length for an array dimension (see the section on <variable> below). The dimension map is cleared when </netcdf> is encountered (though this doesn't matter currently since we allow only one right now, but it will matter for aggregation, potentially). We also do not support <group>, which is the only other legal place in NcML 2.2 for a dimension element.

'''Parse Errors''':
* If the name and length are not both specified.
* If the dimension name already exists in the current scope
* If the length is not an unsigned integer
* If any of the other attributes specified in NcML 2.2 are used. We do not handle them, so we consider them errors now.

== <variable> Element ==

The <variable> element is used to:

* Provide lexical scope for a contained <attribute> or <variable> element
* Rename existing variables
* Add new scalar variables of simple types
* Add new Structure variables
* Add new N-dimensional Array's of simple types
* Specify the coordinate variable for the new dimension in a joinNew aggregation

We describe each in turn in more detail.

=== Specifying Lexical Scope with <variable type=""> ===

Consider the following example:

<variable name="u">
<attribute name="Metadata" type="string">This is metadata!</attribute>
</variable>

This code assumes that a variable named "u" exists (of any type since we do not specify) and provides the lexical scope for the
attribute "Metadata" which will be added or modified within the attribute table for the variable "u" (it's qualified name would be "u.Metadata").

==== Nested DAP Structure and Grid Scopes ====

Scoping variable elements may be nested if the containing variable is a Structure (this includes the special case of Grid)

<pre>
<variable name="DATA_GRANULE" type="Structure">
<variable name="PlanetaryGrid" type="Structure">
<variable name="percipitate">
<attribute name="units" type="String" value="inches"/>
</variable>
</variable>
</variable>
</pre>

This adds a "unit" attribute to the variable "percipitate" within the nested Structure's
("DATA_GRANULE.PlanetaryGrid.percipitate" as fully qualified name).
Note that we '''must''' refer to the type explicitly as a "Structure" so the parser knows
to traverse the tree.

'''Note''' the variable might be of type Grid, but the type "Structure" must be used in the NcML to traverse it.

==== Adding Multiple Attributes to the Same Variable ====

Once the variable's scope is set by the opening <variable> element, more than one attribute can be specified within it. This will make the NcML more readable and also will make the parsing more efficient since the variable will only need to be looked up once.

For example,

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

should be preferred over:

<pre>
<variable name="Foo">
<attribute name="Attr_1" type="string" value="Hello"/>
</variable>

<variable name="Foo">
<attribute name="Attr_2" type="string" value="World!"/>
</variable>
</pre>

although they produce the same result. Any number of attributes can be specified before the variable is closed.

=== Renaming Existing Variables ===

The attribute ''variable@orgName'' is used to rename an existing variable.

For example:

<pre>
<variable name="NewName" orgName="OldName"/>
</pre>

will rename an existing variable at the current scope named "OldName" to "NewName". After this point in the NcML file (such as in constraints specified for the DAP request), the variable is known by "NewName".

Note that the type is not required here --- the variable is assumed to exist and its existing type is used. It is not possible to change the type of an existing variable at this time!

'''Parse Errors''':
* If a variable with ''variable@orgName'' doesn't exist in the current scope
* If the new name ''variable@name'' is already taken in the current scope
* If a new variable is created but does not have exactly one values element

=== Adding a New Scalar Variable ===

The <variable> element can be used to create a new scalar variable of a simple type (i.e. an atomic NcML type such as "int" or "float", or any DAP atomic type, such as "UInt32" or "URL") by specifying an empty ''variable@shape'' (which is the default), a simple type for ''variable@type'', and a contained <values> element with the one value of correct type.

For example:

<pre>
<variable name="TheAnswerToLifeTheUniverseAndEverything" type="double">
<attribute name="SolvedBy" type="String" value="Deep Thought"/>
<values>42.000</values>
</variable>
</pre>

will create a new variable named "TheAnswerToLifeTheUniverseAndEverything" at the current scope. It has no shape so will be a scalar of type "double" and will have the value 42.0.

'''Parse Errors''':
* It is a parse error to not specify a <values> element with exactly one proper value of the variable type.
* It is a parse error to specify a malformed or out of bounds value for the data type

=== Adding a New Structure Variable ===

A new Structure variable can be specified at the global scope or within another Structure. It is illegal for an array to have type structure, so the shape must be empty.

For example:

<pre>
<variable name="MyNewStructure" type="Structure">
<attribute name="MetaData" type="String" value="This is metadata!"/>
<variable name="ContainedScalar1" type="String"><values>I live in a new structure!</values></variable>
<variable name="ContainedInt1" type="int"><values>42</values></variable>
</variable>
</pre>

specifies a new structure called "MyNewStructure" which contains two scalar variable fields "ContainedScalar1" and "ContainedInt1".

Nested structures are allowed as well.

'''Parse Error''':
* If another variable or attribute exists at the current scope with the new name.
* If a <values> element is specified as a direct child of a new Structure --- structures cannot contain values, only attributes and other variables.

=== Adding a New N-dimensional Array ===

An N-dimensional array of a simple type may be created virtually as well by specifying a non-empty ''variable@shape''. The shape contains the array dimensions in left-to-right order of slowest varying dimension first. For example:

<pre>
<variable name="FloatArray" type="float" shape="2 5">

<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will specify a 2x5 dimension array of float values called "FloatArray". The <values> element must contain 2x5=10 values in row major order (slowest varying dimension first). Since whitespace is the default separator, we use a newline to show the dimension boundary for the values, which is easy to see for a 2D matrix such as this.

A dimension name may also be used to refer mnemonically to a length. The DAP response will use this mnemonic in its output, but it is not currently used for shared dimensions, only as a mnemonic. See the section on the <dimension> element for more information. For example:

<pre>
<netcdf>
<dimension name="station" length="2"/>
<dimension name="sample" length="5"/>
<variable name="FloatArray" type="float" shape="station sample">
<values>
0.1 0.2 0.3 0.4 0.5
1.1 1.1 1.3 1.4 1.5
</values>
</variable>
</pre>

will produce the same 2x5 array, but will incorporate the dimension mnemonics into the response. For example, here's the DDS response:

<pre>
Dataset {
Float32 FloatArray[station = 2][samples = 5];
} sample_virtual_dataset.ncml;
</pre>

Note that the <values> element respects the ''values@separator'' attribute if whitespace isn't correct. This is very useful for arrays of strings with whitespace, for example.

<pre>
<variable name="StringArray" type="string" shape="3">
<values separator="*">String 1*String 2*String 3</values>
</variable>
</pre>

creates a length 3 array of string StringArray = {"String 1", "String 2", "String 3"}.

'''Parse Errors''':
* It is an error to specify the incorrect number of values
* It is an error if any value is malformed or out of range for the data type.
* It is an error to specify a named dimension which does not exist in the current <netcdf> scope.
* It is an error to specify an Array whose flattened size (product of dimensions) is > 2^31-1.

=== Specifying the new coordinate variable for a joinNew aggregation ===

In the special case of a joinNew aggregation, the new coordinate variable may be specified with the <variable> element. The new coordinate variable is ''defined'' to have the same name as the new dimension. This allows for several things:

*Explicit specification of the variable type and coordinates for the new dimension
*Specification of the metadata for the new coordinate variable

In the first case, the author can specify explicitly the type of the new coordinate variable and the actual values for each dataset. In this case, the variable ''must'' be specified ''after'' the aggregation element in the file so the new dimension's size (number of member datasets) may be known and error checking performed. Metadata can also be added to the variable here.

In the second case, the author may just specify the variable name, which allows one to specify the metadata for a coordinate variable that is automatically generated by the aggregation itself. This is the only allowable case for a variable element to ''not'' contain a values element! Coordinate variables are generated automatically in two cases:

*The author has specified an explicit list of member datasets, with or without explicit coordVal attributes.
*The author has used a <scan> element to specify the member datasets via a directory scan

In this case, the <variable> element may come before or after the <aggregation>.

'''Parse Errors:'''

*If an explicit variable is declared for the new coordinate variable:
** And it contains explicit values, the number of values must be equal to the number of member datasets in the aggregation.
** It must be specifed ''after'' the <aggregation> element

* If a numeric coordVal is used to specify the first member dataset's coordinate, then ''all'' datasets must contain a numerical coordinate.

* An error is thrown if the specified aggregation variable (variableAgg) is not found in ''all'' member datasets.

* An error is thrown if the specified aggregation variable is not of the same type in ''all'' member datasets. Coercion is ''not'' performed!

* An error is thrown if the specified aggregation variables in all member datasets do not have the same shape

* An error is thrown if an explicit coordinate variable is specified with a shape that is ''not'' the same as the new dimension name (and the variable name itself).

== <values> Element ==

The <values> element can only be used in the context of a '''new''' variable of scalar or array type. We cannot change the values for existing variables in this version of the handler.
The characters content of a <values> element is considered to be a separated list of value tokens valid for the type of the variable of the parent element. The number of specified tokens in the content ''must'' equal the product of the dimensions of the enclosing ''variable@shape'', or be one value for a scalar. It is an error to ''not'' specify a <values> element for a declared new variable as well.

=== Changing the Separator Tokens ===

The author may specify values@separator to change the default value token separator from the default whitespace. This is very useful for specifying arrays of strings with whitespace in them, or if data in CSV form is being pasted in.

=== Autogeneration of Uniform Arrays ===

We also can parse ''values@start'' and ''values@increment'' INSTEAD OF tokens in the content. This will "autogenerate" a uniform array of values of the given product of dimensions length for the containing variable. For example:

<pre>
<variable name="Evens" type="int" shape="100">
<values start="0" increment="2"/>
</variable>
</pre>

will specify an array of the first 100 even numbers (including 0).

'''Parse Errors''':
* If the incorrect number of tokens are specified for the containing variable's shape
* If any value token cannot be parsed as a valid value for the containing variable's type
* If content is specified in addition to start and increment
* If only one of start or increment is specified
* If the values element is placed anywhere except within a NEW variable.

== <attribute> Element ==

As an overview, whenever the parser encounters an <attribute> with a non-existing name (at the current scope),
it creates a new one, whether a container or atomic attribute (see below). If the attribute exists,
its value and/or type is modified to those specified in the <attribute> element. If an attribute
structure (container) exists, it is used to define a nested lexical scope for child attributes.

Attributes may be scalar (one value) or one dimensional arrays. Arrays are specified by using whitespace (default) to separate the different values. The attribute@separator may also be set in order to specify a different separator, such as CSV format or to specify a non-whitespace separator so strings with whitespace are not tokenized. We will give examples of creating array attributes below.

=== Adding New Attributes or Modifying an Existing Attribute ===

If a specified attribute with the attribute@name does not exist at the current lexical scope,
a new one is created with the given type and value. For example, assume "new_metadata" doesn't
exist at the current parse scope. Then:

<attribute name="new_metadata" type="string" value="This is a new entry!"/>

will create the attribute at that scope. Note that value can be specified in the content of the
element as well. This is identical to the above:

<attribute name="new_metadata" type="string">This is a new entry!</attribute>

If the attribute@name already exists at the scope, it is modified to contain the specified type and value.

==== Arrays ====

As in NcML, for numerical types an array can be specified by separating the tokens by whitespace (default) or be
specifying the token separator with attribute@separator. For example,

<attribute name="myArray" type="int">1 2 3</attribute>

and

<attribute name="myArray" type="int" separator=",">1,2,3</attribute>

both specify the same array of three integers named "myArray".

TODO Add more information on splitting with a separator!

==== Structures (Containers) ====

We use attribute@type="Structure" to define a new (or existing) attribute container. So if we
wanted to add a new attribute structure, we'd use something like this:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

Assuming "MySamples" doesn't already exist, an attribute container will be created at the current scope
and the "Location" and "Samples" attributes will be added to it.

Note that we can create nested attribute structures to arbitrary depth this way as well.

If the attribute container with the given name already exists at the current scope,
then the attribute@type="Structure" form is used to define
the lexical scope for the container. In other words, child <attribute> elements will
be processed within the scope of the container. For example, in the above example, if
"MySamples" already exists, then the "Location" and "Samples" will be processed within the
existing container (they may or may not already exist as well).

==== Renaming an Existing Attribute or Attribute Container ====

We also support the attribute@orgName attribute for renaming attributes.

For example,

<attribute name="NewName" orgName="OldName" type="string"/>

will rename an existing attribute "OldName" to "NewName" while leaving its value alone.
If attribute@value is also specified, then the attribute is renamed ''and'' has its value modified.

This works for renaming attribute containers as well:

<attribute name="MyNewContainer" orgName="MyOldContainer" type="Structure"/>

will rename an existing "MyOldContainer" to "MyNewContainer". Note that any children
of this container will remain in it.

=== DAP ''OtherXML'' Extension ===

The module now allows specification of attributes of the new DAP type "OtherXML". This allows the NCML file author to inject arbitrary well-formed XML into an attribute for clients that want XML metadata rather than just string or url. Internally, the attribute is still a string (and in a DAP DAS response will be quoted inside one string). However, since it is XML, the NCMLParser still parses it and checks it for well-formedness (but NOT against schemas). This extension allows the NCMLParser to parse the arbitrary XML within the given attribute without causing errors, since it can be any XML.

The injected XML is most useful in the DDX response, where it shows up directly in the response as XML. XSLT and other clients can then parse it.

==== Errors ====

*The XML '''must''' be in the content of the <attribute type="OtherXML"> element. It is a parser error for ''attribute@value'' to be set if ''attribute@type'' is "OtherXML".
*The XML must also be well-formed since it is parsed. A parse error will be thrown if the OtherXML is malformed.

==== Example ====

Here's an example of the use of this special case.

<pre>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/coverage/200803061600_HFRadar_USEGC_6km_rtv_SIO.nc">

<attribute name="someName" type="OtherXML">
<Domain xmlns="http://www.opengis.net/wcs/1.1"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:gml="http://www.opengis.net/gml/3.2"
>
<SpatialDomain>
<ows:BoundingBox crs="urn:ogc:def:crs:EPSG::4326">
<ows:LowerCorner>-97.8839 21.736</ows:LowerCorner>
<ows:UpperCorner>-57.2312 46.4944</ows:UpperCorner>
</ows:BoundingBox>
</SpatialDomain>
<TemporalDomain>
<gml:timePosition>2008-03-27T16:00:00.000Z</gml:timePosition>
</TemporalDomain>
</Domain>
<SupportedCRS xmlns="http://www.opengis.net/wcs/1.1">urn:ogc:def:crs:EPSG::4326</SupportedCRS>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">netcdf-cf1.0</SupportedFormat>
<SupportedFormat xmlns="http://www.opengis.net/wcs/1.1">dap2.0</SupportedFormat>
</attribute>

</netcdf>
</pre>

'''TODO'' Put the DDX response for the above in here!

====Namespace Closure====

Furthermore, the parser will make the chunk of OtherXML "namespace closed". This means any namespaces specified in parent NCML elements of the OtherXML tree will be "brought down" and added to the ''root'' OtherXML elements so that the subtree may be pulled out and added to the DDX and still have its namespaces. The algorithm doesn't just bring used prefixes, but brings ''all'' of the lexically scoped closest namespaces in all ancestors. In other words, it adds unique namespaces (as determined by prefix) in order from the root of the OtherXML tree as it traverses to the root of the NCML document.

Namespace closure is a syntactic sugar that simplifies the author's task since they can specify the namespaces just once at the top of the NCML file and expect that when the subtree of XML is added to the DDX that these namespaces will come along with that subtree of XML. Otherwise they have to explicitly add the namespaces to each attributes.

'''TODO''' Add an example!

== <remove> Element ==

The <remove> element can remove attributes and variables. For example:

<pre>
<attribute name="NC_GLOBAL" type="Structure">
<remove name="base_time" type="attribute"/>
</attribute>
</pre>

will remove the attribute named "base_time" in the attribute structure named "NC_GLOBAL".

Note that this works for attribute containers as well! We could recursively remove the ''entire''
attribute container (i.e. it and all its children) with:

<pre>
<remove name="NC_GLOBAL" type="attribute"/>
</pre>

It also can be used to remove variables from existing datasets:

</pre>
<remove name="SomeExistingVariable" type="variable"/>
</pre>

This also recurses on variables of type Structure --- the entire structure including all of its children are removed from the dataset's response.

'''Parse Errors''':
* It is a parse error if the given attribute or variable doesn't exist in the current scope

== <aggregation> Element ==

Aggregation involves combining multiple datasets (<netcdf>) into a virtual "single" dataset in various ways. For a tutorial on aggregation in NcML 2.2, the reader is referred to the Unidata page: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

NcML 2.2 supports multiple types of aggregation: union, joinNew, joinExisting, and fmrc (forecast model run collection).

The current version of the NcML module supports two of these aggregations:

*Union [[NCML_Module_Aggregation_Union]]
*JoinNew [[NCML_Module_Aggregation_JoinNew]]

A ''union'' aggregation specifies that the first instance of a variable or attribute (by name) that is found in the ordered list of datasets will be the one in the output aggregation. This is useful for combining two dataset files, each which may contain a single variable, into a composite dataset with both variables.

A JoinNew aggregation joins a variable which exists in multiple datasets (usually samples of a datum over time) into a new variable containing the data from ''all'' member datasets by creating a new outer dimension. The ''i''th component in the new outer dimension is the variable's data from the ''i''th member dataset. It also adds a new coordinate variable of whose name is the new dimension's name and whose shape (length) is the new dimension as well. This new coordinate variable may be explicitly given by the author or may be autogenerated in one of several ways.

== <scan> Element ==

The scan element can be used within an aggregation context to allow a directory to be searched in various ways in order to specify the members of an aggregation. This allows a static NcML file to refer to an aggregation which may change over time, such as where a new data file is generated each day.

We describe usage of the <scan> element in detail in the joinNew aggregation tutorial at [[NCML_Module_Aggregation_JoinNew]].

= Errors =

There are three types of error messages that may be returned:

* Internal Error
* Resource Not Found Error
* Parse Error

== Internal Errors ==
'''Internal errors''' should be reported to support@opendap.org as they are likely bugs.

== Resource Not Found Errors ==

If the netcdf@location specifies a non-existent local dataset (one that is not being served by
the same Hyrax server), it will specify the resource was not found. This may also be returned
if a handler for the specified dataset is not currently loaded in the BES. Users should test that the
dataset to be wrapped already exists and can be viewed on the running server before writing NcML
to add metadata. It's also an error to refer to remote datasets (at this time).

== Parse Errors ==

'''Parse errors''' are user errors in the NcML file. These could be malformed XML, malformed NcML,
unimplemented features of NcML, or could be errors in referring to the wrapped dataset.

The error message should specify
the error condition as well as the "current scope" as a fully qualified DAP name within the
loaded dataset. This should be enough information to correct the parse error as new NcML
files are created.

The parser will generate parse errors in various
situations where it expects to find certain structure in the underlying dataset. Some examples:

* A variable of the given name was not found at the current scope
* attribute@orgName was specified, but the attribute cannot be found at current scope.
* attribute@orgName was specified, but the new name is already used at current scope.
* remove specified a non-existing attribute name

= Grid Metadata Tutorial =

Please see the page [[Grid_Metadata_Tutorial]] for an example of adding metadata to the various parts of a DAP Grid variable.

= Aggregation Tutorials =

The NcML module may also be used to aggregate multiple datasets into one virtual dataset. We currently support two of the NcML aggregtions:

*Union [[NCML_Module_Aggregation_Union]]: combine multiple datasets into one by merging variables together, selecting the first of each unique name.
*JoinNew [[NCML_Module_Aggregation_JoinNew]]: combine variables across multiple datasets by creating a new outer dimension and coordinate variable for each of the sample datasets.

Please see the sections for a tutorial on the various uses of these aggregations.

= Additions/Changes to NcML 2.2 =

This section will keep track of changes to the NcML 2.2 schema. Eventually these
will be rolled into a new schema.

== Attribute Structures (Containers) ==
This module also adds functionality beyond the current NcML 2.2 schema
--- it can handle nested <attribute> elements in order to make
attribute structures. This is done by using the <attribute
type="Structure"> form, for example:

<attribute name="MySamples" type="Structure">
<attribute name="Location" type="string" value="Station 1"/>
<attribute name="Samples" type="int">1 4 6</attribute>
</attribute>

"MyContainer" describes an attribute structure with two attribute fields,
a string "Location" and an array of int's called "Samples".
Note that an attribute structure of this form can
only contain other <attribute> elements and NOT a value.

If the container does not already exist, it will be created at the scope it is declared, which could
be:

* Global (top of dataset)
* Within a variable's attribute table
* Within another attribute container

If an attribute container of the given name already exists at the lexical scope, it is traversed
in order to define the scope for the nested (children) attributes it contains.

== Unspecified Variable Type Matching for Lexical Scope ==

We also allow the type attribute of a variable element (variable@type) to be the empty string
(or unspecified) when using existing variables to define the lexical scope of an <attribute>
transformation. In the schema, variable@type is (normally) required.

== DAP 2 Types ==

Additionally, we allow DAP2 atomic types (such as UInt32, URL) in addition to the NcML types.
The NcML types are mapped onto the closest DAP2 type internally.

== DAP OtherXML Attribute Type ==

We also allow attributes to be of the new DAP type "OtherXML" for injecting arbitrary XML into an attribute as content rather than trying to form a string. This allows the parser to check well-formedness.

== Forward Declaration of Dimensions ==

Since we use a SAX parser for efficiency, we require the <dimension> elements to come ''before'' their use in a ''variable@shape''. One way to change the schema to allow this is to force the dimension elements to be specified in a sequence after explicit and metadata choice and before all other elements.

== Aggregation Element Location and Processing Order Differences ==

NcML specifies that if a dataset (<netcdf> element) specifies an aggregation element, the aggregation element is always processed first, regardless of its ordering within the <netcdf> element. Our parser, since it is SAX and not DOM, modifies this behavior in that order matters in some cases:

* Metadata (<attribute>) elements specified ''prior'' to an aggregation "shadow" the aggregation versions. This is be useful for "overriding" an attribute or variable in a union aggregation, where the first found will take precedence.
* JoinNew: If the new coordinate variable's data is to be set explicitly by specifying the new dimension's shape (either with explicit data or the autogenerated data using values@start and values@increment attributes), the <variable> ''must'' come after the aggregation since the size of the dimension is unknown until the aggregation element is processed.

= Backward Compatibility Issues =

Due to the way shared dimensions were implemented in the NetCDF, HDF4, and HDF5 handlers, the DAS responses did not follow the DAP2 specification. The NcML module, on the other hand, generates DAP2 compliant DAS for these datasets, which means that wrapping some datasets in NcML will generate a DAS with a different structure. This is important for the NcML author since it changes the names of attributes and variables. In order for the module to find the correct scope for adding metadata, for example, the DAP2 DAS must be used.

In general, what this means is that an empty "passthrough" NcML file should be the starting point for authoring an NcML file. This file would just specify a dataset and nothing else:

<pre>
<netcdf location="/data/ncml/myNetcdf.nc"/>
</pre>

The author would then request the DAS response for the NCML file and use that as the starting point for modifications to the original dataset.

More explicit examples are given below.

== NetCDF ==

The NetCDF handler represents some NC datasets as a DAP 2 Grid, but the returned DAS is not consistent with the DAP 2 spec for the attribute hierarchy for such a Grid. The map vector attributes are placed as siblings of the grid attributes rather than within the grid lexical scope. For example, here's the NetCDF Handler DDS for a given file:

<pre>
Dataset {
Grid {
Array:
Int16 cldc[time = 456][lat = 21][lon = 360];
Maps:
Float64 time[time = 456];
Float32 lat[lat = 21];
Float32 lon[lon = 360];
} cldc;
} cldc.mean.nc;
</pre>

showing the Grid. Here's the DAS the NetCDF handler generates:

<pre>
Attributes {
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
}
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
}
</pre>

Note the map vector attributes are in the "dataset" scope.

Here's the DAS that the NcML Module produces from the correctly formed DDX:

<pre>
Attributes {
NC_GLOBAL {
String title "COADS 1-degree Equatorial Enhanced";
String history "";
String Conventions "COARDS";
}
DODS_EXTRA {
String Unlimited_Dimension "time";
}
cldc {
Float32 valid_range 0.000000000, 8.000000000;
Float32 actual_range 0.000000000, 8.000000000;
String units "okta";
Int16 precision 1;
Int16 missing_value 32766;
Int16 _FillValue 32766;
String long_name "Cloudiness Monthly Mean at Surface";
String dataset "COADS 1-degree Equatorial Enhanced\\012AI";
String var_desc "Cloudiness\\012C";
String level_desc "Surface\\0120";
String statistic "Mean\\012M";
String parent_stat "Individual Obs\\012I";
Float32 add_offset 3276.500000;
Float32 scale_factor 0.1000000015;
cldc {
}
time {
String units "days since 1-1-1 00:00:0.0";
String long_name "Time";
String delta_t "0000-01-00 00:00:00";
String avg_period "0000-01-00 00:00:00";
Float64 actual_range 715511.00000000000, 729360.00000000000;
}
lat {
String long_name "Latitude";
String units "degrees_north";
Float32 actual_range 10.00000000, -10.00000000;
}
lon {
String long_name "Longitude";
String units "degrees_east";
Float32 actual_range 0.5000000000, 359.5000000;
}
}
}
</pre>

Here the Grid Structure "cldc" and its contained data array (of the same name "cldc") and map vectors have their own attribute containers as DAP 2 specifies.

What this means for the author of an NcML file adding metadata to a NetCDF dataset that returns a Grid is that they should generate a "passthrough" file and get the DAS and then specify modifications based on that structure.

Here's an example passthrough:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">
</netcdf>
</pre>

For example, to add an attribute to the map vector "lat" in the above, we'd need the following NcML:

<pre>
<netcdf location="data/ncml/agg/cldc.mean.nc" title="This file results in a Grid">

<variable name="cldc" type="Structure">

<variable name="lat">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lat!</attribute>
</variable>
<variable name="lon">
<attribute name="Description" type="string">I am a new attribute in the Grid map vector named lon!</attribute>
</variable>
</variable>
</netcdf>
</pre>

This clearly shows that the structure of the Grid must be used in the NcML: the attribute being added is technically "cldc.lat.Description" in a fully qualified name. The parser would return an error if it was attempted as "lat.Description" as the NetCDF DAS for the original file would have led one to believe.

== HDF4/HDF5 ==

Similarly to the NetCDF case, the Hyrax HDF4 Module produces DAS responses that do not respect the DAP2 specification. If an NcML file is used to "wrap" an HDF4 dataset, the
correct DAP2 DAS response will be generated, however.

This is important for those writing NcML for HDF4 data since the
lexical scope for attributes relies on the correct DAS form --- to
handle this, the user should start with a "passthrough" NcML file (see the above NetCDF example) and
use the DAS from that as the starting point for knowing the structure
the NcML handler expects to see in the NcML file. Alternatively, the DDX has the
proper attribute structure as well (the DAS is generated from it).

= Known Bugs =

There are no known bugs currently.

= Planned Future Enhancements =

Planned enhancements for future versions of the module include:

* New NcML Aggregations
** JoinExisting
*** Joins a variable across multiple datasets by appending the data for a given dimension from each dataset
*** Will also allow directory scans for specifying the aggregation
** Forecast Model Run Collection (FMRC)
*** Special case of JoinNew for forecast data with two time variables
*** See: http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/FmrcAggregation.html

= Copyright =

This software is copyrighted under the GNU Lesser GPL. Please see the
files COPYING and COPYRIGHT that came with this distribution.

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T19:56:27Z

Mjohnson: /* Pointing the Workspace at Shrew Using Existing Makefiles */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shrew Using Existing Makefiles===

'''NOTE''': It is important that you have run autoconf and configure in the shrew directory tree before you start using Eclipse. The simplest way to do this for a debug environment is to:

<tt>source build_hyrax_debug</tt>

in the top-level shrew directory as mentioned above.

Now we will set up projects for the things we expect to use. In our case, we have a separate project for libdap, bes, and the ncml_module. We have tried to create a top-level shrew project in the workspace, but due to the way the Makefile is set up and the manner in which Eclipse finds source files when debugging, having a shrew project plus a module project will cause headaches. It's better to just create one per item you plan to debug.

'''NOTE''': You cannot expect a breakpoint to work unless it is in a project in the workspace! Eclipse is smart enough to find source matching the symbol table in the case of stepping (sometimes), but don't be surprised if a breakpoint you set in, say, the hdf4_handler, doesn't go off (though you expect it should) unless you explicitly add the hdf4_handler project using the section below on creating projects for modules.

First, however, we need a new workspace to put everything into.

==== Create a New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Libdap====

If you plan to modify the BES or libdap, or even if you want to be able to set breakpoints and step through code for these in the debugger, you must make a project for each of these. We'll use libdap as the example, but the bes can be done in the same way.

'''NOTE''': it's important to get the settings right so that Eclipse doesn't overwrite your Makefile's!

* Select File > New C++ Project
* Give it a name: "libdap" (this doesn't need to match the directory structure... it could be "libdap trunk" or "libdap 1.6 release" etc.
* Uncheck "Use Default Location"
* Browse for the libdap directory and choose it. This should be in $shrew/src/libdap (where $shrew is the top-level shrew dir).
* Under "Project Type..." select "Makefile Project"
* Make sure to select "Empty Project" under the "Makefile Project" treeview to tell Eclipse to use existing Makefile's.
* Tell it the toolchain to use if there's a choice: for Mac it's "MaxOSX GCC".
* Select the "Finish" button.

A new project should be in the project view.

Now we need to tell it the make command since libdap must be installed for it to work in shrew.

* Select the new libdap project.
* Select "Project > Properties"
* Select the "C/C++ Build" in the treeview on the left
* Under "Builder" pane, the uncheck "Use default build command" (which is make).
* This ungrays the make command: enter "make install" for the command.

Now when you select the libdap project and then hit the Build command for it, it will compile and then install it into the shrew location properly.

==== New Project for the BES ====

Go through the same steps as for libdap, except just choose "bes" for the name and project location. Also change its build command to "make install".

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

===== Set the Include Path =====

Setting the include path in the workspace will avoid the "can't find include" problems in the IDE. The Makefile may have the includes, but Eclipse doesn't know where to find them.

Under the Project -> Properties dialog (making sure to have the module project selected):

* Click the "C/C++ General" tree to open it.
* Select "Paths and Symbols"
* On the "Includes" tab, add libdap and bes to C++ language include paths:
**Click "Add..."
**Select all checkmarks: "Add to all configurations", "Add to all languages" and "Is a workspace path".
**Hit the "Workspace..." button and browse to the "shrew/src/libdap" and add it. It should show up as "/shrew/src/libdap" since we selected "is a workspace path"
**Do the same for the bes include dir.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T19:26:14Z

Mjohnson: /* Pointing the Workspace at Shrew Using Existing Makefiles */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shrew Using Existing Makefiles===

'''NOTE''': It is important that you have run autoconf and configure in the shrew directory tree before you start using Eclipse. The simplest way to do this for a debug environment is to:

<tt>source build_hyrax_debug</tt>

in the top-level shrew directory as mentioned above.

Now we will set up projects for the things we expect to use. In our case, we have a separate project for libdap, bes, and the ncml_module. We have tried to create a top-level shrew project in the workspace, but due to the way the Makefile is set up and the manner in which Eclipse finds source files when debugging, having a shrew project plus a module project will cause headaches. It's better to just create one per item you plan to debug.

First, however, we need a new workspace for the projects.

==== Create a New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Libdap====

If you plan to modify the BES or libdap, or even if you want to be able to set breakpoints and step through code for these in the debugger, you must make a project for each of these. We'll use libdap as the example, but the bes can be done in the same way.

'''NOTE''': it's important to get the settings right so that Eclipse doesn't overwrite your Makefile's!

* Select File > New C++ Project
* Give it a name: "libdap" (this doesn't need to match the directory structure... it could be "libdap trunk" or "libdap 1.6 release" etc.
* Uncheck "Use Default Location"
* Browse for the libdap directory and choose it. This should be in $shrew/src/libdap (where $shrew is the top-level shrew dir).
* Under "Project Type..." select "Makefile Project"
* Make sure to select "Empty Project" under the "Makefile Project" treeview to tell Eclipse to use existing Makefile's.
* Tell it the toolchain to use if there's a choice: for Mac it's "MaxOSX GCC".
* Select the "Finish" button.

A new project should be in the project view.

Now we need to tell it the make command since libdap must be installed for it to work in shrew.

* Select the new libdap project.
* Select "Project > Properties"
* Select the "C/C++ Build" in the treeview on the left
* Under "Builder" pane, the uncheck "Use default build command" (which is make).
* This ungrays the make command: enter "make install" for the command.

Now when you select the libdap project and then hit the Build command for it, it will compile and then install it into the shrew location properly.

==== New Project for the BES ====

Go through the same steps as for libdap, except just choose "bes" for the name and project location. Also change its build command to "make install".

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

===== Set the Include Path =====

Setting the include path in the workspace will avoid the "can't find include" problems in the IDE. The Makefile may have the includes, but Eclipse doesn't know where to find them.

Under the Project -> Properties dialog (making sure to have the module project selected):

* Click the "C/C++ General" tree to open it.
* Select "Paths and Symbols"
* On the "Includes" tab, add libdap and bes to C++ language include paths:
**Click "Add..."
**Select all checkmarks: "Add to all configurations", "Add to all languages" and "Is a workspace path".
**Hit the "Workspace..." button and browse to the "shrew/src/libdap" and add it. It should show up as "/shrew/src/libdap" since we selected "is a workspace path"
**Do the same for the bes include dir.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T19:08:16Z

Mjohnson: /* Pointing the Workspace at Shew Using Existing Makefiles */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shrew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

===== Set the Include Path =====

Setting the include path in the workspace will avoid the "can't find include" problems in the IDE. The Makefile may have the includes, but Eclipse doesn't know where to find them.

Under the Project -> Properties dialog (making sure to have the module project selected):

* Click the "C/C++ General" tree to open it.
* Select "Paths and Symbols"
* On the "Includes" tab, add libdap and bes to C++ language include paths:
**Click "Add..."
**Select all checkmarks: "Add to all configurations", "Add to all languages" and "Is a workspace path".
**Hit the "Workspace..." button and browse to the "shrew/src/libdap" and add it. It should show up as "/shrew/src/libdap" since we selected "is a workspace path"
**Do the same for the bes include dir.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T19:07:53Z

Mjohnson: /* New Project for Shrew */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

===== Set the Include Path =====

Setting the include path in the workspace will avoid the "can't find include" problems in the IDE. The Makefile may have the includes, but Eclipse doesn't know where to find them.

Under the Project -> Properties dialog (making sure to have the module project selected):

* Click the "C/C++ General" tree to open it.
* Select "Paths and Symbols"
* On the "Includes" tab, add libdap and bes to C++ language include paths:
**Click "Add..."
**Select all checkmarks: "Add to all configurations", "Add to all languages" and "Is a workspace path".
**Hit the "Workspace..." button and browse to the "shrew/src/libdap" and add it. It should show up as "/shrew/src/libdap" since we selected "is a workspace path"
**Do the same for the bes include dir.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T17:16:17Z

Mjohnson: /* New Project for a Module */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

===== Set the Include Path =====

Setting the include path in the workspace will avoid the "can't find include" problems in the IDE. The Makefile may have the includes, but Eclipse doesn't know where to find them.

Under the Project -> Properties dialog (making sure to have the module project selected):

* Click the "C/C++ General" tree to open it.
* Select "Paths and Symbols"
* On the "Includes" tab, add libdap and bes to C++ language include paths:
**Click "Add..."
**Select all checkmarks: "Add to all configurations", "Add to all languages" and "Is a workspace path".
**Hit the "Workspace..." button and browse to the "shrew/src/libdap" and add it. It should show up as "/shrew/src/libdap" since we selected "is a workspace path"
**Do the same for the bes include dir.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T16:16:57Z

Mjohnson: /* Debugger Tab */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-29T15:35:42Z

Mjohnson: /* Debugger Tab */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.
* Under the "Debugger Options" Main tab, check off "Use full file path to set breakpoints." [''NOTE'': I think this is needed to get the breakpoints in a dynamically loaded module, but not sure...]

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T21:17:49Z

Mjohnson: /* Arguments Tab */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setContainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T21:05:41Z

Mjohnson: /* Test It */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setCo
ntainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc''

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T21:05:19Z

Mjohnson: /* Create a New Debug Configuration */

=Eclipse as an IDE for Shew=

It can be useful to use an IDE when debugging and developing for Hyrax due to the graphical interface to gdb as well as the extensive searching and code indexing features that Eclipse provides. This document will go through the steps for getting Eclipse set up pointing into a Shrew checkout. We also explain how we set up debug configurations for debugging modules using the besstandalone program.

==Downloading Eclipse==

Eclipse may be found at http://eclipse.org/downloads/

Download the Eclipse for C/C++ Development (with Mylyn Integration).

If you also plan to do Java development, you can add these bundles in later using the Software Update.

==Creating a Workspace In A Shrew Directory==

You need to have a shrew project checked out and built in debug mode using the standard approach. The first-time debug build is ''required'' after a fresh checkout in order to get autoconf and configure to run properly and set up the Makefile for all the projects. Eclipse will then piggyback on these.

If you do not have this done already, please follow the steps in the next section "Setting Up a Debug Shrew Project"

===Setting Up a Debug Shrew Project===

Go to the directory you want the shrew project and do a checkout. For example, to get the fully trunk shrew:

<tt>svn co http://scm.opendap.org/svn/trunk/shrew</tt>

Note that shrew uses svn externals, so make sure you know which versions you are getting so when you commit you are commiting to the right place!

To check the externals being used, get the property in the shrew directory you checked out:

<tt>svn propget svn:externals</tt>

which for the trunk shrew returns:

<pre>
aries@chinchilla$ svn propget svn:externals
^/trunk/libdap src/libdap
^/trunk/bes src/bes
^/trunk/dap-server src/modules/dap-server
^/trunk/freeform_handler src/modules/freeform_handler
^/trunk/fileout_netcdf src/modules/fileout_netcdf
^/trunk/netcdf_handler src/modules/netcdf_handler
^/trunk/hdf4_handler src/modules/hdf4_handler
^/trunk/hdf5_handler src/modules/hdf5_handler
^/trunk/ncml_module src/modules/ncml_module
^/trunk/wcs_gateway_module src/modules/wcs_gateway_module
^/trunk/wcs_module src/modules/wcs_module
^/trunk/olfs src/olfs
</pre>

==== Module Development/Debugging ====

If you plan to debug or develop modules for the BES in Hyrax 1.6, we suggest you use:

http://scm.opendap.org/svn/branch/shrew/module_dev_1.6

as the shrew starting point as this will be set to use the trunk versions of the modules but to use the fixed, release versions of libdap and the BES for Hyrax 1.6. This ensures that development doesn't use any new APIs from the libdap or bes trunk versions so that modules may be released incrementally and dynamically work with an existing Hyrax 1.6 installation with no other changes necessary.

====Configuring a Debug Build====

Since you likely plan to use the shrew project for debugging in Eclipse, you'll want to run the build_hyrax_debug script in order to make sure the configuration for all the subprojects is set up to perform debug builds (i.e. no optimization and full symbol tables). That way Eclipse will be able to find breakpoints and let you view variables in a running debug.

In a bash terminal in the top-level shrew directory, type:

<tt>source build_hyrax_debug</tt>

This will run autoconf on everything, then configure, then build it. If there are any missing dependencies, make sure to get them until everything builds. This will ensure that all the subprojects are configured properly so that the Eclipse Build command, which simply calls 'make', will succeed in compiling a debug version of the specified project.

===Pointing the Workspace at Shew Using Existing Makefiles===

==== New Workspace====
First, create a new workspace if you have an existing one. You can use:

File > Switch Workspace > Other...

and just create a new directory for the workspace to make the new one.

====New Project for Shrew====

You probably won't use this directly, but making a top-level project for shrew will let Workspace searches find functions and files in all the subprojects. More likely, you will be using a specific subproject directly. (More on this later).

Create a new Project in the workspace.

File > New > C++ Project

In the dialog:

* Enter a project name. Something like "Shrew" should be OK.
* UNcheck "Use Default Location" since we want to point at an existing directory
* Enter the location of the shrew project you checked out and want to debug (you can use Browse...)
* Select the "Makefile Project" tab since we want to use 'make' to build
* Under the "Makefile Project" select "Empty Project" so Eclipse doesn't overwrite the existing Makefile
* Select the "MaxOSX GCC Toolchain"

You will now have a shrew project. Since Eclipse uses 'make all' to build and shrew doesn't have this target, you can't use Build on shrew. In general, you'll want to create a subproject for each project you plan to modify so that you can do make on each one. Otherwise, you will want to do makes in the shell as needed.

====New Project for BES and/or Libdap====

If you plan to modify the BES or libdap, you can make a project for each of these as well in the same way, but by pointing the location of the project at the appropriate subdirectory (src/bes or src/libdap) of the top-level shrew checkout.

====New Project for a Module====

In this section, we'll set up the subproject for a particular module you'll want to develop or debug. I will use the ncml_module as the example since this is the one I normally develop and I have added some tools to the project to make setting up debugging files easier.

Create the project just like the shrew project, using an Empty Project in the Makefile Project tab. Set the location to be the subdirectory of the top-level shrew, e.g.

src/modules/ncml_module

This will create a separate project tab for the module. If you click on this project and press the Build button (with the hammer icon), just that project will be built.

==Debugging a Module==

In order to debug a given module, we need to set up a Debug configuration for it. The way I do this for the ncml_module is by using the '''besstandalone''' application and pointing it at the same generated ''bes.conf'' file that the Autotest testsuite uses (all the modules should have this structure at this point). This makes sure the required modules get loaded, and in particular, the testsuite ''bes.conf'' points at the locally compiled module shared object in the project ''.libs'' directory, so 'make install' isn't required. We will also need a BES XML command file to automatically run the command we want to debug. For the ncml_module, this means we load a given NcML file and ask it for a particular response type (DAS, DDS, DDX, DODS) and perhaps with a constraint. Then we need to make sure Eclipse knows where to find the dynamically loaded library. Let's go through these steps one by one.

===Create a New Debug Configuration===

Go to:

Run > Debug Configurations...

When the dialog pops up, select the "C/C++ Application" entry and "New" to create a new configuration. This should pop up the configuration options for the new Debug Configuration.

We'll want to make changes in several of the tabs.

==== Main Tab: Select '''besstandalone''' as the application to run====

Now we want to set these options to run '''besstandalone''' with the arguments we want. To do this:

* Select the "Browse..." button under C/C++ Application.
* Go up from the module directory to the shrew directory and into its "bin" directory (this should exist if you did a successful build_hyrax_debug).
* Select the executable '''besstandalone'''

The application path will then loom something like this:

<tt>/Users/aries/src/module_dev_1.6/bin/besstandalone</tt>

Leave "Connect process input & output to a terminal" checked. We want to see the output in the Console.

Leave the configuration window open for the next changes.

==== Arguments Tab ====

Now select the "Arguments" tab.

Under the Program Arguments text field, enter the arguments to give to '''besstandalone''' as follows. First, make sure the working directory is the module directory, i.e.

<tt>${workspace_loc:ncml_module}</tt>

in our case.

Our testsuite directory in ncml_module is "tests", so that's where our ''bes.conf'' lives. Note that other modules use the directory ''bes-testsuite'' instead.

Here's the arguments list we will use for the ncml_module:

<tt>-c tests/bes.conf -d"cerr,ncml,ncml:2" -i tests/eclipse_debug.bescmd</tt>

Here we:

* Specify the bes.conf to use in the testsuite directory so that we load the locally compiled (uninstalled) module
* Turn on a few debug channels, here: "cerr", "ncml" and "ncml:2" which the ncml_module uses for debug output.
* Tell '''besstandalone''' to use the file "tests/eclipse_debug.bescmd" as its input file.

Note that the "eclipse_debug.bescmd" file doesn't yet exist. This is a file that we recreate for each test we want to do. It can be made by hand, but the ncml_module has a script in tests called "generate_bescmd" that creates a BES XML command for a given ncml file and response type.

Here's an example eclipse_debug.bescmd for the DAS response for "/data/ncml/fnoc1_improved.ncml":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<request reqID="some_unique_value" >
<setContext name="dap_format">dap2</setContext>
<setContainer name="c" space="catalog">/data/ncml/fnoc1_improved.ncml</setCo
ntainer>
<define name="d">
<container name="c"></container>
</define>
<get type="das" definition="d" />
</request>
</pre>

If you're using some other module, you probably want to copy this into a new file ''eclipse_debug.bescmd'' and modify it as your debug needs change, such as to test different responses.

====Debugger Tab====

Next, select the Debugger tab.

* Decide whether to leave "Stop on startup at: main" selected or not. I usually leave it selected for the first pass just to be sure I can get a breakpoint in the '''besstandalone''' main() function in case there was a problem. You can edit the configuration to deselect it later.

* Under "Debugger Options", select the "Shared Libraries" tab.
* Make sure "Load shared library symbols automatically" is selected since our modules are dynamically loaded.
* Under Directories, click the "Add..." button
* "Browse..." your way to select the ''.libs'' directory in the module directory. In our case the full path ends up being: <tt>/Users/aries/src/module_dev_1.6/src/modules/ncml_module/.libs</tt>. This should match the module loaded in the bes.conf!
* Leave "Stop on shared library events" unchecked since this will break on every library that gets loaded otherwise.

====Apply Changes====

On the bottom of the configuration, click "Apply" to save the changes.

====Test It====

If you're still in the debug configuration window, you can click the Debug button (assuming you have created a valid ''eclipse_debug.bescmd'' file) to run the debugger. If all worked out and you left Stop on startup at main selected, you will eventually get a breakpoint in the bes StandAloneApp.cc main() function!

From now on, if you have the module selected in the project window, you can click the Debug button (the bug icon) (or use Run > Debug).

You might also want to go to the top-level module initialize call and set a breakpoint to make sure you're module gets loaded properly and that you can stop on a breakpoint in it. For the NcML Module, this is the file:

''NCMLModule.cc"

and the function

<tt>NCMLModule::initialize( const string &modname )</tt>

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T20:50:11Z

Mjohnson: /* Debugging a Module */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T20:11:27Z

Mjohnson: /* Eclipse as an IDE for Shew */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T20:10:52Z

Mjohnson: /* New Project for a Module */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T20:06:33Z

Mjohnson: /* Pointing the Workspace at Shew Using Existing Makefiles */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T20:01:32Z

Mjohnson: /* Creating a Workspace In A Shrew Directory */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T19:58:56Z

Mjohnson: /* Pointing the Workspace at Shew Using Existing Makefiles */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T19:17:09Z

Mjohnson: /* Pointing the Workspace at Shew Using Existing Makefiles */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T19:05:55Z

Mjohnson: /* Creating a Workspace In A Shrew Directory */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T18:23:04Z

Mjohnson: /* Creating a Workspace In A Shrew Directory */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T18:18:05Z

Mjohnson: /* Downloading Eclipse */

Eclipse - How to Setup Eclipse in a Shrew Checkout

2010-04-22T18:01:34Z

Mjohnson: /* Eclipse as an IDE for Shew */

2010-04-20T19:28:41Z

Mjohnson: /* Metadata Specification on the New Coordinate Variable */

=== Metadata Specification on the New Coordinate Variable ===
[[Category:Aggregation]]

We can add metadata to the new coordinate variable in two ways:

* Adding it to the <variable> element directly in the case where the new coordinate variable and values is defined explicitly
* Adding the metadata to an ''automatically created'' coordinate variable by leaving the <values> element out

The first case we have already seen, but we will show it again explicitly. The second case is a little different and we'll cover it separately.

==== Adding Metadata to the Explicit New Coordinate Variable ====

We have already seen examples of explicitly defining the new coordinate variable and giving its values. In these cases, the metadata is added to the new coordinate variable exactly like any other variable. Let's see the example again:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation with explicit map">

<aggregation type="joinNew" dimName="sample_time">
<variableAgg name="dsp_band_1"/>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</aggregation>

<variable name="sample_time" shape="sample_time" type="float">

<attribute name="units" type="string">Days since 01/01/2010</attribute>
<values>100 200 400 1000</values>
</variable>

</netcdf>
</pre>

We see that the '''units''' attribute for the new coordinate variable has been specified. This subset of the DAS (we don't show the extensive global metadata) shows this:

<pre>
dsp_band_1 {
Byte dsp_PixelType 1;
Byte dsp_PixelSize 2;
UInt16 dsp_Flag 0;
UInt16 dsp_nBits 16;
Int32 dsp_LineSize 0;
String dsp_cal_name "Temperature";
String units "Temp";
UInt16 dsp_cal_eqnNumber 2;
UInt16 dsp_cal_CoeffsLength 8;
Float32 dsp_cal_coeffs 0.125, -4;
Float32 scale_factor 0.125;
Float32 add_off -4;
sample_time {
---> String units "Days since 01/01/2010";
}
dsp_band_1 {
}
lat {
String name "lat";
String long_name "latitude";
}
lon {
String name "lon";
String long_name "longitude";
}
}
sample_time {
---> String units "Days since 01/01/2010";
}
</pre>

We show the new metadata with the "--->" marker. Note that the metadata for the coordinate variable is also copied into the new map vector of the aggregated Grid.

Metadata can be specified in this way for ''any'' case where the new coordinate variable is listed explicitly.

==== Adding Metadata to An Autogenerated Coordinate Variable ====

If we expect the coordinate variable to be automatically added, we can also specify its metadata by referring to the variable without setting its values. This is useful in the case of using ''netcdf@coordValue'' and we will also see it is very useful when using a <scan> element for dynamic aggregations.

Here's a trivial example using the default case of the filename:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Test of adding metadata to the new map vector in a joinNew Grid aggregation">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
</aggregation>


<variable type="string" name="filename">
<attribute name="units" type="string">Filename of the dataset</attribute>
</variable>

</netcdf>
</pre>

Note here that we just neglected to add a <values> element since we want the values to be generated automatically by the aggregation. Note also that this is ''almost'' the same way we'd modify an existing variable's metadata. The only difference is we need to "declare" the type of the variable here since technically the variable specified here is a placeholder for the generated coordinate variable. So after the aggregation is specified, we are simply modifying the created variable's metadata, in this case the newly generated map vector.

Here is the DAS portion with just the aggregated Grid and the new coordinate variable:

<pre>
dsp_band_1 {
Byte dsp_PixelType 1;
Byte dsp_PixelSize 2;
UInt16 dsp_Flag 0;
UInt16 dsp_nBits 16;
Int32 dsp_LineSize 0;
String dsp_cal_name "Temperature";
String units "Temp";
UInt16 dsp_cal_eqnNumber 2;
UInt16 dsp_cal_CoeffsLength 8;
Float32 dsp_cal_coeffs 0.125, -4;
Float32 scale_factor 0.125;
Float32 add_off -4;
filename {
String units "Filename of the dataset";
}
dsp_band_1 {
}
lat {
String name "lat";
String long_name "latitude";
}
lon {
String name "lon";
String long_name "longitude";
}
}
filename {
String units "Filename of the dataset";
}
</pre>

Here also the map vector gets a copy of the coordinate variable's metadata.

We can also use this syntax in the case that ''netcdf@coordValue'' was used to autogenerate the coordinate variable:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation with coordValue and metadata">

<aggregation type="joinNew" dimName="sample_time">
<variableAgg name="dsp_band_1"/>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf" coordValue="1"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf" coordValue="10"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf" coordValue="15"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf" coordValue="25"/>
</aggregation>


<variable name="sample_time" shape="sample_time" type="double">
<attribute name="units" type="string">Days since 01/01/2010</attribute>
</variable>

</netcdf>
</pre>

Here we see the metadata added to the new coordinate variable and associated map vector:

<pre>
Attributes {
dsp_band_1 {
Byte dsp_PixelType 1;
Byte dsp_PixelSize 2;
UInt16 dsp_Flag 0;
UInt16 dsp_nBits 16;
Int32 dsp_LineSize 0;
String dsp_cal_name "Temperature";
String units "Temp";
UInt16 dsp_cal_eqnNumber 2;
UInt16 dsp_cal_CoeffsLength 8;
Float32 dsp_cal_coeffs 0.125, -4;
Float32 scale_factor 0.125;
Float32 add_off -4;
sample_time {
---> String units "Days since 01/01/2010";
}
dsp_band_1 {
}
lat {
String name "lat";
String long_name "latitude";
}
lon {
String name "lon";
String long_name "longitude";
}
}
sample_time {
---> String units "Days since 01/01/2010";
}
}
</pre>

==== Parse Errors ====

Since the processing of the aggregation takes a few steps, care must be taken in specifying the coordinate variable in the cases of autogenerated variables.

In particular, it is a Parse Error:

* To specify the shape of the autogenerated coordinate variable if <values> are not set
* To leave out the type or to use a type that does not match the autogenerated type

The second can be somewhat tricky to remember since for existing variables it can be safely left out and the variable will be "found". Since aggregations get processed fulled when the <netcdf> element containing them is closed, the specified coordinate variables in these cases are ''placeholders'' for the automatically generated variables, so they must match the name and type, but not specify a shape since the shape (size of the new aggregation dimension) is not known until this occurs.

=== Metadata Specification on the Aggregation Variable Itself ===

It is also possible to add or modify the attributes on the aggregation variable itself. If it is a Grid, metadata can be modified on the contained array or maps as well. Note that the aggregated variable begins with the metadata ''from the first dataset specified in the aggregation'' just like in a union aggregation.

We will use a Grid as our primary example since other datatypes are similar and simpler and this case will cover those as well.

==== An Aggregated Grid example ====

Let's start from this example aggregation:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf>
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</aggregation>
</netcdf>
</pre>

Here is the DAS for this unmodifed aggregated Grid (with the global dataset metadata removed):

<pre>
Attributes {
dsp_band_1 {
Byte dsp_PixelType 1;
Byte dsp_PixelSize 2;
UInt16 dsp_Flag 0;
UInt16 dsp_nBits 16;
Int32 dsp_LineSize 0;
String dsp_cal_name "Temperature";
String units "Temp";
UInt16 dsp_cal_eqnNumber 2;
UInt16 dsp_cal_CoeffsLength 8;
Float32 dsp_cal_coeffs 0.125, -4;
Float32 scale_factor 0.125;
Float32 add_off -4;
filename {
}
dsp_band_1 {
}
lat {
String name "lat";
String long_name "latitude";
}
lon {
String name "lon";
String long_name "longitude";
}
}
filename {
}
}
</pre>

We will now add attributes to all the existing parts of the Grid:

* The Grid Structure itself
* The Array of data within the Grid
* Both existing map vectors ('''lat''' and '''lon''')

We have already seen how to add data to the new coordinate variable as well.

Here's the NcML we will use. Note we have added units data to the subparts of the Grid, and also added some metadata to the grid itself.

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Showing how to add metadata to all parts of an aggregated grid">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</aggregation>

<variable name="dsp_band_1" type="Structure"> 

1) <attribute name="Info" type="String">This is metadata on the Grid itself.</attribute>

<variable name="dsp_band_1"> 
2) <attribute name="units" type="String">Temp (packed)</attribute> 
</variable> 

<variable name="lat"> 
3) <attribute name="units" type="String">degrees_north</attribute>
</variable>

<variable name="lon"> 
4) <attribute name="units" type="String">degrees_east</attribute>
</variable> 
</variable> 


<variable name="filename" type="String">
5) <attribute name="Info" type="String">Filename with timestamp</attribute>
</variable> 

</netcdf
</pre>

Here we show metadata being injected in several ways, denoted by the 1) -- 5) notations.

1) We are inside the scope of the top-level Grid variable, so this metadata will show up in the attribute table inside the Grid Structure.
2) This is the actual data Array of the Grid, '''dsp_band_1.dsp_band_1'''. We specify the units are a packed temperature.
3) Here we are in the scope of a map variable, '''dsp_band_1.lat'''. We add the units specification to this map.
4) Likewise, we add units to the '''lon''' map vector.
5) Finally, we must close the actual grid and specify the metadata for the ''NEW'' coordinate variable as a sibling of the Grid since this will be used as the canonical prototype to be added to all Grid's which are to be aggregated on the new dimension. Note in this case (unlike previous cases) the type of the new coordinate variable is required since we are specifying a "placeholder" variable for the new map until the Grid is actually processed once its containing <netcdf> is closed (i.e. all data is available to it).

The resulting DAS (with global dataset metadata removed for clarity):

<pre>
Attribute {
... global data clipped ...
dsp_band_1 {
Byte dsp_PixelType 1;
Byte dsp_PixelSize 2;
UInt16 dsp_Flag 0;
UInt16 dsp_nBits 16;
Int32 dsp_LineSize 0;
String dsp_cal_name "Temperature";
String units "Temp";
UInt16 dsp_cal_eqnNumber 2;
UInt16 dsp_cal_CoeffsLength 8;
Float32 dsp_cal_coeffs 0.125, -4;
Float32 scale_factor 0.125;
Float32 add_off -4;
1) String Info "This is metadata on the Grid itself.";
filename {
5) String Info "Filename with timestamp";
}
dsp_band_1 {
2) String units "Temp (packed)";
}
lat {
String name "lat";
String long_name "latitude";
3) String units "degrees_north";
}
lon {
String name "lon";
String long_name "longitude";
4) String units "degrees_east";
}
}
filename {
5) String Info "Filename with timestamp";
}
}
</pre>

We have annotated the DAS with numbers representing which lines in the NcML above correspond to the injected metadata.

Dynamic Aggregation Tutorial

2010-04-20T19:23:20Z

Mjohnson: /* Location (Location Location...) */

=== Location (Location Location...) ===
[[Category:Aggregations]]

The most important attribute of the scan element is the ''scan@location'' element that specifies the top-level search directory for the scan, relative to the BES data root directory specified in the BES configuration.

'''IMPORTANT''': ''ALL'' locations are interpreted relative to the BES root directory and '''NOT''' relative to the location of the NcML file itself! This means that all data to be aggregated '''must''' be in a subdirectory of the BES root data directory and that these directories ''must'' be specified fully, not relative to the NcML file.

For example, if the BES root data dir is "/usr/local/share/hyrax", let ${BES_DATA_ROOT} refer to this location. If the NcML aggregation file is in "${BES_DATA_ROOT}/data/ncml/myAgg.ncml" and the aggregation member datasets are in "${BES_DATA_ROOT}/data/hdf4/myAggDatasets", then the location in the NcML file for the aggregation data directory would be:

<pre>
<scan location="data/hdf4/myAggDatasets"/>
</pre>

which specifies the data directory relative to the BES data root as required.

Again, due to security reasons, the data is always searched under the BES data root. Trying to specify an absolute filesystem path, such as:

<pre>
<scan location="/usr/local/share/data"/>
</pre>

will ''NOT'' work. This directory will also be assumed to be a subdirectory of the ${BES_DATA_ROOT}, regardless of the preceding "/" character.

=== Suffix Criterion ===

The simplest criterion is to match only files of a certain datatype in a given directory. This is useful for filtering out text files and other files that may exist in the directory but which do not form part of the aggregation data.

Here's a simple example:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf"/>
</aggregation>

</netcdf>
</pre>

Assuming that the specified location "data/ncml/agg/grids" contains no subdirectories, this NcML will return all files in that directory that end in ".hdf" in alphanumerical order. In the case of our installed example data, there are four HDF4 files in that directory:

<pre>
data/ncml/agg/grids/f97182070958.hdf
data/ncml/agg/grids/f97182183448.hdf
data/ncml/agg/grids/f97183065853.hdf
data/ncml/agg/grids/f97183182355.hdf
</pre>

These will be included in alphanumerical order, so the scan element will in effect be equivalent to the following list of <netcdf> elements:

<pre>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</pre>

By default, scan will search subdirectories, which is why we mentioned "grids has no subdirectories". We discuss this in the next section.

=== Subdirectory Searching (The Default!) ===

If the author specifies the ''scan@subdirs'' attribute to the value "true" (which is the default!), then the criteria will be applied recursively to any subdirectories of the ''scan@location'' base scan directory as well as to any regular files in the base directory.

For example, continuing our previous example, but giving a higher level location:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan location="data/ncml/agg/" suffix=".hdf" subdirs="true"/>
</aggregation>

</netcdf>
</pre>

Assuming that only the "grids" subdir of "/data/ncml/agg" contains HDF4 files with that extension, the same aggregation as prior will be created, i.e. an aggregation isomorphic to:

<pre>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</pre>

The ''scan@subdirs'' attribute is much for useful for turning off the default recursion. For example, if recursion is '''NOT''' desired, but only files with the given suffix in the given directory are required, the following will do that:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false"/>
</aggregation>
</pre>

=== OlderThan Criterion ===

The ''scan@olderThan'' attribute can be used to filter out files that are "too new". This feature is useful for excluding partial files currently being written by a daemon process, for example.

The value of the attribute is a duration specified by a number followed by a basic time unit. The time units recognized are:

* '''seconds''': { s, sec, secs, second, seconds }
* '''minutes''': { m, min, mins, minute, minutes }
* '''hours''': { h, hour, hours }
* '''days''': { day, days }
* '''months''': { month, months }
* '''years''': { year, years }

The strings inside { } are all recognized as referring to the given time unit.

For example, if we are following our previous example, but we suspect a new HDF file may be written at any time and usually takes 5 minutes to do so, we might use the following NcML:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false" olderThan="10 mins"/>
</aggregation>

</netcdf>
</pre>

Assuming the file will always be written withing 10 minutes, this files does what we wish. Only files whose modification date is older than the given duration from the current system time are included.

'''NOTE''' that the modification date of the file, not the creation date, is used for the test.

=== Regular Expression Criterion ===

The ''scan@regExp'' attribute may be used for more complicated filename matching tests where data for multiple variables, for example, may live in the same directory by whose filenames can be used to distinguish which are desired in the aggregation. Additionally, since the pathname including the location is used for the test, a regular expression test may be used in conjunction with a recursive directory search to find files in subdirectories where the directory name itself is specified in the regular expression, not just the filename. We'll give examples of both of these cases.

We also reiterate that this test is used ''in conjunction'' with any other tests --- the author may also include a suffix and an olderThan test if they wish. All criteria must match for the file to be included in the aggregation.

We recognize the POSIX regular expression syntax. For more information on regular expressions and the POSIX syntax, please see: http://en.wikipedia.org/wiki/Regular_expression.

We will give two basic examples:

* Finding all subdirectories with a given name
* Matching a filename starting with a certain substring

==== Matching a Subdirectory Name ====
Here's an example where we use a subdirectory search to find ".hdf" files in all subdirectories named "grids":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan
location="data/"
subdirs="true"
regExp="^.*/grids/.+\.hdf$"
/>
</aggregation>
</netcdf>
</pre>

The regular expression here is "^.*/grids/.+\/hdf". Let's pull it apart quickly (this is not intended to be a regular expression tutorial):

The "^" matching the beginning of the string, so starts at the beginning of the location pathname. (without this we can match substrings in the middle of strings, etc)

We then match ".*" meaning 0 or more of any character.

We then match the "/grids/" string explicitly, meaning we want all pathnames that contain "/grids/" as a subdirectory.

We then match ".+" meaning 1 or more of any character.

We then match "\." meaning a literal "." character (the backslash "escapes" it).

We then match the suffix "hdf".

Finally, we match "$" meaning the end of the string.

So ultimately, this regular expression finds all filenames ending in ".hdf" that exist in some subdirectory named "grids" of the top-level location.

In following with our previous example, if there was only the one "grids" subdirectory in the ${BES_DATA_ROOT} with our four familiar files, we'd get the same aggregation as before.

==== Matching a partial filename ====

Let's say we have a given directory full of data files whose filename prefix specifies which variable they refer to. For example, let's say our "grids" directory has files that start with "grad" as well as the files that start with "f" we have seen in our examples. We still want just the files starting with "f" to filter out the others. Here's an example for that:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan
location="data/"
subdirs="true"
regExp="^.*/grids/f.+\.hdf$"
/>
</aggregation>
</netcdf>
</pre>

Here we match all pathnames ending in "grids" and files that start with the letter "f" and end with ".hdf" as we desire.

=== Date Format Mark and Timestamp Extraction ===

This section shows how to use the ''scan@dateFormatMark'' attribute along with other search criteria in order to extract and sort datasets by a timestamp encoded in the filename. All that is required is that the timestamp be parseable by a pattern recognized by the Java language "SimpleDateFormat" class ([http://java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html java.text.SimpleDateFormat]), which has also been implemented in C++ in the [http://site.icu-project.org/ International Components for Unicode] library which we use.

We base this example from the Unidata site [http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html Aggregation Tutorial]. Here we have a directory with four files whose filenames contain a timestamp describable by a SimpleDataFormat (SDF) pattern. We will also use a regular expression criterion and suffix criterion in addition to the dateFormatMark since we have other files in the same directory and only wish to match those starting with the characters "CG" that have suffix ".nc".

Here's the list of files (relative to the BES data root dir):

<pre>
data/ncml/agg/dated/CG2006158_120000h_usfc.nc
data/ncml/agg/dated/CG2006158_130000h_usfc.nc
data/ncml/agg/dated/CG2006158_140000h_usfc.nc
data/ncml/agg/dated/CG2006158_150000h_usfc.nc
</pre>

Here's the NcML:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Test of joinNew aggregation using the scan element and dateFormatMark">

<aggregation type="joinNew" dimName="fileTime">
<variableAgg name="CGusfc"/>
<scan location="data/ncml/agg/dated"
suffix=".nc"
subdirs="false"
regExp="^.*/CG[^/]*"
dateFormatMark="CG#yyyyDDD_HHmmss"
/>
</aggregation>

</netcdf>
</pre>

So here we joinNew on the new outer dimension ''fileTime''. The new coordinate variable '''fileTime'''[''fileTime''] for this dimension will be an Array of type String that will contain the parsed [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] timestamps we will extract from the matching filenames.

We have specified that we want only Netcdf files (suffix ".nc") which match the regular expression "^.*/CG[^/]*". This means match the start of the string, then any number of characters that end with a "/" (the path portion of the filename), then the letters "CG", then some number of characters that do ''not'' include the "/" character (which is what "[^/]*" means). Essentially, we want files whose basename (path stripped) start with "CG" and end with ".nc". We also do not want to recurse, but only look in the location directory "/data/ncml/agg/dated" for the files.

Finally, we specify the ''scan@dateFormatMark'' pattern to describe how to parse the filename into an ISO 8601 date. The ''dateFormatMark'' is processed as follows:

* Skip the ''number'' of characters prior to the "#" mark in the pattern while scanning the base filename (no path)
* Interpret the next characters of the file basename using the given SimpleDateFormat string
* Ignore any characters after the SDF portion of the filename (such as the suffix)

First, note that we '''do not match''' the characters in the dateFormatMark --- they are simply counted and skipped. So rather than "CG#" specifying the prefix before the SDF, we could have also used "XX#". This is why we must also use a regular expression to filter out files with other prefixes that we do not want in the aggregation. Note that the "#" is just a marker for the start of the SDF pattern and doesn't count as an actual character in the matching process.

Second, we specify the dateFormatMark (DFM) as the following SDF pattern: "yyyyDDD_HHmmss". This means that we use the four digit year, then the day of the year (a three digit number), then an underscore ("_") separator, then the 24 hour time as 6 digits. Let's take the basename of the first file as an example:

"CG2006158_120000h_usfc.nc"

We skip two characters due to the "CG#" in the DFM. Then we want to match the "yyyy" pattern for the year with: "2006".

We then match the day of the year as "DDD" which is "158", the 158th day of the year for 2006.

We then match the underscore character "_" which is only a separator.

Next, we match the 24 hour time "HHmmss" as 12:00:00 hours:mins:secs (i.e. noon).

Finally, any characters after the DFM are ignored, here "h_usfc.nc".

We see that the four dataset files are on the same day, but sampled each hour from noon to 3 pm.

These parsed timestamps are then converted to an ISO 8601 date string which is used as the value for the coordinate variable element corresponding to that aggregation member. The first file would thus have the time value "2006-06-07T12:00:00Z", which is 7 June 2006 at noon in the GMT timezone.

The matched files are then '''sorted using the ISO 8601 timestamp as the sort key''' and added to the aggregation in this order. Since ISO 8601 is designed such that lexicographic order is isomorphic to chronological order, this orders the datasets monotonically in time from past to future. This is different from the <scan> behavior ''without'' a dateFormatMark specified, where files are ordered lexicographically (alphanumerically by full pathname) --- this order may or may not match chronological order.

If we project out the ASCII dods response for the new coordinate variable, we see all of the parsed timestamps and that they are in chronological order:

<pre>
String fileTime[fileTime = 4] = {"2006-06-07T12:00:00Z",
"2006-06-07T13:00:00Z",
"2006-06-07T14:00:00Z",
"2006-06-07T15:00:00Z"};
</pre>

We also check the resulting DDS to see that it is added as a map vector to the Grid as well:

<pre>
Dataset {
Grid {
Array:
Float32 CGusfc[fileTime = 4][time = 1][altitude = 1][lat = 29][lon = 26]
;
Maps:
String fileTime[fileTime = 4];
Float64 time[time = 1];
Float32 altitude[altitude = 1];
Float32 lat[lat = 29];
Float32 lon[lon = 26];
} CGusfc;
String fileTime[fileTime = 4];
} joinNew_scan_dfm.ncml;
</pre>

Finally, we look at the DAS with global metadata removed:

<pre>
Attributes {
CGusfc {
Float32 _FillValue -1.000000033e+32;
Float32 missing_value -1.000000033e+32;
Int32 numberOfObservations 303;
Float32 actual_range -0.2876400054, 0.2763200104;
fileTime {
---> String _CoordinateAxisType "Time";
}
CGusfc {
}
time {
String long_name "End Time";
String standard_name "time";
String units "seconds since 1970-01-01T00:00:00Z";
Float64 actual_range 1149681600.0000000, 1149681600.0000000;
}
altitude {
String long_name "Altitude";
String standard_name "altitude";
String units "m";
Float32 actual_range 0.000000000, 0.000000000;
}
lat {
String long_name "Latitude";
String standard_name "latitude";
String units "degrees_north";
String point_spacing "even";
Float32 actual_range 37.26869965, 38.02470016;
String coordsys "geographic";
}
lon {
String long_name "Longitude";
String standard_name "longitude";
String units "degrees_east";
String point_spacing "even";
Float32 actual_range 236.5800018, 237.4799957;
String coordsys "geographic";
}
}
fileTime {
---> String _CoordinateAxisType "Time";
}
}
</pre>

We see that the aggregation has also automatically added the "_CoordinateAxisType" attribute and set it to "Time" (denoted by the "-->") as defined by the NcML 2.2 specification. The author may add other metadata to the new coordinate variable as discussed previously.

=== Order of Inclusion ===

In cases where a dateFormatMark is ''not'' specified, the member datasets are added to the aggregation in alphabetical order ''on the full pathname''. This is important in the case of subdirectories since the path of the subdirectory is taken into account in the sort.

In cases where a dateFormatMark ''is'' specified, the extracted ISO 8601 timestamp is used as the sorting criterion, with older files being added before newer files.

Dynamic Aggregation Tutorial

2010-04-20T19:22:54Z

Mjohnson: New page: === Location (Location Location...) === The most important attribute of the scan element is the ''scan@location'' element that specifies the top-level search directory for the scan, relat...

=== Location (Location Location...) ===

The most important attribute of the scan element is the ''scan@location'' element that specifies the top-level search directory for the scan, relative to the BES data root directory specified in the BES configuration.

'''IMPORTANT''': ''ALL'' locations are interpreted relative to the BES root directory and '''NOT''' relative to the location of the NcML file itself! This means that all data to be aggregated '''must''' be in a subdirectory of the BES root data directory and that these directories ''must'' be specified fully, not relative to the NcML file.

For example, if the BES root data dir is "/usr/local/share/hyrax", let ${BES_DATA_ROOT} refer to this location. If the NcML aggregation file is in "${BES_DATA_ROOT}/data/ncml/myAgg.ncml" and the aggregation member datasets are in "${BES_DATA_ROOT}/data/hdf4/myAggDatasets", then the location in the NcML file for the aggregation data directory would be:

<pre>
<scan location="data/hdf4/myAggDatasets"/>
</pre>

which specifies the data directory relative to the BES data root as required.

Again, due to security reasons, the data is always searched under the BES data root. Trying to specify an absolute filesystem path, such as:

<pre>
<scan location="/usr/local/share/data"/>
</pre>

will ''NOT'' work. This directory will also be assumed to be a subdirectory of the ${BES_DATA_ROOT}, regardless of the preceding "/" character.

=== Suffix Criterion ===

The simplest criterion is to match only files of a certain datatype in a given directory. This is useful for filtering out text files and other files that may exist in the directory but which do not form part of the aggregation data.

Here's a simple example:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf"/>
</aggregation>

</netcdf>
</pre>

Assuming that the specified location "data/ncml/agg/grids" contains no subdirectories, this NcML will return all files in that directory that end in ".hdf" in alphanumerical order. In the case of our installed example data, there are four HDF4 files in that directory:

<pre>
data/ncml/agg/grids/f97182070958.hdf
data/ncml/agg/grids/f97182183448.hdf
data/ncml/agg/grids/f97183065853.hdf
data/ncml/agg/grids/f97183182355.hdf
</pre>

These will be included in alphanumerical order, so the scan element will in effect be equivalent to the following list of <netcdf> elements:

<pre>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</pre>

By default, scan will search subdirectories, which is why we mentioned "grids has no subdirectories". We discuss this in the next section.

=== Subdirectory Searching (The Default!) ===

If the author specifies the ''scan@subdirs'' attribute to the value "true" (which is the default!), then the criteria will be applied recursively to any subdirectories of the ''scan@location'' base scan directory as well as to any regular files in the base directory.

For example, continuing our previous example, but giving a higher level location:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan location="data/ncml/agg/" suffix=".hdf" subdirs="true"/>
</aggregation>

</netcdf>
</pre>

Assuming that only the "grids" subdir of "/data/ncml/agg" contains HDF4 files with that extension, the same aggregation as prior will be created, i.e. an aggregation isomorphic to:

<pre>
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/>
<netcdf location="data/ncml/agg/grids/f97182183448.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183065853.hdf"/>
<netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
</pre>

The ''scan@subdirs'' attribute is much for useful for turning off the default recursion. For example, if recursion is '''NOT''' desired, but only files with the given suffix in the given directory are required, the following will do that:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false"/>
</aggregation>
</pre>

=== OlderThan Criterion ===

The ''scan@olderThan'' attribute can be used to filter out files that are "too new". This feature is useful for excluding partial files currently being written by a daemon process, for example.

The value of the attribute is a duration specified by a number followed by a basic time unit. The time units recognized are:

* '''seconds''': { s, sec, secs, second, seconds }
* '''minutes''': { m, min, mins, minute, minutes }
* '''hours''': { h, hour, hours }
* '''days''': { day, days }
* '''months''': { month, months }
* '''years''': { year, years }

The strings inside { } are all recognized as referring to the given time unit.

For example, if we are following our previous example, but we suspect a new HDF file may be written at any time and usually takes 5 minutes to do so, we might use the following NcML:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false" olderThan="10 mins"/>
</aggregation>

</netcdf>
</pre>

Assuming the file will always be written withing 10 minutes, this files does what we wish. Only files whose modification date is older than the given duration from the current system time are included.

'''NOTE''' that the modification date of the file, not the creation date, is used for the test.

=== Regular Expression Criterion ===

The ''scan@regExp'' attribute may be used for more complicated filename matching tests where data for multiple variables, for example, may live in the same directory by whose filenames can be used to distinguish which are desired in the aggregation. Additionally, since the pathname including the location is used for the test, a regular expression test may be used in conjunction with a recursive directory search to find files in subdirectories where the directory name itself is specified in the regular expression, not just the filename. We'll give examples of both of these cases.

We also reiterate that this test is used ''in conjunction'' with any other tests --- the author may also include a suffix and an olderThan test if they wish. All criteria must match for the file to be included in the aggregation.

We recognize the POSIX regular expression syntax. For more information on regular expressions and the POSIX syntax, please see: http://en.wikipedia.org/wiki/Regular_expression.

We will give two basic examples:

* Finding all subdirectories with a given name
* Matching a filename starting with a certain substring

==== Matching a Subdirectory Name ====
Here's an example where we use a subdirectory search to find ".hdf" files in all subdirectories named "grids":

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan
location="data/"
subdirs="true"
regExp="^.*/grids/.+\.hdf$"
/>
</aggregation>
</netcdf>
</pre>

The regular expression here is "^.*/grids/.+\/hdf". Let's pull it apart quickly (this is not intended to be a regular expression tutorial):

The "^" matching the beginning of the string, so starts at the beginning of the location pathname. (without this we can match substrings in the middle of strings, etc)

We then match ".*" meaning 0 or more of any character.

We then match the "/grids/" string explicitly, meaning we want all pathnames that contain "/grids/" as a subdirectory.

We then match ".+" meaning 1 or more of any character.

We then match "\." meaning a literal "." character (the backslash "escapes" it).

We then match the suffix "hdf".

Finally, we match "$" meaning the end of the string.

So ultimately, this regular expression finds all filenames ending in ".hdf" that exist in some subdirectory named "grids" of the top-level location.

In following with our previous example, if there was only the one "grids" subdirectory in the ${BES_DATA_ROOT} with our four familiar files, we'd get the same aggregation as before.

==== Matching a partial filename ====

Let's say we have a given directory full of data files whose filename prefix specifies which variable they refer to. For example, let's say our "grids" directory has files that start with "grad" as well as the files that start with "f" we have seen in our examples. We still want just the files starting with "f" to filter out the others. Here's an example for that:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>

<scan
location="data/"
subdirs="true"
regExp="^.*/grids/f.+\.hdf$"
/>
</aggregation>
</netcdf>
</pre>

Here we match all pathnames ending in "grids" and files that start with the letter "f" and end with ".hdf" as we desire.

=== Date Format Mark and Timestamp Extraction ===

This section shows how to use the ''scan@dateFormatMark'' attribute along with other search criteria in order to extract and sort datasets by a timestamp encoded in the filename. All that is required is that the timestamp be parseable by a pattern recognized by the Java language "SimpleDateFormat" class ([http://java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html java.text.SimpleDateFormat]), which has also been implemented in C++ in the [http://site.icu-project.org/ International Components for Unicode] library which we use.

We base this example from the Unidata site [http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html Aggregation Tutorial]. Here we have a directory with four files whose filenames contain a timestamp describable by a SimpleDataFormat (SDF) pattern. We will also use a regular expression criterion and suffix criterion in addition to the dateFormatMark since we have other files in the same directory and only wish to match those starting with the characters "CG" that have suffix ".nc".

Here's the list of files (relative to the BES data root dir):

<pre>
data/ncml/agg/dated/CG2006158_120000h_usfc.nc
data/ncml/agg/dated/CG2006158_130000h_usfc.nc
data/ncml/agg/dated/CG2006158_140000h_usfc.nc
data/ncml/agg/dated/CG2006158_150000h_usfc.nc
</pre>

Here's the NcML:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Test of joinNew aggregation using the scan element and dateFormatMark">

<aggregation type="joinNew" dimName="fileTime">
<variableAgg name="CGusfc"/>
<scan location="data/ncml/agg/dated"
suffix=".nc"
subdirs="false"
regExp="^.*/CG[^/]*"
dateFormatMark="CG#yyyyDDD_HHmmss"
/>
</aggregation>

</netcdf>
</pre>

So here we joinNew on the new outer dimension ''fileTime''. The new coordinate variable '''fileTime'''[''fileTime''] for this dimension will be an Array of type String that will contain the parsed [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] timestamps we will extract from the matching filenames.

We have specified that we want only Netcdf files (suffix ".nc") which match the regular expression "^.*/CG[^/]*". This means match the start of the string, then any number of characters that end with a "/" (the path portion of the filename), then the letters "CG", then some number of characters that do ''not'' include the "/" character (which is what "[^/]*" means). Essentially, we want files whose basename (path stripped) start with "CG" and end with ".nc". We also do not want to recurse, but only look in the location directory "/data/ncml/agg/dated" for the files.

Finally, we specify the ''scan@dateFormatMark'' pattern to describe how to parse the filename into an ISO 8601 date. The ''dateFormatMark'' is processed as follows:

* Skip the ''number'' of characters prior to the "#" mark in the pattern while scanning the base filename (no path)
* Interpret the next characters of the file basename using the given SimpleDateFormat string
* Ignore any characters after the SDF portion of the filename (such as the suffix)

First, note that we '''do not match''' the characters in the dateFormatMark --- they are simply counted and skipped. So rather than "CG#" specifying the prefix before the SDF, we could have also used "XX#". This is why we must also use a regular expression to filter out files with other prefixes that we do not want in the aggregation. Note that the "#" is just a marker for the start of the SDF pattern and doesn't count as an actual character in the matching process.

Second, we specify the dateFormatMark (DFM) as the following SDF pattern: "yyyyDDD_HHmmss". This means that we use the four digit year, then the day of the year (a three digit number), then an underscore ("_") separator, then the 24 hour time as 6 digits. Let's take the basename of the first file as an example:

"CG2006158_120000h_usfc.nc"

We skip two characters due to the "CG#" in the DFM. Then we want to match the "yyyy" pattern for the year with: "2006".

We then match the day of the year as "DDD" which is "158", the 158th day of the year for 2006.

We then match the underscore character "_" which is only a separator.

Next, we match the 24 hour time "HHmmss" as 12:00:00 hours:mins:secs (i.e. noon).

Finally, any characters after the DFM are ignored, here "h_usfc.nc".

We see that the four dataset files are on the same day, but sampled each hour from noon to 3 pm.

These parsed timestamps are then converted to an ISO 8601 date string which is used as the value for the coordinate variable element corresponding to that aggregation member. The first file would thus have the time value "2006-06-07T12:00:00Z", which is 7 June 2006 at noon in the GMT timezone.

The matched files are then '''sorted using the ISO 8601 timestamp as the sort key''' and added to the aggregation in this order. Since ISO 8601 is designed such that lexicographic order is isomorphic to chronological order, this orders the datasets monotonically in time from past to future. This is different from the <scan> behavior ''without'' a dateFormatMark specified, where files are ordered lexicographically (alphanumerically by full pathname) --- this order may or may not match chronological order.

If we project out the ASCII dods response for the new coordinate variable, we see all of the parsed timestamps and that they are in chronological order:

<pre>
String fileTime[fileTime = 4] = {"2006-06-07T12:00:00Z",
"2006-06-07T13:00:00Z",
"2006-06-07T14:00:00Z",
"2006-06-07T15:00:00Z"};
</pre>

We also check the resulting DDS to see that it is added as a map vector to the Grid as well:

<pre>
Dataset {
Grid {
Array:
Float32 CGusfc[fileTime = 4][time = 1][altitude = 1][lat = 29][lon = 26]
;
Maps:
String fileTime[fileTime = 4];
Float64 time[time = 1];
Float32 altitude[altitude = 1];
Float32 lat[lat = 29];
Float32 lon[lon = 26];
} CGusfc;
String fileTime[fileTime = 4];
} joinNew_scan_dfm.ncml;
</pre>

Finally, we look at the DAS with global metadata removed:

<pre>
Attributes {
CGusfc {
Float32 _FillValue -1.000000033e+32;
Float32 missing_value -1.000000033e+32;
Int32 numberOfObservations 303;
Float32 actual_range -0.2876400054, 0.2763200104;
fileTime {
---> String _CoordinateAxisType "Time";
}
CGusfc {
}
time {
String long_name "End Time";
String standard_name "time";
String units "seconds since 1970-01-01T00:00:00Z";
Float64 actual_range 1149681600.0000000, 1149681600.0000000;
}
altitude {
String long_name "Altitude";
String standard_name "altitude";
String units "m";
Float32 actual_range 0.000000000, 0.000000000;
}
lat {
String long_name "Latitude";
String standard_name "latitude";
String units "degrees_north";
String point_spacing "even";
Float32 actual_range 37.26869965, 38.02470016;
String coordsys "geographic";
}
lon {
String long_name "Longitude";
String standard_name "longitude";
String units "degrees_east";
String point_spacing "even";
Float32 actual_range 236.5800018, 237.4799957;
String coordsys "geographic";
}
}
fileTime {
---> String _CoordinateAxisType "Time";
}
}
</pre>

We see that the aggregation has also automatically added the "_CoordinateAxisType" attribute and set it to "Time" (denoted by the "-->") as defined by the NcML 2.2 specification. The author may add other metadata to the new coordinate variable as discussed previously.

=== Order of Inclusion ===

In cases where a dateFormatMark is ''not'' specified, the member datasets are added to the aggregation in alphabetical order ''on the full pathname''. This is important in the case of subdirectories since the path of the subdirectory is taken into account in the sort.

In cases where a dateFormatMark ''is'' specified, the extracted ISO 8601 timestamp is used as the sorting criterion, with older files being added before newer files.