NCML Module Aggregation JoinNew
Join New Aggregation
A joinNew aggregation joins existing datasets along a new outer Array dimension. Essentially, it adds a new index to the existing variable which points into the values in each member dataset. One useful example of this aggregation is for joining multiple samples of data from different times into one virtual dataset containing all the times. We will first provide a basic introduction to the joinNew aggregation, then demonstrate examples for the various ways to specify the members datasets of an aggregation, the values for the new dimension's coordinate variable (map vector), and ways to specify metadata for this aggregation.
The reader is also directed to a basic tutorial of this NcML aggregation which may be found at http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html#joinNew.
PLEASE NOTE that our syntax is slightly different than that of the THREDDS Data Server (TDS), so please refer to this tutorial when using the Hyrax NcML Module! In particular, we do not process the <aggregation> element prior to other elements in the dataset, so in some cases the relative ordering of the <aggregation> and references to variables within the aggregation matters.
Introduction
A joinNew aggregation combines a variable with data across n datasets by creating a new outer dimension and placing the data from aggregation member i into the element i of the new outer dimension of size n. By "outer dimension" we mean a slowest varying dimension in a row major order flattening of the data (an example later will clarify this). For example, the array A[day][sample] would have the day dimension as the outer dimension. The data samples all must have the same data syntax; specifically the DDS of the variables must all match. For example, if the aggregation variable has name sample and is a 10x10 Array of float32, then all the member datasets in the aggregation must include a variable named sample which are all also 10x10 Arrays of float32. If there were 100 datasets specified in the aggregation, the resulting DDS would contain a variable named sample that was now of data shape 100x10x10.
In addition, a new coordinate variable specifying data values for the new dimension will be created at the same scope as (a sibling of) the specified aggregation variable. For example, if the new dimension is called "filename" and the new dimension's values are unspecified (the default), then an Array of type String will be created with one element for each member dataset --- the filename of the dataset. Additionally, if the aggregation variable was represented as a DAP Grid, this new dimension coordinate variable will also be added as a new Map vector inside the Grid to maintain the Grid specification.
There are multiple ways to specify the member datasets of a joinNew aggregation:
- Explicit: Specifying a separate <netcdf> element for each dataset
- Scan: scan a directory tree for files matching a conjunction of certain criteria:
- Specific suffix
- Older than a specific duration
- Matching a specific regular expression
- Either in a specific directory or recursively searching subdirectories
Additionally, there are multiple ways to specify the new coordinate variable's (the new outer dimension's associated data variable) data values:
- Default: An Array of type String containing the filenames of the member datasets
- Explicit Value Array: Explicit list of values of a specific data type, exactly one per dataset
- Dynamic Array: a numeric Array variable specified using start and increment values -- one value is generated automatically per dataset
- Timestamp from Filename: An Array of String with values of ISO 8601 Timestamps extracted from scanned dataset filenames using a specified Java SimpleDataFormat string. (Only works with <scan> element!)
A Simple Self-Contained Example
First, we start with a simple purely virtual (no external datasets) example to give the reader a basic idea of this aggregation. This example will join two one-dimensional Arrays of int's of length 5. The variable they describe will be called V. In this example, we assume we are joining samples of some variable V where each dataset is samples from 5 stations on a single day. We want to join the datasets so the new outer dimension is the day, resulting in a 2x5 array of int values for V.
Here's our NcML, with comments to describe what we are doing:
<?xml version="1.0" encoding="UTF-8"?>
<!-- A simple pure virtual joinNew aggregation of type Array<int>[5][2] -->
<netcdf title="Sample joinNew Aggregation on Pure NCML Datasets">
<!-- joinNew forming new outer dimension "day" -->
<aggregation type="joinNew" dimName="day">
<!-- For variables with this name in child datasets -->
<variableAgg name="V"/>
<!-- Datasets are one-dimensional Array<int> with cardinality 5. -->
<netcdf title="Sample Slice 1">
<!-- Must forward declare the dimension size -->
<dimension name="station" length="5"/>
<variable name="V" type="int" shape="station">
<values>1 3 5 7 9</values>
</variable>
</netcdf>
<!-- Second slice must match shape! -->
<netcdf title="Sample Slice 2">
<dimension name="station" length="5"/>
<variable name="V" type="int" shape="station">
<values>2 4 6 8 10</values>
</variable>
</netcdf>
</aggregation>
<!-- This will be what the expected output aggregation will look like.
We can use the named dimensions for the shape here since the aggregation
comes first and the dimensions will be added to the parent dataset by now -->
<variable name="V_expected" type="int" shape="day station">
<!-- Row major values. Since we create a new outer dimension, the slices are concatenated
since the outer dimension varies the slowest in row major order. This gives a 2x5 Array.
We use the newline to show the dimension separation for the reader's benefit -->
<values>
1 3 5 7 9
2 4 6 8 10
</values>
</variable>
</netcdf>
Notice we specify the name of the aggregation variable V inside the aggregation using a <variableAgg> element --- this allows to to specify multiple variables in the datasets to join. The new dimension, however, is specified by the attribute dimName of <aggregation>. We do NOT need to specify a <dimension> element for the new dimension (in fact, it would be an error to do so). Its size is calculated based on the number of datasets in the aggregation.
Running this file through the module produces the following DDS:
Dataset {
Int32 V[day = 2][station = 5];
Int32 V_expected[day = 2][station = 5];
String day[day = 2];
} joinNew_virtual.ncml;
Notice how the new dimension caused a coordinate variable to be created with the same name and shape as the new dimension. This array will contain the default values for the new outer dimension's map as we shall see if we ask for the ASCII version of the DODS (data) response:
The data:
Int32 V[day = 2][station = 5] = {{1, 3, 5, 7, 9},{2, 4, 6, 8, 10}};
Int32 V_expected[day = 2][station = 5] = {{1, 3, 5, 7, 9},{2, 4, 6, 8, 10}};
String day[day = 2] = {"Virtual_Dataset_0", "Virtual_Dataset_1"};
We see that the resulting aggregation data matches what we expected to create, specified by our V_expected variable. Also, notice that the values for the coordinate variable are "Virtual_Dataset_i", where i is the number of the dataset. Since the datasets did not have the location attribute set (which would have been used if it was), the module generates unique names for the virtual datasets in the output.
We could also have specified the value for the dataset using the netcdf@coordValue attribute:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Sample joinNew Aggregation on Pure NCML Datasets">
<aggregation type="joinNew" dimName="day">
<variableAgg name="V"/>
<netcdf title="Sample Slice 1" coordValue="100">
<dimension name="station" length="5"/>
<variable name="V" type="int" shape="station">
<values>1 3 5 7 9</values>
</variable>
</netcdf>
<netcdf title="Sample Slice 2" coordValue="107">
<dimension name="station" length="5"/>
<variable name="V" type="int" shape="station">
<values>2 4 6 8 10</values>
</variable>
</netcdf>
</aggregation>
</netcdf>
This results in the ASCII DODS of:
The data:
Int32 V[day = 2][station = 5] = {{1, 3, 5, 7, 9},{2, 4, 6, 8, 10}};
Float64 day[day = 2] = {100, 107};
Since the coordValue's could be parsed numerically, the coordinate variable is of type double (Float64). If they could not be parsed numerically, then the variable would be of type String.
Now that the reader has an idea of the basics of the joinNew aggregation, we will create examples for the many different use cases the NcML aggregation author may wish to create.
A Simple Example Using Explicit Dataset Files
Using virtual datasets is not that common. More commonly, the aggregation author wants to specify files for the aggregation. As an introductory example of this, we'll create a simple aggregation explicitly listing the files and giving string coordValue's. Note that this is a contrived example: we are using the same dataset file for each member, but changing the coordValue's. Also notice that we have specified that both the u and v variables be aggregated using the same new dimension name source.
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Aggregation with explicit string coordValue.">
<aggregation type="joinNew" dimName="source">
<variableAgg name="u"/>
<variableAgg name="v"/>
<!-- Same dataset a few times, but with different coordVal -->
<netcdf title="Dataset 1" location="data/ncml/fnoc1.nc" coordValue="Station_1"/>
<netcdf title="Dataset 2" location="data/ncml/fnoc1.nc" coordValue="Station_2"/>
<netcdf title="Dataset 3" location="data/ncml/fnoc1.nc" coordValue="Station_3"/>
</aggregation>
</netcdf>
which produces the DDS:
Dataset {
Int16 u[source = 3][time_a = 16][lat = 17][lon = 21];
Int16 v[source = 3][time_a = 16][lat = 17][lon = 21];
Float32 lat[lat = 17];
Float32 lon[lon = 21];
Float32 time[time = 16];
String source[source = 3];
} joinNew_string_coordVal.ncml;
Since there's so much data we only show the new coordinate variable:
String source[source = 3] = {"Station_1", "Station_2", "Station_3"};
Also notice that other coordinate variables (lat, lon, time) already existed in the datasets along with the u and v arrays. Any variable that is not aggregated over (specified as an aggregationVar) is explicitly union aggregated (please see NCML_Module_Aggregation_Union) into the resulting dataset --- the first instance of every variable found in the order the datasets are listed is used.
Now that we've seen simple cases, let's look at more complex examples.
Examples of Explicit Dataset Listings
In this section we will give several examples of joinNew aggregation with a static, explicit list of member datasets. In particular, we will go over examples of:
- Default values for the new coordinate variable
- Explicitly setting values of any type on the new coordinate variable
- Autogenerating uniform numeric values for the new coordinate variable
- Explicitly setting String or double values using the netcdf@coordValue attribute
There are several ways to specify values for the new coordinate variable of the new outer dimension. If String or double values are sufficient, the author may set the value for each listed dataset using the netcdf@coordValue attribute for each dataset. If another type is required for the new coordinate variable, then the author has a choice of specifying the entire new coordinate variable explicitly (which must match dimensionality of the aggregated dimension) or using the start/increment autogeneration <values> element for numeric, evenly spaced samples.
Please see the turotial: JoinNew Explicit Dataset Tutorial
Adding/Modifying Metadata on Aggregations
It is possible to add or modify metadata on existing or new variables in an aggregation. The syntax for these varies somewhat, so we give examples of the different cases.
We will also give examples of providing metadata:
- Adding/modifying metadata to the new coordinate variable
- Adding/modifying metadata to the aggregation variable itself
- Adding/modifying metadata to existing maps in an aggregated Grid
Please see the Metadata on Aggregations Tutorial.
Dynamic Aggregations Using Directory Scanning
A powerful way to create dynamic aggregations rather than by listing datasets explicitly is by specifying a data directory where aggregation member datasets are stored and some criteria for which files are to be added to the aggregation. These criteria will be combined in a conjunction (an AND operator) to handle various types of searches. The way to specify datasets in an aggregation is by using the <scan> element inside the <aggregation> element.
A key benefit of using the <scan> element is that the NcML file need not change as new datasets are added to the aggregation, say by an automated process which simply writes new data files into a specific directory. By properly specifying the NcML aggregation with a scan, the same NcML will refer to a dynamically changing aggregation, staying up to date with current data, without the need for modifications to the NcML file itself. If the filenames have a timestamp encoded in them, the use of the dateFormatMark allows for automatic creation of the new coordinate variable data values as well, as shown below.
The scan element may be used to search a directory to find files that match the following criteria:
- Suffix : the aggregated files end in a specific suffix, indicating the file type
- Subdirectories: any subdirectories of the given location are to be searched and all regular files tested against the criteria
- Older Than: the aggregated files must have been modified longer than some duration ago (to exclude files that may be currently being written)
- Reg Exp: the aggregated file pathnames must match a specific regular expression
- Date Format Mark: this highly useful criterion, useful in conjunction with others, allows the specification of a pattern in the filename which encodes a timestamp. The timestamp is extracted from the filenames using the pattern and is used to create ISO 8601 date elements for the new dimension's coordinate variable.
We will give examples of each of these criteria in use in our tutorial. Again, if more than one is specified, then ALL must match for the file to be included in the aggregation.
Please see Dynamic Aggregation Tutorial.
First, we discuss the required location attribute which specifies the (top-level) directory to be scanned with the criteria.
Location (Location Location...)
The most important attribute of the scan element is the scan@location element that specifies the top-level search directory for the scan, relative to the BES data root directory specified in the BES configuration.
IMPORTANT: ALL locations are interpreted relative to the BES root directory and NOT relative to the location of the NcML file itself! This means that all data to be aggregated must be in a subdirectory of the BES root data directory and that these directories must be specified fully, not relative to the NcML file.
For example, if the BES root data dir is "/usr/local/share/hyrax", let ${BES_DATA_ROOT} refer to this location. If the NcML aggregation file is in "${BES_DATA_ROOT}/data/ncml/myAgg.ncml" and the aggregation member datasets are in "${BES_DATA_ROOT}/data/hdf4/myAggDatasets", then the location in the NcML file for the aggregation data directory would be:
<scan location="data/hdf4/myAggDatasets"/>
which specifies the data directory relative to the BES data root as required.
Again, due to security reasons, the data is always searched under the BES data root. Trying to specify an absolute filesystem path, such as:
<scan location="/usr/local/share/data"/>
will NOT work. This directory will also be assumed to be a subdirectory of the ${BES_DATA_ROOT}, regardless of the preceding "/" character.
Suffix Criterion
The simplest criterion is to match only files of a certain datatype in a given directory. This is useful for filtering out text files and other files that may exist in the directory but which do not form part of the aggregation data.
Here's a simple example:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element.">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf"/>
</aggregation>
</netcdf>
Assuming that the specified location "data/ncml/agg/grids" contains no subdirectories, this NcML will return all files in that directory that end in ".hdf" in alphanumerical order. In the case of our installed example data, there are four HDF4 files in that directory:
data/ncml/agg/grids/f97182070958.hdf data/ncml/agg/grids/f97182183448.hdf data/ncml/agg/grids/f97183065853.hdf data/ncml/agg/grids/f97183182355.hdf
These will be included in alphanumerical order, so the scan element will in effect be equivalent to the following list of <netcdf> elements:
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/> <netcdf location="data/ncml/agg/grids/f97182183448.hdf"/> <netcdf location="data/ncml/agg/grids/f97183065853.hdf"/> <netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
By default, scan will search subdirectories, which is why we mentioned "grids has no subdirectories". We discuss this in the next section.
Subdirectory Searching (The Default!)
If the author specifies the scan@subdirs attribute to the value "true" (which is the default!), then the criteria will be applied recursively to any subdirectories of the scan@location base scan directory as well as to any regular files in the base directory.
For example, continuing our previous example, but giving a higher level location:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<!-- This will recurse into the "grids" subdir and grab all *.hdf files there -->
<scan location="data/ncml/agg/" suffix=".hdf" subdirs="true"/>
</aggregation>
</netcdf>
Assuming that only the "grids" subdir of "/data/ncml/agg" contains HDF4 files with that extension, the same aggregation as prior will be created, i.e. an aggregation isomorphic to:
<netcdf location="data/ncml/agg/grids/f97182070958.hdf"/> <netcdf location="data/ncml/agg/grids/f97182183448.hdf"/> <netcdf location="data/ncml/agg/grids/f97183065853.hdf"/> <netcdf location="data/ncml/agg/grids/f97183182355.hdf"/>
The scan@subdirs attribute is much for useful for turning off the default recursion. For example, if recursion is NOT desired, but only files with the given suffix in the given directory are required, the following will do that:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<!-- Find *.hdf files ONLY in the given location and NOT in subdirs -->
<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false"/>
</aggregation>
OlderThan Criterion
The scan@olderThan attribute can be used to filter out files that are "too new". This feature is useful for excluding partial files currently being written by a daemon process, for example.
The value of the attribute is a duration specified by a number followed by a basic time unit. The time units recognized are:
- seconds: { s, sec, secs, second, seconds }
- minutes: { m, min, mins, minute, minutes }
- hours: { h, hour, hours }
- days: { day, days }
- months: { month, months }
- years: { year, years }
The strings inside { } are all recognized as referring to the given time unit.
For example, if we are following our previous example, but we suspect a new HDF file may be written at any time and usually takes 5 minutes to do so, we might use the following NcML:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="joinNew Grid aggregation using the scan element.">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false" olderThan="10 mins"/>
</aggregation>
</netcdf>
Assuming the file will always be written withing 10 minutes, this files does what we wish. Only files whose modification date is older than the given duration from the current system time are included.
NOTE that the modification date of the file, not the creation date, is used for the test.
Regular Expression Criterion
The scan@regExp attribute may be used for more complicated filename matching tests where data for multiple variables, for example, may live in the same directory by whose filenames can be used to distinguish which are desired in the aggregation. Additionally, since the pathname including the location is used for the test, a regular expression test may be used in conjunction with a recursive directory search to find files in subdirectories where the directory name itself is specified in the regular expression, not just the filename. We'll give examples of both of these cases.
We also reiterate that this test is used in conjunction with any other tests --- the author may also include a suffix and an olderThan test if they wish. All criteria must match for the file to be included in the aggregation.
We recognize the POSIX regular expression syntax. For more information on regular expressions and the POSIX syntax, please see: http://en.wikipedia.org/wiki/Regular_expression.
We will give two basic examples:
- Finding all subdirectories with a given name
- Matching a filename starting with a certain substring
Matching a Subdirectory Name
Here's an example where we use a subdirectory search to find ".hdf" files in all subdirectories named "grids":
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<!-- Scan from the top for files in a grids subdir and ending in .hdf.
We are also using a regexp to match on a path that ends in "grids"
and has a suffix of hdf. -->
<scan
location="data/"
subdirs="true"
regExp="^.*/grids/.+\.hdf$"
/>
</aggregation>
</netcdf>
The regular expression here is "^.*/grids/.+\/hdf". Let's pull it apart quickly (this is not intended to be a regular expression tutorial):
The "^" matching the beginning of the string, so starts at the beginning of the location pathname. (without this we can match substrings in the middle of strings, etc)
We then match ".*" meaning 0 or more of any character.
We then match the "/grids/" string explicitly, meaning we want all pathnames that contain "/grids/" as a subdirectory.
We then match ".+" meaning 1 or more of any character.
We then match "\." meaning a literal "." character (the backslash "escapes" it).
We then match the suffix "hdf".
Finally, we match "$" meaning the end of the string.
So ultimately, this regular expression finds all filenames ending in ".hdf" that exist in some subdirectory named "grids" of the top-level location.
In following with our previous example, if there was only the one "grids" subdirectory in the ${BES_DATA_ROOT} with our four familiar files, we'd get the same aggregation as before.
Matching a partial filename
Let's say we have a given directory full of data files whose filename prefix specifies which variable they refer to. For example, let's say our "grids" directory has files that start with "grad" as well as the files that start with "f" we have seen in our examples. We still want just the files starting with "f" to filter out the others. Here's an example for that:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">
<aggregation type="joinNew" dimName="filename">
<variableAgg name="dsp_band_1"/>
<!-- Scan from the top for files in a grids subdir and starting with "f" and ending in ".hdf".
-->
<scan
location="data/"
subdirs="true"
regExp="^.*/grids/f.+\.hdf$"
/>
</aggregation>
</netcdf>
Here we match all pathnames ending in "grids" and files that start with the letter "f" and end with ".hdf" as we desire.
Date Format Mark and Timestamp Extraction
This section shows how to use the scan@dateFormatMark attribute along with other search criteria in order to extract and sort datasets by a timestamp encoded in the filename. All that is required is that the timestamp be parseable by a pattern recognized by the Java language "SimpleDateFormat" class (java.text.SimpleDateFormat), which has also been implemented in C++ in the International Components for Unicode library which we use.
We base this example from the Unidata site Aggregation Tutorial. Here we have a directory with four files whose filenames contain a timestamp describable by a SimpleDataFormat (SDF) pattern. We will also use a regular expression criterion and suffix criterion in addition to the dateFormatMark since we have other files in the same directory and only wish to match those starting with the characters "CG" that have suffix ".nc".
Here's the list of files (relative to the BES data root dir):
data/ncml/agg/dated/CG2006158_120000h_usfc.nc data/ncml/agg/dated/CG2006158_130000h_usfc.nc data/ncml/agg/dated/CG2006158_140000h_usfc.nc data/ncml/agg/dated/CG2006158_150000h_usfc.nc
Here's the NcML:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf title="Test of joinNew aggregation using the scan element and dateFormatMark">
<aggregation type="joinNew" dimName="fileTime">
<variableAgg name="CGusfc"/>
<scan location="data/ncml/agg/dated"
suffix=".nc"
subdirs="false"
regExp="^.*/CG[^/]*"
dateFormatMark="CG#yyyyDDD_HHmmss"
/>
</aggregation>
</netcdf>
So here we joinNew on the new outer dimension fileTime. The new coordinate variable fileTime[fileTime] for this dimension will be an Array of type String that will contain the parsed ISO 8601 timestamps we will extract from the matching filenames.
We have specified that we want only Netcdf files (suffix ".nc") which match the regular expression "^.*/CG[^/]*". This means match the start of the string, then any number of characters that end with a "/" (the path portion of the filename), then the letters "CG", then some number of characters that do not include the "/" character (which is what "[^/]*" means). Essentially, we want files whose basename (path stripped) start with "CG" and end with ".nc". We also do not want to recurse, but only look in the location directory "/data/ncml/agg/dated" for the files.
Finally, we specify the scan@dateFormatMark pattern to describe how to parse the filename into an ISO 8601 date. The dateFormatMark is processed as follows:
- Skip the number of characters prior to the "#" mark in the pattern while scanning the base filename (no path)
- Interpret the next characters of the file basename using the given SimpleDateFormat string
- Ignore any characters after the SDF portion of the filename (such as the suffix)
First, note that we do not match the characters in the dateFormatMark --- they are simply counted and skipped. So rather than "CG#" specifying the prefix before the SDF, we could have also used "XX#". This is why we must also use a regular expression to filter out files with other prefixes that we do not want in the aggregation. Note that the "#" is just a marker for the start of the SDF pattern and doesn't count as an actual character in the matching process.
Second, we specify the dateFormatMark (DFM) as the following SDF pattern: "yyyyDDD_HHmmss". This means that we use the four digit year, then the day of the year (a three digit number), then an underscore ("_") separator, then the 24 hour time as 6 digits. Let's take the basename of the first file as an example:
"CG2006158_120000h_usfc.nc"
We skip two characters due to the "CG#" in the DFM. Then we want to match the "yyyy" pattern for the year with: "2006".
We then match the day of the year as "DDD" which is "158", the 158th day of the year for 2006.
We then match the underscore character "_" which is only a separator.
Next, we match the 24 hour time "HHmmss" as 12:00:00 hours:mins:secs (i.e. noon).
Finally, any characters after the DFM are ignored, here "h_usfc.nc".
We see that the four dataset files are on the same day, but sampled each hour from noon to 3 pm.
These parsed timestamps are then converted to an ISO 8601 date string which is used as the value for the coordinate variable element corresponding to that aggregation member. The first file would thus have the time value "2006-06-07T12:00:00Z", which is 7 June 2006 at noon in the GMT timezone.
The matched files are then sorted using the ISO 8601 timestamp as the sort key and added to the aggregation in this order. Since ISO 8601 is designed such that lexicographic order is isomorphic to chronological order, this orders the datasets monotonically in time from past to future. This is different from the <scan> behavior without a dateFormatMark specified, where files are ordered lexicographically (alphanumerically by full pathname) --- this order may or may not match chronological order.
If we project out the ASCII dods response for the new coordinate variable, we see all of the parsed timestamps and that they are in chronological order:
String fileTime[fileTime = 4] = {"2006-06-07T12:00:00Z",
"2006-06-07T13:00:00Z",
"2006-06-07T14:00:00Z",
"2006-06-07T15:00:00Z"};
We also check the resulting DDS to see that it is added as a map vector to the Grid as well:
Dataset {
Grid {
Array:
Float32 CGusfc[fileTime = 4][time = 1][altitude = 1][lat = 29][lon = 26]
;
Maps:
String fileTime[fileTime = 4];
Float64 time[time = 1];
Float32 altitude[altitude = 1];
Float32 lat[lat = 29];
Float32 lon[lon = 26];
} CGusfc;
String fileTime[fileTime = 4];
} joinNew_scan_dfm.ncml;
Finally, we look at the DAS with global metadata removed:
Attributes {
CGusfc {
Float32 _FillValue -1.000000033e+32;
Float32 missing_value -1.000000033e+32;
Int32 numberOfObservations 303;
Float32 actual_range -0.2876400054, 0.2763200104;
fileTime {
---> String _CoordinateAxisType "Time";
}
CGusfc {
}
time {
String long_name "End Time";
String standard_name "time";
String units "seconds since 1970-01-01T00:00:00Z";
Float64 actual_range 1149681600.0000000, 1149681600.0000000;
}
altitude {
String long_name "Altitude";
String standard_name "altitude";
String units "m";
Float32 actual_range 0.000000000, 0.000000000;
}
lat {
String long_name "Latitude";
String standard_name "latitude";
String units "degrees_north";
String point_spacing "even";
Float32 actual_range 37.26869965, 38.02470016;
String coordsys "geographic";
}
lon {
String long_name "Longitude";
String standard_name "longitude";
String units "degrees_east";
String point_spacing "even";
Float32 actual_range 236.5800018, 237.4799957;
String coordsys "geographic";
}
}
fileTime {
---> String _CoordinateAxisType "Time";
}
}
We see that the aggregation has also automatically added the "_CoordinateAxisType" attribute and set it to "Time" (denoted by the "-->") as defined by the NcML 2.2 specification. The author may add other metadata to the new coordinate variable as discussed previously.
Order of Inclusion
In cases where a dateFormatMark is not specified, the member datasets are added to the aggregation in alphabetical order on the full pathname. This is important in the case of subdirectories since the path of the subdirectory is taken into account in the sort.
In cases where a dateFormatMark is specified, the extracted ISO 8601 timestamp is used as the sorting criterion, with older files being added before newer files.
