Dynamic Aggregation Tutorial - Revision history

Ndp: /* Overview */

2015-10-07T23:24:26Z

Overview

← Older revision		Revision as of 23:24, 7 October 2015
Line 1:		Line 1:
	=== Overview ===		=== Overview ===

	Dynamic aggregation is achieved through the use of the ''scan'' element		Dynamic aggregation is achieved through the use of the ''scan'' element.

			The NcML-2.2 ''scan'' element schema:
	<source lang="xml">		<source lang="xml">
	<xsd:element name="scan" minOccurs="0" maxOccurs="unbounded">		<xsd:element name="scan" minOccurs="0" maxOccurs="unbounded">
Line 15:		Line 17:
	</xsd:element>		</xsd:element>
	</source>		</source>

			This page discusses the use and significance of ''scan'' in creating dynamically aggregated datasets.

	=== Location (Location Location...) ===		=== Location (Location Location...) ===

Ndp at 23:22, 7 October 2015

2015-10-07T23:22:42Z

← Older revision		Revision as of 23:22, 7 October 2015
Line 1:		Line 1:
			=== Overview ===

			Dynamic aggregation is achieved through the use of the ''scan'' element
			<source lang="xml">
			<xsd:element name="scan" minOccurs="0" maxOccurs="unbounded">
			<xsd:complexType>
			<xsd:attribute name="location" type="xsd:string" use="required"/>
			<xsd:attribute name="regExp" type="xsd:string" />
			<xsd:attribute name="suffix" type="xsd:string" />
			<xsd:attribute name="subdirs" type="xsd:boolean" default="true"/>
			<xsd:attribute name="olderThan" type="xsd:string" />
			<xsd:attribute name="dateFormatMark" type="xsd:string" />
			<xsd:attribute name="enhance" type="xsd:string"/>
			</xsd:complexType>
			</xsd:element>
			</source>

	=== Location (Location Location...) ===		=== Location (Location Location...) ===
	[[Category:Aggregation]]		[[Category:Aggregation]]

Ndp: /* Date Format Mark and Timestamp Extraction */

2014-02-13T14:10:55Z

Date Format Mark and Timestamp Extraction

← Older revision		Revision as of 14:10, 13 February 2014
Line 219:		Line 219:
	Here's the NcML:		Here's the NcML:

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="Test of joinNew aggregation using the scan element and dateFormatMark">
	<netcdf title="Test of joinNew aggregation using the scan element and dateFormatMark">

	<aggregation type="joinNew" dimName="fileTime">		<aggregation type="joinNew" dimName="fileTime">
	<variableAgg name="CGusfc"/>		<variableAgg name="CGusfc"/>
	<scan location="data/ncml/agg/dated"		<scan
	suffix=".nc"		location="data/ncml/agg/dated"
	subdirs="false"		suffix=".nc"
	regExp="^./CG[^/]"		subdirs="false"
	dateFormatMark="CG#yyyyDDD_HHmmss"		regExp="^./CG[^/]"
	/>		'''dateFormatMark="CG#yyyyDDD_HHmmss"'''
			/>
	</aggregation>		</aggregation>

	</netcdf>		</netcdf>
	~~</pre>~~

	So here we joinNew on the new outer dimension ''fileTime''. The new coordinate variable '''fileTime'''[''fileTime''] for this dimension will be an Array of type String that will contain the parsed [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] timestamps we will extract from the matching filenames.		So here we joinNew on the new outer dimension ''fileTime''. The new coordinate variable '''fileTime'''[''fileTime''] for this dimension will be an Array of type String that will contain the parsed [http://en.wikipedia.org/wiki/ISO_8601 ISO 8601] timestamps we will extract from the matching filenames.

Ndp: /* Matching a partial filename */

2014-02-13T14:08:27Z

Matching a partial filename

← Older revision		Revision as of 14:08, 13 February 2014
Line 185:		Line 185:
	Let's say we have a given directory full of data files whose filename prefix specifies which variable they refer to. For example, let's say our "grids" directory has files that start with "grad" as well as the files that start with "f" we have seen in our examples. We still want just the files starting with "f" to filter out the others. Here's an example for that:		Let's say we have a given directory full of data files whose filename prefix specifies which variable they refer to. For example, let's say our "grids" directory has files that start with "grad" as well as the files that start with "f" we have seen in our examples. We still want just the files starting with "f" to filter out the others. Here's an example for that:

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">
	<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
Line 196:		Line 195:
	location="data/"		location="data/"
	subdirs="true"		subdirs="true"
	regExp="^.*/grids/f.+\.hdf$"		'''regExp="^.*/grids/f.+\.hdf$"'''
	/>		/>
	</aggregation>		</aggregation>
	</netcdf>		</netcdf>
	~~</pre~~>

	Here we match all pathnames ending in "grids" and files that start with the letter "f" and end with ".hdf" as we desire.		Here we match all pathnames ending in "grids" and files that start with the letter "f" and end with ".hdf" as we desire.

Ndp: /* Matching a Subdirectory Name */

2014-02-13T14:08:03Z

Matching a Subdirectory Name

← Older revision		Revision as of 14:08, 13 February 2014
Line 145:		Line 145:
	Here's an example where we use a subdirectory search to find ".hdf" files in all subdirectories named "grids":		Here's an example where we use a subdirectory search to find ".hdf" files in all subdirectories named "grids":

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">
	<netcdf title="Example of joinNew Grid aggregation using the scan element with a regexp">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
Line 157:		Line 156:
	location="data/"		location="data/"
	subdirs="true"		subdirs="true"
	regExp="^.*/grids/.+\.hdf$"		'''regExp="^.*/grids/.+\.hdf$"'''
	/>		/>
	</aggregation>		</aggregation>
	</netcdf>		</netcdf>
	~~</pre~~>

	The regular expression here is "^.*/grids/.+\/hdf". Let's pull it apart quickly (this is not intended to be a regular expression tutorial):		The regular expression here is "^.*/grids/.+\/hdf". Let's pull it apart quickly (this is not intended to be a regular expression tutorial):
Line 182:		Line 180:

	In following with our previous example, if there was only the one "grids" subdirectory in the ${BES_DATA_ROOT} with our four familiar files, we'd get the same aggregation as before.		In following with our previous example, if there was only the one "grids" subdirectory in the ${BES_DATA_ROOT} with our four familiar files, we'd get the same aggregation as before.


	==== Matching a partial filename ====		==== Matching a partial filename ====

Ndp: /* OlderThan Criterion */

2014-02-13T14:07:21Z

OlderThan Criterion

← Older revision		Revision as of 14:07, 13 February 2014
Line 112:		Line 112:
	For example, if we are following our previous example, but we suspect a new HDF file may be written at any time and usually takes 5 minutes to do so, we might use the following NcML:		For example, if we are following our previous example, but we suspect a new HDF file may be written at any time and usually takes 5 minutes to do so, we might use the following NcML:

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="joinNew Grid aggregation using the scan element.">
	<netcdf title="joinNew Grid aggregation using the scan element.">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
	<variableAgg name="dsp_band_1"/>		<variableAgg name="dsp_band_1"/>
	<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false" olderThan="10 mins"/>		<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false" '''olderThan="10 mins"''' />
	</aggregation>		</aggregation>

			</netcdf>

	~~</netcdf>~~
	~~</pre>~~

	Assuming the file will always be written withing 10 minutes, this files does what we wish. Only files whose modification date is older than the given duration from the current system time are included.		Assuming the file will always be written withing 10 minutes, this files does what we wish. Only files whose modification date is older than the given duration from the current system time are included.

	'''NOTE''' that the modification date of the file, not the creation date, is used for the test.		'''NOTE''' that the modification date of the file, not the creation date, is used for the test.

	=== Regular Expression Criterion ===		=== Regular Expression Criterion ===

Ndp: /* Suffix Criterion */

2014-02-13T14:06:48Z

Suffix Criterion

← Older revision		Revision as of 14:06, 13 February 2014
Line 25:		Line 25:
	Here's a simple example:		Here's a simple example:

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="Example of joinNew Grid aggregation using the scan element.">
	<netcdf title="Example of joinNew Grid aggregation using the scan element.">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
	<variableAgg name="dsp_band_1"/>		<variableAgg name="dsp_band_1"/>
	<scan location="data/ncml/agg/grids" suffix=".hdf"/>		<scan location="data/ncml/agg/grids" '''suffix=".hdf"''' />
	</aggregation>		</aggregation>

	</netcdf>		</netcdf>
	~~</pre~~>

	Assuming that the specified location "data/ncml/agg/grids" contains no subdirectories, this NcML will return all files in that directory that end in ".hdf" in alphanumerical order. In the case of our installed example data, there are four HDF4 files in that directory:		Assuming that the specified location "data/ncml/agg/grids" contains no subdirectories, this NcML will return all files in that directory that end in ".hdf" in alphanumerical order. In the case of our installed example data, there are four HDF4 files in that directory:
Line 56:		Line 54:

	By default, scan will search subdirectories, which is why we mentioned "grids has no subdirectories". We discuss this in the next section.		By default, scan will search subdirectories, which is why we mentioned "grids has no subdirectories". We discuss this in the next section.


	=== Subdirectory Searching (The Default!) ===		=== Subdirectory Searching (The Default!) ===

Ndp: /* Location (Location Location...) */

2014-02-13T14:06:14Z

Location (Location Location...)

← Older revision		Revision as of 14:06, 13 February 2014
Line 9:		Line 9:
	For example, if the BES root data dir is "/usr/local/share/hyrax", let ${BES_DATA_ROOT} refer to this location. If the NcML aggregation file is in "${BES_DATA_ROOT}/data/ncml/myAgg.ncml" and the aggregation member datasets are in "${BES_DATA_ROOT}/data/hdf4/myAggDatasets", then the location in the NcML file for the aggregation data directory would be:		For example, if the BES root data dir is "/usr/local/share/hyrax", let ${BES_DATA_ROOT} refer to this location. If the NcML aggregation file is in "${BES_DATA_ROOT}/data/ncml/myAgg.ncml" and the aggregation member datasets are in "${BES_DATA_ROOT}/data/hdf4/myAggDatasets", then the location in the NcML file for the aggregation data directory would be:

	~~<pre>~~		<scan '''location="data/hdf4/myAggDatasets"''' />
	<scan location="data/hdf4/myAggDatasets"/>
	~~</pre~~>

	which specifies the data directory relative to the BES data root as required.		which specifies the data directory relative to the BES data root as required.
Line 17:		Line 15:
	Again, due to security reasons, the data is always searched under the BES data root. Trying to specify an absolute filesystem path, such as:		Again, due to security reasons, the data is always searched under the BES data root. Trying to specify an absolute filesystem path, such as:

	~~<pre>~~		<scan '''location="/usr/local/share/data"''' />
	<scan location="/usr/local/share/data"/>
	~~</pre~~>

	will ''NOT'' work. This directory will also be assumed to be a subdirectory of the ${BES_DATA_ROOT}, regardless of the preceding "/" character.		will ''NOT'' work. This directory will also be assumed to be a subdirectory of the ${BES_DATA_ROOT}, regardless of the preceding "/" character.

Ndp: /* Subdirectory Searching (The Default!) */

2014-02-13T14:05:27Z

Subdirectory Searching (The Default!)

← Older revision		Revision as of 14:05, 13 February 2014
Line 68:		Line 68:
	For example, continuing our previous example, but giving a higher level location:		For example, continuing our previous example, but giving a higher level location:

	~~<pre>~~
	<?xml version="1.0" encoding="UTF-8"?>		<?xml version="1.0" encoding="UTF-8"?>
	<netcdf title="joinNew Grid aggregation using the scan element.">		<netcdf title="joinNew Grid aggregation using the scan element.">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
	<variableAgg name="dsp_band_1"/>		<variableAgg name="dsp_band_1"/>
	<!-- This will recurse into the "grids" subdir and grab all *.hdf files there -->		<!-- This will recurse into the "grids" subdir and grab all *.hdf files there -->
	<scan location="data/ncml/agg/" suffix=".hdf" subdirs="true"/>		<scan location="data/ncml/agg/" suffix=".hdf" '''subdirs="true"'''/>
	</aggregation>		</aggregation>

	</netcdf>		</netcdf>
	~~</pre>~~

	Assuming that only the "grids" subdir of "/data/ncml/agg" contains HDF4 files with that extension, the same aggregation as prior will be created, i.e. an aggregation isomorphic to:		Assuming that only the "grids" subdir of "/data/ncml/agg" contains HDF4 files with that extension, the same aggregation as prior will be created, i.e. an aggregation isomorphic to:
Line 92:		Line 92:
	The ''scan@subdirs'' attribute is much for useful for turning off the default recursion. For example, if recursion is '''NOT''' desired, but only files with the given suffix in the given directory are required, the following will do that:		The ''scan@subdirs'' attribute is much for useful for turning off the default recursion. For example, if recursion is '''NOT''' desired, but only files with the given suffix in the given directory are required, the following will do that:

	~~<pre>~~		<?xml version="1.0" encoding="UTF-8"?>
	<?xml version="1.0" encoding="UTF-8"?>		<netcdf title="joinNew Grid aggregation using the scan element.">
	<netcdf title="joinNew Grid aggregation using the scan element.">

	<aggregation type="joinNew" dimName="filename">		<aggregation type="joinNew" dimName="filename">
	<variableAgg name="dsp_band_1"/>		<variableAgg name="dsp_band_1"/>
	<!-- Find *.hdf files ONLY in the given location and NOT in subdirs -->		<!-- Find *.hdf files ONLY in the given location and NOT in subdirs -->
	<scan location="data/ncml/agg/grids" suffix=".hdf" subdirs="false"/>		<scan location="data/ncml/agg/grids" suffix=".hdf" '''subdirs="false"'''/>
	</aggregation>		</aggregation>
	</pre>		</pre>


	=== OlderThan Criterion ===		=== OlderThan Criterion ===

Mjohnson: /* Location (Location Location...) */

2010-04-20T19:30:01Z

Location (Location Location...)

← Older revision		Revision as of 19:30, 20 April 2010
Line 1:		Line 1:
	=== Location (Location Location...) ===		=== Location (Location Location...) ===
	[[Category:~~Aggregations~~]]		[[Category:Aggregation]]
			[[Category:NCML]]

	The most important attribute of the scan element is the ''scan@location'' element that specifies the top-level search directory for the scan, relative to the BES data root directory specified in the BES configuration.		The most important attribute of the scan element is the ''scan@location'' element that specifies the top-level search directory for the scan, relative to the BES data root directory specified in the BES configuration.