NCML Module Aggregation JoinExisting - Revision history

Ndp: /* New (Under Construction) */

2015-11-18T22:14:48Z

New (Under Construction)

← Older revision		Revision as of 22:14, 18 November 2015
Line 206:		Line 206:

	----		----

	~~== New (Under Construction) ==~~
	~~<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>~~

Ndp: /* NcML Dimension Cache */

2015-11-18T22:14:32Z

NcML Dimension Cache

← Older revision		Revision as of 22:14, 18 November 2015
Line 114:		Line 114:
	==== NcML Dimension Cache ====		==== NcML Dimension Cache ====

	If the scan element is used without the ''ncoords'' extension (see below), then the first time a joinExisiting aggregation is accessed (say by requesting it's DDS) the BES process will open '''every''' file in the aggregation and cache its dimension information in the ~~''ncml_dimension_cache''~~. By default the cache ~~is located in~~ <tt>/tmp</tt> and is limited to a maximum size of 2GB. ~~This~~ can be ~~controlled by the user~~ by modifying the ncml.conf file, typically located in <tt>/etc/bes/modules/ncml.conf</tt>:		If the scan element is used without the ''ncoords'' extension (see below), then the first time a joinExisiting aggregation is accessed (say by requesting it's DDS) the BES process will open '''every''' file in the aggregation and cache its dimension information in the NcML dimension cache. By default the cache files are written into <tt>/tmp</tt> and the total size of the cache is limited to a maximum size of 2GB. These settings can be changed by modifying the <tt>ncml.conf file</tt>, typically located in <tt>/etc/bes/modules/ncml.conf</tt>:
	<source lang="bash">		<source lang="bash">
	#-----------------------------------------------------------------------#		#-----------------------------------------------------------------------#
Line 133:		Line 133:
	# NCML.DimensionCache.maxDimensions=100		# NCML.DimensionCache.maxDimensions=100
	</source>		</source>
	The cache files are small compared to the source dataset files, typically less than 1kb for a ~~file~~ with a few named dimensions. However the cache files are numerous, ~~as in~~ one for each file used in a joinExisiting aggregation. If you have large joinExisiting aggregations it is important to be sure that the '''NCML.DimensionCache.directory''' ~~will have~~ space to contain the cache and ~~be sure to set~~ the '''NCML.DimensionCache.size''' to an appropriately large value.		The cache files are small compared to the source dataset files, typically less than 1kb for a dataset with a few named dimensions. However the cache files are numerous, one for each file used in a joinExisiting aggregation. If you have large joinExisiting aggregations it is important to be sure that the '''NCML.DimensionCache.directory''' has space to contain the cache and that the '''NCML.DimensionCache.size''' to an appropriately large value.

	Because the first access of the aggregation triggers the population of the NcML dimension cache for that aggregation this first access ~~may take a~~ significant ~~amount of time~~. It may be that typical HTTP clients will timeout before that requests completes. ~~And this may also indicate that the~~ dimension cache ~~does~~ not get fully populated, subsequent requests will cause the cache population to pick up where it was left off.		Because the first access of the aggregation triggers the population of the NcML dimension cache for that aggregation the time for this first access can be significant. It may be that typical HTTP clients will timeout before that requests completes. If a client timeout occurs dimension cache may not get fully populated, however subsequent requests will cause the cache population to pick up where it was left off.

	With only a modicum of effort one could write a shell program that utilizes the BES standalone functionality to pre-populate the dimension caches for large joinExisiting aggregations.		With only a modicum of effort one could write a shell program that utilizes the BES standalone functionality to pre-populate the dimension caches for large joinExisiting aggregations.

Ndp at 18:57, 18 November 2015

2015-11-18T18:57:58Z

@@ Line 107: / Line 107: @@
 Also notice that the map vector for the joined dimension, time, has been duplicated as a sibling of the dataset.  This is done automatically by the aggregation and it is copied into the actual map of the Grid.  This copy is to facilitate datasets which have multiple Grid's that are to be joined --- the top-level map vector is used as the canonical template map which is then copied into the maps for all the aggregated Grids.  In the case of the joined data being of type Array, this vector would already exist as the coordinate variable for the data matrix.  Since this is the source map for all aggregated Grid's, any attribute (metadata) changes should be made explicitly on this top-level coordinate variable so that the metadata is shared among all the aggregated Grid map vectors.
 === Using the Scan Element ===
-The collection of member datasets in the aggregation can be specified using the NcML ''scan'' element as described in the  [[Dynamic_Aggregation_Tutorial| dynamic aggregation tutorial]].
+The collection of member datasets in a joinExisiting aggregation can be specified using the NcML ''scan'' element as described in the  [[Dynamic_Aggregation_Tutorial| dynamic aggregation tutorial]].
 ==== NcML Dimension Cache ====
@@ Line 259: / Line 186: @@
 The advantage of this is that the server does not have to inspect all of the member granules to determine their dimensional size which allows server to manufacture responses much more quickly.

Ndp: /* Using the Scan Element */

2015-11-18T18:56:13Z

Using the Scan Element

@@ Line 183: / Line 183: @@
 <font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>
-The collection of member datasets in the aggregation can be specified using the [[Dynamic_Aggregation_Tutorial | NcML ''scan'' element]]
+The collection of member datasets in the aggregation can be specified using the NcML ''scan'' element as described in the  [[Dynamic_Aggregation_Tutorial| dynamic aggregation tutorial]].
 ==== ncoords extension ====

Ndp: /* Source of Data for Agrgegated Coordinate Variable on Join Dimension */

2015-10-08T20:37:51Z

Source of Data for Agrgegated Coordinate Variable on Join Dimension

← Older revision		Revision as of 20:37, 8 October 2015
Line 167:		Line 167:
	As we have seen, the most important limitation to the JoinExisting aggregation support is that the ''ncoords'' attribute should be specified for efficiency reasons. Future versions will continue to relax this requirement. The problem is that the size of the output join dimension is dependent on checking the DDS of ''every'' granule in the aggregation, which is computationally expensive for large aggregations.		As we have seen, the most important limitation to the JoinExisting aggregation support is that the ''ncoords'' attribute should be specified for efficiency reasons. Future versions will continue to relax this requirement. The problem is that the size of the output join dimension is dependent on checking the DDS of ''every'' granule in the aggregation, which is computationally expensive for large aggregations.

	=== Source of Data for ~~Agrgegated~~ Coordinate Variable on Join Dimension ===		=== Source of Data for Aggregated Coordinate Variable on Join Dimension ===

	This version does not allow the join dimension's data to be declared explicitly in the NcML as the NcML tutorial page describes. This version automatically aggregates '''all''' variables with the outer dimension matching the ''dimName''. This includes the coordinate variable (map vector in the case of Grid's) for the join dimension. These data cannot be overridden from those pulled from the files. Currently the TDS lists about 5 ways this data can be specified in addition to pulling them from the granules --- we only can pull them from granules now, which seems the most common use.		This version does not allow the join dimension's data to be declared explicitly in the NcML as the NcML tutorial page describes. This version automatically aggregates '''all''' variables with the outer dimension matching the ''dimName''. This includes the coordinate variable (map vector in the case of Grid's) for the join dimension. These data cannot be overridden from those pulled from the files. Currently the TDS lists about 5 ways this data can be specified in addition to pulling them from the granules --- we only can pull them from granules now, which seems the most common use.

Ndp: /* NEW UNDER CONSTRUCTION */

2015-10-08T19:09:45Z

NEW UNDER CONSTRUCTION

← Older revision		Revision as of 19:09, 8 October 2015
Line 178:		Line 178:
	----		----

	== ~~NEW UNDER CONSTRUCTION~~ ==		== New (Under Construction) ==

	=== Using the Scan Element ===		=== Using the Scan Element ===

Ndp: /* NEW UNDER CONSTRUCTION - Using the Scan Element */

2015-10-08T19:08:57Z

NEW UNDER CONSTRUCTION - Using the Scan Element

← Older revision		Revision as of 19:08, 8 October 2015
Line 178:		Line 178:
	----		----

	== NEW UNDER CONSTRUCTION - Using the Scan Element ==		== NEW UNDER CONSTRUCTION ==

			=== Using the Scan Element ===
	<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>		<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>

Ndp: /* NEW UNDER CONSTRUCTION - Using the Scan Element */

2015-10-08T19:08:35Z

NEW UNDER CONSTRUCTION - Using the Scan Element

← Older revision		Revision as of 19:08, 8 October 2015
Line 178:		Line 178:
	----		----

	=== NEW UNDER CONSTRUCTION - Using the Scan Element ===		== NEW UNDER CONSTRUCTION - Using the Scan Element ==
	<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>		<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>

Ndp: /* NEW UNDER CONSTRUCTION - Using the Scan Element */

2015-10-07T23:30:43Z

NEW UNDER CONSTRUCTION - Using the Scan Element

← Older revision		Revision as of 23:30, 7 October 2015
Line 232:		Line 232:
	</pre>		</pre>

	The advantage of this is that ~~there~~ server does not have to inspect all of the member granules to determine their dimensional size ~~and the result is the~~ server ~~can~~ manufacture responses much more quickly.		The advantage of this is that the server does not have to inspect all of the member granules to determine their dimensional size which allows server to manufacture responses much more quickly.

Ndp: /* NEW Using the Scan Element UNDER CONSTRUCTION */

2015-10-07T23:29:47Z

NEW Using the Scan Element UNDER CONSTRUCTION

← Older revision		Revision as of 23:29, 7 October 2015
Line 178:		Line 178:
	----		----

	=== NEW Using the Scan Element ~~UNDER CONSTRUCTION~~===		=== NEW UNDER CONSTRUCTION - Using the Scan Element ===
	<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>		<font size="3">'''''This section under construction in support of a pending release''''' Current instructions can be found in the previous section.</font>

	The collection of member datasets in the aggregation can be specified		The collection of member datasets in the aggregation can be specified using the [[Dynamic_Aggregation_Tutorial \| NcML ''scan'' element]]




	==== ncoords extension ====		==== ncoords extension ====

	~~Since~~ all of the granules are of uniform size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:		If all of the granules are of uniform dimensional size, we may also use the syntactic sugar provided by a Hyrax-specific extension to NcML -- adding the ''ncoords'' attribute to a '''scan''' element. The behavior of this extension is to set the ''ncoords'' for each granule matching the scan to be this value, as if the datasets were each listed explicitly with this value of the attribute. Here's an example of using the syntactic sugar that results in the same exact aggregation as the previous explicit one:

	<pre>		<pre>
Line 229:		Line 231:
	} mday_joinExist.ncml;		} mday_joinExist.ncml;
	</pre>		</pre>

			The advantage of this is that there server does not have to inspect all of the member granules to determine their dimensional size and the result is the server can manufacture responses much more quickly.