Aggregation enhancements - Revision history

Jimg: /* Performance and Implementation */

2016-07-30T19:40:55Z

Performance and Implementation

← Older revision		Revision as of 19:40, 30 July 2016
Line 75:		Line 75:
	;tabular: building a DAP Sequence from ''N'' arrays, where when ''N'' > 1, each array must be a member of the same DAP4 'coverage'		;tabular: building a DAP Sequence from ''N'' arrays, where when ''N'' > 1, each array must be a member of the same DAP4 'coverage'
	It is possible to access the essential functionality of the Aggregation Service using these functions.		It is possible to access the essential functionality of the Aggregation Service using these functions.

			===== Design =====
			The design of the [[Use_cases_for_swath_and_time_series_aggregation \| Aggregation Service]] is documented as well, although some aspects of that document are old and incorrect. it may also be useful to look at the source code, which can be found on GitHub at [https://github.com/opendap/olfs olfs] and [https://github.com/opendap/bes bes] in the [https://github.com/OPENDAP/olfs/tree/master/src/opendap/aggregation aggregation] and [https://github.com/OPENDAP/bes/tree/master/functions functions] parts of those repos, respectively.

	=== Examples ===		=== Examples ===

Jimg: /* Returning an Archive */

2016-07-29T23:35:15Z

Returning an Archive

← Older revision		Revision as of 23:35, 29 July 2016
Line 104:		Line 104:

	The example files are also available here:		The example files are also available here:
	[http://docs.opendap.org/index.php/File:D1_netcdf3_variable_subset.txt d1_netcdf3_variable_subset.txt]		*[http://docs.opendap.org/index.php/File:D1_netcdf3_variable_subset.txt d1_netcdf3_variable_subset.txt]
			*[http://docs.opendap.org/images/1/1e/D2_netcdf3_bbox_subset.txt d2_netcdf3_bbox_subset.txt]
			*[http://docs.opendap.org/images/4/46/D3_csv_subset.txt d3_csv_subset.txt]
			*[http://docs.opendap.org/images/8/88/D4_csv_subset_dim.txt d4_csv_subset_dim.txt]

	In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:		In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:

Jimg: /* Returning an Archive */

2016-07-29T23:30:38Z

Returning an Archive

← Older revision		Revision as of 23:30, 29 July 2016
Line 101:		Line 101:
	===== Returning an Archive =====		===== Returning an Archive =====

	'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to //resources/aggregation/tests/demo''		'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to ''resources/aggregation/tests/demo''.

			The example files are also available here:
			[http://docs.opendap.org/index.php/File:D1_netcdf3_variable_subset.txt d1_netcdf3_variable_subset.txt]

	In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:		In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:

Jimg: /* Returning an Archive */

2016-07-29T23:25:49Z

Returning an Archive

← Older revision		Revision as of 23:25, 29 July 2016
Line 101:		Line 101:
	===== Returning an Archive =====		===== Returning an Archive =====

	'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to resources/aggregation/tests/demo		'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to //resources/aggregation/tests/demo''

	In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:		In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:

Jimg: /* Returning an Archive */

2016-07-29T23:20:17Z

Returning an Archive

← Older revision		Revision as of 23:20, 29 July 2016
Line 112:		Line 112:
	&file=/data/modis/MOD04_L2.A2015021.0030.051.NRT.hdf		&file=/data/modis/MOD04_L2.A2015021.0030.051.NRT.hdf
	</source>		</source>

			This example shows how the DAP2 projection constraint can be given once and applied to a number of files. It's also possible to provide a unique constraint for each file.

	Each of the parameters begins with an ampersand (''&''). This command, which will be sent to the service using POST, specifies the ''netcdf3'' response, three files, and the DAP projection constraint ''Latitude,Longitude,Optical_Depth_Land_And_Ocean''. It may be that the parameter name ''&var'' is a bit misleading since you can actually provide array subsetting there as well (but not the filtering-type DAP2/DAP4 constraints).		Each of the parameters begins with an ampersand (''&''). This command, which will be sent to the service using POST, specifies the ''netcdf3'' response, three files, and the DAP projection constraint ''Latitude,Longitude,Optical_Depth_Land_And_Ocean''. It may be that the parameter name ''&var'' is a bit misleading since you can actually provide array subsetting there as well (but not the filtering-type DAP2/DAP4 constraints).

Jimg at 23:18, 29 July 2016

2016-07-29T23:18:17Z

← Older revision		Revision as of 23:18, 29 July 2016
Line 173:		Line 173:
	...		...
	</source>		</source>

			=== Potential Extensions to the Service ===
			This service was purpose-built for the NASA CMR system, but it could be extended in several useful ways.
			* Support general DAP2 and DAP4 constraint expressions, including function calls (functions are used behind the scenes already)
			* Increased parallelism.
			* Support for the '''tar.gz''' return type.

Jimg at 23:14, 29 July 2016

2016-07-29T23:14:29Z

← Older revision		Revision as of 23:14, 29 July 2016
Line 173:		Line 173:
	...		...
	</source>		</source>

	~~===== Returning a Table =====~~

Jimg: /* Returning an Archive */

2016-07-29T23:14:10Z

Returning an Archive

@@ Line 132: / Line 132: @@
      testing: MOD04_L2.A2015021.0030.051.NRT.hdf.nc   OK
 No errors detected in compressed data of d1.zip.
 </source>
 ===== Returning a Table =====

Jimg: /* Returning an Archive */

2016-07-29T23:07:56Z

Returning an Archive

← Older revision		Revision as of 23:07, 29 July 2016
Line 103:		Line 103:
	'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to resources/aggregation/tests/demo		'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to resources/aggregation/tests/demo

			In the OLFS repo on github, you'll see a file named ''resources/aggregation/tests/demo/short_names/d1_netcdf3_variable_subset.txt''. Here's what it looks like:
	<source lang="bash">		<source lang="bash">
	edamame:demo jimg$ more short_names/d1_netcdf3_variable_subset.txt		edamame:demo jimg$ more short_names/d1_netcdf3_variable_subset.txt
Line 112:		Line 113:
	</source>		</source>

			Each of the parameters begins with an ampersand (''&''). This command, which will be sent to the service using POST, specifies the ''netcdf3'' response, three files, and the DAP projection constraint ''Latitude,Longitude,Optical_Depth_Land_And_Ocean''. It may be that the parameter name ''&var'' is a bit misleading since you can actually provide array subsetting there as well (but not the filtering-type DAP2/DAP4 constraints).

			To send this command to the service, use ''curl'' like this:
	<source lang="bash">		<source lang="bash">
	edamame:demo jimg$ curl -X POST -d @short_names/d1_netcdf3_variable_subset.txt http://test.opendap.org/opendap/aggregation > d1.zip		edamame:demo jimg$ curl -X POST -d @short_names/d1_netcdf3_variable_subset.txt http://test.opendap.org/opendap/aggregation > d1.zip
Line 119:		Line 123:
	</source>		</source>

			The output of ''curl'' is redirected to a file (''d1.zip'') and we can list its contents
			verifying that it contains the files we expect.
	<source lang="bash">		<source lang="bash">
	edamame:demo jimg$ unzip -t d1.zip		edamame:demo jimg$ unzip -t d1.zip

Jimg: /* Returning an Archive */

2016-07-29T22:59:04Z

Returning an Archive

← Older revision		Revision as of 22:59, 29 July 2016
Line 100:		Line 100:

	===== Returning an Archive =====		===== Returning an Archive =====

			'''NB:''' To get these examples, clone https://github.com/opendap/olfs, then cd to resources/aggregation/tests/demo

			<source lang="bash">
			edamame:demo jimg$ more short_names/d1_netcdf3_variable_subset.txt
			&operation=netcdf3
			&var=Latitude,Longitude,Optical_Depth_Land_And_Ocean
			&file=/data/modis/MOD04_L2.A2015021.0020.051.NRT.hdf
			&file=/data/modis/MOD04_L2.A2015021.0025.051.NRT.hdf
			&file=/data/modis/MOD04_L2.A2015021.0030.051.NRT.hdf
			</source>

			<source lang="bash">
			edamame:demo jimg$ curl -X POST -d @short_names/d1_netcdf3_variable_subset.txt http://test.opendap.org/opendap/aggregation > d1.zip
			% Total % Received % Xferd Average Speed Time Time Time Current
			Dload Upload Total Spent Left Speed
			100 552k 0 552k 100 226 305k 124 0:00:01 0:00:01 --:--:-- 305k
			</source>

			<source lang="bash">
			edamame:demo jimg$ unzip -t d1.zip
			Archive: d1.zip
			testing: MOD04_L2.A2015021.0020.051.NRT.hdf.nc OK
			testing: MOD04_L2.A2015021.0025.051.NRT.hdf.nc OK
			testing: MOD04_L2.A2015021.0030.051.NRT.hdf.nc OK
			No errors detected in compressed data of d1.zip.
			</source>

	===== Returning a Table =====		===== Returning a Table =====