OPULS: UGrid Subsetting - Revision history

Ndp: /* Retired */

2015-12-07T18:25:03Z

Retired

← Older revision		Revision as of 18:25, 7 December 2015
Line 48:		Line 48:


	== Retired ==		== Retired Functions ==
	=== The ugr5 Function ===		=== The ugr5 Function ===
	'''''This function is deprecated.''' Supported by Hyrax 1.12.2 and earlier.''		'''''This function is deprecated.''' Supported by Hyrax 1.12.2 and earlier.''

Ndp: /* Background information on UGrid 0.9 */

2015-12-03T00:18:19Z

Background information on UGrid 0.9

← Older revision		Revision as of 00:18, 3 December 2015
Line 2:		Line 2:
	We have written server functions that utilize Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the Nx3 or 3xN). However, our function only supports triangular (2D) meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification. Our function supports both zero- and ones-based indexing for both the range and domain variables.		We have written server functions that utilize Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the Nx3 or 3xN). However, our function only supports triangular (2D) meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification. Our function supports both zero- and ones-based indexing for both the range and domain variables.

	== Background information on UGrid 0.9 ==		== Background information on UGrid 0.9.0 ==
	The [https://github.com/ugrid-conventions UGrid specification] moved to GitHub in 2013. The 0.9 version of the specification was last edited on 29 Oct 2013. Our function conforms to that specification and to an older version (0.6?), where the major difference is the organization of the mesh array data. For a triangular array, the 0.9 version specifies the array be a Nx3 while the older specification specified a 3xN. Our code supports both.		The [https://github.com/ugrid-conventions UGrid specification] moved to GitHub in 2013. The 0.9.0 version of the specification was last edited on 29 Oct 2013. Our function conforms to that specification and to an older version (0.6?), where the major difference is the organization of the mesh array data. For a triangular array, the 0.9.0 version specifies the array be a Nx3 while the older specification specified a 3xN. Our code supports both.

	== The ugrid subsetting functions ==		== The ugrid subsetting functions ==

Ndp at 00:16, 3 December 2015

2015-12-03T00:16:48Z

Ndp: /* Overview */

2015-12-02T00:48:14Z

Overview

← Older revision		Revision as of 00:48, 2 December 2015
Line 1:		Line 1:
	== Overview ==		== Overview ==
	We have written a server ~~function~~ that ~~uses~~ Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the Nx3 or 3xN). However, our function only supports triangular (2D) meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification. Our function supports both zero- and ones-based indexing for both the range and domain variables.		We have written server functions that utilize Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the Nx3 or 3xN). However, our function only supports triangular (2D) meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification. Our function supports both zero- and ones-based indexing for both the range and domain variables.

	== Background information on UGrid 0.9 ==		== Background information on UGrid 0.9 ==

Ndp: /* ugnr: Subset by node value. */

2015-11-30T22:44:56Z

ugnr: Subset by node value.

← Older revision		Revision as of 22:44, 30 November 2015
Line 21:		Line 21:
	; Subsetting the range variables		; Subsetting the range variables
	: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is		: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is
	::<tt>[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>		::<tt>[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>
	: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like		: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like
	::<tt>[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>		::<tt>[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>

	=== uger: Subset by edge value. ===		=== uger: Subset by edge value. ===

Ndp: /* The ugr5 Function */

2015-11-30T22:43:51Z

The ugr5 Function

@@ Line 5: / Line 5: @@
 The [https://github.com/ugrid-conventions UGrid specification] moved to GitHub in 2013. The 0.9 version of the specification was last edited on 29 Oct 2013. Our function conforms to that specification and to an older version (0.6?), where the major difference is the organization of the mesh array data. For a triangular array, the 0.9 version specifies the array be a Nx3 while the older specification specified a 3xN. Our code supports both.
-== The ugr5 Function ==
+== The ugrid subsetting functions ==
 ;Name
 :ugr5 - Restrict the domain of an unstructured 2D Mesh Topology grid

Jimg: /* Sample Data */

2015-05-16T00:01:40Z

Sample Data

← Older revision		Revision as of 00:01, 16 May 2015
Line 28:		Line 28:
	A server with unstructured grid data and subsetting function on board is here: http://54.172.39.65:8080/opendap/		A server with unstructured grid data and subsetting function on board is here: http://54.172.39.65:8080/opendap/

	In the directory [http://54.172.39.65:8080/opendap/ugrids/ ugrids] You can find a number of test datasets.		In the directory [http://54.172.39.65:8080/opendap/ugrids/ ugrids] You can find a number of test datasets:
			; fvcom_1step.nc, hex.nc
			: These are test data from Bill Howe. The ''hex'' file is small but it lacks the special variable the UGrid uses to encode metadata the function depends on to read the unstructured data.
			; test4.nc, test4-rename-nodedata.ncml, test4-ugrid.ncml
			: ''test4'' also lacks the special 'metadata variable' but we have written NcML (''test4-ugrid'') that adds it to the dataset, so you can use ugr5() with a URL that points at the virtual dataset that NcML file defines. The companion NcML file (''test4-rename-nodedata.ncml'') uses NcML to rename the node data variable and while this works fine for normal DAP accesses, the ugr5() won't work with this - the purpose of this example is to document a known bug where using NcML to rename a range variable breaks the function.
			; NECOFS_GOM3_FORECAST.nc, NECOFS_WAVE_FORECAST.nc
			: These files us a 3xN face node connectivity array (corresponding to the older version of the UGrid specification)
			; Ike/2D_varied_manning_windstress and Ike/2D_varied_manning_windstress_wave
			: These directories have data from c. 2008 that requires (minimal) NcML to work with our function
			; RENCI
			: Data used by the ODSIP EarthCube project. This directory has the same data (or nearly so) where the face node connectivity array is organized a Nx3 and 3xN.
			; lana
			: This is a bit of a mystery; it may have been lumped into the ugrid sample data by mistake.
			; README
			: A brief recap of this information

			<!--
	I adapted their original test data using ncml files. I wrote an ncml file for test4.nc ( called test4-ugrid.ncml in the same directory) that makes it compliant with the Deltares specification and it can be subset with the ugrid restrict function. http://54.172.39.65:8080/opendap/ugrids/test4-ugrid.ncml Also in that directory is the dataset (NECOFS_GOM3_FORECAST.nc) that Rich Signell provided http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc Which can be subset using the ugrid restrict function.		I adapted their original test data using ncml files. I wrote an ncml file for test4.nc ( called test4-ugrid.ncml in the same directory) that makes it compliant with the Deltares specification and it can be subset with the ugrid restrict function. http://54.172.39.65:8080/opendap/ugrids/test4-ugrid.ncml Also in that directory is the dataset (NECOFS_GOM3_FORECAST.nc) that Rich Signell provided http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc Which can be subset using the ugrid restrict function.

Line 36:		Line 51:
	The file test_dir-norename.ncml can be subset using the ugrid restrict function: http://54.172.39.65:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml The file test_dir.ncml is closer to the original ncml file 00_dir.ncml, unfortunately it only works superficially. You can get the metadata responses for this dataset, but there is a bug in the ncml code that prevents ugrid restrict function from returning data values for renamed variables. The test_dir-norename.ncml is simply test_dir.ncml without the renamed variables.		The file test_dir-norename.ncml can be subset using the ugrid restrict function: http://54.172.39.65:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml The file test_dir.ncml is closer to the original ncml file 00_dir.ncml, unfortunately it only works superficially. You can get the metadata responses for this dataset, but there is a bug in the ncml code that prevents ugrid restrict function from returning data values for renamed variables. The test_dir-norename.ncml is simply test_dir.ncml without the renamed variables.





	~~<!--~~
	=== Places for improvement ===		=== Places for improvement ===


	* Make a version in which only the lat and lon boundaries are specified that then proceeds to subset all of the nodes, edges, and faces.		* Make a version in which only the lat and lon boundaries are specified that then proceeds to subset all of the nodes, edges, and faces.
	*: See the "super restrict" function discussion below		*: See the "super restrict" function discussion below
Line 55:		Line 63:
	: where zeta is defined "Float32 zeta[time = 1081][node = 417642];"??		: where zeta is defined "Float32 zeta[time = 1081][node = 417642];"??
	: '''UPDATE: YES! This works now!'''		: '''UPDATE: YES! This works now!'''


	# Make individual functions that can be 'nested'		# Make individual functions that can be 'nested'

Jimg: /* The ugr5 Function */

2015-05-15T23:36:12Z

The ugr5 Function

← Older revision		Revision as of 23:36, 15 May 2015
Line 21:		Line 21:
	; Subsetting the range variables		; Subsetting the range variables
	: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is		: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is
	:: [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>~~http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?~~ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		:: [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]
	: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like		: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like
	::[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>~~http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?~~ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		::[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]

	== Sample Data ==		== Sample Data ==

Jimg: /* The ugr5 Function */

2015-05-15T23:35:42Z

The ugr5 Function

← Older revision		Revision as of 23:35, 15 May 2015
Line 20:		Line 20:

	; Subsetting the range variables		; Subsetting the range variables
	: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL ~~this would look like~~		: In the example dataset [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is
	:: [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		:: [http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]
	: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like		: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like
	::[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		::[http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>http://54.172.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]

	==~~= The~~ Data ===		== Sample Data ==
	A server with unstructured grid data and subsetting function on board is here: http://54.172.39.65:8080/opendap/		A server with unstructured grid data and subsetting function on board is here: http://54.172.39.65:8080/opendap/

Jimg: /* Overview */

2015-05-15T23:31:29Z

Overview

← Older revision		Revision as of 23:31, 15 May 2015
Line 1:		Line 1:
	== Overview ==		== Overview ==
	We have written a server function that uses Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the 3xN ~~or Nx3~~). However, our function only supports triangular meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification.		We have written a server function that uses Bill Howe's Gridfields library to subset unstructured grids (aka irregular meshes) that conform to the UGrid 0.9 specification that is part of the CF-1.6 specification. Our function has a little latitude in its requirements because it also supports features found in earlier versions of the UGrid specification (e.g., the 'mesh array' can be either the Nx3 or 3xN). However, our function only supports triangular (2D) meshes, not the other mesh schemes outlined in the specification. The function will subset one or more ''range variables'' based on the values of the ''domain variables'' as expressed in a simple relational expression. The result of the function is an unstructured grid - that is, the subsetting operation will return a a set of variables that could be written to a netcdf file and the result would conform to the UGrid 0.9 specification. Because the specification has evolved over time and older data are still perfectly valid, we often 'tweak' those older datasets using NcML so that the contain the variables and attributes required by the newer versions of the UGrid specification. Our function supports both zero- and ones-based indexing for both the range and domain variables.

	== Background information on UGrid 0.9 ==		== Background information on UGrid 0.9 ==

← Older revision		Revision as of 00:16, 3 December 2015
Line 17:		Line 17:
	:: <tt>ugnr(depth, "28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")</tt>		:: <tt>ugnr(depth, "28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")</tt>
	: Used in a url (that returns the DDS for the subset data):		: Used in a url (that returns the DDS for the subset data):
	:: <tt>[http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugnr(depth,%2228.0%3Clat&lat%3C29.0&-89.0%3Clon&lon%3C-88.0%22) http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugnr(depth,"28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")]</tt>		:: <tt>[http://52.70.199.67:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugnr(depth,%2228.0%3Clat&lat%3C29.0&-89.0%3Clon&lon%3C-88.0%22) http://52.70.199.67:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugnr(depth,"28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")]</tt>

	; Subsetting the range variables		; Subsetting the range variables
	: In the example dataset [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is		: In the example dataset [http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is
	::<tt>[http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>		::<tt>[http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>
	: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like		: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like
	::<tt>[http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>		::<tt>[http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugnr(salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugnr(salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]</tt>

	=== uger: Subset by edge value. ===		=== uger: Subset by edge value. ===
Line 62:		Line 62:
	:: ugr5(0, depth, "28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")		:: ugr5(0, depth, "28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")
	: Used in a url (that returns the DDS for the subset data):		: Used in a url (that returns the DDS for the subset data):
	:: [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugr5(0,depth,%2228.0%3Clat&lat%3C29.0&-89.0%3Clon&lon%3C-88.0%22) http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugr5(0,depth,"28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")]		:: [http://52.70.199.67:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugr5(0,depth,%2228.0%3Clat&lat%3C29.0&-89.0%3Clon&lon%3C-88.0%22) http://52.70.199.67:8080/opendap/hyrax/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml.dds?ugr5(0,depth,"28.0<lat & lat<29.0 & -89.0<lon & lon<-88.0")]

	; Subsetting the range variables		; Subsetting the range variables
	: In the example dataset [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is		: In the example dataset [http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc NECOFS_GOM3_FORECAST.nc] ([http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds DDS] [http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.das DAS]) we can see that there are range variables associated with the nodes of the mesh that have a ''time'' dimension and a ''siglay'' dimension. Since the ''time'' dimension has 145 values, and ''siglay'' dimension has 40 values it may be that you wish to only retrieve a single time slice and/or siglay slice. This can accomplished by using an array constraint on the variable as it is passed to the ugrid function. For example, to retrieve the 12th siglay slice of the 5th time slice of the variable salinity you would constrain it like ''salinity[4][11][*]'' (note that the indices begin at zero). In the request URL the function call is
	:: [http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		:: [http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B11%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][11][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]
	: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like		: It is also possible to request, say, every 5th time slice ''salinity[0:5:][][]'' or every 3rd siglay slice from 3 to 30 ''salinity[4][3:3:30][]''. In a request URL this would look like
	::[http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]		::[http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc.dds?ugr5(0,salinity%5B4%5D%5B3:3:30%5D%5B%5D,%2238.0%3Clat%20&%20lat%3C42.0%20&%20-64.0%3Clon%20&%20lon%3C-60.0%22) <nowiki>ugr5(0,salinity[4][3:3:30][],"38.0<lat & lat<42.0 & -64.0<lon & lon<-60.0")</nowiki>]

	== Sample Data ==		== Sample Data ==
	A server with unstructured grid data and subsetting function on board is here: http://54.~~172~~.39.65:8080/opendap/		A server with unstructured grid data and subsetting function on board is here: http://52.70.199.67:8080/opendap/

	In the directory [http://54.~~172~~.39.65:8080/opendap/ugrids/ ugrids] You can find a number of test datasets:		In the directory [http://52.70.199.67:8080/opendap/ugrids/ ugrids] You can find a number of test datasets:
	; fvcom_1step.nc, hex.nc		; fvcom_1step.nc, hex.nc
	: These are test data from Bill Howe. The ''hex'' file is small but it lacks the special variable the UGrid uses to encode metadata the function depends on to read the unstructured data.		: These are test data from Bill Howe. The ''hex'' file is small but it lacks the special variable the UGrid uses to encode metadata the function depends on to read the unstructured data.
Line 90:		Line 90:

	<!--		<!--
	I adapted their original test data using ncml files. I wrote an ncml file for test4.nc ( called test4-ugrid.ncml in the same directory) that makes it compliant with the Deltares specification and it can be subset with the ugrid restrict function. http://54.~~172~~.39.65:8080/opendap/ugrids/test4-ugrid.ncml Also in that directory is the dataset (NECOFS_GOM3_FORECAST.nc) that Rich Signell provided http://54.~~172~~.39.65:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc Which can be subset using the ugrid restrict function.		I adapted their original test data using ncml files. I wrote an ncml file for test4.nc ( called test4-ugrid.ncml in the same directory) that makes it compliant with the Deltares specification and it can be subset with the ugrid restrict function. http://52.70.199.67:8080/opendap/ugrids/test4-ugrid.ncml Also in that directory is the dataset (NECOFS_GOM3_FORECAST.nc) that Rich Signell provided http://52.70.199.67:8080/opendap/hyrax/ugrids/NECOFS_GOM3_FORECAST.nc Which can be subset using the ugrid restrict function.

	Also, a while back I tracked down and downloaded on of the larger, multi-time-step fvcom ugrid data files, it's now set up here: http://54.~~172~~.39.65:8080/opendap/ugrids/Ike/ Inside you'll find that the 2D models have useful stuff. In http://54.~~172~~.39.65:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/ You should see the original nmcl files from the source server. These weren't compatible with Hyrax - mostly the result of Hyrax not being as "elastic" as the TDS with its inputs.		Also, a while back I tracked down and downloaded on of the larger, multi-time-step fvcom ugrid data files, it's now set up here: http://52.70.199.67:8080/opendap/ugrids/Ike/ Inside you'll find that the 2D models have useful stuff. In http://52.70.199.67:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/ You should see the original nmcl files from the source server. These weren't compatible with Hyrax - mostly the result of Hyrax not being as "elastic" as the TDS with its inputs.

	The file test_dir-norename.ncml can be subset using the ugrid restrict function: http://54.~~172~~.39.65:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml The file test_dir.ncml is closer to the original ncml file 00_dir.ncml, unfortunately it only works superficially. You can get the metadata responses for this dataset, but there is a bug in the ncml code that prevents ugrid restrict function from returning data values for renamed variables. The test_dir-norename.ncml is simply test_dir.ncml without the renamed variables.		The file test_dir-norename.ncml can be subset using the ugrid restrict function: http://52.70.199.67:8080/opendap/ugrids/Ike/2D_varied_manning_windstress/test_dir-norename.ncml The file test_dir.ncml is closer to the original ncml file 00_dir.ncml, unfortunately it only works superficially. You can get the metadata responses for this dataset, but there is a bug in the ncml code that prevents ugrid restrict function from returning data values for renamed variables. The test_dir-norename.ncml is simply test_dir.ncml without the renamed variables.

	=== Places for improvement ===		=== Places for improvement ===