Expanded arguments for Constraint Expressions - Revision history

Jimg: /* Performance measurements */

2013-09-17T23:52:23Z

Performance measurements

Jimg: /* The bind_name() and bind_shape() functions */

2013-09-17T23:51:19Z

The bind_name() and bind_shape() functions

← Older revision		Revision as of 23:51, 17 September 2013
Line 149:		Line 149:

	==== The bind_name() and bind_shape() functions ====		==== The bind_name() and bind_shape() functions ====
	These functions take a BaseType* object and bind a name or shape to it (in the latter case the BaseType* must be an Array*). They are intended to be used with make_array() and the #type special forms, but they can be used with any variable in a dataset.		These functions take a BaseType* object and bind a name or shape to it (in the latter case the BaseType* must be an Array*). They are intended to be used with make_array() and the $type special forms, but they can be used with any variable in a dataset.

	;bind_name(<name>,<variable>): The <name> must not exist in the dataset; <variable> may be the name of a variable in the dataset (so this function can rename an existing variable) or it can be a variable returned by another function or special form.		;bind_name(<name>,<variable>): The <name> must not exist in the dataset; <variable> may be the name of a variable in the dataset (so this function can rename an existing variable) or it can be a variable returned by another function or special form.

Jimg: /* The array constant special forms */

2013-09-17T23:50:47Z

The array constant special forms

← Older revision		Revision as of 23:50, 17 September 2013
Line 142:		Line 142:
	These special forms can build vectors with specific values and return them as DAP2 Arrays. The Array variables can be named using the ''bind_name()'' function and have their shape set using ''bind_shape()''.		These special forms can build vectors with specific values and return them as DAP2 Arrays. The Array variables can be named using the ''bind_name()'' function and have their shape set using ''bind_shape()''.

	;<nowiki>#</nowiki><type>(<size hint>: <values>, ...): The ''#<type>'' (''#Byte'', ...) literal starts the special form. The first argument <size_hint> provides a way to preallocate the memory needed to hold the vector of values. Following that, the values are listed. Unlike ''make_array()'', it is not necessary to provide the exact size of the vector; the size hint is just that, a hint. If a size hint of zero is supplied, it will be ignored. Any of the DAP2 numeric types can be used with this special form. This is called a 'special form' because it invokes a custom parser that can process values very efficiently.		;<nowiki>$</nowiki><type>(<size hint>: <values>, ...): The ''$<type>'' (''$Byte'', ...) literal starts the special form. The first argument <size_hint> provides a way to preallocate the memory needed to hold the vector of values. Following that, the values are listed. Unlike ''make_array()'', it is not necessary to provide the exact size of the vector; the size hint is just that, a hint. If a size hint of zero is supplied, it will be ignored. Any of the DAP2 numeric types can be used with this special form. This is called a 'special form' because it invokes a custom parser that can process values very efficiently.

	Example: #Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5) will return a one dimensional (i.e., a vector) Array of Bytes with values 2, 3, ... . The vector is named ''g<int>'' just like the array returned by make_array(). The vector can be turned in to a N-dimensional Array using ''bind_shape()'' using ''bind_shape("[4][4]",#Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5)).		Example: $Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5) will return a one dimensional (i.e., a vector) Array of Bytes with values 2, 3, ... . The vector is named ''g<int>'' just like the array returned by make_array(). The vector can be turned in to a N-dimensional Array using ''bind_shape()'' using ''bind_shape("[4][4]",$Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5)).

	The special forms can make a 1,047,572 element vector on Int32 in 0.4 seconds.		The special forms can make a 1,047,572 element vector on Int32 in 0.4 seconds.

Jimg: /* OLFS */

2013-09-17T22:47:24Z

OLFS

← Older revision		Revision as of 22:47, 17 September 2013
Line 122:		Line 122:

	When a client sends a request using POST, the format of the request document body is a function of the value of the request's ''Content-Type'' header. If ''Content-Type'' is ''application/x-www-form-urlencoded'', then the request body should start with the literal ''ce='' (or ''dap:ce='') and then contain the constraint expression. If ''Content-Type'' is not set, do not include the ''ce='' prefix. In both cases the syntax of the CE is unchanged.		When a client sends a request using POST, the format of the request document body is a function of the value of the request's ''Content-Type'' header. If ''Content-Type'' is ''application/x-www-form-urlencoded'', then the request body should start with the literal ''ce='' (or ''dap:ce='') and then contain the constraint expression. If ''Content-Type'' is not set, do not include the ''ce='' prefix. In both cases the syntax of the CE is unchanged.

			'''Important''': Make sure you configure the POST handler and the DAP Dispatch handler with the same optional values.

	'''Note''': If you are using ''curl'' to test this (or as a client), it will set ''Content-Type'' for you (to ''application/x-www-form-urlencoded'') so you must use the ''ce='' literal.		'''Note''': If you are using ''curl'' to test this (or as a client), it will set ''Content-Type'' for you (to ''application/x-www-form-urlencoded'') so you must use the ''ce='' literal.

Jimg: /* Testing */

2013-07-08T18:48:20Z

Testing

← Older revision		Revision as of 18:48, 8 July 2013
Line 262:		Line 262:

	=== Testing ===		=== Testing ===
			'''Note''': To test the #''type'' special form using a browser, make sure to escape the ''#'' character. That is, type ''%23Byte(...'' instead of ''#Byte(...''. The ''#'' character is reserved by [http://tools.ietf.org/html/rfc3986 RFC3986].

	One of the issues with testing POST responses is having a client that will use the HTTP POST verb. I'll show how to use ''curl''.		One of the issues with testing POST responses is having a client that will use the HTTP POST verb. I'll show how to use ''curl''.

Jimg: /* Raw timing data */

2013-07-08T18:40:55Z

Raw timing data

@@ Line 260: / Line 260: @@
 sys 0m6.058s
 -->
 == Period of use ==
 June 2013 onward

Jimg: /* The array constant special forms */

2013-07-08T18:02:48Z

The array constant special forms

← Older revision		Revision as of 18:02, 8 July 2013
Line 140:		Line 140:
	These special forms can build vectors with specific values and return them as DAP2 Arrays. The Array variables can be named using the ''bind_name()'' function and have their shape set using ''bind_shape()''.		These special forms can build vectors with specific values and return them as DAP2 Arrays. The Array variables can be named using the ''bind_name()'' function and have their shape set using ''bind_shape()''.

	;<nowiki>#</nowiki><type>(<size hint>, <values>, ...): The ''#<type>'' (''#Byte'', ...) literal starts the special form. The first argument <size_hint> provides a way to preallocate the memory needed to hold the vector of values. Following that, the values are listed. Unlike ''make_array()'', it is not necessary to provide the exact size of the vector; the size hint is just that, a hint. If a size hint of zero is supplied, it will be ignored. Any of the DAP2 numeric types can be used with this special form. This is called a 'special form' because it invokes a custom parser that can process values very efficiently.		;<nowiki>#</nowiki><type>(<size hint>: <values>, ...): The ''#<type>'' (''#Byte'', ...) literal starts the special form. The first argument <size_hint> provides a way to preallocate the memory needed to hold the vector of values. Following that, the values are listed. Unlike ''make_array()'', it is not necessary to provide the exact size of the vector; the size hint is just that, a hint. If a size hint of zero is supplied, it will be ignored. Any of the DAP2 numeric types can be used with this special form. This is called a 'special form' because it invokes a custom parser that can process values very efficiently.

	Example: #Byte(16,2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5) will return a one dimensional (i.e., a vector) Array of Bytes with values 2, 3, ... . The vector is named ''g<int>'' just like the array returned by make_array(). The vector can be turned in to a N-dimensional Array using ''bind_shape()'' using ''bind_shape("[4][4]",#Byte(16,2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5)).		Example: #Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5) will return a one dimensional (i.e., a vector) Array of Bytes with values 2, 3, ... . The vector is named ''g<int>'' just like the array returned by make_array(). The vector can be turned in to a N-dimensional Array using ''bind_shape()'' using ''bind_shape("[4][4]",#Byte(16:2,3,4,5,2,3,4,5,2,3,4,5,2,3,4,5)).

	The special forms can make a 1,047,572 element vector on Int32 in 0.4 seconds.		The special forms can make a 1,047,572 element vector on Int32 in 0.4 seconds.

Jimg: /* Raw timing data */

2013-07-06T13:24:09Z

Raw timing data

← Older revision		Revision as of 13:24, 6 July 2013
Line 203:		Line 203:
	-->		-->

	~~After improving the~~ scanner based on info from 'instruments' ~~(copying~~ values and applying www2id escaping was		The scanner was improved based on info from 'instruments', an OS/X profiling tool. Copying values and applying www2id escaping was
	moved from the scanner, where it was applied it to every token that matched SCAN_WORD, to the parser, where it		moved from the scanner, where it was applied it to every token that matched SCAN_WORD, to the parser, where it
	was used only for non-numeric tokens). This performance tweak makes a big difference in this case since there		was used only for non-numeric tokens. This performance tweak makes a big difference in this case since there
	are a million SCAN_WORD tokens that are not symbols.		are a million SCAN_WORD tokens that are not symbols.

	{\|		{\| class="wikitable"
	\|+ Runtimes for make_array() and #type, scanner/parser optimized, two trials		\|+ Runtimes for make_array() and #type, scanner/parser optimized, two trials
	! !! colspan="3" \| Time in seconds		! !! colspan="3" \| Time in seconds

Jimg: /* Raw timing data */

2013-07-06T13:21:48Z

Raw timing data

← Older revision		Revision as of 13:21, 6 July 2013
Line 167:		Line 167:
	<!-- -->		<!-- -->

	{\|		{\| class="wikitable"
	\|+ Runtimes for make_array() and #type, before scanner/parser optimization		\|+ Runtimes for make_array() and #type, before scanner/parser optimization
	! What !! Real ~~(s)~~ !! User !! System		! !! colspan="3" \| Time in seconds
			\|-
			! What !! Real !! User !! System
	\|-		\|-
	\| #type, with hint \|\| 37.975 \|\| 37.415 \|\| 0.426		\| #type, with hint \|\| 37.975 \|\| 37.415 \|\| 0.426
Line 208:		Line 210:
	{\|		{\|
	\|+ Runtimes for make_array() and #type, scanner/parser optimized, two trials		\|+ Runtimes for make_array() and #type, scanner/parser optimized, two trials
			! !! colspan="3" \| Time in seconds
			\|-
	! What !! Real (s) !! User !! System		! What !! Real (s) !! User !! System
	\|-		\|-

Jimg: /* Deliverables */

2013-07-06T13:03:51Z

Deliverables

@@ Line 151: / Line 151: @@
 ;bind_name(<name>,<variable>): The <name> must not exist in the dataset; <variable> may be the name of a variable in the dataset (so this function can rename an existing variable) or it can be a variable returned by another function or special form.
 ;bind_shape(<shape expression>,<variable>): The <shape expression> is a string that gives the number and size of the array's dimensions; the <variable> may be the name of a variable in the dataset (so this function can rename an existing variable) or it can be a variable returned by another function or special form.
 == Period of use ==
 June 2013 onward

← Older revision		Revision as of 23:52, 17 September 2013
Line 158:		Line 158:
	was preset to 1,048,576 elements. Times are for 50 repeats.		was preset to 1,048,576 elements. Times are for 50 repeats.

	Summary: Using the special for #Int32(size_hint, values...) is about 10 times faster for a 1 million element vector		Summary: Using the special for $Int32(size_hint, values...) is about 10 times faster for a 1 million element vector
	than make_array(Int32,[1048576],values...). As part of the performance testing, the scanner and parser were run under a sampling runtime analyzer ('Instruments' on OS/X) and the code was optimized so that long sequences of numbers would scan and parse more efficiently. This benefited both the make_array() function and #type() special form.		than make_array(Int32,[1048576],values...). As part of the performance testing, the scanner and parser were run under a sampling runtime analyzer ('Instruments' on OS/X) and the code was optimized so that long sequences of numbers would scan and parse more efficiently. This benefited both the make_array() function and $type() special form.

	==== Raw timing data ====		==== Raw timing data ====
Line 170:		Line 170:

	{\| class="wikitable"		{\| class="wikitable"
	\|+ Runtimes for make_array() and #type, before scanner/parser optimization		\|+ Runtimes for make_array() and $type, before scanner/parser optimization
	! !! colspan="3" \| Time in seconds		! !! colspan="3" \| Time in seconds
	\|-		\|-
	! What !! Real !! User !! System		! What !! Real !! User !! System
	\|-		\|-
	\| #type, with hint \|\| 37.975 \|\| 37.415 \|\| 0.426		\| $type, with hint \|\| 37.975 \|\| 37.415 \|\| 0.426
	\|-		\|-
	\| #type, no hint \|\| 39.388 \|\| 38.600 \|\| 0.397		\| $type, no hint \|\| 39.388 \|\| 38.600 \|\| 0.397
	\|-		\|-
	\| make_array() \|\| 212.697 \|\| 206.894 \|\| 5.771		\| make_array() \|\| 212.697 \|\| 206.894 \|\| 5.771
Line 211:		Line 211:

	{\| class="wikitable"		{\| class="wikitable"
	\|+ Runtimes for make_array() and #type, scanner/parser optimized, two trials		\|+ Runtimes for make_array() and $type, scanner/parser optimized, two trials
	! !! colspan="3" \| Time in seconds		! !! colspan="3" \| Time in seconds
	\|-		\|-
	! What !! Real (s) !! User !! System		! What !! Real (s) !! User !! System
	\|-		\|-
	\| #type, with hint \|\| 19.844 \|\| 19.355 \|\| 0.437		\| $type, with hint \|\| 19.844 \|\| 19.355 \|\| 0.437
	\|-		\|-
	\| #type, with hint \|\| 19.817 \|\| 19.369 \|\| 0.427		\| $type, with hint \|\| 19.817 \|\| 19.369 \|\| 0.427
	\|-		\|-
	\| #type, no hint \|\| 19.912 \|\| 19.444 \|\| 0.430		\| $type, no hint \|\| 19.912 \|\| 19.444 \|\| 0.430
	\|-		\|-
	\| #type, no hint \|\| 19.988 \|\| 19.444 \|\| 0.428		\| $type, no hint \|\| 19.988 \|\| 19.444 \|\| 0.428
	\|-		\|-
	\| make_array() \|\| 195.332 \|\| 189.271 \|\| 6.058		\| make_array() \|\| 195.332 \|\| 189.271 \|\| 6.058