OPULS: NOAA S3 Data Access - Revision history

Ndp at 21:50, 16 September 2014

2014-09-16T21:50:02Z

← Older revision		Revision as of 21:50, 16 September 2014
Line 27:		Line 27:
	1. Install the S3 DAP Service into the same Tomcat server into which you previously installed the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.		1. Install the S3 DAP Service into the same Tomcat server into which you previously installed the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.
	::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>		::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>
	2. Restart Tomcat. This will ~~case~~ the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>		2. Restart Tomcat. This will cause the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>

	== Configure ==		== Configure ==

Ndp: /* Configure */

2014-08-29T22:25:04Z

Configure

← Older revision		Revision as of 22:25, 29 August 2014
Line 31:		Line 31:
	== Configure ==		== Configure ==

			=== S3 DAP Service ===
	The configuration for the S3 DAP Service is located in the file <code>$CATALINA_HOME/content/s3/s3.xml</code>		The configuration for the S3 DAP Service is located in the file <code>$CATALINA_HOME/content/s3/s3.xml</code>

Line 44:		Line 45:

	Note that the bucket name is set to OPeNDAP's test bucket which already has data and /index.xml files waiting. Once you build your own bucket you simply add a new Bucket element to the configuration, restart Tomcat, and off you go.		Note that the bucket name is set to OPeNDAP's test bucket which already has data and /index.xml files waiting. Once you build your own bucket you simply add a new Bucket element to the configuration, restart Tomcat, and off you go.


			=== BES Gateway Module ===

			The S3 Bucket service will need to be added to the Gateway.Whitelist in the BES Gateway Module. Here's the line I added to my gateway.conf file to enable my BES to provide gateway services to the bucket contents:

			<source lang="c">Gateway.Whitelist+=https://opendap.test.s3.amazonaws.com</source>

			[[BES - Modules - Gateway Module \| Click through HERE for a full explanation of the configuration of the BES Gateway module.]]

	== Run ==		== Run ==

Ndp: /* BES Gateway MOdule */

2014-08-29T22:17:48Z

BES Gateway MOdule

← Older revision		Revision as of 22:17, 29 August 2014
Line 157:		Line 157:
	In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.		In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.

	==== BES Gateway ~~MOdule~~ ====		==== BES Gateway Module ====

	The S3 service relies on the BES gateway module to do the heavy lifting of retrieving/caching data objects from S3 and providing DAP services for them. T		The S3 service relies on the BES gateway module to do the heavy lifting of retrieving/caching data objects from S3 and providing DAP services for them. [[BES - Modules - Gateway_Module \| The BES Gateway module and it's configuration are explained here]].

	== Additional Thing's That Could Be Done ==		== Additional Thing's That Could Be Done ==

Ndp: /* BES Gateway */

2014-08-29T22:16:01Z

BES Gateway

← Older revision		Revision as of 22:16, 29 August 2014
Line 157:		Line 157:
	In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.		In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.

	=== BES Gateway ===		==== BES Gateway MOdule ====

	The S3 service relies on the BES gateway module to do the heavy lifting of retrieving/caching data objects from S3 and providing DAP services for them. T		The S3 service relies on the BES gateway module to do the heavy lifting of retrieving/caching data objects from S3 and providing DAP services for them. T

Ndp: /* S3 DAP Service */

2014-08-29T22:15:50Z

S3 DAP Service

← Older revision		Revision as of 22:15, 29 August 2014
Line 127:		Line 127:
	=== The Software ===		=== The Software ===

	=== S3 DAP Service ===		==== S3 DAP Service ====
	The S3 DAP Service is a sibling to the OLFS and utilizes much of the code found there. It requires a running BES with all the appropriate data handlers and the gateway module installed. At startup the service reads it's configuration file which contains the context names for the DAP and Catalog services, the local filesystem location to be used for a catalog cache, and a collection of S3 bucket names to be used as source data collections. Collection names and bucket names are specified independently so that the S3 bucket name need not be a path component in the URLs formed by the S3 DAP Service. Once configured, the /index.xml catalog files are loaded by the server at startup where they are used to generate THREDDS catalogs (with XSLT) for client and browser traversal.		The S3 DAP Service is a sibling to the OLFS and utilizes much of the code found there. It requires a running BES with all the appropriate data handlers and the gateway module installed. At startup the service reads it's configuration file which contains the context names for the DAP and Catalog services, the local filesystem location to be used for a catalog cache, and a collection of S3 bucket names to be used as source data collections. Collection names and bucket names are specified independently so that the S3 bucket name need not be a path component in the URLs formed by the S3 DAP Service. Once configured, the /index.xml catalog files are loaded by the server at startup where they are used to generate THREDDS catalogs (with XSLT) for client and browser traversal.

Ndp: /* The Software */

2014-08-29T21:07:55Z

The Software

← Older revision		Revision as of 21:07, 29 August 2014
Line 127:		Line 127:
	=== The Software ===		=== The Software ===

			=== S3 DAP Service ===
	The S3 DAP Service is a sibling to the OLFS and utilizes much of the code found there. It requires a running BES with all the appropriate data handlers and the gateway module installed. At startup the service reads it's configuration file which contains the context names for the DAP and Catalog services, the local filesystem location to be used for a catalog cache, and a collection of S3 bucket names to be used as source data collections. Collection names and bucket names are specified independently so that the S3 bucket name need not be a path component in the URLs formed by the S3 DAP Service. Once configured, the /index.xml catalog files are loaded by the server at startup where they are used to generate THREDDS catalogs (with XSLT) for client and browser traversal.		The S3 DAP Service is a sibling to the OLFS and utilizes much of the code found there. It requires a running BES with all the appropriate data handlers and the gateway module installed. At startup the service reads it's configuration file which contains the context names for the DAP and Catalog services, the local filesystem location to be used for a catalog cache, and a collection of S3 bucket names to be used as source data collections. Collection names and bucket names are specified independently so that the S3 bucket name need not be a path component in the URLs formed by the S3 DAP Service. Once configured, the /index.xml catalog files are loaded by the server at startup where they are used to generate THREDDS catalogs (with XSLT) for client and browser traversal.

Line 155:		Line 156:

	In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.		In support of the DAP services stack, the S3 DAP Service software maintains lists of the S3 resource ID's gleaned from the catalog and uses these to construct BES requests for DAP services by access the BES Gateway Module which can retrieve/cache HTTP resources and provide DAP services for them.

			=== BES Gateway ===

			The S3 service relies on the BES gateway module to do the heavy lifting of retrieving/caching data objects from S3 and providing DAP services for them. T

	== Additional Thing's That Could Be Done ==		== Additional Thing's That Could Be Done ==

Ndp: /* Configure */

2014-08-29T21:05:24Z

Configure

← Older revision		Revision as of 21:05, 29 August 2014
Line 43:		Line 43:
	</source>		</source>

	Note that the bucket name is set to OPeNDAP's test bucket which already has data and /index.xml files waiting. Once you build ~~you're~~ own bucket you ~~simple~~ add a new Bucket element to the configuration, restart Tomcat, and off you go.		Note that the bucket name is set to OPeNDAP's test bucket which already has data and /index.xml files waiting. Once you build your own bucket you simply add a new Bucket element to the configuration, restart Tomcat, and off you go.

	== Run ==		== Run ==

Ndp: /* Install */

2014-08-29T21:04:42Z

Install

← Older revision		Revision as of 21:04, 29 August 2014
Line 25:		Line 25:
	== Install ==		== Install ==

	# Install the S3 DAP Service into the same Tomcat server into which you previously installed the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.		1. Install the S3 DAP Service into the same Tomcat server into which you previously installed the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.
	::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>		::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>
	# Restart Tomcat. This will case the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>		2. Restart Tomcat. This will case the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>

	== Configure ==		== Configure ==

Ndp: /* Get */

2014-08-29T21:04:09Z

Get

← Older revision		Revision as of 21:04, 29 August 2014
Line 6:		Line 6:

	== Get ==		== Get ==
	You will need to get and install Hyrax. ~~The~~ you'll need to compile and install a new java component.		You will need to get and install Hyrax. Then you'll need to get the OLFS source code tree and compile and install a new java component.

	=== Hyrax ===		=== Hyrax ===

Ndp: /* Install */

2014-08-29T18:30:44Z

Install

← Older revision		Revision as of 18:30, 29 August 2014
Line 25:		Line 25:
	== Install ==		== Install ==

	# Install the S3 DAP Service into the same Tomcat server into which you ~~install~~ the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.		# Install the S3 DAP Service into the same Tomcat server into which you previously installed the OLFS/Hyrax by copying the s3.war file that you just built into Tomcat's webapps directory.
	::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>		::<code>cp build/dist/s3.war $CATALINA_HOME/webapps</code>
	# Restart Tomcat. This will case the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>		# Restart Tomcat. This will case the S3 DAP service to start and install it's default configuration into <code>$CATALINA_HOME/content/s3</code>