Hyrax: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
mNo edit summary
 
(123 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This is the '''OPeNDAP 4 Data Server''', also known as '''Hyrax'''.
<font size="+1">This is the '''OPeNDAP 4 Data Server''', also known as '''Hyrax'''. See our new '''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html Hyrax Installation and Configuration Guide]'''.</font>
----
<font color="red">
Note: The documentation on this is somewhat out of date as of October 2017.
</font>


Hyrax is a new data server which combines the efforts at UCAR/HAO to build a high performance DAP-compliant data server for the Earth System Grid II project with existing software developed by OPeNDAP.
This page leads you through implementing and configuring Hyrax. Work through this page from top to bottom.


=Overview=
== Hyrax Operational Overview ==
[[Image:HyraxArchitecture.jpg|right|thumb]]
[[Image:HyraxArchitecture.jpg|right|thumb]]


Line 9: Line 13:


The BES is the high-performance server software from HAO. It handles reading data from the data stores and returning DAP-compliant responses to the OLFS. In turn, the OLFS may pass these response back to the requestor with little or no modification or it may use them to build more complex responses. The nature of the Inter Process Communication (IPC) between the OLFS and BES is such that they should both be on the same machine or be able to communicate over a very high bandwidth channel.
The BES is the high-performance server software from HAO. It handles reading data from the data stores and returning DAP-compliant responses to the OLFS. In turn, the OLFS may pass these response back to the requestor with little or no modification or it may use them to build more complex responses. The nature of the Inter Process Communication (IPC) between the OLFS and BES is such that they should both be on the same machine or be able to communicate over a very high bandwidth channel.


Both the OLFS and the BES will run and serve test data immediately after a default installation. Additional configuration is required for them to serve site specific data.
Both the OLFS and the BES will run and serve test data immediately after a default installation. Additional configuration is required for them to serve site specific data.


 
{|  class="wikitable"
==Features==
|+ Hyrax Features
 
| '''THREDDS Catalog Support'''
;THREDDS Catalog Support  
| Hyrax supports the THREDDS catalogs. It can serve user supplied static catalogs and it will dynamically generate THREDDS catalogs of it's internal holdings.
:Hyrax supports the THREDDS catalogs. It can serve user supplied static catalogs and it will dynamically generate THREDDS catalogs of it's internal holdings. Our server is designed so that you can set it up to serve data by editing a single line in the bes.conf file (see information about the ''BES.Catalog.catalog.RootDirectory'' configuration parameter). You don't have to write THREDDS catalogs f you don't want to (but if you do, those provide an excellent way to tailor the way data you store locally is present to remote users).
|-
 
| '''Dataset Aggregation'''
;SOAP Messaging Support
| Collections of related data resources can be collected into a single dataset using the aggregation features. Typically these are formed for geographic tiles, time series, etc.
:Hyrax supports SOAP requests. A prototype SOAP interface is included with this release of Hyrax. It is discussed in the [http://rsg.opendap.org:8090/server-4/templates/soapAPI.html RSG template pages here]. Example client code can be found at our [http://www.opendap.org/download/hyrax.html web site], and can be found in our [http://scm.opendap.org:8090/trac/browser/tags/SoapTestClient Subversion archive here].
|-
 
|'''Adding/modifying dataset content'''
;Plugin Format and Response Handlers
| Datasets can be modified by the server without having to actually change the underlying files. These views are independently accessible from the original data. Both dataset metadata and data values may be added or changed.  
:The Back-End Server (BES) used with Hyrax supports run-time loading of handlers for different formats ''and'' different response types (e.g., the ASCII response and the HTML form interface). To replace one of the standard handlers, or to add a new one, edit the ''bes.conf'' file.
|-
 
|'''Multiple source data formats'''
;Simple Conversion from the Old-style Handlers
| Server can ingest source data stored as HDF4, HDF4-EOS, HDF5, HDF5-EOS, NetCDF-3, NetCDF-4, CEDAR, FITS, Comma Separated Values, and raw ASCII and Binary formats. Because of Hyrax's extensible design, it's easy to add new source data formats.
:It is fairly simple to convert handlers written for the older OPeNDAP server to run-time loadable modules for the BES.
|-
|'''Multiple Return Formats For Data rRetrieval in multiple return formats'''
| Hyrax is able to return data in DAP, DAP4, NetCDF-3, NetCDF-4, JSON, CSV, and ASCII formats, Or, you can add your own response types.
|-
|'''Gateway Service'''
| Hyrax supports a gateway service feature that allows it to provide DAP (and other Hyrax) services for remotely held datasets that are stored in any of Hyrax's source data formats.
|-
|'''RDF'''
| Hyrax provides RDF descriptions of it's data holdings. These can enable semantic web tools to operate upon the metadata content held in the server.
|-
|'''[[Server_Side_Processing_Functions | Server side functions]]'''
|Hyrax supports a number of server side functions out of the box including (but not limited to):
* ''[[Server_Side_Processing_Functions#geogrid.28.29|geogrid]]'': Subset applicable DAP Grids using latitude and longitude values.
* ''[[Server_Side_Processing_Functions#grid.28.29|grid]]'': Subset any DAP Grid object using the values of it's map vectors.
* ''[[Server_Side_Processing_Functions#linear_scale|linear_scale]]'': Apply a linear equation to the data returned, including automatic use of CF attributes.
* ''[[Server_Side_Processing_Functions#version.28.29|version]]'': The version function provides a list of the server-side processing functions available.
* New ones are easy to add.
|-
|'''Extensible WebStart functionality for data clients'''
| Hyrax provides WebStart functionality for a number of Java based DAP clients. It's simple to add new clients to the list that Hyrax supports.
|-
|'''Extensible WebStart functionality for data clients'''
|Hyrax provides WebStart functionality for a number of Java based DAP clients. It's simple to add new clients to the list that Hyrax supports.
|-
|'''Extensible/Configurable web interface'''
|The web interface for both Hyrax and the administrator's interface can be customized using CSS and XSL. You can add your organizations logo and specialize the colors and fonts in the presentation of data sets.
|-
|'''Administrator's interface'''
|Control and dynamically update Hyrax from a convenient web interface. See [[Hyrax_-_Administrators_Interface| the Admin interface documentation]].
|-
|'''WMS services'''
|[[Hyrax WMS|Hyrax now supports WMS services via integration with ncWMS.]]
|-
|'''JSON responses'''
|[[Hyrax JSON| Both metadata and data are now available in a JSON encoding.]]
|-
|'''JSON responses'''
|[[Hyrax JSON| Both metadata and data are now available in a JSON encoding.]]
|-
|'''w10n'''
|Hyrax comes with a complete w10n service stack. W10n navigation is supported through the default catalog where all datasets and "structure" variables appear as graph nodes. Data can be acquired for atomic types or arrays of atomic types in a number of formats.
|}


;Feature Request
;Feature Request
:Is there a feature you would like to see but don't? Let us know: [mailto:support@opendap.org support@opendap.org] or [mailto:opendap-tech@opendap.org opendap-tech@opendap.org] (You need to [http://mailman.opendap.org/mailman/listinfo/opendap-tech subscribe] first)
: Is there a feature you would like to see but don't? Let us know: [mailto:support@opendap.org support@opendap.org] or [mailto:opendap-tech@opendap.org opendap-tech@opendap.org] (You need to [http://mailman.opendap.org/mailman/listinfo/opendap-tech subscribe] first)


@TODO: '''Add a more comprehensive listing of the Hyrax feature set.'''
== Downloading and Installing ==


=Downloads=
The download and installation instructions are kept together.
For the latest release look at <font size="2">[https://www.opendap.org/index.php/software/hyrax-data-server the Hyrax downloads page]</font>.


To obtain Hyrax just go to the [http://www.opendap.org/download/hyrax.html  Hyrax download page] and get it.
=== Building And Installing From Source ===


=Documentation=
[[Hyrax GitHub Source Build|Getting and Building The Code From GitHub]]


==How To Guides==
== Server Configuration ==
 
At this point, you should have a running Hyrax server, but it will not be configured for your needs. In this section you will configure your Hyrax server. To do this, you will need to  configure BES and OLFS; optionally, you can configure THREDDS catalogs to provide a custom virtual view of the servers holdings.
===For Users:===
* [[Hyrax - Installation Instructions|Hyrax Installation Instructions]]
* [[Hyrax - Configuration|Hyrax Configuration Instructions]]
* [[Hyrax - Configuration|Hyrax Configuration Instructions]]
** [[Hyrax - BES Configuration|BES Configuration]]
** [[Hyrax - BES Configuration|BES Configuration]]
** [[Hyrax - OLFS Configuration|OLFS Configuration]]
** [[Hyrax - OLFS Configuration|OLFS Configuration]]
** [[Hyrax - THREDDS Configuration|THREDDS Configuration]]
** [[Hyrax - THREDDS Configuration|THREDDS Catalog Configuration]]
** [[Hyrax - Logging Configuration|Logging Configuration]]
 
=== Apache Integration ===
This is an optional step after configuring BES and OLFS. Integrating Hyrax with Apache allows you to integrate your Hyrax server with existing web infrastructure and is the recommended path for enabling user authentication and authorization. 
* [[Hyrax - Apache Integration|Hyrax integration with the Apache Web Server]]
* [[Hyrax - Apache Integration|Hyrax integration with the Apache Web Server]]
* [[BES - Modules|BES Modules]]:
==== User Authentication ====
** [[BES - Modules - The NetCDF Handler|netcdf_data_handler]]
** [[Hyrax_-_User_Identification_(Authentication) | Hyrax User Authentication and Identification]] Configure Hyrax to support user authentication. Explains how to integrate Hyrax with LDAP, Earthdata Login, Shibboleth, etc.
** [[The_FreeForm_Data_Handler|freeform data handler]]
** [[DAP Clients - Authentication]] Instruction to configure various DAP (and other generic HTTP clients) to work with the various authentication schemes
** [[BES - Modules - FileOut Netcdf|file out netcdf]]
** [[BES - Modules - NcML Module|NcML data handler]] (Some features are not complete)
** [[BES - Modules - Gateway Module|gateway handler]] (Provides interoperability between Hyrax and other web services)
** CSV Handler ''Update this and then edit www.opendap.org/download/csv_handler.html''
* [[Hyrax - Customizing Hyrax| Customizing Hyrax]]
* [[Hyrax - Release Notes|Release Notes]]
* [[Australian BOM System Administrator's Agenda and Presentations|System Administrators Workshop]]
*[[Hyrax - Administrators Interface]]


===For Developers:===
=== And then there's more...===
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]]
* [[Hyrax - Customizing Hyrax| Customizing Hyrax]]. Here you will find instructions on how to change the colors, fonts, and logo images presented in the Hyrax web pages.
*BES
* [[Australian BOM System Administrator's Agenda and Presentations|System Administrators Workshop]] Here are slides from a system administrators workshop we gave to the Australian BOM on how to manage a Hyrax instance.
** [[Hyrax - BES Build Instructions| Building the BES from source.]]
* [[Hyrax - Administrators Interface]] Here are the instruction for using the Hyrax Admin Interface, a web UI that allows one to view logs, restart, reconfigure, and otherwise fiddle with a Hyrax instance.
** [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
<!-- * [[Hyrax - Release Notes|Release Notes]] -->
** [[Hyrax - Create BES Module | How to create your own BES Module]]
** [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
** [[Hyrax - Running bescmdln | Running the BES command line client]]
** [[Hyrax - BES Client commands| BES Client commands]]
** [[BES_XML_Commands | BES XML Commands]]
** [[Hyrax - Extending BES Module | Extending your BES Module]]
** [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
** [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]
*OLFS
** [[Hyrax - OLFS Build Instructions|Building the OLFS from source.]]
** [[Hyrax - SOAP Test Client|Using the SOAP Test Client.]]
* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]


==Reference documentation==
== Modules ==
Hyrax has a number of modules that provide the actual functionality of the server: Reading data files, building different kinds of responses and performing different kinds of server processing operations. Most of these modules work with the BES but some are part of the front (web facing) part of the server.


* [http://www.opendap.org/api/bes-pref-html/html/index.html BES Reference (3.6.2)]
=== BES modules ===
* [http://www.opendap.org/api/pref/html/index.html libdap Reference (3.8.2)]
* [[BES - Modules - The NetCDF Handler|NetCDF data handler]]
* [[BES - Modules - The HDF4 Handler|HDF4 data handler]]
* [[BES - Modules - The HDF5 Handler|HDF5 data handler]]
* [[The_FreeForm_Data_Handler|FreeForm data handler]]
* [[BES - Modules - NcML Module|NcML data handler]]
** [http://docs.opendap.org/index.php/BES_-_Modules_-_NcML_Module#Functionality Variable and Metadata modification]  
** [http://docs.opendap.org/index.php/BES_-_Modules_-_NcML_Module#Aggregation_Tutorials Aggregated Datasets]'')
* [[BES - Modules - Gateway Module|Gateway handler]] (Interoperability between Hyrax and other web services)
* [[BES - Modules - CSV Handler | CSV handler ]]
* [[BES - Modules - GeoTiff, GRIB2, JPEG2000 Handler| GeoTiff, GRiB2, JPEG2000 hander]]
* [[BES - Modules - FileOut Netcdf|NetCDF File Response handler]]
* [[BES_-_Modules_-_FileOut_GDAL | GDAL (GeoTIFF, JPEG2000) File Response handler]]


=Thanks!=
=== Additional Java Modules that use the BES ===
* [[Hyrax WMS | WMS]] - Web Mapping Service via integration with ncWMS.
* [[Aggregation enhancements]]


We hope we hope you find this software useful, and we welcome your questions and comments.
Unsupported:
* [[BES - Modules - SQL Hander | SQL handler]]


==Contact Us==
== Technical Support ==


'''Technical Support:'''  
'''Technical Support:'''  
Line 101: Line 141:
* jgallagher <at> opendap <dot> org (''libdap'')
* jgallagher <at> opendap <dot> org (''libdap'')


==Sponsorship==
== Sponsorship ==


====OPeNDAP Hyrax development is sponsored by:====
OPeNDAP Hyrax development is sponsored by:


[[Image:Nasa-logo.jpg|95px]] '''[http://www.nasa.gov  National Aeronautics and Space Administration]'''


[[Image:Nsf-logo.png|95px]]  '''[http://www.nsf.gov National Science Foundation]'''
[[Image:Nsf-logo.png|95px]]  '''[http://www.nsf.gov National Science Foundation]'''


This material is based upon work supported by the National Science Foundation under Grant No. 0430822. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
This material is based on work supported by the National Science Foundation under Grant No. 0430822. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).
 
[[Image:Nasa-logo.jpg|95px]] '''[http://www.nasa.gov  National Aeronautics and Space Administration]'''


[[Image:Noaa-logo.jpg|90px]]  '''[http://www.noaa.gov National Oceanic and Atmospheric Administration]'''
[[Image:Noaa-logo.jpg|90px]]  '''[http://www.noaa.gov National Oceanic and Atmospheric Administration]'''

Latest revision as of 18:47, 26 October 2017

This is the OPeNDAP 4 Data Server, also known as Hyrax. See our new Hyrax Installation and Configuration Guide.


Note: The documentation on this is somewhat out of date as of October 2017.

This page leads you through implementing and configuring Hyrax. Work through this page from top to bottom.

Hyrax Operational Overview

HyraxArchitecture.jpg

Hyrax uses the Java servlet mechanism to hand off requests from a general web daemon to DAP format-specific software. This results in higher performance for small requests. The servlet front end, which we call the OPeNDAP Lightweight Front end Server (OLFS) looks at each request and formulates a query to a second server (which may or may not on the same machine as the OLFS) called the Back End Server (BES).

The BES is the high-performance server software from HAO. It handles reading data from the data stores and returning DAP-compliant responses to the OLFS. In turn, the OLFS may pass these response back to the requestor with little or no modification or it may use them to build more complex responses. The nature of the Inter Process Communication (IPC) between the OLFS and BES is such that they should both be on the same machine or be able to communicate over a very high bandwidth channel.

Both the OLFS and the BES will run and serve test data immediately after a default installation. Additional configuration is required for them to serve site specific data.

Hyrax Features
THREDDS Catalog Support Hyrax supports the THREDDS catalogs. It can serve user supplied static catalogs and it will dynamically generate THREDDS catalogs of it's internal holdings.
Dataset Aggregation Collections of related data resources can be collected into a single dataset using the aggregation features. Typically these are formed for geographic tiles, time series, etc.
Adding/modifying dataset content Datasets can be modified by the server without having to actually change the underlying files. These views are independently accessible from the original data. Both dataset metadata and data values may be added or changed.
Multiple source data formats Server can ingest source data stored as HDF4, HDF4-EOS, HDF5, HDF5-EOS, NetCDF-3, NetCDF-4, CEDAR, FITS, Comma Separated Values, and raw ASCII and Binary formats. Because of Hyrax's extensible design, it's easy to add new source data formats.
Multiple Return Formats For Data rRetrieval in multiple return formats Hyrax is able to return data in DAP, DAP4, NetCDF-3, NetCDF-4, JSON, CSV, and ASCII formats, Or, you can add your own response types.
Gateway Service Hyrax supports a gateway service feature that allows it to provide DAP (and other Hyrax) services for remotely held datasets that are stored in any of Hyrax's source data formats.
RDF Hyrax provides RDF descriptions of it's data holdings. These can enable semantic web tools to operate upon the metadata content held in the server.
Server side functions Hyrax supports a number of server side functions out of the box including (but not limited to):
  • geogrid: Subset applicable DAP Grids using latitude and longitude values.
  • grid: Subset any DAP Grid object using the values of it's map vectors.
  • linear_scale: Apply a linear equation to the data returned, including automatic use of CF attributes.
  • version: The version function provides a list of the server-side processing functions available.
  • New ones are easy to add.
Extensible WebStart functionality for data clients Hyrax provides WebStart functionality for a number of Java based DAP clients. It's simple to add new clients to the list that Hyrax supports.
Extensible WebStart functionality for data clients Hyrax provides WebStart functionality for a number of Java based DAP clients. It's simple to add new clients to the list that Hyrax supports.
Extensible/Configurable web interface The web interface for both Hyrax and the administrator's interface can be customized using CSS and XSL. You can add your organizations logo and specialize the colors and fonts in the presentation of data sets.
Administrator's interface Control and dynamically update Hyrax from a convenient web interface. See the Admin interface documentation.
WMS services Hyrax now supports WMS services via integration with ncWMS.
JSON responses Both metadata and data are now available in a JSON encoding.
JSON responses Both metadata and data are now available in a JSON encoding.
w10n Hyrax comes with a complete w10n service stack. W10n navigation is supported through the default catalog where all datasets and "structure" variables appear as graph nodes. Data can be acquired for atomic types or arrays of atomic types in a number of formats.
Feature Request
Is there a feature you would like to see but don't? Let us know: support@opendap.org or opendap-tech@opendap.org (You need to subscribe first)

Downloading and Installing

The download and installation instructions are kept together. For the latest release look at the Hyrax downloads page.

Building And Installing From Source

Getting and Building The Code From GitHub

Server Configuration

At this point, you should have a running Hyrax server, but it will not be configured for your needs. In this section you will configure your Hyrax server. To do this, you will need to configure BES and OLFS; optionally, you can configure THREDDS catalogs to provide a custom virtual view of the servers holdings.

Apache Integration

This is an optional step after configuring BES and OLFS. Integrating Hyrax with Apache allows you to integrate your Hyrax server with existing web infrastructure and is the recommended path for enabling user authentication and authorization.

User Authentication

And then there's more...

  • Customizing Hyrax. Here you will find instructions on how to change the colors, fonts, and logo images presented in the Hyrax web pages.
  • System Administrators Workshop Here are slides from a system administrators workshop we gave to the Australian BOM on how to manage a Hyrax instance.
  • Hyrax - Administrators Interface Here are the instruction for using the Hyrax Admin Interface, a web UI that allows one to view logs, restart, reconfigure, and otherwise fiddle with a Hyrax instance.

Modules

Hyrax has a number of modules that provide the actual functionality of the server: Reading data files, building different kinds of responses and performing different kinds of server processing operations. Most of these modules work with the BES but some are part of the front (web facing) part of the server.

BES modules

Additional Java Modules that use the BES

Unsupported:

Technical Support

Technical Support:

Hyrax Java Development:

  • ndp <at> opendap <dot> org

Hyrax C++ Development:

  • pwest <at> ucar <dot> edu (bes)
  • jgallagher <at> opendap <dot> org (libdap)

Sponsorship

OPeNDAP Hyrax development is sponsored by:

Nasa-logo.jpg National Aeronautics and Space Administration

Nsf-logo.png National Science Foundation

This material is based on work supported by the National Science Foundation under Grant No. 0430822. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Noaa-logo.jpg National Oceanic and Atmospheric Administration