OPeNDAP Documentation - User contributions [en]

Hyrax GitHub Source Build

2026-01-29T00:18:59Z

Jimg: /* Apple OSX (Mx processor) */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a system to build our code =
== CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>yum -y update</tt>

;You will need to enable power-tools for this setup
:<tt>yum config-manager --set-enabled powertools</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel</tt>

;Tell the machine where to find the tirpc libraries
:<tt>export CPPFLAGS=-I/usr/include/tirpc</tt>
:<tt>export LDFLAGS=-ltirpc</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>sudo yum install -y python3</tt>

;If you're going to build RPMs
:<tt>yum install rpm-devel rpm-build redhat-rpm-config</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== [[Hyrax-Rocky9|Configuring Rocky9]] ==
== [[Hyrax-Rocky8|Configuring Rocky8]] ==

== Rocky 8 ==
''Updated 6/6/2024''

To get the commands ps, which, etc.
dnf install -y procps

C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf config-manager --set-enabled powertools

Install CppUnit and some more development libraries
dnf install -y cppunit cppunit-devel openjpeg2-devel jasper-devel

Install the RPM tools
dnf install -y rpm-devel rpm-build redhat-rpm-config

Install the AWS CLI
dnf install -y awscli

== Apple OSX (M''x'' processor) ==

Computers with the Apple M series chips require dedicated binaries, or binaries with both Intel and M1 contents. In order to get the ''hyrax-dependencies'' project (and the libdap4, and bes projects) to build the following packages need to be installed prior to running the hyrax-dependencies build.

Updated 1/28/2026 using notes from a build on a clean OSX M1 machine running Tahoe 26.2. I loaded some things that are not completely necessary, like 1Password, during very first step. I'm documenting that here just to be complete. jhrg

I installed '''Chrome''', '''1Password''', and '''vscode''' because they make it easier for me, but I did not directly use them for the build. I used the emacs clone '''mg''' that is bundled with OSX to edit files.

'''Xcode'''
* Use the App Store to install Xcode
* Use a terminal window to run the command: ''xcode-select --install'' <-- I'm not sure if I did this. It's likely, but I started Xcode and clicked 'OK' in the dialog that prompts for the various environments to install (e.g., do you want to write code for the Apple Watch). I installed only the development tools for OSX.

'''Homebrew'''
* Needed for later steps
* Set the homebrew install path (/opt/homebrew for me) to HB (or a more wordy alternative ;-)
* export PATH="$HB/bin:$PATH"

'''cmake'''
* brew install cmake

'''pkg-config'''
* brew install pkg-config

'''libpng'''
* brew install libpng
* export CPPFLAGS="$HB/include"
* export LDFLAGS="$HB/lib"

''At this point, the hyrax-dependencies repo should build, but make sure you follow the directions there, e.g., sourcing '''spath.sh''' ''

'''autotools'''
* brew install autoconf automake libtool

'''CPPUNIT'''
* brew install cppunit

''CppUnit is not needed to build the code, but it is needed to run the unit tests''

''At this point the libdap4 repo should build''

'''openssl'''
* brew install openssl

''At this point the bes repo should build''

'''ant'''
* brew install ant

''At this point the OLFS will build the Ant 'server' target (nothing else tested)''

'''Note''': as of 1/28/26 starting the BES reveals that for ''this build'' the gdal module does not work because of -rpath linker issues. However, 'turning off' the gdal module fixes this and a functioning BES exists. I found that homebrew also has Tomcat 11 and will install it, but I could not get the opendap.war to work. This needs a bit more effort.

'''Note''': I made no attempt to build a docker container.

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
If you didn't run hyrax_clone.sh, make sure you're in the top hyrax directory and use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the top level hyrax directory)
<tt>
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.

<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.

Then, run configure:

<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-90.cgi Tomcat 9 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-9.0.105.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-9.0.105/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-9.0.105/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;./apache-tomcat-9.0.105/bin/shutdown.sh
;besctl stop

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Hyrax GitHub Source Build

2026-01-28T23:38:56Z

Jimg: /* Apple OSX (Mx processor) */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a system to build our code =
== CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>yum -y update</tt>

;You will need to enable power-tools for this setup
:<tt>yum config-manager --set-enabled powertools</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel</tt>

;Tell the machine where to find the tirpc libraries
:<tt>export CPPFLAGS=-I/usr/include/tirpc</tt>
:<tt>export LDFLAGS=-ltirpc</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>sudo yum install -y python3</tt>

;If you're going to build RPMs
:<tt>yum install rpm-devel rpm-build redhat-rpm-config</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== [[Hyrax-Rocky9|Configuring Rocky9]] ==
== [[Hyrax-Rocky8|Configuring Rocky8]] ==

== Rocky 8 ==
''Updated 6/6/2024''

To get the commands ps, which, etc.
dnf install -y procps

C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf config-manager --set-enabled powertools

Install CppUnit and some more development libraries
dnf install -y cppunit cppunit-devel openjpeg2-devel jasper-devel

Install the RPM tools
dnf install -y rpm-devel rpm-build redhat-rpm-config

Install the AWS CLI
dnf install -y awscli

== Apple OSX (M''x'' processor) ==

Computers with the Apple M series chips require dedicated binaries, or binaries with both Intel and M1 contents. In order to get the ''hyrax-dependencies'' project (and the libdap4, and bes projects) to build the following packages need to be installed prior to running the hyrax-dependencies build.

Updated 1/28/2026 using notes from a build on a clean OSX M1 machine running Tahoe 26.2. I loaded some things that are not completely necessary, like 1Password, during very first step. I'm documenting that here just to be complete. jhrg

I installed '''Chrome''', '''1Password''', and '''vscode''' because they make it easier for me, but I did not directly use them for the build. I used the emacs clone '''mg''' that is bundled with OSX to edit files.

'''Xcode'''
* Use the App Store to install Xcode
* Use a terminal window to run the command: ''xcode-select --install'' <-- I'm not sure if I did this. It's likely, but I started Xcode and clicked 'OK' in the dialog that prompts for the various environments to install (e.g., do you want to write code for the Apple Watch). I installed only the development tools for OSX.

'''Homebrew'''
* Needed for later steps
* Set the homebrew install path (/opt/homebrew for me) to HB (or a more wordy alternative ;-)
* export PATH="$HB/bin:$PATH"

'''cmake'''
* brew install cmake

'''pkg-config'''
* brew install pkg-config

'''libpng'''
* brew install libpng
* export CPPFLAGS="$HB/include"
* export LDFLAGS="$HB/lib"

''At this point, the hyrax-dependencies repo should build, but make sure you follow the directions there, e.g., sourcing '''spath.sh''' ''

'''autotools'''
* brew install autoconf automake libtool

'''CPPUNIT'''
* brew install cppunit

''CppUnit is not needed to build the code, but it is needed to run the unit tests''

''At this point the libdap4 repo should build''

'''openssl'''
* brew install openssl

''At this point the bes repo should build''

I did not build the OLFS nor did I make docker containers.

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
If you didn't run hyrax_clone.sh, make sure you're in the top hyrax directory and use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the top level hyrax directory)
<tt>
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.

<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.

Then, run configure:

<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-90.cgi Tomcat 9 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-9.0.105.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-9.0.105/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-9.0.105/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;./apache-tomcat-9.0.105/bin/shutdown.sh
;besctl stop

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Developer Info

2025-05-23T20:31:24Z

Jimg: /* General development information */

* [https://github.com/OPENDAP OPeNDAP's GitHub repositories]: OPeNDAP's software is available using GitHub in addition to the downloads from our website.
** Before 2015 we hosted our own SVN repository. It's still online and available, but for read-only access, at [https://scm.opendap.org/svn https://scm.opendap.org/svn].
* [https://travis-ci.org/OPENDAP Continuous Integration builds]: Software that is built whenever new changes are pushed to the master branch. These builds are done on the Travis-CI system.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* We use the Coverity static system to look for common software defects, information on Hyrax is spread across three projects:
** [https://scan.coverity.com/projects/opendap-bes?tab=overview The BES and the standard handlers we distribute]
** [https://scan.coverity.com/projects/opendap-olfs?tab=overview The OLFS - the front end to the Hyrax data server]
** [https://scan.coverity.com/projects/opendap-libdap4?tab=overview libdap - The implementation of DAP2 and DAP4]

== OPeNDAP's FAQ ==
The [http://www.opendap.org/faq-page OPeNDAP FAQ] has a pretty good section on developer's questions.

== C++ Coding Information ==
* [https://google.github.io/styleguide/cppguide.html Google C++ Style Guide]
* [[Include files for libdap | Guidelines for including headers]]
* [[Using lambdas with the STL]]
* [[Better Unit tests for C++]]
* [[Better Singleton classes C++]]
* [[What is faster? stringstream string + String]]
* [[More about strings - passing strings to functions]]

== OPeNDAP Workshops ==
* [http://www.opendap.org/about/workshops-and-presentations/2007-10-12 The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/about/workshops-and-presentations/2008-07-15 ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.

== libdap4 and BES Reference documentation ==
* [https://opendap.github.io/bes/html/ BES Reference]
* [https://opendap.github.io/libdap4/html/ libdap Reference]

== BES Development Information ==
* [[Hyrax - Logging Configuration|Logging Configuration]]

* [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
* [[BES - Debugging Using besstandalone]]
* [[Hyrax - Create BES Module | How to create your own BES Module]]
* Hyrax Module Integration: How to configure your module so it's easy to add to Hyrax instances ([[:File:HyraxModuleIntegration-1.2.pdf|pdf]])
* [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
* [[Hyrax - Running bescmdln | Running the BES command line client]]
* [[Hyrax - BES Client commands| BES Client commands]]. The page [[BES_XML_Commands | BES XML Commands]] repeats this info for a bit more information on the return values. Most of the commands don't return anything unless they return an error and are expected to be used in a group where a ''get'' command closes out the request and obviously does return a response of some kind (maybe an error).
* [[Hyrax:_BES_Administrative_Commands| BES Administrative Commands]]
* [[Hyrax - Extending BES Module | Extending your BES Module]]
* [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
* [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]

* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]

== OPeNDAP Development process information ==
These pages contain information about how we'd like people working with us to use our various on-line tools.

* [[Planning a Program Increment]] This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
* [[Hyrax GitHub Source Build]] This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
* [[Bug Prioritization]]. How we prioritize bugs in our software.

===[[How to Make a Release|Making A Release]] ===
* [[How to Make a Release]] A general template for making a release. This references some of the pages below.

== Software process issues: ==
* [[How to download test logs from a Travis build]] All of our builds on Travis that run tests save those logs to an S3 bucket.
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]] - Updated 12/2014 to include information regarding git.
* [[How to use CLion with our software]]
* [[BES Timing| How to add timing instrumentation to your BES code.]]
* [[UnitTests| How to write unit tests using CppUnit]] NB: See other information under the heading of C++ development
* [[valgrind| How to use valgrind with unit tests]]
* [[Debugging the distcheck target]] Yes, this gets its own page...
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[Managing public and private keys using gpg]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
* [[UserSupport|How to Handle Email-list Support Questions]]
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [http://semver.org/ Software version numbers]
* [[GuideLines| Development Guidelines]]
* [[Apple M1 Special Needs]]

==== Older info of limited value: ====
* [http://gcc.gnu.org/gcc-4.4/cxx0x_status.html C++-11 gcc/++-4.4 support] We now require compilers that support C++-14, so this is outdated (4/19/23).
* [[How to use Eclipse with Hyrax Source Code]] I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.

==== AWS Tips ====
* [[Growing a CentOS Root Partition on an AWS EC2 Instance]]
* [[How Shutoff the CentOS firewall]]

== General development information ==
These pages contain general information relevant to anyone working with our software:

* '''[[Git Hacks and Tricks]]''': Information about using git and/or GitHub that seems useful and maybe not all that obvious.
* [[Git Secrets]]: securing repositories from AWS secret key leaks.
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind Suppression File Howto] How to build a suppressions file for valgrind.
* [[Using a debugger for C++ with Eclipse on OS/X]] Short version: use lldbmi2 **Add info**
* [[Using ASAN]] Short version, look [https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger at the Google/GitHub pages] for useful environment variables **add text** On Centos, use yum install llvm to get the 'symbolizer' and try ''ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer)''
* [https://www.jviotti.com/2024/01/29/using-xcode-instruments-for-cpp-cpu-profiling.html Using Xcode Instruments for C++ CPU profiling]
* [[How to use ''Instruments'' on OS/X to profile]] Updated 7/2018
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind - How to generate suppression files for valgrind] This will quiet valgrind, keeping it from telling you OS/X or Linux (or the BES) is leaking memory.
* [[Migrating source code from SVN to git]]: How to move a large project from SVN to git and keep the history, commits, branches and tags.
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]. Updated in 2017, this is really good but be aware that it's specific to Mozilla so some of the tips don't apply. Hyrax (i.e., libdap4 and BES) also use their own build system (autotools + make) so most of the configuration information here is very apropos. See also [[How to use Eclipse with Hyrax Source Code]] below.
* [https://jfearn.fedorapeople.org/en-US/RPM/4/html/RPM_Guide/index.html RPM Guide] The best one I'm found so far...
* [https://autotools.io/index.html Autotools Myth busters] The best info on autotools I've found yet (covers ''autoconf'', ''automake'', ''libtool'' and ''pkg-config'').
* The [https://www.gnu.org/software/autoconf/autoconf.html autoconf] manual
* The [https://www.gnu.org/software/automake/ automake] manual
* The [https://www.gnu.org/software/libtool/ libtool] manual
* A good [https://lldb.llvm.org/lldb-gdb.html gdb to lldb cheat sheet] for those of us who know ''gdb'' but not ''lldb''.

= Old information =
'''Note''': Old build information
====The Release Process====
# Make sure the <tt>hyrax-dependencies</tt> project is up to date and tar balls on www.o.o. If there have been changes/updates:
## Update version number for the <tt>hyrax-dependencies</tt> in the <tt>Makefile</tt>
## Save, commit, (merge?), and push the changes to the <tt>master</tt> branch.
## Once the <tt>hyrax-dependencies</tt> CI build is finished, trigger CI builds for both <tt>libdap4</tt> and <tt>bes</tt> by pushing change(s) to the master branch of each.
# [[Source_Release_for_libdap | Making a source release of libdap]]
# [[ReleaseGuide | Making a source release of the BES]].
# [[OLFSReleaseGuide| Make the OLFS release WAR file]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
# [[HyraxDockerReleaseGuide|Make the official Hyrax Docker image for the release]] When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.

====Supplemental release guides====
Old - use the packages built using the Continuous Delivery process
# [[RPM |Make the RPM Distributions]]. Follow these steps to create an RPM distribution of the software. '''Note:''' ''Now we use packages built using CI/CD, so this checklist is no longer needed.''

'''Note''': ''The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...''
* [[MergingBranches |How to merge code]]
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]]. Follow these steps when a new Matlab GUI version is ready to be released.
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
* [[TestOpendapOrg | Notes from setting up Hyrax on our new web host]]
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.

Developer Info

2025-05-20T23:18:21Z

Jimg: /* C++ Coding Information */

* [https://github.com/OPENDAP OPeNDAP's GitHub repositories]: OPeNDAP's software is available using GitHub in addition to the downloads from our website.
** Before 2015 we hosted our own SVN repository. It's still online and available, but for read-only access, at [https://scm.opendap.org/svn https://scm.opendap.org/svn].
* [https://travis-ci.org/OPENDAP Continuous Integration builds]: Software that is built whenever new changes are pushed to the master branch. These builds are done on the Travis-CI system.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* We use the Coverity static system to look for common software defects, information on Hyrax is spread across three projects:
** [https://scan.coverity.com/projects/opendap-bes?tab=overview The BES and the standard handlers we distribute]
** [https://scan.coverity.com/projects/opendap-olfs?tab=overview The OLFS - the front end to the Hyrax data server]
** [https://scan.coverity.com/projects/opendap-libdap4?tab=overview libdap - The implementation of DAP2 and DAP4]

== OPeNDAP's FAQ ==
The [http://www.opendap.org/faq-page OPeNDAP FAQ] has a pretty good section on developer's questions.

== C++ Coding Information ==
* [https://google.github.io/styleguide/cppguide.html Google C++ Style Guide]
* [[Include files for libdap | Guidelines for including headers]]
* [[Using lambdas with the STL]]
* [[Better Unit tests for C++]]
* [[Better Singleton classes C++]]
* [[What is faster? stringstream string + String]]
* [[More about strings - passing strings to functions]]

== OPeNDAP Workshops ==
* [http://www.opendap.org/about/workshops-and-presentations/2007-10-12 The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/about/workshops-and-presentations/2008-07-15 ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.

== libdap4 and BES Reference documentation ==
* [https://opendap.github.io/bes/html/ BES Reference]
* [https://opendap.github.io/libdap4/html/ libdap Reference]

== BES Development Information ==
* [[Hyrax - Logging Configuration|Logging Configuration]]

* [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
* [[BES - Debugging Using besstandalone]]
* [[Hyrax - Create BES Module | How to create your own BES Module]]
* Hyrax Module Integration: How to configure your module so it's easy to add to Hyrax instances ([[:File:HyraxModuleIntegration-1.2.pdf|pdf]])
* [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
* [[Hyrax - Running bescmdln | Running the BES command line client]]
* [[Hyrax - BES Client commands| BES Client commands]]. The page [[BES_XML_Commands | BES XML Commands]] repeats this info for a bit more information on the return values. Most of the commands don't return anything unless they return an error and are expected to be used in a group where a ''get'' command closes out the request and obviously does return a response of some kind (maybe an error).
* [[Hyrax:_BES_Administrative_Commands| BES Administrative Commands]]
* [[Hyrax - Extending BES Module | Extending your BES Module]]
* [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
* [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]

* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]

== OPeNDAP Development process information ==
These pages contain information about how we'd like people working with us to use our various on-line tools.

* [[Planning a Program Increment]] This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
* [[Hyrax GitHub Source Build]] This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
* [[Bug Prioritization]]. How we prioritize bugs in our software.

===[[How to Make a Release|Making A Release]] ===
* [[How to Make a Release]] A general template for making a release. This references some of the pages below.

== Software process issues: ==
* [[How to download test logs from a Travis build]] All of our builds on Travis that run tests save those logs to an S3 bucket.
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]] - Updated 12/2014 to include information regarding git.
* [[How to use CLion with our software]]
* [[BES Timing| How to add timing instrumentation to your BES code.]]
* [[UnitTests| How to write unit tests using CppUnit]] NB: See other information under the heading of C++ development
* [[valgrind| How to use valgrind with unit tests]]
* [[Debugging the distcheck target]] Yes, this gets its own page...
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[Managing public and private keys using gpg]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
* [[UserSupport|How to Handle Email-list Support Questions]]
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [http://semver.org/ Software version numbers]
* [[GuideLines| Development Guidelines]]
* [[Apple M1 Special Needs]]

==== Older info of limited value: ====
* [http://gcc.gnu.org/gcc-4.4/cxx0x_status.html C++-11 gcc/++-4.4 support] We now require compilers that support C++-14, so this is outdated (4/19/23).
* [[How to use Eclipse with Hyrax Source Code]] I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.

==== AWS Tips ====
* [[Growing a CentOS Root Partition on an AWS EC2 Instance]]
* [[How Shutoff the CentOS firewall]]

== General development information ==
These pages contain general information relevant to anyone working with our software:

* '''[[Git Hacks and Tricks]]''': Information about using git and/or GitHub that seems useful and maybe not all that obvious.
* [[Git Secrets]]: securing repositories from AWS secret key leaks.
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind Suppression File Howto] How to build a suppressions file for valgrind.
* [[Using a debugger for C++ with Eclipse on OS/X]] Short version: use lldbmi2 **Add info**
* [[Using ASAN]] Short version, look [https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger at the Google/GitHub pages] for useful environment variables **add text** On Centos, use yum install llvm to get the 'symbolizer' and try ''ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer)''
* [[How to use ''Instruments'' on OS/X to profile]] Updated 7/2018
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind - How to generate suppression files for valgrind] This will quiet valgrind, keeping it from telling you OS/X or Linux (or the BES) is leaking memory.
* [[Migrating source code from SVN to git]]: How to move a large project from SVN to git and keep the history, commits, branches and tags.
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]. Updated in 2017, this is really good but be aware that it's specific to Mozilla so some of the tips don't apply. Hyrax (i.e., libdap4 and BES) also use their own build system (autotools + make) so most of the configuration information here is very apropos. See also [[How to use Eclipse with Hyrax Source Code]] below.
* [https://jfearn.fedorapeople.org/en-US/RPM/4/html/RPM_Guide/index.html RPM Guide] The best one I'm found so far...
* [https://autotools.io/index.html Autotools Myth busters] The best info on autotools I've found yet (covers ''autoconf'', ''automake'', ''libtool'' and ''pkg-config'').
* The [https://www.gnu.org/software/autoconf/autoconf.html autoconf] manual
* The [https://www.gnu.org/software/automake/ automake] manual
* The [https://www.gnu.org/software/libtool/ libtool] manual
* A good [https://lldb.llvm.org/lldb-gdb.html gdb to lldb cheat sheet] for those of us who know ''gdb'' but not ''lldb''.

= Old information =
'''Note''': Old build information
====The Release Process====
# Make sure the <tt>hyrax-dependencies</tt> project is up to date and tar balls on www.o.o. If there have been changes/updates:
## Update version number for the <tt>hyrax-dependencies</tt> in the <tt>Makefile</tt>
## Save, commit, (merge?), and push the changes to the <tt>master</tt> branch.
## Once the <tt>hyrax-dependencies</tt> CI build is finished, trigger CI builds for both <tt>libdap4</tt> and <tt>bes</tt> by pushing change(s) to the master branch of each.
# [[Source_Release_for_libdap | Making a source release of libdap]]
# [[ReleaseGuide | Making a source release of the BES]].
# [[OLFSReleaseGuide| Make the OLFS release WAR file]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
# [[HyraxDockerReleaseGuide|Make the official Hyrax Docker image for the release]] When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.

====Supplemental release guides====
Old - use the packages built using the Continuous Delivery process
# [[RPM |Make the RPM Distributions]]. Follow these steps to create an RPM distribution of the software. '''Note:''' ''Now we use packages built using CI/CD, so this checklist is no longer needed.''

'''Note''': ''The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...''
* [[MergingBranches |How to merge code]]
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]]. Follow these steps when a new Matlab GUI version is ready to be released.
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
* [[TestOpendapOrg | Notes from setting up Hyrax on our new web host]]
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.

DMR++

2024-08-29T18:12:27Z

Jimg: /* Building dmr++ files with get_dmrpp */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'', ''netcdf-4'', and (experimental as of 8/29/24) ''HDF4''/''HDF4-EOS2'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested. However, an external group working on the Python Kerchunk software has developed ''VirtualiZarr'' which can parse either Kerchunk or DMR++ documents and read from data those describe using the Zarr API.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

===''HDF4''/''HDF4-EOS2''===

This is a complicated case, and its support as of 8/29/24 is still considered experimental. The HDF4 data model is quite complex, more so than the HDF5 model, and we focusing on complete support for those features used by NASA. To this end, we are also working on support for HDF4-EOS2, data files that can only be read correctly with the HDF4-EOS2 library. The main distinction of that API is the treatment of values for the Domain variables for Latitude and Longitude. Our support handles the HDF4-EOS Grid data type and using DMR++ the Latitude and Longitude values appear as users expect, although some aspects of this are ongoing. We do not yet support the HDF4-EOS2 Swath data type.

Se the section below for information on the tool for building DMR++ files for HDF4 and HDF4-EOS2 data files.

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.
====Using the server to examine data returned by the DMR++====

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

[[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|text-top|The running server shows the DMR++ as a dataset.]]

Note: ''The server caches data catalog information for 5 minutes (configurable) so new items (e..g., DMR++ documents) may not show up right away. To force the display of a DMR++ that you just created, click on the source data file name and edit the URL so that the suffix '''.dmr.html''' is replaced by '''.dmrpp/dmr'''.''

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|text-top|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

To the right is a comparison of the same underlying data, the left window shows the data returned using the DMR++, the right shows the data read directly from the file using the server's builtin HDF4 reader.
[[File:Data-comparison.png|200px|thumb|right|text-top|Comparison of responses from a DMR++ and the native file handler.]]

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/sh
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''DMR++'' files for HDF5/NetCDF4 with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-29T18:11:48Z

Jimg: /* Supported Data Formats */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'', ''netcdf-4'', and (experimental as of 8/29/24) ''HDF4''/''HDF4-EOS2'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested. However, an external group working on the Python Kerchunk software has developed ''VirtualiZarr'' which can parse either Kerchunk or DMR++ documents and read from data those describe using the Zarr API.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

===''HDF4''/''HDF4-EOS2''===

This is a complicated case, and its support as of 8/29/24 is still considered experimental. The HDF4 data model is quite complex, more so than the HDF5 model, and we focusing on complete support for those features used by NASA. To this end, we are also working on support for HDF4-EOS2, data files that can only be read correctly with the HDF4-EOS2 library. The main distinction of that API is the treatment of values for the Domain variables for Latitude and Longitude. Our support handles the HDF4-EOS Grid data type and using DMR++ the Latitude and Longitude values appear as users expect, although some aspects of this are ongoing. We do not yet support the HDF4-EOS2 Swath data type.

Se the section below for information on the tool for building DMR++ files for HDF4 and HDF4-EOS2 data files.

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.
====Using the server to examine data returned by the DMR++====

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

[[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|text-top|The running server shows the DMR++ as a dataset.]]

Note: ''The server caches data catalog information for 5 minutes (configurable) so new items (e..g., DMR++ documents) may not show up right away. To force the display of a DMR++ that you just created, click on the source data file name and edit the URL so that the suffix '''.dmr.html''' is replaced by '''.dmrpp/dmr'''.''

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|text-top|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

To the right is a comparison of the same underlying data, the left window shows the data returned using the DMR++, the right shows the data read directly from the file using the server's builtin HDF4 reader.
[[File:Data-comparison.png|200px|thumb|right|text-top|Comparison of responses from a DMR++ and the native file handler.]]

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/sh
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-29T00:25:00Z

Jimg: /* Simple shell command */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.
====Using the server to examine data returned by the DMR++====

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

[[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|text-top|The running server shows the DMR++ as a dataset.]]

Note: ''The server caches data catalog information for 5 minutes (configurable) so new items (e..g., DMR++ documents) may not show up right away. To force the display of a DMR++ that you just created, click on the source data file name and edit the URL so that the suffix '''.dmr.html''' is replaced by '''.dmrpp/dmr'''.''

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|text-top|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

To the right is a comparison of the same underlying data, the left window shows the data returned using the DMR++, the right shows the data read directly from the file using the server's builtin HDF4 reader.
[[File:Data-comparison.png|200px|thumb|right|text-top|Comparison of responses from a DMR++ and the native file handler.]]

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/sh
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-29T00:20:56Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.
====Using the server to examine data returned by the DMR++====

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

[[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|text-top|The running server shows the DMR++ as a dataset.]]

Note: ''The server caches data catalog information for 5 minutes (configurable) so new items (e..g., DMR++ documents) may not show up right away. To force the display of a DMR++ that you just created, click on the source data file name and edit the URL so that the suffix '''.dmr.html''' is replaced by '''.dmrpp/dmr'''.''

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|text-top|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

To the right is a comparison of the same underlying data, the left window shows the data returned using the DMR++, the right shows the data read directly from the file using the server's builtin HDF4 reader.
[[File:Data-comparison.png|200px|thumb|right|text-top|Comparison of responses from a DMR++ and the native file handler.]]

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/bash
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-29T00:13:39Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

You will see [[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|text-bottom|The running server shows the DMR++ as a dataset.]]

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|text-bottom|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

Here is a comparison of the same underlying data, the left window shows the data returned using the DMR++, the right shows the data read directly from the file using the server's builtin HDF4 reader.
[[File:Data-comparison.png|200px|thumb|right|text-bottom|Comparison of responses from a DMR++ and the native file handler.]]

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/bash
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

ls -l

shows

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

File:Data-comparison.png

2024-08-29T00:11:19Z

Jimg: Data subset using DMR++ and the native file reader (HDF4-EOS2), compared using Panoply. The data were subset and the response was a netCDF4 file in each case.

== Summary ==
Data subset using DMR++ and the native file reader (HDF4-EOS2), compared using Panoply. The data were subset and the response was a netCDF4 file in each case.

DMR++

2024-08-28T23:59:59Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

<nowiki>usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...</nowiki>

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

<nowiki>[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...</nowiki>

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

You will see [[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|The running server shows the DMR++ as a dataset.]]

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

<nowiki>#!/bin/bash
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF</nowiki>

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

ls -l

shows

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-28T23:43:19Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -f <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details. Skip to '''Simple shell command''' to skip over these details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.20130111.09.7.HDF -u 'file:///usr/share/hyrax/3B42.20130111.09.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory) using the '''-i''' option. The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon, two from the way a URL names a protocol and one because the pathname starts at the root directory. Obscure, but it makes sense.

Building the DMR++ and embedding a ''file://'' URL will enable testing the DMR+.

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

You will see [[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|The running server shows the DMR++ as a dataset.]]

Click on the your equivalent of the '''3B42.20130111.09.7.HDF'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|Use the form interface to subset and get a response.]]

You can run batch tests in lots of files by building many DMR++ documents and then asking the server for various responses (nc4, dap) from the DMR++ and the original file. Those could be compared using various schemes, although in its entirety that is beyond this section's scope, the command ''getdap4'' is also included in the container and could be used to compare ''dap'' responses from the data file and the DMR++ document.

====Simple shell command====

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

''In the spirit of a recipe, I'll restate the earlier command for starting the docker container with the '''get_dmrpp_h4''' command and the '''hyrax''' server.''

Start the container:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

Is it running:

docker ps

The command, written for the Bourne Shell, is:

#!/bin/bash
#
# usage get_dmrpp_h4.sh <file>

data_root=/usr/share/hyrax

cat <<EOF | docker exec --interactive hyrax sh
cd $data_root
get_dmrpp_h4 -i $1 -u "file://$data_root/$1"
EOF

Copy that, save it in a file (I named the file ''get_dmrpp_h4.sh'').

Run the command on the host (not the docker container) and in the directory with the HDF4 files (you don't ''have'' to do that, but sorting out the details is left as an exercise for the reader ;-). Run the command like this:

./get_dmrpp_h4.sh AMSR_E_L3_SeaIce25km_V15_20020601.hdf

The DMR++ will appear when the command completes.

ls -l

shows

(hyrax500) hyrax_git/HDF4-dir % ls -l
total 1251240
-rw-r--r--@ 1 jimg staff 1250778 Aug 22 22:31 AMSR_E_L2_Land_V09_200206191112_A.hdf
-rw-r--r--@ 1 jimg staff 20746207 Aug 22 22:32 AMSR_E_L3_SeaIce25km_V15_20020601.hdf
-rw-r--r-- 1 jimg staff 3378674 Aug 28 17:37 AMSR_E_L3_SeaIce25km_V15_20020601.hdf.dmrpp

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-28T22:57:42Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container '''opendap/hyrax:snapshot''' from Docker Hub. Thw ''snapshot'' is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

The volume mount, from $HDF4_DIR to '/usr/share/hyrax' mounts the current directory of the host computer running the container to the directory ''/usr/share/hyrax'' inside the container. That directory is the root of the server's data tree. This means that the HDF4 files you copied into the HDF4_DIR directory will be accessible by the server running in the container. That will be useful for testing later on.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps -a

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -r <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.19980101.00.7.HDF -u 'file:///usr/share/hyrax/3B42.19980101.00.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory). The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon. Obscure, but it makes sense.

Build ing the DMR++ and embedding a ''file://'' URL will enable testing the DMR++ easily.

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

You will see [[File:Hyrax-including-new-DNRpp.png|200px|thumb|right|Caption]]

Click on the your equivalent of the '''3B42.19980101.00.7.HDF.dmrpp'' link, subset, download and open in Panoply or the equivalent. [[File:Hyrax-subsetting.png|200px|thumb|right|Caption]]

===Command line option===

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-28T22:49:43Z

Jimg: /* Building DMR++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your own needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container from Docker Hub '''opendap/hyrax:snapshot''' this is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps -a

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -r <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

''At the end of this, I'll include a shell script that takes away many of these steps, but the script obscures some aspects of the command that you might want to tweak, so the following shows you all the details.''

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...

Lets build a DMR++ now, by explicitly using the container:

docker exec -it hyrax bash

starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF

...

In that directory, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.19980101.00.7.HDF -u 'file:///usr/share/hyrax/3B42.19980101.00.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory). The '''-u''' option tells the command to embed the URL that follows it in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon. Obscure, but it makes sense.

Build ing the DMR++ and embedding a ''file://'' URL will enable testing the DMR++ easily.

Lets look at how the ''hyrax'' service will treat that data file using the DMR++. In a browser, goto

http://localhost:8080/opendap/

You will see [[File:Hyrax-including-new-DNRpp.png|200px|thumb|left|Caption]]

Click on the your equivalent of the '''3B42.19980101.00.7.HDF.dmrpp'' link:
[[File:Hyrax-subsetting.png|200px|thumb|left|Caption]]

Also note that the volume mount, from $HDF4_DIR to '/usr/share/hyrax' is a trick to get the current directory (where we are running these commands) to be the default directory for the server.

===Command line option===

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

File:Hyrax-subsetting.png

2024-08-28T22:47:53Z

Jimg: Click on the file with the .dmrpp extension and use the Hyrax interface to subset the data, as this example shows. You can use a tool like Panoply to plot the data returned.

== Summary ==
Click on the file with the .dmrpp extension and use the Hyrax interface to subset the data, as this example shows. You can use a tool like Panoply to plot the data returned.

File:Hyrax-including-new-DNRpp.png

2024-08-28T22:44:03Z

Jimg: This shows that the hyrax server running in the Docker container that also contains the get_dmrpp_h4 command can be used along with the server's web interface to examine the DMR++. The interface can be used to download various response types using the DMR++.

== Summary ==
This shows that the hyrax server running in the Docker container that also contains the get_dmrpp_h4 command can be used along with the server's web interface to examine the DMR++. The interface can be used to download various response types using the DMR++.

DMR++

2024-08-28T22:01:18Z

Jimg: /* Building dmr++ files for HDF4 and HDF4-EOS2 (experimental) */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''DMR++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

''Because this command is still experimental, I'll write this documentation like a recipe. Modify it to suit your on needs.''

===Using get_dmrpp_h4===
Make a new directory in a convenient place and copy the HDF4 and/or HDF4-EOS2 files in that directory. Once you have the files in that directory, make an environment variable so it can be referred to easily. From inside the directory:

export HDF4_DIR=$(pwd)

Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container from Docker Hub '''opendap/hyrax:snapshot''' this is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

Note: If you want to use a specific container version, just substitute the version info for 'snapshot.'

Check that the container is running using:

docker ps -a

This will show a somewhat hard-to-read bit of information about all the running Docker container on you host:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
2949d4101df4 opendap/hyrax:snapshot "/entrypoint.sh -" 15 seconds ago Up 14 seconds 8009/tcp, 8443/tcp,
10022/tcp, 11002/tcp, 0.0.0.0:8080->8080/tcp hyrax

If you want to stop containers, use

docker rm -r <CONTAINER ID>

where the ''CONTAINER ID'' for the one we just started and shown in the output of ''docker ps -a'' above is ''2949d4101df4''. No need to stop the container now, I'm just pointing out how to do it because it's often useful.

====Lets run the DMR++ builder====

At the end of this, I'll include a shell script that takes away many of these steps, but it obscures some aspects of the command that you might want to tweak, so the following shows you all the deatils.

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

docker exec -it hyrax get_dmrpp_h4 -h

will return:

usage: get_dmrpp_h4 [-h] -i I [-c CONF] [-s] [-u DATA_URL] [-D] [-v]

Build a dmrpp file for an HDF4 file. get_dmrpp_h4 -i h4_file_name. A dmrpp
file that uses the HDF4 file name will be generated.

optional arguments:

...

Lets build a DMR++ now, using the by explicitly using the container:

docker exec -it hyrax bash

which starts the ''bash'' shell in the container, with the current directory as root (/)

[root@hyrax /]#

Change to the directory that is the root of the data (you'll see your HDF4 files in here):

cd /usr/share/hyrax

You will see, roughly:

[root@hyrax /]# cd /usr/share/hyrax
[root@hyrax hyrax]# ls
3B42.19980101.00.7.HDF
3B42.19980101.03.7.HDF
3B42.19980101.06.7.HDF
...

In that directory while inside the container, use the ''get_dmrpp_h4'' command to build a DMR++ document for one of the files:

[root@hyrax hyrax]# get_dmrpp_h4 -i 3B42.19980101.00.7.HDF -u 'file:///usr/share/hyrax/3B42.19980101.00.7.HDF'

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass ''get_dmrpp_h4'' the name of the file (because it's local to the current directory). The '''-u''' option tells the command to embed the following URL in the DMR++. I've used a ''file://'' URL to the file ''/usr/share/hyrax/3B42.19980101.00.7.HDF''.

Note the three slashes following the colon. Obscure, but it makes sense.

Build ing the DMR++ and embedding a ''file://'' URL will enable testing the DMR++ easily.

Also note that the volume mount, from $HDF4_DIR to '/usr/share/hyrax' is a trick to get the current directory (where we are running these commands) to be the default directory for the server.

===Command line option===

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-28T21:10:10Z

Jimg: here

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''dmr++'' files for HDF4 and HDF4-EOS2 (experimental)==
The HDF4 and HDF4-EOS2 (hereafter just HDF4) DMR++ document builder is currently available in the docker container we build for ''hyrax'' server/service. You can get this container from the public Docker Hub repository. You can also get and build the ''Hyrax'' source code, and use the client that way (as part of a source code build) but it's much more complex than getting the Docker container. In addition, the Docker container includes a server that can test the DMR++ documents that are built and can even show you how the files would look when served without using the DMR++.

===Using get_dmrpp_h4===
Get the Docker container from Docker Hub using this command:

docker run -d -h hyrax -p 8080:8080 -v $HDF4_DIR:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot

What the options mean:
-d, --detach Run container in background and print container ID
-h, --hostname Container host name
-p, --publish Publish a container's port(s) to the host
-v, --volume Bind mount a volume
--name Assign a name to the container

This command will fetch the container from Docker Hub '''opendap/hyrax:snapshot''' this is the latest build of the container. It will then ''run'' the container and return the container ID. The ''hyrax'' server is now running on you computer and can be accessed with a web browser, curl, et cetera. More on that in a bit.

If you want to use a specific container, just substitute the version info for 'snapshot.' Also note that the volume mount, from $HDF4_DIR to '/usr/share/hyrax' is a trick to get the current directory (where we are running these commands) to be the default directory for the server.

===Command line option===

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

DMR++

2024-08-28T20:40:23Z

Jimg: /* Building dmr++ files with get_dmrpp */

''How to build & deploy dmr++ files for Hyrax''

==What? Why?==

It is a fast and flexible way to serve data stored in S3.
The dmr++ encodes the location of the data content residing in a binary data file/object (e.g., an hdf5 file) so that it can be directly accessed, without the need for an intermediate library API, by using the file with the location information. The binary data objects may be on a local filesystem, or they may reside across the web in something like an S3 bucket.

==How Does It Work?==

The ''dmr++'' ingest software reads a data file (see '''note''') and builds a document that holds all of the file's metadata (the names and types of all of the variables along with any other information bound to those variables). This information is stored in a document we call the Dataset Metadata Response (DMR). The ''dmr++'' adds some extra information to this (that's the '++' part) about where each variable can be found and how to decode those values. The ''dmr++'' is simply an special annotated DMR document.

This effectively decouples the annotated DMR (''dmr++'') from the location of the granule file itself. Since dmr++ files are typically significantly smaller than the source data granules they represent, they can be stored and moved for less expense. They also enable reading all of the file's metadata in one operation instead of the iterative process that many APIs require.

If the dmr++ contains references to the source granules location on the web, the location of the the dmr++ file itself does not matter.

Software that understands the dmr++ content can directly access the data values held in the source granule file, and it can do so without having to retrieve the entire file and work on it locally, even when the file is stored in a Web Object Store like S3.

If the granule file contains multiple variables and only a subset of them are needed, the dmr++ enabled software can retrieve just the bytes associated with the desired variables parts.

'''note:''' The OPeNDAP software currently supports HDF5 and NetCDF4. Other formats can be supported, such as zarr.

==Supported Data Formats==

The ''dmr++'' software currently works with ''hdf5'' and ''netcdf-4'' files. (The ''netcdf-4'' format is a subset of ''hdf5'' so ''hdf5'' tools are utilized for both.) Other formats like ''zarr'', ''hdf4'', ''netcdf-3'' are not currently supported by the ''dmr++'' software, but support could be added if requested.

===''hdf5''===

The ''hdf5'' data format is quite complex and many of the options and edge cases are not currently supported by the ''dmr++'' software.

These limitations and how to quickly evaluate an ''hdf5'' or ''netcdf-4'' file for use with the ''dmr++'' software are explained below.

====''hdf5'' filters====

The ''hdf5'' format has several filter/compression options used for storing data values.
The ''dmr++'' software currently supports data that utilize the H5Z_FILTER_DEFLATE, H5Z_FILTER_SHUFFLE, and H5Z_FILTER_FLETCHER32 filters.
[https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html You can find more on hdf5 filters here.]

====''hdf5'' storage layouts====

The ''hdf5'' format also uses a number of "storage layouts" that describe various structural organizations of the data values associated with a variable in the granule file.
The ''dmr++'' software currently supports data that utilize the H5D_COMPACT, H5D_CHUNKED, and H5D_CONTIGUOUS storage layouts. These are all of the storage layouts defined by the ''hdf5'' library, but others can be added.
[https://support.hdfgroup.org/HDF5/doc1.6/Datasets.html You can find more on hdf5 storage layouts here.]

====Is my ''hdf5'' or ''netcdf-4'' file suitable for ''dmr++''?====

To determine the ''hdf5'' filters, storage layouts, and chunking scheme used in an ''hdf5'' or ''netcdf-4'' file you can use the command:
<code>h5dump -H -p <filename></code>
To get a human readable assessment of the file that will show the storage layouts, chunking structure, and the filters needed for each variable (aka DATASET in the ''hdf5'' vocabulary) [https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump h5dump info can be found here.]

''h5dump example output:''
$ h5dump -H -p chunked_gzipped_fourD.h5
HDF5 "chunked_gzipped_fourD.h5" {
GROUP "/" {
DATASET "d_16_gzipped_chunks" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 40, 40, 40, 40 ) / ( 40, 40, 40, 40 ) }
STORAGE_LAYOUT {
CHUNKED ( 20, 20, 20, 20 )
SIZE 2863311 (3.576:1 COMPRESSION)
}
FILTERS {
COMPRESSION DEFLATE { LEVEL 6 }
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}

====Is my netcdf file ''netcdf-3'' or ''netcdf-4''?====

It is an unfortunate state of affairs that the file suffix ".nc" is the commonly used naming convention for both ''netcdf-3'' and ''netcdf-4'' files.
You can use the command:
<code>ncdump -k <filename></code>
to determine if a ''netcdf'' file is either classic ''netcdf-3'' (classic) or ''netcdf-4'' (netCDF-4).
* The ''netcdf'' library must be installed on the system upon which the command is issued.

[http://www.bic.mni.mcgill.ca/users/sean/Docs/netcdf/guide.txn_79.html You can learn more in the NetCDF documentation here.]

==Building ''dmr++'' files for HDF4 and HDF4-EOS2 (experimental)==

==Building ''dmr++'' files with get_dmrpp==

The application that builds the ''dmr++'' files is a command line tool called ''get_dmrpp''. It in turn utilizes other executables such as ''build_dmrpp'', ''reduce_mdf'', ''merge_dmrpp'' (which rely in turn on the ''hdf5_handler'' and the ''hdf5'' library), along with a number of UNIX shell commands.

All of these components are install with each recent version of the Hyrax Data Server

You can see the ''get_dmrpp'' usage statement with the command: <code>get_dmrpp -h</code>

===Using ''get_dmrpp''===

The way that ''get_dmrpp'' is invoked controls the way that the data are ultimately represented in the resulting ''dmr++'' file(s).

The ''get_dmrpp'' application utilizes software from the Hyrax data server to produce the base DMR document which is used to construct the ''dmr++'' file.

The Hyrax server has a long list of configuration options, several of which can substantially alter the the structural and semantic representation of the dataset as seen in the dmr++ files generated using these options.

===Command line options===

The command line switches provide a way to control the output of the tool. In addition to common options like verbose output or testing modes, the tool provides options to build extra (aka 'sidecar') data files that hold information needed for CF compliance if the original HDF5 data files lack that information (see the ''missing data'' section ). In addition, it is often desirable to build ''dmr++'' files before the source data files are uploaded to a cloud store like S3. In this case, the URL to the data may not be known when the ''dmr++'' is built. We support this by using placeholder/template strings in the ''dmr++'' and which can then be replaced with the URL at runtime, when the ''dmr++'' file is evaluated. See the '-u' and '-p' options below.

====Inputs====

; -b
: The fully qualified path to the top level data directory. Data files read by ''get_dmrpp'' must be in the directory tree rooted at this location and their names expressed as a path relative to this location. The value may not be set to <code>/</code> or <code>/etc</code> The default value is /tmp if a value is not provided. All the data files to be processed must be in this directory or one of its subdirectories. If ''get_dmrpp'' is being executed from same directory as the data then <code>-b `pwd`</code> or <code>-b .</code> works as well.
;-u
: This option is used to specify the location of the binary data object. It’s value must be an http, https, or file (file://) URL. This URL will be injected into the dmr++ when it is constructed. If option -u is not used; then the template string <code>OPeNDAP_DMRpp_DATA_ACCESS_URL</code> will be used and the dmr++ will substitute a value at runtime.
;-c
:The path to an alternate bes configuration file to use.
;-s
:The path to an optional addendum configuration file which will be appended to the default BES configuration. Much like the site.conf file works for the full server deployment it will be loaded last and the settings there-in will have an override effect on the default configuration.

====Output====

; -o
: The name of the file to create.

====Verbose Output Modes====

; -h
: Show help/usage page.
; -v:
: verbose mode, prints the intermediate DMR.
; -V
: Very verbose mode, prints the DMR, the command and the configuration file used to build the DMR.
; -D
: Just print the DMR that will be used to build the DMR++
; -X
: Do not remove temporary files. May be used independently of the -v and/or -V options.

====Tests====

; -T
: Run ALL hyrax tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -I
: Run hyrax inventory tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.
; -F
: Run hyrax value probe tests on the resulting dmr++ file and compare the responses the ones generated by the source hdf5 file.

====Missing Data Creation====

; -M
: Build a 'sidecar' file that holds missing information needed for CF compliance (e.g., Latitude, Longitude and Time coordinate data).
; -p
: Provide the URL for the Missing data sidecar file. If this is not given (but -M is), then a template value is used in the dmr++ file and a real URL is substituted at runtime.
; -r
: The path to the file that contains missing variable information for sets of input data files that share common missing variables. The file will be created if it doesn't exist and the result may be used in subsequent invocations of get_dmrpp (using -r) to identify the missing variable file.

====AWS Integration====
The <tt>get_dmrpp</tt> application supports both S3 hosted granules as inputs, and uploading generated dmr++ files to an S3 bucket.

; S3 Hosted granules are supported by default,
: When the <tt>get_dmrpp</tt> application sees that the name of the input file is an S3 URL it will check to see if the AWS CLI is configured and if so <tt>get_dmrpp</tt> will attempt retrieve the granule and make a dmr++ utilizing whatever other options have been chosen.
:: Example: '''<tt>get_dmrpp -b `pwd` s3://bucket_name/granule_object_id</tt>'''

; -U
: The '''<tt>-U</tt>''' command line parameter for <tt>get_dmrpp</tt> instructs the <tt>get_dmrpp</tt> application to upload the generated dmr++ file to S3, but only when the following conditions are met:
:* The name of the input file is an S3 URL
:* The AWS CLI has been configured with credentials that provide r+w permissions for the bucket referenced in the input file S3 URL.
:* The <tt>-U</tt> option has been specified.
: If all three of the above are true then <tt>get_dmrpp</tt> will copy the retrieve the granule, create a dmr++ file from the granule, and copy the resulting dmr++ file (as defined by the -o option) to the source S3 bucket using the well known NGAP sidecar file naming convention: '''<tt>s3://bucket_name/granule_object_id.dmrpp</tt>'''
:: Example: '''<tt>get_dmrpp -U -o foo -b `pwd` s3://bucket_name/granule_object_id</tt>'''

===''hdf5_handler'' Configuration===

Because ''get_dmrpp'' uses the ''hdf5_handler'' software to build the ''dmr++'' the software must inject the ''hdf5_handler'''s configuration.

The default configuration is large, but any valued may be altered at runtime.

Here are some of the commonly manipulated configuration parameters with their default values:
H5.EnableCF=true
H5.EnableDMR64bitInt=true
H5.DefaultHandleDimension=true
H5.KeepVarLeadingUnderscore=false
H5.EnableCheckNameClashing=true
H5.EnableAddPathAttrs=true
H5.EnableDropLongString=true
H5.DisableStructMetaAttr=true
H5.EnableFillValueCheck=true
H5.CheckIgnoreObj=false

====''Note to DAACs with existing Hyrax deployments.''====

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into ''get_dmrpp''.

===The ''H5.EnableCF'' option===

Of particular importance is the ''H5.EnableCF'' option, which instructs the ''get_dmrpp'' tool to produce [https://cfconventions.org/ Climate Forecast convention (CF)] compatible output based on metadata found in the granule file being processed.

Changing the value of ''H5.EnableCF'' from '''''false''''' to '''''true''''' will have (at least) two significant effects.

It will:

* Cause ''get_dmrpp'' to attempt to make the dmr++ metadata CF compliant.
* Remove Group hierarchies (if any) in the underlying data granule by flattening the Group hierarchy into the variable names.

By default ''get_dmrpp'' the ''H5.EnableCF'' option is set to false:
H5.EnableCF = false

There is a much more comprehensive discussion of this key feature, and others, in the
'''''[https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_hyrax_handlers HDF5 Handler section of the Appendix in the Hyrax Data Server Installation and Configuration Guide]'''''

===Missing data, the CF conventions and ''hdf5''===

Many of the ''hdf5'' files produced by NASA and others do not contain the domain coordinate data (such as latitude, longitude, time, etc.) as a collection of explicit values. Instead information contained in the dataset metadata can used to reproduce these values.

In order for a dataset to be Climate Forecast (CF) compatible it must contain these domain coordinate data values.

The Hyrax ''hdf5_handler'' software, utilized by the ''get_dmrpp'' application, can create this data from the dataset metadata. The ''get_dmrpp'' application places these generated data in a “sidecar” file for deployment with the source ''hdf5/netcdf''-4 file.

==Hyrax - Serving data using dmr++ files==

There are three fundamental deployment scenarios for using dmr++ files to serve data with the Hyrax data server.

This can be simple categorized as follows:
The dmr++ file(s) are XML files that contain a root <tt>dap4:Dataset</tt> element with a <tt>dmrpp:href</tt> attribute whose value is one of:
# An http(s):// URL referencing to the underlying granule files via http.
# A file:// URL that references the granule file on the local filesystem in a location that is inside the BES' data root tree.
# The template string <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt>

Each will discussed in turn below.

'''Note''': By default Hyrax will automatically associate files whose name ends with ".dmrpp" with the dmr++ handler.

===Using dmr++ with http(s) URLs===

If the dmr++ files that you wish to serve contain <tt>dmrpp:href</tt> attributes whose values are http(s) URLs then there are 2+1 steps to serve the data:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure that the Hyrax AllowedHosts list is configured to allow Hyrax to access those target URLs. This can be accomplished by adding new regex entires to the AllowedHosts list in /etc/bes/site.conf, creating that file as need be.
# If the data URLs require authentication to access then you'll need to configure Hyrax for that too.

===Using dmr++ with file URLs===

Using dmr++ files with locally held files can be useful for verifying that dmr++ functionality is working without relying on network access that may have data rate limits, authenticated access configuration, or security access constraints. Additionally, in many cases the dmr++ access to the locally held data may be significantly faster than through the native netcdf-4/hdf5 data handlers.

In order to use dmr++ files that contain file:// URLs:
# Place the dmr++ files on the local disk inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration
# Ensure the the dmr++ files contain only file:// URLs that refer to data granule files inside of the directory tree identified by the <tt>BES.Catalog.catalog.RootDirectory</tt> in the BES configuration.

Note: For Hyrax, a correctly formatted file URL must start with the protocol <tt>file://</tt> followed by the full qualified path to the data granule, for example:
/usr/share/hyrax/ghrsst/some_granule.h5
so the the completed URL will have three slashes after the first colon:
file:///usr/share/hyrax/ghrsst/some_granule.h5

===Using dmr++ with the template string. (NASA)===

Another way to serve dmr++ files with Hyrax is to build the dmr++ files '''without''' valid URLs but with a template string that is replaced at runtime. If no target URL is supplied to get_drmpp at the time that the dmr++ is generated the template string: <tt>OPeNDAP_DMRpp_DATA_ACCESS_URL</tt> will added to the file in place of the URL. The at runtime it can be replaced withe the correct value.

Currently the only implementation of this is Hyrax's NGAP service which, when deployed in the NASA NGAP cloud, will accept "restified path" URLs that are defined as
having a URL path component with two mandatory and one optional parameters:
MANDATORY: "/collections/UMM-C:{concept-id}"
OPTIONAL: "/UMM-C:{ShortName} '.' UMM-C:{Version}"
MANDATORY: "/granules/UMM-G:{GranuleUR}"
'''Example:''' <tt>https://opendap.earthdata.nasa.gov/collections/C1443727145-LAADS/MOD08_D3.v6.1/granules/MOD08_D3.A2020308.061.2020309092644.hdf.nc</tt>

When encountering this type of URL Hyrax will decompose it and use the content to formulate a query to the NASA CMR in order to retrieve the data access URL for the granule and for the dmr++ file. It then retrieves the dmr++ file and injects the data URL so that data access can proceed as described above.

[https://wiki.earthdata.nasa.gov/display/DUTRAIN/Feature+analysis%3A+Restified+URL+for+OPENDAP+Data+Access More on the Restified Path can be found here],

== Recipe: Building and testing dmr++ files ==
There are two recipes shown here, one using Hyrax docker containers and a second using the container that is part of the EOSDIS Cumulous task.
Prerequisites:
* Docker daemon running on a system that also supports a shell (the examples use <tt>bash</tt> in this section)
=== Recipe: Building dmr++ files using a Hyrax docker container.===
# Acquire representative granule files for the collection you wish to import. Put them on the system that is running the Docker daemon. For this recipe we will assume that these files have been placed in the directory:
#: <tt>/tmp/dmrpp</tt>
# Get the most up to date Hyrax docker image:
#: <tt>docker pull opendap/hyrax:snapshot</tt>
# Start the docker container, mounting your data directory on to the docker image at <tt>/usr/share/hyrax</tt>:
#: <tt>docker run -d -h hyrax -p 8080:8080 --volume /tmp/dmrpp:/usr/share/hyrax --name=hyrax opendap/hyrax:snapshot</tt>
# Get a first view of your data using <tt>get_dmrpp</tt> with it's default configuration.
## If you want you can build a dmr++ for an example "<tt>input_file</tt>" using a <tt>docker exec</tt> command:
##: <tt>docker exec -it hyrax get_dmrpp -b /usr/share/hyrax -o /usr/share/hyrax/input_file.dmrpp -u "file:///usr/share/hyrax/input_file" "input_file"</tt>
## Or if you want more scripting flexibility you can login to the docker conainer to do the same:
### Login to the docker container:
###: <tt>docker exec -it hyrax /bin/bash</tt>
### Change working dir to data dir:
###:<tt>cd /usr/share/hyrax</tt>
### This sets the data directory to the current one (<tt>-b $(pwd)</tt>) and sets the data URL (<tt>-u</tt>) to the fully qualified path to the input file.
###: <tt>get_dmrpp -b $(pwd) -o foo.dmrpp -u "file://"$(pwd)"/your_test_file" "your_test_file"</tt>
#: '''''Now that you have made a dmr++ file, use the running Hyrax server to view and test it by pointing your browser at:'''''
#:: '''<tt>http://localhost:8080/opendap/</tt>'''
# You can also batch process all of your test granules, if you want to go that route. This script assumes your ingestable data files end with '<tt>.h5</tt>'.
#: ''The resulting dmr++ files should contain the correct <tt>file://</tt> URLs and be correctly located so that they may be tested with the Hyrax service running in the docker instance.''
<pre>
#!/bin/bash
# This script will write each output file as a sidecar file into
# the same directory as its associated input granule data file.

# The target directory to search for data files
target_dir=/usr/share/hyrax
echo "target_dir: ${target_dir}";

# Search the target_dir for names matching the regex \*.h5
for infile in `find "${target_dir}" -name \*.h5`
do
echo " Processing: ${infile}"

infile_base=`basename "${infile}"`
echo "infile_base: ${infile_base}"

bes_dir=`dirname "${infile}"`
echo " bes_dir: ${bes_dir}"

outfile="${infile}.dmrpp"
echo " Output: ${outfile}"

get_dmrpp -b "${bes_dir}" -o "${outfile}" -u "file://${infile}" "${infile_base}"
done
</pre>

'''''Remember that you can use the Hyrax server that is running in the docker container to view and test the dmr++ files you just created by pointing your browser at:'''''
:: '''<tt>http://localhost:8080/opendap/</tt>'''

=== Testing and qualifying dmr++ files ===
In the previous section/step we created some initial dmr++ files using the default configuration. It is crucial to make sure that they provide the representation of the data that you and your users are expecting, and that they will work correctly with the Hyrax server. (See the following sections for details). If the generated dmr++ files do not match expectations then the default configuration of the <tt>get_dmrpp</tt> may need to be amended using the <tt>-s</tt> parameter.
If the data are currently being served by your DAAC's on-prem team this is where understanding exactly what the localizations made to the configurations of the on-prem Hyrax instances deployed for the collection is important. These localization will probably need to be injected into <tt>get_drmpp</tt> in order to produce the correct data representation in the dmr++ files.

=== Flattening Groups ===
By default <tt>get_dmrpp</tt> will preserve and show group hierarchies. If this is not desired, say for CF-1.0 compatibility, then you can change this by creating a small amendment to <tt>get_dmrpp</tt>'s default configuration.
First create the amending configuration file:
echo "H5.EnableCF=true" > site.conf
Then, change the invocation of <tt>get_dmrpp</tt> in the above example by adding the <tt>-s</tt> switch:
get_dmrpp -s site.conf -b `pwd` -o "${dmrpp_file}" -u "file://"`pwd`"/${file}" "${file}"
And re-run the dmr++ production as shown above.

=== DAP representations ===
We have test and assurance procedures for DAP4 and DAP2 protocols below. Both are important. For legacy datasets the DAP2 request API is widely used by an existing client base and should continue to be supported. Since DAP4 subsumes DAP2 (but with somewhat different API semantics) It should be checked for legacy datasets as well. For more modern datasets that content DAP4 types such as Int64 that are not part of the DAP2 specification or implementations we will need to relying eliding the instances of unmapped types, or return an error when this is encountered.
# Test Constants:
GRANULE_FILE="some_name.h5"
# Granule URL
gf_url="http://localhost:8080/opendap/${GRANULE_FILE}"

==== Inspect the dmr++ files====
# Do the dmr++ files have the expected dmrpp:href URL(s)?
#: <tt>head -2 ${GRANULE_FILE}.dmrpp</tt>

==== Check DAP4 DMR Response ====
Inspect <tt>${gf_url}.dmrpp.dmr</tt>
# Get the document, save as <tt>foo.dmr</tt>:
#: <tt>curl -L -o foo.dmr "${gf_url}.dmr"</tt>
# Is each variable's data type correct and as expected?
# Are the associated dimensions correct?

==== DAP4 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://docs.opendap.org/index.php?title=DAP4:_Specification_Volume_1#Fully_Qualified_Names VARIABLE_NAME is a full qualified DAP4 name.]):
:<tt> curl -L -o dap4_subset_file "${gf_url}.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> curl -L -o dap4_subset_dmrpp "${gf_url}.dmrpp.dap?dap4.ce=VARIABLE_NAME"</tt>
:<tt> cmp dap4_subset_file dap4_subset_dmrpp</tt>

==== DAP4 UI test ====
* View and exercise the DAP4 Data Request Form <tt>{gf_url}.dmr.html</tt>

==== DAP2 Check DDS Response ====

# Inspect <tt>${gf_url}.dds</tt>
## Is each variable's data type correct and as expected?
## Are the associated dimensions correct?
# Compare DMR++ DDS with granule file DDS.
#:For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
#::<tt> curl -L -o dap2_dds_file "${gf_url}.dds"</tt>
#::<tt> curl -L -o dap2_dds_dmrpp "${gf_url}.dds"</tt>
#::<tt> cmp dap2_dds_file dap2_dds_dmrpp </tt>

==== DAP2 Check binary data response ====

For a particular granule GRANULE_FILE and a particular variable VARIABLE_NAME (Where [https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf VARIABLE_NAME is a DAP2 name.]):
:<tt> curl -L -o dap2_subset_file "${gf_url}.dods?VARIABLE_NAME"</tt>
:<tt> curl -L -o dap2_subset_dmrpp "${gf_url}.dmrpp.dods?VARIABLE_NAME"</tt>
:<tt> cmp dap2_subset_file dap2_subset_dmrpp</tt>

'''Note''': One might consider doing this with two or more variables.

==== DAP2 UI Test ====
* View and exercise the DAP2 Data Request Form located here: <tt>{gf_url}.html</tt>
* Try it in Panoply!
** Open Panoply.
** From the '''File''' menu select '''Open Remote Dataset...'''
** Paste the <tt>{gf_url}.html</tt> into the resulting dialog box.
---

Hyrax GitHub Source Build

2024-06-07T03:12:24Z

Jimg: /* Rocky 8 */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a CentOS machine to build code =
== Setup CentOS-7 ==
Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the CentOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. You can use this to extrapolate how to configure Ubuntu and OSX (We routinely build on those platforms as well). The complete instructions are in [[ConfigureCentos | Configure CentOS]] and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

;Update the VM
:<tt>'''yum -y update'''</tt>

;Load a basic software development environment:
:<tt>'''yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant ant-junit junit'''</tt> (it's likely that you can use more recent versions of Java)
:<tt>'''yum install git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

; The whole thing, with java-1.8.0
:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant ant-junit junit git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

;Optional
:Download, unpack, build and install the GNU autotools (''but '''don't''' do this unless the versions installed using yum don't work'')
* autoconf <tt>'''[http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz autoconf-2.69.tar.gz]'''</tt>
* automake <tt>'''[http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz automake-1.14.1.tar.gz]'''</tt>
* libtool <tt>'''[http://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz libtool-2.4.2.tar.gz]'''</tt>
:build them (<tt>'''''./configure; make; sudo make install '''''</tt> - this should take no more than three minutes).

== Setup CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>yum -y update</tt>

;You will need to enable power-tools for this setup
:<tt>yum config-manager --set-enabled powertools</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel</tt>

;Tell the machine where to find the tirpc libraries
:<tt>export CPPFLAGS=-I/usr/include/tirpc</tt>
:<tt>export LDFLAGS=-ltirpc</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>sudo yum install -y python3</tt>

;If you're going to build RPMs
:<tt>yum install rpm-devel rpm-build redhat-rpm-config</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== Rocky 8 ==
''Updated 6/6/2024''

To get the commands ps, which, etc.
dnf install -y procps

C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf config-manager --set-enabled powertools

Install CppUnit and some more development libraries
dnf install -y cppunit cppunit-devel openjpeg2-devel jasper-devel

Install the RPM tools
dnf install -y rpm-devel rpm-build redhat-rpm-config

Install the AWS CLI
dnf install -y awscli

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
Use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the directory set to ''$prefix'')
<tt>
;git clone https://github.com/opendap/hyrax-dependencies
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.
git submodule update --init # update the submodules</pre>
That will clone some additional modules into the directory ''modules''; you need to do this! (Previously it was an optional step). See [http://git-scm.com/docs/git-submodule git submodule] for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use ''git submodule foreach 'git checkout master' ''.
<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-70.cgi Tomcat 7 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-7.0.57.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-7.0.57/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;besctl stop
;./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Hyrax GitHub Source Build

2024-06-07T02:54:23Z

Jimg: /* Setup CentOS-8 */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a CentOS machine to build code =
== Setup CentOS-7 ==
Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the CentOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. You can use this to extrapolate how to configure Ubuntu and OSX (We routinely build on those platforms as well). The complete instructions are in [[ConfigureCentos | Configure CentOS]] and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

;Update the VM
:<tt>'''yum -y update'''</tt>

;Load a basic software development environment:
:<tt>'''yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant ant-junit junit'''</tt> (it's likely that you can use more recent versions of Java)
:<tt>'''yum install git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

; The whole thing, with java-1.8.0
:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant ant-junit junit git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

;Optional
:Download, unpack, build and install the GNU autotools (''but '''don't''' do this unless the versions installed using yum don't work'')
* autoconf <tt>'''[http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz autoconf-2.69.tar.gz]'''</tt>
* automake <tt>'''[http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz automake-1.14.1.tar.gz]'''</tt>
* libtool <tt>'''[http://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz libtool-2.4.2.tar.gz]'''</tt>
:build them (<tt>'''''./configure; make; sudo make install '''''</tt> - this should take no more than three minutes).

== Setup CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>yum -y update</tt>

;You will need to enable power-tools for this setup
:<tt>yum config-manager --set-enabled powertools</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel</tt>

;Tell the machine where to find the tirpc libraries
:<tt>export CPPFLAGS=-I/usr/include/tirpc</tt>
:<tt>export LDFLAGS=-ltirpc</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>sudo yum install -y python3</tt>

;If you're going to build RPMs
:<tt>yum install rpm-devel rpm-build redhat-rpm-config</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== Rocky 8 ==
''Updated 6/6/2024''

To get the commands ps, which, etc.
dnf install -y procps

C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf config-manager --set-enabled powertools

Install CppUnit and some more development libraries
dnf install -y cppunit cppunit-devel openjpeg2-devel jasper-devel

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
Use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the directory set to ''$prefix'')
<tt>
;git clone https://github.com/opendap/hyrax-dependencies
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.
git submodule update --init # update the submodules</pre>
That will clone some additional modules into the directory ''modules''; you need to do this! (Previously it was an optional step). See [http://git-scm.com/docs/git-submodule git submodule] for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use ''git submodule foreach 'git checkout master' ''.
<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-70.cgi Tomcat 7 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-7.0.57.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-7.0.57/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;besctl stop
;./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Hyrax GitHub Source Build

2024-06-07T02:51:40Z

Jimg: /* Rocky 8 */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a CentOS machine to build code =
== Setup CentOS-7 ==
Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the CentOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. You can use this to extrapolate how to configure Ubuntu and OSX (We routinely build on those platforms as well). The complete instructions are in [[ConfigureCentos | Configure CentOS]] and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

;Update the VM
:<tt>'''yum -y update'''</tt>

;Load a basic software development environment:
:<tt>'''yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant ant-junit junit'''</tt> (it's likely that you can use more recent versions of Java)
:<tt>'''yum install git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

; The whole thing, with java-1.8.0
:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant ant-junit junit git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

;Optional
:Download, unpack, build and install the GNU autotools (''but '''don't''' do this unless the versions installed using yum don't work'')
* autoconf <tt>'''[http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz autoconf-2.69.tar.gz]'''</tt>
* automake <tt>'''[http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz automake-1.14.1.tar.gz]'''</tt>
* libtool <tt>'''[http://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz libtool-2.4.2.tar.gz]'''</tt>
:build them (<tt>'''''./configure; make; sudo make install '''''</tt> - this should take no more than three minutes).

== Setup CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>'''yum -y update'''</tt>

;You will need to enable power-tools for this setup
:<tt>'''yum config-manager --set-enabled powertools'''</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel'''</tt>

;Tell the machine where to find the tirpc libraries
:<tt>'''export CPPFLAGS=-I/usr/include/tirpc'''</tt>
:<tt>'''export LDFLAGS=-ltirpc'''</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>'''sudo yum install -y python3'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== Rocky 8 ==
''Updated 6/6/2024''

To get the commands ps, which, etc.
dnf install -y procps

C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf config-manager --set-enabled powertools

Install CppUnit and some more development libraries
dnf install -y cppunit cppunit-devel openjpeg2-devel jasper-devel

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
Use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the directory set to ''$prefix'')
<tt>
;git clone https://github.com/opendap/hyrax-dependencies
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.
git submodule update --init # update the submodules</pre>
That will clone some additional modules into the directory ''modules''; you need to do this! (Previously it was an optional step). See [http://git-scm.com/docs/git-submodule git submodule] for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use ''git submodule foreach 'git checkout master' ''.
<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-70.cgi Tomcat 7 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-7.0.57.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-7.0.57/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;besctl stop
;./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Hyrax GitHub Source Build

2024-06-07T02:47:33Z

Jimg: /* Rocky 8 */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a CentOS machine to build code =
== Setup CentOS-7 ==
Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the CentOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. You can use this to extrapolate how to configure Ubuntu and OSX (We routinely build on those platforms as well). The complete instructions are in [[ConfigureCentos | Configure CentOS]] and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

;Update the VM
:<tt>'''yum -y update'''</tt>

;Load a basic software development environment:
:<tt>'''yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant ant-junit junit'''</tt> (it's likely that you can use more recent versions of Java)
:<tt>'''yum install git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

; The whole thing, with java-1.8.0
:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant ant-junit junit git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

;Optional
:Download, unpack, build and install the GNU autotools (''but '''don't''' do this unless the versions installed using yum don't work'')
* autoconf <tt>'''[http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz autoconf-2.69.tar.gz]'''</tt>
* automake <tt>'''[http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz automake-1.14.1.tar.gz]'''</tt>
* libtool <tt>'''[http://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz libtool-2.4.2.tar.gz]'''</tt>
:build them (<tt>'''''./configure; make; sudo make install '''''</tt> - this should take no more than three minutes).

== Setup CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>'''yum -y update'''</tt>

;You will need to enable power-tools for this setup
:<tt>'''yum config-manager --set-enabled powertools'''</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel'''</tt>

;Tell the machine where to find the tirpc libraries
:<tt>'''export CPPFLAGS=-I/usr/include/tirpc'''</tt>
:<tt>'''export LDFLAGS=-ltirpc'''</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>'''sudo yum install -y python3'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== Rocky 8 ==
''Updated 6/6/2024''

# To get the commands ps, which, etc.
dnf install -y procps

# C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

# Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

# Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

# Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core
dnf install epel-release
dnf update

cppunit cppunit-devel openjpeg2-devel jasper-devel

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
Use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the directory set to ''$prefix'')
<tt>
;git clone https://github.com/opendap/hyrax-dependencies
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.
git submodule update --init # update the submodules</pre>
That will clone some additional modules into the directory ''modules''; you need to do this! (Previously it was an optional step). See [http://git-scm.com/docs/git-submodule git submodule] for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use ''git submodule foreach 'git checkout master' ''.
<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-70.cgi Tomcat 7 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-7.0.57.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-7.0.57/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;besctl stop
;./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

Hyrax GitHub Source Build

2024-06-07T02:45:32Z

Jimg: /* Setup CentOS-8 and Rocky8 */

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:
# Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
# Build the C++ DAP library (''libdap4'') and the Hyrax BES daemon
# Build the Hyrax OLFS web application

Quick links if you already know the process:
* [https://github.com/opendap/hyrax new all-in-one repo that uses shell scripts]
* [https://github.com/opendap/libdap libdap git repo]
* [https://github.com/opendap/bes BES git repo]
* [https://github.com/opendap/olfs OLFS git repo]
* [https://github.com/opendap/hyrax-dependencies Hyrax dependencies]

= Set up a CentOS machine to build code =
== Setup CentOS-7 ==
Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the CentOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. You can use this to extrapolate how to configure Ubuntu and OSX (We routinely build on those platforms as well). The complete instructions are in [[ConfigureCentos | Configure CentOS]] and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

;Update the VM
:<tt>'''yum -y update'''</tt>

;Load a basic software development environment:
:<tt>'''yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant ant-junit junit'''</tt> (it's likely that you can use more recent versions of Java)
:<tt>'''yum install git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

; The whole thing, with java-1.8.0
:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant ant-junit junit git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

;Optional
:Download, unpack, build and install the GNU autotools (''but '''don't''' do this unless the versions installed using yum don't work'')
* autoconf <tt>'''[http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz autoconf-2.69.tar.gz]'''</tt>
* automake <tt>'''[http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz automake-1.14.1.tar.gz]'''</tt>
* libtool <tt>'''[http://ftp.gnu.org/gnu/libtool/libtool-2.4.2.tar.gz libtool-2.4.2.tar.gz]'''</tt>
:build them (<tt>'''''./configure; make; sudo make install '''''</tt> - this should take no more than three minutes).

== Setup CentOS-8 ==
The CentOS-8 setup is very similar to CentOS-7, but there are some minor differences.

;Update the VM
:<tt>'''yum -y update'''</tt>

;You will need to enable power-tools for this setup
:<tt>'''yum config-manager --set-enabled powertools'''</tt>

;Load the basic software development environment plus the additional packages of openjpeg2, jasper, and libtirpc. Note that you may not need ''openjpeg2'' and ''jasper'' if you build the dependencies successfully. If you determine that you don't need these, please let us know. JUnit support has also been dropped so we dropped the <tt>''ant-junit junit''</tt> packages from the install list.

:<tt>'''yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel ant git gcc-c++ flex bison cmake autoconf automake libtool emacs openssl-devel libuuid-devel readline-devel zlib-devel bzip2 bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel cppunit cppunit-devel vim bc openjpeg2-devel jasper-devel libtirpc-devel'''</tt>

;Tell the machine where to find the tirpc libraries
:<tt>'''export CPPFLAGS=-I/usr/include/tirpc'''</tt>
:<tt>'''export LDFLAGS=-ltirpc'''</tt>

;NB: As of 1/28/22 you should not need to do this. The ''configure'' script should find the correct way to run python on CentOS 8. However, if it does not, our Makefiles (built from ''Makefile.am'' files) use ''python'' but a vanilla CentOS 8 machine only has ''python3''. Until we fix this, you need to make sure ''python'' runs a python program. One way is to make a symbolic link between ''python3'' and ''python'' in a directory that is on your PATH. '''The TODO item here is to make sure ''python'' exists and can run a program'''. It is generally enough to verify that the command exists:
:<tt>'''which python'''</tt>

; Lacking that (which I was on Rocky8) install python
: <tt>'''sudo yum install -y python3'''</tt>

;If you're going to build RPMs
:<tt>'''yum install rpm-devel rpm-build redhat-rpm-config'''</tt>

Once you run through the rest of the hyrax build make sure that both ''gdal'' and ''hdf4'' build correctly (look for their libraries in $prefix/deps/lib). To build them manually, run '''make gdal''', '''make hdf4''', amd '''make netcdf4''' inside the hyrax-dependencies to build and install gdal and hdf4

== Rocky 8 ==
''Updated 6/6/2024''

# To get the commands ps, which, etc.
dnf install -y procps

# C++ environment plus build tools
dnf install -y git gcc-c++ flex bison cmake autoconf automake libtool emacs bzip2 vim bc

# Development library versions
dnf install -y openssl-devel libuuid-devel readline-devel zlib-devel bzip2-devel libjpeg-devel libxml2-devel curl-devel libicu-devel libtirpc-devel

# Java
dnf install -y java-17-openjdk java-17-openjdk-devel ant

# Setup DNF so that we can load in some obscure packages from EPEL, etc., repos
dnf install dnf-plugins-core

cppunit cppunit-devel openjpeg2-devel jasper-devel

= A semi-automatic build =

Use git to clone the https://github.com/opendap/hyrax project and follow the short instructions in the README file.
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Summarized here, those instructions are:
;use bash: The shell scripts in this repo assume you are using bash.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Test the server by loooking at ''<nowiki>http://localhost:8080/opendap</nowiki>'' in a browser. You should see a directory named ''data'' and following that link should lead to more data. The server will be accessible to clients other than a web browser.
:To test the BES function independently of the front end, use ''bescmdln'' and give it the ''show version;'' command, you should see output about different components and their versions.
:Use ''exit'' to leave the command line test client.

As described in the README file that is part of the ''hyrax'' repo, there are some other scripts in the repo and some options to the ''clone'' and ''build'' script that you can investigate by using -h (help).

= The manual build =

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant.

== Get Hyrax from GitHub ==
Use git to clone the https://github.com/opendap/hyrax project and follow the instructions on this page (which differ a bit from ones in the project's README)
:'''<tt>git clone https://github.com/opendap/hyrax'''</tt>

Once you have the ''hyrax'' project cloned:
;set up some environment variables so the server will build an install locally, something that streamlines development
:<tt>'''source spath.sh'''</tt>
;clone the three code repos for the server plus the hyrax dependencies
:<tt>'''./hyrax_clone.sh -v'''</tt>
;proceed with the rest of the build as described in the following sections of this page

== Important Note ==
Many of the problems people have with the build stem from not setting the shell correctly for the build.
In the above section, ''make sure'' you run '''source spath.sh''' before you run any of the building/compiling/testing commands that use the source code or build files. While the ''$prefix'' and ''$PATH'' environment variables are simple to set up, they are needed by most users. When you exit a terminal window and then open a new one, make sure to (re)source the ''spath.sh'' file in the new shell. You don't have to source spath.sh every time you enter the ''hyrax'' directory, but you must run it for every new instance of the shell.

== Compile the Hyrax dependencies ==
Use git to clone the hyrax-dependencies:
git clone https://github.com/opendap/hyrax-dependencies
And then build it. Unlike many source packages, there is no need to run a configure script, just ''make'' will do. However, the Makefile in this package expects ''$prefix'' to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called ''deps''. To build the dependencies for building RPMs, use ''make -j9 for-static-rpm''.
;(make sure you're in the directory set to ''$prefix'')
<tt>
;git clone https://github.com/opendap/hyrax-dependencies
; cd hyrax-dependencies
; make --jobs=9
: ''The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. '''-jN''' is the short form for the option.''
;cd ..: ''Go back up to '''$prefix''' ''
</tt>

'''Note:''' You can get some of the ''dependencies'' for Hyrax like ''netCDF'' from the EPEL repository, but the versions are often older than Hyrax needs. Contact us if you want information about using EPEL. At the risk of throwing people a curve ball, here's a synopsis of the process. Don't do this unless you know EPEL well. Use [http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm epel-release-6-8.noarch.rpm] and install it using ''sudo yum install epel-release-6-8.noarch.rpm''. Then install packages needed to read various file formats: ''yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build''

== Build ''libdap'' and the ''BES'' daemon ==

==== Get and build libdap4 ====
;WARNING: If you have ''libdap'' already, uninstall it before proceeding.
Build, test and install libdap4 into $prefix:

<pre>
git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf -fiv
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd .. # Go back up to ''$prefix''
</pre>


==== Get and build the BES and all of the modules shipped with Hyrax ====
Build, test and install the BES and its modules
<pre>git clone https://github.com/opendap/bes # Clone the BES from GitHub</pre>
cd bes # enter the bes dir.
git submodule update --init # update the submodules</pre>
That will clone some additional modules into the directory ''modules''; you need to do this! (Previously it was an optional step). See [http://git-scm.com/docs/git-submodule git submodule] for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use ''git submodule foreach 'git checkout master' ''.
<pre>autoreconf --force --install --verbose # You can use -fiv instead of the long options.</pre>

These means, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
<pre>./configure --prefix=$prefix --with-dependencies=$prefix/deps --enable-developer</pre>

: Notes:
:* The --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
:* The --enable-developer option will compile in all of the debugging code which may affect performance even if the debugging output is not enabled.
<pre>make -j9
make check -j9</pre>
Some tests may fail and adding ''-k'' ignores that and keeps make marching along. ''Note that you must run '''make''' before '''make check''' in the bes code''.
<pre>make install
cd .. # Go back up to $prefix</pre>

==== Test the BES ====
Start the BES and verify that all of the modules build correctly.
<pre>besctl start # Start the BES.</pre>
Given that ''$prefix/bin'' is on your ''$PATH'', this should start the BES. You will not need to be root if you used the ''--enable-developer'' switch with configure (as shown above), otherwise you should run ''sudo besctl start'' with the caveat that as root ''$prefix/bin'' will probably not be n your ''$PATH''.
:If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is ''not'' owned by root.
:Restart.
<pre>bescmdln # Now that the BES is running, start the BES testing tool
BESClient> show version; # Send the BES the version command to see if it's running </pre>
:Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
<pre> BESClient> exit; # Exit the testing tool</pre>

Note that even though you have exited the ''bescmdln'' test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use ''besctl stop'', or ''besctl pids'' to see the daemon's processes. If the BES is not stopping, ''besctl kill'' will stop all BES processes without waiting for them to complete their current task.

== Build the Hyrax ''OLFS'' web application ==
The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get [http://tomcat.apache.org/download-70.cgi Tomcat 7 from Apache]. Note that if you built the dependencies from source using the ''hyrac-dependencies-1.10.tar'' then there is a copy of Tomcat in the ''hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in ''$prefix''. I'll assume you have the Apache Tomcat tar file in ''$prefix''.

;tar -xzf apache-tomcat-7.0.57.tar.gz: Expand the Tomcat tar ball
;git clone https://github.com/opendap/olfs: Get the OLFS source code
;cd olfs: change directory to the OLFS source
;ant server: Build it
;cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/: Copy the opendap web archive to the tomcat webapps direcotry.
;cd ..: Go up to ''$prefix''
;./apache-tomcat-7.0.57/bin/startup.sh: Start Tomcat

== Test the server ==
You can test the server several ways, but the most fun is to use a web browser. The URL ''http://<machine>:8080/opendap'' should return a page pointing to a collection of test datasets bundled with the server. You can also use ''curl'', ''wget'' or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

== Stopping the server ==
Stop both the BES and Apache

;besctl stop
;./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a ''hyraxctl'' script that provides a way to start and stop Hyrax without you (or ''init.d'') having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

== Building select parts of the BES ==
Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (''git submodule update --init'') and link configure.ac to ''configure_standard.ac''. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if ''libdap'' is found by the configure script).

To build modules for the BES, simply go to ''$prefix'', clone their git repo and build them, taking care to pass set ''$prefix'' when calling the module's ''configure'' script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a ''configure.ac'', ''Makefile.am'', etc., that will support both kinds of builds and [[Configuration of BES Modules]] explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

= Building on Ubuntu =
This was tested using Xenial (Ubuntu 16)

''sudo apt-get update''

Packages needed:

''sudo apt-get install ...''

'''ant junit git flex bison autoconf automake libtool emacs openssl bzip2 libjpeg-dev libxml2-dev curl libicu-dev vim bc make cmake uuid-dev libcurl4-openssl-dev libicu-dev g++ zlib1g-dev libcppunit-dev libssl-dev'''

More about strings - passing strings to functions

2024-02-22T23:26:00Z

Jimg: Created page with "== Is it better to pass as a const reference, or by value and then use std::move()? == It depends. Here's a concise explanation from Stack Overflow. There are three cases: /* (0) */ Creature(const std::string &name) : m_name{name} { } A passed lvalue binds to name, then is copied into m_name. A passed rvalue binds to name, then is copied into m_name. /* (1) */ Creature(std::string name) : m_name{std::move(name)} { } A passed lvalue is copied into name, t..."

== Is it better to pass as a const reference, or by value and then use std::move()? ==

It depends.

Here's a concise explanation from Stack Overflow. There are three cases:

/* (0) */
Creature(const std::string &name) : m_name{name} { }

A passed lvalue binds to name, then is copied into m_name.

A passed rvalue binds to name, then is copied into m_name.

/* (1) */
Creature(std::string name) : m_name{std::move(name)} { }

A passed lvalue is copied into name, then is moved into m_name.

A passed rvalue is moved into name, then is moved into m_name.

/* (2) */
Creature(const std::string &name) : m_name{name} { }
Creature(std::string &&rname) : m_name{std::move(rname)} { }

A passed lvalue binds to name, then is copied into m_name.

A passed rvalue binds to rname, then is moved into m_name.

As move operations are usually faster than copies, (1) is better than (0) if you pass a lot of temporaries. (2) is optimal in terms of copies/moves, but requires code repetition.

Source: https://stackoverflow.com/questions/51705967/advantages-of-pass-by-value-and-stdmove-over-pass-by-reference

== What is an ''lvalue''? What is an ''rvalue''? ==

;LValue: An lvalue refers to an expression that identifies a memory location and can be assigned a value. It essentially acts as a locator for data storage.

Key characteristics:
* Can appear on the left-hand side of an assignment operator (=).
* Represents a persistent object that exists beyond the evaluation of a single expression.

Examples:
* Variable names (e.g., x, name)
* Array elements (e.g., array[3])
* Member variables of objects (e.g., object.value)
* Function calls returning an lvalue reference (rare)

;RValue: An rvalue represents a value itself, but it doesn't have a memory location that can be directly assigned to. It provides the data for an assignment.

Key characteristics:
* Can appear on the right-hand side of an assignment operator (=).
* Represents a temporary value that exists only during the evaluation of an expression.

Examples:
* Literal values (e.g., 42, "hello")
* Arithmetic expressions (e.g., 2 + 3)
* Function calls that don't return an lvalue reference
* The result of certain operators (e.g., increment/decrement)

''Every expression in a program is either an lvalue or an rvalue.''

;Implicit conversions: In some cases, compilers can implicitly convert between lvalues and rvalues. For example, an lvalue can be converted to an rvalue when its value is used in an expression (e.g., x + 5).
;Modern languages: While the core concepts remain similar, some modern languages like C++ introduce additional categories like ''xvalues'' for handling move semantics and resource management.

Source: https://gemini.google.com/app/efaa1d88d284719d, with edits.

Developer Info

2024-02-22T23:09:05Z

Jimg: /* C++ Coding Information */

* [https://github.com/OPENDAP OPeNDAP's GitHub repositories]: OPeNDAP's software is available using GitHub in addition to the downloads from our website.
** Before 2015 we hosted our own SVN repository. It's still online and available, but for read-only access, at [https://scm.opendap.org/svn https://scm.opendap.org/svn].
* [https://travis-ci.org/OPENDAP Continuous Integration builds]: Software that is built whenever new changes are pushed to the master branch. These builds are done on the Travis-CI system.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* We use the Coverity static system to look for common software defects, information on Hyrax is spread across three projects:
** [https://scan.coverity.com/projects/opendap-bes?tab=overview The BES and the standard handlers we distribute]
** [https://scan.coverity.com/projects/opendap-olfs?tab=overview The OLFS - the front end to the Hyrax data server]
** [https://scan.coverity.com/projects/opendap-libdap4?tab=overview libdap - The implementation of DAP2 and DAP4]

== OPeNDAP's FAQ ==
The [http://www.opendap.org/faq-page OPeNDAP FAQ] has a pretty good section on developer's questions.

== C++ Coding Information ==
* [[Include files for libdap | Guidelines for including headers]]
* [[Using lambdas with the STL]]
* [[Better Unit tests for C++]]
* [[Better Singleton classes C++]]
* [[What is faster? stringstream string + String]]
* [[More about strings - passing strings to functions]]

== OPeNDAP Workshops ==
* [http://www.opendap.org/about/workshops-and-presentations/2007-10-12 The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/about/workshops-and-presentations/2008-07-15 ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.

== libdap4 and BES Reference documentation ==
* [https://opendap.github.io/bes/html/ BES Reference]
* [https://opendap.github.io/libdap4/html/ libdap Reference]

== BES Development Information ==
* [[Hyrax - Logging Configuration|Logging Configuration]]

* [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
* [[BES - Debugging Using besstandalone]]
* [[Hyrax - Create BES Module | How to create your own BES Module]]
* Hyrax Module Integration: How to configure your module so it's easy to add to Hyrax instances ([[:File:HyraxModuleIntegration-1.2.pdf|pdf]])
* [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
* [[Hyrax - Running bescmdln | Running the BES command line client]]
* [[Hyrax - BES Client commands| BES Client commands]]. The page [[BES_XML_Commands | BES XML Commands]] repeats this info for a bit more information on the return values. Most of the commands don't return anything unless they return an error and are expected to be used in a group where a ''get'' command closes out the request and obviously does return a response of some kind (maybe an error).
* [[Hyrax:_BES_Administrative_Commands| BES Administrative Commands]]
* [[Hyrax - Extending BES Module | Extending your BES Module]]
* [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
* [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]

* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]

== OPeNDAP Development process information ==
These pages contain information about how we'd like people working with us to use our various on-line tools.

* [[Planning a Program Increment]] This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
* [[Hyrax GitHub Source Build]] This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
* [[Bug Prioritization]]. How we prioritize bugs in our software.

===[[How to Make a Release|Making A Release]] ===
* [[How to Make a Release]] A general template for making a release. This references some of the pages below.

== Software process issues: ==
* [[How to download test logs from a Travis build]] All of our builds on Travis that run tests save those logs to an S3 bucket.
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]] - Updated 12/2014 to include information regarding git.
* [[How to use CLion with our software]]
* [[BES Timing| How to add timing instrumentation to your BES code.]]
* [[UnitTests| How to write unit tests using CppUnit]] NB: See other information under the heading of C++ development
* [[valgrind| How to use valgrind with unit tests]]
* [[Debugging the distcheck target]] Yes, this gets its own page...
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[Managing public and private keys using gpg]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
* [[UserSupport|How to Handle Email-list Support Questions]]
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [http://semver.org/ Software version numbers]
* [[GuideLines| Development Guidelines]]
* [[Apple M1 Special Needs]]

==== Older info of limited value: ====
* [http://gcc.gnu.org/gcc-4.4/cxx0x_status.html C++-11 gcc/++-4.4 support] We now require compilers that support C++-14, so this is outdated (4/19/23).
* [[How to use Eclipse with Hyrax Source Code]] I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.

==== AWS Tips ====
* [[Growing a CentOS Root Partition on an AWS EC2 Instance]]
* [[How Shutoff the CentOS firewall]]

== General development information ==
These pages contain general information relevant to anyone working with our software:

* '''[[Git Hacks and Tricks]]''': Information about using git and/or GitHub that seems useful and maybe not all that obvious.
* [[Git Secrets]]: securing repositories from AWS secret key leaks.
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind Suppression File Howto] How to build a suppressions file for valgrind.
* [[Using a debugger for C++ with Eclipse on OS/X]] Short version: use lldbmi2 **Add info**
* [[Using ASAN]] Short version, look [https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger at the Google/GitHub pages] for useful environment variables **add text** On Centos, use yum install llvm to get the 'symbolizer' and try ''ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer)''
* [[How to use ''Instruments'' on OS/X to profile]] Updated 7/2018
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind - How to generate suppression files for valgrind] This will quiet valgrind, keeping it from telling you OS/X or Linux (or the BES) is leaking memory.
* [[Migrating source code from SVN to git]]: How to move a large project from SVN to git and keep the history, commits, branches and tags.
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]. Updated in 2017, this is really good but be aware that it's specific to Mozilla so some of the tips don't apply. Hyrax (i.e., libdap4 and BES) also use their own build system (autotools + make) so most of the configuration information here is very apropos. See also [[How to use Eclipse with Hyrax Source Code]] below.
* [https://jfearn.fedorapeople.org/en-US/RPM/4/html/RPM_Guide/index.html RPM Guide] The best one I'm found so far...
* [https://autotools.io/index.html Autotools Myth busters] The best info on autotools I've found yet (covers ''autoconf'', ''automake'', ''libtool'' and ''pkg-config'').
* The [https://www.gnu.org/software/autoconf/autoconf.html autoconf] manual
* The [https://www.gnu.org/software/automake/ automake] manual
* The [https://www.gnu.org/software/libtool/ libtool] manual
* A good [https://lldb.llvm.org/lldb-gdb.html gdb to lldb cheat sheet] for those of us who know ''gdb'' but not ''lldb''.

= Old information =
'''Note''': Old build information
====The Release Process====
# Make sure the <tt>hyrax-dependencies</tt> project is up to date and tar balls on www.o.o. If there have been changes/updates:
## Update version number for the <tt>hyrax-dependencies</tt> in the <tt>Makefile</tt>
## Save, commit, (merge?), and push the changes to the <tt>master</tt> branch.
## Once the <tt>hyrax-dependencies</tt> CI build is finished, trigger CI builds for both <tt>libdap4</tt> and <tt>bes</tt> by pushing change(s) to the master branch of each.
# [[Source_Release_for_libdap | Making a source release of libdap]]
# [[ReleaseGuide | Making a source release of the BES]].
# [[OLFSReleaseGuide| Make the OLFS release WAR file]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
# [[HyraxDockerReleaseGuide|Make the official Hyrax Docker image for the release]] When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.

====Supplemental release guides====
Old - use the packages built using the Continuous Delivery process
# [[RPM |Make the RPM Distributions]]. Follow these steps to create an RPM distribution of the software. '''Note:''' ''Now we use packages built using CI/CD, so this checklist is no longer needed.''

'''Note''': ''The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...''
* [[MergingBranches |How to merge code]]
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]]. Follow these steps when a new Matlab GUI version is ready to be released.
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
* [[TestOpendapOrg | Notes from setting up Hyrax on our new web host]]
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.

Source Release For hyrax-dependencies

2024-01-24T23:25:36Z

Jimg: /* Tag The Release */

This task is to ensure that the ''hyrax-dependencies'' project is up to date and tar balls on www.o.o. are current.

== Update ChangeLog, NEW, and release version ==
=== Update the '''ChangeLog''' file. ===
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
Save the result to a temp file and combine the two files: 
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

=== Update the Version Numbers ===
If the review of the ChangeLog indicates that there have been changes since the last release, increment the version number in the Makefile.

Make sure any change in version number is also reflected in the NEWS file.

=== Update the NEWS file ===
To update the NEWS file, just read over the new ChangeLog entries and summarize.

== Commit And Push ==
# Save, commit, and push the changes to master branch.
# Once the ''hyrax-dependencies'' CI build is finished
## Trigger a CI build ''libdap4'' by pushing a small change to the ''libdap4'' master branch. When that CI build has completed successfully,
## Trigger a CI build in the ''bes'' by pushing a small change to the ''bes'' master branch.
# Wait for the successful completion.
#: If there's a problem with the CI builds at this point you may wish to follow the advice of '''''Herman Wouk''''': ''"When in danger or in doubt, run in circles, scream and shout"''

== Publish and Sign ==

All you need do is build the tar file using <tt>make dist</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''hyrax-dependencies''' project on your local machine and run <tt>make dist</tt> which will make a hyrax-dependencies-x.y.tar.gz file in the directory above the top level of the '''hyrax-dependencies''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org ../hyrax-dependencies-x.y.tar</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''hyrax-dependencies''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put hyrax-dependencies-x.y.tar.sig</tt>
#: <tt>put hyrax-dependencies-x.y.tar</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify hyrax-dependencies-x.y.tar.sig hyrax-dependencies-x.y.tar</tt>

== Tag The Release ==
# Tag, and push the tag.
#* ''git tag -m "version-<number>" -a <numbers>''
#* ''git push origin <numbers>''

== Make The Release On GitHub ==
# Goto the [https://github.com/OPENDAP/hyrax-dependencies/tags GitHub 'tags' page for ''hyrax-dependencies''].
# Click the "Create release from tag" button
# Enter a title for the release (looks at previous releases for examples)
# Copy the most recent text from the NEWS file into the describe field
# Click Save/Update this release.

Source Release for BES

2024-01-24T19:15:23Z

Jimg: /* Get the BES DOI from Zenodo */

This pages covers the steps required to release the BES software for Hyrax.

We now depend on the CI/CD process to build binary packages and to test the source builds.

:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch if you don't '''need''' it, since it complicates making tags.

== Verify the code base ==
# We release using the ''master'' branch. The code on ''master'' must have passed the CI builds. '''This includes the hyrax-docker builds since that CI build runs the full server regression tests!'''
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')

== Update the Version Numbers ==

=== Version for Humans ===
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.

:; Affected Files
:* '''''configure.ac''''' - Look for:
::: <tt>AC_INIT(bes, ###.###.###, opendap-tech@opendap.org)</tt>
:* ''''' debian/changelog''''' (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
:* '''''ChangeLog'''''
:* '''''NEWS'''''
:* '''''README.md'''''
:* '''''INSTALL'''''

=== Update the internal library (API/ABI) version numbers. ===
The BES is '''''not''''' a shared library, it is a set of c++ applications that are typically built as statically linked binaries. Because of this the usual CURRENT:REVISION:AGE tuples used to express the binary compatibility state of a c++ shared object library have little meaning for the BES code. So, what we choose to do is simply bump the REVISION numbers by one value for each release.

* In the '''configure.ac''' file locate each of:
** LIB_DIS_REVISION
** LIB_PPT_REVISION
** LIB_XML_CMD_REVISION
* Increase the value of each by one (1).
* Save the file.
* Update the text documentation files and version numbers in the configuration files:

Example of the relevant section from configure.ac:
<source lang-="bash">
LIB_DIS_CURRENT=18
LIB_DIS_AGE=3
LIB_DIS_REVISION=3
AC_SUBST(LIB_DIS_CURRENT)
AC_SUBST(LIB_DIS_AGE)
AC_SUBST(LIB_DIS_REVISION)
LIBDISPATCH_VERSION="$LIB_DIS_CURRENT:$LIB_DIS_REVISION:$LIB_DIS_AGE"
AC_SUBST(LIBDISPATCH_VERSION)

LIB_PPT_CURRENT=5
LIB_PPT_AGE=1
LIB_PPT_REVISION=2
AC_SUBST(LIB_PPT_CURRENT)
AC_SUBST(LIB_PPT_AGE)
AC_SUBST(LIB_PPT_REVISION)
LIBPPT_VERSION="$LIB_PPT_CURRENT:$LIB_PPT_REVISION:$LIB_PPT_AGE"
AC_SUBST(LIBPPT_VERSION)

LIB_XML_CMD_CURRENT=5
LIB_XML_CMD_AGE=4
LIB_XML_CMD_REVISION=2
AC_SUBST(LIB_XML_CMD_CURRENT)
AC_SUBST(LIB_XML_CMD_AGE)
AC_SUBST(LIB_XML_CMD_REVISION)
LIBXMLCOMMAND_VERSION="$LIB_XML_CMD_CURRENT:$LIB_XML_CMD_REVISION:$LIB_XML_CMD_AGE"
AC_SUBST(LIBXMLCOMMAND_VERSION)
</source>

== Update the '''ChangeLog''' file. ==
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: <tt style="font-size: 1.1em; font-weight: bold;">gitlog-to-changelog --since="1970-01-01"</tt>
:: (''Specify a date one day later than the one at the top of the existing ChangeLog file.'')
Save the result to a temp file and combine the two files: 
: <tt style="font-size: 1.1em; font-weight: bold;">cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog</tt>
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

== Update the '''NEWS''' file ==
To update the NEWS file, just read over the new ChangeLog entries and summarize.

The new entries to the NEWS file will be used later when making the GitHub release and when writing the server's release page on www.opendap.org.

We might replace this:
* It's also helpful to have, in the '''NEWS''' file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.)

With instructions about making an associated release in JIRA using version tagging.

== Update the Version Numbers for Humans ==
;Affected Files:
: configure.ac
: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog] Debian ChangeLog)
: NEWS
: README.md
: INSTALL

# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.
# In the '''README.md''' file be sure to update the description of how to locate the DOI for the release with the new version number.

== Update the libdap version ==
Determine the libdap version associated with this release by checking the contents of the file <tt>libdap4-snapshot</tt> The <tt>libdap4-snapshot</tt> file should contain a single line like this example:
<pre>
libdap4-3.20.9-0 2021-12-28T19:23:45+0000
</pre>

The libdap version for the above example is: <tt>libdap-3.20.9</tt> (The version is NOT <tt>libdap4-3.20.9</tt>)

=== Update the libdap version in the .travis.yml file ===
;Affected Files
: .travis.yml

In the .travis.yml file update the value of ''LIBDAP_RPM_VERSION'' in the ''env: global:'' section so that it contains the complete numerical value of the libdap version you located in the previous step. Using the previous example the value would be:
<pre>
- LIBDAP_RPM_VERSION=3.20.9-0
</pre>

=== Update the libdap version in the RPM spec files ===
;Affected Files
: ''bes.spec*.in''
Update the <tt>bes.spec*.in</tt> files by changing the <tt>Requires</tt> and <tt>BuildRequires</tt> entries for libdap. Based on our example the result would be:

<pre>
Requires: libdap >= 3.20.9
BuildRequires: libdap-devel >= 3.20.9
</pre>

(''These lines may not be adjacent to each other in the spec files'')

=== Update the libdap version in the README.md file ===
;Affected Files
: README.md
# [https://zenodo.org Get the DOI markdown from Zenodo] by using the search bar and searching for the libdap version string that you determined at the beginning of this section.
# Update the '''README.md''' file with libdap version and the associated DOI link (using the markdown you got from Zenodo).

; Note
: You will also need this DOI markdown when making the GitHub release page for the BES.

See the section on this page titled "''Get the BES DOI from Zenodo''" for more details about getting the DOI markdown.

== Update the RPM dependencies ==
;Affected Files:
:''bes.spec*.in''

In the RPM ''.spec'' file, update the dependencies as needed.
* The libdap version dependency was covered in a previous step.
* Be attentive to changes that have been made to the hyrax-dependencies since the last release.

== Update the module version numbers for humans ==
In bes/modules/common, check that the file all-modules.txt is complete and update as needed. Then run:

* Remove the sentinel files that prevent the version updater from being run multiple times in succession without specific intervention:
:: '''<tt>rm -v ../*/version_updated</tt>'''
* Now run the version updater:
:: '''<tt>./version_update_modules.sh -v </tt>'''

This will update the patch number (x.y.patch) for each of the named modules.

If a particular module has significant fixes, hand edit the number, in the Makefile.am.

See below for special info about the HDF4/5 modules (which also applies to any modules not in the BES GitHub repo).

== <del>For the BES HDF4/5 modules (BES only) </del>==
# <del>''Make sure that you are working on the master branch of each module!!''</del>
# <del> Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).</del>
# <del> Update the module version numbers in their respective Makefile.am files.</del>
# <del> Commit and Push these changes.</del>

== Update the Build Offset ==
''Setting the build offset correctly will set the build number for the new release to "0".''

In the file <tt>travis/travis_bes_build_offset.sh</tt> set the value of <tt>BES_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/bes the TravisCI page for the BES] and use that build number plus 1.

== Commit Changes ==
''Be sure that you have completed all of the changes to the various ChangeLog, NEWS, INSTALL, configure.ac, <tt>travis/travis_bes_build_offset.sh</tt>, and other files before proceeding!''

# Commit and push the BES code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.

== Tag the BES code ==

The build process automatically tags builds of the master branch. The Hyrax-version tag is a placeholder for us so we can sort out what code goes with various Hyrax source releases.

# If this is part of a Hyrax Release, then tag this point in the master branch with the Hyrax release number
#* <tt style="font-size: 1.1em; font-weight: bold;">git tag -m "hyrax-<number>" -a hyrax-<numbers></tt>
#* <tt style="font-size: 1.1em; font-weight: bold;">git push origin hyrax-<numbers></tt>
#: '''NB:''' ''Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.''

== Create the BES release on Github ==
# [https://github.com/OPENDAP/bes Goto the BES project page in GitHub]
# Choose the '''releases''' tab.
# On the [https://github.com/OPENDAP/bes/releases Releases page] click the 'Tags' tab.
# On the [https://github.com/OPENDAP/bes/tags Tags page], locate the tag (created above) associated with this new release.
# Click the ellipses (...) located on the far right side of the ''version-x.y.z'' tag 'frame' for this release and and choose ''Create release''.
#* Enter a ''title'' for the release
#* Copy the most recent text from the NEWS file into the ''describe'' field
#* Click '''Publish release''' or '''Save draft'''.
#** If you have previously edited the release page you can click '''Update this release'''

== Publish and Sign ==

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.

All you need do is build the tar file using <tt>make list</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''bes''' project on your local machine and run ''<tt>make dist</tt>'' which will make a bes-x.y.z,tar.gz file at the top level of the '''bes''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org bes-x.y.z.tgz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put bes-x.y.z.tgz.sig</tt>
#: <tt>put bes-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify bes-x.y.z.tgz.sig bes-x.y.z.tgz</tt>

== Get the BES DOI from Zenodo ==
Get the Zenodo DOI for the newly created BES release and add it to the associated GitHub BES release page.

# [https://zenodo.org Goto Zenodo]
# Look at the 'upload' page. If there is nothing there (perhaps because you are not ''jhrg'' or whoever set up the connection between the BES project and Zenodo) you can use the search bar to search for '''bes'''.
#: Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up.
# Edit the GitHub release page for the BES release you just created and paste the DOI markdown into the top of the description.

'''Tip:''' ''If you are trying to locate the '''libdap''' releases in Zenodo you have to search for the string:'' <tt style="font-size: 1.1em; font-weight: bold;">libdap4</tt>

=== Images ===
[[File:Screenshot 2018-12-06 11.06.44.png|none|thumb|400px|border|left|Zenodo upload page]]

== Update the online reference documentation ==

# ''make gh-docs''

Source Release for libdap

2024-01-24T18:54:46Z

Jimg: /* Get the DOI from Zenodo */

This page covers the step needed to release the libdap software for Hyrax. There is are separate pages for the BES and OLFS code and an overview page that describes how the website is updated and lists are notified.

We now depend on the CI/CD process to build binary packages and to test the source builds. When the source code is tagged and marked as a release in GitHub, our linked Zenodo account archives that software and mints a DOI for it.

== The Release Process ==
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch unless you need to since it complicates making tags.

=== Verify the code base ===
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')

=== Update Release Files ===
Update the text documentation files and version numbers in the configuration files:

; '''Note'''
:It's helpful to have, in the '''NEWS''' file, the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

==== Update the '''ChangeLog''' file. ====
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
Save the result to a temp file and combine the two files: 
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize.

==== Update the Version Numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like version-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0

===== Version for Humans =====
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.

:;Affected Files:
:: '''''configure.ac''''' - Look for:
::: <tt>AC_INIT(libdap, ###.###.###, opendap-tech@opendap.org)</tt>
:: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
:: '''README.md'''
:: '''INSTALL'''

===== API/ABI Version =====
The library API/ABI version is represented as CURRENT:REVISION:AGE.

;The rules for shared image version numbers:
:# No interfaces changed, only implementations (good): Increment REVISION.
:# Interfaces added, none removed (good): Increment CURRENT, set REVISION to 0, increment AGE.
:# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set REVISION to 0 , and set AGE to 0.

See the ''Appendix: How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.

;Affected Files:
: '''''configure.ac''''' - Look for
:: DAPLIB_CURRENT=###
:: DAPLIB_REVISION=###
:: DAPLIB_AGE=###

=== Commit ===
* Commit and push the code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.

=== Update the Build Offset ===
''Setting the build offset correctly will set the build number for the new release to "0".''

In the file <tt>travis/travis_libdap_build_offset.sh</tt> set the value of <tt>LIBDAP_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/libdap4 the TravisCI page for libdap4] and use that build number plus 1.

This is not the build number for the package. It is the build number used by Travis, which is the the total number of times Travis has build the code. This number is the build number on the left-hand TOC

Once you have updated the <tt>travis/travis_libdap_build_offset.sh</tt> commit and push this change. Do NOT use a <tt>[skip ci]</tt> string in the commit message as it is important that this commit run through the entire CI process.

=== Tag The Release ===
In the past we manually made the tags for builds. Since we started a 'build number release' for NASA, we automated that.

If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:
# '''git tag -m "hyrax-<number>" -a hyrax-<numbers>''' I think we can leave this tag as ''hyrax-<version>'' since it's for our own bookkeeping.
# '''git push origin hyrax-<numbers>'''
#: NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

=== Create the release on Github ===
Goto the 'tags' page ('code' then 'tags' at the top of the directory window) and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
# Enter a ''title'' for the release
# Copy the most recent text from the NEWS file into the ''describe'' field
# Click ''Update this release'' or ''Save draft''

This will trigger a 'archive and DOI' process on the Zenodo system.

=== Publish and Sign ===

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.

All you need do is build the tar file using <tt>make dist</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''libdap4''' project on your local machine and run <tt>make dist</tt> which will make a libdap-x.y.z.tar.gz file at the top level of the '''libdap4''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org libdap-x.y.z.tar.gz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put libdap-x.y.z.tgz.sig</tt>
#: <tt>put libdap-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify libdap-x.y.z.tgz.sig libdap-x.y.z.tgz</tt>

=== Get the DOI from [https://zenodo.org Zenodo] ===

# Goto [https://zenodo.org Zenodo] and look at the 'upload' page. Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
# Also paste that into the README file. Commit using ''[skip ci]'' so we don't do a huge build (or do the build, it really doesn't matter that much).

Images for the above steps to help with the web UI: coming soon

=== Update the online reference documentation ===

# ''make gh-docs''

== Appendix: How to see the scope of API/ABI changes in C++ sources ==
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T $.*$@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of the <tt>CURRENT:REVISION:AGE</tt> strings then:
;Edit the configure.ac and update the version values to the new ones.

Source Release for libdap

2024-01-24T18:39:22Z

Jimg: /* Get the DOI from Zenodo */

This page covers the step needed to release the libdap software for Hyrax. There is are separate pages for the BES and OLFS code and an overview page that describes how the website is updated and lists are notified.

We now depend on the CI/CD process to build binary packages and to test the source builds. When the source code is tagged and marked as a release in GitHub, our linked Zenodo account archives that software and mints a DOI for it.

== The Release Process ==
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch unless you need to since it complicates making tags.

=== Verify the code base ===
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')

=== Update Release Files ===
Update the text documentation files and version numbers in the configuration files:

; '''Note'''
:It's helpful to have, in the '''NEWS''' file, the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

==== Update the '''ChangeLog''' file. ====
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
Save the result to a temp file and combine the two files: 
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize.

==== Update the Version Numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like version-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0

===== Version for Humans =====
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.

:;Affected Files:
:: '''''configure.ac''''' - Look for:
::: <tt>AC_INIT(libdap, ###.###.###, opendap-tech@opendap.org)</tt>
:: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
:: '''README.md'''
:: '''INSTALL'''

===== API/ABI Version =====
The library API/ABI version is represented as CURRENT:REVISION:AGE.

;The rules for shared image version numbers:
:# No interfaces changed, only implementations (good): Increment REVISION.
:# Interfaces added, none removed (good): Increment CURRENT, set REVISION to 0, increment AGE.
:# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set REVISION to 0 , and set AGE to 0.

See the ''Appendix: How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.

;Affected Files:
: '''''configure.ac''''' - Look for
:: DAPLIB_CURRENT=###
:: DAPLIB_REVISION=###
:: DAPLIB_AGE=###

=== Commit ===
* Commit and push the code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.

=== Update the Build Offset ===
''Setting the build offset correctly will set the build number for the new release to "0".''

In the file <tt>travis/travis_libdap_build_offset.sh</tt> set the value of <tt>LIBDAP_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/libdap4 the TravisCI page for libdap4] and use that build number plus 1.

This is not the build number for the package. It is the build number used by Travis, which is the the total number of times Travis has build the code. This number is the build number on the left-hand TOC

Once you have updated the <tt>travis/travis_libdap_build_offset.sh</tt> commit and push this change. Do NOT use a <tt>[skip ci]</tt> string in the commit message as it is important that this commit run through the entire CI process.

=== Tag The Release ===
In the past we manually made the tags for builds. Since we started a 'build number release' for NASA, we automated that.

If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:
# '''git tag -m "hyrax-<number>" -a hyrax-<numbers>''' I think we can leave this tag as ''hyrax-<version>'' since it's for our own bookkeeping.
# '''git push origin hyrax-<numbers>'''
#: NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

=== Create the release on Github ===
Goto the 'tags' page ('code' then 'tags' at the top of the directory window) and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
# Enter a ''title'' for the release
# Copy the most recent text from the NEWS file into the ''describe'' field
# Click ''Update this release'' or ''Save draft''

This will trigger a 'archive and DOI' process on the Zenodo system.

=== Publish and Sign ===

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.

All you need do is build the tar file using <tt>make dist</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''libdap4''' project on your local machine and run <tt>make dist</tt> which will make a libdap-x.y.z.tar.gz file at the top level of the '''libdap4''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org libdap-x.y.z.tar.gz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put libdap-x.y.z.tgz.sig</tt>
#: <tt>put libdap-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify libdap-x.y.z.tgz.sig libdap-x.y.z.tgz</tt>

=== Get the DOI from [https://zenodo.org Zenodo] ===

# Goto [https://zenodo.org Zenodo] and look at the 'upload' page. Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
# Also paste that into the README file. Commit using ''[skip ci]'' so we don't do a huge build (or do the build, it really doesn't matter that much).

Images for the above steps to help with the web UI: coming soon

== Appendix: How to see the scope of API/ABI changes in C++ sources ==
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T $.*$@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of the <tt>CURRENT:REVISION:AGE</tt> strings then:
;Edit the configure.ac and update the version values to the new ones.

Jira Release Process

2024-01-23T21:14:32Z

Jimg:

[[ReleaseSprintNotes | back]]

This is a work in progress... (jhrg 12/05/18)

== OPeNDAP JIRA ==

===Get all the closed tickets for this release ===

[[File:Screenshot 2018-12-05 15.36.01.png|200px|thumb|right|Jira Recently Closed Tickets]]

Get closed tickets, numbers, et c., from Jira. Go to the Issues and Filters page and Look at recently closed issues. Edit the query by clicking on the 'Advanced' link at the upper right, and then get the list of issues as an Excel/CSV file. This makes the issues, their obscure ticket numbers, et c., easy to copy and paste into the release web page.

These steps worked when were not also using Jira for our 'Build releases,' but now that we are, editing the 'fixVersion' makes those more work and does not seem to add much to the 'releases for people outside of NGAP' since they cannot see the 'NASA Jira' and thus have no way of knowing about this change.

Let's stop making this change unless/until a way to do so that works with the Build Releases presents itself.

== Don't Do this <s>Update the Release Version</s> ==

[[Image:Screenshot 2018-12-05 15.16.27.png|200px|thumb|right|Jira Releases]]

Add a new 'Release version' on the Jira 'Releases' page. Then 'release' the current version, moving all unclosed tickets up to the next logical version.

== Don't do this <s>NASA JIRA</s> ==

# Go to the [https://bugs.earthdata.nasa.gov/secure/RapidBoard.jspa?rapidView=901&projectKey=HYRAX&view=planning.nodetail&issueLimit=100 Hyrax JIRA page at NASA].
# Find the Version for the pending Hyrax release. If you can only find the Version for a previous release make a new Version title Hyrax-<numbers> for the new release.
# Locate the tickets closed since the last Hyrax release date. You can easily find this out by using the [https://bugs.earthdata.nasa.gov/issues/?filter=23327 saved issue filter "Closed Since"]. You may have to edit the filter so the "since date" reflects the date of the previous Hyrax release.
# Examine each ticket to determine if the work done resulted in a change to Hyrax or to a change to some part of the NGAP stack. Add the Hyrax Version tag to the tickets that have become part of the publicly available Hyrax code. Do not include changes that are associated with NGAP CI/CD, NGAP deployments, or NGAP Cloud provisioning. The idea is to only tag changes that were made to Hyrax that took place in our GitHub repository and that changed the server that we release publicly. In theory this process should be happening as we go along in the NGAP JIRA process, so this step should really be a "sweep" to make sure nothing important was missed.
# When this has been completed you can go to the Hyrax Version page and make it a Release.
# Copy the URL for the release page for inclusion in the Hyrax release page on www.opendap.org

Source Release for BES

2024-01-05T02:31:44Z

Jimg: /* Tag the BES code */

Source Release for libdap

2024-01-04T22:44:56Z

Jimg: /* Tag The Release */

This page covers the step needed to release the libdap software for Hyrax. There is are separate pages for the BES and OLFS code and an overview page that describes how the website is updated and lists are notified.

We now depend on the CI/CD process to build binary packages and to test the source builds. When the source code is tagged and marked as a release in GitHub, our linked Zenodo account archives that software and mints a DOI for it.

== The Release Process ==
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch unless you need to since it complicates making tags.

=== Verify the code base ===
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')

=== Update Release Files ===
Update the text documentation files and version numbers in the configuration files:

; '''Note'''
:It's helpful to have, in the '''NEWS''' file, the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

==== Update the '''ChangeLog''' file. ====
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
Save the result to a temp file and combine the two files: 
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize.

==== Update the Version Numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like version-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0

===== Version for Humans =====
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.

:;Affected Files:
:: '''''configure.ac''''' - Look for:
::: <tt>AC_INIT(libdap, ###.###.###, opendap-tech@opendap.org)</tt>
:: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
:: '''README.md'''
:: '''INSTALL'''

===== API/ABI Version =====
The library API/ABI version is represented as CURRENT:REVISION:AGE.

;The rules for shared image version numbers:
:# No interfaces changed, only implementations (good): Increment REVISION.
:# Interfaces added, none removed (good): Increment CURRENT, set REVISION to 0, increment AGE.
:# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set REVISION to 0 , and set AGE to 0.

See the ''Appendix: How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.

;Affected Files:
: '''''configure.ac''''' - Look for
:: DAPLIB_CURRENT=###
:: DAPLIB_REVISION=###
:: DAPLIB_AGE=###

=== Commit ===
* Commit and push the code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.

=== Update the Build Offset ===
''Setting the build offset correctly will set the build number for the new release to "0".''

In the file <tt>travis/travis_libdap_build_offset.sh</tt> set the value of <tt>LIBDAP_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/libdap4 the TravisCI page for libdap4] and use that build number plus 1.

This is not the build number for the package. It is the build number used by Travis, which is the the total number of times Travis has build the code. This number is the build number on the left-hand TOC

Once you have updated the <tt>travis/travis_libdap_build_offset.sh</tt> commit and push this change. Do NOT use a <tt>[skip ci]</tt> string in the commit message as it is important that this commit run through the entire CI process.

=== Tag The Release ===
In the past we manually made the tags for builds. Since we started a 'build number release' for NASA, we automated that.

If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:
# '''git tag -m "hyrax-<number>" -a hyrax-<numbers>''' I think we can leave this tag as ''hyrax-<version>'' since it's for our own bookkeeping.
# '''git push origin hyrax-<numbers>'''
#: NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

=== Create the release on Github ===
Goto the 'tags' page ('code' then 'tags' at the top of the directory window) and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
# Enter a ''title'' for the release
# Copy the most recent text from the NEWS file into the ''describe'' field
# Click ''Update this release'' or ''Save draft''

This will trigger a 'archive and DOI' process on the Zenodo system.

=== Publish and Sign ===

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.

All you need do is build the tar file using <tt>make dist</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''libdap4''' project on your local machine and run <tt>make dist</tt> which will make a libdap-x.y.z.tar.gz file at the top level of the '''libdap4''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org libdap-x.y.z.tar.gz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put libdap-x.y.z.tgz.sig</tt>
#: <tt>put libdap-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify libdap-x.y.z.tgz.sig libdap-x.y.z.tgz</tt>

=== Get the DOI from [https://zenodo.org Zenodo] ===
'''We should stop putting the DOIs and the cool badge in the README because the copy of README archived by Zenodo will be one step out of sync.'''

''Instead, put a note in the README that the DOI can be found at Zenodo under name XYZ. jhrg 5/8/20''

# Goto [https://zenodo.org Zenodo] and look at the 'upload' page. Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
# Also paste that into the README file. Commit using ''[skip ci]'' so we don't do a huge build (or do the build, it really doesn't matter that much).

Images for the above steps to help with the web UI: coming soon

== Appendix: How to see the scope of API/ABI changes in C++ sources ==
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T $.*$@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of the <tt>CURRENT:REVISION:AGE</tt> strings then:
;Edit the configure.ac and update the version values to the new ones.

Source Release for libdap

2024-01-04T22:40:53Z

Jimg: /* Update the Build Offset */

This page covers the step needed to release the libdap software for Hyrax. There is are separate pages for the BES and OLFS code and an overview page that describes how the website is updated and lists are notified.

We now depend on the CI/CD process to build binary packages and to test the source builds. When the source code is tagged and marked as a release in GitHub, our linked Zenodo account archives that software and mints a DOI for it.

== The Release Process ==
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch unless you need to since it complicates making tags.

=== Verify the code base ===
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')

=== Update Release Files ===
Update the text documentation files and version numbers in the configuration files:

; '''Note'''
:It's helpful to have, in the '''NEWS''' file, the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

==== Update the '''ChangeLog''' file. ====
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
Save the result to a temp file and combine the two files: 
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''

==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize.

==== Update the Version Numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like version-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0

===== Version for Humans =====
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.

:;Affected Files:
:: '''''configure.ac''''' - Look for:
::: <tt>AC_INIT(libdap, ###.###.###, opendap-tech@opendap.org)</tt>
:: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
:: '''README.md'''
:: '''INSTALL'''

===== API/ABI Version =====
The library API/ABI version is represented as CURRENT:REVISION:AGE.

;The rules for shared image version numbers:
:# No interfaces changed, only implementations (good): Increment REVISION.
:# Interfaces added, none removed (good): Increment CURRENT, set REVISION to 0, increment AGE.
:# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set REVISION to 0 , and set AGE to 0.

See the ''Appendix: How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.

;Affected Files:
: '''''configure.ac''''' - Look for
:: DAPLIB_CURRENT=###
:: DAPLIB_REVISION=###
:: DAPLIB_AGE=###

=== Commit ===
* Commit and push the code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.

=== Update the Build Offset ===
''Setting the build offset correctly will set the build number for the new release to "0".''

In the file <tt>travis/travis_libdap_build_offset.sh</tt> set the value of <tt>LIBDAP_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/libdap4 the TravisCI page for libdap4] and use that build number plus 1.

This is not the build number for the package. It is the build number used by Travis, which is the the total number of times Travis has build the code. This number is the build number on the left-hand TOC

Once you have updated the <tt>travis/travis_libdap_build_offset.sh</tt> commit and push this change. Do NOT use a <tt>[skip ci]</tt> string in the commit message as it is important that this commit run through the entire CI process.

=== Tag The Release ===
# '''git tag -m "version-<number>" -a <numbers>''' (this was '''git tag -m "version-<number>" -a version-<numbers> ''' but we have had a request to switch to plan version numbers to be more conformant with common practice WRT git version tags).
# '''git push origin <numbers>'''

If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:
# '''git tag -m "hyrax-<number>" -a hyrax-<numbers>''' I think we can leave this tag as ''hyrax-<version>'' since it's for our own bookkeeping.
# '''git push origin hyrax-<numbers>'''
#: NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

=== Create the release on Github ===
Goto the 'tags' page ('code' then 'tags' at the top of the directory window) and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
# Enter a ''title'' for the release
# Copy the most recent text from the NEWS file into the ''describe'' field
# Click ''Update this release'' or ''Save draft''

This will trigger a 'archive and DOI' process on the Zenodo system.

=== Publish and Sign ===

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.

All you need do is build the tar file using <tt>make dist</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

# Go to the '''libdap4''' project on your local machine and run <tt>make dist</tt> which will make a libdap-x.y.z.tar.gz file at the top level of the '''libdap4''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org libdap-x.y.z.tar.gz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put libdap-x.y.z.tgz.sig</tt>
#: <tt>put libdap-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify libdap-x.y.z.tgz.sig libdap-x.y.z.tgz</tt>

=== Get the DOI from [https://zenodo.org Zenodo] ===
'''We should stop putting the DOIs and the cool badge in the README because the copy of README archived by Zenodo will be one step out of sync.'''

''Instead, put a note in the README that the DOI can be found at Zenodo under name XYZ. jhrg 5/8/20''

# Goto [https://zenodo.org Zenodo] and look at the 'upload' page. Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
# Also paste that into the README file. Commit using ''[skip ci]'' so we don't do a huge build (or do the build, it really doesn't matter that much).

Images for the above steps to help with the web UI: coming soon

== Appendix: How to see the scope of API/ABI changes in C++ sources ==
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T $.*$@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of the <tt>CURRENT:REVISION:AGE</tt> strings then:
;Edit the configure.ac and update the version values to the new ones.

Better Singleton classes C++

2023-12-28T17:50:46Z

Jimg: /* Explanation */

NB: This was ripped from Google Bard and edited. jhrg 12/27/23

The Meyers Singleton pattern is a popular way to implement the singleton design pattern in C++ using a static member variable declared within a function. It leverages the properties of static functions and objects to guarantee only one instance of the class is ever created.

== Overview ==

This pattern, developed by Scott Meyers (although G'bard didn't say so...) uses the basic properties of static methods and static variables within functions/methods. The singleton object initialization should happen in the object's default constructor, which is shielded from use by anything other than the singleton class. That's the one thing that makes this pattern somewhat tricky as constructors should not throw exceptions.

== Example ==

<pre>
class Singleton {
private:
// Private constructor to prevent direct instantiation
Singleton() {}

public:
// Delete the copy constructor and assignment operator to prevent copying
Singleton(const Singleton&) = delete;
Singleton& operator=(const Singleton&) = delete;

// Static member function that returns the singleton instance
static Singleton& getInstance() {
// Create a local static object the first time the function is called
static Singleton instance;
return instance;
}

// Other member functions of the Singleton class...
};

int main() {
// Access the singleton instance through the getInstance() function
Singleton& instance1 = Singleton::getInstance();
Singleton& instance2 = Singleton::getInstance();

// Verify that both instances are the same object
if (&instance1 == &instance2) {
std::cout << "Both instances refer to the same Singleton object!" << std::endl;
}

// Use the Singleton instance...

return 0;
}
</pre>

== Explanation ==

The Singleton class has a private constructor to prevent direct instantiation.
The getInstance() function is static and declared within the Singleton class.
Inside getInstance(), a static member object instance is declared. This object is only created the first time the function is called.
Subsequent calls to getInstance() simply return the existing instance object.
The copy constructor and assignment operator are deleted to prevent copying the singleton object.

=== Benefits of the Meyers Singleton ===

;Thread-safe: Initialization is guaranteed to happen only once, even in multithreaded environments.
;Lazy initialization: The singleton object is only created when it is first needed.
;Simple and concise: The implementation is relatively easy to understand and maintain.

=== Drawbacks of the Meyers Singleton ===

;Overuse: Singletons can lead to tight coupling and reduced testability. Use them sparingly and only when truly necessary.
;No explicit destruction: The singleton object will be destroyed only when the program exits. This can be problematic if resources need to be explicitly released earlier.

=== Alternatives to the Meyers Singleton ===

;Static local variables: This approach can be used within a single file to create a thread-safe singleton.
;Resource acquisition is initialization (RAII): This technique can be used to manage resources associated with the singleton and ensure proper cleanup.
;Dependency injection: This approach can improve testability and decouple the singleton from its dependent classes.

== Thread safety ==

The Meyers Singleton pattern is thread-safe in C++11 and later, even without explicit synchronization mechanisms like mutexes or locks.

The thread safety of the Meyers Singleton pattern relies on language-level guarantees introduced in C++11.
For older C++ versions (pre-C++11), additional synchronization mechanisms would be needed for thread safety.
Even though the initialization is thread-safe, it's important to note that any methods of the singleton class itself still need to be thread-safe if accessed concurrently by multiple threads.

=== How it achieves thread safety ===

;Static Local Variable:
The getInstance() function uses a static local variable to hold the singleton instance.
Static local variables are guaranteed to be initialized only once, even in multithreaded environments.
This initialization happens in a thread-safe manner due to language-level guarantees in C++11 and beyond.

;Magic Statics:
This feature of C++11 and later ensures that static variables with block scope (like those within functions) are initialized in a thread-safe way.
The compiler and runtime collaborate to handle potential race conditions during initialization, ensuring that only one thread initializes the variable at a time.

;No Explicit Locking Needed:
Because of these language-level guarantees, the Meyers Singleton pattern doesn't require any explicit locking mechanisms (like mutexes or locks) to ensure thread safety.
This makes it a relatively simple and efficient way to implement a thread-safe singleton.

Better Singleton classes C++

2023-12-28T00:14:21Z

Jimg:

Better Singleton classes C++

2023-12-27T22:05:12Z

Jimg:

NEVER USE THIS

We use lots of Singleton classes in the BES Framework. One issue with that pattern is that memory is usually not returned to the heap before the process exits, leaving tools like valgrind to report the memory as leaked. This is misleading and can be ignored, except that it's a great way to hide ''real'' leaks behind the noise in a sea of false positives. Here's a way around that using C++'s unquie_ptr.

== The basic plan ==

NEVER USE THIS

I'll use a real example of this from the BES; the BES 'Keys' key-value-pair configuration database. The idea is that we store a static pointer to the single copy of the class to be built. We access that pointer whenever the class is to used with a static method. That method returns a pointer to the class if it has been allocated or allocates a pointer and constructs the object if not. The pattern builds the single instance in a way that is thread safe (using the C++ 11 concurrency features).

== What goes in the header ==

In the header for the singleton class, include a static member that is a ''unique pointer'' to an instance of the object.

<pre>
class TheBESKeys: public BESObj {

...

TheBESKeys() = default;

explicit TheBESKeys(const std::string &keys_file_name);

static std::unique_ptr<TheBESKeys> d_instance;
static std::once_flag d_euc_init_once;

public:

/// Access to the singleton.
static TheBESKeys *TheKeys();

~TheBESKeys() override = default;

...
}
</pre>

To adapt most existing code to this pattern, move an existing static pointer so that it is a private static '''std::unique_ptr<'''class'''>'''.

== What goes in the implementation ==

Define the singleton instance a global/file level scope:

<pre>std::unique_ptr<TheBESKeys> TheBESKeys::d_instance = nullptr;
std::once_flag TheBESKeys::d_euc_init_once;</pre>

The accessor for the static pointer to the instance is likely the only code that needs to be changed in the implementation. Note that the ''std::once_flag'' is a static class member like the unique_ptr<>.

<pre>
TheBESKeys *TheBESKeys::TheKeys()
{
if (d_instance == nullptr) {
std::call_once(d_euc_init_once, []() {
d_instance.reset(new TheBESKeys(get_the_config_filename()));
});
}

return d_instance.get();
}
</pre>

Note that the constructor for the class is called inside the ''reset()'' method of the ''uniqe_ptr'' object: ''d_instance.reset(new TheBESKeys(get_the_config_filename()));'' and that that is called inside ''std::call_once()''. The use of ''std::call_once()'' ensure that if two threads simultaneously call the accessor, only one instance will be made. In the code above, a lambda was used to pass the 'runnable' to call_once(). If the pointer to the instance (''d_intance'') is not null, then the accessor uses ''unique_ptr::get()'' to return the 'raw' pointer to the instance, which will greatly simplify using this pattern with our existing code.

Git Hacks and Tricks

2023-10-20T22:30:47Z

Jimg: /* Cheat sheet items */

== Git resources ==
* The [http://git-scm.com/book/en/v2 Pro GIT] book is online at: git-scm.com
* Good cheat sheet: http://ndpsoftware.com/git-cheatsheet.html#loc=workspace;
* Info on branching from git.com: http://git-scm.com/book/en/Git-Branching-Remote-Branches
* Migration to git: http://git-scm.com/book/en/Git-and-Other-Systems-Migrating-to-Git

== Setup a username and access token for GitHub ==

:git config --global github.user <name>
:git config --global github.token <token>

:where the token is made using the instructions at https://help.github.com/articles/creating-an-access-token-for-command-line-use

If you want to configure a token for use with the OSX keychain, get the credential-osxkeychain tool with brew if you need to. Test if you have it by running '''git credential-osxkeychain''' and look for the credential-osxkeychain help message. To ''use'' the git extension, you need to enter '''git credential-osxkeychain <command>''' and then, on the next line, enter '''host=github.com''' and maybe '''protocol=https''' and other key/value pairs and then a blank line. See the examples below.

To use the osx keychain, first check if you have a password/token already saved:

:git credential-osxkeychain get
::host=github.com
::protocol=https
::<cr>

Erase the password/token

:git credential-osxkeychain erase
::host=github.com
::protocol=https
::<cr>

Then set the new token

:git credential-osxkeychain store
::host=github.com
::protocol=https
::username=<your login>
::password=<your token>
::<cr>

Then use the '''get''' command to verify.

== Git Secrets ==

Use this tool, which is run automatically before each commit, to keep from adding AWS and other secret keys to code that is destined for a public repository. Doing that will 'leak' the key and Amazon _will_ notice. The remedy will involve every account changing its password and every key pair being 'rotated' (i.e., every key pair has to be replaced with t anew one).

https://github.com/awslabs/git-secrets

Scroll down to the bottom for installation instructions. On OSX, you can use brew and do not have to clone the repo. Here's what I did:

:brew install git-secrets # install _git secrets_
:git secrets --register-aws --global # configure git so all future cloned repos will use it
:git secrets --install ~/.git-templates/git-secrets # set up all existing repos so they use it.
:git config --global init.templateDir ~/.git-templates/git-secrets

You can read a bit more of the docs in the _git secrets_ repo and configure a more fine grained approach.

== Subtrees: how to incorporate code from other repositories ==
Git subtrees are an alternative to submodules and are easier for the users of the parent repository (in our case, typically the BES repo). There is a fair amount of information about subtrees, although the main thing to know is that once the code for the child repo is part of the parent, in most cases there's nothing else to do. Only when changes get made in the code from the child repo are extra steps to keep the parent's copy of the code and child repo in sync needed (and then, only if you want to keep them in sync). For normal branch-PR-merge operations, there is no need to think about the subtree management commands.

Here's a [https://winstonkotzan.com/blog/2016/09/26/git-submodule-vs-subtree.html discussion about the differences between submodules and subtrees].

=== How to incorporate code from another repo ===
To incorporate code from another repo (the ''child'') into an existing repo (the ''parent''), use these steps:

# name the other project ''child'', and fetch: '''git remote add -f ''child'' <nowiki>https://github.com/</nowiki>...'''
:: The ''-f'' option runs ''git fetch'' automatically after the remote repo is added. See [https://git-scm.com/docs/git-remote git remote].
# prepare for the later step to record the result as a merge.: '''git merge -s ours --no-commit --allow-unrelated-histories ''child''/master'''
:: The ''-s'' option to ''git merge'' selects the ''ours'' strategy for the merge; ''--no-commit'' does not commit the merge automatically. See [https://git-scm.com/docs/git-merge#_merge_strategies git merge]. If you are using git 2.9+, add the option ''--allow-unrelated-histories'', but older versions of git don't support that (as of Jan. 2022, OSX was using git 2.32).
# read "master" branch of ''child'' to the subdirectory ''dir-child'': '''git read-tree --prefix=''dir-child''/ -u ''child''/master'''
:: The ''-u'' option causes ''git read-tree'' to update the files in the working directory. See [https://git-scm.com/docs/git-read-tree git read-tree].
# record the merge result: '''git commit -m "Merge ''child'' project as our subdirectory"'''

=== About Git subtree merges ===
To pull in subsequent update from ''child'' using "subtree" merge: '''git pull -s subtree ''child'' master'''

To learn more about subtree merges, see [https://docs.github.com/en/get-started/using-git/about-git-subtree-merges About Git subtree merges].

=== How to remove a submodule ===
If a child repo was included in a parent repo using git submodules, here's how to remove it so that the child repo can be included using subtrees as documented above.

* '''git rm -r ''path/to/submodule'' '''
* '''rm -rf .git/modules/''path/to/submodule'' '''

If the second line isn't used, even if you removed the submodule for now, the remnant .git/modules/the_submodule folder will prevent the same submodule from being added back or replaced in the future.

Also, using just these two commands will leave an entry for the submodule in ''.git/config''. To remove that,

* '''git config -f .git/config --remove-section submodule.''path/to/submodule'' '''

Here is an older way that illustrates where all the information is held:
;How do I delete a submodule?
NB: Ignore the submodule help info about ''deinit'' since that seems to leave too much undone.

To remove a submodule you need to:
* Delete the relevant line from the ''.gitmodules'' file.
* Delete the relevant section from ''.git/config''.
* Delete the submodule info in ''.git/modules'': '''rm -rf .git/modules/<path_to_submodule>'''
* Run '''git rm --cached path_to_submodule''' (no trailing slash).
* Commit the parent repo ('''git commit -m "Removed submodule ..."''')
* Delete the now untracked submodule files.

== Someone forked and issued a PR on our repo, but ... ==
The Travis build failed because that person is not allowed access to our AWS.

One way is to copy their branch to our remote (aka the 'origin' remote) and issue a PR on it.

# Set up their remote as one you can reference.
# Fetch the branches of that remote (you can fetch just the one branch)
# Checkout that remote/branch combo.
# Checkout with branching to move it to your default remote (which is liley called 'origin').
# Push that branch to github

Here are the commands (with a real example):

# git remote add https://github.com/Bo98/libdap4.git
# git fetch Bo98
# git checkout Bo98/libtirpc-fix // Bo98 is the remote, libtirpc-fix is the branch
# git checkout -b libtirpc-fix // That makes the code just checked out a branch for the 'origin' remote
# get push -u origin libtirpc-fix // Now that code is a branch in our repo and Travis will work.

== Cheat sheet items ==
These are simple things that are not really obvious from the git book or other sources

; How do I see the most recent commit date for all my branches?
:: git branch --sort=creatordate --sort=committername --format "%(align:20) %(creatordate:relative) %(end) %(align:25) %(committername) %(end) %(refname:lstrip=-1)"

;About ''git rebase''; I have a branch with lots of commits and I want to squash them. How? Oh, I pushed those commits to github...
:: The trick is to use ''git log'' to find the commit hash of the starting point for ''rebase'' and ''git rebase --interactive'' and then ''git push origin +<branch>''.
:: '''''Here's a HowTo on [[Squashing commits]]'''''

;I forked someone's repo, now I want to synch to their 'master' branch. How?
: Follow these steps
:: Set up an 'upstream' remote: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork
:: Then do these operations to get and merge the changes to 'master': https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork

;I keep getting a 'Permission Denied (publickey)' error when I push!
: Follow these steps to make and use a public/private key pair for github
:: cd ~/.ssh.
:: Within .ssh, there should be these two files: id_rsa and id_rsa.pub. If those two files are not there...
:: To create the SSH keys, type ssh-keygen -t rsa -C "your_email@example.com".
:: Open id_rsa.pub in a text editor, and copy the contents, exactly as it appears, of id_rsa.pub
:: and paste it into GitHub and/or BitBucket under the Account Menu (upper right corner) Settings > SSH Keys.

;I just made a perfectly good commit to the wrong branch. How do I undo the last commit in my master branch and then take those same changes and get them into my upgrade branch?
:If you haven't yet pushed your changes, you can also do a soft reset:
:''git reset --soft HEAD^''
:This will revert the commit, but put the committed changes back into your index. Assuming the branches are relatively up-to-date with regard to each other, git will let you do a checkout into the other branch, whereupon you can simply commit:
:''git checkout [-b] branch''
:''git commit''
:The disadvantage is that you need to re-enter your commit message.

;How to see a list of 'conflicted' files after a merge
:git diff --name-only --diff-filter=U
;How to see the difference between to commits
:git diff <commit-hash-1> <commit-hash-2>, e.g., git diff 0da94be 59ff30c
:...for a specific file: git diff <commit-hash-1> <commit-hash-2> -- <file>
:...and don't forget the shorthand for the hashes: git diff HEAD^^..HEAD -- main.c where ''HEAD^'' is the parent of HEAD. HEAD{n} is the Nth parent.
;How to see the different remote branches:
:git remote show origin
;Fetch all the branches on ''origin''
:git fetch origin
;How do I list the remote branches (that have been fetched)?
:git branch -a
;How do I switch to a branch from a remote origin?
:git checkout -b test origin/test
:or, with newer versions of git<nowiki>:</nowiki> git checkout test
;How do I see what would be pushed to a remote repo?
:git push --dry-run
:git diff origin/master # Assumes you have run git fetch, I think
:git diff --stat origin/master # --stat just shows the file names stats, not the diffs
;To get a specific file from a specific branch
:git show dap4:./gdal_dds.cc > gdal_dds.dap4.cc ''You can use checkout instead of show and that will overwrite the file.''
:the general syntax is ''object'' (that's the 'dap4:./gdal_dds.cc' part) and it can use the ^ and ~n syntax to specify various commits on the given branch. A SHA can also be used.
;How to change the 'origin' for a remote repo
:git remote set-url origin <nowiki>git://new.url.here</nowiki> (https URLs work too...)
;How to push a local branch to a remote repo
:git push -u origin feature_branch_name
;How to make and track a new (local) branch
;How to cause Travis CI to skip a build
:Add ''[ci skip]'' to the log text. See the about topic on amending a commit log, which can be handy
:git checkout -b <branch name>
;How to track a remote branch
:git checkout --track origin/serverfix '' or'' git checkout -b sf origin/serverfix
;How do I make an ''existing'' local branch track an existing remote branch?
:git branch --set-upstream upstream/foo foo where ''upstream'' is probably actually ''origin''.
;Commited my code, then made a bunch of changes that just seem like a bad idea in retrospect. How do I go back to my previous commit for everything in a directory? ''I don't care if I loose all my changes since the last commit.''
:git reset HEAD --hard (Note that this is one of the very few git commands where you really cannot undo what you have done).
;How to undo a commit (that has not been pushed)
:git reset --soft HEAD~1. This leaves the files in their changed state in your working dir so that you can edit them and recommit. You can also change to a different branch and commit there, then change back.
;In the above case, To reuse the old commit message
:git commit -c ORIG_HEAD <-- This works because 'reset' copied the old head to .git/ORIG_HEAD. If you don't need to edit the old message, use -C instead of -c.
;How to delete a remote brnach
:git push origin --delete serverfix ''The data are kept for a little bit - before git runs garbage collection - so it may be possible to undo this.''
;How to delete a local branch
:git branch -d the_local_branch ''and delete the remote branch you were tracking with the same name'' git push origin :the_remote_branch
;How to I set up a git cloned repo on a remote machine so I don't have to type my password all the time?
:This page shows how to make a PKI key-pair with a secure password, configure the machine to remember the password using ssh-agent and upload the public key to your github account so it'll use the key for authentication. https://help.github.com/articles/generating-ssh-keys/
;How can I know which branches are already merged into the master branch?
:''git branch --merged master'' lists branches merged into master
:''git branch --merged'' lists branches merged into HEAD (i.e. tip of current branch)
:''git branch --no-merged'' lists branches that have not been merged
:By default this applies to only the local branches. The -a flag will show both local and remote branches, and the -r flag shows only the remote branches.
;Switching remote URLs from HTTPS to SSH
:''git remote -v''
: # origin <nowiki>https://github.com/USERNAME/REPOSITORY.git</nowiki> (fetch)
: # origin <nowiki>https://github.com/USERNAME/REPOSITORY.git</nowiki> (push)
:''git remote set-url origin git@github.com:USERNAME/OTHERREPOSITORY.git''
:''git remote -v
: # Verify new remote URL
: # origin git@github.com:USERNAME/OTHERREPOSITORY.git (fetch)
: # origin git@github.com:USERNAME/OTHERREPOSITORY.git (push)
;Amending the commit message
:''git commit --amend''
:''git commit --amend -m "New commit message"''
; How do I revert a commit after if it has been pushed?:
:Given:
::''e512d38 Adding taunts to management.''
::''bd89039 Adding kill switch in case I'm fired.''
::''da8af4d Adding performance optimizations to master loop.''
::''db0c012 Fixing bug in the doohickey''
:If you just want to revert the commits without modifying the history, you can do the following:
:
::''git revert e512d38''
::''git revert bd89039''
:Alternatively, if you don’t want others to see that you added the kill switch and then removed it, you can roll back the repository using the following (however, this will cause problems for others who have already pulled your changes from the remote):
:
::''git reset --hard da8af4d''
::''git push origin -f localBranch:remoteBranch''
;The gitlog-to-changelog script comes in handy to generate a GNU-style ChangeLog.
:As shown by gitlog-to-changelog --help, you may select the commits used to generate a ChangeLog file using either the option --since:
:
::''gitlog-to-changelog --since=2008-01-01 > ChangeLog''
:or by passing additional arguments after --, which will be passed to git-log (called internally by gitlog-to-changelog):
:
::''gitlog-to-changelog -- -n 5 foo > last-5-commits-to-branch-foo''
;Amending the commit message
:''git commit --amend''
:
;Tagging stuff
:''git tag'' will list the existing tags
:''git tag -a <tag name>'' adds a new tag
:''git push origin <tag name>'' pushes that tag up to the server ''origin''
:''git push origin --tags'' pushes all new tags up to ''origin''
;How to resolve conflicts in a submodule when you've just merged master down to a branch
:
:Run git status - make a note of the submodule folder with conflicts
:Reset the submodule to the version that was last committed in the current branch:
:
:git reset HEAD path/to/submodule
: At this point, you have a conflict-free version of your submodule which you can now update to the latest version in the submodule's repository:
:
: cd path/to/submodule
:
: git pull origin SUBMODULE-BRANCH-NAME
: And now you can commit that and get back to work.
; How to move a submodule into the main repo
:If all you want is to put your submodule code into the main repository, you just need to remove the submodule and re-add the files into the main repo, follow the prescription below. If you want to see how to add the branches, history, etc. to the repo, see http://stackoverflow.com/questions/1759587/un-submodule-a-git-submodule:
:
:''git rm --cached submodule_path'' '''# delete reference to submodule HEAD (no trailing slash)'''
:''git rm .gitmodules'' '''# if you have more than one submodules, you need to edit this file instead of deleting!'''
:''rm -rf submodule_path/.git'' '''# make sure you have backup!!'''
:''git add submodule_path'' '''# will add files instead of commit reference'''
:''git commit -m "remove submodule"''
; Checking out a tag
:You will not be able to checkout the tags if its not locally in your repo so first you have to fetch it all.
:
:First make sure that the tag exists locally by doing
:
:# --all will fetch all the remotes.
:# --tags will fetch all tags as well
:''git fetch --all --tags --prune''
:Then check out the tag by running
:
:''git checkout tags/<tag_name> -b <branch_name>''
:Instead of origin use the tags/ prefix.
;How to remove old/unused/deleted branches
:''git remote prune origin'' prunes tracking branches not on the remote.
:''git branch --merged'' lists branches that have been merged into the current branch )but maybe including '''master''' so be careful about the next part).
:''xargs git branch -d'' deletes branches listed on standard input.
:Be '''careful''' deleting branches listed by ''git branch --merged''. The list could include master or other branches you'd prefer not to delete.
;How do I merge just one file?
:A simple command already solved the problem for me if I assume that all changes are committed in both branches A and B
:''git checkout A''
:''git checkout --patch B f''
:The first command switches into branch ''A'', into where I want to merge ''B'' 's version of the file ''f''. The second command patches the file ''f'' with ''f'' of HEAD of ''B''. You may even accept/discard single parts of the patch. Instead of ''B'' you can specify any commit here, it does not have to be HEAD.

== Continuous Integration builds involving submodules ==
There are two ways to handle getting a CI build to run when you've editing a submodule used by the BES. The **best** way is to use a branch of the BES to run the build as part of a **pull request** and is described as option number one below. Another way is to use the master branch of the BES and is describe as the second choice.

==== Using a BES branch and a GitHub pull request ====
This is the better of the two ways. It requires a bit more work but does not introduce code to the master branch before that code has passed the CI build.

Once your work on the submodule is complete:
* Commit and push the code in your submodule's branch
: git commit
: git push
* Goto the top of the BES and checkout a new branch. Choose a name that is similar to the name of the branch used for the submodule's changes
: cd bes
: git checkout -b <name>
* Commit the submodule's current commit hash to the .gitmodules file (easier than is sounds)
: 'git commit -a' or 'git add <path to submodule>' and then 'git commit'
* Push this to GitHub
: git push
* Goto Github and issue a pull request for the BES <name> branch.

This will trigger a CI build of that branch. This does not change the BES master branch at all, which is the goal here - to build without affecting the master branch.

Once the build works, merge the submodule branch to the submodule's master. Then delete the BES <name> branch and make sure to update the BES master so that it references the new master branch for the submodule.

==== An alternative, that uses the BES master branch ====

Once your code is in committed and pushed on the master branch, go to the top of the bes project and run ‘git commit -a’. This will prompt you with a commit that shows a new HDF5 handler version. Add a commit message (e.g., “New HDF5 handler version”) and then push. This works because the new hash code for the commit that is on the master branch is recorded in a hidden file managed by git (named ‘.gitmodules’). Since that file is changed, the push results in changed files in bes and than triggers a new build. The new build will reference the new commit hash for your latest changes on master and that hash will be checked out for the build.

== Merging in a branch where submodules have been removed and replaced with directories ==
Recently (02/03/17) we dropped most of the submodules in the '''bes''' (except for the ''hdf*_handler'' modules) and replaced them with regular code directories.

What to do if you have an existing branch the you need to update with the new master:

# Check in all of your changes.
# Push your branch to github.
# Start with a brand new clone for the bes repo from github:
#: ''git clone https://github.com/opendap/bes''
# DO NOT RUN ''submodule init''
# In the new clone, checkout your branch.
# In the ''modules'' directory remove crucial submodules:
#: ''rm -rf csv_handler dap-server debug_functions fileout_* fits_handler freeform_handler gateway_module gdal_handler ncml_module netcdf_handler ugrid_functions w10n_handler xml_data_handler''
# Now, merge the master to your branch:
#: ''git merge master''
# Cleanup any trouble (there should not be much)
# Build and test
# Merge your branch to master when you're ready.

File:OPeNDAP-Logo Large.png

2023-10-18T19:08:06Z

Jimg: Jimg reverted File:OPeNDAP-Logo Large.png to an old version

The rally big logo with a clear background.

File:OPeNDAP-Logo Large.png

2023-10-18T19:05:55Z

Jimg: Jimg uploaded a new version of File:OPeNDAP-Logo Large.png

The rally big logo with a clear background.

File:OPeNDAP-Logo Large.png

2023-10-18T19:03:34Z

Jimg: Jimg uploaded a new version of File:OPeNDAP-Logo Large.png

The rally big logo with a clear background.

Planning a Program Increment

2023-10-16T19:30:11Z

Jimg: /* How to Plan a Feature */

The overall scope of the work that management thinks should be done for a quarter (i.e., a PI) is given in the Candidate Feature list. The Planning Agenda describes how the 200+ people in the ESDIS project will coordinate the planning process for that work.

Information sources:
* Overall information with schedule: https://wiki.earthdata.nasa.gov/display/EPS/ESDIS+Program+SAFe
* Planning Agenda: https://wiki.earthdata.nasa.gov/display/EPS/PI+Planning+for+23.4+Agenda
* Candidate features: https://wiki.earthdata.nasa.gov/display/EPS/PI+23.4+Candidate+Features#tab-Transformation

But... The Candidate Feature list is not the only place where features are found. Other places include:
* Features that were not completed in the previous quarter (''Carryover'');
* Features that are internal to the group (''Team''); and
* Features that are important to other groups in the larger collection of groups (aka, 'train', ''Train-level'')

Regardless of source, all features are planned similarly.

== How to Plan a Feature ==
Based on the introduction, there are four kinds of features: Candidate, Carryover, Team and Train. All these are planned the same way ''except'' for the start of the process. The Candidate features have a more formal origin than the remaining three feature types.

* In Jira, write an Epic for the feature.
** Include the Acceptance Criteria (AC) for the feature in the Epic's AC - you may have to hunt around Jira's various 'Edit' features to find how to add/edit an AC
** Bind this to the PI using a Fix Version label for the PI (e.g., ''Transformation PI 23.4'')
* For that feature, write one or more tickets that describe what to do. For each ticket (normally a ''Task''):
** write an AC
** assign points
** make it visible as part of the P by assigning a Fix Version label for the given PI (e.g., ''Transformation PI 23.4'')
** assign a person
** map those tickets across the Sprints in the PI
* If there is a dependency between our work on the feature and some other team, link that using a ticket in the Epic, not the Epic itself.

If another team has feature that will depend on us in some way, make a non-Epic (e.g., Task) ticket and link that up with the non-Epic ticket in their Jira.

Why do all this labeling and linking? Because that's how the page of Objectives and Risks (e.g., https://wiki.earthdata.nasa.gov/display/EPS/Transformation+Train+-+PI+23.4+-+Objectives%2C+Risks%2C+and+Dependencies+Dashboard#tab-OPeNDAP) gets the stuff it displays.

== Candidate Features are Special ==
Because they are derived from perceived needs that span the whole ESDIS group and are blessed by NASA management, the Candidate Features are special. The only way they differ from the other three types is that their ACs are given in a Feature Planning ticket that can be found as a link on the Candidate Feature page. That AC should be used to write the AC for the Epic we make for the feature. However, we need to be circumspect about what can be done in a single quarter versus what is shown on the Feature Request page. Make sure to write an AC that' achievable. During review if something that is required has been removed, that will become apparent.

Planning a Program Increment

2023-10-16T19:08:36Z

Jimg: /* How to plan a feature */

The overall scope of the work that management thinks should be done for a quarter (i.e., a PI) is given in the Candidate Feature list. The Planning Agenda describes how the 200+ people in the ESDIS project will coordinate the planning process for that work.

Information sources:
* Overall information with schedule: https://wiki.earthdata.nasa.gov/display/EPS/ESDIS+Program+SAFe
* Planning Agenda: https://wiki.earthdata.nasa.gov/display/EPS/PI+Planning+for+23.4+Agenda
* Candidate features: https://wiki.earthdata.nasa.gov/display/EPS/PI+23.4+Candidate+Features#tab-Transformation

But... The Candidate Feature list is not the only place where features are found. Other places include:
* Features that were not completed in the previous quarter (''Carryover'');
* Features that are internal to the group (''Team''); and
* Features that are important to other groups in the larger collection of groups (aka, 'train', ''Train-level'')

Regardless of source, all features are planned similarly.

== How to Plan a Feature ==
Based on the introduction, there are four kinds of features: Candidate, Carryover, Team and Train. All these are planned the same way ''except'' for the start of the process. The Candidate features have a more formal origin than the remaining three feature types.

* In Jira, write an Epic for the feature.
** Include the Acceptance Criteria (AC) for the feature in the Epic's AC - you may have to hunt around Jira's various 'Edit' features to find how to add/edit an AC
** Bind this to the PI using a Fix Version label for the PI (e.g., ''Transformation PI 23.4'')
* For that feature, write one or more tickets that describe what to do
** For each ticket, assign a person
** assign points
** write an AC
** make it visible as part of the P by assigning a Fix Version label for the given PI (e.g., ''Transformation PI 23.4'')
** map those tickets across the Sprints in the PI
* If there is a dependency between our work on the feature and some other team, link that using a ticket in the Epic, not the Epic itself.

If another team has feature that will depend on us in some way, make a non-Epic (e.g., Task) ticket and link that up with the non-Epic ticket in their Jira.

Why do all this labeling and linking? Because that's how the page of Objectives and Risks (e.g., https://wiki.earthdata.nasa.gov/display/EPS/Transformation+Train+-+PI+23.4+-+Objectives%2C+Risks%2C+and+Dependencies+Dashboard#tab-OPeNDAP) gets the stuff it displays.

== Candidate Features are Special ==
Because they are derived from perceived needs that span the whole ESDIS group and are blessed by NASA management, the Candidate Features are special. The only way they differ from the other three types is that their ACs are given in a Feature Planning ticket that can be found as a link on the Candidate Feature page. That AC should be used to write the AC for the Epic we make for the feature. However, we need to be circumspect about what can be done in a single quarter versus what is shown on the Feature Request page. Make sure to write an AC that' achievable. During review if something that is required has been removed, that will become apparent.

Planning a Program Increment

2023-10-16T19:08:06Z

Jimg: /* How to plan a feature */

The overall scope of the work that management thinks should be done for a quarter (i.e., a PI) is given in the Candidate Feature list. The Planning Agenda describes how the 200+ people in the ESDIS project will coordinate the planning process for that work.

Information sources:
* Overall information with schedule: https://wiki.earthdata.nasa.gov/display/EPS/ESDIS+Program+SAFe
* Planning Agenda: https://wiki.earthdata.nasa.gov/display/EPS/PI+Planning+for+23.4+Agenda
* Candidate features: https://wiki.earthdata.nasa.gov/display/EPS/PI+23.4+Candidate+Features#tab-Transformation

But... The Candidate Feature list is not the only place where features are found. Other places include:
* Features that were not completed in the previous quarter (''Carryover'');
* Features that are internal to the group (''Team''); and
* Features that are important to other groups in the larger collection of groups (aka, 'train', ''Train-level'')

Regardless of source, all features are planned similarly.

== How to plan a feature ==
Based on the introduction, there are four kinds of features: Candidate, Carryover, Team and Train. All these are planned the same way ''except'' for the start of the process. The Candidate features have a more formal origin than the remaining three feature types.

* In Jira, write an Epic for the feature.
** Include the Acceptance Criteria (AC) for the feature in the Epic's AC - you may have to hunt around Jira's various 'Edit' features to find how to add/edit an AC
** Bind this to the PI using a Fix Version label for the PI (e.g., ''Transformation PI 23.4'')
* For that feature, write one or more tickets that describe what to do
** For each ticket, assign a person
** assign points
** write an AC
** make it visible as part of the P by assigning a Fix Version label for the given PI (e.g., ''Transformation PI 23.4'')
** map those tickets across the Sprints in the PI
* If there is a dependency between our work on the feature and some other team, link that using a ticket in the Epic, not the Epic itself.

If another team has feature that will depend on us in some way, make a non-Epic (e.g., Task) ticket and link that up with the non-Epic ticket in their Jira.

Why do all this labeling and linking? Because that's how the page of Objectives and Risks (e.g., https://wiki.earthdata.nasa.gov/display/EPS/Transformation+Train+-+PI+23.4+-+Objectives%2C+Risks%2C+and+Dependencies+Dashboard#tab-OPeNDAP) gets the stuff it displays.

== Candidate Features are Special ==
Because they are derived from perceived needs that span the whole ESDIS group and are blessed by NASA management, the Candidate Features are special. The only way they differ from the other three types is that their ACs are given in a Feature Planning ticket that can be found as a link on the Candidate Feature page. That AC should be used to write the AC for the Epic we make for the feature. However, we need to be circumspect about what can be done in a single quarter versus what is shown on the Feature Request page. Make sure to write an AC that' achievable. During review if something that is required has been removed, that will become apparent.

Planning a Program Increment

2023-10-16T18:45:51Z

Jimg: /* How to plan a feature */

The overall scope of the work that management thinks should be done for a quarter (i.e., a PI) is given in the Candidate Feature list. The Planning Agenda describes how the 200+ people in the ESDIS project will coordinate the planning process for that work.

Information sources:
* Overall information with schedule: https://wiki.earthdata.nasa.gov/display/EPS/ESDIS+Program+SAFe
* Planning Agenda: https://wiki.earthdata.nasa.gov/display/EPS/PI+Planning+for+23.4+Agenda
* Candidate features: https://wiki.earthdata.nasa.gov/display/EPS/PI+23.4+Candidate+Features#tab-Transformation

But... The Candidate Feature list is not the only place where features are found. Other places include:
* Features that were not completed in the previous quarter (''Carryover'');
* Features that are internal to the group (''Team''); and
* Features that are important to other groups in the larger collection of groups (aka, 'train', ''Train-level'')

Regardless of source, all features are planned similarly.

== How to plan a feature ==
Based on the introduction, there are four kinds of features: Candidate, Carryover, Team and Train. All these are planned the same way ''except'' for the start of the process. The Candidate features have a more formal origin than the remaining three feature types.

* In Jira, write an Epic for the feature.
** Include the Acceptance Criteria (AC) for the feature in the Epic's AC - you may have to hunt around Jira's various 'Edit' features to find how to add/edit an AC
** Bind this to the PI using a Fix Version label for the PI (e.g., ''Transformation PI 23.4'')
* For that feature, write one or more tickets that describe what to do
** For each ticket, assign a person
** assign points
** write an AC
** make it visible as part of the P by assigning a Fix Version label for the given PI (e.g., ''Transformation PI 23.4'')
** map those tickets across the Sprints in the PI
* If there is a dependency between our work on the feature and some other team, link that using a ticket in the Epic, not the Epic itself.

If another team has feature that will depend on us in some way, make a non-Epic (e.g., Task) ticket and link that up with the non-Epic ticket in their Jira.

Why do all this labeling and linking? Because that's how the page of Objectives and Risks (e.g., https://wiki.earthdata.nasa.gov/display/EPS/Transformation+Train+-+PI+23.4+-+Objectives%2C+Risks%2C+and+Dependencies+Dashboard#tab-OPeNDAP) gets the stuff it displays.

Planning a Program Increment

2023-10-16T18:34:23Z

Jimg: Created page with "The overall scope of the work that management thinks should be done for a quarter (i.e., a PI) is given in the Candidate Feature list. The Planning Agenda describes how the 200+ people in the ESDIS project will coordinate the planning process for that work. Information sources: * Overall information with schedule: https://wiki.earthdata.nasa.gov/display/EPS/ESDIS+Program+SAFe * Planning Agenda: https://wiki.earthdata.nasa.gov/display/EPS/PI+Planning+for+23.4+Agenda * Ca..."

Developer Info

2023-10-16T16:54:15Z

Jimg: /* OPeNDAP Development process information */

* [https://github.com/OPENDAP OPeNDAP's GitHub repositories]: OPeNDAP's software is available using GitHub in addition to the downloads from our website.
** Before 2015 we hosted our own SVN repository. It's still online and available, but for read-only access, at [https://scm.opendap.org/svn https://scm.opendap.org/svn].
* [https://travis-ci.org/OPENDAP Continuous Integration builds]: Software that is built whenever new changes are pushed to the master branch. These builds are done on the Travis-CI system.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* We use the Coverity static system to look for common software defects, information on Hyrax is spread across three projects:
** [https://scan.coverity.com/projects/opendap-bes?tab=overview The BES and the standard handlers we distribute]
** [https://scan.coverity.com/projects/opendap-olfs?tab=overview The OLFS - the front end to the Hyrax data server]
** [https://scan.coverity.com/projects/opendap-libdap4?tab=overview libdap - The implementation of DAP2 and DAP4]

== OPeNDAP's FAQ ==
The [http://www.opendap.org/faq-page OPeNDAP FAQ] has a pretty good section on developer's questions.

== C++ Coding Information ==
* [[Include files for libdap | Guidelines for including headers]]
* [[Using lambdas with the STL]]
* [[Better Unit tests for C++]]
* [[Better Singleton classes C++]]
* [[What is faster? stringstream string + String]]

== OPeNDAP Workshops ==
* [http://www.opendap.org/about/workshops-and-presentations/2007-10-12 The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/about/workshops-and-presentations/2008-07-15 ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.

== libdap4 and BES Reference documentation ==
* [https://opendap.github.io/bes/html/ BES Reference]
* [https://opendap.github.io/libdap4/html/ libdap Reference]

== BES Development Information ==
* [[Hyrax - Logging Configuration|Logging Configuration]]

* [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
* [[BES - Debugging Using besstandalone]]
* [[Hyrax - Create BES Module | How to create your own BES Module]]
* Hyrax Module Integration: How to configure your module so it's easy to add to Hyrax instances ([[:File:HyraxModuleIntegration-1.2.pdf|pdf]])
* [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
* [[Hyrax - Running bescmdln | Running the BES command line client]]
* [[Hyrax - BES Client commands| BES Client commands]]. The page [[BES_XML_Commands | BES XML Commands]] repeats this info for a bit more information on the return values. Most of the commands don't return anything unless they return an error and are expected to be used in a group where a ''get'' command closes out the request and obviously does return a response of some kind (maybe an error).
* [[Hyrax:_BES_Administrative_Commands| BES Administrative Commands]]
* [[Hyrax - Extending BES Module | Extending your BES Module]]
* [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
* [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]

* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]

== OPeNDAP Development process information ==
These pages contain information about how we'd like people working with us to use our various on-line tools.

* [[Planning a Program Increment]] This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
* [[Hyrax GitHub Source Build]] This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
* [[Bug Prioritization]]. How we prioritize bugs in our software.

===[[How to Make a Release|Making A Release]] ===
* [[How to Make a Release]] A general template for making a release. This references some of the pages below.

== Software process issues: ==
* [[How to download test logs from a Travis build]] All of our builds on Travis that run tests save those logs to an S3 bucket.
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]] - Updated 12/2014 to include information regarding git.
* [[How to use CLion with our software]]
* [[BES Timing| How to add timing instrumentation to your BES code.]]
* [[UnitTests| How to write unit tests using CppUnit]] NB: See other information under the heading of C++ development
* [[valgrind| How to use valgrind with unit tests]]
* [[Debugging the distcheck target]] Yes, this gets its own page...
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[Managing public and private keys using gpg]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
* [[UserSupport|How to Handle Email-list Support Questions]]
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [http://semver.org/ Software version numbers]
* [[GuideLines| Development Guidelines]]
* [[Apple M1 Special Needs]]

==== Older info of limited value: ====
* [http://gcc.gnu.org/gcc-4.4/cxx0x_status.html C++-11 gcc/++-4.4 support] We now require compilers that support C++-14, so this is outdated (4/19/23).
* [[How to use Eclipse with Hyrax Source Code]] I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.

==== AWS Tips ====
* [[Growing a CentOS Root Partition on an AWS EC2 Instance]]
* [[How Shutoff the CentOS firewall]]

== General development information ==
These pages contain general information relevant to anyone working with our software:

* '''[[Git Hacks and Tricks]]''': Information about using git and/or GitHub that seems useful and maybe not all that obvious.
* [[Git Secrets]]: securing repositories from AWS secret key leaks.
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind Suppression File Howto] How to build a suppressions file for valgrind.
* [[Using a debugger for C++ with Eclipse on OS/X]] Short version: use lldbmi2 **Add info**
* [[Using ASAN]] Short version, look [https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger at the Google/GitHub pages] for useful environment variables **add text** On Centos, use yum install llvm to get the 'symbolizer' and try ''ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer)''
* [[How to use ''Instruments'' on OS/X to profile]] Updated 7/2018
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind - How to generate suppression files for valgrind] This will quiet valgrind, keeping it from telling you OS/X or Linux (or the BES) is leaking memory.
* [[Migrating source code from SVN to git]]: How to move a large project from SVN to git and keep the history, commits, branches and tags.
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]. Updated in 2017, this is really good but be aware that it's specific to Mozilla so some of the tips don't apply. Hyrax (i.e., libdap4 and BES) also use their own build system (autotools + make) so most of the configuration information here is very apropos. See also [[How to use Eclipse with Hyrax Source Code]] below.
* [https://jfearn.fedorapeople.org/en-US/RPM/4/html/RPM_Guide/index.html RPM Guide] The best one I'm found so far...
* [https://autotools.io/index.html Autotools Myth busters] The best info on autotools I've found yet (covers ''autoconf'', ''automake'', ''libtool'' and ''pkg-config'').
* The [https://www.gnu.org/software/autoconf/autoconf.html autoconf] manual
* The [https://www.gnu.org/software/automake/ automake] manual
* The [https://www.gnu.org/software/libtool/ libtool] manual
* A good [https://lldb.llvm.org/lldb-gdb.html gdb to lldb cheat sheet] for those of us who know ''gdb'' but not ''lldb''.

= Old information =
'''Note''': Old build information
====The Release Process====
# Make sure the <tt>hyrax-dependencies</tt> project is up to date and tar balls on www.o.o. If there have been changes/updates:
## Update version number for the <tt>hyrax-dependencies</tt> in the <tt>Makefile</tt>
## Save, commit, (merge?), and push the changes to the <tt>master</tt> branch.
## Once the <tt>hyrax-dependencies</tt> CI build is finished, trigger CI builds for both <tt>libdap4</tt> and <tt>bes</tt> by pushing change(s) to the master branch of each.
# [[Source_Release_for_libdap | Making a source release of libdap]]
# [[ReleaseGuide | Making a source release of the BES]].
# [[OLFSReleaseGuide| Make the OLFS release WAR file]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
# [[HyraxDockerReleaseGuide|Make the official Hyrax Docker image for the release]] When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.

====Supplemental release guides====
Old - use the packages built using the Continuous Delivery process
# [[RPM |Make the RPM Distributions]]. Follow these steps to create an RPM distribution of the software. '''Note:''' ''Now we use packages built using CI/CD, so this checklist is no longer needed.''

'''Note''': ''The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...''
* [[MergingBranches |How to merge code]]
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]]. Follow these steps when a new Matlab GUI version is ready to be released.
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
* [[TestOpendapOrg | Notes from setting up Hyrax on our new web host]]
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.

Using lambdas with the STL

2023-10-16T16:50:24Z

Jimg:

= Using lambdas with the STL =

Let's say you want to find the first instance of the variable named 'name' in one of our containers. The thing in the container is not a string, but a complex object with lots of fields, one of which happens to be a string that holds the name of the object. Here's the old, pre-c++-11 way using a function that returns a boolean and some adapters like bind2nd() and ptr_fun().

Change code like:

<pre>
// Note that in order for this to work the second argument must not be a reference.
// jhrg 8/20/13
static bool
name_eq(D4Group *g, const string name)
{
return g->name() == name;
}

groupsIter g = find_if(grp_begin(), grp_end(), bind2nd(ptr_fun(name_eq), grp_name));
</pre>

to:
<pre>
auto g = find_if(grp_begin(), grp_end(), [name](const D4Group *g) { return g->name() == name; });
^ ^ ^
1 2 3
</pre>

Where ''[name](const D4Group *g) { return g->name() == name; }'' is a C++ Lambda function (an anonymous function).
This lambda function use uses a 'capture.' The square braces at #1 name the captured variable. Its value is taken from the current environment when the lambda is instantiated at runtime and used at #3 in the function.
At #2 the argument to the lambda function is declared as a ''const pointer'' so the compiler knows the function won't be modifying the object.
C++ STL functions like ''find_if()'' take predicates (which this lambda function is) and that can streamline code quite a bit.

The return value from find_if(...) is the iterator that references the first instance in the D4Group with a name that matches ''name''.

Note that there a many algorithms in the STL that can perform operations like searching on all the elements of a container given the beginning and ending iterators.

Git Hacks and Tricks

2023-09-26T18:37:04Z

Jimg: /* Cheat sheet items */

== Git resources ==
* The [http://git-scm.com/book/en/v2 Pro GIT] book is online at: git-scm.com
* Good cheat sheet: http://ndpsoftware.com/git-cheatsheet.html#loc=workspace;
* Info on branching from git.com: http://git-scm.com/book/en/Git-Branching-Remote-Branches
* Migration to git: http://git-scm.com/book/en/Git-and-Other-Systems-Migrating-to-Git

== Setup a username and access token for GitHub ==

:git config --global github.user <name>
:git config --global github.token <token>

:where the token is made using the instructions at https://help.github.com/articles/creating-an-access-token-for-command-line-use

If you want to configure a token for use with the OSX keychain, get the credential-osxkeychain tool with brew if you need to. Test if you have it by running '''git credential-osxkeychain''' and look for the credential-osxkeychain help message. To ''use'' the git extension, you need to enter '''git credential-osxkeychain <command>''' and then, on the next line, enter '''host=github.com''' and maybe '''protocol=https''' and other key/value pairs and then a blank line. See the examples below.

To use the osx keychain, first check if you have a password/token already saved:

:git credential-osxkeychain get
::host=github.com
::protocol=https
::<cr>

Erase the password/token

:git credential-osxkeychain erase
::host=github.com
::protocol=https
::<cr>

Then set the new token

:git credential-osxkeychain store
::host=github.com
::protocol=https
::username=<your login>
::password=<your token>
::<cr>

Then use the '''get''' command to verify.

== Git Secrets ==

Use this tool, which is run automatically before each commit, to keep from adding AWS and other secret keys to code that is destined for a public repository. Doing that will 'leak' the key and Amazon _will_ notice. The remedy will involve every account changing its password and every key pair being 'rotated' (i.e., every key pair has to be replaced with t anew one).

https://github.com/awslabs/git-secrets

Scroll down to the bottom for installation instructions. On OSX, you can use brew and do not have to clone the repo. Here's what I did:

:brew install git-secrets # install _git secrets_
:git secrets --register-aws --global # configure git so all future cloned repos will use it
:git secrets --install ~/.git-templates/git-secrets # set up all existing repos so they use it.
:git config --global init.templateDir ~/.git-templates/git-secrets

You can read a bit more of the docs in the _git secrets_ repo and configure a more fine grained approach.

== Subtrees: how to incorporate code from other repositories ==
Git subtrees are an alternative to submodules and are easier for the users of the parent repository (in our case, typically the BES repo). There is a fair amount of information about subtrees, although the main thing to know is that once the code for the child repo is part of the parent, in most cases there's nothing else to do. Only when changes get made in the code from the child repo are extra steps to keep the parent's copy of the code and child repo in sync needed (and then, only if you want to keep them in sync). For normal branch-PR-merge operations, there is no need to think about the subtree management commands.

Here's a [https://winstonkotzan.com/blog/2016/09/26/git-submodule-vs-subtree.html discussion about the differences between submodules and subtrees].

=== How to incorporate code from another repo ===
To incorporate code from another repo (the ''child'') into an existing repo (the ''parent''), use these steps:

# name the other project ''child'', and fetch: '''git remote add -f ''child'' <nowiki>https://github.com/</nowiki>...'''
:: The ''-f'' option runs ''git fetch'' automatically after the remote repo is added. See [https://git-scm.com/docs/git-remote git remote].
# prepare for the later step to record the result as a merge.: '''git merge -s ours --no-commit --allow-unrelated-histories ''child''/master'''
:: The ''-s'' option to ''git merge'' selects the ''ours'' strategy for the merge; ''--no-commit'' does not commit the merge automatically. See [https://git-scm.com/docs/git-merge#_merge_strategies git merge]. If you are using git 2.9+, add the option ''--allow-unrelated-histories'', but older versions of git don't support that (as of Jan. 2022, OSX was using git 2.32).
# read "master" branch of ''child'' to the subdirectory ''dir-child'': '''git read-tree --prefix=''dir-child''/ -u ''child''/master'''
:: The ''-u'' option causes ''git read-tree'' to update the files in the working directory. See [https://git-scm.com/docs/git-read-tree git read-tree].
# record the merge result: '''git commit -m "Merge ''child'' project as our subdirectory"'''

=== About Git subtree merges ===
To pull in subsequent update from ''child'' using "subtree" merge: '''git pull -s subtree ''child'' master'''

To learn more about subtree merges, see [https://docs.github.com/en/get-started/using-git/about-git-subtree-merges About Git subtree merges].

=== How to remove a submodule ===
If a child repo was included in a parent repo using git submodules, here's how to remove it so that the child repo can be included using subtrees as documented above.

* '''git rm -r ''path/to/submodule'' '''
* '''rm -rf .git/modules/''path/to/submodule'' '''

If the second line isn't used, even if you removed the submodule for now, the remnant .git/modules/the_submodule folder will prevent the same submodule from being added back or replaced in the future.

Also, using just these two commands will leave an entry for the submodule in ''.git/config''. To remove that,

* '''git config -f .git/config --remove-section submodule.''path/to/submodule'' '''

Here is an older way that illustrates where all the information is held:
;How do I delete a submodule?
NB: Ignore the submodule help info about ''deinit'' since that seems to leave too much undone.

To remove a submodule you need to:
* Delete the relevant line from the ''.gitmodules'' file.
* Delete the relevant section from ''.git/config''.
* Delete the submodule info in ''.git/modules'': '''rm -rf .git/modules/<path_to_submodule>'''
* Run '''git rm --cached path_to_submodule''' (no trailing slash).
* Commit the parent repo ('''git commit -m "Removed submodule ..."''')
* Delete the now untracked submodule files.

== Someone forked and issued a PR on our repo, but ... ==
The Travis build failed because that person is not allowed access to our AWS.

One way is to copy their branch to our remote (aka the 'origin' remote) and issue a PR on it.

# Set up their remote as one you can reference.
# Fetch the branches of that remote (you can fetch just the one branch)
# Checkout that remote/branch combo.
# Checkout with branching to move it to your default remote (which is liley called 'origin').
# Push that branch to github

Here are the commands (with a real example):

# git remote add https://github.com/Bo98/libdap4.git
# git fetch Bo98
# git checkout Bo98/libtirpc-fix // Bo98 is the remote, libtirpc-fix is the branch
# git checkout -b libtirpc-fix // That makes the code just checked out a branch for the 'origin' remote
# get push -u origin libtirpc-fix // Now that code is a branch in our repo and Travis will work.

== Cheat sheet items ==
These are simple things that are not really obvious from the git book or other sources

;About ''git rebase''; I have a branch with lots of commits and I want to squash them. How? Oh, I pushed those commits to github...
:: The trick is to use ''git log'' to find the commit hash of the starting point for ''rebase'' and ''git rebase --interactive'' and then ''git push origin +<branch>''.
:: '''''Here's a HowTo on [[Squashing commits]]'''''

;I forked someone's repo, now I want to synch to their 'master' branch. How?
: Follow these steps
:: Set up an 'upstream' remote: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork
:: Then do these operations to get and merge the changes to 'master': https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/syncing-a-fork

;I keep getting a 'Permission Denied (publickey)' error when I push!
: Follow these steps to make and use a public/private key pair for github
:: cd ~/.ssh.
:: Within .ssh, there should be these two files: id_rsa and id_rsa.pub. If those two files are not there...
:: To create the SSH keys, type ssh-keygen -t rsa -C "your_email@example.com".
:: Open id_rsa.pub in a text editor, and copy the contents, exactly as it appears, of id_rsa.pub
:: and paste it into GitHub and/or BitBucket under the Account Menu (upper right corner) Settings > SSH Keys.

;I just made a perfectly good commit to the wrong branch. How do I undo the last commit in my master branch and then take those same changes and get them into my upgrade branch?
:If you haven't yet pushed your changes, you can also do a soft reset:
:''git reset --soft HEAD^''
:This will revert the commit, but put the committed changes back into your index. Assuming the branches are relatively up-to-date with regard to each other, git will let you do a checkout into the other branch, whereupon you can simply commit:
:''git checkout [-b] branch''
:''git commit''
:The disadvantage is that you need to re-enter your commit message.

;How to see a list of 'conflicted' files after a merge
:git diff --name-only --diff-filter=U
;How to see the difference between to commits
:git diff <commit-hash-1> <commit-hash-2>, e.g., git diff 0da94be 59ff30c
:...for a specific file: git diff <commit-hash-1> <commit-hash-2> -- <file>
:...and don't forget the shorthand for the hashes: git diff HEAD^^..HEAD -- main.c where ''HEAD^'' is the parent of HEAD. HEAD{n} is the Nth parent.
;How to see the different remote branches:
:git remote show origin
;Fetch all the branches on ''origin''
:git fetch origin
;How do I list the remote branches (that have been fetched)?
:git branch -a
;How do I switch to a branch from a remote origin?
:git checkout -b test origin/test
:or, with newer versions of git<nowiki>:</nowiki> git checkout test
;How do I see what would be pushed to a remote repo?
:git push --dry-run
:git diff origin/master # Assumes you have run git fetch, I think
:git diff --stat origin/master # --stat just shows the file names stats, not the diffs
;To get a specific file from a specific branch
:git show dap4:./gdal_dds.cc > gdal_dds.dap4.cc ''You can use checkout instead of show and that will overwrite the file.''
:the general syntax is ''object'' (that's the 'dap4:./gdal_dds.cc' part) and it can use the ^ and ~n syntax to specify various commits on the given branch. A SHA can also be used.
;How to change the 'origin' for a remote repo
:git remote set-url origin <nowiki>git://new.url.here</nowiki> (https URLs work too...)
;How to push a local branch to a remote repo
:git push -u origin feature_branch_name
;How to make and track a new (local) branch
;How to cause Travis CI to skip a build
:Add ''[ci skip]'' to the log text. See the about topic on amending a commit log, which can be handy
:git checkout -b <branch name>
;How to track a remote branch
:git checkout --track origin/serverfix '' or'' git checkout -b sf origin/serverfix
;How do I make an ''existing'' local branch track an existing remote branch?
:git branch --set-upstream upstream/foo foo where ''upstream'' is probably actually ''origin''.
;Commited my code, then made a bunch of changes that just seem like a bad idea in retrospect. How do I go back to my previous commit for everything in a directory? ''I don't care if I loose all my changes since the last commit.''
:git reset HEAD --hard (Note that this is one of the very few git commands where you really cannot undo what you have done).
;How to undo a commit (that has not been pushed)
:git reset --soft HEAD~1. This leaves the files in their changed state in your working dir so that you can edit them and recommit. You can also change to a different branch and commit there, then change back.
;In the above case, To reuse the old commit message
:git commit -c ORIG_HEAD <-- This works because 'reset' copied the old head to .git/ORIG_HEAD. If you don't need to edit the old message, use -C instead of -c.
;How to delete a remote brnach
:git push origin --delete serverfix ''The data are kept for a little bit - before git runs garbage collection - so it may be possible to undo this.''
;How to delete a local branch
:git branch -d the_local_branch ''and delete the remote branch you were tracking with the same name'' git push origin :the_remote_branch
;How to I set up a git cloned repo on a remote machine so I don't have to type my password all the time?
:This page shows how to make a PKI key-pair with a secure password, configure the machine to remember the password using ssh-agent and upload the public key to your github account so it'll use the key for authentication. https://help.github.com/articles/generating-ssh-keys/
;How can I know which branches are already merged into the master branch?
:''git branch --merged master'' lists branches merged into master
:''git branch --merged'' lists branches merged into HEAD (i.e. tip of current branch)
:''git branch --no-merged'' lists branches that have not been merged
:By default this applies to only the local branches. The -a flag will show both local and remote branches, and the -r flag shows only the remote branches.
;Switching remote URLs from HTTPS to SSH
:''git remote -v''
: # origin <nowiki>https://github.com/USERNAME/REPOSITORY.git</nowiki> (fetch)
: # origin <nowiki>https://github.com/USERNAME/REPOSITORY.git</nowiki> (push)
:''git remote set-url origin git@github.com:USERNAME/OTHERREPOSITORY.git''
:''git remote -v
: # Verify new remote URL
: # origin git@github.com:USERNAME/OTHERREPOSITORY.git (fetch)
: # origin git@github.com:USERNAME/OTHERREPOSITORY.git (push)
;Amending the commit message
:''git commit --amend''
:''git commit --amend -m "New commit message"''
; How do I revert a commit after if it has been pushed?:
:Given:
::''e512d38 Adding taunts to management.''
::''bd89039 Adding kill switch in case I'm fired.''
::''da8af4d Adding performance optimizations to master loop.''
::''db0c012 Fixing bug in the doohickey''
:If you just want to revert the commits without modifying the history, you can do the following:
:
::''git revert e512d38''
::''git revert bd89039''
:Alternatively, if you don’t want others to see that you added the kill switch and then removed it, you can roll back the repository using the following (however, this will cause problems for others who have already pulled your changes from the remote):
:
::''git reset --hard da8af4d''
::''git push origin -f localBranch:remoteBranch''
;The gitlog-to-changelog script comes in handy to generate a GNU-style ChangeLog.
:As shown by gitlog-to-changelog --help, you may select the commits used to generate a ChangeLog file using either the option --since:
:
::''gitlog-to-changelog --since=2008-01-01 > ChangeLog''
:or by passing additional arguments after --, which will be passed to git-log (called internally by gitlog-to-changelog):
:
::''gitlog-to-changelog -- -n 5 foo > last-5-commits-to-branch-foo''
;Amending the commit message
:''git commit --amend''
:
;Tagging stuff
:''git tag'' will list the existing tags
:''git tag -a <tag name>'' adds a new tag
:''git push origin <tag name>'' pushes that tag up to the server ''origin''
:''git push origin --tags'' pushes all new tags up to ''origin''
;How to resolve conflicts in a submodule when you've just merged master down to a branch
:
:Run git status - make a note of the submodule folder with conflicts
:Reset the submodule to the version that was last committed in the current branch:
:
:git reset HEAD path/to/submodule
: At this point, you have a conflict-free version of your submodule which you can now update to the latest version in the submodule's repository:
:
: cd path/to/submodule
:
: git pull origin SUBMODULE-BRANCH-NAME
: And now you can commit that and get back to work.
; How to move a submodule into the main repo
:If all you want is to put your submodule code into the main repository, you just need to remove the submodule and re-add the files into the main repo, follow the prescription below. If you want to see how to add the branches, history, etc. to the repo, see http://stackoverflow.com/questions/1759587/un-submodule-a-git-submodule:
:
:''git rm --cached submodule_path'' '''# delete reference to submodule HEAD (no trailing slash)'''
:''git rm .gitmodules'' '''# if you have more than one submodules, you need to edit this file instead of deleting!'''
:''rm -rf submodule_path/.git'' '''# make sure you have backup!!'''
:''git add submodule_path'' '''# will add files instead of commit reference'''
:''git commit -m "remove submodule"''
; Checking out a tag
:You will not be able to checkout the tags if its not locally in your repo so first you have to fetch it all.
:
:First make sure that the tag exists locally by doing
:
:# --all will fetch all the remotes.
:# --tags will fetch all tags as well
:''git fetch --all --tags --prune''
:Then check out the tag by running
:
:''git checkout tags/<tag_name> -b <branch_name>''
:Instead of origin use the tags/ prefix.
;How to remove old/unused/deleted branches
:''git remote prune origin'' prunes tracking branches not on the remote.
:''git branch --merged'' lists branches that have been merged into the current branch )but maybe including '''master''' so be careful about the next part).
:''xargs git branch -d'' deletes branches listed on standard input.
:Be '''careful''' deleting branches listed by ''git branch --merged''. The list could include master or other branches you'd prefer not to delete.
;How do I merge just one file?
:A simple command already solved the problem for me if I assume that all changes are committed in both branches A and B
:''git checkout A''
:''git checkout --patch B f''
:The first command switches into branch ''A'', into where I want to merge ''B'' 's version of the file ''f''. The second command patches the file ''f'' with ''f'' of HEAD of ''B''. You may even accept/discard single parts of the patch. Instead of ''B'' you can specify any commit here, it does not have to be HEAD.

== Continuous Integration builds involving submodules ==
There are two ways to handle getting a CI build to run when you've editing a submodule used by the BES. The **best** way is to use a branch of the BES to run the build as part of a **pull request** and is described as option number one below. Another way is to use the master branch of the BES and is describe as the second choice.

==== Using a BES branch and a GitHub pull request ====
This is the better of the two ways. It requires a bit more work but does not introduce code to the master branch before that code has passed the CI build.

Once your work on the submodule is complete:
* Commit and push the code in your submodule's branch
: git commit
: git push
* Goto the top of the BES and checkout a new branch. Choose a name that is similar to the name of the branch used for the submodule's changes
: cd bes
: git checkout -b <name>
* Commit the submodule's current commit hash to the .gitmodules file (easier than is sounds)
: 'git commit -a' or 'git add <path to submodule>' and then 'git commit'
* Push this to GitHub
: git push
* Goto Github and issue a pull request for the BES <name> branch.

This will trigger a CI build of that branch. This does not change the BES master branch at all, which is the goal here - to build without affecting the master branch.

Once the build works, merge the submodule branch to the submodule's master. Then delete the BES <name> branch and make sure to update the BES master so that it references the new master branch for the submodule.

==== An alternative, that uses the BES master branch ====

Once your code is in committed and pushed on the master branch, go to the top of the bes project and run ‘git commit -a’. This will prompt you with a commit that shows a new HDF5 handler version. Add a commit message (e.g., “New HDF5 handler version”) and then push. This works because the new hash code for the commit that is on the master branch is recorded in a hidden file managed by git (named ‘.gitmodules’). Since that file is changed, the push results in changed files in bes and than triggers a new build. The new build will reference the new commit hash for your latest changes on master and that hash will be checked out for the build.

== Merging in a branch where submodules have been removed and replaced with directories ==
Recently (02/03/17) we dropped most of the submodules in the '''bes''' (except for the ''hdf*_handler'' modules) and replaced them with regular code directories.

What to do if you have an existing branch the you need to update with the new master:

# Check in all of your changes.
# Push your branch to github.
# Start with a brand new clone for the bes repo from github:
#: ''git clone https://github.com/opendap/bes''
# DO NOT RUN ''submodule init''
# In the new clone, checkout your branch.
# In the ''modules'' directory remove crucial submodules:
#: ''rm -rf csv_handler dap-server debug_functions fileout_* fits_handler freeform_handler gateway_module gdal_handler ncml_module netcdf_handler ugrid_functions w10n_handler xml_data_handler''
# Now, merge the master to your branch:
#: ''git merge master''
# Cleanup any trouble (there should not be much)
# Build and test
# Merge your branch to master when you're ready.

What is faster? stringstream string + String

2023-08-16T18:45:19Z

Jimg:

== Question ==

I have two string objects:string str_1, str_2. I want to concatenate to them. I can use two methods:

method 1:

<pre>
std::stringstream ss;
ss << "hello"<< "world"；
const std::string dst_str = std::move(ss.str());
</pre>

method 2:

<pre>
std::string str_1("hello");
std::string str_2("world");
const std::string dst_str = str_1 + str_2;
</pre>

Because the string's buffer is read only, when you change the string object, its buffer will destroy and create a new one to store new content. So method 1 is better than method 2? Is my understanding correct?

== Answer ==
From StackOverflow (https://stackoverflow.com/questions/30254175/is-stringstream-better-than-strings-operator-for-string-objects-concatenati)

stringstreams are complex objects compared to simple strings. Everythime you use method 1, a stringstream must be constructed, and later destructed. If you do this millions of time, the overhead will be far from neglectible.

The apparently simple <pre>ss << str_1 << str_2</pre> is in fact equivalent to <pre>std::operator<<(sst::operator<<(ss, str_1), str_2);</pre> which is not optimized for in memory concatenation, but common to all the streams.

I've done a small benchmark :

* In debug mode, method 2 is almost twice as fast as method1.

* In optimized build (verifying in the assembler file that nothing was optimized away), it's more then 27 times faster.