Hyrax GitHub Source Build: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
mNo edit summary
Line 20: Line 20:


== Build Hyrax Using Bash Scripts ==
== Build Hyrax Using Bash Scripts ==
This should work for a correctly configured CentOS 6 computer if you have bison 3 installed. However, if it does not work proceed to the manual build. In fact, these scripts ''hyrax_clone.sh'' and ''hyrax_build.sh'' simply automate the handful of commands you need to build the code 'by-hand'. We have these because they are part of our nightly-build process.  
This should work for a correctly configured CentOS 6 computer. However, if it does not work proceed to the manual build. In fact, these scripts ''hyrax_clone.sh'' and ''hyrax_build.sh'' simply automate the handful of commands you need to build the code 'by-hand'.


;use bash: The shell scripts in this repo assume you are using ''bash''.
;use bash: The shell scripts in this repo assume you are using ''bash''.
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;set up some environment variables so the server will build an install locally, something that streamlines development: ''source spath.sh''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;clone the three code repos for the server plus the hyrax dependencies: ''./hyrax_clone.sh -v''
;install bison 3.x: Use ''bison --version'' to test the version number and if it's less than 3.x, get bison 3 (Google will find it) and install it somewhere on your PATH
;build the code, including the dependencies (note that this repo includes CMake and Bison 3): ''./hyrax_build.sh -v''
;build the code, including the dependencies: ''./hyrax_build.sh -v''
;test the server: Start the BES using  ''besctl start''
;test the server: Start the BES using  ''besctl start''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''
:Start the OLFS using''./build/apache-tomcat-7.0.57/bin/startup.sh''

Revision as of 22:51, 21 September 2015

This describes how to get and build Hyrax from our GitHub repositories. Hyrax is a data server that implements the DAP2 and DAP4 protocols, works with a number of different data formats and supports a wide variety of customization options from tailoring the look of the server's web pages to complex server-side processing operations. This page describes how to build the server's source code. If you're working on a Linux or OS/X computer, the process is similar so we describe only the linux case; we do not support building the server on Windows operating systems.

To build and install the server, you need to perform three steps:

  1. Set up the computer to build source code (Install a Java compiler; install a C/C++ compiler; add some other tools)
  2. Build, install the software. It installs with a default configuration that includes sample data.
  3. You may need to configure the server for your data

Quick links if you already know the process:


A GitHub semi-automatic build

Use git to clone the hyrax project and follow the short instructions in the README file. Summarized here, those instructions are described in the following two sections. For this to work you will need a Linux/OSX/Unix computer that is 'set up for software development.' In practice, OSX 10 with the developer tools or CentOS 6 meet these requirements with the exception that libdap requires bison 3.0++ at this time and the possible exception that you need to get Apache Ant. If you don't plan on modifying the OLFS source code, you can get the opendap.war file and skip the olfs build step.

Build Hyrax Using Bash Scripts

This should work for a correctly configured CentOS 6 computer. However, if it does not work proceed to the manual build. In fact, these scripts hyrax_clone.sh and hyrax_build.sh simply automate the handful of commands you need to build the code 'by-hand'.

use bash
The shell scripts in this repo assume you are using bash.
set up some environment variables so the server will build an install locally, something that streamlines development
source spath.sh
clone the three code repos for the server plus the hyrax dependencies
./hyrax_clone.sh -v
build the code, including the dependencies (note that this repo includes CMake and Bison 3)
./hyrax_build.sh -v
test the server
Start the BES using besctl start
Start the OLFS using./build/apache-tomcat-7.0.57/bin/startup.sh
Test the server by loooking at http://localhost:8080/opendap in a browser. You should see a directory named data and following that link should lead to more data. The server will be accessible to clients other than a web browser.
To test the BES function independently of the front end, use bescmdln and give it the show version; command, you should see output about different components and their versions.
Use exit to leave the command line test client.

GitHub build

In the following, we describe only the build process for CentOS; the one for OS/X is similar and we note the differences where they are significant. We assume you are starting with a 'bare' machine, i.e., one not set up for software development. OSX builders will have to extrapolate from the CentOS commands.

Make a Place to Build the Server

You need to make a new directory to build the server and set the environment variable prefix to be that directory name. Then export $prefix and add $prefix/bin to $PATH. I'll use hyrax-git for the new directory and assume you're using bash.

mkdir hyrax
cd hyrax
export prefix=`pwd`
export PATH=$prefix/bin:$PATH

Set up a CentOS machine to build code

Note that I don't like clicking around to different pages to follow simple directions, so what follows is a short version of the ContOS 6 configuration information we've compiled for people that help us by building RPM packages for Hyrax. The complete instructions are in Configure CentOS and describe how to to set up a CentOS 6 machine to build software. What follows is the condensed version:

Load a basic software development environment
yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel ant git gcc-c++ flex bison openssl-devel libuuid-devel readline-devel zlib-devel libjpeg-devel libxml2-devel curl-devel emacs
Check the version of bison
Use bison --version to see the version of bison. If it is less than 3.0, get and build the bison 3.x source. Make sure to update PATH so the build will use the new version of bison. The flex from yum should be OK.
Get GNU autotools
You probably have what's needed (autoconf 2.63 and automake 1.10). If you get more recent versions you can run the regression tests in parallel using make check TESTSUITEFLAGS=-jN where N roughly the number of cores on you machine. ...not needed but faster by about N/2.

Choice point: You either need to get and build the Hyrax dependencies from source (see Compile the Hyrax dependencies) or install EPEL for CentOS and then use yum to install the dependencies, with the caveat that as of Hyrax 1.10, EPEL does not have, or does not have a recent enough version of GDAL, Gridfields, HDF-EOS2 or FITS to build our code, so those parts of Hyrax won't build if you go the EPEL route. Neither way is too hard, but building the dependencies from source takes longer.

EPEL
Get epel-release-6-8.noarch.rpm and install it using sudo yum install epel-release-6-8.noarch.rpm.
Using EPEL...
Then install packages needed to read various file formats: yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build
or, Compile the dependencies
See the instructions below...

Compile the Hyrax dependencies

Note: If you installed packages for netcdf, etc., using EPEL as described above, skip this step.

Get the tar ball hyrax-dependencies and build it. Unlike many source packages, there is no need to run a configure script, just make will do. However, the Makefile in this package expects $prefix to be set as described above. It will put all of the Hyrax server dependencies in a subdirectory called $prefix/deps. Get the latest tarball from http://www.opendap.org/pub/source/, currently hyrax-dependencies-1.11.tar

(make sure you're in the directory set to $prefix)
wget http://www.opendap.org/pub/source/hyrax-dependencies-1.11.tar
tar -xf hyrax-dependencies-1.11.tar
cd hyrax-dependencies
make --jobs=9
The --jobs=N runs a parallel build with at most N simultaneous compile operations. This will result in a huge performance improvement on multi-core machines. -jN is the short form for the option.
cd ..
Go back up to $prefix
Add $prefix/deps/bin to PATH
export PATH=$prefix/deps/bin:$PATH
On Linux export LD_LIBRARY_FLAGS=$prefix/deps/libs
You will need to set the environment variable LD_LIBRARY_FLAGS to reference the directory where the dependencies package stored shared libraries or the NcML handler will not find the library. This environment variable needs to be defined in the shell used to start the server. For OS/X, skip this step.

Build libdap and the Hyrax BES daemon

Get and build libdap4

WARNING
If you have libdap already uninstall it before proceeding.

Build, test and install libdap4 into $prefix:

git clone https://github.com/opendap/libdap4
cd libdap4
autoreconf --force --install --verbose
./configure --prefix=$prefix --enable-developer
make -j9
make check -j9
make install
cd ..
Go back up to $prefix

Get and build the BES and all of the modules shipped with Hyrax

Build, test and install the BES and its modules

git clone https://github.com/opendap/bes
Clone the BES from GitHub
cd bes
git submodule update --init
This will clone each of the N modules into the directory modules. If you don't want to clone all of the modules, skip this step and see the note for the next step. See git submodule for information about all you can do with git's submodule command. Also note that this does not checkout a particular branch for the submodules; the modules are left in the 'detached head' state. To checkout a particular branch like 'master', which is important if you'll be making changes to that code, use git submodule foreach 'git checkout master' .
git submodule foreach 'git checkout master'
This will get all of the modules (which are "headless" images of the master branch) correctly associated with the github master branch.
ln -s configure_modules.ac configure.ac
This is the one non-standard operation in the clone, configure and build of the BES. If you don't want to build the modules, you would use ln -s configure_standard.ac configure.ac instead (and not clone the modules in the step above)
autoreconf --force --install --verbose
You can use -fiv instead of the long options. These mean, when starting from a freshly cloned repo, run all of the autotools commands and install all of the needed scripts.
./configure --prefix=$prefix --enable-developer --with-dependencies=$prefix/deps
Note that the --with-deps... is not needed if you load the dependencies from RPMs or otherwise have them installed an generally accessible on the build machine.
make -j9
make check -k
Some tests may fail and -k ignores that and keeps make marching along. We are not using --j9 as we did with libdap because some of the BES tests fail when run from a parallel build.
make install
cd ..
Go back up to $prefix

Test the BES

Start the BES and verify that all of the modules build correctly.

besctl start
Start the BES. Given that $prefix/bin is on your $PATH, this should start the BES. You will not need to be root if you used the --enable-developer switch with configure (as shown above), otherwise you should run sudo besctl start with the caveat that as root $prefix/bin will probably not be n your $PATH.
If there's an error (e.g., you tried to start as a regular user but need to be root), edit bes.conf to be a real user (yourself?) in a real group (use 'groups' to see which groups you are in) and also check that the bes.log file is not owned by root.
Restart.
bescmdln
Now that the BES is running, start the BES testing tool
show version;
Send the BES the version command to see if it's running
Take a quick look at the output. There should be entries for libdap, bes and all of the modules.
exit
Exit the testing tool

Note that even though you have exited the bescmdln test tool, the BES is still running. That's fine - we'll use it in just a bit - but if you want to shut it down, use besctl stop, or besctl pids to see the daemon's processes. If the BES is not stopping, besctl kill will stop all BES processes without waiting for them to complete their current task.

Build the Hyrax OLFS web application

The OLFS is a java servlet built using ant. The OLFS is a java servlet web application and runs with Tomcat, Glassfish, etc. You need a copy of Tomcat, but our servlet does not work with the RPM version of Tomcat. Get Tomcat 7 from Apache. Note that if you built the dependencies from source using the hyrac-dependencies-1.10.tar then there is a copy of Tomcat in the hyrax-dependecies/extra_downloads directory. You can unpack the Tomcat tar file in $prefix. I'll assume you have the Apache Tomcat tar file in $prefix.

tar -xzf apache-tomcat-7.0.57.tar.gz
Expand the Tomcat tar ball
git clone https://github.com/opendap/olfs
Get the OLFS source code
cd olfs
change directory to the OLFS source
ant server
Build it
cp build/dist/opendap.war ../apache-tomcat-7.0.57/webapps/
Copy the opendap web archive to the tomcat webapps direcotry.
cd ..
Go up to $prefix
./apache-tomcat-7.0.57/bin/startup.sh
Start Tomcat

Test the server

You can test the server several ways, but the most fun is to use a web browser. The URL http://<machine>:8080/opendap should return a page pointing to a collection of test datasets bundled with the server. You can also use curl, wget or any application that can read from OpenDAP servers (e.g., Matlab, Octave, ArcGIS, IDL, ...).

Stopping the server

Stop both the BES and Apache

besctl stop
./apache-tomcat-7.0.57/bin/shutdown.sh

Note that there is also a hyraxctl script that provides a way to start and stop Hyrax without you (or init.d) having to type separate commands for both the BES and OLFS. This script is part of the BES software you cloned from git.

Building select parts of the BES

Building just the BES and one of more of its handlers/modules is not at all hard to do with a checkout of code from git. In the above section on building the BES, simply skip the step where the submodules are cloned (git submodule update --init) and link configure.ac to configure_standard.ac. The rest of the process is as shown. The end result is a BES daemon without any of the standard Hyrax modules (but support for DAP will be built if libdap is found by the configure script).

To build modules for the BES, simply go to $prefix, clone their git repo and build them, taking care to pass set $prefix when calling the module's configure script.

Note that it is easy to combine the 'build it all' and 'build just one' processes so that a complete Hyrax BES can be built in one go and then a new module/handler not included in the BES git repo can be built and used. Each module we have on GitHub has a configure.ac, Makefile.am, etc., that will support both kinds of builds and Configuration of BES Modules explains how to take a module/handler that builds as a standalone module and tweak the build scripts so that it's fully integrated into the Hyrax BES build, too.

Building RPMs

There are some tricks to building the RPMs, but for the most part, its just make rpm. However, you can make this easier if you do two things:

  1. Install the EPEL yum repository. Get epel-release-6-8.noarch.rpm and install it using sudo yum install epel-release-6-8.noarch.rpm. Then install packages needed to read various file formats: yum install netcdf-devel hdf-devel hdf5-devel libicu-devel cfitsio-devel cppunit-devel rpm-devel rpm-build
  2. Set up the rpmbuild directory structure in $HOME and then build as yourself and not root. (Here are instructions for CentOS)
build the libdap rpms
cd libdap4, make rpm
install (temporarily) the two libdap rpm packages
You need to do this because libdap and libdap-devel are dependencies for the BES rpms and their builds. Use sudo yum install ~/rpmbuild/RPMS/x86_64/libdap*.rpm
build the bes rpms
cd bes, make rpm
test the bes rpms (if you want)
sudo yum install ~/rpmbuild/RPMS/x86_64/bes.rpm, then run besctl start and bescmdln. Type show version; in to the latter, you should get an xml document that lists the BES's components. Stop the server with besctl stop
remove the packages you just installed
Do this step if you're on a build host an you don't want libdap, etc., permanently installed. sudo yum remove bes bes-devel libdap libdap-devel

BES RPM for NASA

Build RPMs that are usable for NASA takes a bit more work because NASA uses hdf4/hdfeos2 for lots of its data and there is no rpm for hdfeos2. As a result, you will need to build a version of the hdf4 handler that is statically linked to the hdfeps2 and hdf4 libraries. Not a huge problem, but it's an extra step. However, there is a bonus for doing this extra work: You also get the handlers that build GeoTIFF and JPEG2000 responses, an updated FITS handler and a handler that uses GDAL to read any of the formats it supports, including data stored in GeoTIFF files.

Note: There are actually two different 'static' RPM build options for the BES, and here I describe the one that builds every handler using static linking so that EPEL is not required at all. There is also a NASA static RPM build option that is a mix of the two - it requires EPEL but also statically links some handlers, but that is an older option - use the for-static-rpm and all-static-rpm options described below. Those options will build the dependencies as static libraries only and then the BES modules so they use only those static libraries. This eliminates the need for a separate install step for the dependencies when people install the RPM - which is great because not everyone has the EPEL repo configured and, even if they do, not all of the dependencies have packages in the RPM 'ecosystem.'

There's considerable overlap with the instructions from the above section, but there are key differences for the BES parts, highlighted in red below.

Build the dependencies for the BES handlers that have no (usable) RPM in CentOS 6
cd hyrax-dependencies, make -j9 for-static-rpm CONFIGURE_FLAGS=--disable-shared
Build the libdap rpms
cd libdap4, make rpm
install (temporarily) the two libdap rpm packages
You need to do this because libdap and libdap-devel are dependencies for the BES rpms and their builds. Use sudo yum install ~/rpmbuild/RPMS/x86_64/libdap*.rpm
build the bes rpms
cd bes, make all-static-rpm
test the bes rpms (if you want)
sudo yum install ~/rpmbuild/RPMS/x86_64/bes*.rpm, then run besctl start and bescmdln. Type show version; in to the latter, you should get an xml document that lists the BES's components. Stop the server with besctl stop
remove the packages you just installed
Do this step if you're on a build host an you don't want libdap, etc., permanently installed. sudo yum remove bes bes-devel libdap libdap-devel