ConfigureAmazonLinuxAMI

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽

<< back to HowTo Guides

How to configure an Amazon Linux AMI for EC2 to build Hyrax

This describes how to set up a vanilla Amazon Linux AMI virtual machine so that we can build the Hyrax data server.

Amazon Linux AMI versions

These instructions have been amended to cover:

  • Amazon Linux AMI 2012.09

Initial configuration

Set up general build stuff that you need:

  • yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel
  • yum install junit
  • yum install ant
  • yum install ant-junit.noarch
  • yum install make
  • yum install subversion
  • yum install gcc-c++
  • yum install flex
  • yum install bison
  • yum install curl-devel
  • yum install libxml2-devel
  • yum install libjpeg-devel
  • yum install zlib-devel
  • yum install readline-devel
  • yum install libuuid-devel
  • yum install openssl-devel

Other dependencies

  • yum install libicu-devel

Autotools

  • download the latest versions of autoconf, automake and libtool
  • ./configure, make, make install

Building Hyrax from the shrew project

In a (ba)sh shell:

Check out the shrew project
svn co https://scm.opendap.org/sv/trunk/shrew
Change your working directory to the shrew directory.
cd shrew
Source the spath file
. spath
Build the dependancies
cd src/dependencies
make


Using the shrew top level Makefile

Change your working directory to the shrew directory.
cd $prefix
Copy the Makefile.am.master to Makefile.am
cp Makefile.am.master Makefile.am
Edit the Makefile.am to be certain that the various dependancies are correctly configured.
vim Makefile.am
(Since I am trying to get a new package up I edited this file, mostly I think you should just be able to make the copy and proceed.)
Engage the autotools process and see how far you get...
autoreconf -vif
./configure --prefix=$prefix
make

If the above fails then you need to build each Hyrax component by hand.'

Building by hand

Build and install libdap
cd $prefix/src/libdap
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the BES
cd $prefix/src/bes
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the dap-server module
cd $prefix/src/modules/dap-server
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the csv_handler module
cd $prefix/src/modules/csv_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the fileout_gdal module
cd $prefix/src/modules/fileout_gdal
autoreconf -vif
./configure --prefix=$prefix --with-gdal=$prefix/deps/gdal-12.15.12
make install
Build and install the fileout_netcdf module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-4.1.2
make install
Build and install the fits_handler module
cd $prefix/src/modules/fits_handler
autoreconf -vif
./configure --prefix=$prefix --with-cfits=$prefix/deps/cfitsio
make install
Build and install the freeform_handler module
cd $prefix/src/modules/freeform_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the gateway_module module
cd $prefix/src/modules/gateway_module
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the gdal_handler module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-gdal=$prefix/deps/gdal-12.15.12
make install
Build and install the hdf4_handler module
cd $prefix/src/modules/hdf4_handler
autoreconf -vif
./configure --prefix=$prefix --with-hdf4=$prefix/deps/hdf-4.2.8
make install
Build and install the hdf5_handler module
cd $prefix/src/modules/hdf5_handler
autoreconf -vif
./configure --prefix=$prefix --with-hdf5=$prefix/deps/hdf5-1.8.6
make install
Build and install the ncml_module module
cd $prefix/src/modules/ncml_module
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the netcdf_handler module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-4.1.2
make install
Build and install the xml_data_handler module
cd $prefix/src/modules/xml_data_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the ugrid_functions module
cd $prefix/src/modules/ugrid_functions
autoreconf -vif
./configure --prefix=$prefix --with-gridfields=$prefix/deps/gridfields-1.0.1
make install


Building the ncml_handler when incompatible versions of ICU are resident on the system

Is there a system ICU?

Check to see what's on the machine in terms of ICU:

[root@ip-10-152-156-244 shrew]# rpm -qa | grep icu
libicu-4.2.1-9.9.amzn1.x86_64
libicu-devel-4.2.1-9.9.amzn1.x86_64
[root@ip-10-152-156-244 shrew]#

(no need to be root)

So we can see that icu is there but it's the version 4 code and that is massively incompatible with the version 3 code. You probably knew all of that, I think we talked about it, but I just thought I'd show the useful 'rpm -qa' (query all) command.

Where for art thou, ICU files?

Here's how to see where stuff is:

[ec2-user@ip-10-152-156-244 shrew]$ rpm -qil libicu-devel-4.2.1-9.9.amzn1.x86_64Name  : libicu-devel Relocations: (not relocatable)
Version  : 4.2.1 Vendor: Amazon.com
Release  : 9.9.amzn1 Build Date: Mon 09 Jan 2012 04:52:37 PM UTC
Install Date: Wed 30 Jan 2013 05:39:05 AM UTC Build Host: build-31003.build
Group  : Development/Libraries Source RPM: icu-4.2.1-9.9.amzn1.src.rpm
.
.
.
Description :
Includes and definitions for developing with icu.
/usr/bin/icu-config
/usr/include/layout
/usr/include/layout/LEFontInstance.h
/usr/include/layout/LEGlyphFilter.h
/usr/include/layout/LEGlyphStorage.h
/usr/include/layout/LEInsertionList.h
/usr/include/layout/LELanguages.h
/usr/include/layout/LEScripts.h
/usr/include/layout/LESwaps.h
/usr/include/layout/LETypes.h
/usr/include/layout/LayoutEngine.h
/usr/include/layout/ParagraphLayout.h
/usr/include/layout/RunArrays.h
/usr/include/layout/loengine.h
/usr/include/layout/playout.h
/usr/include/layout/plruns.h
/usr/include/unicode
/usr/include/unicode/basictz.h
/usr/include/unicode/bms.h
/usr/include/unicode/bmsearch.h
.
.
.

So if configure looks for icu-config, it'll find the one for icu 4.x which our handler cannot use - at least not without some tweaking things. I think the includes are all different (nice, eh?)

Make the build find the ICU in our dependencies

Put our icu-3.6/bin dir on the PATH in front of everything:

[ec2-user@ip-10-152-156-244 shrew]$ export PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/bin:$PATH

And configure in the ncml_handler should work just fine

But when you try to start the server, massive library not found crap. That's because on linux you have to run ldconfig for a library to be found 'automatically'. As a convenience to users, icu left this step out…

Make the dynamic library load find our depencencies ICU at runtime

Add our icu-3.6/lib dir to LD_LIBRARY_PATH

[root@ip-10-152-156-244 shrew]# export LD_LIBRARY_PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/lib/
[root@ip-10-152-156-244 shrew]# besctl startStarting the BES
OK: Successfully started the BES
PID: 11975 UID: 0
[root@ip-10-152-156-244 shrew]#

Start tomcat and the server is up!

http://ec2-54-242-224-73.compute-1.amazonaws.com:8080/opendap/data/ncml/fnoc1_improved.ncml

Why we don't have this these issues on CentOS is likely that they use icu-3.6 and even if we build our own version, it's not being used. In fact, the reason we used 3.6 and not 4.2 was that most linux distros came with 3.6.