Difference between revisions of "ConfigureAmazonLinuxAMI"

From OPeNDAP Documentation
(Building the ncml_handler when incompatible versions of ICU are resident on the system)
Line 176: Line 176:
 
== Building the ncml_handler when incompatible versions of ICU are resident on the system ==
 
== Building the ncml_handler when incompatible versions of ICU are resident on the system ==
  
# Check to see what's on the machine in terms of ICU:
+
=== Is there a system ICU? ===
[root@ip-10-152-156-244 shrew]# rpm -qa | grep icu
+
Check to see what's on the machine in terms of ICU:  
libicu-4.2.1-9.9.amzn1.x86_64
+
<font size="2"><code>
libicu-devel-4.2.1-9.9.amzn1.x86_64
+
: [root@ip-10-152-156-244 shrew]# rpm -qa | grep icu
[root@ip-10-152-156-244 shrew]#  
+
: libicu-4.2.1-9.9.amzn1.x86_64
:
+
: libicu-devel-4.2.1-9.9.amzn1.x86_64
:(no need to be root)
+
: [root@ip-10-152-156-244 shrew]#  
:
+
</code></font>
: So we can see that icu is there but it's the version 4 code and that is massively incompatible with the version 3 code. You probably knew all of that, I think we talked about it, but I just thought I'd show the useful 'rpm -qa' (query all) command.
+
(no need to be root)
# Here's how to see where stuff is:
 
[ec2-user@ip-10-152-156-244 shrew]$ rpm -qil libicu-devel-4.2.1-9.9.amzn1.x86_64Name        : libicu-devel                Relocations: (not relocatable)
 
Version    : 4.2.1                            Vendor: Amazon.com
 
Release    : 9.9.amzn1                    Build Date: Mon 09 Jan 2012 04:52:37 PM UTC
 
Install Date: Wed 30 Jan 2013 05:39:05 AM UTC      Build Host: build-31003.build
 
Group      : Development/Libraries        Source RPM: icu-4.2.1-9.9.amzn1.src.rpm
 
.
 
.
 
.
 
Description :
 
Includes and definitions for developing with icu.
 
/usr/bin/icu-config
 
/usr/include/layout
 
/usr/include/layout/LEFontInstance.h
 
/usr/include/layout/LEGlyphFilter.h
 
/usr/include/layout/LEGlyphStorage.h
 
/usr/include/layout/LEInsertionList.h
 
/usr/include/layout/LELanguages.h
 
/usr/include/layout/LEScripts.h
 
/usr/include/layout/LESwaps.h
 
/usr/include/layout/LETypes.h
 
/usr/include/layout/LayoutEngine.h
 
/usr/include/layout/ParagraphLayout.h
 
/usr/include/layout/RunArrays.h
 
/usr/include/layout/loengine.h
 
/usr/include/layout/playout.h
 
/usr/include/layout/plruns.h
 
/usr/include/unicode
 
/usr/include/unicode/basictz.h
 
/usr/include/unicode/bms.h
 
/usr/include/unicode/bmsearch.h
 
.
 
.
 
.
 
# So if configure looks for icu-config, it'll find the one for icu 4.x which our handler cannot use - at least not without some tweaking things. I think the includes are all different (nice, eh?)
 
  
4. Put our icu-3.6/bin dir on the PATH in front of everything:
+
So we can see that icu is there but it's the version 4 code and that is massively incompatible with the version 3 code. You probably knew all of that, I think we talked about it, but I just thought I'd show the useful 'rpm -qa' (query all) command.
[ec2-user@ip-10-152-156-244 shrew]$ export PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/bin:$PATH
 
  
And configure works just fine
+
=== Where for art thou, ICU? ===
 +
Here's how to see where stuff is:
 +
<font size="2"><code>
 +
: [ec2-user@ip-10-152-156-244 shrew]$ rpm -qil libicu-devel-4.2.1-9.9.amzn1.x86_64Name        : libicu-devel                Relocations: (not relocatable)
 +
: Version    : 4.2.1                            Vendor: Amazon.com
 +
: Release    : 9.9.amzn1                    Build Date: Mon 09 Jan 2012 04:52:37 PM UTC
 +
: Install Date: Wed 30 Jan 2013 05:39:05 AM UTC      Build Host: build-31003.build
 +
: Group      : Development/Libraries        Source RPM: icu-4.2.1-9.9.amzn1.src.rpm
 +
: .
 +
: .
 +
: .
 +
: Description :
 +
: Includes and definitions for developing with icu.
 +
: /usr/bin/icu-config
 +
: /usr/include/layout
 +
: /usr/include/layout/LEFontInstance.h
 +
: /usr/include/layout/LEGlyphFilter.h
 +
: /usr/include/layout/LEGlyphStorage.h
 +
: /usr/include/layout/LEInsertionList.h
 +
: /usr/include/layout/LELanguages.h
 +
: /usr/include/layout/LEScripts.h
 +
: /usr/include/layout/LESwaps.h
 +
: /usr/include/layout/LETypes.h
 +
: /usr/include/layout/LayoutEngine.h
 +
: /usr/include/layout/ParagraphLayout.h
 +
: /usr/include/layout/RunArrays.h
 +
: /usr/include/layout/loengine.h
 +
: /usr/include/layout/playout.h
 +
: /usr/include/layout/plruns.h
 +
: /usr/include/unicode
 +
: /usr/include/unicode/basictz.h
 +
: /usr/include/unicode/bms.h
 +
: /usr/include/unicode/bmsearch.h
 +
: .
 +
: .
 +
: .
 +
</code></font>
 +
 
 +
So if configure looks for icu-config, it'll find the one for icu 4.x which our handler cannot use - at least not without some tweaking things. I think the includes are all different (nice, eh?)
 +
 
 +
=== Make the build find the ICU in our dependencies ===
 +
Put our icu-3.6/bin dir on the PATH in front of everything:
 +
<font size="2"><code>
 +
: [ec2-user@ip-10-152-156-244 shrew]$ export PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/bin:$PATH
 +
</code></font>
 +
And configure in the ncml_handler should work just fine
  
 
But when you try to start the server, massive library not found crap. That's because on linux you have to run ldconfig for a library to be found 'automatically'. As a convenience to users, icu left this step out…
 
But when you try to start the server, massive library not found crap. That's because on linux you have to run ldconfig for a library to be found 'automatically'. As a convenience to users, icu left this step out…
  
5. Add our icu-3.6/lib dir to LD_LIBRARY_PATH
+
=== Make the dynamic library load find our depencencies ICU at runtime ===
[root@ip-10-152-156-244 shrew]# export LD_LIBRARY_PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/lib/
+
Add our icu-3.6/lib dir to LD_LIBRARY_PATH
[root@ip-10-152-156-244 shrew]# besctl startStarting the BES
+
<font size="2"><code>
OK: Successfully started the BES
+
: [root@ip-10-152-156-244 shrew]# export LD_LIBRARY_PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/lib/
PID: 11975 UID: 0
+
: [root@ip-10-152-156-244 shrew]# besctl startStarting the BES
[root@ip-10-152-156-244 shrew]#  
+
: OK: Successfully started the BES
 
+
: PID: 11975 UID: 0
 +
:[root@ip-10-152-156-244 shrew]#  
 +
</code></font>
 
Start tomcat and the server is up!
 
Start tomcat and the server is up!
  
 
http://ec2-54-242-224-73.compute-1.amazonaws.com:8080/opendap/data/ncml/fnoc1_improved.ncml
 
http://ec2-54-242-224-73.compute-1.amazonaws.com:8080/opendap/data/ncml/fnoc1_improved.ncml
  
Why we don't have this BS on CentOS is likely that they use icu-3.6 and even if we build our own version, it's not being used. In fact, the reason I had Michael use 3.6 and not 4.2 was that most linux distros came with 3.6.
+
Why we don't have this these issues on CentOS is likely that they use icu-3.6 and even if we build our own version, it's not being used. In fact, the reason we used 3.6 and not 4.2 was that most linux distros came with 3.6.
 
 
Really, if we could just use TDS for aggregation, that would be fine with me! Oe maybe there's a better library out there now for the date stuff…
 

Revision as of 16:51, 4 March 2013

<< back to HowTo Guides

1 How to configure an Amazon Linux AMI for EC2 to build Hyrax

This describes how to set up a vanilla Amazon Linux AMI virtual machine so that we can build the Hyrax data server.

2 Amazon Linux AMI versions

These instructions have been amended to cover:

  • Amazon Linux AMI 2012.09

3 Initial configuration

Set up general build stuff that you need:

  • yum install java-1.6.0-openjdk java-1.6.0-openjdk-devel
  • yum install junit
  • yum install ant
  • yum install ant-junit.noarch
  • yum install make
  • yum install subversion
  • yum install gcc-c++
  • yum install flex
  • yum install bison
  • yum install curl-devel
  • yum install libxml2-devel
  • yum install libjpeg-devel
  • yum install zlib-devel
  • yum install readline-devel
  • yum install libuuid-devel
  • yum install openssl-devel

4 Other dependencies

  • yum install libicu-devel

4.1 Autotools

  • download the latest versions of autoconf, automake and libtool
  • ./configure, make, make install

5 Building Hyrax from the shrew project

In a (ba)sh shell:

Check out the shrew project
svn co https://scm.opendap.org/sv/trunk/shrew
Change your working directory to the shrew directory.
cd shrew
Source the spath file
. spath
Build the dependancies
cd src/dependencies
make


5.1 Using the shrew top level Makefile

Change your working directory to the shrew directory.
cd $prefix
Copy the Makefile.am.master to Makefile.am
cp Makefile.am.master Makefile.am
Edit the Makefile.am to be certain that the various dependancies are correctly configured.
vim Makefile.am
(Since I am trying to get a new package up I edited this file, mostly I think you should just be able to make the copy and proceed.)
Engage the autotools process and see how far you get...
autoreconf -vif
./configure --prefix=$prefix
make

If the above fails then you need to build each Hyrax component by hand.'

5.2 Building by hand

Build and install libdap
cd $prefix/src/libdap
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the BES
cd $prefix/src/bes
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the dap-server module
cd $prefix/src/modules/dap-server
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the csv_handler module
cd $prefix/src/modules/csv_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the fileout_gdal module
cd $prefix/src/modules/fileout_gdal
autoreconf -vif
./configure --prefix=$prefix --with-gdal=$prefix/deps/gdal-12.15.12
make install
Build and install the fileout_netcdf module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-4.1.2
make install
Build and install the fits_handler module
cd $prefix/src/modules/fits_handler
autoreconf -vif
./configure --prefix=$prefix --with-cfits=$prefix/deps/cfitsio
make install
Build and install the freeform_handler module
cd $prefix/src/modules/freeform_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the gateway_module module
cd $prefix/src/modules/gateway_module
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the gdal_handler module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-gdal=$prefix/deps/gdal-12.15.12
make install
Build and install the hdf4_handler module
cd $prefix/src/modules/hdf4_handler
autoreconf -vif
./configure --prefix=$prefix --with-hdf4=$prefix/deps/hdf-4.2.8
make install
Build and install the hdf5_handler module
cd $prefix/src/modules/hdf5_handler
autoreconf -vif
./configure --prefix=$prefix --with-hdf5=$prefix/deps/hdf5-1.8.6
make install
Build and install the ncml_module module
cd $prefix/src/modules/ncml_module
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the netcdf_handler module
cd $prefix/src/modules/fileout_netcdf
autoreconf -vif
./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-4.1.2
make install
Build and install the xml_data_handler module
cd $prefix/src/modules/xml_data_handler
autoreconf -vif
./configure --prefix=$prefix
make install
Build and install the ugrid_functions module
cd $prefix/src/modules/ugrid_functions
autoreconf -vif
./configure --prefix=$prefix --with-gridfields=$prefix/deps/gridfields-1.0.1
make install


6 Building the ncml_handler when incompatible versions of ICU are resident on the system

6.1 Is there a system ICU?

Check to see what's on the machine in terms of ICU:

[root@ip-10-152-156-244 shrew]# rpm -qa | grep icu
libicu-4.2.1-9.9.amzn1.x86_64
libicu-devel-4.2.1-9.9.amzn1.x86_64
[root@ip-10-152-156-244 shrew]#

(no need to be root)

So we can see that icu is there but it's the version 4 code and that is massively incompatible with the version 3 code. You probably knew all of that, I think we talked about it, but I just thought I'd show the useful 'rpm -qa' (query all) command.

6.2 Where for art thou, ICU?

Here's how to see where stuff is:

[ec2-user@ip-10-152-156-244 shrew]$ rpm -qil libicu-devel-4.2.1-9.9.amzn1.x86_64Name  : libicu-devel Relocations: (not relocatable)
Version  : 4.2.1 Vendor: Amazon.com
Release  : 9.9.amzn1 Build Date: Mon 09 Jan 2012 04:52:37 PM UTC
Install Date: Wed 30 Jan 2013 05:39:05 AM UTC Build Host: build-31003.build
Group  : Development/Libraries Source RPM: icu-4.2.1-9.9.amzn1.src.rpm
.
.
.
Description :
Includes and definitions for developing with icu.
/usr/bin/icu-config
/usr/include/layout
/usr/include/layout/LEFontInstance.h
/usr/include/layout/LEGlyphFilter.h
/usr/include/layout/LEGlyphStorage.h
/usr/include/layout/LEInsertionList.h
/usr/include/layout/LELanguages.h
/usr/include/layout/LEScripts.h
/usr/include/layout/LESwaps.h
/usr/include/layout/LETypes.h
/usr/include/layout/LayoutEngine.h
/usr/include/layout/ParagraphLayout.h
/usr/include/layout/RunArrays.h
/usr/include/layout/loengine.h
/usr/include/layout/playout.h
/usr/include/layout/plruns.h
/usr/include/unicode
/usr/include/unicode/basictz.h
/usr/include/unicode/bms.h
/usr/include/unicode/bmsearch.h
.
.
.

So if configure looks for icu-config, it'll find the one for icu 4.x which our handler cannot use - at least not without some tweaking things. I think the includes are all different (nice, eh?)

6.3 Make the build find the ICU in our dependencies

Put our icu-3.6/bin dir on the PATH in front of everything:

[ec2-user@ip-10-152-156-244 shrew]$ export PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/bin:$PATH

And configure in the ncml_handler should work just fine

But when you try to start the server, massive library not found crap. That's because on linux you have to run ldconfig for a library to be found 'automatically'. As a convenience to users, icu left this step out…

6.4 Make the dynamic library load find our depencencies ICU at runtime

Add our icu-3.6/lib dir to LD_LIBRARY_PATH

[root@ip-10-152-156-244 shrew]# export LD_LIBRARY_PATH=/home/ec2-user/opendap/shrew/deps/icu-3.6/lib/
[root@ip-10-152-156-244 shrew]# besctl startStarting the BES
OK: Successfully started the BES
PID: 11975 UID: 0
[root@ip-10-152-156-244 shrew]#

Start tomcat and the server is up!

http://ec2-54-242-224-73.compute-1.amazonaws.com:8080/opendap/data/ncml/fnoc1_improved.ncml

Why we don't have this these issues on CentOS is likely that they use icu-3.6 and even if we build our own version, it's not being used. In fact, the reason we used 3.6 and not 4.2 was that most linux distros came with 3.6.