Hyrax Package for OS-X: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
No edit summary
 
(52 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Overview ==
==Overview ==
The goal of this build was to produce a set of OS-X package installers for Hyrax. This includes bundling all of the Hyrax dependencies in an installer. In addition, all of the installed files (Hyrax and dependencies) should be confined to a single directory (tree) for the release. For any given release I chose ''<code>/usr/local/opendap/servers/hyrax-<version></code>''. This abbreviated is throughout this section as '''$prefix'''.
The goal of this build was to produce a set of OS-X package installers for Hyrax. This includes bundling all of the Hyrax dependancies in an installer. In addition, all of the installed files (Hyrax and dependancies) should be confined to a single directory (tree) for the release. For this release I chose ''<code>/usr/local/opendap./servers/hyrax-1.6.2</code>''. This abbreviated is throughout this section as '''$prefix'''.


=== Development System ===
'''Note:''' This is based on Nathan's original detailed instructions for building on OS/X so that a MetaPackage installer can be made for Hyrax. I modified the ''shrew'' project for the Hyrax 1.6.2 release to do most of this in a semi-automated way. See [[BuildHyraxOnOSX | Building Hyrax Dependencies]] for Nathan's original narrative of the process required.
;Operating System
: OS-X 10.5.8


;Hardware
You can check out the shrew project at [https://scm.opendap.org/svn/trunk/shrew https://scm.opendap.org/svn/trunk/shrew]. We often do not use that but a copy of the shrew project on a branch. This enables everyone to know when changes are going to be slated for the next (trunk) or current (branch) release. The branch for a particular set of releases can be found by browsing the SVN sources using [https://scm.opendap.org/trac/browser Trac]. A typical example is: [https://scm.opendap.org/trac/browser/branch/shrew/hyrax_1.7_release https://scm.opendap.org/trac/browser/branch/shrew/hyrax_1.7_release], where the libdap, bes, et c., projects are referenced by the shrew project.
: MacBook Pro, 2.53GHz Intel Core 2 Duo, 4GB RAM


== Building Hyrax Dependancies ==
==Build a Hyrax MetaPackage for OS/X Using Shrew==


In an effort to build a binary distribution for Hyrax I built all of our dependancies from source and rolled them up with PackageMaker. Here's an ordered list what got built an installed. Note that with some noted exceptions, each of the dependancies gets installed into its own directory.
Based on the work that Nathan did, I've modified the '''shrew''' project to accommodate most of this process. It's almost automatic...


=== Preconditions ===
A completely up to date checkout


=== Options ===
You can use these targets to build a set of dependencies that are then used to build the various handlers and/or you can build a set of OS/X packages that can be combined into a single, huge, metapackage. There may be some utility in just doing the former, but this process has been developed with the idea that it will be used to make an OS/X metapackage for Hyrax.


=== szip ===
=== Set environment variables ===
'''Required by:''' HDF4, hdf4_handler <br/>
You must set '''$prefix''', '''$PATH''', '''$TOMCAT_DIR''' and '''$CATALINA_HOME''' for this all to work. There are several scripts that will do this for you; you need to ''source'' these for them to work. In the top-level shrew directory, ''source spath.sh'' or ''source spath.sh <prefix>'' will set all of these (the former sets $prefix to the current working directory and the latter sets it to ''<prefix>''). The ''sptah.sh'' script is for the bash shell only. There are also two scripts in shrew/src/dependencies/scripts, one for bash and one for csh, that set these variables as well. Below we show you how to do this by hand...
'''File:''' szip-2.1.tar.gz*


The '''hdf4_handler'''s configure script does not provide options to allow the location of szip to be specified. When the '''hdf4_handler'''s configure script is run it looks in the <code>--with-hdf4</code> location of szip, after that it looks in the usual places. In order to ensure that the correct szip (the one sent with the installer) was located we needed to install the szip library into the HDF4 library location.
==== $prefix ====
; Build process
Set '''$prefix''' to the full path of the desired installation location. This should be '''''/usr/local/opendap/servers/hyrax-<version>'''''. We chose this because we want to make sure several things are true:
: ./configure --prefix=$prefix/deps/hdf-4.2.5
* When the build is done, components of the server must be linked to the libraries they use and those same libraries must be in the same place when the server is run.
: make
* Since OS/X's various installers don't have an uninstall option, putting the entire server in one place makes it easy to remove
: make check
* We're Unix geeks and we like things to be in /usr/local ;-)
: make install


=== zlib ===
==== $PATH ====
'''Required by:''' HDF4, hdf4_handler <br/>
Set '''$PATH''' to '''''$prefix/bin:$PATH'''''
'''File:''' zlib-1.2.3.tar.gz


The '''hdf4_handler'''s configure script does not provide options to allow the location of zlib to be specified. When the '''hdf4_handler'''s configure script is run it looks in the <code>--with-hdf4</code> location of zlib, after that it looks in the usual places. In order to ensure that the correct zlib (the one sent with the installer) was located we needed to install the zlib library into the HDF4 library location.
==== $TOMCAT_DIR ====
; Build process
Set '''$TOMCAT_DIR''' to '''''$prefix/apache-tomcat-6.0.29'''''
: ./configure --prefix=$prefix/deps/hdf-4.2.5
: make
: make check
: make install


=== jpeg ===
:NB: Building the dependencies installs Tomcat for you.
'''Required by:''' HDF4, hdf4_handler <br/>
'''File:''' jpegsrc.v6b.tar.gz


The '''hdf4_handler'''s configure script does not provide options to allow the location of jpeg to be specified. When the '''hdf4_handler'''s configure script is run it looks in the <code>--with-hdf4</code> location of jpeg, after that it looks in the usual places. In order to ensure that the correct jpeg (the one sent with the installer) was located we needed to install the jpeg library into the HDF4 library location.
We use the evironment variable ''$TOMCAT_DIR'' instead of ''$CATALINA_HOME'' because the latter is actually used by tomcat. That said, make sure that if you have a value set for CATALINA_HOME it is the same as TOMCAT_DIR. If that's not the case, if CATALINA_HOME points to a different copy of tomcat, when you run the start up script (''shrew/apache-tomcat-6.0.29/bin/startup.sh'') it will start the copy of tomcat that CATALINA_HOME points to and not the one in ''shrew/apache-tomcat-6.0.29''.
; Build process
: ./configure --prefix=$prefix/deps/hdf-4.2.5
: make
: make check
: mkdir -p /usr/local/opendap/servers/hyrax-1.6.2/bin
: mkdir -p /usr/local/opendap/servers/hyrax-1.6.2/man/man1
: make install
: cp libjpeg.a $prefix/lib
: cp *.h $prefix/include


=== hdf5 ===
* In any case, make sure that the copy of tomcat you run actually has the '''opendap.war''' file in its ''webapps'' directory.
'''Required by:''' hdf5_handler <br/>
* If you're building a package/metapackage for a software distribution, make sure that the copy of the OLFS has a version number! To do that, build using ''ant -DHYRAX_VERSION=1.6.2 -DOLFS_VERSION=1.7.1'' for example. Note that the ''shrew'' Makefile does this for you if you have set the '''$(olfs_version_arg)''' Makefile variable in ''Makefile.am'' correctly. That is, following this prescription, ''make world'' will build the OLFS, using the version numbers set in the ''Makefile'' and put the ''opendap.war'' file in tomcat's ''webapps'' directory.
'''File:''' hdf5-1.8.5-patch1.tar.gz
; Build process
: ./configure --prefix=$prefix/deps/hdf5-1.8.5-patch1
: make
: make check
: make install


=== netcdf ===
=== Edit the Makefile.am ===
'''Required by:''' netcdf_handler <br/>
The Makefile.am used by ''shrew'' must be generic. It has a number of configuration values set for OS/X, but they are commented out since they break the Linux builds. These are particularly important for this package build because they are designed to help with configuring and building the handlers to use a completely self-contained set of dependencies. The '''make depends-pkg''' call below will build these and install them in '''$prefix/deps'''; these variables will ensure that the handlers use those and not some other set of libraries.
'''File:''' netcdf-3.6.3.tar.gz
; Build process
: ./configure --prefix=$prefix/deps/netcdf-3.6.3
: make
: make check
: make install


=== hdf4 ===
Here's what the relevant part of the ''Makefile.am'' looks like in svn:
'''Required by:''' hdf4_handler <br/>
'''File:''' hdf-4.2.5.tar.gz
; Build process
:./configure --prefix=$prefix/deps/hdf-4.2.5 --disable-fortran --with-szlib=$prefix/deps/hdf-4.2.5 --with-netcdf=$prefix/deps/netcdf-3.6.3 --enable-production --enable-shared -with-jpeg=$prefix/deps/hdf-4.2.5
: make
: make check
: make install


=== icu ===
olfs_version_arg = -DHYRAX_VERSION=1.6.2 -DOLFS_VERSION=Nightly.Build
'''Required by:''' ncml_handler <br/>
prefix_arg = --prefix=$(prefix)
'''File:''' icu4c-3_6-src.tgz
# Only use these if you need them and make sure to double check the values.                                                               
# If they are wrong, configure in the modules dir will fail and that means                                                               
# other parts of the build will fail too.<font  color="red">                                                                                                 
#netcdf_arg = --with-netcdf=$(prefix)/deps/netcdf-3.6.3
#hdf4_arg = --with-hdf4=$(prefix)/deps/hdf-4.2.5 --with-hdfeos2=$(prefix)/deps/hdf-4.2.5 --enable-short-name
#hdf5_arg = --with-hdf5=$(prefix)/deps/hdf5-1.8.5-patch1
#icu_arg = --with-icu-prefix=$(prefix)/deps/icu-3.6</font>
# Users may change add the external lib variables                                                                                         
# to specify locations of the external libraries as needed.                                                                                
#
release_params = $(prefix_arg) $(netcdf_arg) $(hdf4_arg) $(hdf5_arg) $(icu_arg)
# release_params = $(prefix_arg)                                                                                                         
#
debug_params = $(debug_flags) $(prefix_arg) $(netcdf_arg) $(hdf4_arg) $(hdf5_arg) $(icu_arg)
# debug_params = $(debug_flags) $(prefix_arg)                                                                                             


At run time ICU dependents failed to link correctly. Using <code>otool -L</code> we could see in our libraries all of the ICU dependancies were shown like:
Where the ''$(release_params)'' or ''$(debug_params)'' are used as the value to ''./configure'' for each of the handlers. So, uncommenting the lines shown in read will cause the handlers to look for the HDF4, HDF5, NetCDF, etc., libraries in ''$(prefix)/deps'' which was made by the ''depends-pkg'' target (in fact, the ''depends-install'' target also builds that directory).
:<code>libicui18n.dylib.36 (compatibility version 36.0.0, current version 36.0.0)</code>


When all the other libs had a full path like this:
=== Build some code ===
:<code>/usr/local/opendap/servers/hyrax-1.6.2/lib/libbes_dap.3.dylib (compatibility version 5.0.0, current version 5.5.0)</code>
To build the OS/X packages for the entire C/C++ part of the server,
run the following commands:
The ICU build was producing only dynamic libraries where the links were only to the package name (no path). This also caused programs linking to one of theICU libs to get "path free" links.  
;autoreconf --verbose --force --include: This rebuilds ''configure'' from ''configure.ac'' and ''Makefile.in'' from ''Makefile.am''
;./configure --prefix=$prefix : This builds ''Makefile'' from ''Makefile.in'' using the value of '''$prefix''' for the autoconf directory ''prefix''
;make depends-install : Build the dependencies, install them in '''$prefix''' and then make an OSX package to be used later
;make world : Make the server and install the bits and pieces in '''$prefix'''. This makes sure that the modules all link against the correct versions of the libraries (but it depeends on having '''$PATH''' set as described above.
;make pkg : Given that the code is all built and installed (and this includes the OLFS), make OSX packages for all the pieces. The packages will be stored in the different directories, but that doesn't matter so long as you followed the advice above about starting out in you OSX home directory and not /usr/local/src, or some other place that OSX file chooser dialogs won't find. What you wind up with is an OS/X Package for each of: libdap, bes, and the handlers in a top-level directory called ''osx-packages''. In a subsequent step these will be loaded into a Metapackage by hand, but there's more to do...
;make depends-pkg : This step builds a single package that holds all of the dependencies, something not technically needed but a practical necessity for OS/X where things like netcdf4 and hdf5 are not standard parts of the OS. This package is left in the ''<top level>/src/dependencies'' directory.


James found the following in the last reply to this thread: http://bugs.php.net/bug.php?id=34089 (Thanks James!)
=== Make the MetaPackage ===
<!-- [[Image:PackageMaker Configuration.png|thumb|150px|Components tab]][[Image:PackageMaker Contents.png|thumb|150px|Contents tab]] -->


:: ''The icu build system produces MAC platform libraries with an install name which does not include a path, this can be seen by running "otool -L" on any of the libraries or anything you build that links against any of the libraries.''
Installing all of the packages one-by-one is too complicated for most mac users, even those who are going to run a server. To get around this we combine all these packages in one giant blob called a ''metapackage''. To build the metapackage, you need to combine the discrete packages by hand, using ''PackageMaker''. The PackageMaker application is not hard to use, but you must attend to a fair amount of detail to get a MetaPackage that works correctly. Also, be wary of PackageMaker's tendency to crash when making large things like the metapackage; save often.
::
::''My solution was to edit icu's darwin config file found at "<icu-source-directory>/config/mh-darwin" ''
::
:: ''By changing line 28 from:''
::
::: ''<code>LD_SONAME = -Wl,-compatibility_version -Wl,$(SO_TARGET_VERSION_MAJOR) -Wl,-current_version -Wl,$(SO_TARGET_VERSION) -install_name $(notdir $(MIDDLE_SO_TARGET))</code>''
::
::''To:''
::
::: ''<code>LD_SONAME = -Wl,-compatibility_version -Wl,$(SO_TARGET_VERSION_MAJOR) -Wl,-current_version -Wl,$(SO_TARGET_VERSION) -install_name $(libdir)/$(notdir $(MIDDLE_SO_TARGET))</code>''


I made the recommended change and everything worked great.
==== Edit the OS/X Resources for the MetaPackage ====


; Build process
Prepare the ''.txt'' files in ''OSX_Resources'' at the top level of the shrew project:
: ./runConfigureICU MacOSX --prefix=$prefix/deps/icu-3.6
* Edit ''Introduction.txt''. This is probably the most important thing to edit since it's the first thing the installer shows a user.
: make
* Edit ''ReadMe.txt''.
: make check
* Don't edit ''License.txt''
: make install


==== Open PackageMaker and build the metapackage by hand ====


Start PackageMaker and set the ''Organization'' to '''org.opendap''' and the ''Minimum Target'' to '''10.5'''.
[[File:Basic PM Window.png|thumb|none|250px|Basic PM Window]]


== Building Hyrax C++ (libdap, bes, handlers and modules) ==
==== Configure the installer UI ====
'''URL:''' http://scm.opendap.org/svn/branch/shrew/hyrax_1.6.2_release<br/>
'''Revision:''' 23904


; Summary
In the upper right corner, click ''Edit Interface''. This is where you set the installer's background image, license file, introductory text, etc. These are the files in ''OSX_Resources'' that were edited in the steps before starting PackageMaker. Click through the panes of the installer's interface and set the Welcome text to ''Introduction.txt'', Read Me text to ''ReadMe.txt an License to ''License.txt''. [[File:Edit interface.jpg|thumb|none|250px|Edit interface]]
:I patched all of the Makefile.am targets for the OS-X packages (''<code>make pkg</code>''). I found numerous problems with small details such as incorrect package names for PackageMaker, Failure to preserve configure options in the package build. The code works now, but there are soma caveats for some of the packages. Details in each section below.  


=== libdap ===
==== Add and configure the dependencies ====
;Build process
: ./configure --prefix=$prefix
: make pkg
: make check
: make install


=== bes ===
Now add the ''dependencies'' package as the first 'child' package of this metapackage. Use drag and drop from a Finder window to copy the package into PackageMaker's window for the new metapackage. [[File:Dependencies first.png|thumb|none|250px|Dependencies First]]
;Build process
: ./configure --prefix=$prefix
: make pkg
: make check
: make install


=== dap-server ===
Make sure to set the options for the child package options so that the metapackage does not use any resources present in the child package
;Build process
[[File:Do not use res.png|thumb|none|250px|Do not use resources from child packages]]
: ./configure --prefix=$prefix
Also make sure to check that the paths used when this child package is installed are correct (e.g., may sure the child package won't install into your ''/Users/...'' directory!). [[File:Check paths.png|thumb|none|250px|Check Paths]]
: make pkg
Lastly, and very important, make sure the child package won't trigger a restart - this is very tiring and is not necessary for Hyrax. [[File:Restart none.png|thumb|none|250px|Restart none]]
: make check
: make install


=== fileout_netcdf ===
==== Now add the Hyrax packages ====
The <code>configure</code> command accepts:
<code>
:  --with-netcdf=ARG netcdf directory
:  --with-netcdf-include=ARG netcdf include directory
:  --with-netcdf-libdir=ARG    netcdf library directory
</code>
'''However for OS-X packages ONLY''' <code>--with-netcdf</code> '''will work!'''
;Build process
: ./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-3.6.3
: make pkg
: make check
: make install
 
=== freeform_handler ===
;Build process
: ./configure --prefix=$prefix
: make pkg
: make check
: make install
 
=== hdf4_handler ===
The <code>configure</code> command accepts:
<code>
:  --with-hdf4=ARG        hdf4 directory
:  --with-hdf4-include=ARG hdf 4 include directory
:  --with-hdf4-libdir=ARG  hdf 4 library directory
</code>
'''However for OS-X packages ONLY''' <code>--with-hdf4</code> '''will work!'''
;Build process
:./configure --prefix=$prefix --with-hdf4=$prefix/deps/hdf-4.2.5
: make pkg
: make check
: make install
 
=== hdf5_handler ===
The <code>configure</code> command accepts:
<code>
:  --with-hdf5=ARG        hdf5 directory
:  --with-hdf5-include=ARG hdf5 include directory
:  --with-hdf5-libdir=ARG  hdf5 library directory
</code>
'''However for OS-X packages ONLY''' <code>--with-hdf5</code> '''will work!'''
;Build process
:./configure --prefix=$prefix --with-hdf5=$prefix/deps/hdf5-1.8.5-patch1
: make pkg
: make check
: make install
 
===nectdf_handler ===
The <code>configure</code> command accepts:
<code>
:--with-netcdf=ARG      netcdf directory
:--with-netcdf-include=ARG
:::netcdf include directory
:--with-netcdf-libdir=ARG
:::netcdf library directory
</code>
'''However for OS-X packages ONLY''' <code>--with-netcdf</code> '''will work!'''
;Build process
: ./configure --prefix=$prefix --with-netcdf=$prefix/deps/netcdf-3.6.3
: make pkg
: make check
: make install
 
=== ncml_handler ===
The <code>configure</code> command accepts:
<code>
:--with-icu-prefix=PREFIX
:::Prefix where icu is installed (optional)
: --with-icu-exec-prefix=EPREFIX
:::Exec prefix where icu is installed (optional)
</code>
'''However for OS-X packages ONLY''' <code>--with-icu-prefix</code> '''will work!'''
;Build process
: ./configure --prefix=$prefix--with-icu-prefix=$prefix/deps/icu-3.6
: make pkg
: make check
: make install
 
== Building Hyrax Java (olfs)  ==
 
This required no additional code building.
 
=== Tomcat ===
 
I got the latest apache Tomcat (6.0.29) from http://tomcat.apache.org/download-60.cgi
 
;Build process
: cd $prefix
: tar -xvf $downloads/apache-tomcat-6.0.29.tar.gz
: find . -exec xattr -d com.apple.quarantine {} \; -print
:: Because I used Safari to grab the tomcat archive file and Finder to open it OS-X marked everything in the file system with the ''com.apple.quarantine'' [http://en.wikipedia.org/wiki/Extended_file_attributes| extended file attribute]. This causes OS-X  to produce the ''This is a web downloaded file, are you sure you want to do this'' user dialogs when it is first run. You can see this using an <code>ls -l</code> command. The presence of extended attributes is denoted by a trailing <code>@</code> sign at the end of the permissions string.
:: For example
::: <code>drwxr-xr-x@ 9 ndp  staff  476 Dec  3 14:59 apache-tomcat-6.0.29</code>
:: I removed this using <code>find</code> and the <code>xattr</code> command:
:::<code>find . -exec xattr -d com.apple.quarantine {} \; -print</code>
 
=== OLFS ===
 
I downloaded OLFS-1.7.1 from http://www.opendap.org/pub/olfs/olfs-1.7.1-webapp.tgz and unpacked it. From the bundle I took the opendap.war file and copied it to $prefix/tomcat/webapps
 
;Build process
: cd $prefix
: cp $downloads/olfs-1.7.1-webapp/opendap.war tomcat/webapps
 
== Shell Scripts ==
 
I also added a shell script that can be used to configure the users shell so that they can easily control Hyrax. There is both a bash and a c-shell version in the top distribution directory.
 
== PackageMaker ==
 
=== Hyrax Dependancies ===
 
In order to bundle the dependancies into an OSX package I used the pattern established by James in http://scm.opendap.org/svn/trunk/libdap/Makefile.am where he used the automake DESTDIR property to deploy the linked libraries to a directory of his choosing. I also created an OSX_Resources directory based on James' work and used it to generate the dependancies package.
 
;Build process
: cd $project_dir
: mkdir -p macosx/$prefix
: cd $prefix
: tar -cv $project_dir/deps.tar  deps
: cd $project_dir/macosx/$prefix
: tar -xvf project_dir/deps.tar
: cd $project
: /Developer/usr/bin/packagemaker --root macosx --id org.opendap.hyrax_dependancies-1.6.2 --title "Hyrax-1.6.2 dependancies" --version "1.6.2" --out hyrax-1.6.2-depends.pkg --resources OSX_Resources
 
=== Hyrax C++ (libdap, bes, modules and handlers) ===
 
All of the C++ components have a OS-X package target. I did have to make modifications to the production rules (Makefile.am) and some of the OSX_Resources for every one of these. They are all pretty clean now.
 
;Things to do:
: Most of the OSX_Resources/InstallationCheck scripts have hardwired PATH environments that point to this (hyrax-1.6.2) install. The production rules need to be tweaked to make this dependent on the build prefix.
 
=== Hyrax Java (The OLFS) ===
 
In order to bundle the dependancies into an OSX package I used the pattern established by James in http://scm.opendap.org/svn/trunk/libdap/Makefile.am where he used the automake DESTDIR property to deploy the linked libraries to a directory of his choosing. I also created an OSX_Resources directory based on James' work and used it to generate the dependancies package.
 
SP Basically I grabbed the tomcat tree (with the olfs opendap.war file in place) from $prefix, p[opped it into the macosx directory in my olfs package project and called packagemaker.
 
;Build process
: cd $project_dir
: mkdir -p macosx/$prefix
: cd $prefix
: tar -cv $project_dir/olfs.tar  apache-tomcat-6.0.29
: cd $project_dir/macosx/$prefix
: tar -xvf project_dir/olfs.tar
: cd $project
: /Developer/usr/bin/packagemaker --root macosx --id org.opendap.olfs-1.7.1 --title "OPeNDAP olfs-1.7.1" --version "1.7.1" --out olfs-1.7.1.pkg --resources OSX_Resources
 
=== Hyrax-1.6.2 ===
 
==== Meta-Package ====
Once I was able to build all all of the component packages I was able to use the PackageMaker GUI to build a meteapackage that combined all of the previously packaged components. I allowed the use to choose which handlers they wish to install, but they must install the dependancies, libdap, bes, dap-server, and olfs.
 
==== OneBigPackage ====
I was also able to use the PackageMaker GUI to build a single package based on the body of installed code in $prefix.
 
 
 
 
== Notes ==


Now add the Hyrax packages: libdap, bes, handlers and olfs. ''In that order''. Make sure to set all of the options for ''each'' package (not to use the child package's resources, not to require a restart and to check the install paths) using the same values as were set for the dependencies package.


<!-- [[File:Find component pkgs.png|thumb|none|250px|Find component packages]] -->


----
=== Notes ===
These things would complete the hyrax configuration as part of the installation process:
These things would complete the hyrax configuration as part of the installation process:
# If possible we should ask for user and group names in the install process.
# If possible we should ask for user and group names in the install process.
# We should create the logs directory as part of the install and configure the bes to use it.
# We should create the logs directory as part of the install and configure the bes to use it.
# Notes on the process for [[BuildHyraxOnOSX|building Hyrax on OS/X]] that has now been mostly codified in the ''shrew'' project's Makefile.am and related scripts and/or here (for the parts regarding building the OS/X MetaPackage).

Latest revision as of 17:33, 29 April 2013

Overview

The goal of this build was to produce a set of OS-X package installers for Hyrax. This includes bundling all of the Hyrax dependencies in an installer. In addition, all of the installed files (Hyrax and dependencies) should be confined to a single directory (tree) for the release. For any given release I chose /usr/local/opendap/servers/hyrax-<version>. This abbreviated is throughout this section as $prefix.

Note: This is based on Nathan's original detailed instructions for building on OS/X so that a MetaPackage installer can be made for Hyrax. I modified the shrew project for the Hyrax 1.6.2 release to do most of this in a semi-automated way. See Building Hyrax Dependencies for Nathan's original narrative of the process required.

You can check out the shrew project at https://scm.opendap.org/svn/trunk/shrew. We often do not use that but a copy of the shrew project on a branch. This enables everyone to know when changes are going to be slated for the next (trunk) or current (branch) release. The branch for a particular set of releases can be found by browsing the SVN sources using Trac. A typical example is: https://scm.opendap.org/trac/browser/branch/shrew/hyrax_1.7_release, where the libdap, bes, et c., projects are referenced by the shrew project.

Build a Hyrax MetaPackage for OS/X Using Shrew

Based on the work that Nathan did, I've modified the shrew project to accommodate most of this process. It's almost automatic...

Preconditions

A completely up to date checkout

Options

You can use these targets to build a set of dependencies that are then used to build the various handlers and/or you can build a set of OS/X packages that can be combined into a single, huge, metapackage. There may be some utility in just doing the former, but this process has been developed with the idea that it will be used to make an OS/X metapackage for Hyrax.

Set environment variables

You must set $prefix, $PATH, $TOMCAT_DIR and $CATALINA_HOME for this all to work. There are several scripts that will do this for you; you need to source these for them to work. In the top-level shrew directory, source spath.sh or source spath.sh <prefix> will set all of these (the former sets $prefix to the current working directory and the latter sets it to <prefix>). The sptah.sh script is for the bash shell only. There are also two scripts in shrew/src/dependencies/scripts, one for bash and one for csh, that set these variables as well. Below we show you how to do this by hand...

$prefix

Set $prefix to the full path of the desired installation location. This should be /usr/local/opendap/servers/hyrax-<version>. We chose this because we want to make sure several things are true:

  • When the build is done, components of the server must be linked to the libraries they use and those same libraries must be in the same place when the server is run.
  • Since OS/X's various installers don't have an uninstall option, putting the entire server in one place makes it easy to remove
  • We're Unix geeks and we like things to be in /usr/local ;-)

$PATH

Set $PATH to $prefix/bin:$PATH

$TOMCAT_DIR

Set $TOMCAT_DIR to $prefix/apache-tomcat-6.0.29

NB: Building the dependencies installs Tomcat for you.

We use the evironment variable $TOMCAT_DIR instead of $CATALINA_HOME because the latter is actually used by tomcat. That said, make sure that if you have a value set for CATALINA_HOME it is the same as TOMCAT_DIR. If that's not the case, if CATALINA_HOME points to a different copy of tomcat, when you run the start up script (shrew/apache-tomcat-6.0.29/bin/startup.sh) it will start the copy of tomcat that CATALINA_HOME points to and not the one in shrew/apache-tomcat-6.0.29.

  • In any case, make sure that the copy of tomcat you run actually has the opendap.war file in its webapps directory.
  • If you're building a package/metapackage for a software distribution, make sure that the copy of the OLFS has a version number! To do that, build using ant -DHYRAX_VERSION=1.6.2 -DOLFS_VERSION=1.7.1 for example. Note that the shrew Makefile does this for you if you have set the $(olfs_version_arg) Makefile variable in Makefile.am correctly. That is, following this prescription, make world will build the OLFS, using the version numbers set in the Makefile and put the opendap.war file in tomcat's webapps directory.

Edit the Makefile.am

The Makefile.am used by shrew must be generic. It has a number of configuration values set for OS/X, but they are commented out since they break the Linux builds. These are particularly important for this package build because they are designed to help with configuring and building the handlers to use a completely self-contained set of dependencies. The make depends-pkg call below will build these and install them in $prefix/deps; these variables will ensure that the handlers use those and not some other set of libraries.

Here's what the relevant part of the Makefile.am looks like in svn:

olfs_version_arg = -DHYRAX_VERSION=1.6.2 -DOLFS_VERSION=Nightly.Build
prefix_arg = --prefix=$(prefix)
# Only use these if you need them and make sure to double check the values.                                                                
# If they are wrong, configure in the modules dir will fail and that means                                                                 
# other parts of the build will fail too.                                                                                                  
#netcdf_arg = --with-netcdf=$(prefix)/deps/netcdf-3.6.3
#hdf4_arg = --with-hdf4=$(prefix)/deps/hdf-4.2.5 --with-hdfeos2=$(prefix)/deps/hdf-4.2.5 --enable-short-name
#hdf5_arg = --with-hdf5=$(prefix)/deps/hdf5-1.8.5-patch1
#icu_arg = --with-icu-prefix=$(prefix)/deps/icu-3.6
# Users may change add the external lib variables                                                                                          
# to specify locations of the external libraries as needed.                                                                                 
#
release_params = $(prefix_arg) $(netcdf_arg) $(hdf4_arg) $(hdf5_arg) $(icu_arg) 
# release_params = $(prefix_arg)                                                                                                           
#
debug_params = $(debug_flags) $(prefix_arg) $(netcdf_arg) $(hdf4_arg) $(hdf5_arg) $(icu_arg)
# debug_params = $(debug_flags) $(prefix_arg)                                                                                              

Where the $(release_params) or $(debug_params) are used as the value to ./configure for each of the handlers. So, uncommenting the lines shown in read will cause the handlers to look for the HDF4, HDF5, NetCDF, etc., libraries in $(prefix)/deps which was made by the depends-pkg target (in fact, the depends-install target also builds that directory).

Build some code

To build the OS/X packages for the entire C/C++ part of the server, run the following commands:

autoreconf --verbose --force --include
This rebuilds configure from configure.ac and Makefile.in from Makefile.am
./configure --prefix=$prefix
This builds Makefile from Makefile.in using the value of $prefix for the autoconf directory prefix
make depends-install
Build the dependencies, install them in $prefix and then make an OSX package to be used later
make world
Make the server and install the bits and pieces in $prefix. This makes sure that the modules all link against the correct versions of the libraries (but it depeends on having $PATH set as described above.
make pkg
Given that the code is all built and installed (and this includes the OLFS), make OSX packages for all the pieces. The packages will be stored in the different directories, but that doesn't matter so long as you followed the advice above about starting out in you OSX home directory and not /usr/local/src, or some other place that OSX file chooser dialogs won't find. What you wind up with is an OS/X Package for each of: libdap, bes, and the handlers in a top-level directory called osx-packages. In a subsequent step these will be loaded into a Metapackage by hand, but there's more to do...
make depends-pkg
This step builds a single package that holds all of the dependencies, something not technically needed but a practical necessity for OS/X where things like netcdf4 and hdf5 are not standard parts of the OS. This package is left in the <top level>/src/dependencies directory.

Make the MetaPackage

Installing all of the packages one-by-one is too complicated for most mac users, even those who are going to run a server. To get around this we combine all these packages in one giant blob called a metapackage. To build the metapackage, you need to combine the discrete packages by hand, using PackageMaker. The PackageMaker application is not hard to use, but you must attend to a fair amount of detail to get a MetaPackage that works correctly. Also, be wary of PackageMaker's tendency to crash when making large things like the metapackage; save often.

Edit the OS/X Resources for the MetaPackage

Prepare the .txt files in OSX_Resources at the top level of the shrew project:

  • Edit Introduction.txt. This is probably the most important thing to edit since it's the first thing the installer shows a user.
  • Edit ReadMe.txt.
  • Don't edit License.txt

Open PackageMaker and build the metapackage by hand

Start PackageMaker and set the Organization to org.opendap and the Minimum Target to 10.5.

Basic PM Window

Configure the installer UI

In the upper right corner, click Edit Interface. This is where you set the installer's background image, license file, introductory text, etc. These are the files in OSX_Resources that were edited in the steps before starting PackageMaker. Click through the panes of the installer's interface and set the Welcome text to Introduction.txt, Read Me text to ReadMe.txt an License to License.txt.

Edit interface

Add and configure the dependencies

Now add the dependencies package as the first 'child' package of this metapackage. Use drag and drop from a Finder window to copy the package into PackageMaker's window for the new metapackage.

Dependencies First

Make sure to set the options for the child package options so that the metapackage does not use any resources present in the child package

Do not use resources from child packages

Also make sure to check that the paths used when this child package is installed are correct (e.g., may sure the child package won't install into your /Users/... directory!).

Check Paths

Lastly, and very important, make sure the child package won't trigger a restart - this is very tiring and is not necessary for Hyrax.

Restart none

Now add the Hyrax packages

Now add the Hyrax packages: libdap, bes, handlers and olfs. In that order. Make sure to set all of the options for each package (not to use the child package's resources, not to require a restart and to check the install paths) using the same values as were set for the dependencies package.



Notes

These things would complete the hyrax configuration as part of the installation process:

  1. If possible we should ask for user and group names in the install process.
  2. We should create the logs directory as part of the install and configure the bes to use it.
  3. Notes on the process for building Hyrax on OS/X that has now been mostly codified in the shrew project's Makefile.am and related scripts and/or here (for the parts regarding building the OS/X MetaPackage).