Developer Info: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
 
(102 intermediate revisions by 2 users not shown)
Line 1: Line 1:
* [http://test.opendap.org/cgi-bin/build_reader.pl?show=current&sort=yes Nightly Builds]: Software we build every night on several platforms. The link provides the most recent report for each build we do. Use ''show=current'' to get the most recent; ''show=<date>'' to get a particular date, ''show=<platform>'' to see a particular platform. The ''sort=yes'' option runs the results through Unix sort before building the table. The nightly build software is part of the ''nbuilds'' project and is available from our trac/svn repository.


* [https://github.com/OPENDAP OPeNDAP's GitHub repositories]: OPeNDAP's software is available using GitHub in addition to the downloads from our website.
** Before 2015 we hosted our own SVN repository. It's still online and available, but for read-only access, at [https://scm.opendap.org/svn https://scm.opendap.org/svn].
* [https://travis-ci.org/OPENDAP Continuous Integration builds]: Software that is built whenever new changes are pushed to the master branch. These builds are done on the Travis-CI system.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* [http://test.opendap.org/ test.opendap.org]: Test servers with data files.
* We use the Coverity static system to look for common software defects, information on Hyrax is spread across three projects:
** [https://scan.coverity.com/projects/opendap-bes?tab=overview The BES and the standard handlers we distribute]
** [https://scan.coverity.com/projects/opendap-olfs?tab=overview The OLFS - the front end to the Hyrax data server]
** [https://scan.coverity.com/projects/opendap-libdap4?tab=overview libdap - The implementation of DAP2 and DAP4]


== Developer Guidelines and Information ==
== OPeNDAP's FAQ ==
The [http://www.opendap.org/faq OPeNDAP FAQ] has a pretty good section on developer's questions, especially regarding SVN.
The [http://www.opendap.org/faq-page OPeNDAP FAQ] has a pretty good section on developer's questions.


== General development information ==
== C++ Coding Information ==
These pages contain general information relevant to anyone working with our software:
* [[Include files for libdap | Guidelines for including headers]]
* [[Using lambdas with the STL]]
* [[Better Unit tests for C++]]
* [[Better Singleton classes C++]]
* [[What is faster? stringstream string + String]]
* [[More about strings - passing strings to functions]]
 
== OPeNDAP Workshops ==
* [http://www.opendap.org/about/workshops-and-presentations/2007-10-12 The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/about/workshops-and-presentations/2008-07-15 ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.
 
== libdap4 and BES Reference documentation ==
* [https://opendap.github.io/bes/html/ BES Reference]
* [https://opendap.github.io/libdap4/html/ libdap Reference]
 
== BES Development Information ==
* [[Hyrax - Logging Configuration|Logging Configuration]]


* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
* [[BES_-_How_to_Debug_the_BES| How to debug the BES]]
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]]
* [[BES - Debugging Using besstandalone]]
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
* [[Hyrax - Create BES Module | How to create your own BES Module]]
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
* Hyrax Module Integration: How to configure your module so it's easy to add to Hyrax instances ([[:File:HyraxModuleIntegration-1.2.pdf|pdf]])
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
* [[Hyrax - Starting and stopping the BES| Starting and stopping the BES]]
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]
* [[Hyrax - Running bescmdln | Running the BES command line client]]
* [[Hyrax - BES Client commands| BES Client commands]]. The page [[BES_XML_Commands | BES XML Commands]] repeats this info for a bit more information on the return values. Most of the commands don't return anything unless they return an error and are expected to be used in a group where a ''get'' command closes out the request and obviously does return a response of some kind (maybe an error).
* [[Hyrax:_BES_Administrative_Commands| BES Administrative Commands]]
* [[Hyrax - Extending BES Module | Extending your BES Module]]
* [[Hyrax - Example BES Modules | Example BES Modules]] - the Hello World example and the CSV data handler
* [[Hyrax - BES PPT | BES communication protocol using PPT (point to point transport)]]


Workshops:
* [[Australian BOM Software Developer's Agenda and Presentations|Software Developers Workshop]]
* [http://www.opendap.org/apac_workshop_vm The APAC/BOM Workshops]: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
* [http://www.opendap.org/ESIP_Federation_Hyrax_workshop ESIP Federation Server Workshop]: This half-day workshop focused on server installation and configuration. Summer 2008
* [[A One-day Course on Hyrax Development | Server Functions]]: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012.


== Development process information (OPeNDAP Specific) ==
== OPeNDAP Development process information ==
These pages contain information about how we'd like people working with us to use our various on-line tools.
These pages contain information about how we'd like people working with us to use our various on-line tools.


===== Using revision control: =====
* [[Planning a Program Increment]] This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
* [[Hyrax GitHub Source Build]] This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
* [[Bug Prioritization]]. How we prioritize bugs in our software.
* [[ReleaseGuide | Making a Source Release]]. Once software is ready for distribution, use this checklist to make sure you do all of the steps needed to make a source release.
* [[MergingBranches |How to merge code]]


===== Making specific kinds of packages for release: =====
===[[How to Make a Release|Making A Release]] ===
* [[RPM |Making RPM Distributions]]. Follow these steps to create an RPM distribution of the software.
* [[How to Make a Release]] A general template for making a release. This references some of the pages below.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
* [[OLFSReleaseGuide| Making a Release of OLFS]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]].  Follow these steps when a new Matlab GUI version is ready to be released.  


===== Software process issues: =====
== Software process issues: ==
* [[UnitTests| How to write unit tests using CppUnit]]
* [[How to download test logs from a Travis build]] All of our builds on Travis that run tests save those logs to an S3 bucket.
* [[ConfigureCentos| How to configure a CentOS machine for production of RPM binaries]] - Updated 12/2014 to include information regarding git.
* [[How to use CLion with our software]]
* [[BES Timing| How to add timing instrumentation to your BES code.]]
* [[UnitTests| How to write unit tests using CppUnit]] NB: See other information under the heading of C++ development
* [[valgrind| How to use valgrind with unit tests]]
* [[valgrind| How to use valgrind with unit tests]]
* [[Debugging the distcheck target]] Yes, this gets its own page...
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[CopyRights| How to copyright software written for OPeNDAP]]
* [[GuideLines| Development Guidelines]]
* [[Managing public and private keys using gpg]]
* [[Managing public and private keys using gpg]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
* [[SecureEmail |How to Setup Secure Email and Sign Software Distributions]]
Line 47: Line 70:
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [[NetworkServerSecurity |Security Policy and Related Procedures]]
* [http://semver.org/ Software version numbers]
* [http://semver.org/ Software version numbers]
* [[GuideLines| Development Guidelines]]
* [[Apple M1 Special Needs]]


== About Subversion ==
==== Older info of limited value: ====
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
* [http://gcc.gnu.org/gcc-4.4/cxx0x_status.html C++-11 gcc/++-4.4 support] We now require compilers that support C++-14, so this is outdated (4/19/23).
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.
* [[How to use Eclipse with Hyrax Source Code]] I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.


== Trac install ==
==== AWS Tips ====
Todo:
* [[Growing a CentOS Root Partition on an AWS EC2 Instance]]
* Figure out if the account manager plugin will work with Agilo
* [[How Shutoff the CentOS firewall]]
* turn on ''https''
* copy more set up info from the notebook
* <del>Install Agilo plugin</del> DONE
* <del>figure out what else to instal and do it (goto http://scm.opendap.org/trac/about to see what's installed)</del> DONE but there are issues to be resolved
* copy over the old data base
* edit the value of base_url in [trac] section of trac.ini


Updated with information about the update to Trac 1.0.1 and the Agilo plugin.
== General development information ==
These pages contain general information relevant to anyone working with our software:


=== Authentication ===
* '''[[Git Hacks and Tricks]]''': Information about using git and/or GitHub that seems useful and maybe not all that obvious.
* [[Git Secrets]]: securing repositories from AWS secret key leaks.
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind Suppression File Howto] How to build a suppressions file for valgrind.
* [[Using a debugger for C++ with Eclipse on OS/X]] Short version: use lldbmi2 **Add info**
* [[Using ASAN]] Short version, look [https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger at the Google/GitHub pages] for useful environment variables **add text** On Centos, use yum install llvm to get the 'symbolizer' and try ''ASAN_OPTIONS=symbolize=1 ASAN_SYMBOLIZER_PATH=$(shell which llvm-symbolizer)''
* [[How to use ''Instruments'' on OS/X to profile]] Updated 7/2018
* [https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto Valgrind - How to generate suppression files for valgrind] This will quiet valgrind, keeping it from telling you OS/X or Linux (or the BES) is leaking memory.
* [[Migrating source code from SVN to git]]: How to move a large project from SVN to git and keep the history, commits, branches and tags.
* [https://developer.mozilla.org/en-US/docs/Eclipse_CDT Eclipse - Detailed information about running Eclipse on OSX from the Mozzilla project]. Updated in 2017, this is really good but be aware that it's specific to Mozilla so some of the tips don't apply. Hyrax (i.e., libdap4 and BES) also use their own build system (autotools + make) so most of the configuration information here is very apropos. See also [[How to use Eclipse with Hyrax Source Code]] below.
* [https://jfearn.fedorapeople.org/en-US/RPM/4/html/RPM_Guide/index.html RPM Guide] The best one I'm found so far...
* [https://autotools.io/index.html Autotools Myth busters] The best info on autotools I've found yet (covers ''autoconf'', ''automake'', ''libtool'' and ''pkg-config'').
* The [https://www.gnu.org/software/autoconf/autoconf.html autoconf] manual
* The [https://www.gnu.org/software/automake/ automake] manual
* The [https://www.gnu.org/software/libtool/ libtool] manual
* A good [https://lldb.llvm.org/lldb-gdb.html gdb to lldb cheat sheet] for those of us who know ''gdb'' but not ''lldb''.


Added code to the trac.conf httpd conf file in /etc/httpd/conf.d/trac.conf:
= Old information =
'''Note''': Old build information
====The Release Process====
# Make sure the <tt>hyrax-dependencies</tt> project is up to date and tar balls on www.o.o. If there have been changes/updates:
## Update version number for the <tt>hyrax-dependencies</tt> in the <tt>Makefile</tt>
## Save, commit, (merge?), and push the changes to the <tt>master</tt> branch.
## Once the <tt>hyrax-dependencies</tt> CI build is finished, trigger CI builds for both <tt>libdap4</tt> and <tt>bes</tt> by pushing change(s) to the master branch of each.
# [[Source_Release_for_libdap | Making a source release of libdap]]
# [[ReleaseGuide | Making a source release of the BES]].
# [[OLFSReleaseGuide| Make the OLFS release WAR file]]. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
# [[HyraxDockerReleaseGuide|Make the official Hyrax Docker image for the release]] When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.


<source lang="xml">
====Supplemental release guides====
<Location "/trac/login">
<font color="red">Old - use the packages built using the Continuous Delivery process</font>
  AuthType Basic
# [[RPM |Make the RPM Distributions]]. Follow these steps to create an RPM distribution of the software. '''Note:''' ''Now we use packages built using CI/CD, so this checklist is no longer needed.''
  AuthName "Trac"
  AuthUserFile /usr/local/scm/svn-httpd-passwd
  Require valid-user
</Location>
</source>


And the login button worked. Since the passwd file was copied over the old user names and passwords are all there.
'''Note''': ''The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...''
 
* [[MergingBranches |How to merge code]]
=== Logging ===
* [[TrunkDevelBranchRel | Using the SVN trunk, branches and tags to manage releases]].
 
* [[ShrewBranchGuide | Making a Branch of Shrew for a Server Release]]. Releases should be made from the trunk and moved to a branch once they are 'ready' so that development can continue on the trunk and so that we can easily go back to the software that mad up a release, fix bugs, and (re)release those fixes. In general, it's better to fix things like build issues, etc., discovered in the released software ''on the trunk'' and merge those down to the release branch to maintain consistency, re-release, etc. This also means that virtually all new feature development should take place on special ''feature'' branches, not the trunk.
In the [logging] section of the trac.ini file. Need to set the log type to ''file'' to get log info in a file. The default is ''none'' and that turns off logging.
* [[Hyrax Package for OS-X]]. This describes how to make a new OS/X 'metapackage' for Hyrax.
 
* [[XP| Making Windows XP distributions]]. Follow these directions to make Windows XP binaries.
=== Serving static pages ===
* [[ReleaseToolbox |Making a Matlab Ocean Toolbox Release]]. Follow these steps when a new Matlab GUI version is ready to be released.
 
* [[Eclipse - How to Setup Eclipse in a Shrew Checkout]] This includes some build instructions
First, use trac-admin to deploy the htdocs. I copied the result to /var/www/trac. dropping the ''htdocs'' dir and just copying the ''common'' and ''site'' dirs into /var/www/trac. Then set up the two directories so that the aliases /chrome/{common,site} can be used to access the stuff they contain. Note that ''site'' is supposed to hold the site specific stuff like a logo.
* [[LinuxBuildHostConfig| How to configure a Linux machine to build Hyrax from SVN]]
 
* [[ConfigureSUSE| How to configure a SUSE machine for production of RPM binaries]]
I also copied the trac.wsgi file in deploy/cgi-bin to /var/www/cgi-bin.
* [[ConfigureAmazonLinuxAMI| How to configure an Amazon Linux AMI for EC2 Instance To Build Hyrax]]
 
* [[TestOpendapOrg | Notes from setting up Hyrax on our new web host]]
In trac.ini, set the logo (in the section  [header_logo]0 using ''src = /chrome/site/opendap_scm_2.png'' and the htdocs location for the common pages (in the section [trac]) using ''htdocs_location = /chrome/common''
* [http://svnbook.red-bean.com/en/1.7/index.html Subversion 1.7 documentation] -- The official Subversion documentation; [http://svnbook.red-bean.com/en/1.1/svn-book.pdf PDF] and [http://svnbook.red-bean.com/en/1.1/index.html HTML].
 
* [[OPeNDAP's Use of Trac]] -- How to use Trac's various features in the software development process.
<source lang="xml">
# This configures the web server to serve Trac pages so the static             
# parts don't get processed by python. The paths /chrome/common and            
# /chrome/site are used in the trac.ini file.                                 
Alias /chrome/common /var/www/trac/common
 
<Directory /var/www/trac/common>
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>
 
<Location /chrome/common>
  SetHandler None
</Location>
 
Alias /chrome/site /var/www/trac/site
 
<Directory /var/www/trac/site>
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>
 
<Location /chrome/site>
  SetHandler None
</Location>
 
</source>
=== Agilo Plugin ===
 
Goto http://www.agilofortrac.com/, get an account and download the Agilo egg file, checking the hosts python version (2.6.6 right now) and making sure to get the egg file with the matching version (2.6)
 
The egg file was ziped, so I uncompressed it (unzip binary_agilo-PRO-py2.6.egg.zip) and restarted the web server.
 
I then ran ''trac-admin /usr/local/scm/trac/opendap upgrade'' and ''trac-admin /usr/local/scm/trac/opendap wiki upgrade'' as instructed.
 
borked
 
easy_install binary_agilo-1.3.12_PRO-py2.6.egg
 
trac-admin /usr/local/scm/trac/opendap permission add jimg TRAC_ADMIN
 
The easy_install step edited the trac.ini file, else add the following to [components] ''agilo.* = enabled''
 
Re-run the upgrade command for/with trac-admin
 
Works; should look at opitons and tweak the setup/config
 
=== XML/RPM Plugin ===
 
This is used by Eclipse and the MyLyn task manager
 
Source: http://trac-hacks.org/wiki/XmlRpcPlugin
 
''easy_install -Z -U http://trac-hacks.org/svn/xmlrpcplugin/trunk''
 
then in trac.ini:
 
<source lang="bash">
[components]
tracrpc.* = enabled
</source>
 
Not sure why this doesn't show up in the admin's plugin panel.
 
=== <del>Table of Contents Plugin</del> ===
 
This adds a TOC to the wiki pages. I'm adding it because we used it before and I want to smooth migration of old pages.
 
Source: http://trac-hacks.org/wiki/TocMacro
 
Looks like it does not support Trac 1.0++
 
=== <del>Master tickets Plugin</del> ===
 
Enables ticket dependency.
 
Source: http://trac-hacks.org/wiki/MasterTicketsPlugin
 
''easy_install http://trac-hacks.org/svn/masterticketsplugin/trunk/''
 
Then in trac.ini:
<source lang="bash">
[components]
mastertickets.* = enabled
 
[ticket-custom]
blocking = text
blocking.label = Blocking
blockedby = text
blockedby.label = Blocked By
</source>
 
Upgrade the Trac environment:
 
''trac-admin /usr/local/scm/trac/opendap/ upgrade''
 
And
 
''trac-admin /usr/local/scm/trac/opendap wiki upgrade''
 
Agilo seems to override this...
 
=== account manager plugin ===
 
Makes it easier for people to manage their accounts
 
Source: http://trac-hacks.org/wiki/AccountManagerPlugin
 
''easy_install https://trac-hacks.org/svn/accountmanagerplugin/tags/acct_mgr-0.4.3/''
 
No idea if this is working - Agilo seems to bork other plugins.
 
=== Configuration notes ===
 
In the trac.ini I can turn off or on the 'native' ticket system of Trac using these settings:
<source lang="bash">
trac.ticket.api.ticketsystem = enabled
trac.ticket.roadmap.roadmapmodule = enabled
trac.ticket.web_ui.ticketmodule = enabled
</source>
 
Agilo seems to want to have only four types of tickets; not sure if turning on the 'regular' ticket system will break it or what. Enabling these does add a 'New Ticket' menu item to the menu bar.
 
This seems to break things...

Latest revision as of 23:09, 22 February 2024

OPeNDAP's FAQ

The OPeNDAP FAQ has a pretty good section on developer's questions.

C++ Coding Information

OPeNDAP Workshops

  • The APAC/BOM Workshops: This workshop spanned several days and covered a number of topics, including information for SAs and Developers. Oct 2007.
  • ESIP Federation Server Workshop: This half-day workshop focused on server installation and configuration. Summer 2008
  • Server Functions: This one-day workshop is all about writing and debugging server-side functions. It also contains a wealth of information about Hyrax, the BES and debugging tricks for the server. Spring 2012. Updated Fall 2014 for presentation to Ocean Networks Canada.

libdap4 and BES Reference documentation

BES Development Information

OPeNDAP Development process information

These pages contain information about how we'd like people working with us to use our various on-line tools.

  • Planning a Program Increment This is a checklist for the planning phase that precedes a Program Increment (PI) when using SAFe with the NASA ESDIS development group.
  • Hyrax GitHub Source Build This explains how to clone our software from GitHub and build our code using a shell like bash. It also explains how to build the BES and all of the Hyrax 'standard' handlers in one operation, as well as how to build just the parts you need without cloning the whole set of repos. Some experience with 'git submodule' will make this easier, although the page explains everything.
  • Bug Prioritization. How we prioritize bugs in our software.

Making A Release

Software process issues:

Older info of limited value:

  • C++-11 gcc/++-4.4 support We now require compilers that support C++-14, so this is outdated (4/19/23).
  • How to use Eclipse with Hyrax Source Code I like Eclipse, but we now use CLion because it's better (4/19/23) . Assuming you have cloned our Hyrax code from GitHub, this explains how to setup eclipse so you can work fairly easily and switch back and forth between the shell, emacs and eclipse.

AWS Tips

General development information

These pages contain general information relevant to anyone working with our software:

Old information

Note: Old build information

The Release Process

  1. Make sure the hyrax-dependencies project is up to date and tar balls on www.o.o. If there have been changes/updates:
    1. Update version number for the hyrax-dependencies in the Makefile
    2. Save, commit, (merge?), and push the changes to the master branch.
    3. Once the hyrax-dependencies CI build is finished, trigger CI builds for both libdap4 and bes by pushing change(s) to the master branch of each.
  2. Making a source release of libdap
  3. Making a source release of the BES.
  4. Make the OLFS release WAR file. Follow these steps to create the three .jar files needed for the OLFS release. Includes information on how to build the OLFS and how to run the tests.
  5. Make the official Hyrax Docker image for the release When the RPMs and the WAR file(s) are built and pushed to their respective download locations, make the Docker image of the release.

Supplemental release guides

Old - use the packages built using the Continuous Delivery process

  1. Make the RPM Distributions. Follow these steps to create an RPM distribution of the software. Note: Now we use packages built using CI/CD, so this checklist is no longer needed.

Note: The following is all about using Subversion and is out of date as of November 2014 when we switched to git. There are still good ideas here...

Retrieved from "https://docs.opendap.org/index.php?title=Developer_Info&oldid=13526"