Difference between revisions of "Source Release for BES"

From OPeNDAP Documentation
m (Tag The Release)
(Tag the BES code)
 
(117 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Note that the old pages with the tables of version numbers are gone. Use the web pages themselves as documentation of what has or has not been built. For the current release, include grayed-out links to binaries we do plan to make but keep the grayed-out links in place for the current version only!
 
  
== When we decide to release software ==
+
This pages covers the steps required to release the BES software for Hyrax.  
Use the developers at opendap.org list to send out notices to all the developers about pending releases.
 
  
=== Planned release ===
+
We now depend on the CI/CD process to build binary packages and to test the source builds.
If this is a planned release, send out a notice at least two weeks in advance to anyone who likely has made changes to the software. More lead time is, of course, better. This will allow developers time to get their code and documentation changes into github and onto the master branch in time for the release.  
 
  
Run Coverity!
+
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch if you don't '''need''' it, since it complicates making tags.
  
=== Emergency releases ===
+
== Verify the code base ==
If this is an 'emergency' release, then send out a notice to developers as soon as the decision to release is made, since this will give a chance for other developers to lets us know if there are new changes in the master branch that are ready for release. if we don't get an email from those developers of a particular component, then we should assume that the code might not be 100% ready for release and we should use the version tagged with/for the last release if possible.  
+
# We release using the ''master'' branch. The code on ''master'' must have passed the CI builds. '''This includes the hyrax-docker builds since that CI build runs the full server regression tests!'''
 +
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')
  
Maybe run Coverity - depends on the scope of the changes.
+
== Update the Version Numbers ==
  
== The Release Process ==
+
=== Version for Humans ===
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. If you don't need to do this, skip it. Also, note that the handlers that are ''git submodules'' really make this a hassle, so feel free to skip this step or do it only for the BES itself, etc.
+
# Determine the human version number. This appears to be a somewhat subjective process.
:'''Create the release branch.'''
+
# Edit each of the ''Affected Files'' and update the human version number.
:* ''git checkout -b hyrax_release_0.0.0''
 
:* ''git push --set-upstream origin hyrax_release_0.0.0''
 
: For the bes, also make the branch for each submodule:
 
:* ''git submodule foreach 'git checkout -b hyrax_release_0.0.0' ''
 
===  Verify the code base ===
 
# Make sure that the source code you're using is up-to-date. (''git pull'')
 
# Have all tickets for the release milestone been resolved?
 
# Does the code pass it's tests? The Travis builds should be running the distcheck targets for these packages.
 
#;Note
 
#: For the BES (''or any software that requires parameters passed to configure'') use <tt>DISTCHECK_CONFIGURE_FLAGS=...</tt> when you run <tt>make</tt> For running tests, using <tt>TESTSUITEFLAGS=-j9</tt> will speed up the autconf/autotest tests if the version of autoconf supports parallel testing (but not on CentOS 6 because it's not supported by the stock version of autoconf on that OS).
 
#* ''libdap4''
 
#:: '''make check -j9'''
 
#:: '''make distcheck -j9'''
 
#* ''bes''
 
#:: '''make check -j9'''
 
#:: '''make distcheck -j9 DISTCHECK_CONFIGURE_FLAGS=--with-dependencies=$prefix/deps''' (DISTCHECK_CONFIGURE... no longer needed with automake 1.14 or greater. jhrg 9/21/17)
 
#: How did that go, with the <tt>distcheck</tt> target there? [[Debugging the distcheck target|Look here for help]]
 
# Once the tests pass merge the '''master''' branch to the '''coverity-scan''' branch. Read the resulting report and deal with the identified issues. Once completed return to the top of this section and repeat until everything is clean.
 
  
=== Check the<tt> make rpm </tt>target ===
+
:; Affected Files
Running the <tt>make rpm</tt> target is useful because it will find problems that can slow the process once the final edits are done and the RPMs are being built.
+
:* '''''configure.ac''''' - Look for:
# Make sure to install the ''libdap rpms'' before you build the bes rpms and uninstall them afterwards.:
+
::: <tt>AC_INIT(bes, ###.###.###, opendap-tech@opendap.org)</tt>
#: '''sudo yum install ~/rpmbuild/RPMS/x86_64/libdap-*'''  
+
:* ''''' debian/changelog''''' (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog Debian ChangeLog])
# Make sure that the bes dependencies have been built using the static option so that the BES rpm will have modules that are statically linked to their dependencies. To check this look at the libraries in <tt>$prefix/deps/lib</tt>. There should be no <tt>*.so</tt> files in the <tt>lib</tt> dir, only <tt>*.a</tt>, <tt>*.la</tt>, and some other directories. To build a new set of dependencies that provide only static libraries, use:
+
::: '''Take Note!''' ''The <tt>debian/changelog</tt> is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.''
#: '''make for-static-rpm -j9 CONFIGURE_FLAGS=--disable-shared'''  
+
:* '''''ChangeLog'''''
# Build the BES rpm using those snazzy static libraries, use
+
:* '''''NEWS'''''
#: '''make all-static-rpm -j9'''
+
:* '''''README.md'''''
# When finished, uninstall the libdap rpms:
+
:* '''''INSTALL'''''
#: '''sudo yum uninstall ~/rpmbuild/RPMS/x86_64/libdap-*'''
 
  
----
+
=== Update the internal library (API/ABI) version numbers. ===
 +
The BES is '''''not''''' a shared library, it is a set of c++ applications that are typically built as statically linked binaries. Because of this the usual CURRENT:REVISION:AGE tuples used to express the binary compatibility state of a c++ shared object library have little meaning for the BES code. So, what we choose to do is simply bump the REVISION numbers by one value for each release.
  
=== Checkpoint ===
+
* In the '''configure.ac''' file locate each of:
----
+
** LIB_DIS_REVISION
Up until now you have only verified the software's readiness for release. Beginning with the follow section you will be making the edits and changes required to update the version numbers and build the actual release files. Do not proceed unless you have passed all of the previous process checks. Thanks.
+
** LIB_PPT_REVISION
 +
** LIB_XML_CMD_REVISION
 +
* Increase the value of each by one (1).
 +
* Save the file.
 +
* Update the text documentation files and version numbers in the configuration files:
  
=== Update Release Related Files ===
+
Example of the relevant section from configure.ac:
Update the text documentation files and version numbers in the configuration files:
+
<source lang-="bash">
 +
LIB_DIS_CURRENT=18
 +
LIB_DIS_AGE=3
 +
LIB_DIS_REVISION=3
 +
AC_SUBST(LIB_DIS_CURRENT)
 +
AC_SUBST(LIB_DIS_AGE)
 +
AC_SUBST(LIB_DIS_REVISION)
 +
LIBDISPATCH_VERSION="$LIB_DIS_CURRENT:$LIB_DIS_REVISION:$LIB_DIS_AGE"
 +
AC_SUBST(LIBDISPATCH_VERSION)
  
==== Update the '''ChangeLog''' file. ====
+
LIB_PPT_CURRENT=5
 +
LIB_PPT_AGE=1
 +
LIB_PPT_REVISION=2
 +
AC_SUBST(LIB_PPT_CURRENT)
 +
AC_SUBST(LIB_PPT_AGE)
 +
AC_SUBST(LIB_PPT_REVISION)
 +
LIBPPT_VERSION="$LIB_PPT_CURRENT:$LIB_PPT_REVISION:$LIB_PPT_AGE"
 +
AC_SUBST(LIBPPT_VERSION)
 +
 
 +
LIB_XML_CMD_CURRENT=5
 +
LIB_XML_CMD_AGE=4
 +
LIB_XML_CMD_REVISION=2
 +
AC_SUBST(LIB_XML_CMD_CURRENT)
 +
AC_SUBST(LIB_XML_CMD_AGE)
 +
AC_SUBST(LIB_XML_CMD_REVISION)
 +
LIBXMLCOMMAND_VERSION="$LIB_XML_CMD_CURRENT:$LIB_XML_CMD_REVISION:$LIB_XML_CMD_AGE"
 +
AC_SUBST(LIBXMLCOMMAND_VERSION)
 +
</source>
 +
 
 +
== Update the '''ChangeLog''' file. ==
 
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.  
 
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.  
: '''gitlog-to-changelog --since="1970-01-01"''' (''Specify a date one day later than the one at the top of ChangeLog'')
+
: <tt style="font-size: 1.1em; font-weight: bold;">gitlog-to-changelog --since="1970-01-01"</tt>
 +
:: (''Specify a date one day later than the one at the top of the existing ChangeLog file.'')
 
Save the result to a temp file and combine the two files: <br/>
 
Save the result to a temp file and combine the two files: <br/>
: '''cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog'''
+
: <tt style="font-size: 1.1em; font-weight: bold;">cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog</tt>
 
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. <br/>
 
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. <br/>
 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''
 
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''
  
==== Update the NEWS file ====
+
== Update the '''NEWS''' file ==
To update the NEWS file, just read over the new ChangeLog entries and summarize. In the old days of CVS, the logs automatically included the names of the changed files, but subversion and git don't do that.
+
To update the NEWS file, just read over the new ChangeLog entries and summarize.  
==== Update the internal library version numbers ====
 
There are really 2 version numbers for each of these project items. The ''human'' version (like bes-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0
 
  
 +
The new entries to the NEWS file will be used later when making the GitHub release and when writing the server's release page on www.opendap.org.
 +
 +
We might replace this:
 +
* It's also helpful to have, in the '''NEWS''' file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.)
 +
 +
With instructions about making an associated release in JIRA using version tagging.
 +
 +
== Update the Version Numbers for Humans ==
 
;Affected Files:  
 
;Affected Files:  
 
: configure.ac
 
: configure.ac
 +
: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog] Debian ChangeLog)
 +
: NEWS
 +
: README.md
 +
: INSTALL
  
The rules for shared image version numbers:
+
# Determine the human version number. This appears to be a somewhat subjective process.
# No interfaces changed, only implementations (good): Increment REVISION.
+
# Edit each of the ''Affected Files'' and update the human version number.
# Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
+
# In the '''README.md''' file be sure to update the description of how to locate the DOI for the release with the new version number.
# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.
+
 
 +
== Update the libdap version ==
 +
Determine the libdap version associated with this release by checking the contents of the file <tt>libdap4-snapshot</tt> The <tt>libdap4-snapshot</tt> file should contain a single line like this example:
 +
<pre>
 +
libdap4-3.20.9-0 2021-12-28T19:23:45+0000
 +
</pre>
 +
 
 +
The libdap version for the above example is: <tt>libdap-3.20.9</tt> (The version is NOT <tt>libdap4-3.20.9</tt>)
 +
 
 +
=== Update the libdap version in the .travis.yml file ===
 +
;Affected Files
 +
: .travis.yml
 +
 
 +
In the .travis.yml file update the value of  ''LIBDAP_RPM_VERSION'' in the ''env: global:'' section so that it contains the complete numerical value of the libdap version you located in the previous step. Using the previous example the value would be:
 +
<pre>
 +
    - LIBDAP_RPM_VERSION=3.20.9-0
 +
</pre>
 +
 
 +
=== Update the libdap version in the RPM spec files ===
 +
;Affected Files
 +
: ''bes.spec*.in''
 +
Update the <tt>bes.spec*.in</tt> files by changing the <tt>Requires</tt> and <tt>BuildRequires</tt> entries for libdap. Based on our example the result would be:
 +
 
 +
<pre>
 +
Requires:      libdap >= 3.20.9
 +
BuildRequires: libdap-devel >= 3.20.9
 +
</pre>
  
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
+
(''These lines may not be adjacent to each other in the spec files'')
: For C++, build a file of the methods and their arguments using:
+
 
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs'''
+
=== Update the libdap version in the README.md file ===
: and compare that using <tt>diff</tt> on the previous release's library.
+
;Affected Files
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
+
: README.md
* No interfaces changed, only implementations (good): ==> Increment REVISION.
+
# [https://zenodo.org Get the DOI markdown from Zenodo] by using the search bar and searching for the libdap version string that you determined at the beginning of this section.  
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
+
# Update the '''README.md''' file with libdap version and the associated DOI link (using the markdown you got from Zenodo).
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
+
 
The current value of  <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
+
; Note
<source lang="bash">
+
: You will also need this DOI markdown when making the GitHub release page for the BES.  
LIB_DIS_CURRENT=14
+
 
LIB_DIS_AGE=6
+
See the section on this page titled "''Get the BES DOI from Zenodo''" for more details about getting the DOI markdown.
LIB_DIS_REVISION=1
 
</source>
 
Once you have determined the new values of  the <tt>CURRENT:REVISION:AGE</tt> strings then:
 
;Edit the configure.ac and update the version values to the new ones.
 
  
===== Update The BES dependencies =====
+
== Update the RPM dependencies ==
In the BES, update the requirements (''Requires'' line in spec file) for ''libdap'' as needed.
 
 
;Affected Files:  
 
;Affected Files:  
: *.spec
+
:''bes.spec*.in''
 +
 
 +
In the RPM ''.spec'' file, update the dependencies as needed.
 +
* The libdap version dependency was covered in a previous step.
 +
* Be attentive to changes that have been made to the hyrax-dependencies since the last release.
 +
 
 +
== Update the module version numbers for humans ==
 +
In bes/modules/common, check that the file all-modules.txt is complete and update as needed. Then run:
 +
 
 +
* Remove the sentinel files that prevent the version updater from being run multiple times in succession without specific intervention:
 +
:: '''<tt>rm -v ../*/version_updated</tt>'''
 +
* Now run the version updater:
 +
:: '''<tt>./version_update_modules.sh -v </tt>'''
 +
 
 +
This will update the patch number (x.y.patch) for each of the named modules.
 +
 
 +
If a particular module has significant fixes, hand edit the number, in the Makefile.am.
 +
 
 +
See below for special info about the HDF4/5 modules (which also applies to any modules not in the BES GitHub repo).
 +
 
 +
== <del>For the BES HDF4/5 modules (BES only) </del>==
 +
# <del>''Make sure that you are working on the master branch of each module!!''</del>
 +
# <del> Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).</del>
 +
# <del> Update the module version numbers in their respective Makefile.am files.</del>
 +
# <del> Commit and Push these changes.</del>
 +
 
 +
== Update the Build Offset ==
 +
''Setting the build offset correctly will set the build number for the new release to "0".''
 +
 
 +
In the file <tt>travis/travis_bes_build_offset.sh</tt> set the value of <tt>BES_TRAVIS_BUILD_OFFSET</tt> to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in [https://app.travis-ci.com/github/OPENDAP/bes the TravisCI page for the BES] and use that build number plus 1.
 +
 
 +
== Commit Changes ==
 +
''Be sure that you have completed all of the changes to the various ChangeLog, NEWS, INSTALL, configure.ac,  <tt>travis/travis_bes_build_offset.sh</tt>, and other files before proceeding!''
 +
 
 +
# Commit and push the BES code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.
  
===== Updating the Human Version Number =====
+
== Tag the BES code ==
# Determine the human version number. This appears to be a somewhat subjective process.
+
# Tag the bes code using command line git in your local (up-to-date) '''bes''' project
# Edit each of the ''Affected Files'' and update the human version number.
+
#* <tt style="font-size: 1.1em; font-weight: bold;">git tag -m "bes-<number>" -a <numbers> </tt>
:;Affected Files:  
+
#* <tt style="font-size: 1.1em; font-weight: bold;">git push origin <numbers></tt>
:: configure.ac
+
#: <br/>
:: *.spec
+
# If this is part of a Hyrax Release, then tag this point in the master branch with the Hyrax release number
:: debian/changelog (see https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog)
+
#* <tt style="font-size: 1.1em; font-weight: bold;">git tag -m "hyrax-<number>" -a hyrax-<numbers></tt>
:: NEWS
+
#* <tt style="font-size: 1.1em; font-weight: bold;">git push origin hyrax-<numbers></tt>
:: README.md
+
#: '''NB:''' ''Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.''
:: INSTALL
 
  
Note that it's helpful to have, in the '''NEWS''' file, a list of the Jira ''Bug'' tickets that have been closed since the last release. Make a filter for this by going to ''Advanced search'' on the ''Issues'' page/menu. Here's what the JQL looks like:
+
== Create the BES release on Github ==
  ''project = HYRAX AND updatedDate >= "2017/05/31" AND resolution = Done AND Type = Bug ORDER BY Key''
+
# [https://github.com/OPENDAP/bes Goto the BES project page in GitHub]
Edit the date, of course. If you ''Save as'' this search, it will appear in the ''Issues'' menu. The Jira interface makes editing the date easy. You can find the date of the last release from our web page or from the GitHub tag/release information.
+
# Choose the '''releases''' tab.
 +
# On the [https://github.com/OPENDAP/bes/releases Releases page] click the 'Tags' tab.
 +
# On the [https://github.com/OPENDAP/bes/tags Tags page], locate the tag (created above) associated with this new release.
 +
# Click the ellipses (...) located on the far right side of the ''version-x.y.z'' tag 'frame' for this release and and choose ''Create release''.
 +
#* Enter a ''title'' for the release
 +
#* Copy the most recent text from the NEWS file into the ''describe'' field
 +
#* Click '''Publish release''' or  '''Save draft'''.  
 +
#** If you have previously edited the release page you can click '''Update this release'''
  
==== Commit & Push these changes. ====
+
== Publish and Sign ==
''Srsly.''
 
  
==== For the BES modules ====
+
When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.
''This will change when the BES project is restructured.''
 
  
Do the release items above for the source release and then, for the modules, do the following:
+
All you need do is build the tar file using <tt>make list</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.  
* Confirm that the modules are on the correct branch:
 
*: '''git submodule foreach 'git status''''
 
* If modules have been added or removed you will need to edit the file <tt>bes/modules/all_modules.txt</tt> to reflect the changes.
 
* Use the magic version update shell scripty to update the various module's ChangeLog and version numbers:
 
*: '''cd modules; ./version_update_modules.sh -v `cat all_modules.txt`'''
 
* Have a look at what happened:
 
*: '''git submodule foreach 'git status''''
 
* In each module:
 
**Review <tt>ChangeLog</tt> file and adjust the libtool version numbers as needed.
 
** Edit the module Makefile.am look for and update  '''M_VER=''' line. (Human Version Number)
 
* Commit and push module changes:
 
*: '''git submodule foreach 'git commit -a -m "updated for Hyrax-xx.xx.xx"''''
 
*: '''git submodule foreach 'git push''''
 
That completes the update of the modules (sub-projects)
 
  
''Lastly commit and push the changes to the BES!''
+
# Go to the '''bes''' project on your local machine and run ''<tt>make dist</tt>'' which will make a bes-x.y.z,tar.gz file at the top level of the '''bes''' project.
: '''git commit -a -m "Updated for Hyrax-xx.xx.xx" '''
+
# Use '''gpg''' to sign the tar bundle:
: '''git push'''
+
#: <tt>gpg --detach-sign --local-user security@opendap.org bes-x.y.z.tgz</tt>
 +
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
 +
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
 +
#: <tt>sftp opendap@www.opendap.org</tt>
 +
#: <tt>cd httpdocs/pub/source</tt>
 +
#: <tt>put bes-x.y.z.tgz.sig</tt>
 +
#: <tt>put bes-x.y.z.tgz</tt>
 +
#: <tt>quit</tt>
 +
# Check your work!
 +
## Download the source tar bundle and signature from www.opendap.org.
 +
## Verify the signature:
 +
##: <tt> gpg --verify bes-x.y.z.tgz.sig bes-x.y.z.tgz</tt>
  
=== Build And Upload Distribution Files ===
+
== Get the BES DOI from Zenodo ==
# Make the tar file
+
Get the Zenodo DOI for the newly created BES release and add it to the associated GitHub BES release page.
#* '''make dist'''
 
#* '''Sign''' the tar file. [[SecureEmail | See the help item on the main trac wiki page if you need instructions.]]
 
#* Move the file to the web site for distribution.
 
# For the BES, see below
 
# Make the [[RPM]]
 
#*'''Sign''' the RPM
 
#*Move to a directory on the web site and use script to move the RPM around (later, once all the RPMs are uploaded). -->
 
# Update the web pages: A complex and (at this time - 2/21/06) changing process.
 
  
=== Tag The Release ===
+
# [https://zenodo.org Goto Zenodo]
# '''git tag -a version-<numbers> -m "Version <number>"'''
+
# Look at the 'upload' page. If there is nothing there (perhaps because you are not ''jhrg'' or whoever set up the connection between the BES project and Zenodo) you can use the search bar to search for '''bes'''.
# '''git push origin version-<numbers>'''
+
#: Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
 +
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
 +
# Copy the DOI as markdown from the window that pops up.
 +
# Edit the GitHub release page for the BES release you just created and paste the DOI markdown into the top of the  description.
  
If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:
+
'''Tip:''' ''If you are trying to locate the '''libdap''' releases in Zenodo you have to search for the string:'' <tt style="font-size: 1.1em; font-weight: bold;">libdap4</tt>
# '''git tag -a hyrax-<numbers> -m "Hyrax <number>"'''
 
# '''git push origin hyrax-<numbers>'''
 
#: NB: Instead of tagging all of the modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error; recording the wrong version for one of the 17 or so modules. thus, there is no need to tag the code in the modules.
 
# '''Mark the release on Github:'''
 
#* Goto the 'releases' page and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'vrsion-*' tag and:
 
#** Enter a ''title'' for the release
 
#** Copy the most recent text from the NEWS file into the ''describe'' field
 
#** Click ''Update this release'' or ''Save draft''
 
  
== [[RPM#Building_RPMs_that_have_Statically_Linked_Handlers|Build The Release RPM and Source Distributions]] ==
+
=== Images ===
 +
[[File:Screenshot 2018-12-06 11.06.44.png|none|thumb|400px|border|left|Zenodo upload page]]

Latest revision as of 14:50, 22 July 2022

This pages covers the steps required to release the BES software for Hyrax.

We now depend on the CI/CD process to build binary packages and to test the source builds.

Tip: If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a release branch that you can merge and discard later. Do not make a release branch if you don't need it, since it complicates making tags.

1 Verify the code base

  1. We release using the master branch. The code on master must have passed the CI builds. This includes the hyrax-docker builds since that CI build runs the full server regression tests!
  2. Make sure that the source code you're using for the following steps is up-to-date. (git pull)

2 Update the Version Numbers

2.1 Version for Humans

  1. Determine the human version number. This appears to be a somewhat subjective process.
  2. Edit each of the Affected Files and update the human version number.
Affected Files
  • configure.ac - Look for:
AC_INIT(bes, ###.###.###, opendap-tech@opendap.org)
Take Note! The debian/changelog is the "single source of truth" for the libdap4 version in the debian packaging. If this does not agree with the version being packaged the package build will fail.
  • ChangeLog
  • NEWS
  • README.md
  • INSTALL

2.2 Update the internal library (API/ABI) version numbers.

The BES is not a shared library, it is a set of c++ applications that are typically built as statically linked binaries. Because of this the usual CURRENT:REVISION:AGE tuples used to express the binary compatibility state of a c++ shared object library have little meaning for the BES code. So, what we choose to do is simply bump the REVISION numbers by one value for each release.

  • In the configure.ac file locate each of:
    • LIB_DIS_REVISION
    • LIB_PPT_REVISION
    • LIB_XML_CMD_REVISION
  • Increase the value of each by one (1).
  • Save the file.
  • Update the text documentation files and version numbers in the configuration files:

Example of the relevant section from configure.ac:

LIB_DIS_CURRENT=18
LIB_DIS_AGE=3
LIB_DIS_REVISION=3
AC_SUBST(LIB_DIS_CURRENT)
AC_SUBST(LIB_DIS_AGE)
AC_SUBST(LIB_DIS_REVISION)
LIBDISPATCH_VERSION="$LIB_DIS_CURRENT:$LIB_DIS_REVISION:$LIB_DIS_AGE"
AC_SUBST(LIBDISPATCH_VERSION)

LIB_PPT_CURRENT=5
LIB_PPT_AGE=1
LIB_PPT_REVISION=2
AC_SUBST(LIB_PPT_CURRENT)
AC_SUBST(LIB_PPT_AGE)
AC_SUBST(LIB_PPT_REVISION)
LIBPPT_VERSION="$LIB_PPT_CURRENT:$LIB_PPT_REVISION:$LIB_PPT_AGE"
AC_SUBST(LIBPPT_VERSION)

LIB_XML_CMD_CURRENT=5
LIB_XML_CMD_AGE=4
LIB_XML_CMD_REVISION=2
AC_SUBST(LIB_XML_CMD_CURRENT)
AC_SUBST(LIB_XML_CMD_AGE)
AC_SUBST(LIB_XML_CMD_REVISION)
LIBXMLCOMMAND_VERSION="$LIB_XML_CMD_CURRENT:$LIB_XML_CMD_REVISION:$LIB_XML_CMD_AGE"
AC_SUBST(LIBXMLCOMMAND_VERSION)

3 Update the ChangeLog file.

Use the script gitlog-to-changelog (which can be found with Google) to update the ChangeLog file by running it using the --since="<date>" option with a date one day later in time than the newest entry in the current ChangeLog.

gitlog-to-changelog --since="1970-01-01"
(Specify a date one day later than the one at the top of the existing ChangeLog file.)

Save the result to a temp file and combine the two files:

cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog

If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first.
Tip: When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.

4 Update the NEWS file

To update the NEWS file, just read over the new ChangeLog entries and summarize.

The new entries to the NEWS file will be used later when making the GitHub release and when writing the server's release page on www.opendap.org.

We might replace this:

  • It's also helpful to have, in the NEWS file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto Jira's Issues page and look at the Tickets closed recently item. From there, click on Advanced and edit the time range so it matches the time range since the past release to now, then Export that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.)

With instructions about making an associated release in JIRA using version tagging.

5 Update the Version Numbers for Humans

Affected Files
configure.ac
debian/changelog (see [1] Debian ChangeLog)
NEWS
README.md
INSTALL
  1. Determine the human version number. This appears to be a somewhat subjective process.
  2. Edit each of the Affected Files and update the human version number.
  3. In the README.md file be sure to update the description of how to locate the DOI for the release with the new version number.

6 Update the libdap version

Determine the libdap version associated with this release by checking the contents of the file libdap4-snapshot The libdap4-snapshot file should contain a single line like this example:

libdap4-3.20.9-0 2021-12-28T19:23:45+0000

The libdap version for the above example is: libdap-3.20.9 (The version is NOT libdap4-3.20.9)

6.1 Update the libdap version in the .travis.yml file

Affected Files
.travis.yml

In the .travis.yml file update the value of LIBDAP_RPM_VERSION in the env: global: section so that it contains the complete numerical value of the libdap version you located in the previous step. Using the previous example the value would be:

    - LIBDAP_RPM_VERSION=3.20.9-0

6.2 Update the libdap version in the RPM spec files

Affected Files
bes.spec*.in

Update the bes.spec*.in files by changing the Requires and BuildRequires entries for libdap. Based on our example the result would be:

Requires:       libdap >= 3.20.9
BuildRequires:  libdap-devel >= 3.20.9

(These lines may not be adjacent to each other in the spec files)

6.3 Update the libdap version in the README.md file

Affected Files
README.md
  1. Get the DOI markdown from Zenodo by using the search bar and searching for the libdap version string that you determined at the beginning of this section.
  2. Update the README.md file with libdap version and the associated DOI link (using the markdown you got from Zenodo).
Note
You will also need this DOI markdown when making the GitHub release page for the BES.

See the section on this page titled "Get the BES DOI from Zenodo" for more details about getting the DOI markdown.

7 Update the RPM dependencies

Affected Files
bes.spec*.in

In the RPM .spec file, update the dependencies as needed.

  • The libdap version dependency was covered in a previous step.
  • Be attentive to changes that have been made to the hyrax-dependencies since the last release.

8 Update the module version numbers for humans

In bes/modules/common, check that the file all-modules.txt is complete and update as needed. Then run:

  • Remove the sentinel files that prevent the version updater from being run multiple times in succession without specific intervention:
rm -v ../*/version_updated
  • Now run the version updater:
./version_update_modules.sh -v

This will update the patch number (x.y.patch) for each of the named modules.

If a particular module has significant fixes, hand edit the number, in the Makefile.am.

See below for special info about the HDF4/5 modules (which also applies to any modules not in the BES GitHub repo).

9 For the BES HDF4/5 modules (BES only)

  1. Make sure that you are working on the master branch of each module!!
  2. Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).
  3. Update the module version numbers in their respective Makefile.am files.
  4. Commit and Push these changes.

10 Update the Build Offset

Setting the build offset correctly will set the build number for the new release to "0".

In the file travis/travis_bes_build_offset.sh set the value of BES_TRAVIS_BUILD_OFFSET to the number of the last TravisCI build plus one. The previous commit and push will have triggered a TravisCI build. Find the build number for the previous commit in the TravisCI page for the BES and use that build number plus 1.

11 Commit Changes

Be sure that you have completed all of the changes to the various ChangeLog, NEWS, INSTALL, configure.ac, travis/travis_bes_build_offset.sh, and other files before proceeding!

  1. Commit and push the BES code. Wait for the CI/CD builds to complete. You must be working on the master branch to get the CD package builds to work.

12 Tag the BES code

  1. Tag the bes code using command line git in your local (up-to-date) bes project
    • git tag -m "bes-<number>" -a <numbers>
    • git push origin <numbers>

  2. If this is part of a Hyrax Release, then tag this point in the master branch with the Hyrax release number
    • git tag -m "hyrax-<number>" -a hyrax-<numbers>
    • git push origin hyrax-<numbers>
    NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

13 Create the BES release on Github

  1. Goto the BES project page in GitHub
  2. Choose the releases tab.
  3. On the Releases page click the 'Tags' tab.
  4. On the Tags page, locate the tag (created above) associated with this new release.
  5. Click the ellipses (...) located on the far right side of the version-x.y.z tag 'frame' for this release and and choose Create release.
    • Enter a title for the release
    • Copy the most recent text from the NEWS file into the describe field
    • Click Publish release or Save draft.
      • If you have previously edited the release page you can click Update this release

14 Publish and Sign

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is not the one we wish to publish because it requires people to have autoconf installed. Rather we want to use the result of "make dist" which will have the configure script pre-generated.

All you need do is build the tar file using make list, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

  1. Go to the bes project on your local machine and run make dist which will make a bes-x.y.z,tar.gz file at the top level of the bes project.
  2. Use gpg to sign the tar bundle:
    gpg --detach-sign --local-user security@opendap.org bes-x.y.z.tgz
  3. Use sftp to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
    (Assuming your current working directory is the top of the bes project)
    sftp opendap@www.opendap.org
    cd httpdocs/pub/source
    put bes-x.y.z.tgz.sig
    put bes-x.y.z.tgz
    quit
  4. Check your work!
    1. Download the source tar bundle and signature from www.opendap.org.
    2. Verify the signature:
      gpg --verify bes-x.y.z.tgz.sig bes-x.y.z.tgz

15 Get the BES DOI from Zenodo

Get the Zenodo DOI for the newly created BES release and add it to the associated GitHub BES release page.

  1. Goto Zenodo
  2. Look at the 'upload' page. If there is nothing there (perhaps because you are not jhrg or whoever set up the connection between the BES project and Zenodo) you can use the search bar to search for bes.
    Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
  3. Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
  4. Copy the DOI as markdown from the window that pops up.
  5. Edit the GitHub release page for the BES release you just created and paste the DOI markdown into the top of the description.

Tip: If you are trying to locate the libdap releases in Zenodo you have to search for the string: libdap4

15.1 Images

Zenodo upload page