Source Release for BES: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
Line 15: Line 15:


== The Release Process ==
== The Release Process ==
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. If you don't need to do this, skip it. Also, note that the handlers that are ''git submodules'' really make this a hassle, so feel free to skip this step or do it only for the BES itself, etc.
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. If you don't '''need''' to do this, do not make a release branch since it complicates making tags.
:'''Create the release branch.'''
 
:* ''git checkout -b hyrax_release_0.0.0''
:* ''git push --set-upstream origin hyrax_release_0.0.0''
: For the bes, also make the branch for each submodule:
:* ''git submodule foreach 'git checkout -b hyrax_release_0.0.0' ''
===  Verify the code base ===
===  Verify the code base ===
# Make sure that the source code you're using is up-to-date. (''git pull'')
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.  
# Have all tickets for the release milestone been resolved?
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')
# Does the code pass it's tests? The Travis builds should be running the distcheck targets for these packages.
#;Note
#: For the BES (''or any software that requires parameters passed to configure'') use <tt>DISTCHECK_CONFIGURE_FLAGS=...</tt> when you run <tt>make</tt> For running tests, using <tt>TESTSUITEFLAGS=-j9</tt> will speed up the autconf/autotest tests if the version of autoconf supports parallel testing (but not on CentOS 6 because it's not supported by the stock version of autoconf on that OS).
#* ''libdap4''
#:: '''make check -j9'''
#:: '''make distcheck -j9'''
#* ''bes''
#:: '''make check -j9'''
#:: '''make distcheck -j9 DISTCHECK_CONFIGURE_FLAGS=--with-dependencies=$prefix/deps''' (DISTCHECK_CONFIGURE... no longer needed with automake 1.14 or greater. jhrg 9/21/17)
#: How did that go, with the <tt>distcheck</tt> target there? [[Debugging the distcheck target|Look here for help]]
# Once the tests pass merge the '''master''' branch to the '''coverity-scan''' branch. Read the resulting report and deal with the identified issues. Once completed return to the top of this section and repeat until everything is clean.
 
=== Check the<tt> make rpm </tt>target ===
Running the <tt>make rpm</tt> target is useful because it will find problems that can slow the process once the final edits are done and the RPMs are being built.  
# Make sure to install the ''libdap rpms'' before you build the bes rpms and uninstall them afterwards.:
#: '''sudo yum install ~/rpmbuild/RPMS/x86_64/libdap-*'''
# Make sure that the bes dependencies have been built using the static option so that the BES rpm will have modules that are statically linked to their dependencies. To check this look at the libraries in <tt>$prefix/deps/lib</tt>. There should be no <tt>*.so</tt> files in the <tt>lib</tt> dir, only <tt>*.a</tt>, <tt>*.la</tt>, and some other directories. To build a new set of dependencies that provide only static libraries, use:
#: '''make for-static-rpm -j9 CONFIGURE_FLAGS=--disable-shared'''
# Build the BES rpm using those snazzy static libraries, use
#: '''make all-static-rpm -j9'''
# When finished, uninstall the libdap rpms:
#: '''sudo yum uninstall ~/rpmbuild/RPMS/x86_64/libdap-*'''
 
----
 
=== Checkpoint ===
----
Up until now you have only verified the software's readiness for release. Beginning with the follow section you will be making the edits and changes required to update the version numbers and build the actual release files. Do not proceed unless you have passed all of the previous process checks. Thanks.


=== Update Release Related Files ===
=== Update Release Related Files ===
Line 63: Line 31:
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. <br/>
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. <br/>
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''
==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize. In the old days of CVS, the logs automatically included the names of the changed files, but subversion and git don't do that.
==== Update the internal library version numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like bes-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0


;Affected Files:  
;Affected Files:  
: configure.ac
: ChangeLog


The rules for shared image version numbers:
==== Update the NEWS file ====
# No interfaces changed, only implementations (good): Increment REVISION.
To update the NEWS file, just read over the new ChangeLog entries and summarize.
# Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.
 
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of  <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of  the <tt>CURRENT:REVISION:AGE</tt>  strings then:
;Edit the configure.ac and update the version values to the new ones.


===== Update The BES dependencies =====
In the BES, update the requirements (''Requires'' line in spec file) for ''libdap'' as needed.
;Affected Files:  
;Affected Files:  
: *.spec
: NEWS


===== Updating the Human Version Number =====
==== Updating the Human Version Number ====
# Determine the human version number. This appears to be a somewhat subjective process.
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.
# Edit each of the ''Affected Files'' and update the human version number.
:;Affected Files:  
:;Affected Files:  
:: configure.ac
:: configure.ac
Line 110: Line 53:
:: INSTALL
:: INSTALL


Note that it's helpful to have, in the '''NEWS''' file, a list of the Jira ''Bug'' tickets that have been closed since the last release. Make a filter for this by going to ''Advanced search'' on the ''Issues'' page/menu. Here's what the JQL looks like:
It's helpful to have, in the '''NEWS''' file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.
  ''project = HYRAX AND updatedDate >= "2017/05/31" AND resolution = Done AND Type = Bug ORDER BY Key''
 
Edit the date, of course. If you ''Save as'' this search, it will appear in the ''Issues'' menu. The Jira interface makes editing the date easy. You can find the date of the last release from our web page or from the GitHub tag/release information.
==== Update the RPM dependencies ====
In the RPM ''.spec'' file, update the dependencies as needed.
 
;Affected Files:
: *.spec
 
==== Update the internal library version numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like bes-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0
 
The rules for shared image version numbers:
# No interfaces changed, only implementations (good): Increment REVISION.
# Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.


==== Commit & Push these changes. ====
See ''How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.
''Srsly.''


==== For the BES modules ====
;Affected Files:
''This will change when the BES project is restructured.''
: configure.ac


Do the release items above for the source release and then, for the modules, do the following:
* Confirm that the modules are on the correct branch:
*: '''git submodule foreach 'git status''''
* If modules have been added or removed you will need to edit the file <tt>bes/modules/all_modules.txt</tt> to reflect the changes.
* Use the magic version update shell scripty to update the various module's ChangeLog and version numbers:
*: '''cd modules; ./version_update_modules.sh -v `cat all_modules.txt`'''
* Have a look at what happened:
*: '''git submodule foreach 'git status''''
* In each module:
**Review <tt>ChangeLog</tt> file and adjust the libtool version numbers as needed.
** Edit the module Makefile.am look for and update  '''M_VER=''' line. (Human Version Number)
* Commit and push module changes:
*: '''git submodule foreach 'git commit -a -m "updated for Hyrax-xx.xx.xx"''''
*: '''git submodule foreach 'git push''''
That completes the update of the modules (sub-projects)


''Lastly commit and push the changes to the BES!''
==== For the BES HDF4/5 modules ====
: '''git commit -a -m "Updated for Hyrax-xx.xx.xx" '''
# Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).
: '''git push'''
# Update the module version numbers in their respective Makefile.am files.
# Commit and Push these changes.


=== Build And Upload Distribution Files ===
====
# Make the tar file
#* '''make dist'''
#* '''Sign''' the tar file. [[SecureEmail | See the help item on the main trac wiki page if you need instructions.]]
#* Move the file to the web site for distribution.
# For the BES, see below
# Make the [[RPM]]
#*'''Sign''' the RPM
#*Move to a directory on the web site and use script to move the RPM around (later, once all the RPMs are uploaded). -->
# Update the web pages: A complex and (at this time - 2/21/06) changing process.


=== Tag The Release ===
=== Tag The Release ===
Line 158: Line 89:
# '''git tag -a hyrax-<numbers> -m "Hyrax <number>"'''
# '''git tag -a hyrax-<numbers> -m "Hyrax <number>"'''
# '''git push origin hyrax-<numbers>'''
# '''git push origin hyrax-<numbers>'''
#: NB: Instead of tagging all of the modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error; recording the wrong version for one of the 17 or so modules. thus, there is no need to tag the code in the modules.
#: NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.
# '''Mark the release on Github:'''
# Mark the release on Github:
#* Goto the 'releases' page and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
#* Goto the 'releases' page and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
#** Enter a ''title'' for the release
#** Enter a ''title'' for the release
#** Copy the most recent text from the NEWS file into the ''describe'' field
#** Copy the most recent text from the NEWS file into the ''describe'' field
#** Click ''Update this release'' or ''Save draft''
#** Click ''Update this release'' or ''Save draft''
=== How to see the scope of API/ABI changes in C++ sources ===
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
: For C++, build a file of the methods and their arguments using:
:: '''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs'''
: and compare that using <tt>diff</tt> on the previous release's library.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
* No interfaces changed, only implementations (good): ==> Increment REVISION.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
The current value of  <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
<source lang="bash">
LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1
</source>
Once you have determined the new values of  the <tt>CURRENT:REVISION:AGE</tt>  strings then:
;Edit the configure.ac and update the version values to the new ones.


== [[RPM#Building_RPMs_that_have_Statically_Linked_Handlers|Build The Release RPM and Source Distributions]] ==
== [[RPM#Building_RPMs_that_have_Statically_Linked_Handlers|Build The Release RPM and Source Distributions]] ==

Revision as of 22:32, 26 November 2018

Note that the old pages with the tables of version numbers are gone. Use the web pages themselves as documentation of what has or has not been built. For the current release, include grayed-out links to binaries we do plan to make but keep the grayed-out links in place for the current version only!

When we decide to release software

Use the developers at opendap.org list to send out notices to all the developers about pending releases.

Planned release

If this is a planned release, send out a notice at least two weeks in advance to anyone who likely has made changes to the software. More lead time is, of course, better. This will allow developers time to get their code and documentation changes into github and onto the master branch in time for the release.

Run Coverity!

Emergency releases

If this is an 'emergency' release, then send out a notice to developers as soon as the decision to release is made, since this will give a chance for other developers to lets us know if there are new changes in the master branch that are ready for release. if we don't get an email from those developers of a particular component, then we should assume that the code might not be 100% ready for release and we should use the version tagged with/for the last release if possible.

Maybe run Coverity - depends on the scope of the changes.

The Release Process

Tip: If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a release branch that you can merge and discard later. If you don't need to do this, do not make a release branch since it complicates making tags.

Verify the code base

  1. We release using the master branch. The code on master must pass the CI build.
  2. Make sure that the source code you're using for the following steps is up-to-date. (git pull)

Update Release Related Files

Update the text documentation files and version numbers in the configuration files:

Update the ChangeLog file.

Use the script gitlog-to-changelog (which can be found with Google) to update the ChangeLog file by running it using the --since="<date>" option with a date one day later in time than the newest entry in the current ChangeLog.

gitlog-to-changelog --since="1970-01-01" (Specify a date one day later than the one at the top of ChangeLog)

Save the result to a temp file and combine the two files:

cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog

If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first.
Tip: When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.

Affected Files
ChangeLog

Update the NEWS file

To update the NEWS file, just read over the new ChangeLog entries and summarize.

Affected Files
NEWS

Updating the Human Version Number

  1. Determine the human version number. This appears to be a somewhat subjective process.
  2. Edit each of the Affected Files and update the human version number.
Affected Files
configure.ac
*.spec
debian/changelog (see https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog)
NEWS
README.md
INSTALL

It's helpful to have, in the NEWS file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto Jira's Issues page and look at the Tickets closed recently item. From there, click on Advanced and edit the time range so it matches the time range since the past release to now, then Export that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

Update the RPM dependencies

In the RPM .spec file, update the dependencies as needed.

Affected Files
*.spec

Update the internal library version numbers

There are really 2 version numbers for each of these project items. The human version (like bes-3.17.5) and the library API/ABI version which is represented as CURRENT:REVISION:AGE. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like 25:0:0 but the human version might only change from bes-3.17.3 to bes-3.18.0

The rules for shared image version numbers:

  1. No interfaces changed, only implementations (good): Increment REVISION.
  2. Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
  3. Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.

See How to see the scope of API/ABI changes in C++ sources below for gruesome details. Often basic knowledge of the edits is good enough.

Affected Files
configure.ac


For the BES HDF4/5 modules

  1. Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).
  2. Update the module version numbers in their respective Makefile.am files.
  3. Commit and Push these changes.

==

Tag The Release

  1. git tag -a version-<numbers> -m "Version <number>"
  2. git push origin version-<numbers>

If this is part of Hyrax, also tag this point in the master branch with the Hyrax release number:

  1. git tag -a hyrax-<numbers> -m "Hyrax <number>"
  2. git push origin hyrax-<numbers>
    NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.
  3. Mark the release on Github:
    • Goto the 'releases' page and click the 'Tags' tab. There, click the ellipses (...) on the right of the 'version-*' tag and:
      • Enter a title for the release
      • Copy the most recent text from the NEWS file into the describe field
      • Click Update this release or Save draft

How to see the scope of API/ABI changes in C++ sources

Determine the new software version (assuming you don't already know the extent of the changes that have been made)

For C++, build a file of the methods and their arguments using:
nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs
and compare that using diff on the previous release's library.

Assess the changes you find based on the following rules for the values of CURRENT,REVISION, and AGE

  • No interfaces changed, only implementations (good): ==> Increment REVISION.
  • Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
  • Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.

The current value of CURRENT,REVISION, and AGE can be found in configure.ac:

LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1

Once you have determined the new values of the CURRENT:REVISION:AGE strings then:

Edit the configure.ac and update the version values to the new ones.

Build The Release RPM and Source Distributions