Source Release for BES: Difference between revisions

From OPeNDAP Documentation
⧼opendap2-jumptonavigation⧽
mNo edit summary
(119 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Note that the old pages with the tables of version numbers are gone. Use the web pages themselves as documentation of what has or has not been built. For the current release, include grayed-out links to binaries we do plan to make but keep the grayed-out links in place for the current version only!


Here's how to make a source release:
This pages covers the steps required to release the BES software for Hyrax.


#First, make sure that the source code you're using is on a branch, up-to-date (''svn up'') and that the software passes its tests and no outstanding tickets remain for the release milestone.
We now depend on the CI/CD process to build binary packages and to test the source builds.
#Update the text documentation files and version numbers in the configuration files:
 
#*Update the '''ChangeLog''' file using the script ''update_cl.sh'' which can be found in the ''svn-tools'' project. (e.g., ~/svn-tools/update_cl.sh ChangeLog). If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. Here's a key tip: When you're making the commit log entries, always include the file name(s) of the affected files in your entry. Then to write the NEWS file, just read over the new ChangeLog entries and summarize. In the old days of CVS, the logs automatically included the names of the changed files, but subversion doesn't do that. A little discipline with those log entries will go a long way toward streamlining the release process!
== The Release Process ==
#*Make sure that the version number is set in the '''configure.ac''' and '''*.spec''' files! When the version number changes, reset the RPM release number to '1' in the '''*.spec''' files ("Release:    1"). For several releases of the same version, increment the release number (the number after the dash in the RPM file name; we don't have release numbers for the tar files). ''I generally update the ChangeLog before updating the version numbers or NEWS, et c., files because the ChangeLog gives me a bird's eye view of what happened since the last release.''
:'''Tip''': If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a ''release branch'' that you can merge and discard later. Do not make a release branch if you don't '''need''' it, since it complicates making tags.
#*libtool versioning rules: What the ''CURRENT[:REVISION[:AGE]]'' string passed to ''libtool means:'' (Note these are set using variables in the configure.ac script.)
 
#**No interfaces changed, only implementations (good): ==> Increment REVISION.
===  Verify the code base ===
#**Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
# We release using the ''master'' branch. The code on ''master'' must pass the CI build.  
#**Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
# Make sure that the source code you're using for the following steps is up-to-date. (''git pull'')
#**So how do you know? For C++, build a file of the methods and their arguments using ''nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs'' and compare that using diff on the previous release's library.
 
#*Make sure that '''README''', '''INSTALL''' and '''NEWS''' are updated (setting ''check-news'' in ''Makefile.am'' ''AUTOMAKE_OPTIONS'' will enforce this).
=== Determine the scope of API/ABI changes in C++ sources ===
#Check in these changes.
Determine the new software version (assuming you don't already know the extent of the changes that have been made)
#Tag the release:
: For C++, build a file of the methods and their arguments using the command:
#*I use svn="https://scm.opendap.org/svn" in the following and in my shell.
:: <tt style="font-size: 1.1em; font-weight: bold;">nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs</tt>
#*Use ''svn copy $svn/branch/<<package>> $svn/tags/<<package-name>>/<<release-number>>''
: and compare that using <tt>diff</tt> on the previous release's library.
#*If this is the first release of <<package-name>>, you have to create the directory under ''tags'' using ''svn mkdir''.
Assess the changes you find based on the following rules for the values of <tt>CURRENT</tt>,<tt>REVISION</tt>, and <tt>AGE</tt>
#'''Sign''' the tar file. [https://scm.opendap.org/trac/wiki/SecureEmail See the help item on the main trac wiki page if you need instructions.]
* No interfaces changed, only implementations (good): ==> Increment REVISION.
#Also check that the RPM and DMG targets are working. This is a lower priority item then the ''distcheck'' target because these targets seem pretty stable unless the packaging and distribution files are changed. If that is the case, these targets should be checked.
* Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
#Move the file to the web site for distribution.
* Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.
#Update the web pages:
The current value of  <tt>CURRENT</TT>,<tt>REVISION</tt>, and <tt>AGE</tt> can be found in <tt>configure.ac</tt>:
#*Edit the package-specific page
<source lang="bash">
#*Edit the download ''index.html'' page
LIB_DIS_CURRENT=14
#*Edit the ''whatsnew.html'' page
LIB_DIS_AGE=6
#*Add the edits from ''Whatsnew.html'' to the main ''index.html'' page under the News heading - the <tt><tr></tt> blocks can be copied verbatim and then the <tt><name></tt> element changed to a <tt><href></tt>. Follow the examples of the other entries.
LIB_DIS_REVISION=1
#*edit the ''news.xml'' page to '''update the RSS news feed.''' One very useful tool is the RSS [http://feedvalidator.org/ Feed Validator].
</source>
#*Edit the What's New section of the front page.
 
#*Check in those changes and ask someone else in the group to look them over. ''The more people from our group that look over prospective changes, the better.'' Then find someone who has write access to the web site and have them update the served pages. People who have write access are Yuan, James and Dan (at least, maybe others).
=== Update Release Files ===
#Build binaries and upload to the web site as time permits. See [[Distributions]].
Once you have determined the new values of  the <tt>CURRENT:REVISION:AGE</tt>  strings then:
* Edit the configure.ac and update the version values to the new ones.
* Update the text documentation files and version numbers in the configuration files:
 
==== Update the '''ChangeLog''' file. ====
Use the script <tt>gitlog-to-changelog</tt> (which can be found with Google) to update the '''ChangeLog''' file by running it using the <tt>--since="<date>"</tt> option with a date one day later in time than the newest entry in the current ChangeLog.  
: <tt style="font-size: 1.1em; font-weight: bold;">gitlog-to-changelog --since="1970-01-01"</tt>
:: (''Specify a date one day later than the one at the top of the existing ChangeLog file.'')
Save the result to a temp file and combine the two files: <br/>
: <tt style="font-size: 1.1em; font-weight: bold;">cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog</tt>
If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first. <br/>
'''Tip''': ''When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.''
 
==== Update the NEWS file ====
To update the NEWS file, just read over the new ChangeLog entries and summarize.
 
==== Update the Version Numbers for Humans ====
# Determine the human version number. This appears to be a somewhat subjective process.
# Edit each of the ''Affected Files'' and update the human version number.
 
:;Affected Files:
:: configure.ac
:: *.spec (In the BES it's ''bes.spec.*'')
:: debian/changelog (see [https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#changelog] Debian ChangeLog)
:: .travis.yml (for BES only)
:: NEWS
:: README.md (Add the libdap version the README.md)
:: INSTALL
 
It's helpful to have, in the '''NEWS''' file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto ''Jira's Issues'' page and look at the ''Tickets closed recently'' item. From there, click on ''Advanced'' and edit the time range so it matches the time range since the past release to now, then ''Export'' that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.
 
==== Update the RPM dependencies ====
In the RPM ''.spec'' file, update the dependencies as needed.
 
;Affected Files:
: *.spec (In the BES it's ''bes.spec.*''')
 
==== Update the internal library version numbers ====
There are really 2 version numbers for each of these project items. The ''human'' version (like bes-3.17.5) and the ''library'' API/ABI version which is represented as <tt>CURRENT:REVISION:AGE</tt>. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like <tt>25:0:0</tt> but the human version might only change from bes-3.17.3 to bes-3.18.0
 
The rules for shared image version numbers:
# No interfaces changed, only implementations (good): Increment REVISION.
# Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
# Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.
 
See ''How to see the scope of API/ABI changes in C++ sources'' below for gruesome details. Often basic knowledge of the edits is good enough.
 
;Affected Files:
: configure.ac
 
==== For the BES HDF4/5 modules (BES only) ====
# Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).
# Update the module version numbers in their respective Makefile.am files.
# Commit and Push these changes.
 
=== Commit Changes ===
# Commit and push the BES code. Wait for the CI/CD builds to complete. You must be working on the ''master'' branch to get the CD package builds to work.
 
=== Tag & Release ===
==== Tag the BES code ====
# Tag the bes code using command line git in your local (up-to-date) '''bes''' project
#* <tt style="font-size: 1.1em; font-weight: bold;">git tag -a version-<numbers> -m "Version <number>"</tt>
#* <tt style="font-size: 1.1em; font-weight: bold;">git push origin version-<numbers></tt>
#: <br/>
# If this is part of a Hyrax Release, then tag this point in the master branch with the Hyrax release number
#* <tt style="font-size: 1.1em; font-weight: bold;">git tag -a hyrax-<numbers> -m "Hyrax <number>"</tt>
#* <tt style="font-size: 1.1em; font-weight: bold;">git push origin hyrax-<numbers></tt>
#: '''NB:''' ''Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.''
 
==== Create the release on Github ====
# [https://github.com/OPENDAP/bes Goto the BES project page in GitHub]
# Choose the '''releases''' tab.
# On the [https://github.com/OPENDAP/bes/releases Releases page] click the 'Tags' tab.
# On the [https://github.com/OPENDAP/bes/tags Tags page], locate the tag (created above) associated with this new release.
# Click the ellipses (...) located on the far right side of the ''version-x.y.z'' tag 'frame' for this release and and choose ''Create release''.
#* Enter a ''title'' for the release
#* Copy the most recent text from the NEWS file into the ''describe'' field
#* Click '''Publish release''' or  '''Save draft'''.
#** If you have previously edited the release page you can click '''Update this release'''
 
=== Publish and Sign ===
 
When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is '''not''' the one we wish to publish because it requires people to have ''autoconf'' installed. Rather we want to use the result of "<tt>make dist</tt>" which will have the <tt>configure</tt> script pre-generated.
 
All you need do is build the tar file using <tt>make list</tt>, sign it, and push (or pull) these files onto www.opendap.org/pub/source.
 
# Go to the '''bes''' project on your local machine and run <tt>make dist</tt> which will make a bes-x.y.z,tar.gz file at the top level of the '''bes''' project.
# Use '''gpg''' to sign the tar bundle:
#: <tt>gpg --detach-sign --local-user security@opendap.org bes-x.y.z.tgz</tt>
# Use '''sftp''' to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
#: ''(Assuming your current working directory is the top of the '''bes''' project)''
#: <tt>sftp opendap@www.opendap.org</tt>
#: <tt>cd httpdocs/pub/source</tt>
#: <tt>put bes-x.y.z.tgz.sig</tt>
#: <tt>put bes-x.y.z.tgz</tt>
#: <tt>quit</tt>
# Check your work!
## Download the source tar bundle and signature from www.opendap.org.
## Verify the signature:
##: <tt> gpg --verify bes-x.y.z.tgz.sig bes-x.y.z.tgz</tt>
 
=== Get the DOI from Zenodo ===
# [https://zenodo.org Goto Zenodo]
# Look at the 'upload' page. If there is nothing there (perhaps because you are not ''jhrg'' or whoever set up the connection between the BES project and Zenodo) you can use the search bar to search for '''bes'''.
#: Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
# Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
# Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
# Also paste that into the README file. Commit using ''[skip ci]'' so we don't do a huge build (or do the build, it really doesn't matter that much).
 
'''Tip:''' ''If you are trying to locate the '''libdap''' releases in Zenodo you have to search for the string:'' <tt style="font-size: 1.1em; font-weight: bold;">libdap4</tt>
 
===== Images =====
[[File:Screenshot 2018-12-06 11.06.44.png|none|thumb|400px|border|left|Zenodo upload page]]

Revision as of 17:15, 22 March 2019

This pages covers the steps required to release the BES software for Hyrax.

We now depend on the CI/CD process to build binary packages and to test the source builds.

The Release Process

Tip: If, while working on the release, you find you need to make changes to the code and you know the CI build will fail, do so on a release branch that you can merge and discard later. Do not make a release branch if you don't need it, since it complicates making tags.

Verify the code base

  1. We release using the master branch. The code on master must pass the CI build.
  2. Make sure that the source code you're using for the following steps is up-to-date. (git pull)

Determine the scope of API/ABI changes in C++ sources

Determine the new software version (assuming you don't already know the extent of the changes that have been made)

For C++, build a file of the methods and their arguments using the command:
nm .libs/libdap.a | c++filt | grep ' T .*::' | sed 's@.* T \(.*\)@\1@' > libdap_funcs
and compare that using diff on the previous release's library.

Assess the changes you find based on the following rules for the values of CURRENT,REVISION, and AGE

  • No interfaces changed, only implementations (good): ==> Increment REVISION.
  • Interfaces added, none removed (good): ==> Increment CURRENT, increment AGE, set REVISION to 0.
  • Interfaces removed or changed (BAD, breaks upward compatibility): ==> Increment CURRENT, set AGE and REVISION to 0.

The current value of CURRENT,REVISION, and AGE can be found in configure.ac:

LIB_DIS_CURRENT=14
LIB_DIS_AGE=6
LIB_DIS_REVISION=1

Update Release Files

Once you have determined the new values of the CURRENT:REVISION:AGE strings then:

  • Edit the configure.ac and update the version values to the new ones.
  • Update the text documentation files and version numbers in the configuration files:

Update the ChangeLog file.

Use the script gitlog-to-changelog (which can be found with Google) to update the ChangeLog file by running it using the --since="<date>" option with a date one day later in time than the newest entry in the current ChangeLog.

gitlog-to-changelog --since="1970-01-01"
(Specify a date one day later than the one at the top of the existing ChangeLog file.)

Save the result to a temp file and combine the two files:

cat tmp ChangeLog > ChangeLog.tmp; mv ChangeLog.tmp ChangeLog

If you're making the first ChangeLog entries, then you'll need to create the ChangeLog file first.
Tip: When you're making the commit log entries, use line breaks so ChangeLog will be readable. That is, use lines < 80 characters long.

Update the NEWS file

To update the NEWS file, just read over the new ChangeLog entries and summarize.

Update the Version Numbers for Humans

  1. Determine the human version number. This appears to be a somewhat subjective process.
  2. Edit each of the Affected Files and update the human version number.
Affected Files
configure.ac
*.spec (In the BES it's bes.spec.*)
debian/changelog (see [1] Debian ChangeLog)
.travis.yml (for BES only)
NEWS
README.md (Add the libdap version the README.md)
INSTALL

It's helpful to have, in the NEWS file and the Web site and the release notes, a list of the Jira tickets that have been closed since the last release. The best way to do this is to goto Jira's Issues page and look at the Tickets closed recently item. From there, click on Advanced and edit the time range so it matches the time range since the past release to now, then Export that info as an excel spreadsheet (the icon with a hat and a down arrow). YMMV regarding how easy this is and Jira's UI changes often.

Update the RPM dependencies

In the RPM .spec file, update the dependencies as needed.

Affected Files
*.spec (In the BES it's bes.spec.*')

Update the internal library version numbers

There are really 2 version numbers for each of these project items. The human version (like bes-3.17.5) and the library API/ABI version which is represented as CURRENT:REVISION:AGE. There are special rules for when each of the numbers in the library API/ABI version get incremented that are triggered by the kinds of changes that where made to the code base. The human version number is more arbitrary. So for example, we might make a major API/ABI change and have to change to a new Libtool version like 25:0:0 but the human version might only change from bes-3.17.3 to bes-3.18.0

The rules for shared image version numbers:

  1. No interfaces changed, only implementations (good): Increment REVISION.
  2. Interfaces added, none removed (good): Increment CURRENT, increment AGE, set REVISION to 0.
  3. Interfaces removed or changed (BAD, breaks upward compatibility): Increment CURRENT, set AGE and REVISION to 0.

See How to see the scope of API/ABI changes in C++ sources below for gruesome details. Often basic knowledge of the edits is good enough.

Affected Files
configure.ac

For the BES HDF4/5 modules (BES only)

  1. Goto those directories and update the ChangeLog, NEWS, README, and INSTALL files (even though INSTALL is not used by many).
  2. Update the module version numbers in their respective Makefile.am files.
  3. Commit and Push these changes.

Commit Changes

  1. Commit and push the BES code. Wait for the CI/CD builds to complete. You must be working on the master branch to get the CD package builds to work.

Tag & Release

Tag the BES code

  1. Tag the bes code using command line git in your local (up-to-date) bes project
    • git tag -a version-<numbers> -m "Version <number>"
    • git push origin version-<numbers>

  2. If this is part of a Hyrax Release, then tag this point in the master branch with the Hyrax release number
    • git tag -a hyrax-<numbers> -m "Hyrax <number>"
    • git push origin hyrax-<numbers>
    NB: Instead of tagging the HDF4/5 modules, use the saved commit hashes that git tracks for submodules. This cuts down on the bookkeeping for releases and removes one source of error.

Create the release on Github

  1. Goto the BES project page in GitHub
  2. Choose the releases tab.
  3. On the Releases page click the 'Tags' tab.
  4. On the Tags page, locate the tag (created above) associated with this new release.
  5. Click the ellipses (...) located on the far right side of the version-x.y.z tag 'frame' for this release and and choose Create release.
    • Enter a title for the release
    • Copy the most recent text from the NEWS file into the describe field
    • Click Publish release or Save draft.
      • If you have previously edited the release page you can click Update this release

Publish and Sign

When the release is made on GitHub the source tar bundle is made automatically. However, this bundle is not the one we wish to publish because it requires people to have autoconf installed. Rather we want to use the result of "make dist" which will have the configure script pre-generated.

All you need do is build the tar file using make list, sign it, and push (or pull) these files onto www.opendap.org/pub/source.

  1. Go to the bes project on your local machine and run make dist which will make a bes-x.y.z,tar.gz file at the top level of the bes project.
  2. Use gpg to sign the tar bundle:
    gpg --detach-sign --local-user security@opendap.org bes-x.y.z.tgz
  3. Use sftp to push the signature file and the tar bundle to the /httpdocs/pub/source directory on www.opendap.org
    (Assuming your current working directory is the top of the bes project)
    sftp opendap@www.opendap.org
    cd httpdocs/pub/source
    put bes-x.y.z.tgz.sig
    put bes-x.y.z.tgz
    quit
  4. Check your work!
    1. Download the source tar bundle and signature from www.opendap.org.
    2. Verify the signature:
      gpg --verify bes-x.y.z.tgz.sig bes-x.y.z.tgz

Get the DOI from Zenodo

  1. Goto Zenodo
  2. Look at the 'upload' page. If there is nothing there (perhaps because you are not jhrg or whoever set up the connection between the BES project and Zenodo) you can use the search bar to search for bes.
    Since the libdap, BES and OLFS repositories are linked to Zenodo, the newly-tagged code is uploaded to Zenodo automatically and a DOI is minted for us.
  3. Click on the new version, then click on the DOI tag in the pane on the right of the page for the given release.
  4. Copy the DOI as markdown from the window that pops up and paste that into the info for the version back in Github land.
  5. Also paste that into the README file. Commit using [skip ci] so we don't do a huge build (or do the build, it really doesn't matter that much).

Tip: If you are trying to locate the libdap releases in Zenodo you have to search for the string: libdap4

Images
Zenodo upload page