Skip to content

Release Steps Jobs On Jenkins

yishayw edited this page Jul 25, 2021 · 11 revisions

The Release Steps jobs on Jenkins are an attempt to break the creation of release artifacts into several smaller steps and only require that one computer (the Jenkins server) be configured to create the release artifacts. The Release Manager must still be configured to VERIFY that the release artifacts are correct.

Historically, only one person has been able to run the releasecandidate.xml scripts successfully. Others reported issues with network connections and unreliable upload and download of artifacts and dependencies. So, having one machine configured that we know will work should help others become release managers and not have to figure out how to configure their machines and networks.

Verification

The purpose of verification is to convince the RM that it is safe to GPG sign the artifacts created on a shared server like the CI Server. This is done by having the release scripts create JARs and SWCs that are binary identical no matter what computer builds the binary (Mac or Win). This assumes that the RM has the same versions of Java and Ant (for sure) and probably Maven.

Current versions are:

  • Java: 1.8.0_152
  • Ant: 1.9.9
  • Maven: Apache Maven 3.8.1

The Ant source package is verified in a slightly different way. The verification script clones the tags from the 3 repos onto the RM's computer and builds its own source package. The result will not binary match what the CI server generated because making it match would require running a Maven plug-in and we don't want to require that everyone who wants to build a release package (for nightly builds and other reasons) have both Ant and Maven. We only want to require that of RMs for now. So the verification script runs the Maven plugin on both the CI-generated Ant source package and the RM's generated source package and verifies that they are binary identical.

Security

One reason for so many release steps is that we do not want to store any credentials on the CI Server, so every time there is something to be pushed to Git, the release step has to end and request that the RM login to the CI Server and enter their Github username and password. In order to use 2FA authentication user has to generate token and use it as password.

Similarly, when GPG signatures are required, the RM must download the artifacts to be signed and do the signing locally. The RM's keys should never be transferred/copied to the CI Server

Job Details

The various jobs all end by sending an email to dev@royale.a.o if successful. If the emails are not clear, please improve them. The template for the emails are sort of hidden in a Jenkins job configuration. Go to "Post-Build Actions" at the bottom of the config page, then choose "Advanced Setting", then find the "Success" email and choose "Advanced".

Fail and Revert

If you fail on comparing artifacts you will probably have to revert all the changes which has been done by couple of previous steps. Changes has to be reverted on created branch for release purposes ex. release/0.9.6

  1. If you fail on comparing artifacts on your machine in step Release Step 003 you should revert everything what has been pushed during perform of steps from Release Step 001 If Utils to Release Step 003
  2. If you fail on comparing artifacts on your machine in step Release Step 007 you should revert everything what has been pushed during perform of steps from Release Step 005 to Release Step 007

Release Step 001

This step creates the release branch for royale-compiler from the develop branch using Maven. Maven updates the develop branch to the next version number which is why it needs a "git push". Then other release steps will modify the release branch so the email at the end of this step requests that the RM run two git commands to set up remote origin tracking of the release branch.

Release Step 001 If Utils

The "Utils" are the royale-compiler projects in the compiler-jburg-types and compiler-build-tools. These two projects are used to process other royale-compiler source code. Due to the way Maven works, you can't have some sub-projects in a Maven build be tools for other sub-projects, so if you need to compile the source in compiler-jburg-types and/or compiler-build-tools with Maven you have to first run the main pom.xml with the "utils" profile, which just builds those two sub-projects, then run the main pom.xml file with the "main" profile to build the rest of the projects. (Another option would be to move these two projects to a different repo and release them separately).

Fortunately, the code in the utils projects rarely changes and they have a different version numbering scheme so we don't have to update their version numbers on each release and thus re-release the utils projects with every release.

In theory, if the code in the utils projects changes, the developer making the change will update the version of the utils to a next SNAPSHOT version. Then if you skip the utils steps, one of the steps should fail because it detects a SNAPSHOT version still in a POM. And that will remind you to run the "utils" release steps.

This step changes the SNAPSHOT versions in the POM to non-SNAPSHOT versions and commits them. The RM must then "git push" them to the remote server.

FWIW, there is a way to use Maven's release:prepare to change the SNAPSHOT versions, but it was easier to automate using Maven's set-versions and set-property step for this different release versioning scheme.

Release Step 002

This step runs Maven's release:prepare step on the royale-compiler release branch. This step changes the other royale-compiler project's SNAPSHOT versions to non-SNAPSHOT versions and then tags that with the RC number. Maven does not rely on a repo's HEAD in a release branch, it builds the release artifacts from a tag.

Release Step 002 If Utils

This step runs Maven's release:perform step on the utils projects in royale-compiler. This results in release-able artifacts for compiler-build-tools and compiler-jburg types.

Release Step 003

This step runs Maven's release:perform step on the other royale-compiler projects. When this step completes, and email is sent describing the set of steps in the Ant script releasesteps.xml (in the royale-compiler repo) to run to verify that the artifacts the CI server built are correct. This involves downloading the Maven source and binary artifacts into a temporary folder, building the source artifact and comparing the resulting binary artifacts with the CI-built binary artifacts. Because the scripts use a temporary folder, you can run the releasesteps.xml in your local working copy of the royale-compiler repo. If the artifacts are bit-for-bit the same, then the RM should feel safe PGP signing those artifacts. One Ant target in releasesteps.xml will sign all of the artifacts. Another will upload the signed artifacts to The ASF Nexus staging repository.

Release Step 004

This step creates the release branch for royale-typedefs from the develop branch using Maven. Maven updates the develop branch to the next version number which is why it needs a "git push". Then other release steps will modify the release branch so the email at the end of this step requests that the RM run two git commands to set up remote origin tracking of the release branch.

Release Step 005

This step updates some properties in the pom.xml to use the release version of the compiler jars, and set the date for all SWCs so they will be bit-for-bit the same on any computer that builds them. I recommend using the current date and then picking a time that represents the release number, so 9:06 for the 0.9.6 release. And 1:00 for the 1.0.0 release. The date and time isn't really that important.

Release Step 005 If Utils

This step updates a property in the pom.xml if the royale-compiler utils version is changing.

Release Step 006

This step runs Maven's release:prepare step on the royale-typedefs release branch. This step changes royale-typedefs project's SNAPSHOT versions to non-SNAPSHOT versions and then tags that with the RC number. Maven does not rely on a repo's HEAD in a release branch, it builds the release artifacts from a tag.

Release Step 007

This step runs Maven's release:perform step on the royale-typedefs projects. When this step completes, and email is sent describing the set of steps in the Ant script releasesteps.xml (in the royale-typedefs repo) to run to verify that the artifacts the CI server built are correct. This involves downloading the Maven source and binary artifacts into a temporary folder, building the source artifact and comparing the resulting binary artifacts with the CI-built binary artifacts. Because the scripts use a temporary folder, you can run the releasesteps.xml in your local working copy of the royale-typedefs repo. If the artifacts are bit-for-bit the same, then the RM should feel safe PGP signing those artifacts. One Ant target in releasesteps.xml will sign all of the artifacts. Another will upload the signed artifacts to The ASF Nexus staging repository.

Release Step 008

This step creates the release branch for royale-asjs from the develop branch using Maven. Maven updates the develop branch to the next version number which is why it needs a "git push". Then other release steps will modify the release branch so the email at the end of this step requests that the RM run two git commands to set up remote origin tracking of the release branch.

Release Step 009

This step updates some properties in the pom.xml to use the release version of the compiler jars, and set the date for all SWCs so they will be bit-for-bit the same on any computer that builds them. I recommend using the current date and then picking a time that represents the release number, so 9:06 for the 0.9.6 release. And 1:00 for the 1.0.0 release. The date and time isn't really that important.

Release Step 010

This step runs Maven's release:prepare step on the royale-asjs release branch. This step changes the royale-asjs project's SNAPSHOT versions to non-SNAPSHOT versions and then tags that with the RC number. Maven does not rely on a repo's HEAD in a release branch, it builds the release artifacts from a tag.

Release Step 011

This step runs Maven's release:perform step on the royale-asjs projects. When this step completes, and email is sent describing the set of steps in the Ant script releasesteps.xml (in the royale-asjs repo) to run to verify that the artifacts the CI server built are correct. This involves downloading the Maven source and binary artifacts into a temporary folder, building the source artifact and comparing the resulting binary artifacts with the CI-built binary artifacts. Because the scripts use a temporary folder, you can run the releasesteps.xml in your local working copy of the royale-asjs repo. If the artifacts are bit-for-bit the same, then the RM should feel safe PGP signing those artifacts. One Ant target in releasesteps.xml will sign all of the artifacts. Another will upload the signed artifacts to The ASF Nexus staging repository.

At this point, the Maven artifacts are complete, and the Nexus staging repository should be "Closed". Instructions are in the email. If some new artifact was added to the release and not signed then Nexus will report an error when closing the repo, and the appropriate releasesteps.xml scripts should be updated.

Release Step 012

This step is the first step to prepare the Ant artifacts that go on dist.a.o. The ASF dist.a.o uses Subversion not Git, so this step checks out the Royale folder from dist.a.o and prepares a folder to store the release artifacts.

Release Step 013

This step creates the Ant artifacts that go on dist.a.o. It checks out the develop branch just to get an Ant script for building, but doesn't use other source. It grabs the source packages from Maven and uses those instead. When this step completes, the email sent out will describe how to verify those artifacts and sign them and upload them to dist.a.o

Release Step 014

This step generates the [Vote] thread

Release Step 015

This step generates the [Discuss] thread

Troubleshooting

If one of your step fail with following stack trace:

> > Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 52.191.135.173/52.191.135.173:63013 > > at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) > > at hudson.remoting.Request.call(Request.java:202) > > at hudson.remoting.Channel.call(Channel.java:954) > > at hudson.FilePath.act(FilePath.java:1161) > > at

It is highly possible that you need to restart Jenkins slave agent on our Remote machine. Close opened application and double click on slave-agent app placed on our desktop.

If for some reason you did not get the 'SUCCEEDED' email after the completion of step XX, you can find the template for that email under

http://apacheroyaleci2.westus2.cloudapp.azure.com:8080/job/Royale_Release_Step_0XX/configure -> Post-build Actions -> Advanced Settings... -> Triggers -> Success -> Advanced...

Future Improvements

Uploading failures

The current steps still require that the RM be able to upload a lot of artifacts to the Nexus Staging Repository and Apache's dist.apache.org. If folks still have issues with uploads, we can alter the scripts to upload only the PGP signature files from the RM's computer and upload the other artifacts from the CI Server

However, it should be possible to "retry" the uploads by re-running the upload steps locally.

Clone this wiki locally