Continuous Integration without SNAPSHOTs

Maven includes a feature called SNAPSHOT which allows you to tell Maven that a particular artifact is a “work in progress”, not a finished version.  This also acts as a hint to let Maven know that it should pick up the newest version of artifacts referred to as SNAPSHOTs.  This is all very useful, but it can present some challenges in a Continuous Integration environment.

When we are automating our builds and testing with CI, one thing that we really want is deterministic builds.  Said another way, we want to be able to run a particular job again and get the same result.  SNAPSHOTs introduce an element of non-determinism to the build – artifacts with SNAPSHOT versions will resolve to the latest snapshot version that happens to be available when the build is run, and they all have the same ‘version’ – snapshot versions look like ‘2.1.2-SNAPSHOT’.

So this means that SNAPSHOTs and CI do not play well together.

But there is a relatively simple way to get the behavior we want without using SNAPSHOTs.  To be clear on that, the behavior we want is this:

• whenever we do a build, we want to pick up the latest available version of the (in development) dependencies,
• we want to know which particular version/build of a dependency we actually used in any given build,
• we want to be able to repeat the build and get the same outcome.

For the purpose of discussion, let’s assume our source code is in git, we build our projects with Maven, perform CI on a Hudson server, and we publish our artifacts to Artifactory.

So let’s consider two projects that are both under active development at the same time – i.e. they are both changing day to day:

• super-common, version 2.0.1– this contains common code that is used by a number of applications,
• super-webapp, version 1.0.0 – this is a ‘Super’ web application that we are building, it depends on super-common.

Let’s look at super-common first, since it is at the bottom of the dependency tree.

We need to do two things to make this work:

First, in the POM, set the version to something like 2.0.1-CIVERSION – this creates a placeholder that we will later replace with a unique identifier made up from the timestamp and the build number.

Then, update the Hudson job as follows:

Add a new Execute Shell build step at the beginning of the job (before the Maven step) and set it up as follows:

export DATE=`date +"%y%m%d.%H%M"`
export BUILDVERSION="$DATE.$BUILD_NUMBER"
sed -i -e "s#2.0.1-CIVERSION#2.0.1-$BUILDVERSION#g" pom.xml

Install the Maven3-Artifactory integration Hudson plugin (from Manage Hudson/Plugins) if you do not already have it.  Then edit your job and select the chcekbox for Maven3-Artifactory Integration.  Select your Artifactory server and repository (it should be a repository that holds releases, not snapshots) and make sure you select two options – Deploy artifacts to Artifactory and Capture and publish build info.

Finally, make sure your Maven step’s goals includes the Maven install goal, and does not include the Maven deploy goal.  This will ensure that the maven-metadata.xml is generated/updated, but will allow the Mavn2-Artifactory plugin to control the deployment to Artifactory, rather than Maven trying to do it.

Note: You do not need and should not have a distributionManagment section in you project POM.

Now, when your job runs, it is going to update the version number in your POM to include the time and the Hudson build number.  If you browse your Artifactory repository, you will find each build creates a new artifact.  So this solves the first part of the problem – now we have a reliable way to describe which particular version/build of the artifact we want to use when we are expressing a dependency.

Now, let’s look at super-webapp.

Here we have a dependency on super-common.  In your POM, go ahead and put in any valid version number – just copy the latest one from Artifactory for example.   We will see how we update this later.

Next, make sure you have the scm entry in your POM, and in particular its developerConnection element is specified.

Now, we go to the Hudson job that builds super-webapp.  You can, of course do the same things we did above so that we have unique builds of super-webapp as well.  But for now, let’s just focus on consuming the latest super-common.

In the Maven step in the build, update the goals to look like this:

clean versions:use-latest-versions 
      scm:checkin -Dmessage="update versions" -DpushChanges 
      install

Of course, you can add any others that you need.  Here we are using the Maven ‘Versions’ Plugin to go find the latest available version of the dependencies and to update the POM for us.  Without any further configuration, it will check and update all dependencies.  If this is not what you want, you can add configuration to the build section with include and exclude lists, like most Maven plugins.  Check out the documentation for more details.

Now, this is going to change our POM – so we need to make sure we don’t lose that change – or we still wont be sure what versions of dependencies were used to build this version of super-webapp.  So we need to commit this change back to our version control system.  The scm:checkin goal takes care of that for us.  Of course, we need to provide a commit message, and then we also define pushChanges to tell it to not just commit, but also push to the upstream repository, otherwise we would lose it when the workspace is replaced during the next execution of the build.

Finally, we have install for the same reason as before.

When this job runs, Maven is going to go to Artifactory and find the latest version of super-common, update the POM for super-webapp with that version, commit that change to git, and then proceed with the build.

So now we have what we wanted.  We have no SNAPSHOTs, but each build picks up the latest available version of its dependencies, and we know exactly what version was used (we have the git revision ID and the build number), and we can rerun the exact same build if we wanted to.  Of course, to do that we would have to take the use-latest-versions out of the Maven goals so that it does not grab newer dependencies.

This is a much more CI-friendly approach than using Maven snapshots.

I am indebted to Adam Dent and Leon Franzen for parts of this approach.