Advertisement

Continuous Delivery is for Open Source Too

This is a guest post from Stephen Connolly, CloudBees, Inc.

 

If you start to push for Continuous Delivery in the context of an open source project, you will often get push-back such as:

“One of the nice things about Open Source projects is that, well, it’s NOT work, it’s NOT a job.”

and

“[Open source projects] aren’t factories, churning out releases and code like widgets. We do that enough at our day-jobs.”

These are valid arguments. But they miss some key points.

  • People write commercial software for money.
  • People write open source software for passion.

There is still a transaction going on. It may not be one you can see or account for on a traditional balance sheet… but you can still measure it.

Passion is measured in engagement. When an open source project has lots of passion there are lots of contributors and a vibrant community is built up around it. When an open source project runs out of passion it becomes the tumbleweed of the open source forges: A repository with the last commit five years ago and nobody can checkout and build the code any more.

“If you have an apple and I have an apple and we exchange apples then you and I will still each have one apple. But if you have an idea and I have an idea and we exchange these ideas, then each of us will have two ideas.” – George Bernard Shaw

Passion does not work the same way as money does.

If I spend my passion developing a feature for an open source project and the community grows more passionate because of that feature, then my passion is restored and re-invigorated as part of the community… there is more passion after than before.

If I spend my passion developing a feature for an open source project and it gets dropped on the floor and ignored, however, my passion was wasted.

When you feel passionate for an open source project, then it is your responsibility to help grow the passion of that project’s community.

There are things we can do to help grow our project’s community and passion, one of those ways is to get releases out more often.

Leaving unreleased features / fixes in your source repository burns the passion of those members of the community who need those features / fixes – and of those who created them. The longer those changes stay unreleased, the more passion you are losing. To put it another way:

Continuous delivery – not just for your day job

Your day job wants continuous delivery because the new features / fixes are like inventory left half finished on the shop floor… it’s consumed resources but still cannot be sold. It’s overhead.

Your open source project wants continuous delivery because the new features / fixes are like inventory left half finished on the shop floor… it’s consumed community passion rather than growing it.

Now of course these the passion balance sheet works differently than a fiscal one… but there are some things that stay the same:

  • Continuous delivery is about making sure that improvements are delivered as efficiently as possible. Not as fast as possible because then you are cutting corners and perhaps lowering quality (which will cost you money / passion).
  • Continuous delivery becomes easier if you can automate – as much as possible – the early parts of the process. For most processes, the expensive parts are the human parts. In some sense the commercial projects have easier accounting here… it is easy to measure the fiscal cost of human processes, e.g. using time and motion studies. It is very difficult to measure the passion cost of human processes on volunteers in your community… but our intuition tells us that the cost is not small.
  • We maximise our return by getting new features / fixes out as early as possible.

The decision to run a release through your continuous delivery pipeline has a cost.

A good pipeline is almost completely automated with a few sanity manual checks at the penultimate step. These pipelines are therefore low cost… both in fiscal and in passion accounting terms.

A bad pipeline has lots of manual checks and steps scattered throughout. These pipelines are high cost… and if prone to failure, the cost can grow while you fight to get a release all the way through to the end.

You need to weigh the expected cost of executing your pipeline against the expected net gain you will get by having the new release delivered. Once that measure gives a net gain in your respective accounting metric… then it is time to cut a release.

Every situation is different, so it is no surprise that the balance will play out differently for different projects.

jenkins

One of the projects I am involved with is the Jenkins CI project. When it started out initially there were releases as often as necessary. There were some occasions where the project got a new release several times during the same day as Kohsuke added a new feature and then people found critical bugs in the new feature. This very fast release cadence was great for getting people engaged with the project… but it came with a cost… people just could not keep up to date with the releases. After a while, Kohsuke settled down to a weekly release cadence. There may not be a release every single week (for example, Kohsuke may not have access to cut a release while travelling abroad) but on average about 48 weeks of the year there is a release.

Is that weekly release schedule right for every project? No, in fact it isn’t even right for everyone in the Jenkins community. If we look at the usage statistics there are a lot of people using the newer versions. There is also a significant group of people in the community who don’t want to live on the bleeding edge but still need bug fixes. The solution is a second, slower train of releases in what is called the Jenkins LTS release line.

The fact that there are regular releases with the Jenkins project has, in my view, helped to grow the community. If somebody finds a bug in Jenkins and provides a patch to fix that bug, there is a very strong possibility that the patch will be applied in the next week’s release.

★★★★★ Excellent: I would contribute again

★★★★

★★★☆☆

★★☆☆☆

☆☆☆☆ Poor: I would discourage others from contributing

Part of what makes the weekly releases work for Jenkins is that Kohsuke has heavily automated the release process.

 maven

Another project I am involved with is Apache Maven. On the Maven project, our current release process has a lot of manual steps. If we count the steps for releasing Maven itself… I lost count after at least 26 steps. Another issue is that, for legal reasons, the Apache board requires that Apache projects have to vote on releases. So each PMC member casting a vote on releasing has to

“…download the signed source code package, compile it as provided, and test the resulting executable on their own platform, along with also verifying that the package meets the requirements of the ASF policy on releases.” – http://www.apache.org/dev/release.html#approving-a-release

In short, we burn a lot of passion getting a release out the door. The result of that is that it is probably not currently possible for Apache Maven to maintain a release cadence faster than about once every 4-8 weeks… so we are looking into automating as much of the process as we can. It may be that Apache Maven will never have a faster cadence than once every 4-8 weeks… but making it cheaper to release at that cadence can only help grow the community passion.

Continuous delivery is more a collective state of mind than any one, set process or technology. If a project releases at all then it is practicing delivery. To see the benefits of continuous delivery, you just need to up your cadence. Things to look out for are:

  • Manual tasks that can be moved later in the process
  • Automated tasks that can be moved earlier in the process
  • Automated checks that can be created / moved as early as possible in the process
  • Redundant tasks and checks that can be removed entirely

 

Upping your cadence will come with some pain. The point is to identify the causes of the pain and reduce or remove them. If, at a later time, you decide the higher cadence of continuous delivery is not for you, the reduced pain of releasing will still remain as a benefit.

 jenkins-promotions

One example is a set of Jenkins jobs that I set up to help cut releases of Apache Maven. They are:

  • There is an initial job. This job is responsible for determining if there are any changes to the code since the last release. If there are changes then it triggers an evaluation of those changes.
  • There is a build job. This job is responsible for determining if an exact GIT hash identified by the initial job builds and passes all the unit tests. If that job passes, then it triggers an integration test
  • Finally the test job takes the distribution of Maven built by the build job and runs that through the integration test suite.

If the downstream jobs are successful, they add a badge to the initial job that kicked them off. When a build has all the badges, then an email is sent to the Maven developers list to notify them that there is something that could be released. This actually removes about 10 of the 26+ steps from the current Maven core release process.

If you want to say you are doing continuous delivery I would suggest you aim for the following levels:

Level 0: A new feature / confirmed fix of a bug committed to source control waits no longer than 3 months before a release with that new feature / fix is available. If you cannot meet this target don’t even try to claim your processes are continuous delivery

Level 1: A new feature / confirmed fix of a bug committed to source control waits no longer than 1 month before a release with that new feature / fix is available.

Level 2: A new feature / confirmed fix of a bug committed to source control waits no longer than 1 week before a release with that new feature / fix is available.

Level 3: A new feature / confirmed fix of a bug committed to source control waits no longer than 1 day before a release with that new feature / fix is available. You are a continuous delivery evangelist, people may hate you. You may need processes to support those in your community who cannot handle such a fast cadence.

Summary

Many open source projects have explicitly moved to time-based mostly fixed cadence releases. It may not be right for your project, but you cannot know until you try.

Experimenting with continuous delivery can help your project and may significantly grow your community’s passion… even if you don’t end up using continuous delivery after an experiment!

 

—Stephen Connolly
CloudBees, Inc.
www.cloudbees.com

sconnolly

Stephen Connolly has over 20 years experience in software development. He currently works for CloudBees, Inc as an Elite developer and architect. He is involved in a number of open source projects, including Jenkins. Stephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland – where the weather icons are particularly useful. Follow Stephen on Twitter and on his blog.

4 Responses to “Continuous Delivery is for Open Source Too”

  1. Paul Apr 6, 2014 at 8:03 am #

    Does the Sourceforge platform make any features available to project admins to help with this? Releasing on Sourceforge is not the simplest task! I need to:Use FTP to release the website (using FileZilla in my case)From the Files webpage, create a new folder, upload the release file and make it the default Go to each Ticket Tracker’s admin page and add the new version number to the listetc I don’t know the best way of automating this yet. If the Sourceforge platform  has a good API, then perhaps some of this could be scripted which would lower the barrier to do a release – at the moment, it;s just a lot of manual work!!

  2. ctsai Apr 15, 2014 at 9:40 am #

    @Paul  Hello, We do have some features that can help automate releases. For example, we offer uploads via SFTP (which you already use), SCP, and rsync over SSH. The latter two in particular are well suited for incorporating into some sort of a release script (I personally like rsync the best for this). See our documentation here: https://sourceforge.net/p/forge/documentation/FileManagement/ Note that you can use those to upload to both the project web space as well as the file release system. As for making a file default, we have some instruction on doing that via an API here: http://sourceforge.net/p/forge/community-docs/UsingtheReleaseAPI/ For tickets, it’s not currently possible to change admin settings via an API, but we do have an API that can be used to interact with tickets, wiki pages, etc. http://sourceforge.net/p/forge/documentation/AlluraAPI/ Hopefully this information helps, please let us know how else we can advise. If you can elaborate on the “etc”, perhaps I can provide some insight on that as well. Regards, Chris Tsai, SourceForge.net Support

  3. Paul Apr 21, 2014 at 2:35 pm #

    ctsaiThanks Chris, that’s helpful. I’m documenting my release procedure here: https://sourceforge.net/p/utplsql/wiki/ReleaseProcedure/ This is very a much a “work in progress” – I’ve only done one release so far and will no doubt refine the procedure each time I do it.

  4. Paul Apr 25, 2014 at 5:20 pm #

    ctsai(I replied earlier, but that comment appears to have gone missing). Thanks for the links, I’ll take a look. My release procedure is at: https://sourceforge.net/p/utplsql/wiki/ReleaseProcedure/ though it’s very much a “work in progress” and will get refined over time.