or: How we learned to stop worrying and ship software every day

Just one year ago we were releasing once or twice per month and this September we’re at 55 releases a month. All with the same development team. What does this mean to our customers? They get updates from us every day. Sometimes every hour, sometimes more than once an hour. What does that mean to us? No iterations, no useless stalling, no situations where one ticket breaks something and is poisoning the whole branch.

What was holding us back? Premature integration to the master branch turned out to be the culprit. Let me provide some context about our previous branching model and ticket workflow. For those eager ones, you can skip to the solution.

Old Workflow

In terms of code versioning and servers we had a few predefined branches that also matched QA servers - a production (prod) and a development (dev) branches that would be deployed on respective qa servers (prod-qa and dev-qa). We would patch production with things tested on prod qa-server and do feature development and non-high priority bugs in dev.

In terms of ticket flow, our development team works (this has not changed in the new process) according to these guidelines:

Problem

Major problem with such workflow for us was few tickets poisoning the whole batch. Once ticket got merged to dev, if it turned out that it was not good to go, it was keeping the whole batch until some action was taken on it - either fix the issues or revert the poisonous code.

This was severe, given that Assembla employs people from all over the world and we are big on asynchronous development. Possible “easy solutions” for the problem:

1. Have very thorough specifications.
2. Have a rigid peer review process
3. Have a release manager who maintains the dev branch and either fixes or reverts the poisonous code.

All of these do not work, or are short term solutions, or bring slowness as side effects. In essence, these solutions scale poorly. Is there a solution for this problem that scales well?

Solution

Lets analyze this Cardwall view. Can you spot the problem?

There are tickets to be released, but the whole batch can not be released because some of the tickets are not ready. However, some (12) of them are ready. These tickets are not even related, but Ticket #15711 (as well as some of the tickets in Accepted/Review/Test, if they have code committed to master) stop us from releasing all the tickets in Deploy. Recognizing this leads us to the most important insight - every action, that can put a ticket back in status must happen before integration to master. We decided to change just this one thing in our process. Master integration must happen at the last possible step. The result...Did we show you the graph?

The later the merge, the better - this means that issues will be caught before the integration most of the time. Have you heard of those companies, that do not let their developers go home until master build is green? Have you stayed late fixing someone else’s code after they are not there and their code is failing user acceptance testing? Yeah, forget about those ruined evenings.

We’ll continue to write about our development workflow and how we managed to move integration to be the last step of the process on this blog, so if you read until here, you might as well subscribe to our blog.

See how Assembla can help you with Continuous Delivery.