Continuous delivery is hot. Facebook recently announced that they are releasing changes twice per day, and every week they release code from 600 engineers. Even if you don’t have the same resources as Facebook, you can have a release process that is just as fast and effective. Six months ago, we were releasing once every four weeks, and as our system got more complicated, the release cycle got slower, and we spent more time patching production bugs after each release. Now we release every day, or many times per day, with fewer bugs. It’s a vast improvement. It didn’t come naturally. We had to figure out the process secret, which I can now share with you.
The secret ingredient is the location where you test your new code prior to release.
A typical software release process looks like the diagram below. Developers have been using this pattern for at least 50 years. They merge all of their changes together, and then they test the “release candidate” to make sure everything works. They iterate this process until they have a release candidate that looks bug free. Then, they release.
This has the advantage of using one test system and presenting one place for testers to test. So, it makes sense if your test systems or testers are in short supply.
When you merge everything together, you can test it to make sure that one set of changes does not cause a problem with a different set of changes. This sounds like a good idea, but we shall see later that this is not a real advantage and doesn't leave you with clear guidance about who should fix the problem.
This pattern also creates some obvious scaling problems. If you have a lot of changes (our goal with scalable agile is to increase the velocity of changes) then it can take a long time to test everything. I suspect that it would take months to test changes from 600 engineers. Contributors with simple or reliable changes will be held back from release by problems from any source. Often there is contention for space in the build and test process, with some changes held out while an earlier set is processed.
The Continuous Delivery test pattern
A continuous release process uses a different test pattern, as diagrammed below. Each contributing team or developer has a separate test system that is a lightweight clone of the production system. They merge their changes with a copy of the current production code, test, and when everything is working, merge into the production version.
If you use this process, you can release anytime. It’s also much more scalable, since the testing doesn’t get more complicated when more teams are contributing more changes. In the extreme case, 600 engineers can release twice per day instead of once per quarter.
This procedure requires many test systems. That’s probably why it’s only now becoming popular as on-demand cloud systems have become more readily available. One of our biggest challenges was creating an on-demand test environment. However, this accelerates our work because developers can get test systems on-demand, rather than waiting for a slot in a shared QA environment.
The next picture shows the process in more detail.
- You will want to do code review for two reasons. First, you will need a place to ask for tests that your continuous integration system will require. Second, you should have a buddy test any code because it will go straight through to release. I have illustrated a code review phase within a feature team before going into the team-level test environment.
- It makes sense to release a small number of changes at a time. The merge pattern above shows that a contributor will grab a copy of the production code, test it with his/her/its changes and put it into the production version.
This pattern just avoids the situation where you find a problem in the way that two sets of changes work together, and then try to figure out who is responsible for a fix. You only test changes against the current production system, not combined with other changes. On the diagram, I have labeled the merges in order, and you can see that if Contributor 1 moves a change through to release, then Contributor 2 is clearly responsible for figuring out any problems result from changes that are added later.
There is a variation on this pattern in which you test in production. This is practical if you have a high-volume Web site, and it is used by Github and probably by Facebook. In this variation, the branch (labeled "test system X") gets deployed to production and monitored. If it works well, THEN it gets merged to the master or trunk or production revision. Obviously, testing in production is tricky. However, the advantage of this scheme is that the production revision is cleaner (and better for other developers and rollbacks) because it's been through the final test - a production deployment.
The continuous delivery pattern leads to a change in the role of QA professionals. With the centralized “release candidate” system, the QA team can be a gatekeeper. The QA team can be responsible for saying when the release candidate is ready. And, the QA team only needs to look in one place. In the new continuous pattern, a QA professional is more of a consultant. Each contributing developer or team can call on a QA professional to test changes in any localized environment.
In every continuous release process that I have seen, a developer, not a QA professional, is responsible for approving the release. Why? Because only developers can quickly fix problems that slip through to production. They are incentivized to release with high quality by the following bargain: “If you approve a change for release, and we send it out, and there is a problem, you will instantly come back from Friday night beers to fix it.”
In almost every successful continuous release process, the “QA” box is partly occupied by automated tests, written by developers, that run “continuous integration” when each set of changes is merged. For example, we configured Jenkins to run tests on each change and “downvote” changes that don’t pass all tests. I’ll write about continuous integration in a future article.
You can run this process without any QA professionals, if your developers are sufficiently motivated to preserve quality. In this case, you might be able to motivate your developers to improve the automated tests, which accelerates the release cycle. It’s become standard in open source projects for a single “maintainer” to merge, test, and accept changes.
We have built the lessons of this analysis into our product and our Scalable Agile process. We built our merge request features with clear instructions that guide the user to merge individual changes with a target branch, test them, review them, and accept or reject the changes. The merge request process keeps track of affected tickets, making it easier to test appropriately. We’ll be posting more about test automation. And, we are now launching services to provide hands-on development for clients that want to release every day.
Get modern code workflows - merge requests, server side hooks and much more - for free with Assembla Renzoku.