2 Flavors of Continuous Delivery: Centralized versus Distributed

Posted by Andy Singleton on Jul 29, 2013 12:54:00 PM

When we made a commitment to Continuous Delivery, we struggled with some buzzword bingo.  The word “Continuous Delivery” actually applies to two very different strategies.  The “centralized” approach uses one code version for all developers and tests.  It is simpler and more efficient.  The “distributed” approach puts each change in a different version of the code.  is more complicated, more scalable, and it allows you to actually release every change. Read this article to learn how to use these strategies.

"Centralized" or "Single Stream Continuous Integration"

In "Centralized" or "Single Stream Continuous Integration" every developer commits changes to one version of the code – trunk, master, mainline.  A continuous integration system is constantly running automated tests on this shared version.  The developers are trained to structure their changes to avoid breaking the build.

Key concept: Early as possible integration

Important input: Automated tests

centralized ci resized 600

You can see the key concepts of continuous delivery at work.

* Automated tests.  You need enough automated tests to tell you immediately if something is broken.  You run them frequently (continuous integration).

* Developer responsibility.  It is the responsibility of the developers to make sure that their code doesn't break anything.  It should be releasable.  They can't just throw it over to QA and wait for bug reports.

* Feature switches.  Developers are constantly releasing code for the new features that they are working on, but they need to hide any features that aren't ready.  They put switches in the code to hide changes that aren't officially released or "unveiled".


Centralization is efficient.  There is one test system, which is important if your test environment is complicated.  You don’t have to move code between multiple versions.   You know where to go for manual testing.  Because of its efficiency, this is a great pattern for a small team.

It relies on the principle of “as early as possible” integration and discovery of problems.  People are committing frequently, and tests are running frequently, and any conflicts between changes are discovered quickly.  This minimizes the amount of wasted programming effort and improves efficiency.

When developers are building new features or major refactoring, they use switches to hide them and prevent them from interfering with the working version.  Hiding and unveiling code is a tactic that is used in any continuous delivery process.

Single Stream Continuous Integration works well with Subversion, where it is called “active trunk”.  Subversion does a good job at synchronizing the developers onto the single stream.


Centralized CD doesn’t scale very well if you have many contributors, new contributors, or distributed teams.  The more changes that you put into the shared version, the more often you will have a “broken build” where it doesn’t work, or possibly doesn’t even compile.

As you add contributors, the process becomes more stressful.  Every contributor has to be alert to a broken test or a broken build, and fix it immediately, regardless of time zone, so that the other contributors aren’t delayed.  This puts extra stress on distributed team members who have responsibilities after the end of their day.  That's why Humble doesn't recommend it for truly distributed teams.

The "early as possible" integration process limits the frequency of your releases.  Whenever you put two changes together, you expect to find problems.  It's not releasable until you have tested for the possibility of new problems.  Centralized CD well suited to a two-week release cycle that will include user acceptance testing, and batched releases.  It's really a continuous integration system, and not a continuous delivery system.  If you want to release every day, or every change, you may decide make a major leap from “as early as possible” integration to “as late as possible”.

The fix: Test and review

You an eliminate almost all of these disadvantages and go to continuous release, if est and review the incoming commits.  We will discuss this below.

Distributed or Continuous Release

Some modern online services have hundreds or thousands of developers, releasing once, twice, or 50 times per day.  They do it with a “Continuous Release” process.  In this process, each change gets tested and released independently. 

Key concept: Late as possible integration

Important input: Cloud computing with on-demand test environments

If you want to release every change, you can’t take the time to test it with every other change that a big group of developers is working on.  To solve this problem, Distributed CD makes the leap from “as early as possible” integration to “as late as possible” integration.  Each change is tested and released before you try it out with the other candidate changes. 

distributed cd resized 600

1) You put changes into a branch or fork for an individual or team committer.

2) For testing, you merge from the production version, to your change branch.  This gives you a release candidate for testing.  Git has a special operation called “rebase” that helps you merge the target version, and then batches your changes up into a single package for delivery.

3) You have multiple test environments.  QA acts in a consulting role, looking at a test environment if a developer needs help verifying that the change is good and desirable.

4) You release.  This is highly automated, and it may be incremental.  You are likely to be releasing some features that have switches to make them visible only to selected beta users.


With the distributed process, you can release every change immediately, which provides a competitive advantage for online service providers.  The continuous release process reduces stress by getting fixes out more quickly, and by allowing big changes to take as much time as they need.  The distributed process can include a large number of contributors, and they can be distributed in multiple time zones.


Test environments are more complicated.  There is more DevOps load to set up multiple test environments.  Developers are responsible for maintaining versions and shepherding changes through tests.  They need to know more about the test environments.

Critics often observe that developers using this approach can work a long time on branches, and then cause integration problems when they merge big batches of code.  Finding problems in these big batches of code is difficult.  If you use this approach, you should:

  • Merge frequently "back" from the production version, to your development version.  So, you will do frequent integration and test in your development version, and you will not find problems when you merge "forward" into the production version.  Git has a special operation for this called "rebase", which is radical in the VCS world because it "rewrites history" to bundle your history of changes into one commit for forward delivery.
  • Release changes as frequently as possible.  When you release smaller amounts of code, you have fewer places to look for problems and you will find them faster.

Gerrit-style change review – the middle ground

You can improve the scalability of the centralized process by adding a code review.  This will give you some of the simplicity of the centralized process, with some of the scalability of the distributed / continuous release process.   We like a review process where every contribution aimed at the master branch goes into a temporary branch, where it can be tested and reviewed by tests systems and other developers.  This has a lot of good effects.  First, it increases the quality and stability of the target branch.  Second, it gives you very convenient place to put “precommit” automated tests, so that your automated test system can reject any changes that don’t pass tests.   Third, it helps you bring in new contributors through reviews and comments.  Fourth, and perhaps most importantly, it improves your automated tests because you can ask for tests in the review process.

review branch resized 600

This change review workflow was originally built into Gerrit, the code contribution system used by the Google Android project and many related telecomm companies.  At Assembla, we think this type of change review is an important step for continuous delivery teams, so we implemented a streamlined version of this workflow that we call "protected branches."  The Assembla imlpementation gives you a very clean workflow, and you can add it to any git master with a few clicks.

Convergence:  How distributed starts to look like centralized

In a distributed CD process, you have an incentive to release frequent, small changes.  Why?  Because small changes are easier to debug.  If you discover a problem, you only have to look through a small amount of changed code to find it and fix it.  When problems show up in a large release, they are harder to fix and they create more stress.  So, over time, distributed, continuous release teams will move to frequent, small releases, with a trunk or master that is constantly updated, and is similar to a single centralized version.

In future articles, we’ll cover the hiding and “unveil” tactics that make it possible to smoothly sneak big changes into a continuously released system.

Topics: continuous delivery, development process, renzoku, continuous integration

Follow Assembla

Get Started


Subscribe to Email Updates