Current Articles | RSS Feed RSS Feed

Distributed Continuous Integration - Keep the Mainline Clean

Posted by Michael Chletsos on Fri, Apr 05, 2013
  
  

Distributed Continuous Integration

I recently defined Continuous Integration as the practice of merging development work with a Master/Trunk/Mainline branch constantly so that you can test changes, and test that changes work with other changes, this is the "as early as possible" integration methodology. The idea here is to test your code as often as possible to catch issues early (Continuous Delivery vs Continuous Deployment vs Continuous Integration)  

Watching a presentation by Jez Humble of ThoughtWorks, who defines Continuous Integration (CI) in relationship to Continuous Delivery, I realized that my definition was in direct opposition to 2 minutes of his presentation: http://www.youtube.com/watch?v=IBghnXBz3_w&feature=youtu.be&t=10m  But why is Jez so adamant about these points, whereas I feel I am doing CI without everyone committing to a Mainline daily, with having Feature branches and often with working locally, making commits without having ALL integrated tests running.  Can I be?  I say yes, and the reason is because of where the integration points are and what kind of code is moving forward to Mainline - unstable code or stable code. These differences keep a clean stable Mainline which in turns gives the ability to deploy code to Production anytime.  Jez’s definition of CI (Centralized CI) causes bottlenecks and is in direct contradiction with the process of Continuous Deployment (CD), whereas Distributed CI is able to remove these barriers while still giving confidence good code is moving to Production.

Argument

The argument goes, that a Continuous Integration process is the process of constantly integrating all development work across the entire project into Mainline to detect issues early on in a code’s lifecycle.  I argue that when utilizing Continuous Deployment, this is detrimental and the proper way is to integrate only Production ready code to Mainline, while merging Mainline back onto the isolated development work.  Recently, I explained this concept to another developer and their response: “I have never thought about merging backwards from master to branches in order to run test, amazing”.  Continuous Deployment is not achievable using traditional CI methodologies as described by ThoughtWorks and others because the bottlenecks will prevent the flow of code from developer to Production, but the simple notion of merging backwards to run tests in order to see a view of Mainline before you integrate upwards, will allow you to achieve Continuous Deployment

Daily Commits to Mainline

Checking in only stable code is necessary for Continuous Deployment, however it is in direct opposition of the Centralized CI evangelists’ rule #1: (http://www.youtube.com/watch?v=IBghnXBz3_w&feature=youtu.be&t=10m) you must check into Mainline daily.  The idea here is that you are integrating with all development code daily, whether it is Production ready code or not.  This means that development work must be hidden if released, but it also means that development work, that may be thrown out tomorrow, may not integrate properly with Production ready work today, causing an unstable Mainline.  Any practice that forces Mainline to become unstable worries me a little, it just seems unnecessary.

In Distributed CI a developer commits locally to the developer branch and integrates often (daily) backwards from Mainline, saving the trouble of integration bottlenecks in Mainline.  Anything a developer takes from Mainline in a Distributed CI process is considered stable and releasable code - maybe not immediately used code.  Centralized CI may have a good build but still may have unreleasable code.  You must coordinate the Centralized CI release, a Distributed CI release does not have this issue, since Mainline is always considered stable and a release may be performed at anytime.  

Broken Builds

Oh and lets not forget broken builds - broken builds prevent anyone from being able to reliably take code from Mainline or move code to Production from Mainline.  Centralized CI has a way to “fix” this, rule #2: (http://eugenedvorkin.com/10-continuous-integration-practices/) never break the build, if you do you must fix it, and not leave until you do.  OK, so nevermind the insanity of the first part of this rule, since the only way to ensure satisfying it is by never committing: I cannot control how my code integrates with other unknown code..  And sure, if I break something, I understand that I must fix it.  But how do we know it was my code that broke the build and not yours, we don’t.  But assume it is me who broke it, now everyone is waiting for me to fix it.  I have now become a bottleneck and no code can move past Mainline, shoots, I guess Friday night beers are not in my future.  And the pipeline from developer to Production has just halted dead in its tracks, no code can be Continuously Deployed.  Some places call this a lesson (http://www.hanselman.com/blog/FirstRuleOfSoftwareDevelopment.aspx) and assume that you will learn from this ordeal, I think you will fear it, yes, but to work in fear . . . I prefer not to.  I prefer to have systems that do require me to stay late to keep Mainline stable unduly, but rather a system that always ensures Mainline is stable, before and after my code is merged.

Distributed CI deals with this with a “mergeback” from Mainline to your developer branch - ensuring that you have a clean build after running unit tests on the branch, then merging up to Mainline. Otherwise, having failed a developer branch unit test - do not merge to Mainline, do not become the bottleneck, go out for beers, rest well, and fix it tomorrow.  After the developer branch is merged to Mainline, do not run the unit tests again - the test suite was already run against a copy of Mainline in your development branch, already knowing that all tests have passed, the code is deployed right out to Production.  Then Mainline is merged backwards to other developers’ branches, and unit tests are run individually on each developer branch.  If any tests fail - the owner of that branch must fix the issue.

Feature Branches

Rule #3 (http://www.youtube.com/watch?v=IBghnXBz3_w&feature=youtu.be&t=11m49s minutes 10-12 are definitely my favorite) - no feature branches.  Wait what? Hold on, I like my feature branches.  But Jez did just say: “But you can’t say you’re doing Continuous Integration and be doing feature branching. It’s just not possible by definition (while waving hands)”, didn’t he?  Looking at the definition of CI from Martin Fowler of ThoughtWorks:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.  (http://martinfowler.com/articles/continuousIntegration.html)

I do not actually see anywhere that says CI cannot have feature branches, just that all developers work must integrate frequently - not even daily.  Well in Jez’s Centralized CI - feature branches are a no go - you must integrate daily ALL changes to a centralized Mainline.  But hold on - this code can’t be integrated into Mainline, its weeks away from delivery.  In distributed CI - a feature branch is nothing more than a developer’s branch.  Feature branch away. Mainline will be integrated back into the Feature branch after any Production deploy and all new stable code will be integrated and tested immediately.  

Hmmm, sounds like feature branching and CI are not mutually exclusive - nice. But Jez was very clear on this.  Yes, his worry is about long running code not being integrated with.  OK - so you want a true feature branch and not integrate up to Mainline for longer than today.  Well you can.  Since you are always integrating backwards from Mainline.  But you must continually do this.  Once we are ready to move the feature branch into Mainline - we already have a snapshot of what Mainline would look like, the feature branch is Mainline plus all new code of the feature branch, and it has been Continually Integrated with since creation every time Mainline is updated.  If two developers want to integrate with each other, then they are able to in the Distributed CI World, they can actually be moved behind another integration point, where their work is merged up to it and then to Mainline.  This is basically utilizing the ability to create a distributed network of developers working off of various localized Mainlines that merge up to a Master Mainline.

Distributed CI Reigns Supreme with CD

Distributed CI has the advantage for Continuous Deployment because it keeps a clean and stable Mainline branch that can always be deployed to Production.  During a Centralized CI process, an unstable Mainline will exist if code does not integrate properly (broken build) or if there is unfinished work integrated.  This works quite well with iteration release planning, but creates a bottleneck for Continuous Deployment.  The direct line from developer branch to Production must be kept clean in CD, Distributed CI does this by only allowing Production ready code to be put into the Mainline.

Is Distributed CI really CI?

Yes.  if you are implementing Distributed CI and CD together, you will always be integrating your developer code with the latest stable release to Production.  If an integration test fails after the release to Production, it will fail for a developer’s branch, not everyone, so we know the integration failure is isolated to that developer’s branch and the new code; the developer whose failed branch must attend to it.  In Centralized CI, the developers must discuss and work out this issue to see who is at fault.  CD demands that you deliver code to Production often.  In Distributed CI, anytime you deliver code to Production, the Mainline branch is merged back to developer branches and tested (thank goodness for automation).  What does this mean, lets look at the life cycle of a commit a little closer:

Centralized vs Distributed CI

The top diagram shows a Centralized CI process, where the developer merges into a Mainline integrated with other developers commits before running Unit Tests.  Mainline cannot necessarily be deployed since some work from developers may not be Production ready.  The lower diagram show a Distributed CI process where the developer merges Mainline backwards, then integrates up to Mainline, then pushes on through to Production.

If you are performing CD, you are releasing very often, each stable commit is still tested with every developer’s development work.  Except in distributed CI - its tested individually, so detection of issues is easier - the amount of conflicting code is less, and isolated - to a developer’s branch.  Distributed CI is a more thorough form of CI when implemented with CD - which attempts to break down processes into smaller automated pieces.  Centralized CI is suited for iteration release planning, but distributed CI will work just as well for this and better.  If you are doing centralized CI - then you are not doing Continuous Deployment.  Dstributed CI removes all the bottlenecks and constraints placed on the workflow from developer to production that centralized CI requires.  

So why all this confusion around CI and the artificial constraints put on it?  Clearly, the centralized VCS is the basis for these constraints.  Basically, Distributed CI is treating each developer branch like it is Mainline and testing there instead of up to a centralized Mainline.  Perhaps its because CI grew up in a time of non-distributed VCS and has not modernized since.  Now is the time to utilize the advantage of a distributed VCS.  Unleash those feature branches, and commit code now, merge to Mainline to see it in Production moments later - yes, moments later, every time.

Anyone else performing CI like this, or have any other questions, please leave a comment below.

To read more about Continuous Deployment and how it can help you, check out Assembla and CD.

Tags: , , , ,

COMMENTS

Awesome article - I really enjoyed reading it

posted @ Friday, April 05, 2013 6:34 AM by Stanislav


Good article. This is why I'm never overly dogmatic about a practice. A co-worker came from NASA and used a similar integration process and I think it makes sense. 
 
We are unfortunately asked to demo our product regularly and because of the instability of the mainline, we're always nervous about demo because of this and scrambling at the last minute to work out the kinks.  
 
We want to move to this kind of approach and I think it makes sense.

posted @ Friday, April 05, 2013 7:16 AM by Ryan


The problem with each developer working on a separate branch, even when they all merge frequently from mainline, is that you're not integrating between developers. So you've still got un-integrated work to merge later when someone finally commits their work and you pull their changes down. 
 
Quibbling over the definition of "continuous" misses the point. The more work you do without integrating, the more work you store up for later - resolving conflicts, stabilizing changes, and testing.  
 
There's no hard and fast rule of how frequently you need to integrate everyone's work to be officially doing CI, but the longer you leave it, the less effective it becomes.  
 
This reminds me of someone I once heard who said their team was "doing" agile, with iterations that vary from 3 to 5 months depending on how long it took to get the planned work done. Nobody's going to arrest them for not being "pure" agile, but they're kind of missing the point. 
 
The idea of Continuous Delivery is to push yourself to keep your code production ready at all times, so that you're not leaving any work for "later". CI is totally necessary for this, since you need to make sure each and every change is integrated and tested. If you're distinguishing between "production quality" code and "in-development" code, you're probably missing the point of Continuous Delivery.  
 
The problem with all of this stuff (Agile, CI, CD) is that it means changing habits. These habits are different from what most of us learned, they can be hard to get used to, and until you really get into the swing of it, they trip you up. 
 
No big deal, if these things don't seem useful for you, don't do them. But I also wouldn't lose sleep trying to justify why the way you work doesn't fit the label. 

posted @ Friday, April 05, 2013 9:12 AM by Kief


I am little sceptical about how much distributed CI would scale. I am saying this from experience. I worked on a project with more than a hundred developers all working on different parallel streams. In the end all these features had to go live and live together. This was a few years ago and we did try distributed CI (though we didn't use the term distributed CI). We had five teams each working on one stream. We had five branches and a mainline (each with their own CI build). We reverse merged back to mainline and ran tests and all worked fine. The first team to integrate back to mainline was the happiest. No merge conflicts and no test failures. The second team would get some merge conflicts and spend some time fixing them. The third team would get even more conflicts and spend more time integrating. So almost every team faced some merge conflict. If Team A felt that code was bad, they just refactored it in one way. Team B, who had no idea about what Team A was doing, also found the code was bad and refactored it another way. Now we have a code merge conflict as well as a design merge conflict. Since both had diverged so much, fixing it meant that both Team A and Team B had to talk to each other (and for a longer time) to fix the conflict. The time advantage that team A (of not fixing merge conflicts was effectively cancelled). We had to move Trunk based development. Yes we had more broken builds, but we fixed that problem by improving our practices and build pipeline. Distributed CI just postponed the integration problem and gave us a false safety net. I don't know how we would managed if each developer had her own branch, it would have exponentially magnified the problem. I do however agree that it can work well on small projects with short lived feature branches (the shorter the life better). I use local feature branches (where I can't afford dedicated CI for my branch) and reverse merge from the mainline. It can however bite you back if it goes beyond control. For the record I work for Thoughtworks and I learnt this from my own experience long before I read Jez or Martin's opinions.

posted @ Friday, April 05, 2013 9:29 AM by Uttam Kini


Hello Kief, 
 
Thank you for your observations. Let me try to outline a couple of practices that are essential for dCI to work. These practices mitigate the issues you are raising. 
 
Yes, we always have some unintegrated work work between developers. However, our developers do releases for their code at least once a week and most of them do a release 3-4 times a week. The key is to get your code to production ready and push it to production frequently. If we would not do that, we would indeed find ourselves in the situation where developers work for a week and pull from somewhat idle Mainline only to later face integration conflicts. Releasing code very frequently mitigates this issue. 
 
An essential quality for releasing that frequently is feature flags. Code can be on production, but visible only to a subset of users, only to developers, only to alpha, or beta testers. Code can be on production, but not visible to anyone. 
 
The Continuous Delivery mantra of "keep your code production ready at all times" is a lie. Code is either in production or not. This is where Continuous Deployment comes in. What is "production ready" if its not in production? It is the same as holding inventory in manufacturing. 
 
I see a lot of companies working with similar workflows and what baffles me is that the mainstream thought of CI, Continuous Delivery and Continuous Deployment is still in denial of these successful models.

posted @ Friday, April 05, 2013 10:01 AM by Titas Norkūnas


Hello Uttam, 
 
Nice to have a Thoughtworker on our blog! 
 
I have no knowledge of the project you were on or the specifics of it, but looking at the problems you describe, I can relate to some of them. When we started doing this, we indeed split our development into three teams and worked as your Team A/B/C worked. We were facing similar problems. One additional problem that we had was that a team lead was responsible for release of the code for the whole team and even though he knew the code good, he could not fix issues if they appeared on production as fast as original developers and he could not solve conflicts as effectively as the original developer. This might sound counterintuitive at first, but splitting work to individual developers worked better, because: 
* developers can work on one thing at a time until they release it to production 
* developers know their code and have very fresh code - so even if they have merge conflicts (which they have), they are fresh and usually very easily solved. This is no different than conflict solving in the standard centralized model in a lot of respects 
* we removed all batching, which is one of the big issues with the centralized model - one rotten apple preventing the whole basket of fresh apples to be sold until the rotten one is picked out. This is what used to happen in our experience with the centralized Continuous Deployment model (how do you mitigate this issue?) 
 
While dCI might not be universal, there is one critical benefit when its coupled with Continuous Deployment - you can release code to production very frequently - meaning code is inventory for the shortest possible period and starts working for you and your customers as soon as its ready (meaning, in production, not production ready
 
One thing I am picking up from your writeup is In the end all these features had to go live and live together. Did you have any initiative and benefit to always have the Mainline with all recent changes in production?

posted @ Friday, April 05, 2013 10:22 AM by Titas Norkūnas


Having one person responsible for code is the first anti pattern in your team. The entire team/teams should own the code (even when in production). When ever an issue crops up in the production, the person with the best knowledge fixes it by working very closely with the operations team. We pair program in general. This means more people familiar with same piece of code. Yes it may disrupt your dev cycle, but if there are a lot of defects in production, then you have bigger quality problems which probably means more money lost compared to what you would lose if a feature gets delayed. 
 
About the rotten apple question; Yes you will have rotten apples, but the key is to throw it away as soon as you find each one. By delaying, you are just going to find ten rotten apples at the same time which means more time spent in clean up. Our rotten apples were flaky builds, integration tests and build time. We tackled these issues by fixing bad tests, parallelizing the build etc. Since everybody checked into mainline, the merge conflicts were hardly rotten apples for us. They were minor glitches which got fixed in minutes or an hour max with people talking to each other. 
 
The biggest initiative for trunk based development was that your mainline got the latest features (even incomplete or with feature toggles) and regression defects could be discovered sooner rather than later. We never were always production ready (we were getting closer and closer though), but it helped us get production ready quicker and more predictably. 
One flaw which I find in the inventory argument is that you still have inventory in your unmerged feature branches, it just doesn't show up in your mainline. Even if you use feature toggles and release the code, the inventory is still present in the unused features. A feature is ready when it is ready, until then it is always inventory regardless of centralized/distributed CI. 
 
regards, 
 
Uttam

posted @ Friday, April 05, 2013 10:53 AM by Uttam Kini


Sorry for being unclear - when we started with multiple teams, people owned their code (we do peer reviews usually instead of pair programming), but when release time comes, we can't gather the whole team (we're distributed around the world in different timezones). Also, when someone is mergning code from team branch to Mainline, usually one person is merging. Have you been doing it in a different way? 
 
I agree about the defects in production reducing quality and counting towards money loss - but only with Continuous Deployment we can fix and deploy bugfixes in no-time - we don't have to wait until mainline is stabilized to release a bugfix, or resort to non-standard measures such as ad-hoc release branches. I'm not saying that bugs in production are good (they are inevitable however), I'm saying that proper Continuous Delivery allows to fix them very fast thus providing value. 
 
About rotten apples - well, if mainline is unstable and not production ready at a certain point, even if the build passes, by definition you have a rotten apple in the mainline. Question is how you detect it. Of course, its a no-brainer to get your automatic tests to pick them out (but then again, why do testing after committing to Mainline, why not a pre-commit hook that denies the commit if build fails?), but if you do manual QA or UX testing afterwards when code is integrated in mainline, that is a potential step to show that mainline is unstable. I think it makes sense to move all activities that can pick out rotten apples to happen before the branch -> mainline integration (a simple commit to mainline is also the same idea). This allows to have an always deployable mainline. 
 
I somewhat agree with the flow you notice in my inventory argument. Yes, getting rid of inventory fully is not possible. Let me try to explain how I see code in different steps generating value: 
* code that is not visible to anyone, but on production: generates minimal value. Only value is that it is in mainline and integrated - meaning anyone will get it when they do a mergeback from Mainline 
* code that is visible to our developers: starts to generate real value. we are the first customer and we are reaping the benefit already. It might not be as pretty or stable as it will be when we release it, but we are already using something more than an MVP 
* code that is in alpha/beta: generates medium value. Customers who desperately need the feature are able to use it sooner. Most of them will be willing to put up with a couple of rough edges only to get the benefit of losing their pain sooner. Wouldn't you? 
 
So, while we don't get rid of inventory fully, we do minimize it.

posted @ Friday, April 05, 2013 12:11 PM by Titas Norkūnas


Hi Uttam 
 
So here's the short answer: CI is defined - by Wikipedia ("Everyone commits to the baseline every day") and Martin Fowler ("Everyone Commits To the Mainline Every Day") and others - as everybody merging all their changes into trunk on a regular basis (at least once a day). I urge you to go to these articles and search for the text I have quoted above. It's there. I know, it's easy to miss it if you're in denial ;-) 
 
You can actually see me on stage with Martin Fowler defining it here: http://yow.eventer.com/events/1004/talks/1062 
 
You are welcome to follow another process such as the one you describe, and if it works, that's great, but don't call it continuous integration (or "distributed continuous integration") because it's not, and you're just going to confuse everybody. As Abraham Lincoln said, "If you call a tail a leg, how many legs has a dog? Five? No, calling a tail a leg don't make it a leg." 
 
I know you are skeptical that CI (as it's really defined) and CD can work together. A lot of people feel that way until they have tried it. But they can. It's described in detail in my book, and in this talk, and it's a technique that is used, as described, by Etsy, Google, Facebook and others. There's a forthcoming video where John Penix of Google described how they do this, based on his talk on my continuous delivery track at QCon last year. Everybody in Google regularly checks into a single trunk on Perforce. 
 
Also, CI works perfectly great with distributed version control systems. I have been working on teams who do this this since 2008. I talk about that here: http://continuousdelivery.com/2011/07/on-dvcs-continuous-integration-and-feature-branches/ (and this is probably the most important one for you to read)

posted @ Friday, April 05, 2013 12:50 PM by Jez Humble


@Jez - first off, I want to say that I respect the work that you and Thoughtworks have done for quite sometime now - its impressive. I am glad that you are willing to engage in a healthy debate over what is CI and what works well with CD. 
 
To take your first argument, that that is how Wikipedia defines it, is to say that is how the current thought is on the subject matter, but it does not mean that it cannot change. The thought leadership that you and Martin Fowler have done has definitely led the movement of CI to the traditional definition set out in the Wikipedia article, but that does not mean it cannot advance or change as technology changes. 
 
I have read your articles and watched your talks. But I am still not convinced that merging backwards is worse than merging forwards to a mainline. All code is tested against all other code in a mergeback process. Yes this requires frequent releases, similar to the idea of frequent check-ins. If you do not have a frequent check-in, you still integrate long running branches behind a centralized point and have the advantage of merging back onto it. The long running branches are integrated at this point to ensure the development work is not conflicting. 
 
Why would moving the merge point backwards be any different than keeping it in a centralized point? Perhaps Google would like to give it a try if they understood it were an option. We used to have a Centralized CI system, it caused bottlenecks - and the first thing I like to do with bottlenecks is remove them. 
 
As for the name, calling it distributed CI is no different than calling it dvcs - we had vcs's now we have a dvcs - why not have dCI to match it? 
 
The important part to me is the result not the process - whatever works to keep clean, flowing code to Production should be the goal of any organization. To be able to accomplish this without bottlenecks and with efficiency should be the next goal. The advantage of moving the bottleneck out of the line of good code to Production far outweighs any strict process for me.

posted @ Friday, April 05, 2013 1:29 PM by Michael Chletsos


The process you describe in this article is not "new and improved". It is exactly the process ClearCase was pushing in the 1990s (ClearCase's "dynamic views" give you the merge from mainline into your branch automatically). 
 
The whole reason continuous integration was invented was to replace the process you're describing because of the problems with it. That's the reason why calling it "distributed CI" is deeply ironic. You can be sure that Google (and everyone else who knows the history of version control process) understands it is an option, and explicitly rejected it. 
 
As to which process is better, I have tried to explain that in my book and in the article I referenced on DVCS, CI and feature branches. Clearly I have done a bad job. 
 
What you describe as a "bottleneck" is only a bottleneck if you're optimizing for the creation of unintegrated code (where "integrated" means "integrated with everyone else's work", not "integrated with what's currently on mainline"). 
 
I am pretty sure I'm not going to be able to convince you of this, and that's fine - all I ask is that you don't describe this process as a "new and improved" continuous integration - it's not.

posted @ Friday, April 05, 2013 3:54 PM by Jez Humble


I don't think that I called it "new and improved" in the article, I pointed out that the power of the dvcs lends itself to this process and makes it more feasible with Continuous Delivery. 
 
It is distributed CI since it Integrates all code prior to going into Production Mainline before it gets there. It is Continuous when coupled with Continuous Delivery which wants small batches of work to constantly go into Production. This prevents the long runs without Integration, once a commit is made, its tested against the Production+this commit, then pushed to Production and merged back on all other in-progress development work to Integrate and be tested. At the same time, the development work is now Integrated with the Current Production work and is developed upon. 
 
It is not only integrated with what's currently in Mainline, its also integrated with all other development work. It just becomes the problem of the developer who has not pushed to Production to fix the conflict if there is any, without holding anyone up to push to Production next. 
 
Its not about convincing me, its about what is actually happening and what is important to happen, if anything you can help me refine the process or abandon it if it is not feasible, though I see it working in action every day.

posted @ Friday, April 05, 2013 4:24 PM by Michael Chletsos


The process you describe is feasible, it's just not optimal. Here's what I hear when I read your blog: "We tried moving to continuous integration. But it was too painful. So instead of trying to fix the underlying problem which made it painful, we moved back to something less painful. We're going to call that a better version of the thing we couldn't get to work, because makes us feel better about giving up on something that was too hard." 
 
Continuous integration exerts a force on your developers. It makes it painful to work in big batches. That's why you found it hard. The correct response to that is to do all your work - new features, architectural changes, bug fixes - in small increments (a few hours work max) each of which keeps trunk releasable and gets you a little closer to your goal, and to check them in to trunk and deploy them. 
 
This is difficult. It requires great discipline. It's hard to work out how to decompose a feature or architectural change into small, incremental bits of code. Until you get good at it, it takes longer than doing it in a big batch on a branch. 
 
But working in this way pays off big time. It prevents you from shaving yaks. It means you don't do three days' work and then find out you have to roll back two days of it because you can't finish the refactoring without changing big chunks of the codebase. It stops you gold plating features. It stops the developers from having to merge in days worth of other people's work before they can get stuff out. Once you've spent a few months practicing this I promise you won't want to go back. You'll feel like you just gained a superpower. 
 
Pretty much every good methodology in modern software development - kanban, continuous integration, continuous delivery, hypothesis-driven development - is all about making you work in smaller batches. So when it hurts, that's good. The process is working. It's telling you something. Ride the pain. Challenge yourself. Work out how to solve it. 
 
DVCSs are fabulous. But unfortunately they make it easier for you to cheat and take the easy way out. And what makes me sad (and angry) is that so many developers got fooled into thinking the easy way out was actually a better way.

posted @ Friday, April 05, 2013 10:28 PM by Jez Humble


Exactly - smaller batches moving forward faster. It not that we experienced Pain from Traditional CI - its that we saw how it was slowing us down particularly when you have to determine where in the large CI process something failed. It created an integration bottleneck that slowed down every developer, not just the bad one. So instead of punishing everyone and forced ridicule that Traditional CI puts on the developer, we looked to remove this overburden and the waste associated with Traditional CI. 
 
Look at a timeline for Traditional CI: 
 
I commit, I wait for everyone else to commit, CI runs, we deal with conflicts and integration problems, then move code to Production. 
 
A Timeline for Distributed CI: 
I commit, I integrate with all code on Production, wait for CI, then I push to Production, then all code is mergedback to other developers and CI is run and the integration problems are fixed. 
 
The difference is when I am dealing with integration problems and how quickly I can get code out to Production. 
 
You absolutely must use smaller batches and absolutely must commit often. When you do not, however, you are not punished as harshly, since you have been integrating often. 
 
It is the goal of every developer in Continuous Delivery to work in smaller batches, then why batch all developers work up into one large CI batch? This seems contrary to the small batch argument. Where is the conflict? Is it my code or your code. By running smaller CI batches per developer, I know exactly where the integration problem occurred. How does Traditional CI handle this? It does not and is a flaw in the system. 
 
Working on a Remote Team compounds this problem, since you must work out conflicts between developers, sometimes asynchronously. But in Distributed CI, you know exactly who must fix the issues and when - immediately. 
 
Advancements to the technology is not cheating, its a change in the game. I am still not sure why you argue that this is not full integration when each code is integrated at exactly the right time, anytime there is a change to the code in Production. Since it is using a Continuous Delivery model, that means very often throughout the day. And all code is integrated at this time, but only the necessary is amount to move to Production is integrated before it is released to Production, we check the other code integration afterwards since it does not matter. 
 
Worse case scenario is that you have to patch your code that was just released for upcoming code - the dev about to release the upcoming code can do this. This makes it exactly the same as Traditional CI, except I do not have to wait to push out the first round of code. In Traditional CI, my first round of code would have been delayed till after the second round of code was fixed and released. 
 

posted @ Saturday, April 06, 2013 3:21 AM by Michael Chletsos


Great post, I'm interested on how you might include a functional test loop as well though. Our functional tests take a long time now. Certainly too long to run the complete suite for each developer.

posted @ Monday, April 08, 2013 5:42 AM by Hansel Dunlop


Hi Hansel- 
 
Each developer takes on more responsibility and should be running these functional tests themselves as they develop. Barring automation, which will of course help you with testing. The developer should be able to break the test suite down into the smaller subsets of tests that matter for the code that they are working on - so not all functional tests need to be dealt with. It is important to be able to identify the functional tests that will be affected and which will not. 
 
Then they will use a QA as a contributor to their project, not as a gatekeeper - this is important as well. The developer consults with the QA contributor to help them determine if their code is functionally proper. 
 
Then if you must have a UAT step, you will have this before code goes into Production, you can batch this up before Production for an aggregate test suite run. In other words, in the diagram, change the Production bubble with a UAT bubble, then add another Production bubble below it.

posted @ Monday, April 08, 2013 8:30 AM by Michael Chletsos


Consider this abstraction. As soon as two or more developers have some sort of incompatible thoughts the product under development becomes "virtually broken" even if they have not yet written any code. The longer these issues go unresolved the tougher they are to resolve. 
 
One of the goals of CI is to make these issues apparent, and to fix them, as soon as possible. Therefore what we want is a place where we meet frequently where we can fix these problems. A branch if you will. Let's call it Branch-X (because it's name is unimportant) 
 
Interestingly deployment and conflict resolution are related. One way of testing if developer conflicts have truly been resolved is testing whether the merging of their work can be deployed. Therefore in practice a team strives to keep Branch-X deployment ready. 
 
In parallel a company/team/project might have the need to retain something stable for deployment or branching purposes. That is not the goal of Branch-X. If you are lucky then Branch-X could do that double duty, but if you aren't then you should have another branch for this stability function. 
 
The article sort of implies that wanting such a stable branch is a reason for changing how Branch-X should operate. I think that's red-herring and it's an unrelated topic. 
 
Our reason for wanting Branch-X deployment ready is to have an indicator of a certain level of harmony on the project. However that indicator becomes less meaningful the longer we delay putting stuff into Branch-X. 
 
 

posted @ Sunday, April 21, 2013 7:43 PM by Tony Mowers


I want to amend my comment a bit. 
 
I agree with the gist of the article. Developer's should merge down and then copy up their tested work into the mainline. I would expect most developer's would do that to some degree. 
 
However, I think the degree of testing that should be done off the mainline depends upon how important the continuous deployment capability is for the project/company.

posted @ Sunday, April 21, 2013 8:15 PM by Tony Mowers


Tony- 
 
I like your amendment and maybe I did not make it clear enough, however, the important part to understand is that this form of testing will not work for everyone, as you state. 
 
Its just another tool in the toolbox and is useful when wanting to deploy to Production several or very many times in a day. 
 
One point I think that I did not explain clearly enough is that all code going into Production is being tested against all developers code, immediately after pushing to Production. This ensures that we are testing the code against code in development. If there is an issue, then the developer who still has code in development must work it out (sometimes by talking with the developer who pushed code into Production). 
 
This form of testing basically takes the ready to go code (or thought) and says its adequate enough to test against everyone else's code or thoughts, but releasing it immediately. It can always be changed later if the upcoming code or thoughts need it to be. We are admitting that we are in a Continuous world and therefore things change. Having a central meeting place does not necessarily change this, it only delays it by having too many people in the conversation. By isolating the work out to developer branches, you know who is incompatible.  
 
There is no silver bullet for how everyone should be doing things, the right tool for the right job, and its good to have several tools to choose among. As a team or as a business you will have to decide what is more important, always having everything tested and decided upon before moving forward, thereby slowing down the process or if most decisions can be adequate, then move forward and course correct when you have to.

posted @ Sunday, April 21, 2013 9:34 PM by Michael Chletsos


Comments have been closed for this article.

Follow Assembla

twitter facebook youtube linkedin googleplus

Subscribe by Email

Your email: