Awesome article - I really enjoyed reading it
Good article. This is why I'm never overly dogmatic about a practice. A co-worker came from NASA and used a similar integration process and I think it makes sense.
We are unfortunately asked to demo our product regularly and because of the instability of the mainline, we're always nervous about demo because of this and scrambling at the last minute to work out the kinks.
We want to move to this kind of approach and I think it makes sense.
The problem with each developer working on a separate branch, even when they all merge frequently from mainline, is that you're not integrating between developers. So you've still got un-integrated work to merge later when someone finally commits their work and you pull their changes down.
Quibbling over the definition of "continuous" misses the point. The more work you do without integrating, the more work you store up for later - resolving conflicts, stabilizing changes, and testing.
There's no hard and fast rule of how frequently you need to integrate everyone's work to be officially doing CI, but the longer you leave it, the less effective it becomes.
This reminds me of someone I once heard who said their team was "doing" agile, with iterations that vary from 3 to 5 months depending on how long it took to get the planned work done. Nobody's going to arrest them for not being "pure" agile, but they're kind of missing the point.
The idea of Continuous Delivery is to push yourself to keep your code production ready at all times, so that you're not leaving any work for "later". CI is totally necessary for this, since you need to make sure each and every change is integrated and tested. If you're distinguishing between "production quality" code and "in-development" code, you're probably missing the point of Continuous Delivery.
The problem with all of this stuff (Agile, CI, CD) is that it means changing habits. These habits are different from what most of us learned, they can be hard to get used to, and until you really get into the swing of it, they trip you up.
No big deal, if these things don't seem useful for you, don't do them. But I also wouldn't lose sleep trying to justify why the way you work doesn't fit the label.
I am little sceptical about how much distributed CI would scale. I am saying this from experience. I worked on a project with more than a hundred developers all working on different parallel streams. In the end all these features had to go live and live together. This was a few years ago and we did try distributed CI (though we didn't use the term distributed CI). We had five teams each working on one stream. We had five branches and a mainline (each with their own CI build). We reverse merged back to mainline and ran tests and all worked fine. The first team to integrate back to mainline was the happiest. No merge conflicts and no test failures. The second team would get some merge conflicts and spend some time fixing them. The third team would get even more conflicts and spend more time integrating. So almost every team faced some merge conflict. If Team A felt that code was bad, they just refactored it in one way. Team B, who had no idea about what Team A was doing, also found the code was bad and refactored it another way. Now we have a code merge conflict as well as a design merge conflict. Since both had diverged so much, fixing it meant that both Team A and Team B had to talk to each other (and for a longer time) to fix the conflict. The time advantage that team A (of not fixing merge conflicts was effectively cancelled). We had to move Trunk based development. Yes we had more broken builds, but we fixed that problem by improving our practices and build pipeline. Distributed CI just postponed the integration problem and gave us a false safety net. I don't know how we would managed if each developer had her own branch, it would have exponentially magnified the problem. I do however agree that it can work well on small projects with short lived feature branches (the shorter the life better). I use local feature branches (where I can't afford dedicated CI for my branch) and reverse merge from the mainline. It can however bite you back if it goes beyond control. For the record I work for Thoughtworks and I learnt this from my own experience long before I read Jez or Martin's opinions.
Thank you for your observations. Let me try to outline a couple of practices that are essential for dCI to work. These practices mitigate the issues you are raising.
Yes, we always have some unintegrated work
work between developers. However, our developers do releases for their code at least once a week and most of them do a release 3-4 times a week. The key is to get your code to production ready and push it to production frequently. If we would not do that, we would indeed find ourselves in the situation where developers work for a week and pull from somewhat idle Mainline only to later face integration conflicts. Releasing code very frequently mitigates this issue.
An essential quality for releasing that frequently is feature flags. Code can be on production, but visible only to a subset of users, only to developers, only to alpha, or beta testers. Code can be on production, but not visible to anyone.
The Continuous Delivery mantra of "keep your code production ready at all times" is a lie. Code is either in production or not. This is where Continuous Deployment comes in. What is "production ready" if its not in production
? It is the same as holding inventory in manufacturing.
I see a lot of companies working with similar workflows and what baffles me is that the mainstream thought of CI, Continuous Delivery and Continuous Deployment is still in denial of these successful models.
Nice to have a Thoughtworker on our blog!
I have no knowledge of the project you were on or the specifics of it, but looking at the problems you describe, I can relate to some of them. When we started doing this, we indeed split our development into three teams and worked as your Team A/B/C worked. We were facing similar problems. One additional problem that we had was that a team lead was responsible for release of the code for the whole team and even though he knew the code good, he could not fix issues if they appeared on production as fast as original developers and he could not solve conflicts as effectively as the original developer. This might sound counterintuitive at first, but splitting work to individual developers worked better, because:
* developers can work on one thing at a time until they release it to production
* developers know their code and have very fresh code - so even if they have merge conflicts (which they have), they are fresh and usually very easily solved. This is no different than conflict solving in the standard centralized model in a lot of respects
* we removed all batching, which is one of the big issues with the centralized model - one rotten apple preventing the whole basket of fresh apples to be sold until the rotten one is picked out. This is what used to happen in our experience with the centralized Continuous Deployment model (how do you mitigate this issue?)
While dCI might not be universal, there is one critical benefit when its coupled with Continuous Deployment - you can release code to production very frequently - meaning code is inventory for the shortest possible period and starts working for you and your customers as soon as its ready (meaning, in production
, not production ready
One thing I am picking up from your writeup is In the end all these features had to go live and live together
. Did you have any initiative and benefit to always have the Mainline with all recent changes in production?
Having one person responsible for code is the first anti pattern in your team. The entire team/teams should own the code (even when in production). When ever an issue crops up in the production, the person with the best knowledge fixes it by working very closely with the operations team. We pair program in general. This means more people familiar with same piece of code. Yes it may disrupt your dev cycle, but if there are a lot of defects in production, then you have bigger quality problems which probably means more money lost compared to what you would lose if a feature gets delayed.
About the rotten apple question; Yes you will have rotten apples, but the key is to throw it away as soon as you find each one. By delaying, you are just going to find ten rotten apples at the same time which means more time spent in clean up. Our rotten apples were flaky builds, integration tests and build time. We tackled these issues by fixing bad tests, parallelizing the build etc. Since everybody checked into mainline, the merge conflicts were hardly rotten apples for us. They were minor glitches which got fixed in minutes or an hour max with people talking to each other.
The biggest initiative for trunk based development was that your mainline got the latest features (even incomplete or with feature toggles) and regression defects could be discovered sooner rather than later. We never were always production ready (we were getting closer and closer though), but it helped us get production ready quicker and more predictably.
One flaw which I find in the inventory argument is that you still have inventory in your unmerged feature branches, it just doesn't show up in your mainline. Even if you use feature toggles and release the code, the inventory is still present in the unused features. A feature is ready when it is ready, until then it is always inventory regardless of centralized/distributed CI.
Sorry for being unclear - when we started with multiple teams, people owned
their code (we do peer reviews usually instead of pair programming), but when release time comes, we can't gather the whole team (we're distributed around the world in different timezones). Also, when someone is mergning code from team branch to Mainline, usually one person is merging. Have you been doing it in a different way?
I agree about the defects in production reducing quality and counting towards money loss - but only with Continuous Deployment we can fix and deploy bugfixes in no-time - we don't have to wait until mainline is stabilized to release a bugfix, or resort to non-standard measures such as ad-hoc release branches. I'm not saying that bugs in production are good (they are inevitable however), I'm saying that proper Continuous Delivery allows to fix them very fast thus providing value.
About rotten apples - well, if mainline is unstable and not production ready at a certain point, even if the build passes, by definition you have a rotten apple in the mainline. Question is how you detect it. Of course, its a no-brainer to get your automatic tests to pick them out (but then again, why do testing after committing to Mainline, why not a pre-commit hook that denies the commit if build fails?), but if you do manual QA or UX testing afterwards when code is integrated in mainline, that is a potential step to show that mainline is unstable. I think it makes sense to move all activities that can pick out rotten apples to happen before the branch -> mainline integration (a simple commit to mainline is also the same idea). This allows to have an always deployable mainline.
I somewhat agree with the flow you notice in my inventory argument. Yes, getting rid of inventory fully is not possible. Let me try to explain how I see code in different steps generating value:
* code that is not visible to anyone, but on production: generates minimal value. Only value is that it is in mainline and integrated - meaning anyone will get it when they do a mergeback from Mainline
* code that is visible to our developers: starts to generate real value. we are the first customer and we are reaping the benefit already. It might not be as pretty or stable as it will be when we release it, but we are already using something more than an MVP
* code that is in alpha/beta: generates medium value. Customers who desperately need the feature are able to use it sooner. Most of them will be willing to put up with a couple of rough edges only to get the benefit of losing their pain sooner. Wouldn't you?
So, while we don't get rid of inventory fully, we do minimize it.
So here's the short answer: CI is defined
- by Wikipedia ("Everyone commits to
the baseline every day") and Martin Fowler
("Everyone Commits To
the Mainline Every Day") and others
- as everybody merging all their changes into
trunk on a regular basis (at least once a day). I urge you to go to these articles and search for the text I have quoted above. It's there. I know, it's easy to miss it if you're in denial ;-)
You can actually see me on stage with Martin Fowler defining it here: http://yow.eventer.com/events/1004/talks/1062
You are welcome to follow another process such as the one you describe, and if it works, that's great, but don't call it continuous integration (or "distributed continuous integration") because it's not, and you're just going to confuse everybody. As Abraham Lincoln said, "If you call a tail a leg, how many legs has a dog? Five? No, calling a tail a leg don't make it a leg."
I know you are skeptical that CI (as it's really defined) and CD can work together. A lot of people feel that way until they have tried it. But they can. It's described in detail in my book
, and in this talk
, and it's a technique that is used, as described, by Etsy, Google, Facebook and others. There's a forthcoming video where John Penix of Google described how they do this, based on his talk on my continuous delivery track
at QCon last year. Everybody in Google regularly checks into a single trunk on Perforce.
Also, CI works perfectly great with distributed version control systems. I have been working on teams who do this this since 2008. I talk about that here: http://continuousdelivery.com/2011/07/on-dvcs-continuous-integration-and-feature-branches/ (and this is probably the most important one for you to read)
@Jez - first off, I want to say that I respect the work that you and Thoughtworks have done for quite sometime now - its impressive. I am glad that you are willing to engage in a healthy debate over what is CI and what works well with CD.
To take your first argument, that that is how Wikipedia defines it, is to say that is how the current thought is on the subject matter, but it does not mean that it cannot change. The thought leadership that you and Martin Fowler have done has definitely led the movement of CI to the traditional definition set out in the Wikipedia article, but that does not mean it cannot advance or change as technology changes.
I have read your articles and watched your talks. But I am still not convinced that merging backwards is worse than merging forwards to a mainline. All code is tested against all other code in a mergeback process. Yes this requires frequent releases, similar to the idea of frequent check-ins. If you do not have a frequent check-in, you still integrate long running branches behind a centralized point and have the advantage of merging back onto it. The long running branches are integrated at this point to ensure the development work is not conflicting.
Why would moving the merge point backwards be any different than keeping it in a centralized point? Perhaps Google would like to give it a try if they understood it were an option. We used to have a Centralized CI system, it caused bottlenecks - and the first thing I like to do with bottlenecks is remove them.
As for the name, calling it distributed CI is no different than calling it dvcs - we had vcs's now we have a dvcs - why not have dCI to match it?
The important part to me is the result not the process - whatever works to keep clean, flowing code to Production should be the goal of any organization. To be able to accomplish this without bottlenecks and with efficiency should be the next goal. The advantage of moving the bottleneck out of the line of good code to Production far outweighs any strict process for me.
The process you describe in this article is not "new and improved". It is exactly the process ClearCase was pushing in the 1990s (ClearCase's "dynamic views" give you the merge from mainline into your branch automatically).
The whole reason continuous integration was invented was to replace the process you're describing because of the problems with it. That's the reason why calling it "distributed CI" is deeply ironic. You can be sure that Google (and everyone else who knows the history of version control process) understands it is an option, and explicitly rejected it.
As to which process is better, I have tried to explain that in my book and in the article I referenced on DVCS, CI and feature branches
. Clearly I have done a bad job.
What you describe as a "bottleneck" is only a bottleneck if you're optimizing for the creation of unintegrated code (where "integrated" means "integrated with everyone else's work", not "integrated with what's currently on mainline").
I am pretty sure I'm not going to be able to convince you of this, and that's fine - all I ask is that you don't describe this process as a "new and improved" continuous integration - it's not.
I don't think that I called it "new and improved" in the article, I pointed out that the power of the dvcs lends itself to this process and makes it more feasible with Continuous Delivery.
It is distributed CI since it Integrates all code prior to going into Production Mainline before it gets there. It is Continuous when coupled with Continuous Delivery which wants small batches of work to constantly go into Production. This prevents the long runs without Integration, once a commit is made, its tested against the Production+this commit, then pushed to Production and merged back on all other in-progress development work to Integrate and be tested. At the same time, the development work is now Integrated with the Current Production work and is developed upon.
It is not only integrated with what's currently in Mainline, its also integrated with all other development work. It just becomes the problem of the developer who has not pushed to Production to fix the conflict if there is any, without holding anyone up to push to Production next.
Its not about convincing me, its about what is actually happening and what is important to happen, if anything you can help me refine the process or abandon it if it is not feasible, though I see it working in action every day.
The process you describe is feasible, it's just not optimal. Here's what I hear when I read your blog: "We tried moving to continuous integration. But it was too painful. So instead of trying to fix the underlying problem which made it painful, we moved back to something less painful. We're going to call that a better version of the thing we couldn't get to work, because makes us feel better about giving up on something that was too hard."
Continuous integration exerts a force on your developers. It makes it painful to work in big batches. That's why you found it hard. The correct response to that is to do all your work - new features, architectural changes, bug fixes - in small increments (a few hours work max) each of which keeps trunk releasable and gets you a little closer to your goal, and to check them in to trunk and deploy them.
This is difficult. It requires great discipline. It's hard to work out how to decompose a feature or architectural change into small, incremental bits of code. Until you get good at it, it takes longer than doing it in a big batch on a branch.
But working in this way pays off big time. It prevents you from shaving yaks. It means you don't do three days' work and then find out you have to roll back two days of it because you can't finish the refactoring without changing big chunks of the codebase. It stops you gold plating features. It stops the developers from having to merge in days worth of other people's work before they can get stuff out. Once you've spent a few months practicing this I promise you won't want to go back. You'll feel like you just gained a superpower.
Pretty much every good methodology in modern software development - kanban, continuous integration, continuous delivery, hypothesis-driven development - is all about making you work in smaller batches. So when it hurts, that's good. The process is working. It's telling you something. Ride the pain. Challenge yourself. Work out how to solve it.
DVCSs are fabulous. But unfortunately they make it easier for you to cheat and take the easy way out. And what makes me sad (and angry) is that so many developers got fooled into thinking the easy way out was actually a better way.
Exactly - smaller batches moving forward faster. It not that we experienced Pain from Traditional CI - its that we saw how it was slowing us down particularly when you have to determine where in the large CI process something failed. It created an integration bottleneck that slowed down every developer, not just the bad one. So instead of punishing everyone and forced ridicule that Traditional CI puts on the developer, we looked to remove this overburden and the waste associated with Traditional CI.
Look at a timeline for Traditional CI:
I commit, I wait for everyone else to commit, CI runs, we deal with conflicts and integration problems, then move code to Production.
A Timeline for Distributed CI:
I commit, I integrate with all code on Production, wait for CI, then I push to Production, then all code is mergedback to other developers and CI is run and the integration problems are fixed.
The difference is when I am dealing with integration problems and how quickly I can get code out to Production.
You absolutely must use smaller batches and absolutely must commit often. When you do not, however, you are not punished as harshly, since you have been integrating often.
It is the goal of every developer in Continuous Delivery to work in smaller batches, then why batch all developers work up into one large CI batch? This seems contrary to the small batch argument. Where is the conflict? Is it my code or your code. By running smaller CI batches per developer, I know exactly where the integration problem occurred. How does Traditional CI handle this? It does not and is a flaw in the system.
Working on a Remote Team compounds this problem, since you must work out conflicts between developers, sometimes asynchronously. But in Distributed CI, you know exactly who must fix the issues and when - immediately.
Advancements to the technology is not cheating, its a change in the game. I am still not sure why you argue that this is not full integration when each code is integrated at exactly the right time, anytime there is a change to the code in Production. Since it is using a Continuous Delivery model, that means very often throughout the day. And all code is integrated at this time, but only the necessary is amount to move to Production is integrated before it is released to Production, we check the other code integration afterwards since it does not matter.
Worse case scenario is that you have to patch your code that was just released for upcoming code - the dev about to release the upcoming code can do this. This makes it exactly the same as Traditional CI, except I do not have to wait to push out the first round of code. In Traditional CI, my first round of code would have been delayed till after the second round of code was fixed and released.
Great post, I'm interested on how you might include a functional test loop as well though. Our functional tests take a long time now. Certainly too long to run the complete suite for each developer.
Each developer takes on more responsibility and should be running these functional tests themselves as they develop. Barring automation, which will of course help you with testing. The developer should be able to break the test suite down into the smaller subsets of tests that matter for the code that they are working on - so not all functional tests need to be dealt with. It is important to be able to identify the functional tests that will be affected and which will not.
Then they will use a QA as a contributor to their project, not as a gatekeeper - this is important as well. The developer consults with the QA contributor to help them determine if their code is functionally proper.
Then if you must have a UAT step, you will have this before code goes into Production, you can batch this up before Production for an aggregate test suite run. In other words, in the diagram, change the Production bubble with a UAT bubble, then add another Production bubble below it.
Consider this abstraction. As soon as two or more developers have some sort of incompatible thoughts the product under development becomes "virtually broken" even if they have not yet written any code. The longer these issues go unresolved the tougher they are to resolve.
One of the goals of CI is to make these issues apparent, and to fix them, as soon as possible. Therefore what we want is a place where we meet frequently where we can fix these problems. A branch if you will. Let's call it Branch-X (because it's name is unimportant)
Interestingly deployment and conflict resolution are related. One way of testing if developer conflicts have truly been resolved is testing whether the merging of their work can be deployed. Therefore in practice a team strives to keep Branch-X deployment ready.
In parallel a company/team/project might have the need to retain something stable for deployment or branching purposes. That is not the goal of Branch-X. If you are lucky then Branch-X could do that double duty, but if you aren't then you should have another branch for this stability function.
The article sort of implies that wanting such a stable branch is a reason for changing how Branch-X should operate. I think that's red-herring and it's an unrelated topic.
Our reason for wanting Branch-X deployment ready is to have an indicator of a certain level of harmony on the project. However that indicator becomes less meaningful the longer we delay putting stuff into Branch-X.
I want to amend my comment a bit.
I agree with the gist of the article. Developer's should merge down and then copy up their tested work into the mainline. I would expect most developer's would do that to some degree.
However, I think the degree of testing that should be done off the mainline depends upon how important the continuous deployment capability is for the project/company.
I like your amendment and maybe I did not make it clear enough, however, the important part to understand is that this form of testing will not work for everyone, as you state.
Its just another tool in the toolbox and is useful when wanting to deploy to Production several or very many times in a day.
One point I think that I did not explain clearly enough is that all code going into Production is being tested against all developers code, immediately after pushing to Production. This ensures that we are testing the code against code in development. If there is an issue, then the developer who still has code in development must work it out (sometimes by talking with the developer who pushed code into Production).
This form of testing basically takes the ready to go code (or thought) and says its adequate enough to test against everyone else's code or thoughts, but releasing it immediately. It can always be changed later if the upcoming code or thoughts need it to be. We are admitting that we are in a Continuous world and therefore things change. Having a central meeting place does not necessarily change this, it only delays it by having too many people in the conversation. By isolating the work out to developer branches, you know who is incompatible.
There is no silver bullet for how everyone should be doing things, the right tool for the right job, and its good to have several tools to choose among. As a team or as a business you will have to decide what is more important, always having everything tested and decided upon before moving forward, thereby slowing down the process or if most decisions can be adequate, then move forward and course correct when you have to.