Current Articles | RSS Feed RSS Feed

Avoiding Premature Integration to Reach Continuous Delivery

Posted by Michael Chletsos and Titas Norkunas on Mon, Oct 08, 2012
  
  

or: How we learned to stop worrying and ship software every day

We have been writing about testing and tools in our day-to-day Continuous Delivery process. However, we did not write about our development workflow yet. So, here you go. Did we show you the graph?

 

Just one year ago we were releasing once or twice per month and this September we’re at 55 releases a month. All with the same development team. What does this mean to our customers? They get updates from us every day. Sometimes every hour, sometimes more than once an hour. What does that mean to us? No iterations, no useless stalling, no situations where one ticket breaks something and is poisoning the whole branch.

What was holding us back? Premature integration to the master branch turned out to be the culprit. Let me provide some context about our previous branching model and ticket workflow. For those eager ones, you can skip to the solution.

Old Workflow

In terms of code versioning and servers we had a few predefined branches that also matched QA servers - a production (prod) and a development (dev) branches that would be deployed on respective qa servers (prod-qa and dev-qa). We would patch production with things tested on prod qa-server and do feature development and non-high priority bugs in dev.

In terms of ticket flow, our development team works (this has not changed in the new process) according to these guidelines:

Ticket Status

Description

New

Ticket gets created, and Tech Leads prioritize the ticket in the Agile Planner

Accepted

Developer picks up the ticket and writes code for it

Review

If first developer does not feel sure that ticket is ready to go, another developer reviews the code

Test

If after peer review, developers do not feel sure that ticket is ready to go, they ask for help from our QA team who does some testing.

Failed

Ticket goes to this status if it fails testing

Deploy

Admin deploys code to production

Fixed

Ticket is recognized as resolved.

If at any point a ticket does not pass a test it gets set to Failed. The original developer usually picks failed tickets and works on them to fix the issues.

Problem

Major problem with such workflow for us was few tickets poisoning the whole batch. Once ticket got merged to dev, if it turned out that it was not good to go, it was keeping the whole batch until some action was taken on it - either fix the issues or revert the poisonous code.

This was severe, given that Assembla employs people from all over the world and we are big on asynchronous development. Possible “easy solutions” for the problem:

  1. Have very thorough specifications.
  2. Have a rigid peer review process
  3. Have a release manager who maintains the dev branch and either fixes or reverts the poisonous code.

All of these do not work, or are short term solutions, or bring slowness as side effects. In essence, these solutions scale poorly. Is there a solution for this problem that scales well?

Solution

Lets analyze this Cardwall view. Can you spot the problem?

There are tickets to be released, but the whole batch can not be released because some of the tickets are not ready. However, some (12) of them are ready. These tickets are not even related, but Ticket #15711 (as well as some of the tickets in Accepted/Review/Test, if they have code committed to master) stop us from releasing all the tickets in Deploy. Recognizing this leads us to the most important insight - every action, that can put a ticket back in status must happen before integration to master. We decided to change just this one thing in our process. Master integration must happen at the last possible step. The result...Did we show you the graph?

The later the merge, the better - this means that issues will be caught before the integration most of the time. Have you heard of those companies, that do not let their developers go home until master build is green? Have you stayed late fixing someone else’s code after they are not there and their code is failing user acceptance testing? Yeah, forget about those ruined evenings.

We’ll continue to write about our development workflow and how we managed to move integration to be the last step of the process on this blog, so if you read until here, you might as well subscribe to our blog.

Tags: , , ,

COMMENTS

A great example of how making work in progress and blockers visible can lead to actionable outcomes for feature pipeline throughput.

posted @ Monday, October 08, 2012 2:17 PM by Thomas Schranz


Sorry, this sounds exciting but I'm not following your grammar: "every action, that can put a ticket back in status must happen before integration to master."  
 
Can you clarify what you mean by "back in status"?

posted @ Monday, October 08, 2012 2:28 PM by Hal


Hi Hal, 
 
"back in status" means that ticket moves backwards, not forward, in the workflow given example could be "from Test to Failed" or "from Test back to Review". Any action that can do that (say a test by a QA tester) should be happening before integration to master branch.

posted @ Monday, October 08, 2012 3:06 PM by Titas Norkūnas


Isn't this just each developer playing the role of a release manager (of their own contributions)? Who merges to the master branch?  
Is there an intermediary branch that QA is testing, or do they use the dev branch?

posted @ Wednesday, October 10, 2012 7:46 PM by Michael


Hi Michael, 
 
Yes, you are right. Every developer plays the role of release manager for their own code. At this moment we have tech leads merging to the master branch, but ideally, everyone should. I think that this only makes sense - who better knows how to releas, later monitor and bugfix the code than the original developer?

posted @ Thursday, October 11, 2012 3:53 AM by Titas Norkūnas


Am I misunderstanding or are you saying that code review only happens if the dev thinks it's necessary and QA testing only happens if the dev and reviewer think it's necessary? This implies changes can go live without any QA regression / exploratory testing. How are your defect / rework rates?

posted @ Thursday, October 11, 2012 11:20 AM by Rob Fletcher


Titas, 
Thanks, that is a good way to put it. Rob brings up a good point though; where does QA fit into the picture?

posted @ Thursday, October 11, 2012 12:18 PM by Michael


Hi Michael/Rob- 
QA should be a consultant to developers helping them learn how to do testing on their products and perform testing when a developer asks them for help. Otherwise, they should be focused on customer experience and performing exploratory testing, finding edge cases and hard to find bugs. The quality of the product is ensured by Product Managers and developers who should know the specs and features the best. 
Code Review should happen often (except in unnecessary cases, i.e. small changes)  
Regression Testing should be mostly automated with Continuous Integration tests.

posted @ Thursday, October 11, 2012 1:11 PM by Michael Chletsos


Hi Titas, 
How is your workflow with branches. Has every developer their own branch? Or do you have dev/prod branch? 
How are you working with branching. What is workflow. 
 
Thanks. 
Jeroen

posted @ Friday, November 02, 2012 6:46 AM by Jeroen


Hi, Jeroen, 
 
Actually every developer or team (depending on their preference) get their own fork, where they do their work. Every ticket is usually developed in a separate branch in those forks. We don't have an unclean dev/prod branch. Whenever a ticket gets done in its own (ticket_12345) branch, it gets merged straight to master and released to prod immediately, 
 
Titas

posted @ Friday, November 02, 2012 7:03 AM by Titas


This sounds good and all but I have to admit I get pretty frustrated with your software changing all the time - and sometimes being less stable or more quirky than it should be. I don't think it's the best approach.

posted @ Thursday, December 13, 2012 10:07 PM by Dan Connor


Hello Dan 
 
Which areas of the app do you think are the most quirky?

posted @ Friday, December 14, 2012 6:56 AM by titas


Comments have been closed for this article.

Follow Assembla

twitter facebook youtube linkedin googleplus

Subscribe by Email

Your email: