Quality on the fly: How to stabilize and release
My startup clients often experience a moment of terror when they realize that I am recommending what looks like a worst-case scenario for release quality. How will they make sure the system actually works on release? My recommendations are usually as follows:
- We will release every two weeks, on a schedule, whether the intended features are done, or not. Releasing on time and not on feature completion is important. It ensures that the feedback machine is always running at full speed.
- We will release to real users who will critique us.
- Worse, we will be releasing "trunk" code, the code we were just fingering, rather than code that was specially branched and code-frozen.
The trick is in stabilization
. We don't do a code freeze, because we have to keep moving. But, there has to be a stabilization period, during which the developers agree not to make major changes. Here is how we do it for our Web projects:
Tips and Tricks
- When we begin the stabilization period, look at the application, and decide what will be in the release by selecting the things that are finished or mostly finished. Move all tickets that will not be in the next release to the following release. So, we should see a short ticket list for the top release. New bug reports will be added to this short list.
- On the day before the release, remove features, edit them, or roll back changes if there are problems. If we are working on big features that take more than two weeks to build and stabilize, we hide them in the production release, but show them in the test and development releases. You can use statements like "IF not Production then do this..." to show them in test and development releases and hide them in the production release.
- If there are problems on release day, then release on the following day.
Your stabilization period needs to be long enough to test the system. If you have a new product that is simple, you may be able to do it in a day. As your system gets bigger and more complicated, there is more to test. As more people rely on it, the tolerance for errors decreases. So, this time will get longer.
The length of the stabilization period actually determines the length of your release cycle / iterations / sprints. You should allow about twice as much time for new development as stabilization. So, if your stabilization period is four days, you will want 8 days for development (12 days total), rounding up to 2 weeks.
It helps A LOT to have automated testing. The more automated testing that you can do, the shorter your test and stabilization period.
Stabilization time must be short. If the stabilization period becomes longer than a week, then you are delaying your developers who need to do new development. You probably will not be able to continue releasing your shared "trunk" code. You will have to create a source code branch that separates the stabilized version and the new development. This should be avoided as long as possible because it costs you time and energy. You have two development efforts and you have to fix code branches. That's the price you pay for being slow. You get slower.
There is a use for code freezes, but it is mostly psychological. If you need your team to pay more attention to some particular aspect of quality, you can inflict a code freeze on them. Stop the music. If you do it infrequently, you will get their attention.