This article provides an introduction to Scalable Agile, a non-Scrum process that is designed to help you scale a development effort from 2 contributors to 2000 without abrupt transitions, work with diverse and distributed contributors, and release whenever you want.
Table of Contents:
When I talk to corporate people about agile software development, they are usually talking about "Scrum." 10 out of 12 of the software managers that we brought together for our Scalable Agile Design Group describe their process as Scrum or Scrum-like. Here's a description of Scrum:
A team gets together in a short planning session and decides what they will work on in the next iteration. Then they work for a fixed period of time – often several weeks – and they end up with a "potentially shippable release." Here's a link to a picture. There's more - you'll hear about three roles, three meetings, three artifacts - but not too much more. If you have a team of 5 to 10 people, all in one place, releasing at known intervals, Scrum can help you achieve the agile goal of frequent, high-quality releases.
If your development effort, like most, does not exactly meet the description of a "team of 5 to 10 people, all in one place, releasing at known intervals," Scrum will leave you with some problems. If you have only two or three people, the cycle has too much overhead. If you have more than ten people, you will need multiple teams, and you will need to make an effort to coordinate them for an iteration plan. You won't be co-located. Most modern development projects, and all big ones, include people from multiple locations. As you get close to a release, you merge stuff together for testing, and there is never enough time for that. If you run an online service, or you have a lot of components to update, you face pressure to release more frequently.
Most development efforts will face some of these issues. Cloud development projects face most of these issues. On Assembla projects, we always have all of these issues.
Problems cluster around two things that happen in a batch – iteration planning and testing for release. These activities don't scale well. Iteration planning is a problem for any distributed team or multi-team project that is not meeting in one place. Test and release in a fixed time interval is a problem if your system is big, or if your developers are exceptionally productive.
So, one of our goals it so eliminate these batch activities and set up a continuous process that allows us to plan every day and release every day. We want to relieve the stress that builds up around a fixed release schedule. That goal is especially relevant in the age of cloud software development. Our research shows that cloud and online product developers are very concerned with continuous integration and rapid release cycles.
Another goal is to include diverse types of contributors. Training people to work together in self-managed Scrum teams is a laborious process. If you have such a team, that's fantastic. If not, if you are a lone hacker, or you work in a hierarchical organization, we still want you to be an effective contributor.
A related goal is to include contributors from any location. Distributed work is a requirement for any modern project.
Finally, we want to scale. The value of process and methodology is much greater for a large group, as getting efficient work from such a group is a difficult and expensive task. An agile process should be designed to help us with that task.
"Scaling" doesn't only mean going to larger project sizes. Most software projects are smaller than a cross-functional Scrum team. An effective process for 2 or 3 people can be summed up as "prioritize and communicate." For these groups, the transition to a Scrum process with "3 artifacts" and "3 rituals" can be harsh. Shouldn't we instead try to bring easy and informal cooperation into larger groups? A truly scalable agile process should provide some guidance to a team of 2, and allow 100 more contributors to join, without abrupt transitions.
I applied some basic principles for managers interested in scaling any effort:
- Recognize reality. For example, teams are distributed, specialists are shared, and projects will cost as much time and effort as they are worth, and not what you estimate or recommend.
- Let contributors decide what to do and get the resources they need. Most management problems occur because managers have limited knowledge. They just don't know as much as the people doing the work, and it is unreasonable to expect them (or expect yourself) to make a perfect project plan, or to know what each of 20 other contributors can do or should do. 21 people are smarter than one person.
- Expand the resource base. Add new contributors when you need them. Try to get everything on-demand. The Internet is very large and it connects you to a lot of people and things that can help you.
We're going to start with what is basically a Kanban process. Here is one of my articles with a picture of Scrum, Kanban, and a recommended variant I call "ScrumBan." I try to remember three important things about Kanban:
1) People "pull" work. This is a simple way to make sure that work never piles up in the wrong place, and it saves a lot of time and effort for managers who would otherwise have to figure out who to assign things to, and when. See my discussion under "empower individual contributors."
2) It's a continuous process. Instead of planning a batch of tasks, each contributor takes one task, works on it until it is finished, and then gets one new task.3) It's based on a Lean principle which says that you should not keep "inventory." In a software project, inventory is the stuff you worked on that isn't finished. In principle, you should finish each task before you start a new one. The software Kanban board takes care of this in a very simple way by limiting "Work in Progress", or WIP. WIP is similar to a sprint backlog in Scrum.
Scalable agile adds specific recommendations which will help you with continuous delivery - releasing every day or whenever you want. Scalable agile also provides a structure for including diverse types of contributors.
In a Scalable Agile process, you plan your release at the end, and not at the beginning. You just skip all of the iteration planning and other batch processes, which is an immediate productivity booster. People work at their own pace, which you measure for forecasting and incentive purposes. Then, when you are ready to release, you take what is ready, and you release it.
Here is a chart that shows the process.
(1) We make a prioritized backlog of features, tasks, and bugs that you want to work on. This is the same backlog you will use for Scrum or any other agile process. The "product owner" selects a subset of items that we are ready to work on, which we call "Current" work. "Current" is very similar to a sprint backlog, with the difference that you don't have estimate its size.
(2) Contributors grab tasks out of Current and start work. They can grab a task of any size (tasks have very different sizes – that's one of the realities we need to recognize). In keeping with Kanban principles, they should only grab tasks that they are sure they are going to work on it in the very near future, and they should never be allowed to hoard tasks just because they are experts in a particular subject. We haven't specified what a "Contributor" means. It could be a lone hacker, or the person in the next desk. It could be an established high-performance Scrum team, which can grab a sprint backlog, and commit to delivering it on their normal sprint cycle. This process is 100% Scrum compatible. A contributor could also be an outsourced team, or a big partner, that uses hierarchical management and grabs individual tasks. They can all work with their own process as long as they plug in correctly at the next step.
3) Each contributor has a separate environment for building and testing. You can do this step manually on developer workstations if you have a small team and a highly skilled tech lead or "maintainer". However, this is normally the step where you pull out the high technology, such as automated tests and on-demand cloud based copies of your production environment. You will also want to reorganize your QA professionals to serve as resources that can test on any of these environments. This type of organization is a necessary (and probably sufficient) step toward releasing every day. You can read about it in more detail in my article "Continuous Delivery and Scalable Agile - Find Out the Secret."
4) Contributors submit code (and other changes) for review and release. There is a centralized release manager or release team, which has ultimate power at this stage. The system should be set up to handle changes in the form of specific merge requests (Assembla) / changes (Gerrit) / pull requests (Github), which can be reviewed and accepted or rejected. This review process is usually used to put changes into the contributor test builds, and then again to move changes into the production release. When a change makes it through both reviews, we assume that it is ready to release.
It's faster. It's continuous delivery. We enable releases "on demand" by having multiple "on-demand" test environments and accepting changes when they are ready.
Statistically, we have found that smaller changes result in fewer bugs with smaller bug impact.
We make the process work for distributed groups by eliminating mass meetings and the requirement for mass consensus. We do this with "the principle of pull" where contributors pull and contribute when ready, knowledgeable tech leads who can manage both code and people, and clear online visibility of all work. This allows many discussions to take place asynchronously, or one-to-one.
We make the process scalable to an indefinite number of contributing individuals and teams by eliminating centralized iteration planning and centralized testing.
We open the door to many types of contributors, not just Scrum teams, or people trained in a particular process. At Assembla we like to think about "blended teams" that can include our tech-lead driven distributed teams, partners or outsourcers with hierarchically managed teams, shared functional specialists (like design and ops), lone hackers, and individual experts. It's Scrum compatible! Established Scrum teams can grab a sprint backlog, and work as contributors with no process change.
If we are bringing in contributors we don't know, how do we control them? We aren't necessarily managing them. They are managing themselves in unknown ways! Will our project fly out of control? Actually, we have two very strong points of control. First, we control the backlog, so we control what people work on. Second, we control the code changes and other changes that we accept. So, we have an absolute control over what goes in the product. The review and acceptance process actually gives us a much stronger way to enforce standards for quality, architecture, and testing than we get in most other methodologies. Finally, the continuous process prevents contributors from causing problems if they get stuck. Work from other contributors will just flow around them.
There are modern projects, mostly cloud-based, that use variations on the Scalable Agile process that I describe here. However, it is a recent development, because it is dependent on four key technologies, and some of those technologies are new.
1) Ticketing and task management
In its simplest form, the list of tasks is a stack of cards, or post-it notes, or lines in a spreadsheet. The quaint stack of cards is idealized as a good focal point for Scrum or Kanban teams, but it's not what we want for scalable agile. I'm going to describe it as a technology because a distributed team always has an online ticket or issue list that is globally visible. It may contain some of the following features:
- Customers and stakeholders adding requests
- Ways to organize large backlogs
- Requirements and mockups added with increasing detail
- Ongoing real-time discussion and collaboration
- Workflow steps, including testing
- "Traceability" linking a ticket to related code commits, tests, and deployments
2) Global team collaboration and management
Internet workspaces for distributed teams (like Assembla) and related management tools.
3) On-demand build and test systems
Continous delivery requires separate integration test environments for each contributor, so you can test each change. To make it scalable, you want to satisfy "the principle of pull" that a contributor should be able to get what he needs without going through some centralized system for building and testing. Fortunately, there are now many suppliers of on-demand cloud servers.
You will want to invest in automated testing and continuous integration. Test and integration environments are specialized for each project. Fortunately, there are also many new systems and platforms for automated testing, continuous integration, and automated deployment.
4) Code management and merge
Source code management and code review tools are an essential part of the process. New merge workflows have given a big boost to the type of continuous delivery process that I describe here. Code management tools were neglected in the last generation of agile methodologies, which focused on interactions between people. However, the reality is that managing code is a lot easier than managing people. It's difficult to personally manage 20 developers, and it gets more difficult if they are in 20 time zones, but you can scale a code contribution and review system to include thousands of contributors.
The Assembla product will provide tools to help you run "Scalable Agile" and other continuous delivery processes. Here is an overview of relevant new and planned features:
Continuous flow for tasks: Traditionally, the Assembla ticket system organized tickets into multiple "milestones," or iterations. In the new Planner view, there is only one Backlog and one Current container (WIP or Sprint Backlog), so work can flow through Current in a continuous process. You can see Current on the cardwall, a traditional Kanban view. This week we will put a continuous flow diagram on the metrics page to provide a measure of progress that is useful in the continuous flow system.
Large backlog management (unreleased): We found that when we put everything in one backlog, the list was too long to visualize, and we needed some new tools. We are compressing it with epics, stories, and tasks. We are also testing in filtered views, so that you can see only the tasks that apply to your team or epic on the Planner and Cardwall.
Multi-team task management (unreleased): You will be able to throw a filtered view of tickets out into a related "child space" for a new team to work on. You will also be able to report on the progress of the complete backlog, or any contributor or team.
Merge requests / code review: A few months ago, we released advanced merge requests for Git. They allow code contributors to package up a set of changes, and submit them for code review and merge. We have used and refined this feature as we moved our release interval from 2 times per month to many times per day. We have improved the integration with tickets so that changes can move from idea to release in one smooth process. We added "tickets affected" by a marge request- a type of automated release note makes my life much better - and in the new ticket layout, a view of merge requests related to a ticket. We have implemented merge requests for Perforce, and we have been working with the Perforce team on their own highly scalable form of continuous delivery. We are currently working on merge requests for Subversion, which will fit with the "symmetric merge" improvements coming in Subversion 1.8 later this year.
Continuous integration plugins for Jenkins: Jenkins can help you build and test every proposed change. These plugins help you configure Jenkins to run continuous integration with tests for any merge requests, authenticate using Assembla accounts, and even include it as a space tab.
Professional services, under the Studios label: These services range from basic DevOps services, to setting up scripts for continuous integration and continuous delivery, through full management and project delivery.
Talent: You should be able to expand with new individuals and teams. We're working to make it easier.