Why are programmers kicking butt? Because they are using powerful machines. Vast data centers filled with computers do much of their work. Ultimately, these machines will do experiments, and make data-driven decisions that replace a lot of work all the way up the chain to the CEO.
Decades ago, I heard the story that software developers (we called them programmers) would never improve their productivity, because programming is a labor intensive business, done by people, and people don’t change much over the generations.
That was a pretty good analysis from a time when software got delivered slowly. It’s clearly wrong now. We’re building software a lot faster now. Small teams of 5 people are building systems that 20 years ago would have required 50 people, and 50 person teams are making talking advisory devices and self-driving cars with millions of lines of code. Since software is eating the world, the world depends on our ability to build software faster and with bigger scale.
How are we doing it? Here is a diagram of the stages of software development:
- The strategizing and fuzzy front end, where you decide what you want to do
- Task/issue management, team management, or project management, where you figure out who will work on what
- Code contribution, where you actually write code and evaluate changes to code
- Build, test, and deploy
In my observation, the amount of innovation and productivity improvement increases dramatically from left to right.
This picture helps us see how “agile” became a dead zone. The typical Scrum-based agile process recommendations are all about how to organize teams and tasks. It’s an art that may be very helpful for an individual organization, but overall, it’s advancing at a snail’s pace. As a friend of mine said, “the last thing they came up with was the cardwall.”
However, there is a lot of innovation in the area of coding workflows. We have new workflows for merge contribution, code review, and continuous integration, which all add up to much faster release cycles and more scalable teams. And, when we get to build, test, and deploy methods, we find an avalanch of new capabilities from cloud computing providers.
Agile consultants and enthusiasts like to think of themselves as being important to the “C-suite” of chief managers. They want to contribute to strategy, and make it more “agile”. But, when they do that, they don’t have any sources of innovation. They become the same as everyone else who is discussing management and strategy. They want to improve productivity, but their desire for importance leads them in EXACTLY THE WRONG DIRECTION. The world of strategy is even more static, with few innovations since the heyday of the industrial revolution 200 years ago.
The new agile techniques based on continuous delivery are tapping into a huge source of innovation by exploiting new coding and testing workflows. This is important. At Assembla, we believe that we can add a lot to agile just by reaching out and bringing a bunch of coding and testing innovation into the “agile” frame. That’s why we make an integrated system that links tasks with leading edge code management, review, and test.
It seems to me that innovation and productivity improvement are closely related to the degree of mechanization. On the left hand side, you have people talking. On the right hand side, you have arrays of machines that run scripts to process changes, thousands of times.
So, it turns out that productivity improvement in software development works the same way as productivity improvement in car manufacturing, or coal mining. You can improve productivity by adding machines. And, the more machines you can add, the more you can improve.
John Henry, watch out! This a megatrend that’s not going to stop for a long time.
The man to machine megatrend even gives us a route back to the C-suite, strategy, and planning.
On the long planning loop, you send all of your product initiatives past the C-suite to get aligned with strategy. Your CEO and your VP of Marketing will argue about what to try next. They will have caveman-level productivity. They’ll do the same thing that two cavemen used to do when they strategized around the fire, trying to figure out whether they’ll have better luck hunting mastodons up in the mountains, or down by the river.
On the short loop, you are using rapid development cycles to test options and gather data. The shorter you can make this loop, the faster it will run. It will also become more productive because you are moving to the right side of the process where you can use more machines.
If the short loop gets good enough, you can decide more and more things with testing, and you will decide fewer and fewer things by having management discussions to decide what bets to make. Eventually, you can get to the point where you can pass a lot of management work off to machines.