Last Sunday I introduced a theory of innovation by defining innovation as a production process that produces something called “design”. Today I will introduce the raw materials that feed this process, optimization and exploration. We will confront fountain of innovation itself, the dueling horsemen of genesis, the yin and the yang, order and chaos.
In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. And God said, Let there be light: and there was light. And God saw the light, that it was good: and God divided the light from the darkness.
King James Bible
If yin and yang do not exist, the One cannot be revealed. If the One cannot be revealed then the function of the two forces will cease, Reality and unreality, motion and rest, integration and disintegration, and clearness and turbidity are two different substances. In the final analysis, however, they are one.
Chinese philosopher Chang Tsai
Innovation is a process of trial and error. You try something new, and if it works, or it survives, or if you like it, you keep it. Otherwise, you call it an error and you throw it away.
We can also visualize innovation as a search process. When you try something new, you are exploring. When you find something you like, you keep it, and then you improve it. You start optimizing. In a simple model of this, you are hiking through a mountain range, looking for the highest peak. You roam around looking for a high slope. That’s your exploration. Then, you climb up. That’s your optimization.
Our “hike to the top of the mountain” task sounds easy enough, because you can just look up to see the mountain tops. However, I do this type of hiking in Maine, where it is often very foggy, and difficult to see through the trees. If you are trying to do real innovation, the fog will be very thick, and you won’t be able to see the future. In fact, you may not be able to see which direction is uphill. In these circumstances, you can’t just go uphill. You have to do little explorations, striking out 20 paces in one direction, and then coming back and trying a different direction if you didn’t find the way up. Maybe one person holds a flashlight, and another circles, as I found myself doing one foggy night. In this case, optimization is a type of exploration, only with smaller distances.
Optimization and hill climbing are just explorations/trial-and-error over small distances. We aren’t going very far, so we are pretty sure that we will end up almost as high as the place we started, and there is about a 50% chance we will end up higher. We have a good chance of success. It seems a lot easier than doing large-distance exploration, in which we have to climb down through gullies, across plains, and the occasional grand canyon. However, there is a problem. We eventually get to the top of our hill, and then we don’t know if we are on the highest hill. Maybe if we strike out over a longer distance, we will find a much higher peak. It is a constant tradeoff.
So, we have a dual strategy. We explore. We try some big things. We roam around and map out some mountains. We start new businesss or new product lines. Then, if we find opportunities that are good enough, we shrink the options and start optimizing. And, we are never sure. Should this button on the web site be blue, or red, or maybe throw in some quotes about order and chaos from ancient texts? That’s just presentation. We can use Google Web Optimizer to test that with customers and find out what works best. What if I offer a different feature set? That’s a lot more work to test. Should I even be in the business that I am in? I might get a million dollars with this product launch, but there is a 26-year old guy over there who is making billions of dollars with a different approach.
At this point, if you are a troublemaker, you might raise your hand and say “What about the impact of creativity? Is it fair to compare leaps of human creativity with purely mechanical processes like evolution or A/B market testing?” My answer would be, yes. I’ll define a creative solution as something that is found very efficiently (aha! we don’t know exactly how the creator found it), and that is good but not provably optimal (it’s good as art or innovation or practice, not mathematically). You can make a purely mechanical process that find solutions with these attributes. For example, it is not practical to find an optimal solution to NP-hard problems like the traveling salesman problem – the shortest path to visit many unique cities. However, you can make a computer that can find very good solutions with trial, error, and careful selection. It looks for short paths that go to a subset of cities, and then runs trials where it combines them together, and keeps the shorter results, and makes variations, and evolves shorter and shorter solutions. These solutions are found efficiently, and are not provably optimal. The key input is a sort of domain knowledge that tells the computer what to optimize and when to take a creative leap, basically, when to grab a sub-path solution and search in its vicinity. This is the same type of domain knowledge and “discernment” and boldness that we find in creative individuals.
When we think of innovation as trial and error, it’s clear why it is not efficient. The more trials we do, the more errors we get. We can reduce the number of errors by doing more optimization – more trials of close-by ideas that we are pretty sure will work as well as what we have, but then we don’t get the big gains. There is an uncertainty principle of innovation that is similar to the quantum uncertainty principle. If you want to make bigger gains, you have to accept more and worse errors. There are two things that you can do to make your search more efficient. Make sure that you are not repeating yourself – not working with too much order. And, make sure you are searching in the right general area – not working with too much chaos. You can control the size of your steps. After that, you are dependent on the underlying landscape to yield its treasure – and a lot of errors.
In our innovation manufacturing system, we have dial that we can turn from order to chaos. If we turn it toward order, nothing happens. We make only very small improvements. If we turn it toward chaos, we generate a lot of big steps. Our product is constantly changing, and not for the better, and nothing works right.
Order and chaos are the major inputs to evolution. Mutations are the product of chaos, and most large mutations are fatal. A creature born without a heart, or with extra heart parts, will die. Most living things are highly optimized – they are the top of their little hills – so any step away is likely a step down. Evolution adds order, usually in the structure of the genome. Genomes are carefully constructed so that the result of reproduction operations is very similar to the inputs. I saw this very strongly in my genetic programming experiments, where evolved software quickly evolved redundancy that made it behave the same in future generations, even after cutting it in half and inserting random operations. Try doing that to non-evolved code. This defensiveness is why natural evolution is often very slow. It’s evolved to slow itself down. It can be speeded up.
Chaos alone is bad for innovation. It rips you away from good things before you can use them and refine them. Everything changes, and nothing improves. Order alone is bad for innovation. It stifles innovation, freezing you in place.
As innovators, we need a balance. As we will see on a future Sunday, we need a system on the edge of chaos.
Next in series: Backing away from the edge of chaos