Your Source/SVN tool now supports a modern code review and code contribution process. You can see it under the new "Review code" subtab.  It is similar to the workflows enjoyed by git users to include a large number of contributors or to review code for continuous integration and continuous delivery.  You can put your changes in a branch, review them, and merge them through a Web-based UI.

This first release has a simple workflow that requires users to manage their own branches.  Please leave a comment with your feedback.  After we are sure that this is working smoothly, we will enhance it with much-requested features like extra permissions on trunk (to push contributions into reviewed branches), one-button branch creation, and inline code commenting.

How it Works:

Make a temporary branch.  Write your changes and commit them.

Make a merge request.  You only need to select the source branch. We already know how to merge it to the destination it was branched from.

Contributions show up on the review list

Select a merge request to review.  You can open the "Get Changes" link to see instructions for merging and testing the changes.  In most cases, you want to test the result of merging the change with trunk (or the source that you branched from).  So, the instructions will ask you to switch to trunk, and merge the changes.

Use the controls on the merge request to write comments (you can @mention the contributor), vote, and view changed files and diffs.

Subversion merge requests have a lot of the same great features that are in our git merge requests.  For example, they get linked to tickets when you include a #ticket link in your commits.  So, when you can see the status of the relevant code changes.  And, when you look at a merge request, you can see the tickets you are working on.

Patch or branch?

A lot of Subversion code review systems use patches.  You make a patch (a diff file with the changes) and you upload it to a code review system like Reviewboard, or you email it to a list of reviewers (Subversion developers do this on the Apache Subversion developers list).  The advantage of a patch is that it is easy for a reviewer to apply it to whatever version of code he is working on, and it doesn't need to be integrated with your repository server.  However, it is annoying to make and upload a patch, and you can't fix it.  If you see a problem in a branch, you just switch to the branch and fix it.  We discussed this with our own team, and with some of the Apache Subversion core developers, and we decided that we prefer reviewing and sharing real branches.

We hope that you see some benefit in the branch-based workflow.  Get it for free with Assembla Renzoku.

Hold on for increasing returns... In the last Sunday article on a theory of innovation, I observed that innovation is a very wasteful process.  It includes a lot of trial and error, mostly error.  So, if you need to be efficient, you cut back on innovation.  Necessity is the mother of efficiency, and the enemy of innovation.

So, one of the prerequisites of innovation is abundance: subsidies; risk capital;  extravagance;  youth;  total war, whimsy; bubbles; money to burn; free food.  One of the things that I noticed when visiting Silicon Valley during the last few years was free food at most of the events and offices that I went to.   Last week I did a tour of Kendall Square, Cambridge, at which I was offered free food at every stop, so maybe the East Coast is on the way back.  Don't starve your innovators.

Abundance doesn’t need to come in the form of money.  The software industry is currently floating on a tide of code delivered by the Internet.

However, if abundance was the only requirement for innovation, then the emperor or superpower of the day could just order up some innovation.  That’s clearly not what happens.  Great powers don’t necessarily stay ahead of the curve.

The curve looks like this:

History implies that both evolution and innovation have a punctuated dynamic.  They can be stagnant for a long time, and then suddenly take off in an exponential curve.  One striking example of this is the evolution of human technology.  It’s easy to imagine that the top-level production curve that counts the number of different devices used by humans worldwide was pretty flat for tens of  thousands of years, was on  a slow exponential increase through the middle ages, and hocky-sticked about 200 years  ago.   We have better data for specific industries, for example, we could count the number of different silicon chips after 1950, and we would see exponential curves.

This is a picture of increasing returns.  Innovation leads to increasing innovation.  Invention is the mother of invention.

Why does innovation tend to accelerate from an initial spark?  There is no established research or theory about this.  However, I can list four factors that I think accelerate innovation.

An abundance effect:  Successful innovations generate resources that can be burned up in the quest for more innovation.

A network effect:  The more things that you have, the more ways you can combine them.  If you start inventing machines that produce new materials, you can combine the results into even more new types of machines.  It’s clear that this effect is strong in industrial economies, and in software production, and it is used extensively by bacteria, which swap DNA.

A platform effect:  Innovation builds on previous innovation.  Its primary quality is that it produces a very durable good (which I called design) that lasts forever, builds up in layers, and can be reused an indefinite number of times.  This can create a new platform for further innovation.  This is a supply-side effect.  If you invent multi-cellular life, you can supply a Cambrian explosion in species diversity.   If  you invent silicon chips, you can make a lot of new stuff.

Evolutionary space:  When creatures moved from sea to land, they found a lot of empty space.  There was room for a lot of new species.  When users start reading the Internet, they create a lot of empty space for  new media.  This is a close relative of the platform effect, but on the demand side instead of the supply side.

In this list we have two factors with increasing returns.  The network effect gets stronger as the network gets bigger, and the abundance effect gets stronger as you invent more things that deliver abundance.  So, that can explain increasing return to innovation.  We also have two factors with decreasing returns.  A platform has limited width.  Eventually you use up the new space on top of a big innovation.  And, you also use up evolutionary space.  So, it’s not surprising that industrial innovation has a decreasing returns phase.  If you look at the number of different products in an industry over a longer period of time, you get a leveling off, the famous S-curve.   Eventually, there is a decline.

So far, we have only seen the exponential phase of our great experiment in human industrial production.  Will this continue to accelerate, producing more amazing things, and more abundance?  If it is not bounded, it will eventually go vertical, and we will burst through to a new future in a great rush.  This is the hypothesis popularized by Ray Kurzweill. We might find a great arc of technology moving from the slow evolution of DNA, to our current use of silicon, to an evolutionary technology that constantly accelerates itself.  This is implied by my claim that there is a theory and a technology that can accelerate innovation.

However, I don’t see this as a foregone conclusion.  Subjectively, it seems that we have reached a phase where industrial innovation is decelerating.  The difference in my grandmother’s lifestyle between 1900 and 1955 was enormous.  The difference in lifestyle between 1955 and 2010 is much smaller – some new gadgets, and a small increase in the effectiveness of medical treatments.  Pretty much all economic growth in that period can be traced back to one “platform” – the increased use of oil and electric energy.  This continues today.  Chinese economic growth can be mostly linked to their rapidly increasing use of energy, not innovation.

I recently saw an economist who noted that the American economy (which is not using increasing amounts of energy) and stock market have stagnated in the last ten years because we didn’t get a big lift from new innovations.  On the other side, Louis  C.K. tells us about stuff that really is amazing... but planes were invented 100 years ago.

So, where should we invest to get lift?  I hypothesize that you want to invest where you see increasing returns.  The competition will be more intense than in decreasing return industries, because of the “non-rivalry” attribute that  allows anyone to use new design.  However, the lift from increasing returns is powerful enough to counter that. 

One obvious case is in biotechnology.  Most biotechnology investment goes into developing drugs.  However, the productivity of this investment has been declining rapidly.   Increasing amounts of money and ever-more ingenious new science and technology produce fewer and fewer new drugs.  They tend to produce new drugs that aren’t as good as the old ones.  So, we can see that this effort has diminishing returns because of space.  The “drug targets” in a human body have been used up.

On the other hand, you could use biotechnology to produce new materials and new organisms.  This is the approach taken by companies making new types of biofuels.  I would expect this effort to lead in an increasing number of directions, and benefit from increasing returns.

So why aren’t more biotech companies investing in new materials and organisms?  Because it’s risky, and wasteful, and they have no idea where it will go.  It requires real innovation.

In a previous Sunday article on the theory of innovation, I emphasized the balance between optimization and exploration, and between order and chaos.  In order to innovate, we need to do two things:

1) We need to make something new. (chaos)

2) That new thing needs to persist over time. (order)

Christopher Langton coined the phrase “edge of chaos” to describe how this happens in a computer model called a cellular automata (CA).  A CA is a grid that uses rules to calculate changes in the color of cells on the grid, over time.  There can be a very large number of rules, but Langton found that they can be described by a parameter he called “lambda”.  Lambda tells you what proportion of the cells on the grid will not become black in each time period.  If lambda is low, the grid just goes to black – perfect order.  If lambda is high, the cells on the grid oscillate – chaos or noise.  In between, you can see lines and patterns on the grid, representing things that were created, and then persisted.

Follow this link to find an applet that actually runs a CA, and lets you play with the lambda variable (the slider on top).  Select example “ca_params_4” under the “examples” menu, and let it run for a while.  You will see how patterns can start, spread, and interact.  In example 4, you will get random noise if the lambda slider is over about .44, static black under .14, stable triangles between .14 and .22, and interesting behavior between .22 and .43

A business organization is like this “params_4” system just under .22.  Most of the time, it is out of the chaotic zone (except on Mondays) and it produces the same products and services (the red triangles).

Sometimes, you can push it closer to the edge of chaos, and it starts to do new and colorful things.  However, you generally need to push it.

It turns out that if you pump energy into a system you can get to “self-organized criticality” – a natural progression to the edge of chaos state where interesting things can both happen, and persist.

In practice, most organizations stop innovating and exploring, and start optimizing in a very small area, as soon as they find something profitable to do. They back off from the critical point, toward order.  As noted before, innovation is very inefficient and wasteful.  It involves trial and error, mostly error. Businesses want to be efficient.

It’s often laughable when big, successful organizations say that they want to be more innovative. Sometimes, they say they want to be “efficient innovators” which is almost a perfect contradiction.  These organizations are big and successful because they found something profitable to do, and they do it efficiently. They hire people with specific job descriptions, to fit into this efficient process. They don’t want employees to do something completely different. It would wreak havoc with profitability.

This isn’t just a human tendency.  When I run computer models of genetic programming, one of the most powerful forces is an evolution toward stability.  This can happen very fast – within three or four generations of starting.  And, it can produce a population that basically stays the same for hundreds of generations after that.  In genetic programming, the mechanism is a buildup of redundancy in the genome, to the point where  you can cut and splice the computer program “genomes”, and almost always get the same algorithm.  Back in 1992, I described this as “defense against crossover”.  The objects of a genetic operation tend to evolve the genome to defend themselves from changes.

We see the same thing in biological evolution.  Species are actually very stable.  It is not common to see entire species evolve continuously, as Darwin suggested they might do.  What we actually see is “punctuated equilibrium” – a species that is the same for a long time, and then suddenly displaced by a new species.

That’s why I am so sure that we can make evolution run faster. It’s designed to be slow.

Does innovation stop when we switch to optimization?  No.  It doesn’t stop.  It just gets delayed, and pressure builds up, like the pressure for an earthquake.  This pressure buildup leads to the signature phenomenon of self-organized criticality, which is called a “power law distribution”.  Basically, the system builds up enough pressure so that innovation doesn’t just happen incrementally.  It can happen at any scale.

One visible effect in biological evolution is punctuated equilibrium and mass extinctions.  One visible effect in industrial economics is a distribution of startup sizes, where some startups can grow to be as big as the biggest previously existing companies.

So, there are forces which stop innovation.  Necessity – a lack of resources – is one of them.  Necessity is NOT the mother of invention.  Necessity is the mother of efficiency.  Efficiency is in some ways the enemy of invention.  Where we see poor people, poor countries, and declining industries, we do not see innovation. Where we see "east coast" venture capitalism, we do not see invention.  We see efficient and conservative lifestyles and operations.

What is the real mother of invention?  I will cover that in the next Sunday article on the theory of innovation.

Last Sunday I introduced a theory of innovation by defining innovation as a production process that produces something called “design”.  Today I will introduce the raw materials that feed this process, optimization and exploration.  We will confront fountain of innovation itself, the dueling horsemen of genesis, the yin and the yang, order and chaos.

In the beginning God created the heaven and the earth. And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters. And God said, Let there be light: and there was light.  And God saw the light, that it was good: and God divided the light from the darkness.
King James Bible

If yin and yang do not exist, the One cannot be revealed. If the One cannot be revealed then the function of the two forces will cease, Reality and unreality, motion and rest, integration and disintegration, and clearness and turbidity are two different substances. In the final analysis, however, they are one.
Chinese philosopher Chang Tsai

Innovation is a process of trial and error.  You try something new, and if it works, or it survives, or if you like it, you keep it. Otherwise, you call it an error and you throw it away.

We can also visualize innovation as a search process.  When you try something new, you are exploring.  When you find something you like, you keep it, and then you improve it.  You start optimizing.  In a simple model of this, you are hiking through a mountain range, looking for the highest peak.  You roam around looking for a high slope.  That’s your exploration. Then, you climb up.  That’s your optimization.

Our “hike to the top of the mountain” task sounds easy enough, because you can just look up to see the mountain tops.  However, I do this type of hiking in Maine, where it is often very foggy, and difficult to see through the trees.   If you are trying to do real innovation, the fog will be very thick, and you won’t be able to see the future.  In fact, you may not be able to see which direction is uphill.  In these circumstances, you can’t just go uphill.  You have to do little explorations, striking out 20 paces in one direction, and then coming back and trying a different direction if you didn’t find the way up.    Maybe one person holds a flashlight, and another circles, as I found myself doing one foggy night.  In this case, optimization is a type of exploration, only with smaller distances.

Optimization and hill climbing are just explorations/trial-and-error over small distances.  We aren’t going very far, so we are pretty sure that we will end up almost as high as the place we started, and there is about a 50% chance we will end up higher.  We have a good chance of success.  It seems a lot easier than doing large-distance exploration, in which we have to climb down through gullies, across plains, and the occasional grand canyon.  However, there is a problem.  We eventually get to the top of our hill, and then we don’t know if we are on the highest hill.  Maybe if we strike out over a longer distance, we will find a much higher peak.  It is a constant tradeoff.

So, we have a dual strategy.  We explore.  We try some big things.  We roam around and map out some mountains.  We start new businesss or new product lines.  Then, if we find opportunities that are good  enough, we shrink the options and start optimizing.  And, we are never sure.  Should this button on the web site be blue, or red, or maybe throw in some quotes about order and chaos from ancient texts?  That’s just presentation.  We can use Google Web Optimizer to test that with customers and find out what works best.  What if I offer a different feature set?  That’s a lot more work to test.  Should I even be in the business that I am in?  I might get a million dollars with this product launch, but there is a 26-year old guy over there who is making billions of dollars with a different approach.

At this point, if you are a troublemaker, you might raise your hand and say “What about the impact of creativity?  Is it fair to compare leaps of human creativity with purely mechanical processes like evolution or A/B market testing?”  My answer would be, yes.  I’ll define a creative solution as something that is found very efficiently (aha! we don’t know exactly how the creator found it), and that is good but not provably optimal (it’s good as art or innovation or practice, not mathematically).  You can make a purely mechanical process that find solutions with these attributes.  For example, it is not practical to find an optimal solution to NP-hard problems like the traveling salesman problem – the shortest path to visit many unique cities. However, you can make a computer that can find very good solutions with trial, error, and careful selection.  It looks for short paths that go to a subset of cities, and then runs trials where it combines them together, and keeps the shorter results, and makes variations, and evolves shorter and shorter solutions.  These solutions are found efficiently, and are not provably optimal.  The key input is a sort of domain knowledge that tells the computer what to optimize and when to take a creative leap, basically, when to grab a sub-path solution and search in its vicinity.  This is the same type of domain knowledge and “discernment” and boldness that we find in creative individuals.

When we think of innovation as trial and error, it’s clear why it is not efficient.  The more trials we do, the more errors we get. We can reduce the number of errors by doing more optimization – more trials of close-by ideas that we are pretty sure will work as well as what we have, but then we don’t get the big gains. There is an uncertainty principle of innovation that is similar to the quantum uncertainty principle. If you want to make bigger gains, you have to accept more and worse errors. There are two things that you can do to make your search more efficient. Make sure that you are not repeating yourself – not working with too much order.  And, make sure you are searching in the right general area – not working with too much chaos. You can control the size of your steps.  After that, you are dependent on the underlying landscape to yield its treasure – and a lot of errors.

In our innovation manufacturing system, we have dial that we can turn from order to chaos.  If we turn it toward order, nothing happens.  We make only very small improvements.  If we turn it toward chaos, we generate a lot of big steps.  Our product is constantly changing, and not for the better, and nothing works right.

Order and chaos are the major inputs to evolution.  Mutations are the product of chaos, and most large mutations are fatal.  A creature born without a heart, or with extra heart parts, will die.  Most living things are highly optimized – they are the top of their little hills – so any step away is likely a step down. Evolution adds order, usually in the structure of the genome. Genomes are carefully constructed so that the result of reproduction operations is very similar to the inputs.  I saw this very strongly in my genetic programming experiments, where evolved software quickly evolved redundancy that made it behave the same in future generations, even after cutting it in half and inserting random operations.  Try doing that to non-evolved code.  This defensiveness is why natural evolution is often very slow.  It’s evolved to slow itself down.  It can be speeded up.

Chaos alone is bad for innovation.  It rips you away from good things before you can use them and refine them.  Everything changes, and nothing improves.  Order alone is bad for innovation.  It stifles innovation, freezing you in place.

As innovators, we need a balance.  As we will see on a future Sunday, we need a system on the edge of chaos. 

Next in series: Backing away from the edge of chaos

When we talk about accelerating software development, we’re really talking about accelerating innovation.  We can learn from other forms of innovation, like evolution, which share a common mechanism of creation.

Birds can fly about 100 miles per hour.  A jet airplane designed from similar principles, but optimized for speed rather than efficiency, can go 2000 miles per hour.  Evolution is famously a very slow process.  But, it’s not always slow.  And, with a theory of innovation, it’s likely that we can make an artificial version of evolution that roars like a jet airplane.

I’ve been collecting ideas about a theory of innovation for years, and I will take some time on Sundays to share them here.

These ideas come from a variety of sources.  The most important source is the study of natural evolution.  A man can make a better mousetrap, but evolution can make a cat.  And evolution is the starting point for humans, and hence all human activity.  Some argue that biological evolution produces humans, and humans produce silicon and DNA technology, and we’ll engineer that so that it can create and evolve, and it will all be one big continuum of creation.  

If that sounds scary, you can can comfort yourself with the fact that we really don’t understand evolution yet.  Evolution turns out to be very intricate.  It’s not just competition, and survival of the fittest, with one caveman clubbing another caveman over the head and a selfish gene emerging smugly from the fray. It’s also cooperation, and symbiosis, and evolvability of genomes, and so on.

Some of the other inputs are:

• Artificial evolution, including experiments with genetic programming – evolving software inside a computer.
• Thermodynamics.   This is the really big picture.  Is there anything new under the sun?  How do new things come out of undifferentiated “macro states”?   In thermodynamic theory, noise is the raw material of information, but how does noise become meaningful information?  While we’re at it, is there a meaning of life, and will innovation ever come to an end?
• Human creativity.  What makes an idea “creative” as opposed to logical or obvious?
• “Dynamical systems theory” as it was so exuberantly discussed at the Santa Fe Institute, with universal ideas like power laws, emergence, chaos, and order.
• Industrial innovation, including the art and business of invention.
• Economics.  We need an economic theory of innovation. We don't have one yet, but we have a few ideas.

Innovation is clearly an economic activity.  We spend a bunch of money on  it, and we expect a return  on  investment.  It’s the most important factor  of production, the most important product, and the most important determinant of the wealth of nations. When I started studying innovation, I wondered why economics doesn’t have much to say about it.  If you start from a theory of economics, innovation is a very strange beast.  I tried to tame the beast by defining some economic properties. Here is the first draft.

First, innovation is a process of production, sort of like pumping oil or making widgets.  Only, we aren’t producing tangible goods like oil or widgets.  Innovation produces something that I will call “design”.

Design has some strange properties, which will be confounding to economists, but useful for understanding the software business.

Design has the property of “non-rivalry”. That means that two or more people can use it at the same time.  Once I figure out how to make a wheel, you and I and some guy in China can all make similar wheels.

Although innovation can be expensive, it’s output, design, doesn’t have any scarcity or marginal cost.  As an innovator, you don’t get paid for the work that you do, and as a buyer of design goods, you don’t pay for any particular cost.  Instead, you pay for things like “ecosystems”. This leads to increasing returns, which is why our world is so crazy and constantly flying out of equilibrium. 

Design also has the property of durability.  It can last forever. I can still make a wheel thousands of years after someone figured out how to do it.  Because design is durable, it builds up in layers that are very deep and impossible to measure. How much design is in a computer?  Well, how much design is in the paint on the computer, or the chemicals in the paint? We can deal with these questions using tricks like fractal math, but that’s beyond typical economics.

Software is a pure design good.  We can use as many copies as we want, and it never wears out.  Therefore, making software is a process of pure innovation.  It seems strange to say so, since much of the work of making software isn’t lightbulbs going off with “aha” moments, and the work of making software is in fact usually a lot like making widgets, and most software isn’t very innovative.  But, that only proves that innovation is a very inefficient process.   I’ll write about that on some other Sunday.

Next in series: Optimization, exploration, and the dueling horsemen of genesis

I'm in Berlin at SAP's Innovation Weekend, where we are introducing SAP Code Exchange, a portal for shared source projects related to the SAP platform.  For the first time, hundreds of thousands of professional SAP developers will have a place to find utility code and work together on bigger projects.  To build Code Exchange, we integrated SAP's developer network with a customized Assembla instance.

Open source code and processes are a key component of modern fast, lean, and global software development.  I like to think about practices, things that you need to know to get benefits from this component.

• First, you need to know how to do research to find and adopt appropriate code and frameworks.  That makes development faster.
• Then, you learn to contribute to open source projects, so your customizations benefit from shared maintenance.  At a company level, this leads to sharing code with customers.
• At an advanced level, you organize open source and shared source projects to support your vision of the world, your platform, and your users.   Every big software vendor will eventually have to learn and deploy these advanced processes, as SAP is doing.

SAP's Innovation Challenge

Project leader Rui Nogueira says "We built Code Exchange to encourage co-innovation, with innovations moving more quickly from customers to SAP, and from SAP to customers."  Kaj van de Loo, SVP of Technology strategy, talked about SAP's openness initiatives, and quoted Ronald Reagan, "tear down this wall".

Code Exchange was shaped to fit with SAP's shared-source licensing, their proprietary code management, and their active developer network.

SAP has a long tradition of sharing code, predating the open source movement, because when you buy their installed ERP apps, you get the source code.   Very cool.  In the license agreement, customer and integrators must agree to certain terms.  Code Exchange creates a compatible environment by asking all users sign a “terms of use” agreement that is compatible with the SAP shared source license.  Once they sign the terms of use, they get into a walled garden where everyone shares with common license terms.  They are breaking new ground with a type of shared-source licensing that can open up high-value, proprietary systems to the power of  sharing.

SAP apps are beyond the size that I can imagine managing, with about 150,000 screens.  SAP has specialized systems for version control and deployment to handle this beast.  We need ways to move code between their Netweaver development environment and internal ABAP repositories, and standard svn and git repositories.  For example, SAP advisor Gregor Wolf has been working with collaborators on the SAPlink import and export mechanism.

SAP has existing portals for their developer network, and the customer network (with 2M users).  They just released a new project collaboration environment based on Jive SBS.  The SAP team wanted to add repositories and ticketing, without confusing anyone.  Their vision was radical simplification.  We stripped Assembla down to one project template with just the extra tools they wanted – Subversion, Issues (our Tickets), Releases (our Milestones), and Chat.

What did we do?

So, how do we customize and deploy Assembla for an enterprise customer?

1) Start with our Private Assembla configuration.  This is a stand-alone configuration of Assembla without any of the subscription features, with source code.

2) Set up staging and production servers.  We got permission to use cloud hosting (see below) and started some servers in the Amazon EC2 datacenter in Ireland, bringing us inside the EU.

3) Fork the Assembla code into a new git repository.  With this structure and thoughtful engineering, we can easily merge new versions with our SAP customizations.  We also set up ticketing in this new space and invited the SAP team to work with us. We set up build scripts and continuous integration for the staging server.

4) Configure and skin.  We brought in a header and styles from the SAP Developer Network, as shown in the screen shot.  Strip it down to one project template and configure that template space.

5) We made a number of customizations, including: login/authentication to implement single-sign-on with their domain cookie; simplified start page;  release file posting; API authentication, and some API upgrades.

6) The Jive portal team at SAP integrated Assembla using the REST API.  They used the Space API to add matching projects, and the Tickets API to list  and post Issues, a subversion browser, and a new Milestones API to show current release files.

7) SLA and security compliance.  We asked for some adaptations to SAP's normal security policies, so that we could use cloud hosting.  We also took advantage of some of the work that Amazon is doing to meet enterprise security requirements. This is by itself an interesting subject, if you are involved in moving enterprise apps to cloud hosts.

Here's a video interview from Innovation Weekend.

Juan Enriquez stepped up to remind software developers that the most significant future work in coding is likely to be done with genomes.  He works in the emerging industry of synthetic biology.  He's suggesting applications like "improving the climate on Mars, designing human organs that fend off disease or cloning cattle that produce powerful vaccines in lieu of dairy products."

This is indeed a trend.  Last year I went to visit Tom Knight, one of the inventors of computer science, at his office in the MIT's Stata Center.  I was surprised to find him surrouned by petri dishes.  Always the pioneer, he has moved his attention from digital code to genomic code.

Coders can bring their skills to synthetic biology.  We can turn this around.  Biology can also bring it's magic to coding. 

I propose the basic idea that innovation has multiple forms - biological evolution, industrial/economic innovation, and human creativity - but a single underlying set of mechanisms.  The mechanisms are based on the simple variation and selection process that we learn about when we study darwinian evolution.  It's called "exploration and optimization" in engineering circles. 

However, the machinery is actually a lot more complicated than that, which is why we still don't understand it fully.  We can run evolution on computer programs, "genetic programming," but the results aren't even close to what we get with real evolution, yet.  That's why it's such a good area for further study.  Code is the product of innovation in its pure form, and if we learn how to make one type of code, we can make other types of code.

Evolution takes a long time to deliver results.  It can go 1.8 billion years just working on bacteria.  However, it appears to be "punctuated equilibrium".  Most of the action is packed into short bursts.  We see the same pattern when we run genetic programming.  By creating an "arms race" scenario and taking away the tendency to equilibrium, we can prod evolution to run much faster.  This creates an artificial process that is related to normal evolution in the same way that a Mach 3 jet airplane is related to a bird.

This is a long term answer to how we are going to generate bigger projects and bigger wins than Web services.  Once we understand how innovation and evolution work, we can apply those ideas to computer code, to synthetic biology, and to hardware.  How, as a species, we are going to deal with the effects of this takes us into the realm of science fiction, but at least the current innovation deficit will be out of the way.

Here’s another post on my long and winding attempt to crack the code of genetic programming.  Genetic programming, or the evolution of computer programs, will revolutionize the speed and effectiveness of solving hard problems, and it will be something that we can plug into systems like Assembla to spit out useful subroutines.  First, we must get over some obstacles.  After 17 years of research and 10,000 times the computing power, our ability to make complex software has not measurably improved.  It appears that there are a lot of things that we don’t know about how evolution works.  I will crack the code yet.  However, it will take a while.  On the way, I hope to learn a lot of amazing things about evolution and it’s anthropic twin, innovation.

Saturday I went to see real genetic geneticists at a “Future of Life” event at Harvard and got to see amazingly smart researchers say something about their latest work.  

Craig Venter gave an overview of his recent work – collecting and sequencing 20M genes from ocean bacteria, approaches to growing biofuels, and experiments in building a bacteria from a naturally-inspired but minimum-sized synthetic genome.

George Church showed some even more recent work, including his hot-off-the-press in-vitro ribosome (a ribosome builds proteins and is extremely useful if you want to, for instance, generate a lot of different proteins and see what they do) which can be constructed outside the cell with 158 genes.  He also had very interesting things to say about biofuels.  He likes the idea of using algae for making biofuels because they can make exactly the fuel you want – typically clean-burning diesel oil.  He has worked out some clever ways to get them to emit the oil, so in theory the oil can be skimmed off the top of the pond, rather than pressed out with great waste.

He taunted me for my interest in evolvable software, observing that future bio-reactors should be able to test 20 trillion combinations in a batch, a scale that software will never match.  Venter has his 20M marine genes to sort through.  Church has a machine, a chip, that can spit out genes with many slight variations.  So, once they crank up the synthesis machinery, they will have a lot of raw material.  And, they have moved far beyond drug discovery in the search for new fuels and materials.

However, as they described their struggles to build a complete organism, and I contrast it with the relative ease of manipulating gene variants, it seems to me that they are running up against the same problem that we ran up against with genetic programming.  They can make simple things, and select for them, but not complex things.  There is a missing link in this artificial evolution, too.

Even so, I would never bet against these guys.  They are using machinery that is getting exponentially more powerful, at rates much faster than Moore’s law.  I would bet on algae as a fuel source even though the VC are pulling back (probably a good sign for returns) – 

Yesterday I had a chance to talk to one of my heroes, Stuart Kauffman.  As I am one of the six people to actually finish his amazing book Origins of Order, we had a lot to talk about.  He’s moved up to “Reinventing the Sacred” and doing a stint at Harvard Divinity School.  His basic point,  as he explained it (I haven’t read the book) is that the deterministic world of Laplace doesn’t have any room for innovation.  In a mechanical, predictable universe, everything that can be is prefigured and can be precomputed by its initial conditions.  The mathematics of chaos makes this unlikely, however, and the emergence of new kinds of life is unexpected.  My interpretation of his theory is that innovation is real, that new things come from life, things that didn’t exist before, and that progress exists in the universe.  I agree with that.

What’s so great about The Origins of Order?  It was one of the first things that I read which acknowledged that natural selection is actually a weak force compared with the innate order in genetic machinery.  That’s one of the things we need to realize about artificial evolution.  The genetic machinery is just as important as the selection machinery.

Kauffman agreed that most of the a-life experimenters he talked to have discussed the same issue – the difficulty of making anything beyond a certain level of complexity.

I proposed to him that our current emphasis on selecting the best item out of a genetic algorithm is like trying to build a house by selecting good bricks.  You keep pulling apart the house to find the best brick.  To build the house, you need to let the bricks fit together.  You need collaboration, co-evolution, and symbiosis.  The trick to determining the brick size and relationships might be found in his N-K network theory.  We’re going to get together to think about that.

One of the interesting things that I learned is that climate change can be very extreme.  For example, our oxygen-rich atmosphere is chemically unstable and is maintained only because we are surrounded by plants that emit oxygen.  The early earth had an atmosphere that was “anoxic”.  What I didn’t know is that scientists believe that earth went through about a billion years of a highly sulfuric atmosphere.  I had previously heard that eventually, when a lot of continents moved to the equator and started reflecting sunlight, the earth went into a super ice-age called “snowball earth” in which the oceans froze one mile thick.  These ice ages were interspersed with periods of scalding heat and acid rain.  Various kinds of bacteria happily lived through these various conditions, and contributed to the Permian explosion of multicellular life after the snowball earth thawed.  So, be happy with today’s weather.

I have occasionally posted on the topic of genetic programming, as a possible way to drive rapid software development.  Some of these posts have drawn requests for more material.  For those who are interested, I posted a summary of my (very old) research and source code here.   The material includes Doodle Garden, a windows application that evolves pictures, the GPQuick C++ framework, sample code for using a Meta-GA to evolve financial trading rules, and a discussion of advanced techniques like distributed clusters, Meta-GA, and symbiosis.

 

The announcement of a petaflop computer has resurrected my dream of computers that write their own software and come up with their own ideas.  That would take us a long way toward delivering our promise of "accelerating software development", with a completely different mechanism than the organizational mechanism I am using now. I've written before about ways that computer-generated code could be woven into a rapid application development process.

 

15 years ago I worked on genetic programming – the evolution of computer programs. However, I ran into a severe shortage of computing power, eventually filling a rented house with racks of single board computers (top of the line, 40 megahertz 80486) in clusters.  Genetic programming requires vast amounts of computing power to generate and simulate millions and millions of functional cases, and the computing power of my homemade cluster was pathetic by the standard of today's desktops. Genetic programming is ideally suited to make use of parallel computing, and it can make good use of almost any topology – clustered, or MIMD, or SIMD/vector. The IBM monster is a useful combination of these architectures - Cell vector processors out of the Playstation combined with independent Opteron processors out of a PC.

 

If we can't get useful results from these new computers, we're doing something wrong. And, in fact, we are doing something wrong. We fundamentally don't understand evolution. The evidence is in that increases in programming capability of 1000 or more over previous systems have not significantly increased the rate of new discoveries. There seems to be a limit on the complexity of the evolved artifacts, and with the faster computers, we are just hitting those complexity limits faster.

 

There is an "existence proof" for the effectiveness of evolutionary methods.  DNA-based life forms are amazingly complex.  A human genome has at least 30,000 genes.  The most complex genetic programming artifacts have only about 100 active elements.

 

What will get us over this hurdle?  I have some ideas, a three point plan.

Cooperative mechanisms 

The high-school depiction of evolution as a survival-of-the-fittest competition is crude and limiting.  In the real world, organisms and genes co-evolve and cooperate in expanding systems.  We still haven't fully absorbed the lessons of Lynn Margulis and others studying symbiosis and cooperation.

Genome structure

If you try genetic programming, you will find that running the evolutionary algorithm has an immediate and powerful effect on the genomes - the code fragments that you are evolving.  The first thing that they do is expand and add redundancy - something like the introns that make up most of an animal genome.  This change in the genotype moderates the effect of mutation operations, without changing the phenotype - the output - and it's crucial for making the process run smoothly.  So, we see evidence that the structure of the genome is a hugely important player in the game of evolution.  It's no accident that our animal genomes contain chromosomes which cooperate and co-evolve, or that they contain lots of "junk" DNA.  These structural features are what make complexity possible.  If we understood why the genomes are set up this way (we don't, mostly don't think about it), we could make much better genetic programs.

Discovery with Meta-Genetic Algorithms

We have questions cooperating systems and appropriate genomic  structures. Now that we have big computers, we can answer these questions with a meta-genetic algorithm machine.  This machine will generate new genomic structures and population parameters (new types of Genetic Programming), and run the lower level genetic programming runs to see the results for each set of parameters.  This is exactly the kind of machine that I built to run on my primitive rack clusters.  In one run, it could answer questions a the level of PhD thesis.  For example, the Meta-GA very quickly identified the programming operations and inputs that would be most effective.  However, it had to run for days because it would run thousands of runs with the lower level GP to find the higher level parameters.  Hence the problem with computing power.

 

With the more powerful machines, we can resurrect the meta-evolution layer, look at a range of genome structures, and find a way to harness cooperative evolution.  Did anyone follow that?