The announcement of a petaflop computer has resurrected my dream of computers that write their own software and come up with their own ideas.  That would take us a long way toward delivering our promise of "accelerating software development", with a completely different mechanism than the organizational mechanism I am using now. I've written before about ways that computer-generated code could be woven into a rapid application development process.

15 years ago I worked on genetic programming – the evolution of computer programs. However, I ran into a severe shortage of computing power, eventually filling a rented house with racks of single board computers (top of the line, 40 megahertz 80486) in clusters.  Genetic programming requires vast amounts of computing power to generate and simulate millions and millions of functional cases, and the computing power of my homemade cluster was pathetic by the standard of today's desktops. Genetic programming is ideally suited to make use of parallel computing, and it can make good use of almost any topology – clustered, or MIMD, or SIMD/vector. The IBM monster is a useful combination of these architectures - Cell vector processors out of the Playstation combined with independent Opteron processors out of a PC.

If we can't get useful results from these new computers, we're doing something wrong. And, in fact, we are doing something wrong. We fundamentally don't understand evolution. The evidence is in that increases in programming capability of 1000 or more over previous systems have not significantly increased the rate of new discoveries. There seems to be a limit on the complexity of the evolved artifacts, and with the faster computers, we are just hitting those complexity limits faster.

There is an "existence proof" for the effectiveness of evolutionary methods.  DNA-based life forms are amazingly complex.  A human genome has at least 30,000 genes.  The most complex genetic programming artifacts have only about 100 active elements.

What will get us over this hurdle?  I have some ideas, a three point plan.

Cooperative mechanisms 

The high-school depiction of evolution as a survival-of-the-fittest competition is crude and limiting.  In the real world, organisms and genes co-evolve and cooperate in expanding systems.  We still haven't fully absorbed the lessons of Lynn Margulis and others studying symbiosis and cooperation.

Genome structure

If you try genetic programming, you will find that running the evolutionary algorithm has an immediate and powerful effect on the genomes - the code fragments that you are evolving.  The first thing that they do is expand and add redundancy - something like the introns that make up most of an animal genome.  This change in the genotype moderates the effect of mutation operations, without changing the phenotype - the output - and it's crucial for making the process run smoothly.  So, we see evidence that the structure of the genome is a hugely important player in the game of evolution.  It's no accident that our animal genomes contain chromosomes which cooperate and co-evolve, or that they contain lots of "junk" DNA.  These structural features are what make complexity possible.  If we understood why the genomes are set up this way (we don't, mostly don't think about it), we could make much better genetic programs.

Discovery with Meta-Genetic Algorithms

We have questions cooperating systems and appropriate genomic  structures. Now that we have big computers, we can answer these questions with a meta-genetic algorithm machine.  This machine will generate new genomic structures and population parameters (new types of Genetic Programming), and run the lower level genetic programming runs to see the results for each set of parameters.  This is exactly the kind of machine that I built to run on my primitive rack clusters.  In one run, it could answer questions a the level of PhD thesis.  For example, the Meta-GA very quickly identified the programming operations and inputs that would be most effective.  However, it had to run for days because it would run thousands of runs with the lower level GP to find the higher level parameters.  Hence the problem with computing power.

With the more powerful machines, we can resurrect the meta-evolution layer, look at a range of genome structures, and find a way to harness cooperative evolution.  Did anyone follow that?

Last weekend I spent 1.5 days at Railsconf. It was a 6,000 mile round trip squeezed into a weekend between busy working weeks, and interrupted by the datacenter explosion that took down our site. So, I didn't get to see many technical talks. I did get a feeling for what's hot: cloud computing.

The vendor booths were dominated cloud computing infrastructure providers. Amazon was there representing EC2, the key enabling infrastructure. Heroku showed rails development and deployment on EC2, backed by their recently announced VC funding. Morph showed a more enterprise-oriented service for hosting and scaling Rails and java applications on EC2. Rightscale showed a polished version of their more general-purpose EC2 cluster management tools, backed by their recent VC investment. Engineyard, which runs their own full-service, fully staffed virtual server cloud, was doing a booming business, backed by their recent VC investment. They talked about releasing some of their management tools as open source projects. Clearly the Silicon Valley money is going into infrastructure at this stage. Sun was showing off an Assembla-like workspace that will showcase their ability to host team applications on their cloud computing infrastructure. Google was the big player that wasn't there, as their recently announced Google App Engine runs only Python and faces off against the Ruby-centric Amazon cloud.

I'm a convert to cloud infrastructure. We have started moving our clients onto Amazon servers. Assembla.com was hurt by our dependence on a more traditional server topology, and we'll be moving it into a virtual server cloud over the next month.

There are number of implications for software developers:

• You can get instant staging servers. Start releasing even faster. We find this very useful, and we will build it into Assembla workspaces.

• You can put high-volume fragments of your app on high-volume hosts. For example, the big selling point of Google App Engine is that it can handle a very large number of HTTP requests and survive a slashdotting or a facebook-scale rampup. If you are considering adding a facebook front end, or a component like polling chat that is very request intensive, you have an incentive to throw that onto Google App Engine and link it to the rest of your systems with web services.

• You need to think about file management. Files in a system like Amazon EC2 don't just live on a permanent, homogeneous SAN. You need to back them up to a storage cloud (S3) and restore them when you move to new virtual servers. Amazon and Google offer BigDB and SimpleDB, which aren't exactly the same as the databases and persistent stores you are used to.

• We have to move beyond HTTP request based applications to new mesh-computing architectures that send messages in real time. I saw some interesting projects based on the XMPP jabber protocol, Erlang, and the eJabberd server, which is a jabber server written in Erlang. To back up, XMPP is a protocol for sending messages to jabber instant messengers, but it can also be used to send messages between servers, or to route messages through a high-volume cluster to handle Twitter style applications. Erlang is a computer language that naturally distributes processing to multiple servers. We are seeing these tools deployed to manage servers in the cloud. However, we are also seeing applications in which browser ajax panels connect directly to XMPP servers for real-time chat or dashboards.

• Having a professional-quality repository with a build, stage, release process is important for cloud applications, because your team and your code are likely to be distributed in the cloud just as your servers are.

The “User story” or “use case” is a powerful tool for designing software. You pick a user, or “actor”, with a specific goal, and you diagram the process / workflow by which the user employs your system to accomplish the goal. Many of the projects that I see do not use this tool effectively, and as a result end up lengthening development cycles. Here is a list of issues, and fixes, that will make your stories more useful.

Generic user: This is the single most damaging user story mistake. You define an imaginary user with very broad needs, a “scientist” or a “mother”. You run into two problems that kill your ability to release software early and often. First, this generic user has a very broad set of needs. You end up looking at a huge number of features. Second, you can't actually call this user and ask specific questions - “red or blue?”, “do we need a lookup here?”. So, you don't get any of the details that you will need for implementation. When you diagram a user story, you should have a specific user in mind. This user only has a small number of specific requirements, and this user can answer detailed questions.

One demanding primary or reference user:  You can go too far and pick one user. This causes delays because this one user may have some one particular feature they really really want, even if it is irrelevant for most other users, and hard to build. They probably volunteered to be the reference user because they have an agenda. It pays to shop around and find less demanding users for your early releases.

Unfamiliar user: Software design is most effective when you are designing software for yourself. The user stories in this case are easy to get, accurate, and nuanced. A high proportion of really good software is built by people for their own needs. If you find yourself building software for users that you don't understand, you have a difficult problem. You can learn a huge amount from interviews with the intended users, but you will miss some things that will only come out when they test the prototype. I suggest two maneuvers in this case. First, you should make yourself into the user by walking through the process yourself. You will be like the hotel company CEO that takes a day a year to work the reservation desk, and ends up leading a rebuild of the entire reservation system. How many times have we heard that story? One way to get started is with a paper trail. I like to collect all of the pieces of paper that go into or come out of a business process, and pin them into a storyboard. You find a lot of data structures that way, and some workflow. Second, you need to partition your user base and find users that will take early alpha and beta releases. Your user stories will come more by feedback than by initial specification, so getting releases in front of external users becomes more important.

Non-user features:  The user story concept isn't very useful when you are working on features that users don't see: infrastructure, architecture, middleware, administration, etc. You can make it more useful in these cases by remembering two things:

• System administrators and developers are users too. They have user stories.

• If you really want to be agile, you build the underlying infrastructure As Late As Possible. You get releases to users faster, and you save yourself a lot of work in the cases where you find you don't end up needing the infrastructure. So, there is an argument to build the minimum amount of stuff that supports a visible user workflow.

Vague stories: Vague user stories without a specific user or actual example task will not give you the details that you need to do an implementation. You aren't doing any damage with them, but you are wasting your time, because you aren't learning very much.

Textual stories: If you have been in software development business long enough, you will eventually encounter a specification that has been built with use cases and formal requirements based on something like the Rational Unified Process. The first encounter is very impressive. It drops onto the table with a loud THUD, 400 pages of printouts. In some ways, this is a very useful document, filled with details that will help the developers go forward. However, it incomprehensible to the human brain. It has long lists of labeled requirements and step by step processes. You cannot absorb information in this format, and you cannot explain it to your users and project sponsors. It fails the first purpose of a user story, which is to get feedback from the subject of the story. So, you end up getting diminished benefit from the (expensive) planning process, since the user will ignore the specification and instead provide feedback only after seeing the prototype.

The solution: Make pictures, storyboards, and not text. Everyone will understand the storyboard. A storyboard is simple, even simpler than a flowchart. It's basically a flowchart with only the components that a user will see. It shows screens that the user will encounter while working with the system, and links that represent transitions between screens. It can be converted directly to a mockup. You can skip most of the text and write it in later as ticket detail and comments.

Our site started misbehaving on Saturday when we got this message from our datacenter: "...electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room ... This is a significant outage, impacting approximately 9,000 servers"

An explosion. How cool is that? It's way cooler than having a guy accidentally dig up your wiring with a backhoe, or run it over with a truck. It's an outright privilege compared with the squirrel that ate the fiber cable. Imagine spending nights and weekends rallying your troops around a disaster recovery plan, all to do battle with a squirrel. That would be humiliating. My wall of flame incinerates your squirrel.

The flame walled us off from three servers out of the approximately 8 that we need to run Assembla.com at full power. In theory, the rest of the system, including the www.assembla.com site itself, should have continued to work properly. In practice, we found that processes on the remaining servers would hang while waiting for responses from the missing servers, and users would get “Unavailable” errors on www.assembla.com or find themselves unable to access trac.

It's better now. The missing servers are back online or replaced. We'll be moving to improved virtual server topologies that will hold up under explosive attack.

Many of you are experiencing problems accessing assembla servers.  If you get a "Service Unavailable" error on www.assembla.com, you will be able to get through, but you may have to refresh.  If your svn repository is on the server "svn.assembla.com", it is offline.  Other svn servers are working correctly.  Some trac services may be hanging intermittently. 

We hope that everything will be fixed in time for the work day on Monday.

We have received a message from our datacenter that says:

"...electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room   Thankfully, no one was injured.  In addition, no customer servers were damaged or lost. 

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers.  Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday."

Fortunately, most of the Assembla servers are still online.  However, there seem to be communications with the servers that are offline that can hang some parts of the system.  Again, we hope to resolve this quickly.

Rebuilding your software to update the architecture is the riskiest development project you will ever dive into. Big, successful companies have been crushed by this task. The right tactics are essential. Recently, a more moderately sized company called me in to advise them on a much-needed rebuild, and we broke the tactical options down into three categories - "standard", "incremental", and "buy", each with its own advantages and disadvantages.

A personal story illustrates the risks involved. In 2000, I launched a rebuild of my PowerSteering product. There were all sorts of reasons to rebuild the product, the least important but the most motivating being that we wanted to make the product extensible in some exciting new ways. Over the long term, this turned out to be the right thing to do. We added a lot of new capabilities that customers appreciated. However, this was a crushing mistake for me personally. We were coming to market with this product in 2001, just as the big recession hit, and we were carrying the extra expenses of the product rebuild. We got some VC's involved, and they set to work with a well-oiled plan to fire me, strip me of the assets that I had invested in the company, and dilute me out of a meaningful shareholding. It was hard on my pregnant wife.

In this case, we had a rare controlled experiment. Someone else took the same code and went into an alternate universe without a rewrite. An entrepreneur had approached me to purchase full ownership of some project management code, so that he could turn it into a product for his startup. I sold him the OLD code for a nominal sum, with a stipulation that he should send me $200K if he ever sold it. He changed the front end to be more industry specific, but he never made any upgrades to the underlying code. A few years later, my (former) company got a check in the mail. It turns out he sold the company and the product for $6M.

Dharmesh Shah elaborates upon this lesson in Why you should almost never rewrite your software.

Why take the time and expense to rebuild software? Because after a while, it becomes harder and harder to do the things that you want to do. There is an ever-increasing amount of code that is structured incorrectly for the new demands you are placing on it. Eventually, it looks easier and faster to rebuild the software with an updated architecture, than to continue working with the old code.

Here are the three major options:

1) Standard Approach - Prototype and expand

The standard approach is to build a prototype of the new product, and then expand it into a complete application.

Advantages: This approach has the advantage of being relatively cost efficient. During the prototyping phase, you can work with a very small team, or even your best individual architect. And, you can make significant changes to build the product you actually want.

Disadvantages: The start of the project might be delayed by work on specification, to ensure that the important details of the old product make their way into the new product. And, it's risky. Once you commit to expanding the prototype into a complete application, you enter a potentially long period in which you are spending extra money on development, and you do not have an up-to-date product. In the worst case, you have a situation like the one faced by Microsoft when they release a new operating system, where the new product needs to do everything the old product did, or it will break customer installations. You could hold your breath for a long time waiting for this to happen.

2) Incremental

In the incremental approach, you replace big components of your software with more modern components, or you refactor the existing code. However, you do this in a series of steps that leave you with a releasable and saleable product after each step. This is often compared to "rebuilding the plane in the air".

Advantages: The big payoff is that you have a much lower risk of ending up without an updated product. A more subtle advantage is that you don't need to do as much specification, if you are willing to say that the new product should do basically what the old product did. That saves you time.

Disadvantages: Compared with building software from scratch, this is more complicated to do, and it takes longer. You are working with a larger codebase, and you have to figure out how to keep the plane flying while you rip off the engines.

3) Buy

You might be able to acquire the rights to something that does most of what you need to do (see my story). In an open source world, you might be able to take something off the shelf and adapt it. In fact, modern product rebuild plans almost always contain a significant component of buying or borrowing from open source. As the amount of software in the world grows, it's increasingly important to do research to find out what you can acquire or adopt.

Advantages: Much faster, and probably cheaper.

Disadvantages: Might not meet all requirements. Might place some restrictions on the ultimate value of the asset.

In this case, we decided on the "Incremental" option. The team was able to come up with an amazing plan in which every aspect of the architecture was replaced, but an existing application would still run correctly at each step. If you find yourself contemplating a rebuild, the reduction in risk from the incremental approach is so great that is worth applying your best brainpower to figuring out how to do it.

An increasing number of people blog, add to their facebook page, twitter, and post pictures to document their entire life, and they call it Lifestreaming. It seems like an unstoppable trend. In the future it is likely that we will have phones or similar devices that do record an entire life. Be afraid, be very afraid, and amazed, and curious, and productive. Here at Assembla we are looking at a similar trend that I call Workstreaming. The goal of Workstreaming is to share what you are working on, so that you can work more effectively with your team members. In Assembla, we provide email, RSS, and Web views of Alerts, our list of Workstream activity. As Lifestreaming and Workstreaming infrastructure improves, we should be able to provide a much more connected team experience.

Already, your phone will track where you are.  Customers have asked me for this kind of tracking for their team members. 

Typical workstream events include code commits, comments, wiki edits, ticket edits, and chat. We get all of those things inside Assembla spaces. However, we are taking Workstreaming serious, so we are also developing tools to suck in information from other systems.

What about sharing the other way? I have been asking myself if we can take Workstream data and feed it into the Lifestreaming infrastructure. Can we put your workspace activity on your Facebook page? Unfortunately, the permissioning does not match. A user owns his or her own Lifestream data and can decide what friends to share it with. When the user is working in a team project, the project owner or funder controls this information, and may not want to share it with the same friends.

We can share events from public spaces in the Lifestreaming infrastructur, and we will do that. However, 90% of our projects are private. So, we will need to work on permissioned ways to share that.

Assembla preconfigured spaces give you a fully configured workspace, with related code, in one click.  For example, if you drop into the Just Chat preconfigured space and copy it, and you get a chat room.  Too lightweight?  Preconfigured paces can also contain complete code repositories, with wiki documentation that explains how a developer can get started, and the initial milestones.  Any user can copy, improve, customize, and share a space.

The actual feature is simple.  In the footer of every space page is a link that says "Copy this space".  You can click on that link, and make a copy of the space.  The following picture shows you where to find the link.

The copy operation pulls in tools (the extra feature tabs), wiki pages, milestones, styles, and the complete subversion repository.  It does not copy content that you will not want to reuse - flow messages, tickets, tasks, or chat messages.

Why is this important? 

Preconfigured spaces will save hours of work on a typical development or customization project, because they can be set up with best practices, and can include extras like deployment scripts.  Just in the current Assembla user base alone, that will save staffing-years of effort.  

Assembla will build a list of featured spaces.  We're leading off with Ruby on Rails to demonstrate elements that we think are important:  Build documentation for the team, source code with build scripts and svn configurations that help code move smoothly from developers to the staging and production servers.

Assembla will also be working with other open source frameworks to produce a bigger library of preconfigured spaces.

If you run a lot of projects, you can set up your own starter space and copy it every time you launch a new project.

Users with frameworks to share can post their own preconfigured spaces.  To share, just enter "preconfigured" as a tag under the Admin tab.  I don't know all of the ways that people will use it, but I suspect we will learn something as we make Assembla more configurable.

Here's an idea:  If you are setting up an application process or contest, you can provide the applicants with a preconfigured space that contains their instructions and template documents, and ask them to attach the completed space to your portfolio.

Coming attractions 

The instructions for building your own server are can be dramatically simplified.  It  could be one button.  We plan on adding a "Rails server" tool that gives you a virtual server, on demand.  This will link to a virtual Rails server provider like Morph or the magical Heroku.  With this tool instructions will be "Push the deploy button".  You push the button, and you click on the provided URL to see your app running on the full-bore EC2 cluster.  We will have moved the team, code, and servers all the way into the on-demand cloud.  The rapture will arrive as we effortlessly float into our deployments.

I am particularly interested in the development challenge posed by applications like Drupal and SugarCRM.  These popular applications are often customized.  However, they aren't designed for a professional, version-controlled development process where the developers build and test on a development system, and then deploy their tested changes to the production system.  These applications want you to configure to the production system manually - not a reliable or acceptable process for a big site.  It's a big problem that we can solve by providing preconfigured spaces with documentation and build scripts from developers that have figured out the best practices for this type of customization.  I'm going to create public workspaces for each of these frameworks and solicit help from experts to complete the configuration.

If this goes well, we'll be able to build a foundry for each of these frameworks and applications.  It will really smooth out the application life cycle.  There will be a documented process that makes any customization maintainable, an people that own those customizations can always return to the foundry to get the code, tools, and top talent to do serious development work or maintenance.

The tools on Assembla.com right now are designed to be used without configuration - instant on.  With the preconfigured spaces, we can start introducing tools with more complex configurations.  We'd like to add workflow configuration to the ticket tool, and make a generic workflow tool (user-defined fields and states) that would handle a variety of tasks from use cases to customer support.

It used to be complicated to set up a project correctly, but it isn't any more. There's no excuse not to do it right.

So take my space ... please. 

The March 15 release included dozens of fixes and design enhancements. If you are Assembla user, here are few things that might help you:

A lot of help pages are filled in.  We are making a lot of progress on both help pages and a user manual. Thank you for your patience. 

We improved the format for the Trac/SVN tool page.  We're making it easier to configure, and easier to bring a new user into a version-controlled proejct.  The big new feature here is a button for  importing Trac tickets into Assembla Tickets.  So, you can import the Trac tickets that are already hosted on Assembla, or you can upload your Trac, and then import.  Soon, we will offer an export to Trac tickets, so you will have a migration path both ways.

If you use Assembla tickets, you can update the tickets with Subversion commit comments.  Use "closes #99" to close ticket 99 (also closed, fixed, and fixes), or "Re #99" or "to append a comment (also references and see), or  "test #99" to move it to test status.  You will also find a much-requested "Group by" option in the reports. And, as mentioned in my last post, we added a Notification list option, so that you can select the people who will be alerted by email about a ticket.

The big new alpha/beta test feature in tickets is the  "drag and drop sorter".  It gives you a grid view of the tickets where you can drag them between milestones, or priorities, or users.  I am looking forward to using it for roadmapping.  Try this new link, above the ticket list near "batch update", and tell us what you think.

We made some improvements in the layout of Milestones, and in integration with the Tickets tool.  Milestones is a bit of an awkward feature  because it spans the gap between Basecamp-style milestones and to-do's, a very simple feature for non-technical users, and the release milestones used by developers in the Tickets tool.  It's worth doing because both constituencies work on the same milestones.  I think we are making it flow more smoothly.

The chat implementation has been tightened up.  It will take a big step forward next week when we add HTTP polling to support users that are behind proxy firewalls.  How many of you think we should add audio and video chat?

I won't mention the two BIG features that we have been working on now.  We will cover them in posts over the next few days.