Our users in Europe and India were inconvenienced last night by an error - SSL Certificate expired. Our servers were working correctly. This particular problem was caused not by technology, but by a combination of human error and politics. The politics is particularly interesting.
The human error was mine. I did not notice the expiration reminders, because they go to an email box where they are mixed in with our credit card transaction notices, which we get thousands of. I think that this in turn is a usability problem in the Paypal/Payflow system, which apparently cannot turn off email transaction alerts. I can filter them and will do so in the future.
The politics comes from Moldova, a small country, renowned for its homegrown food and wine, where Assembla’s night system administrators live. Yesterday, the government of Moldova shut down the Internet and took our guys offline. So, when I went to bed last night, I knew that for extraordinary reasons, we didn’t have night coverage, and I was worried. Sure enough, I was barely asleep when the ticking time bomb went off and the SSL certificate expired. A call from an alert user finally woke me up, at which point Moldova was back online, and we fixed the problem.
Moldova is a peaceful and friendly country with a strong engineering tradition, which is remarkable when you consider two things. First, it is the poorest country in Europe, and mostly agricultural. Second, it is a tiny country wedged between Russia, which recently governed Moldova and occupies a slice of Moldova called Transdniestria, and Romania, which speaks the same language and was in the past joined with Moldova. Romania has been stealthily absorbing Moldovans into the EU by issuing them Romanian passports.
Moldova held an election on Sunday, and Monday, the results came in. The incumbent communists won about 50% of the vote and 60% of parliament. This is a credible result, given that the communists have during the past two years delivered peace and prosperity, albeit with corruption. The election was certified as "orderly" by international observers. The opposition claims that one reason for the calm and orderly voting was that thousands of dead people voted. A lot of young people in Moldova are not happy with the prospect of spending another 4 years with one of the world's only elected communist governments.
Students in Moldova fired up their computers and mobile phones, got on Twitter / facebook / SMS, and organized a protest with the keyword #pman. They converged on the presidential building, swarmed around the guards, broke the windows, and went inside, where they raised the Romanian flag, asked the incumbent president to leave the country, etc.
The incumbent government of Moldova then shut down Internet and mobile phone service in the capital, Chisinau. This tactic limited the formation of new flashmobs.
Spreading admin work into different time zones is a good way to get 24 hour coverage, but this example shows that it has some problems. I am not sure that it offers a general lesson, because I have not seen an entire country go offline before. It seems pretty rare.