On Friday and Saturday, many of our users experienced 504 error and dropped connections. We apologize for these annoying events. We have a plan to make sure that it does not happen again. Thank you for your patience.
We were overloaded. We got too many requests for data from Subversion, and our firewall and proxy servers were overloaded. Load spikes from the repository servers disrupted both repository traffic, and Web traffic. The load came from game updates. Some of our users are hosting game content, and on Friday night, when a lot of gamers start up, thousands of them were pulling updates from Assembla Subversion servers.
What are we doing?
- Rate limiting our repository servers, so that the servers in our private, high-availability datacenter cannot become overloaded in this way.
- Organizing our proxy servers so that repository problems cannot affect the Web app.
- Adding a high-throughput server in a public cloud datacenter, so that we can host games and other high-throughput applications.