Cloud Balancer

The latest Amazon S3 blackout made me think what could be done to try to solve this problem, when your application depends on S3 or any other cloud service.

Does this mean that you should all stop using cloud services and go back to your own data center? No way! There are better and more reasonable solutions. They just need a bit of thought and some experimentation.

Some people, like Dave Winer, can even see that a new service can emerge from this need. On his article “Amazon S3 down all day”, Dave proposes a possible solution:

It seems there is a business opportunity here – it would be easy to hook up an external service to S3, and for a fee, keep a mirror on another server. Then it would be a matter of redirecting domains to point at the other server when S3 goes down.

So, here’s my proposed solution for this specific problem: use as many equivalent storage solutions as redundacy. Technicaly speaking, you should write your data to multiple services at once and create a read procedure that selects the fastest service and uses it.

This solution can use tarpipe to let you write data to multiple destinations at once, and Gnip can inform your application about the best service to read from.