Amazon EC2 storage remains problematic

choose two

A well-known concept in professional services is the quality triangle: the client wants Good, Cheap, and Fast. Now choose two. You can't have all three.

There's a similar trade-off triangle in high-end hosting solutions: Scaleability, High-Availability, Data Integrity. Now choose two.

• If you want to scale infinitely with maximal uptime, you'll have to accept some data loss on failover.
• If you want absolute data integrity and scale infinitely, you'll have to accept some downtime windows to enable data replication to complete.
• If you want absolute data integrity with absolutely no downtime, scaling up hits speed-of-light constraints where network latencies start to impact synchronization.

Amazon EC2 is almost infinitely scaleable and now moves into high availability territory. Data integrity is woefully inadequate though.

Let me repeat that: EC2 data integrity is a joke. If your image has downtime, all persistent data is gone. That's the equivalent of having to restore a server from backup just because it's rebooted. Also, the backup solution sucks.

The Amazon S3 storage solution can best be described as a HTTP filesystem with a max-filesize of 5 GB. This is great for storing video content cheaply. But not of much use if you want to rsync large data volumes frequently.

The best way of backing up an EC2 image is by setting up a second EC2 image as a slave replicator. That is, by making it highly available. In terms of price per storage GB that's quite expensive, and it scales badly if you want multiple backups, for example to enable point-in-time recovery.