Don MacAskill: “Which brings us to fourth, we aren’t 100% cloud yet. We’re working as quickly as possible to get there, but the lack of a performant, predictable cloud database at our scale has kept us from going there 100%. As a result, the exact types of data that would have potentially been disabled by the EBS meltdown don’t actually live at AWS at all – it all still lives in our own datacenters, where we can provide predictable performance. This has its own downsides – we had two major outages ourselves this week (we lost a core router and its redundancy earlier, and a core master database server later). I wish I didn’t have to deal with routers or database hardware failures anymore, which is why we’re still marching towards the cloud.”
Something to remember when you put your data in the hands of someone else, they don’t care about it as much as you do.
Case in point, an ECG monitoring company had a life and death situation because of the failure. They should’ve been a bit better prepared, but I’m sure they thought “The Mythical Cloud” would never fail.
Bottom line: It’s still a network, they’re still just computers, they’re over used, humans are in charge, and you should have a backup plan.
