Real Life Disaster Planning (and Recovery) – It’s No Joke
It’s the week before I hit the road for PASS and Australia, and I have a visitor here in MA: Hurricane Sandy (well, the outer edge of it). Here in the Northeastern US, we get Nor’easters, but rarely events like hurricanes. Just the other week we had a magnitude 5.x earthquake. So we’re not immune to natural stuff happening, it just happens infrequently. As I’m typing this, the wind is howling outside and I’ve heard things hit my building. So there’s a pretty good chance that I may lose power at some point in the next 24 hours, so I’m trying to get as much done work-wise as I can before that happens because next week is PASS and I leave to go Down Under after that. Yes, I have spare batteries for my computer (3 to be exact) but that’s not really the point.
Sandy has virtually shut down the Northeast Corridor, one of the most densely populated areas of the USA. The storm is affecting roughly 66,000,000 people. SEPTA (Philadephia), the MBTA (Boston), and the NYC transport system – the major public transportation in those cities – are all shut down. Amtrak isn’t running, and plane flights are cancelled (although the Patriots somehow made it back home from London, but I know people stuck there … go figure). According to Yahoo, Sandy is costing us $10 billion (yes, with a b) per day. This is serious stuff, and we haven’t even gotten to the most important part.
I’ve written quite a bit on disaster recovery (D/R) over the years, and how you need to be prepared. It really is not a technical exercise to a large degree. For people, it’s making sure you can do simple things like eat, drink, and be OK until things pass and are restored (not limited to, but including losing power). That means you need things like water, non-perishable food, flashlights/lanterns, batteries, you name it. Around here, most places were sold out of D batteries by some point on Saturday afternoon – a good 2 or 3 days ago. Anyone looking last night was out of luck. A business must have similar preparations. It’s not just about recovering from a disaster. What it really is, to be honest, is business continuity. When stuff happens, you need to go on be it a minute, a day, or a week later.
It’s funny – I talk a lot with customers about how they will handle some sort of disaster or more serious outage, be it man made or natural. Most are not that concerned or can’t quantify what will happen, which surprises me given the fact that they all want to be 24×7. D/R and business continuity is very much an afterthought in most planning, but I think Sandy will hopefully change the minds of many (at least here in the Northeast). I know MANY businesses that not only don’t have D/R plans, but if they do, their data centers are mere miles apart. Granted, with the whole of the Northeast Corridor being affected here, even 400 – 500 miles may not make a difference with Sandy the way some things are going. A geographically dispersed failover cluster is not going to help you if both of your data centers are down.
Coming full circle, this all brings back Irene for me, which unfortunately took the life of my friend Mike Kenwood (just do a search here …). He made the ultimate sacrifice as an EMT, so I know all too well the impact an event like this can have. Yes, bringing back your business is important, but not if you don’t have the people to do the work. Putting systems in place won’t bring help those who are injured (or worse). If you’re a business, make sure your employees are safe. Most employers (including the Commonwealth of MA) told their employees to stay home today (unless they were essential). That was the right move; heck, one of the stats here in New England just ordered all cars off the road as of 3PM. If your people are where they need to be and come through OK, coupled with the right business continuity plans, you’ll be back up and running as soon as possible.
I’ve been a bit heartened by the fact that the various states and cities – especially the big ones – took all of this seriously. Are people going to experience damage (trees down, flooding, etc.) and power outages? You bet. But it seems like Irene taught many a lesson and that the impact felt from Sandy will be big, but it’s going to be dealt with. There was a lot of backlash after Irene (and a few other things) here in MA with regards to the power companies. Some of that was definitely justified, and it’s been nice to see e-mails from both NStar and National Grid not only telling people what’s going on, but giving tips on preparing.
If you don’t have any D/R plans, maybe Sandy has scared you into doing it. Need help? Contact us.
Stay safe, everyone. I’m hunkered down for the duration.