data loss Archives

February 14, 2017

Outages In An Increasingly Connected World

I’ve been in the availability game a long time. It seems like at least once a week there is a story about a fairly major outage somewhere in the world due to software, hardware, public cloud failures, human error/incompetence, DDoS, hacking, or ransomware. In the dark ages of IT, when Twitter, Facebook, or any of the other social media platforms did not exist,sometimes you heard about these events. Today, we hear about them in real time. Unfortunately, they can also become PR nightmares, too. Below is a sample of some recent events, all of which happened after January 1, 2017:

Instapaper (The Register’s story; full postmortem)
Delta Airlines (Washington Post’s story)
Australian Tax Office (The Register’s story; what the ATO has to say)
Licking County, Ohio (Newark Adovcate story)
BlueMix/SoftLayer (The Register’s story)
Lloyd’s Bank (one of The Register’s stories)
GitLab (their own postmortem)
Code.org (what they had to say)

The costs associated with these problems also seems to be increasing. Computer problems are no longer just small “glitches”. There are real consequences, whether these are man made outages, systems going down for some type of hardware or software failure, or something else completely. The elephant in the proverbial room is that this is all caused by “the (public) cloud”.<insert picture of old man shaking his fist> This is not true in many cases, but let’s be clear: public cloud providers have had their missteps, too. Technology is only as good as the humans implementing it and the processes around them. Technology won’t save you from stupidity.

Let’s examine some of these very public failures.

RIP Storage

The ATO has had some high profile outages over the last year, a lot of them storage related. Their latest run in with trouble was just a few days ago. This outage didn’t have data loss associated with it, but if you look at the ATO’s statement of February 8th, it’s pretty clear they are unhappy with their vendor. They’ve even hired an additional firm to come in and get to the bottom of things. In other words: in addition to all of the inconvenience and impact to end users and taxpayers, they’re spending more money to figure out what went wrong and hopefully fix the problem.

We’re database folks. Storage is fundamental to us in three ways: capacity (i.e. having enough space), performance (helps get those results back quicker, among other things), and availability (no storage, no database). You need to know where you are in relation to all of those for your database solutions – especially the mission critical ones. You also need to have a good relationship with whoever is responsible for your storage. I’ve been involved with and heard too many horror stories around storage outages and the mess they caused, some of which could have been prevented with good communication.

Oops

Ah, GitLab. What started out as something well intentioned (putting servers up in a staging environment to test reducing load), became the perfect storm. Before I go any further, let me say up front I applaud their transparency. How many would admit this?

Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed.

Hoping they could restore the database the engineers involved went to look for the database backups, and asked for help on Slack. Unfortunately the process of both finding and using backups failed completely.

Read the “Broken Recovery Procedures” section of that postmortem. It is very telling. Things went nuclear and recovery was messy, but in my opinion, something that could have been avoided. Most failures of this magnitude are always based in poor processes in place – it’s something I see time and time again. Kudos to GitLab owning it and committing to fixing them, but assumptions made along the way (and you know what they say about assumptions …) helped seal their fate. This cautionary tale highlights the importance of making backups and ultimately, restores.

Wait, There Are Limitations?

A few of these tales of woe are related to not knowing the platforms used.

I can’t say whose fault it was (developers? person who purchased said product? There could be lots of culprits …), but look at Instapaper. Check out the Root Cause section of the post mortem: they went down because they hit, and then exceeded, the 2TB file size limit that existed in an older verson of their underlying platform. That is something that anyone who touched that solution should have known, but had specific monitoring in place so that when things got close, it could be mitigated. I call a bit of shenanigans on this statement, but applaud Brian taking full responsibility at the end:

Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue.

It’s your job to ask the right questions when you take over (hence the accountability part). Now, to be fair, it’s a legitimate gripe assuming what is said is true and RDS does not alert you. Shame on Amazon if that is the case, but situations like that are the perfect storm of being blind to a problem that is a ticking time bomb.

Similarly, Code.org suffered the same fate. I’m not a developer by nature, but even I know that 4 billion lines of code is not a lot, especially when you have a shared model. Their own webpage alludes to over 20 billion lines of code written on the platform. Their issue is that they were using a 32-bit index which had a max of 4 billion rows of coding activity, and had no idea they were hitting their limit. I would bet that some dev along the way said, “Hey, 4 billion seems like a lot. We’ll never hit that!” Part of the fix was switching to a 64-bit index which holds more (18 quintillion rows), but famous last words …

On the plus side, this new table will be able to store student coding information for millions of years.

A Very Real World Example

In the US right now, there is the chance of a major catastrophe in Oroville, CA because of the Oroville Dam. There is nothing more serious than possible loss of life and major impact on human lives. I was reading an article “Alarms raised years ago about risks of Oroville Dam’s spillways” on the San Francisco Chronicle site, and like a lot of other things, it appears that there is a chance this all could have been avoided. Of course, with things of this nature, there’s a political aspect and a bit of finger pointing (as there can be in businesses, too), but here is a quote I want to highlight

Bill Croyle, the agency’s acting director, said Monday, “This was a new, never-happened-before event.”

It only takes once. I’ve seen this time and time again at customers for nearly twenty years. Nothing is a problem … until it’s a problem. No source control and you can’t roll back an application? No change management and updates kill your deployments and you have to rebuild from bare metal? You bet I’ve seen those scenarios and customers implemented source control and change management after.

Let me be crystal clear: hundreds of thousands of people and animals being displaced is very different than losing a few documents or some data. I have no real evidence they did not do the right repairs at some point (I’m not an Oroville Dam expert, nor have I studied the reports), and yes, there is always something that you do not know that can take you down, but statements like that look bad.

Pay Now or Pay Later – Don’t Be The Headline

Outages are costly, and to fix them will take more time, money, and possibly downtime to fix. Here are five tips on how to avoid being put in these situations and giving yourself a potential resume generating event:

1. Backups are the most important task you can do as an administrator. At the end of the day, that may be all you have when your fancy platform’s features fail (for any number of reasons, including implementing them incorrectly). More important than generating backups is testing them. You do not have a good backup without a successful restore. With very large sets of data (not just SQL Server databases – data can mean much more such as files associated with metadata that is in a RDBMS), finding ways to restore is not trivial due to the costs (storage, time, etc.). However, when you are down and possibly closed for good, was it worth it the risk only to find out you have nothing? No.

2. Technical debt will kill you. Having systems that are long out of support (and probably on life support) and/or on old hardware is a recipe for disaster. It is definitely a matter of when, not if, they will fail. Old hardware will malfunction. If you’re going to have to search eBay for parts to resuscitate a server, you’re doing it wrong. Similarly, for the most mission critical systems, planned obsolescence every few years (usually 3 – 5 in most companies) needs to be in the roadmap.

3. Have the right processes in place. Do things like test disaster recovery plans. How do you know your plans work if you do not run them? I have a lot to say here but that’s a topic for another day.

4. Assess risk and mitgitate as best as possible. Don’t bury your head in the sand and act surprised when something happens. (“Golly gee, we’ve never seen this before!”)

5. Making decision on bad information and advice will hurt you down the road. I can’t tell you how many times Max and I have come in after the fact and clean up somebody else’s mess. I don’t relish that or get any glee from it. My goal is to just fix the problem, but if you have the wrong architecture or technology/feature implemented, it will cause you years of pain and a lot of money.

Are you struggling with any of the above? Have you had or do you currently have availability issues? Contact us today so SQLHA can help you avoid becoming the next headline.

August 16, 2012

Drawing Inspiration and Data Aplenty

Hello everyone! I just came off a bit of a vacation where I went to the Disney D23 Destination D 75 Years of Animated Features event out in California. Of course I didn’t give myself much of a transition – I am finishing up a 3-day private training course today.

Anyway, I’ve talked about it before in a previous blog post, but growing up I wanted to be an animator for Disney. For me, Disney has always been in my life as far back as I can remember; I mean, I can vividly remember coming home in the afternoons and watching the old Mickey Mouse Club reruns on, if memory serves me correctly, Channel 48 in Philadelphia. Disney is liked for many reasons by different people, but for me it’s always been about the animation. So the opportunity to go to an event focused on the animation and while not talked about in that link above, there was a 3rd day that was limited to a certain number of people which allowed you to tour the Walt Disney Studio (but different than the normal D23 tour they do periodically) and see things like the Animation Research Library – no brainer. Going there has always been one of my holy grails. (Before you start rolling your eyes, I promise there will be IT/SQL related content in this post. No, really.)

That third day was simply amazing and worth every penny I paid. In the AM we got to tour the Roy E. Disney Animation building where the animators put together a whole program for us showing the animation process based on the upcoming Wreck It Ralph movie. They took us through the different processes, and not surprisingly, the one that interested me the most was recording the voices and such. Don’t get me wrong, using the real Disney animation tools in a lab to mess with Wreck It Ralph was fun, but I could have sat in that recording studio all day and absorbed info about the process and asked questions since I’m a music/audio guy.

In the afternoon, we got to do different things, one of which was visiting the aforementioned Animation Research Library – arguably the place if you care at all about Disney animation. It’s locked down tighter than Fort Knox. More than a holy grail, it’s proverbially sacred ground and something I thought I’d never see. 65 million pieces of artwork live there. They took a small selection out for us and we got to look inside a few of the vaults. Seeing things like actual pencil drawings from “Steamboat Willie” (1928), the original maquettes used by animators for various movies, the wooden Pinocchio marionette from 1940 seen here (lower right picture), and original Mary Blair artwork (among many other things) was a once-in-a-lifetime experience. Most Disney people can’t even get in, let alone see some of what we saw. I am still thinking about it days later.

Another standout thing was that we got various presentations, one of which was on the preservation and restoration of the Disney vault of films (animated and live action) and shorts. This was something right up my alley since I’m also a video guy (I still have and use my laserdisc player). It was clear I was the techie guy asking the questions and people were looking at me like I had two heads. It was definitely a magical day all around.

As much as I was on vacation and loving this experience, my IT/SQL mind started to creep in and come up to the foreground every now and then. When we were in the studio, some folks go to do some ADR work to see what that process was (and it was fun to observe), but I was a bit more interested in the gear and process. What was interesting to me is that besides the takes they recorded in one rig, they had a continuous session running in parallel on a different one to capture literally everything. One reason for that is simple: if something is missed, it is captured elsewhere (think backups and HA/DR – gotta love redundancy).

That begs the question: how much data is that? Well, potentially a lot for each project over its lifetime. So of course I asked the question – at what resolution do they record at? For you non-audio geeks, to put anything I’m going to say in perspective, a CD’s resolution has a sampling rate of 44.1 kHz at a depth of 16-bit (an 80 minute CD at 44/16 is roughly 700MB of data). In the audio world, higher sampling rates (48, 88.2, 96, 192, or 384 being the other common options) and more depth (either 24- or 32-bit) is generally a better thing. Not surprisingly, that all chews up a lot of space the more resolution and the higher the bit depth you go. They record at 48/24, and everything is stored on a server with a lot of space. Considering how much audio they must have for each project, that’s a lot to deal with (even at 48/24) not only in terms of needing it near term for the active movie project, but then to back it up and ultimately archive it for other uses down the road. This is not unlike a DBA needing to manage a total of giga-, tera-, or petabytes of information both in databases as well as the various copies (including backups). This is just the audio data – can you imagine all of the animation, too? Wow. Oh, and every bit of that audio needs to be meticulously cataloged, too. Much like you need to keep inventory of all of your servers, instances, and databases.

During the video restoration presentation, we (royal we here) got to talking shop a bit (probably to the dismay of the non-audio and video folks who were in my group) and I asked why they only scan in at 4K and not 8K, as well as discussed audio sample rates. The reason was similar: it’s a data problem and for some old film types, you’re theoretically not going to really get better with higher scans (MuppetVision 3D in Disneyland was scanned in at 6K if you care about such things, but that decision had a lot to do with the original format; this forum discusses this topic more if you want to see some opinions and other links). This puts a whole new spin on Big Data, eh? It may not be schemas, but it sure is a ton of information that is unwieldy to deal with.

What was also interesting about that presentation was the talk about the film stock and not only how flammable it could be (depending on its type), but no matter what they do, at some point, the originals will be completely unusable (much like how I related backups to audio which I blogged about awhile back). Digital, film, analog tape – stored properly it will have a long shelf life, but it may not … it all depends since there are so many factors that contribute to that. If you have important data that needs to live for a long time, preserve it properly and have adequate redundancy, too. Archiving is no joke. A company like Disney is built on its history, and they need to preserve it.

Shifting gears a bit, the Animation Research Library as I said houses 65,000,000 pieces of art. I didn’t ask, but I could tell it was all cataloged and scanned in (I thought I heard someone say 150 dpi which was good enough years ago; I believe they’re working on scanning in at either 600 or 1200dpi now that tech has moved on and storage has gotten cheaper). Anyway, point is, we’ve got another data thing going on here. This is data not only used by animators today that needs to be accessible (hello HA!), but quickly and organized. Side note: cool perk of working there is you can apparently choose pieces from the archive to put up in your office. Drool. This is Art on Demand on steroids! Where do I sign up at HR for framed pencil drawings of “The Band Concert“?

Would I still love to be an animator? Maybe, but some of what goes on at Disney isn’t too far off from what I do today. I help customers ensure their data is available when they need it, be it near line or offline. Your data is your IP … and legacy. So maybe I really did choose the right career path after all – just getting a glimpse of how Disney deals with things like archiving and redundancy gives me inspiration in what I do right now, and is more than just a doodle on a page or a ride in an amusement park.

June 24, 2012

Sometimes You Do Get a Fairy Tale Ending

Just an update to my blog post yesterday (“Even HA Guys Do Dumb Things”) – I have ZERO – yes ZERO data loss. The program I used to recover my pictures from a bad SD card didn’t handle anything than pictures and videos. So I downloaded another program (iCare Data Recovery if you’re curious) and lo and behold, I have all of the files I needed from the blown away partition. Heck, it even found some pictures I had deleted and moved to another hard drive to clear up some space.

I had a total happy ending to my story, but you won’t always get one when something happens like a drive failure or an accidental reformatting. I was up and running with no data loss in less than 48 hours which is good by many standards. Granted, I didn’t have a proverbial gun to my head, it was just one system, and I did things like eat dinner and go out to the movies last night. A true downtime drill won’t have any fun or slack associated with it.

When it comes to disaster recovery, data loss has to be expected. That is a very difficult conversation to have – the business always wants zero data loss. Is zero data loss possible even in scenarios not like mine? Sure. Time was certainly on my side – my recovery point objective (most if possible) as well as recovery time objective (I had all the time in the world) were easy to meet. When you’ve got a real recovery objective of a few hours, a 2TB database, and no idea what your last transaction log backup is that’s available to you – it’s a whole different story. Businesses of all kinds can afford some data loss – you just need to be realistic and either be told the number or make that conversation happen. This is a big factor in planning your backup and restore/recovery strategy.

Software like the one I used may even be able to be used depending on your outage scenario, but realize not only may the package not work, it would take time to run (general rule of thumb: the larger the hard drive, the longer it takes). When the goal is to be back up and running quickly, time is not on your side. It’s about doing the right thing in that situation. Is business continuity more important than data loss? Something has to give somewhere.

Never say never. It happened to me. And it may happen to you. How will you deal with your situation?

June 23, 2012

Even HA Guys Do Dumb Things

Hi again, everyone! It’s been a little bit since my last blog post but I’ve been here, there, and everywhere doing customer work as well as doing a lot of presenting (including my first mutli-day public high availability class in a long time this past week). I can finally take a moment and breathe.

One thing I’m catching up on is a bit of maintenance. As some of you who have seen me know, I have my little powerhouse laptop (Panasonic CF-J10) from Japan. Right now it’s coming up on its birthday. Hard to imagine that I’ve kept a laptop for a year without getting a serious case of upgradeitis, but I have. I have had one nagging problem that I always do – drive space. Even with a 512GB SSD inside, I still carried an external 512GB SSD and another 256GB SSD for a total of 1.25TB of space. This is largely due to the sheer numbers of virtual machines (VMs) that I use depending on the situation and which hypervisor I decide to use (either Hyper-V or VMware Workstation). The only downside to my J10 is that it isn’t truly a 6Gb SATA III bus.

I was very excited when OCZ announced their 1TB SSD quite a few months back. I’m no stranger to being an early adopter of hardware – especially SSDs. When I got my Toshiba R600 back in 2009 or so, it had a 512GB SSD. This is at least 2 years before they were mainstream and “somewhat” reasonable in cost. Toshiba I believe was the only one at the time doing them, and it was easily well over half the cost of the laptop (if not more). Same with the SSD (32GB I believe) in my Sony VGN-G1 in 2006 or 7. The Octane line came out but the 1TB version didn’t really appear until springtime of this year – and the price was stupidly expensive. I mean, even I flinched. A good 512GB SSD will still usually cost in the $700 – $900 range, and I figured double that, and maybe a little more. My 512GB in the Toshiba was about the same, so that’s what I figured. Boy was I wrong! List price on this thing is about $3,000 US, and is generally $2,500 (give or take) street price. Ouch. I can get 1.5TB in three drives for that (albeit not as convenient …). I pretty much decided that I’d sit this one out until prices came down to where I thought they should be. I know I’m weak on this stuff. This past week, I saw that a seller on Amazon had the drive at a price I was willing to pay, and I snagged it. The drive showed up on Friday.

I also usually put on pre-release versions of Microsoft OSes once they are fairly stable. This goes back to at least Vista where I ran it probably for 6 months to a year on my old Sony Vaio SZ, and the same could be said with Windows 7. With the Japanese laptops, this is always an interesting proposition especially since I like to make sure things like the function buttons work. With Sony, most things always worked and once the OS was officially released, everything did. Until now, I haven’t taken the plunge on Windows 8. My disdain for Metro on a non-touchscreen laptop is palpable, but I also know it’s the future. I’ve been using it a bit on Windows Server 2012, and I figured at some point I’d put it on at least one of my Panasonics (I still have the J9 and S9 in addition to the S10 – I really need to sell some!).

Disclaimer: I have this stupid penchant for making non-trivial changes (OS switches, drive swaps) to my laptops before doing something big like my training class this past week. I’ve fought that urge for the past few months because I’ve been on the road and didn’t want to cause myself unnecessary pain and heartache. Keep this in mind. I should have learned a drive change isn’t simple. And boy, this time I almost completely shot myself in the foot.

The original goal was to just change out the 512GB for the 1TB and leave my current Windows 7/Windows Server 2008 R2 dual boot. I have Paragon Partition Manager, so I was just going to clone the drive and change the partition sizes. Oh, if it had been that easy you wouldn’t be reading this blog post. No matter what I tried, I was getting errors. I ran surface tests – the new drive itself was fine. After about five hours of this, I was deleting the partitions (yet again) on the OCZ. Or so I thought. Unfortunately, I must have not been paying attention because I basically blew away my partitions on the 512GB SSD. Cue sad trombone.

My first thought when I knew I was in a bad state: crap – when was the last time I made a backup of any of this stuff and would I need to use it? It was somewhat recently, but not completely up to date. Worst case, I would lose minimal data. My work e-mail is all in Exchange and on a server, so I knew I had that (and since the documentation I do I send to customers and such, a lot of newer stuff would be there even if not in a backup). All was not bleak.

Before you ask, “Hey dummy, why didn’t you make a backup first of your data?” Well, the primary reason is that I was working mainly with the new drive and I wasn’t changing anything on the old and I knew I had a lot of it elsewhere where I could get it or recreate it. So in my risk mitigation, I deemed that low. Since I had been doing things right for hours, the odds of me screwing up were small. Well, I hit the magic window alright. To be honest, I’ve done this a lot in the past so again, the risk from my perspective was minimal.

I stopped and took stock of where I was: how would I go forward, regardless of the backup situation and what I had. In any disaster recovery drill, it’s one thing you need to do. Be calm, have a level head, and make smart decisions. I already did something monumentally stupid. The first decision I made was to take the 512GB out, put the 1TB in, and then put an OS on it just to get started (should have done that to begin with in hindsight). I installed Windows 7, and then made “the” decision – since I basically was starting fresh and we’re at RC, I’ll install Windows 8 – Metro UI warts and all. No time like the present. I got my OS up and running, and then installed Windows Server 2008 R2 for dual booting. I even got most of the Panasonic utilities working under Windows 8 – the only thing that isn’t working at the moment is the touch pad which is a shame because of the way the rounded touchpad should work. But the important stuff – function keys changing brightness, audio volume, selecting display for presentations – all work. Most of the hardware (sound card, display, USB 3) had drivers built-in. This was arguably better than the Windows 7 experience. In the roughly 24 hours since installing Windows 8, I still dislike Metro but have found ways to work around most of the stuff I hate and overall, it’s much faster than Windows 7 on this laptop. I’m surprised by it because Windows 7 was pretty zippy. If I didn’t bring up a menu to select between the two OSes (8 and Windows Server 2008 R2), it would boot even faster. This switch has also gotten rid of a nagging issue I had with the Crucial M4 where it would randomly spin up I/O for a minute or two (usually right after logging in, but sometimes before) and I’d have to wait until I could do anything on the laptop.

However, at this point was the difficult part: where the hell was I going to get my files from and what, if anything, would I lose? Paragon has an Undelete Partition feature. Fingers crossed, I ran it on my 512GB drive. The thing felt like it ran for 10 years. When it was done, I was able to get back two of the three partitions – my old Windows 7 with most of my files (so no data loss there – yay!) and the one with my Hyper-V VMs and various installs I use when using VMs. What I did lose completely was my Windows Server 2008 R2 partition and any of the recent scripts I had written for Windows Server 2012 (including my TechEd ones and one I was working on to change the binding order of the NICs). I lost a few other files, but in the grand scheme of things, I got very, VERY lucky. It could have been much worse than things turned out.

Moral of this story: even HA guys have bad days and do stupid things to their own equipment or installs where I felt “invincible”. Had this been a customer environment, I would have definitely made a backup before doing anything, but I decided to be a bit more reckless with my own setup and I paid the price. I was fortunate. Not everyone is in these scenarios. At least I’m honest enough to admit my mistake. Outside of some of the scripts, I may need to recreate some of my VM environments but I enjoy that so it’s not a big deal. I needed to reconfigure my multi-site cluster demo anyway.

Last night none of this was funny, but now I can look back and chuckle, roll my eyes, and proverbially slap myself for falling into the trap many of us do. Although to be fair, things, while they were not working, were not going poorly all night. I needed to find a way to get from A to B, and my mistake was a completely honest one. I selected the wrong thing once and all hell broke loose.

EDIT: As a friend pointed out on Twitter, I had a D/R plan that worked. To a degree, that’s very true. I knew what I had to do, where I could get things, from, etc. Can you say the same thing? If not, you would really be in trouble in this scenario. I’m being very critical of myself because I did make a super dumb mistake.

Going forward and doing my own post-mortem, I’m going to take my own advice and look for a way to not have this happen again. It will probably involve putting things like decks and scripts up on the SQLHA SharePoint site Ben and I use for our business stuff. Lesson learned. I may run a program I’ve used in the past to retrieve files from a pretty messed up SD card to see if I can get the remaining files back from the SSD, but I may leave well enough alone at this point, take my own lumps, and be done with things. I’m up and running with my new 1TB drive … albeit about a day late.

Oh, as an aside, I have both Hyper-V enabled and VMware Workstation installed and both work under Windows 8 (unlike Windows Server 2008 R2). This is good news, but I’ll still need to dual boot since there’s no software routing in Windows 8 (client OS; Windows Server 2012 does).

Categories