mission critical Archives

January 3, 2018

So You Think You Know It All

Happy New Year, everyone! Sorry I’ve been a bit lax on blogging, but it was a crazy busy last half of the year. I will be doing more blogging this year and there will be some other new things which I’ll talk about soon. All in good time …

Anyway, I’m at the car dealer this morning having my car serviced and I overheard an exchange between a tech and a customer that inspired me to write this blog post. The service person who is handling this customer’s case talks to the gentleman explain what the tech found (or didn’t, in this case). Said customer did not believe him, so he asked for the tech to come out. The tech explains things and how he does his process, including to the point of explaining how he could possibly be seeing what he is. Now, I’m not a deep car guy, but here’s this tech trying to explain how the systems are working together. The guy was having none of it and pulled the “Well, it’s a brand new car. I don’t see why this is relevant.” HE then starts asking the tech if they have a rental car or a loaner which isn’t his responsibility. At no time did I hear the tech raise his voice, and it was not a shouting match but clearly the customer felt like he was being wrong and lied to.

I’ve seen this in our end of the world in different ways. I’ve even experienced it.

I love working with customers. Heck, I’ve built a career on it and wouldn’t have survived this long if I sucked at my job. Ostensibly you’re hiring myself or Max (or someone else, if not SQLHA) because you want expertise. I certainly want to provide that, and would turn down an engagement if I felt you knew more than me or I could be of no help (or didn’t have the bandwidth). Why would I take on an engagement that would ultimately be a problem? The money isn’t worth it.

However, there have been those handful of cases over the years where no matter what you say to someone, they’re in denial. Their problem can’t possibly be the problem, right? Sometimes it is what it is, but people don’t like the answer. This devolves – like the situation I witnessed this morning – into a no win situation. Having said that, if you’re going to keep fighting me, why did you hire me? Why would you hire any expert if you’re not going to listen to them? Could we be wrong? Sure. We’re not infallible. I will admit and own my mistakes or if I am wrong. At the same time, I stand by my track record. You’re not hiring me only for my dashing good looks, you know.

Recently I was working with one of our customers who hit a problem. They sent me an e-mail and I knew immediately what their issue was – it was something I had seen a million times. So based on the little info they gave me, I replied, and lo and behold, problem solved. THAT is why you hire folks like me. Would I have dug in more to see what the issue was if it wasn’t what I suggested? You bet. They were happy and they were not blocked.

I would be lying if I said I know and retain every minutiae about Windows Server, SQL Server, Linux, storage, networking, and so on. It’s just not possible since I do not have a photographic memory. I retain a heck of a lot, and over the years, I joke but it’s probably true: I’ve forgotten more about clustering SQL Server and Windows Server than most people knew. It’s not an ego thing. I’ve just been doing it for 20 years. I still remember lots of little details – even about NT4 – but not everything. It all comes back to me when I’m hands on with the older stuff.

Some things to leave you with:

Asking for help is not a sign of weakness, whether you are an expert or not. I’m at the car dealer because I’m not a mechanic. If I was an expert, would I be sitting here? NO! So if their customer this morning knew more than the tech, why didn’t he just fix it himself? Which leads into …
Being a jerk is not called for in these scenarios whether you are the customer or the person working with him or her. Having been in in the tech’s shoes, I felt for the him. The service rep’s job is to handle these scenarios. The customer asked to speak to the tech, but the customer got indignant. Sometimes you get your dander up and no matter how you break things down, how nice you are, you’re attacking them. The right thing to do at that point is disengage.
When you’re hiring someone, do your due diligence. When we get on a call before we do an engagement with a customer, it’s usually pretty clear we’ve been around the block a few times. It’s up to them at the end of the day whether or not they want to hire us. Some will just consider cost above all. We get that and always work with a customer’s budget whenever possible. But if you want the sun, moon, and stars for the price of a candy bar, chances are we may not be able to help you. The problem with putting budget above all is that often leads to bigger problems. Many times we come in after you’ve hired the wrong person and clean up an even bigger mess. Hiring the right resource up front saves you both time (and often downtime) and money. We’re mission critical guys. We get it. Time really is money – on a whole lot of levels. Work with people who understand the technical and non-technical factors and are invested in working with you.
Good consultants don’t drain your proverbial blood like a vampire and will say no to work not in their wheelhouse. I’m not working for charity, but SQLHA isn’t going to take your money “just because”. We’ve had companies contact us who we said no to that come back later BECAUSE we said no and they liked that. We were up front and honest with them. No is not a bad word or negative in consulting, contrary to popular belief.
Someone you hire’s job isn’t to insult your employees nor be a threat to them. Fun fact: I can tell you with 100% certainty I’m not looking to replace you as a DBA or admin, nor staff your company with my cronies. That’s not what we do at SQLHA.

Bottom line: trust your instincts. They are often right. We all need to ask for help, and we can’t know everything about everything, but be smart about where you get your advice and who you bring in to help. If you need some help, contact us and we’d be happy to see what we can do.

February 14, 2017

Outages In An Increasingly Connected World

I’ve been in the availability game a long time. It seems like at least once a week there is a story about a fairly major outage somewhere in the world due to software, hardware, public cloud failures, human error/incompetence, DDoS, hacking, or ransomware. In the dark ages of IT, when Twitter, Facebook, or any of the other social media platforms did not exist,sometimes you heard about these events. Today, we hear about them in real time. Unfortunately, they can also become PR nightmares, too. Below is a sample of some recent events, all of which happened after January 1, 2017:

Instapaper (The Register’s story; full postmortem)
Delta Airlines (Washington Post’s story)
Australian Tax Office (The Register’s story; what the ATO has to say)
Licking County, Ohio (Newark Adovcate story)
BlueMix/SoftLayer (The Register’s story)
Lloyd’s Bank (one of The Register’s stories)
GitLab (their own postmortem)
Code.org (what they had to say)

The costs associated with these problems also seems to be increasing. Computer problems are no longer just small “glitches”. There are real consequences, whether these are man made outages, systems going down for some type of hardware or software failure, or something else completely. The elephant in the proverbial room is that this is all caused by “the (public) cloud”.<insert picture of old man shaking his fist> This is not true in many cases, but let’s be clear: public cloud providers have had their missteps, too. Technology is only as good as the humans implementing it and the processes around them. Technology won’t save you from stupidity.

Let’s examine some of these very public failures.

RIP Storage

The ATO has had some high profile outages over the last year, a lot of them storage related. Their latest run in with trouble was just a few days ago. This outage didn’t have data loss associated with it, but if you look at the ATO’s statement of February 8th, it’s pretty clear they are unhappy with their vendor. They’ve even hired an additional firm to come in and get to the bottom of things. In other words: in addition to all of the inconvenience and impact to end users and taxpayers, they’re spending more money to figure out what went wrong and hopefully fix the problem.

We’re database folks. Storage is fundamental to us in three ways: capacity (i.e. having enough space), performance (helps get those results back quicker, among other things), and availability (no storage, no database). You need to know where you are in relation to all of those for your database solutions – especially the mission critical ones. You also need to have a good relationship with whoever is responsible for your storage. I’ve been involved with and heard too many horror stories around storage outages and the mess they caused, some of which could have been prevented with good communication.

Oops

Ah, GitLab. What started out as something well intentioned (putting servers up in a staging environment to test reducing load), became the perfect storm. Before I go any further, let me say up front I applaud their transparency. How many would admit this?

Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed.

Hoping they could restore the database the engineers involved went to look for the database backups, and asked for help on Slack. Unfortunately the process of both finding and using backups failed completely.

Read the “Broken Recovery Procedures” section of that postmortem. It is very telling. Things went nuclear and recovery was messy, but in my opinion, something that could have been avoided. Most failures of this magnitude are always based in poor processes in place – it’s something I see time and time again. Kudos to GitLab owning it and committing to fixing them, but assumptions made along the way (and you know what they say about assumptions …) helped seal their fate. This cautionary tale highlights the importance of making backups and ultimately, restores.

Wait, There Are Limitations?

A few of these tales of woe are related to not knowing the platforms used.

I can’t say whose fault it was (developers? person who purchased said product? There could be lots of culprits …), but look at Instapaper. Check out the Root Cause section of the post mortem: they went down because they hit, and then exceeded, the 2TB file size limit that existed in an older verson of their underlying platform. That is something that anyone who touched that solution should have known, but had specific monitoring in place so that when things got close, it could be mitigated. I call a bit of shenanigans on this statement, but applaud Brian taking full responsibility at the end:

Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue.

It’s your job to ask the right questions when you take over (hence the accountability part). Now, to be fair, it’s a legitimate gripe assuming what is said is true and RDS does not alert you. Shame on Amazon if that is the case, but situations like that are the perfect storm of being blind to a problem that is a ticking time bomb.

Similarly, Code.org suffered the same fate. I’m not a developer by nature, but even I know that 4 billion lines of code is not a lot, especially when you have a shared model. Their own webpage alludes to over 20 billion lines of code written on the platform. Their issue is that they were using a 32-bit index which had a max of 4 billion rows of coding activity, and had no idea they were hitting their limit. I would bet that some dev along the way said, “Hey, 4 billion seems like a lot. We’ll never hit that!” Part of the fix was switching to a 64-bit index which holds more (18 quintillion rows), but famous last words …

On the plus side, this new table will be able to store student coding information for millions of years.

A Very Real World Example

In the US right now, there is the chance of a major catastrophe in Oroville, CA because of the Oroville Dam. There is nothing more serious than possible loss of life and major impact on human lives. I was reading an article “Alarms raised years ago about risks of Oroville Dam’s spillways” on the San Francisco Chronicle site, and like a lot of other things, it appears that there is a chance this all could have been avoided. Of course, with things of this nature, there’s a political aspect and a bit of finger pointing (as there can be in businesses, too), but here is a quote I want to highlight

Bill Croyle, the agency’s acting director, said Monday, “This was a new, never-happened-before event.”

It only takes once. I’ve seen this time and time again at customers for nearly twenty years. Nothing is a problem … until it’s a problem. No source control and you can’t roll back an application? No change management and updates kill your deployments and you have to rebuild from bare metal? You bet I’ve seen those scenarios and customers implemented source control and change management after.

Let me be crystal clear: hundreds of thousands of people and animals being displaced is very different than losing a few documents or some data. I have no real evidence they did not do the right repairs at some point (I’m not an Oroville Dam expert, nor have I studied the reports), and yes, there is always something that you do not know that can take you down, but statements like that look bad.

Pay Now or Pay Later – Don’t Be The Headline

Outages are costly, and to fix them will take more time, money, and possibly downtime to fix. Here are five tips on how to avoid being put in these situations and giving yourself a potential resume generating event:

1. Backups are the most important task you can do as an administrator. At the end of the day, that may be all you have when your fancy platform’s features fail (for any number of reasons, including implementing them incorrectly). More important than generating backups is testing them. You do not have a good backup without a successful restore. With very large sets of data (not just SQL Server databases – data can mean much more such as files associated with metadata that is in a RDBMS), finding ways to restore is not trivial due to the costs (storage, time, etc.). However, when you are down and possibly closed for good, was it worth it the risk only to find out you have nothing? No.

2. Technical debt will kill you. Having systems that are long out of support (and probably on life support) and/or on old hardware is a recipe for disaster. It is definitely a matter of when, not if, they will fail. Old hardware will malfunction. If you’re going to have to search eBay for parts to resuscitate a server, you’re doing it wrong. Similarly, for the most mission critical systems, planned obsolescence every few years (usually 3 – 5 in most companies) needs to be in the roadmap.

3. Have the right processes in place. Do things like test disaster recovery plans. How do you know your plans work if you do not run them? I have a lot to say here but that’s a topic for another day.

4. Assess risk and mitgitate as best as possible. Don’t bury your head in the sand and act surprised when something happens. (“Golly gee, we’ve never seen this before!”)

5. Making decision on bad information and advice will hurt you down the road. I can’t tell you how many times Max and I have come in after the fact and clean up somebody else’s mess. I don’t relish that or get any glee from it. My goal is to just fix the problem, but if you have the wrong architecture or technology/feature implemented, it will cause you years of pain and a lot of money.

Are you struggling with any of the above? Have you had or do you currently have availability issues? Contact us today so SQLHA can help you avoid becoming the next headline.

July 23, 2015

Lots Of Upcoming Training and Speaking – Come See Me!

Happy Thursday, everyone! If you haven’t noticed, I have been pretty silent on the blog front. I am heads down writing the book (which is coming along nicely) as well as doing customer work. Fingers crossed, the book will be done before Summit. I will get back to a more normal blogging schedule when the book is done. Also, be sure to check out Max’s blog posts as he is now also regularly blogging here on the site.

It hasn’t been all book writing and customer work, though. Today I updated our Training and Events section with lots of new stuff. I am pretty excited, but it also means that I will be on the road from late September until early December. The things I do for you guys 🙂

First up is my 4-day Mission Critical SQL Server class in London via Technitrain from September 28 – October 1. As with all my 4-day deliveries, there are lots of hands on labs (bring your own device, though!). This has been on the books for quite some time. I love London, and my class in 2014 was a blast. I expect this delivery and the students to be the same. I may also be doing something else that week … stay tuned!

I will only be home a few days before heading down to Orlando (a place I *clearly* hate if you know me at all – Disney here I come!) for SQL Saturday 442. On Thursday, October 8, I will be doing a brand new preconference session entitled “Planning Highly Available SQL Server Deployments in a Cloudy, Virtualized World“. This is a “normal” precon, meaning there are no labs like some of my other precons. This helps to keep the price down while I get to deliver a really fun – and timely – topic. On the Saturday of the event, I will be delivering the talk What’s New for High Availability in SQL Server 2016 and Windows Server 2016 – another new one I am putting together.

The next thing on the books is the big kahuna – PASS Summit 2015. I am honored to once again be doing a preconference before Summit. This year I will be delivering “Advanced SQL Server Availability Architectures and Deployments“. I could have done the safe thing and just submitted an updated version of what I did last year. Why rest on your laurels? Don’t get me wrong – having 100 people doing labs at the same time was amazing, but been, there, done that. How could I kick it up a notch? Do something more difficult! Everyone gets their own lab environment (i.e. multiple VMs) that they connect to via a browser pm a device they bring with them. I am finalizing what the labs look like right now and will be building the VMs in the next month. I truly believe for topics like this you need to touch and feel. No hands on makes advanced topics more difficult to learn. Space is limited and I expect another sold out session, so register early. Don’t miss the PASS Summit 2015 Speaker Q&A I did for the precon posted up at the Summit site.

During Summit, I was asked to participate in the panel discussion “Will the DBA Job of the Future Still Involve DBAs?“. That should be a lively panel. For the third year in a row, SQLHA will have a booth (this year once again with Denny Cherry & Associates) called Consultant’s Corner. Drop by to say hello to Max or myself – we’ll both be there. Unlike last year where I was juggling speaking as well as the MVP Summit over in Redmond, I will be in the booth quite a bit this year.

Finally, I am very excited to be returning to Australia for two weeks. The last time I was there was 2013. WardyIT is once again bringing me back as part of their Platinum Training series. I will be delivering my 4-day Mission Critical SQL Server Workshop in Brisbane (November 24 – 27) and Canberra (November 30 – December 3). Every time I go to Australia it is two different cities; the last time I did this combination was in 2012. The last time I was in Brisbane I also spoke at the user group, so we’ll see what happens.

As an aside, both the London and Australia deliveries of the Mission Critical SQL Server class will also be the first time that I will be rolling in some 2016 (both SQL Server and Windows Server) content formally into the curriculm where it is appropriate. My classes are always evolving.

Whew! Are you tired from looking at my schedule yet? I know I’ll be hitting at least Platinum on American with all of those miles in the air. And if all of this was not enough, there are a few other things in the works which are not quite ready to be announced yet. As soon as they are, they will be on the events page.

June 11, 2015

Features Should Not Dictate Mission Critical System Designs

I was working up a new presentation for the upcoming 24 Hours of PASS on mission critical system design (a bit more on that later) when one of those puzzles of life we all have cooking away on the back burners of the mind suddenly snapped into focus. The story of that puzzle goes a little something like this…

The development team ships a great feature called ladders for changing those light bulbs in high ceilings. They also have a fashionable feature called yellow polka dot cloth, which is just perfect for those retro 70s swimming trunks and bikinis that are all the rage. And they put in a cleverly-engineered tricycle feature loved by our active 4 year olds. Nothing at all wrong here.

Yet many times I arrive onsite to help a new customer with their mission critical system only to find that they’ve currently got a three wheeled, yellow polka dot, monster truck jacked up on stilts. I always used to wonder just how on earth they ever got to this point.

Now, thanks to last week’s epiphany, I finally understand. There is a natural tendency of humans to map what they need to what they have – regardless of any actual correlation between the two. There’s probably a law of needs way down the list about this already, and if not there should be because it really is true.

I guess this was correct in most natural situations. If a caveman needed a weapon to fight off a sabertooth tiger, picking one that was close at hand would be a critical survival skill. Clearly this is no time to start with a blank page and work through exactly what the best weapon would be to kill a huge, hungry cat that is looking at you like you are the answer to all his culinary dreams.

But therein lies the real moral of the story, and it’s just as true for DBAs.

Under pressure you’ll have to take the existing features which are, after all, designed to implement high availability and disaster recovery. The funny thing is that this is relative cheap and fast, but just always seems to fit like that sweater your aunt knitted you for Christmas – one sleeve too short, one too long, a bit tight around the neck, and never draping just right.

The features that ship in the product are great. However, chances are that if you sat down and wrote out real wants and needs for your particular applications and environment you would find that those existing features can get you only part of the way to success. Suddenly the gaps that you never saw when you just implemented HA/DR features become all too noticeable. The plan becomes a bit more complex and you wind up using a feature for this, part of a feature for that, writing some scripts over here, and adding a bit of custom hardware over there. It starts to be, well, work. But now the solution fits. It just feels right and does what it should when it should.

Of course, some knowledge always helps, and I mentioned that I was working on a new session for 24 Hours of PASS called “The Art and the Science of Designing a Mission Critical SQL Server Solution”. If you’ve ever thought about taking out that blank page and starting from scratch, I believe you’ll find it gives you some fresh new ideas and a nice starting point.

I hope some of you will join me there on Wednesday, June 24^th at 8PM Pacific/11PM Eastern. Alternatively, feel free to try to fend off that big cat with a convenient tree branch – but we both know down deep that it’s probably going to end with one very happy feline.

If I’ve achieved nothing else, I hope you take away that what you really need is rarely what you have conveniently at hand and that you must actively resist human nature to see that this is the case.

November 14, 2013

CONTEST! Win A Seat To Allan’s Mission Critical SQL Server Class in Chicago

We here at SQLHA had a chat and we decided that besides what we already do already, we want to give back to the SQL Server community at large. We started somewhere and people gave us a helping hand – we’d like to do the same and pay it forward. That’s why we decided for selected deliveries of my in-person Mission Critical SQL Server class we put on, we are going to give away one seat to a lucky winner … and we’re doing it for Chicago from December 9 – 12. Classes like the upcoming London one (and Australia which I just finished) would always be out of scope since those are not put on directly by SQLHA. Remember that there is still a discount (see the link above) for the Chicago class if you want to outright purchase a seat (and I encourage you to sign up!).

If you don’t win for Chicago, don’t fret – we’re looking at the 2014 calendar now and scheduling additional dates in the US and maybe Canada. Outside of North America – stay tuned.

Good luck!

The Rules

To win the seat is simple: send us an e-mail as to why you think you deserve the seat and how it would impact you. It’s just that simple. Don’t send us one sentence, or just “Pick me!” We also don’t need a tome. As I said above, we want to give back so give us a heartfelt answer as to why you should be chosen. We are not going to make you do anything outlandish or record a video (if you want to, that’s fine but we are not asking for it); we’re going to keep this simple.

Send all entries to: sales@sqlha.com with the this as your e-mail subject: Windy City Here I Come

All entries must be received by 11:59 PM Eastern US time on Friday, November 22, 2013. No extensions or amount of begging will change that.

The Fine Print

One entry per person per class. Entry is only good for the class submitted. You must submit individual entries for different classes.
Winners will not be eligible for a free seat in a future class and are ineligible for winning any other free SQLHA giveaway for 12 months after winning the seat in the class (excluding any giveaways in the class). If you cannot attend the class where you are chosen as a winner, you forfeit the prize. Please do not enter if you cannot attend; it is not fair to those who can.
You (or your company) are responsible for all travel and expenses including, but not limited to: airfare, taxis, food, hotel, and so on. If you cannot meet this obligation for the class you are thinking of entering, please save it for one you can.
Entries without the proper subject will be disqualified. Sorry.
While we do not have delicate sensibilities, keep your entries clean.
You are responsible for any taxes you may need to pay as a result of winning this contest.
You must be eligible to win. For example, some who work in certain jobs or roles (such as some government agencies) would be ineligible. Know if you can before you enter. I apologize in advance if what you do rules you out, but we don’t want to waste anyone’s time or cause issues for you OR us.
While we understand that writing is not everyone’s forte and English may not be your first language, anyone who uses text speak such as ur will be disqualified as well.

12 3

Categories