Blog

The No Good, Terrible Processor Flaw and SQL Server Deployments – Nearly Everything You Need To Know

By: on January 4, 2018 in Linux, Security, SQL Server, Windows Server | 2 Comments

UPDATED JANUARY 18

If you haven’t been paying attention, a serious security flaw in nearly every processor made in the last ten years was discovered. Initially it was thought to be just Intel, but it appears it’s everyone. Official responses:

  • AMD (downplaying the issue)
  • ARM (great response)
  • Intel (oy)

There are two bugs which are known as Meltdown and Spectre. The Register has a great summarized writeup here – no need for me to regurgitate. This is a hardware issue – nothing short of new chips will eradicate it. That said, pretty much everyone who has written an OS, hypervisor, or software has (or will have) patches to hopefully eliminate this flaw. This blog post covers physical, virtualized, and cloud-based deployments of Windows, Linux, and SQL Server.

The fact every vendor is dealing with this swiftly is a good thing. The problem? Performance will most likely be impacted. No one knows the extent, especially with SQL Server workloads. You’re going to have to test and reset any expectations/performance SLAs. You’ll need new baselines and benchmarks. There is some irony here that it seems virtualized workloads will most likely take the biggest hit versus ones on physical deployments. Time will tell – no one knows yet.

What do you need to do? Don’t dawdle or bury your head in the sand thinking you don’t need to do anything and you are safe. If you have deployed anything in the past 10 – 15 years, it probably needs to be patched. Period. PATCH ALL THE THINGS! However, keep in mind that besides this massive scope, there’s pretty much a guarantee – even on Linux – you will have downtime associated with patching.

Below is a summarized list of the biggest players for SQL Server-related deployments covering physical, virtualized, and cloud. Finding all these links took some time, so I figured I should put them all in one convenient place for everyone. Each vendor and product has its own guidance and response, and there may be updates to what I’ve posted but this should get you started. What I did not list is all the hardware vendors. Check with Dell, HP, Hitachi, etc. to see if there are firmware/BIOS/UEFI updates as well.

If you want help with new baselines and benchmarks, or just assistance in sorting this out and coming up with a plan, contact us. If you are on an older, unsupported version of one of the things below that will not be patched, you should strongly consider accelerating your upgrade/migration plans. This is also something we can help with.

AWS

If you’re running workloads using Amazon Web Services, their response can be found here. It appears that their stuff has been patched, but if you’re running IaaS VMs with EC2, you’re going to have to patch your OSes and software in them.

Azure

Microsoft’s response for Azure customers can be found here. They also did a KB article (4073235) which can be found here. Like AWS, they’ve patched the underlying stuff. If you are running IaaS VMs, you’ll need to make sure they are patched properly unless you have automatic patching and running WIndows Server (see below).

GCP

If you’re using the Google Cloud for your workloads, their response is here. As with AWS and Azure, they took care of the base, but you’re responsible for your IaaS VMs/workloads.

Red Hat Enterprise Linux

Red Hat’s response can be found here which talks more about the impact and the performance. To understand the patching side of things, refer to this. SQL Server is supported on 7.3 or later, and those builds have patches available (although I didn’t see 7.4 listed as of the writing of this post, just 7.3). CentOS had its patches released on January 5th.

SQL Server

Microsoft did a great KB (4073225) article summarizing your options which you can read here. Microsoft is patching SQL Server 2008 and later, but reality is because SQL Server 2005 can technically run on Windows Server 2008 and 2008 R2, it would be affected but it’s out of support. I don’t see Microsoft doing anything for it. This would be a good time to consider when you are planning to upgrade or migrate. As of January 18th, patches are available for 2008, 2008 R2, 2012, 2014, 2016, and 2017.

Microsoft lists five scenarios in the KB. Please read them carefully and make the right choice(s), but the absolute wrong choice is to patch nothing.

SLES

If you’re using SLES for your SQL Server deployment, their information can be found here and here (KB). It appears they’ve patched 11 SP3-LTSS through 12 SP3. Although not officially supported for SQL Server, the OpenSUSE info can be found here.

Ubuntu

Here is Ubuntu’s high level response. Here is the link to where to get the patches. 16.04 is covered, which is important for SQL Server.

VMware

VMware posted a security announcement with regards to this issue as well as a blog post. So if you’re using ESXi as your hypervisor, you need to read it. As of the writing of this blog post, it looks like they patched ESXi 5.5, 6.0, and 6.5. It does not look like they are patching anything older than 5.5. There are two vulnerability alerts: VMSA-2018-002.1 and VMSA-2018-0004.2. VMware patched CVE-2017-5715 and CVE-2017-5753. VMware is not affected by CVE-2017-5754, so no patch exists for that.

If you are not on ESXi 5.5 or later, I strongly encourage you to upgrade as soon as possible, and you want that anyway since 6.0 is the first version of ESXi to support vMotion of clustered configurations of SQL Server.

Windows Server

Similar to SQL Server, Microsoft wrote a KB article (4072698) for this issue that can be found here. As of the writing of this blog post, Microsoft has released patches for Windows Server 2008 R2, 2012 R2, 2016, and RS3 (AKA 1709). Hopefully 2008 and 2012 will get patches soon (still the case as of 1/18). If you have automatic updating enabled, the fixes should be picked up by Windows Update. If not, apply them manually. If you’re still running Windows Server 2003/R2 or earlier, I don’t see Microsoft going back and patching. You’re on your own there. The mitigation would be to upgrade ASAP to something that is patched. If you’re running 2008 or 2012 and MS does not release a patch, I strongly urge you to consider upgrading/migrating your deployments to something that is patched.

More information about the January 3rd patch can be found in KB 4072699. Note that due to some anti-virus vendors, unless the registry is changed, you may not automatically see the patch.

XEN

If you’re using XEN as your hypervisor, they did a writeup as well. Things don’t look as rosy right there for now because they don’t seem to have patches for everything yet as of the time I’m writing this blog post. I’m sure that will change.

Other Stuff

Apple – If you’re running High Sierra, Sierra, or El Capitan, it looks like Apple took care of this back in December of 2017. See this for more infomation.

Browsers

  • Chrome – It looks like Google is going to release a patch for Chrome later in January. See this link for more information.
  • Firefox – Version 57 or later has the proper fixes. See this blog for more information, so patch away!
  • Edge and Internet Explorer – Microsoft has a blog post here. It looks like the January security update (KB4056890) takes care of that. So if you’re using either of these browsers, please update your OSes as soon as possible.

Hardware Vendors

This isn’t an exhaustive list, but will hopefully help some of you. A full list of vendors can be found here.

  • Cisco (thanks to the commenter below)
  • Dell Dell’s list of servers and storage is here. Here is a link for Dell’s Data Security product.
  • Hewlett Packard Enterprise HPE is continually updating this post with the various servers and such they sell with compliance and patch links.

So You Think You Know It All

By: on January 3, 2018 in Advice, Mission Critical | No Comments

Happy New Year, everyone! Sorry I’ve been a bit lax on blogging, but it was a crazy busy last half of the year. I will be doing more blogging this year and there will be some other new things which I’ll talk about soon. All in good time …

Anyway, I’m at the car dealer this morning having my car serviced and I overheard an exchange between a tech and a customer that inspired me to write this blog post. The service person who is handling this customer’s case talks to the gentleman explain what the tech found (or didn’t, in this case). Said customer did not believe him, so he asked for the tech to come out. The tech explains things and how he does his process, including to the point of explaining how he could possibly be seeing what he is. Now, I’m not a deep car guy, but here’s this tech trying to explain how the systems are working together. The guy was having none of it and pulled the “Well, it’s a brand new car. I don’t see why this is relevant.” HE then starts asking the tech if they have a rental car or a loaner which isn’t his responsibility. At no time did I hear the tech raise his voice, and it was not a shouting match but clearly the customer felt like he was being wrong and lied to.

I’ve seen this in our end of the world in different ways. I’ve even experienced it.

I love working with customers. Heck, I’ve built a career on it and wouldn’t have survived this long if I sucked at my job. Ostensibly you’re hiring myself or Max (or someone else, if not SQLHA) because you want expertise. I certainly want to provide that, and would turn down an engagement if I felt you knew more than me or I could be of no help (or didn’t have the bandwidth). Why would I take on an engagement that would ultimately be a problem? The money isn’t worth it.

However, there have been those handful of cases over the years where no matter what you say to someone, they’re in denial. Their problem can’t possibly be the problem, right? Sometimes it is what it is, but people don’t like the answer. This devolves – like the situation I witnessed this morning – into a no win situation. Having said that, if you’re going to keep fighting me, why did you hire me? Why would you hire any expert if you’re not going to listen to them? Could we be wrong? Sure. We’re not infallible. I will admit and own my mistakes or if I am wrong. At the same time, I stand by my track record. You’re not hiring me only for my dashing good looks, you know.

Recently I was working with one of our customers who hit a problem. They sent me an e-mail and I knew immediately what their issue was – it was something I had seen a million times. So based on the little info they gave me, I replied, and lo and behold, problem solved. THAT is why you hire folks like me. Would I have dug in more to see what the issue was if it wasn’t what I suggested? You bet. They were happy and they were not blocked.

I would be lying if I said I know and retain every minutiae about Windows Server, SQL Server, Linux, storage, networking, and so on. It’s just not possible since I do not have a photographic memory. I retain a heck of a lot, and over the years, I joke but it’s probably true: I’ve forgotten more about clustering SQL Server and Windows Server than most people knew. It’s not an ego thing. I’ve just been doing it for 20 years. I still remember lots of little details – even about NT4 – but not everything. It all comes back to me when I’m hands on with the older stuff.

Some things to leave you with:

  1. Asking for help is not a sign of weakness, whether you are an expert or not. I’m at the car dealer because I’m not a mechanic. If I was an expert, would I be sitting here? NO! So if their customer this morning knew more than the tech, why didn’t he just fix it himself? Which leads into …
  2. Being a jerk is not called for in these scenarios whether you are the customer or the person working with him or her. Having been in in the tech’s shoes, I felt for the him. The service rep’s job is to handle these scenarios. The customer asked to speak to the tech, but the customer got indignant. Sometimes you get your dander up and no matter how you break things down, how nice you are, you’re attacking them. The right thing to do at that point is disengage.
  3. When you’re hiring someone, do your due diligence. When we get on a call before we do an engagement with a customer, it’s usually pretty clear we’ve been around the block a few times. It’s up to them at the end of the day whether or not they want to hire us. Some will just consider cost above all. We get that and always work with a customer’s budget whenever possible. But if you want the sun, moon, and stars for the price of a candy bar, chances are we may not be able to help you. The problem with putting budget above all is that often leads to bigger problems. Many times we come in after you’ve hired the wrong person and clean up an even bigger mess. Hiring the right resource up front saves you both time (and often downtime) and money. We’re mission critical guys. We get it. Time really is money – on a whole lot of levels. Work with people who understand the technical and non-technical factors and are invested in working with you.
  4. Good consultants don’t drain your proverbial blood like a vampire and will say no to work not in their wheelhouse. I’m not working for charity, but SQLHA isn’t going to take your money “just because”. We’ve had companies contact us who we said no to that come back later BECAUSE we said no and they liked that. We were up front and honest with them. No is not a bad word or negative in consulting, contrary to popular belief.
  5. Someone you hire’s job isn’t to insult your employees nor be a threat to them. Fun fact: I can tell you with 100% certainty I’m not looking to replace you as a DBA or admin, nor staff your company with my cronies. That’s not what we do at SQLHA.

Bottom line: trust your instincts. They are often right. We all need to ask for help, and we can’t know everything about everything, but be smart about where you get your advice and who you bring in to help.  If you need some help, contact us and we’d be happy to see what we can do.

 

A Look Back at Speaker Idol 2017

By: on November 9, 2017 in PASS Summit 2017, Speaker Idol | No Comments

Hard to believe it’s been nearly a week since the end of PASS Summit 2017 and the last round of Speaker Idol. Congratulations to Jeremy Frye for winning – even though he is a Pirates fan.

For those of you unfamiliar with Speaker Idol, here’s the condensed version:

  • 3 rounds of 5-minute lightning talks from people who have never spoken at PASS Summit
  • 4 contestants each round, 1 winner, 1 runner up
  • 1 wildcard selected from the 3 runner ups to fill the 4th slot in the final round
  • Winner of the final round gets a guaranteed speaking slot at PASS Summit 2018 (unless you go to work for MS … which has happened, hence this rule)

The judging panel expanded this year to include Kendra Little (blog | Twitter), instead of four judges plus one extra for the finals who did no see any of the heats. The other judges were myself, Joey D’Antoni (blog | Twitter), Mark Simms (Twitter), and Karen Lopez (blog | Twitter). Bottom line: if you’ve ever seen American Idol, the Voice, or any show, you basically know the format. The emcee this year was Tom LaRock (blog | Twitter) filling in for Denny Cherry (Twitter) as Denny could not be at Summit this year. Denny’s on the mend, and I’m glad he’ll be as good as new soon. Tom had big shoes to fill, and did it well.

Fun fact: I used to judge Speaker Idol back in the day at TechEd before Denny started it at Summit. We did the heats around lunch time. It was a good idea then and still a good idea now. It’s interesting looking back on doing it at TechEd versus at Summit. One thing that I can say for certainty: the quality of speakers has gone up tremendously. TechEd was obviously a more general conference and diverse audience, but with the rise of user groups and things like SQL Saturday in our corner of the world, there’s been a big uptick in quality. There were many more crash and burn moments earlier on with Idol, but now? Not so much. This year’s crop of contestants was particularly good. Our job was not easy – especially for that final round. The smallest of details separated winner from runner up. It was that close.

To me, a five minute lightning talk to me is much harder than a 60, 75, 90, half day, or precon in terms of speaking. To tell a full story end-to-end that is coherent in five minutes or under is not easy. Even harder is cramming a live demo in there. I know people who would say the opposite – especially about a full day precon or a multi-day class. They are hard for different reasons, but I will always contend that being super concise is one of the hardest exercises you can ever do. So kudos to everyone who had the proverbial cojones to not only do that, but willingly be judged by us judge-y types.

Selected notable improvements across board included:

  • Nearly everyone attributed their graphics. Whenever you speak, if you use a picture from somewhere else, give credit. Shame on you if you don’t.
  • A lot of the contestants had much better stage presence this year. Even veteran speakers get nervous, but very few folks just planted themselves like tree or didn’t use hand gestures, etc.
  • Sure, some folks didn’t do the calls and responses right (i.e. always give a number/percentage/whatever if you poll the room as an answer to said question), but there was more audience interaction this year.
  • We had more live demos than in the past. You are brave souls!
  • Even when people had missteps, there were no moments that devolved into total disasters. Give yourselves a round of applause.

I’m guessing they heeded some advice or attended over the past few years … or just had much better practice, like at SQL Saturday. (Side note – submit for one if you have not.)

Tips for future contestants:

  • Make sure your slides are readable. Whether a small room or a big room, a wall of text, small fonts, and bad color choices will give people an unpleasant experience.
  • Make sure you get feature names right, down to caps/no caps, etc. We’re looking at that, and may not call you out on it in your initial rounds, but reserve the right to hold it against you in the finals if you make it.
  • Since the Idol room is known, maybe at some point go and test your laptop. I know I’ve had laptops that won’t connect (for whatever reason) to some rooms over the years, and best to iron that out ahead of time. When it happened this year things worked out, but they don’t always. You’ll at least have a Plan B – which should always be having your slides on a USB key or something.
  • Remember that PASS Summit is not just for US attendees. I keep making this point year after year. If you’re going to use a sports analogy, baseball or football (the US kind, not what we call soccer but the rest of the world calls football) may not translate. Neither will cricket to a US audience, for that matter. Use examples and analogies that make sense and audiences of numerous backgrounds can relate to if possible.
  • We do not require you to do a whole new presentation for the finals – you can if you want. That has its own risk/reward. What we generally look for is that you synthesized our feedback and incorporated it for a better delivery in the finals. Remember that you’re going against three other people who either won their rounds or came in second, so they are no slouches.
  • Don’t be afraid to tell some downsides/risks/personal experiences/give tips along with the facts. People aren’t there to hear you regurgitate documentation. Why should we care? I am not the deepest guy when it comes to what he talked about, but Jeremy won because of what he did, not the subject.
  • We can tell if you’re enthusiastic or not. I don’t need fake cheerleader stuff. Passion goes a long way.
  • Remember we’re judging you to speak at PASS Summit, not a backyard BBQ. You don’t have to do a technical talk or even a data-related topic, but it’s hard to judge if you have the chops if we don’t see some technical meat or tie what your doing into something data/SQL Server-related.
  • I’m the pedantic judge 🙂 Remember that.

One more thing: from a diversity standpoint, it was nice to see people from all over the world and different backgrounds at Idol, but as Karen mentioned at one point in one of the rounds, there were no female entries this year. That makes me sad, since WIT is a big part of the SQL Server community. I would strongly encourage women to enter for next year’s competition. We have lots of strong women speakers in our community.

Hope to see (and judge) you next year.

Free SQL Server Disaster Recovery Round Table Webinar

By: on October 10, 2017 in Disaster Recovery, SQL Server | No Comments

There are two concepts we deal with every day: high availability (HA) and disaster recovery (D/R). Being highly available generally means you can recovery from a smaller, more localized failure with realitvely little pain. When your primary data center is a smoking hole in the ground, that is when you need to invoke D/R. Even though D/R may use a lot of the same features and methods as high availability, implementing and executing it is a different story and definitely more complex.

Join Allan and Max along with the experts at Denny Cherry and Associates on Tuesday, October 24 at 2 PM Eastern/11 AM Pacific for a free round table webinar on disaster recovery for SQL Server. Click here for more info and how to register.

 

Off to dataMinds Connect, A Peek into PASS Summit 2017, and Other Updates

By: on September 27, 2017 in PASS, PASS Summit, PASS Summit 2017, Pre-conferece, Preconference, Presenting, SQLHA, Teaching | No Comments

Can you believe it’s nearly October? I blinked and realized I hadn’t written a blog post since June. Max and I have been super busy with customer work here at the cozy confines of SQLHA. Some of that you’ll see publicly – like the documentation I’ve been writing for Microsoft on the availability features for SQL Server 2017 on Linux. Fun fact: it was announced that it will be generally available (GA) next Monday, October 2! It’s been a fun few months and what I’m working on will be out soon.

I’m heading to Europe to speak at dataMinds Connect next week. I’m looking forward to it – I’ve heard great things about the conference and it’l be my first time in Belgium. I’ll be delivering both a full day precon and a brand new session “Top Troubleshooting Tips for Clustered Implementations of SQL Server“. It would have been nice to also go to SQL Saturday Copenhagen (lovely town – I’ve been there a few times for conferences), but too much going on here to extend the trip.

I’ve also got three other major speaking things coming up (one of which will get its own section below): PASS Summit, SQL Server Live! in Orlando in November, and  for the only time this year, the 4-day Mission Critical SQL Server class with labs. The class will be just outside of Philadelphia. There are still a few seats left. Use the code SEPBLOG for 25% off.

For SQL Server Live! I’ll be doing something new a full day of a bit of hybrid lab/instructional on November 12. It’s not a traditional precon, and it should be fun. If you plan on attending, use the code LSPK36 to get $500 off the standard price of $2,395 – you’ll only pay $1,895, or use one of the links I’ve provided. This link goes directly to the registration page.

If you’re in Minneapolis, go see Max at SQL Saturday next weekend, October 7, and definitely come up and say hi to both of us at PASS Summit since we’ll have a booth again with Denny Cherry and Associates.

RunAs Radio #51

I recorded a RunAs Radio session with my old friend Richard Campbell a few weeks ago. It had been way too long, and we had a blast catching up and talking about SQL Server, availability, and a bunch of stuff inbetween. It was just posted today – listen to it here.

PASS Summit and My Upcoming Precon

I want to talk for a moment about PASS as an organization as well as PASS Summit and my upcoming precon. I have only missed two or three Summits since its inception (Chicago, Orlando, and I think one of the Seattle ones early on). Sure, people can complain about the change of the meaning of the letters, etc., but under any guise, PASS – both Summit and the organization – has been a building block in my career. I could have never guessed that attending Summit in 2000 in San Francisco would be the start of a long journey for me from attendee to speaker to precon speaker to having a booth with SQLHA as a vendor.

Years ago the conference landscape 20 years ago was more stuff like Tech Ed than anything else. It’s funny to see now how many options we have now to speak and attend. Some of those events are downright fantastic. However, just because Summit is one of the biggest events in the Microsoft data platform world and one I still love going every year. Let me be clear: by no means do I think Summit is a perfect conference. I have my gripes, and always will, but for me, it is one of the “must do” data platform events even if I was not a speaker. Just the networking alone is fantastic. No, PASS is not paying me to say any of this, but it’s easy to get cynical about PASS as an organization and Summit as an event because it’s always just been there for most of us along with many other events including SQL Saturday. Many folks you know and love today in our small corner of the world got their “big breaks” at PASS Summit. Not all of them still attend Summit, which I understand, but reality is that Summit, and maybe PASS as an organization, worked for them at one point but they grew beyond it. There’s nothing wrong with that, but Summit still does a lot right for many folks. I mean it when I say I’m honored to be picked to speak at Summit and any event like dataMinds Connect, SQL Bits, SQL Saturday … you get the picture; I never assume, nor do not take an opportunity for granted. I’m not a DYKWIA (Do You Know Who I Am?) type. I’ve said no, but it’s for reasons like I have scehedule conflicts, etc.

As for my journey as a speaker, PASS Summit was integral for me. I remember my first speaking slot all those years ago, and then finally getting a nod for a precon. Could I fill a whole day’s worth of content? Seems silly now to think I couldn’t, and for those who really know me at this point, you are probably laughing your tuchus off because at this point, I can talk with no slides, no demo, and not even a whiteboard for as long as I need to do it.If you saw my SQL Server on Linux talk at SQL Saturday Boston this year, you know what I mean.

Getting opportunities like a precon challenged me. It made me better. Sometimes I failed. Most times, I succeed. Last year was phenomenal – the most people I’ve had doing labs simultaneously to date. When I introduced a lab component a few years ago into some of my precons, I started way before I did it at a big conference like PASS Summit in smaller doses as a proof of concept. No one else had tried to attempt 100 people doing labs at the same time where there is often crap connectivity. Crazy talk! I had no idea if it would be great or terrible, but I knew I wanted to do more. PASS put their faith in me and I’ve been lucky to be given a chance to do labs ever since, this year included.

I’m putting the finishing touches on the lab for the Summit precon in the next few days . This year’s precon is “Advanced SQL Server Availability and Storage Solutions”. Designing lab content and the instructional design around labs is one of the hardest things I do, and it’s even harder finding that balance in a one day event like a precon where you have a lot to talk about and demo, too. In one of my multi-day classes, it’s much easier. I do enjoy the challenge, and I urge you to reserve your spot because the last few years have completely sold out (and we’re nearly there already). I do cap the number of students who can attend since there’s only one of me and I can only have so many proctors. I hope to see you there.

Other Updates

  • I hit a big milestone. In August, I celebrated 10 years of going independent. I remember when I did it I thought let me give this a go for a year and see how it goes. One became two and the rest is history. Thanks to everyone who has supported me over the years and done business with me. I look forward to many more years doing this.
  • I was re-awarded my dual Microsoft MVP. I can’t believe I’m coming up on nearly 10 years as a Microsoft MVP. Time flies!
  • I did three sessions at VMworld in Las Vegas. Hope to be back next year.