patching Archives

May 16, 2019

Navigating the May 2019 Security Patches for SQL Server Deployments

This past week has been a doozy for security updates. There have been quite a few security alerts and patches. In this post, I’ll guide you through them as they relate to SQL Server and the operating systems that it runs on.

More Intel Processor Woes – Side-Channel Vulnerability/Microarchitectural Data Sampling (MDS)

After Spectre and Meltdown a few months back (which I cover in this blog post from January 4), another round of processor issues has hit the chipmaker. This one is for MDS (also known as a ZombieLoad) This one comprises the following security issues: CVE-2019-11091, CVE-2018-12126, CVE-2018-12127, and CVE-2018-12130. Whew! Fun fact: CVE stands for “Common Vulnerabilities and Exposures”.

As of now, this is only known to be an Intel, not AMD, issue. That is an important distinction here. The official Intel page on this issue can be found at this link. This issue does not exist in select 8th and 9th generation Intel Core processors as well as the 2nd generation Xeon Scalable processor family. (read: the latest stuff) This link in the Intel page is the detailed processor family list and what is/is not affected. If there are hardware-level fixes, those would come from your vendors such as Cisco, Dell, and HP. Many OSes as well as SQL Server have issued statements, guidance, and/or patches for this new Intel processor flaw.

Below is everything that relates to SQL Server or your underlying server configurations. Oh, and patch your systems RIGHT NOW.

Hyperthreading (Processor-level)

This is the biggie. If you look at most of the mitigations beyond patching, nearly everyone is, depending on the circumstance, recommending disabling hyperthreading (HT) on the physical processors. Guidance will vary from vendor to vendor. Doing this will require downtime as you will need to boot into BIOS/UEFI to do this. It will most likely impact performance if you were relying on HT, so check your SQL Server workloads. Before disabling HT, check the guidance from your vendors.

SQL Server

The Spectre/Meltdown KB article (4073225) was updated to include this new flaw. SQL Server does not require any specific patches for MDS. Please see the Windows Server section and specifically ADV190013 (linked below) for Microsoft’s full view on this.

Red Hat Enterprise Linux (RHEL)

SQL Server is currently supported on RHEL 7.3, 7.4, and 7.6. Red Hat did an excellent page explaining things which even has a video on how the attack could work. Even if you don’t use RHEL, the video is a good watch. Their specific vulnerability page for this new flaw can be found here. Read it, but if you want to skip to the conclusion, switch to the Resolve tab.

SUSE Linux Enterprise Server (SLES)

SQL Server is supported on SLES 12 SP2.

Here are their specific pages for each of the vulnerabilities and how to deal with them:

Ubuntu

SQL Server is supported on Ubuntu 16.04 which has a patch for MDS (the plain English stuff). The specificl Ubuntu Knowledge Base article for MDS can be found here which covers any updates that are needed.

VMware

VMware published a security advisory (VMSA-2019-0008) for the MDS issue. They cover all the versions affected and most importantly, which versions are the fixed ones. The main VMware Knowledge Base article linked you should be concerned with is KB67577. That article covers HT.

Windows Server

Microsoft updated their KB article on side-channel vulnerabilities (4072698) to include MDS. It has two very specific links: ADV190013 and KB4457591. Microsoft also has information for Azure IaaS VMs. I would read both carefully as they talk about the potential impact to performance. KB4457591 has a great section on whether or not you would need to disable HT, especially as it relates to Hyper-V.

From a fix perspective, the good news is that Microsoft shipped patches as part of the latest OS update released for Patch Tuesday covering Windows Server 2008, 2008 R2, 2012, 2016, and 2019. The link to the respective software is found at the bottom of ADV190013.

Amazon Web Services

AWS also had to deal with this flaw. Their information can be found in Security Bulletin AWS-2019-004. If you are using their IaaS services (EC2), you will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.

Google Cloud Platform

Like AWS and Azure, GCP has this problem, too. Similar to AWS (and Azure), the hosts were taken care of according to their posted notice. You will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.

Remode Code Execution Vulnerability in Remote Desktop Services (Terminal Services)

On May 14, Microsoft published security bulletin CVE-2019-0708 which details how a worm could happen with older versions of Windows desktop and server with Remote Desktop Services. The security bulletin describes how it would work, and what is – and is not – vulnerable. The good news is that if you are running Windows 8, 8.1, or 10 for the desktop or Windows Server 2012, 2012 R2, 2016, or 2019, you are safe. However, If your SQL Server instances are running on Windows 7 or Windows Server 2003, 2008, and 2008 R2 you are not. This one is bad enough that Microsoft patched Windows 7 and Windows Server 2003 which are out of support. While we do not see much Windows Server 2003 at our customers, there is a fair amount of Windows Server 2008 and 2008 R2 out there. The last version that was supported by Windows Server 2008 R2 was SQL Server 2014.

The security patches for Windows Server 2008 and 2008 R2 can be downloaded from this link. Note that this also affects Itanium processors if you still have those.

The security patch for WIndows Server 2003 can be downloaded from this link.

SQL Server 2017 – SQL Server Analysis Services

SQL Server released a rare security patch recently for SQL Server Analysis Services. According to KB4497700, There is a potential leak of restricted data that is not protected correctly by the Object-Level Security (OLS) system. This does not affect any other version of Analysis Services other than 2017, and is fixed in a GDR for RTM if you have not applied any CUs and a security update for CU14 (CU14 + GDR). CU14 is a re-release from March 25.

Other Stuff

There have been fixes to other things such as Adobe Flash, Internet Explorer, and more. I’ve coverd the major things, but anything that ships as part of the underlying OS is important to patch if it’s installed.

Call To Action

Security flaws need to be patched right away. Yes, that means downtime you probably didn’t plan, but do you want to be the next company to land in the headlines because you didn’t patch? We would be happy to help you figure out the right patching strategy and help you migrate to later platforms/versions that are more secure. Contact us today.

We have an upcoming webinar on June 5 Why Upgrades Matter where we will discuss the reasons you should do your best to stay up to date with releases and patching since there can be benefits, too. Read the abstract here and register today.

January 18, 2013

Patching Windows Server 2012 – The New Wild West

A week or so ago, I was excited to find out that Microsoft finally published a KB article that listed the recommended hotfixes and updates for Windows Server 2012 failover clusters (KB 2784261). Microsoft has done this for the major versions of Windows for the past few years (example: Windows Server 2008 R2 SP1 [KB 2545685]). These are great articles because they are very focused on WSFC. However, my excitement for the Windows Server 2012 article is severely dampered. Why?

If you compare the two, notice that W2K8 R2 SP1’s article lists all hotfixes. Look at what W2012 shows in Figure 1:

Figure 1. Cumulative Update for Windows Server 2012

Cumulative Updates? Oy! Let me preface this comment: I’m a Cluster MVP so inherently as much as I’m a SQL guy, I’m a Windows guy, too. I’m not about biting the hands that feed me but this scares the bejeezus out of me. Cumulative updates are a new concept for Windows, and I’m not sure what cumulative means in Windows. I know what it means in SQL Server. And if you click on the link in that article, you get no information (zero, nada, zilch, nil) about what is in that package. Compare that to a SQL Server CU such as SQL Server 2012 CU5. Even if Microsoft decided to have a hidden fix or two for whatever reason (and I have no reason to believe they have), you’re still getting a list of everything in there. That Windows update? You have no idea. Microsoft is basically asking for blind trust here.

WSFCs (as well as clustered instances of SQL Server [FCIs] and availability groups [AGs]) are usually put in because someone has a need to be up and running – hence mission critical. Something like a cumulative update sends off warning bells for me because instead of directed fixes like the W2K8 R2 SP1 KB which lists some individual hotfixes, you’re getting a ton more fixes that you will eventually get in, say, a service pack. Yes, you will consume them in a service pack, but that is a different beast than something like a CU. For example, on Windows, certain things you don’t want to touch if you don’t have to or until you need to – like TCP/IP. The stability of the networking stack is somewhat important, wouldn’t you say? This is why you don’t apply hotfixes willy nilly, and you certainly don’t apply a metric ton of them (we don’t even know how many fixes are in that CU) “just because”.

And let’s not even get into things like GDR branching (which I wrote about a little here). With a hotfix, you’ll only blow a few things off of the GDR branch. With a CU, you may blow many things off the GDR branch – some of which you didn’t intend to. That is not a good thing in my opinion, and when I have no clue what is in the update – that’s why this new patching method scares me. I don’t know why Microsoft does not list what’s in the CU for Windows, but right now, I would be wary of applying it unless you do your due diligence and test fully in a non-production environment.

My sincere hope is that Microsoft goes full disclosure with these new CUs for Windows and tells us what’s in them just like the SQL Server CUs show (and also link to the individual KBs). If not, even I will have a hard time recommending them to my customers unless I absolutely, 100% with no question know it fixes a specific problem of theirs. This to me is a step in the wrong direction in how we will patch our Windows Server 2012 boxes.

June 28, 2012

Maintaining Quorum During Windows Patching and Updating

I’ve been working with a customer to deploy their new Windows Server 2008 R2 SP1-based failover cluster that will have bot SQL Server 2008 R2 and SQL Server 2012 instances. One thing I always talk about in my classes as well as the presentations I focus on patching is this: you must maintain quorum during the patching and updating cycle. Quorum is one of the most important considerations in general for your clustered deployments (whether you’re implementing traditional failover clustering instances or availability groups), and it’s something that quite frankly, I find there’s a real lack of knowledge out there. It’s so important it’s most likely getting a whole chapter in my upcoming book Mission Critical SQL Server 2012.

I have talked to customers that have had SQL Server outages because their Windows guys when patching the underlying nodes for the Windows Server failover cluster (WSFC) rebooted too many at once, so there were not enough voters us, thus bringing the WSFC (and SQL) down when it wasn’t expected. This is NOT a scenario you want. While in theory it’s easy to control and keep track of when things reboot, this little gremlin in Figure 1 pops up after a little while reminding you it needs to be done:

Figure 1. Reboot needed warning

That’s all well and good, but the unskilled administrator who happens to see that message if they have to log onto the server may think nothing of it and just click Restart now. Yikes! The value of the reminder can be altered as shown in Figure 2, but you still need to worry about the reboot being done.

Figure 2. Changing the reminder value

There’s another dark underbelly I’ve seen these dialog boxes associated with: automatic updating of Windows via Windows Update. That is a much scarier situation because that means your servers could be updating 24×7 on their own, and some of those updates may even force reboots. Check your settings! I still see production servers at client sites that have automatic updating enabled. If they are, you need to have a serious discussion with your server admins as to why this is being done. Automatic updating is not in line with mission critical. Yes, it’s important to have a patch management strategy that keeps you up-to-date, but doing it smartly is the key. Not blindly.

February 15, 2012

Stop Automatically Updating Production Servers

Everything old is new again – well, not really. Many of the core concepts I teach or talk about be it in a session or onsite as part of knowledge transfer on a consulting engagement haven’t changed over the years. Implementations may change, but ultimately, the things that drive them do not. If you care about such arcane concepts like uptime, availability, and mission critical, you understand that downtime is bad. Very bad. Yet I still see and talk to quite a few customers who like to cause themselves unnecessary pain. I mean, it’s perfectly legal here in the USA to own a gun and shoot yourself in the foot, but why would you? It makes no sense.

Move Forward, Not Backward

A big part of staying mission critical is patching your servers – be it an application, the OS, hardware drivers, BIOS … you get the idea. You want to stay supported and up to date. The farther behind you are, the tougher it is to catch up or make the next major upgrade (say, SQL Server 2008 to SQL Server 2012) because you are not at the right minimum level. But that’s not the focus here – the focus is on those more regular patches that get installed such as security updates, browser updates, and so on. Those are the thorns in the proverbial rose bush.

Most companies have outage windows to apply updates and perform other maintenance to servers regardless of OS and other choices. That makes sense and is definitely needed – 100% uptime is largely a fallacy. (Sorry to burst your bubble if you really believed it was possible.) But whether it’s a security patch or a “routine”, more infrequent service pack, care must be taken.

Summertime … And The Updates Are Easy

George Gershwin is probably rolling in his grave, but I parody with the deepest respect. Microsoft has made it painfully easy (emphasis on painful) to update your servers. If you’re connected to the interwebz, you can set the handy, dandy Windows Update feature to download updates automatically. Even better – you can even have Windows automatically install them for you, too. What’s next? A tiled interface that makes Windows harder to use and makes your Server look like Windows Phone? Nah, something that silly could never never happen in a million years. Oh wait …

All kidding and sarcasm aside, look at your production servers: are they set to automatically update? If so, you are NOT mission critical. You may think you are, but you’re not. You’ll never convince me otherwise. Why?

Mission critical is a mindset. Part of that mindset is caring about things like SLAs, RTOs, and RPOs. Allowing servers to spontaneously download and update themselves – possibly requiring a reboot or worse causing one without you even asking – is 180 degrees from that mindset. You might as well just remove any redundancy and leave your chances to fate, who is a cruel mistress. You NEVER want to see this should you have to log into a production server:
Updating for the sake of updating is never the right approach. File under “just because you can, doesn’t mean you should”. Key tenets of mission critical are predictability, consistency, and stability. Servers, especially those participating in a clustered configuration (Windows and/or SQL Server), should be evaluated before applying an update to see if they actually need them. Some do NOT apply, and others may be specific to a configuration (such as a cluster). All patches should be tested prior to going on a production box, and some should be given more weight than others. For example, a fix that touches, say, TCP/IP would be one I would test the heck out of since servers don’t rely on networking or anything. 🙂
To add to #2, I’m going to stick on the consistency point for a moment: servers, especially ones considered mission critical, should all be configured the same. You don’t want varying patch levels and such. That spells trouble with a capital T. (Thank you, Meredith Wilson … I’m on a musical roll tonight!)
If you are going to do anything automatic, just download the updates but for heaven’s sake, don’t let the server automatically apply them. That would be a huge step forward.

The Bottom Line

Look, I realize many of you struggle on a daily basis with your servers and doing this one thing – automatic updates – seemingly makes your lives easier by taking a big task off your proverbial plate. In reality, it may be making it harder. And I also know that in smaller shops especially, you don’t always have the time or (wo)manpower to evaluate each patch/update. I get it. There’s a happy medium somewhere. Don’t turn short term gain into long term pain. There’s no easy or right answer here per se. You have to update. Do what’s best for you, but realize you may blow anything relating to uptime and reliability in the process.

Categories