Security Archives

January 17, 2020

Security Is an Availability Problem

I’ve been hearing more and more from people I know as well as news stories that security is on people’s minds in one way or another. It’s a fundamental IT tenet that security should be a priority. With the rise of ransomware as well as the number of hacks and data leaks going up, things have reached a fever pitch – and I do not think we have hit a peak yet.

The reason I’m writing this blog is because every data breach and ransomware incident becomes an availability problem in one way or another. Mission critical encompasses many things – including both security and availability. Until recently not many saw the two as being in the same boat. Welcome to my world. Come in, have a dip in the pool – the water’s fine!

Let’s take the example of one of the most recent victims of ransomware, Travelex, one of the world’s largest travel-related companies. Travelex was hit with ransomware on December 31, 2019. Happy New Year, right? It hasn’t been for Travelex. They do business with lots of people including direct consumers. One of their businesses is money exchange at airports – I’ve seen their kiosks and storefronts at numerous ones including Heathrow.

They have been silent until today. On their customer information page in the UK about the incident, a video was posted from Tony D’Souza, the CEO of Travelex. First, taking nearly three weeks for a response to a very public incident is not necessarily a good thing in my estimation. I get the need to deal with things. I’m not perfect in all my communications but I’m also not Travelex. A big part of these incidents is incident management – including the public side of things. The customer page is reach, but I’m talking about the overall PR effort.

As of today, January 17, 2020, the screen grab below is their website’s front page in the USA (click to make larger).

The Travelex home page as of 1/17/20

They are partially back up and working. According to a BBC report,

However, while he said the system used by staff is now working, there was no word on when the firm’s main UK website would be returned to service.

That means customers are still unable to order currency online, either from Travelex itself or through the network of banks that use its services, including Barclays, Lloyds, RBS, and the finance websites of Sainsbury’s and Tesco.

Ouch. That is a lot of lost business for their partners AND Travelex themselves with seemingly no end in sight. You get the idea.

Security starts well before someone accesses a databases or a system. FedEx had an issue a few years ago that cost them hundreds of millions of dollars because it got in through a subsidiary in Europe. IT needs to be able to cope with the modern threats. Most do a great job, but sadly, there will always be one attack vector you may not have thought of. Covering as many as you can is important. Not doing anything or saying, “No one will care” should have people polishing their resumes. Just look at the City of Baltimore which got hit with ransomware; it cost them at least $18.2 million.

Security is more than worrying about PII, credit card numbers, HIPAA, but as data professionals, that is one of our primary concerns. For every company that worries about what port to configure for SQL Server and frets that 1433 is insecure, worry about bigger problems like prioritizing the security of data at rest (including backups) as well as on the wire. A port scanner will find that SQL Server instance pretty quickly. Obfuscation may slow a hacker down and annoy them, not stop them. Put some effort into a robust backup strategy that has your backups safe, offsite, and tested so you know you can restore them if you get hit with something like ransomware. Even if you secure SQL Server but you have a ransomware incursion, you still may need those backups if you can’t access the systems. Everything matters here.

We always take security into account when we’re helping our customers design or evaluate solutions. Security is multi-layered. You do not want to let a security problem become an availability issue. Could your business survive being nearly completely down for over two weeks and counting like Travelex? I doubt it. Want to make sure you never find out the answer to that question? Contact us today.

May 16, 2019

Navigating the May 2019 Security Patches for SQL Server Deployments

This past week has been a doozy for security updates. There have been quite a few security alerts and patches. In this post, I’ll guide you through them as they relate to SQL Server and the operating systems that it runs on.

More Intel Processor Woes – Side-Channel Vulnerability/Microarchitectural Data Sampling (MDS)

After Spectre and Meltdown a few months back (which I cover in this blog post from January 4), another round of processor issues has hit the chipmaker. This one is for MDS (also known as a ZombieLoad) This one comprises the following security issues: CVE-2019-11091, CVE-2018-12126, CVE-2018-12127, and CVE-2018-12130. Whew! Fun fact: CVE stands for “Common Vulnerabilities and Exposures”.

As of now, this is only known to be an Intel, not AMD, issue. That is an important distinction here. The official Intel page on this issue can be found at this link. This issue does not exist in select 8th and 9th generation Intel Core processors as well as the 2nd generation Xeon Scalable processor family. (read: the latest stuff) This link in the Intel page is the detailed processor family list and what is/is not affected. If there are hardware-level fixes, those would come from your vendors such as Cisco, Dell, and HP. Many OSes as well as SQL Server have issued statements, guidance, and/or patches for this new Intel processor flaw.

Below is everything that relates to SQL Server or your underlying server configurations. Oh, and patch your systems RIGHT NOW.

Hyperthreading (Processor-level)

This is the biggie. If you look at most of the mitigations beyond patching, nearly everyone is, depending on the circumstance, recommending disabling hyperthreading (HT) on the physical processors. Guidance will vary from vendor to vendor. Doing this will require downtime as you will need to boot into BIOS/UEFI to do this. It will most likely impact performance if you were relying on HT, so check your SQL Server workloads. Before disabling HT, check the guidance from your vendors.

SQL Server

The Spectre/Meltdown KB article (4073225) was updated to include this new flaw. SQL Server does not require any specific patches for MDS. Please see the Windows Server section and specifically ADV190013 (linked below) for Microsoft’s full view on this.

Red Hat Enterprise Linux (RHEL)

SQL Server is currently supported on RHEL 7.3, 7.4, and 7.6. Red Hat did an excellent page explaining things which even has a video on how the attack could work. Even if you don’t use RHEL, the video is a good watch. Their specific vulnerability page for this new flaw can be found here. Read it, but if you want to skip to the conclusion, switch to the Resolve tab.

SUSE Linux Enterprise Server (SLES)

SQL Server is supported on SLES 12 SP2.

Here are their specific pages for each of the vulnerabilities and how to deal with them:

Ubuntu

SQL Server is supported on Ubuntu 16.04 which has a patch for MDS (the plain English stuff). The specificl Ubuntu Knowledge Base article for MDS can be found here which covers any updates that are needed.

VMware

VMware published a security advisory (VMSA-2019-0008) for the MDS issue. They cover all the versions affected and most importantly, which versions are the fixed ones. The main VMware Knowledge Base article linked you should be concerned with is KB67577. That article covers HT.

Windows Server

Microsoft updated their KB article on side-channel vulnerabilities (4072698) to include MDS. It has two very specific links: ADV190013 and KB4457591. Microsoft also has information for Azure IaaS VMs. I would read both carefully as they talk about the potential impact to performance. KB4457591 has a great section on whether or not you would need to disable HT, especially as it relates to Hyper-V.

From a fix perspective, the good news is that Microsoft shipped patches as part of the latest OS update released for Patch Tuesday covering Windows Server 2008, 2008 R2, 2012, 2016, and 2019. The link to the respective software is found at the bottom of ADV190013.

Amazon Web Services

AWS also had to deal with this flaw. Their information can be found in Security Bulletin AWS-2019-004. If you are using their IaaS services (EC2), you will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.

Google Cloud Platform

Like AWS and Azure, GCP has this problem, too. Similar to AWS (and Azure), the hosts were taken care of according to their posted notice. You will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.

Remode Code Execution Vulnerability in Remote Desktop Services (Terminal Services)

On May 14, Microsoft published security bulletin CVE-2019-0708 which details how a worm could happen with older versions of Windows desktop and server with Remote Desktop Services. The security bulletin describes how it would work, and what is – and is not – vulnerable. The good news is that if you are running Windows 8, 8.1, or 10 for the desktop or Windows Server 2012, 2012 R2, 2016, or 2019, you are safe. However, If your SQL Server instances are running on Windows 7 or Windows Server 2003, 2008, and 2008 R2 you are not. This one is bad enough that Microsoft patched Windows 7 and Windows Server 2003 which are out of support. While we do not see much Windows Server 2003 at our customers, there is a fair amount of Windows Server 2008 and 2008 R2 out there. The last version that was supported by Windows Server 2008 R2 was SQL Server 2014.

The security patches for Windows Server 2008 and 2008 R2 can be downloaded from this link. Note that this also affects Itanium processors if you still have those.

The security patch for WIndows Server 2003 can be downloaded from this link.

SQL Server 2017 – SQL Server Analysis Services

SQL Server released a rare security patch recently for SQL Server Analysis Services. According to KB4497700, There is a potential leak of restricted data that is not protected correctly by the Object-Level Security (OLS) system. This does not affect any other version of Analysis Services other than 2017, and is fixed in a GDR for RTM if you have not applied any CUs and a security update for CU14 (CU14 + GDR). CU14 is a re-release from March 25.

Other Stuff

There have been fixes to other things such as Adobe Flash, Internet Explorer, and more. I’ve coverd the major things, but anything that ships as part of the underlying OS is important to patch if it’s installed.

Call To Action

Security flaws need to be patched right away. Yes, that means downtime you probably didn’t plan, but do you want to be the next company to land in the headlines because you didn’t patch? We would be happy to help you figure out the right patching strategy and help you migrate to later platforms/versions that are more secure. Contact us today.

We have an upcoming webinar on June 5 Why Upgrades Matter where we will discuss the reasons you should do your best to stay up to date with releases and patching since there can be benefits, too. Read the abstract here and register today.

January 30, 2018

Downtime Due to Poor Security Will Cost You Money

Sometimes when I speak or in some of my writings, I discuss the cost of downtime and how knowing that number can help you devise a better solution. That number is often company-, and sometimes industry-specific. For example, if processing credit cards, a company may have a financial hit from the customer if it cannot process a transaction either fast enough or, worst case, not at all. That adds up when you have even a five or ten minute outage. Processing a credit card transaction is not the same as loss of life in a hospital, hence needing to account for a system and its solution individually.

However, as of this week, if you have a company or work in the UK, things just got a whole lot more interesting. The UK government officially released a statement on January 28 which affects “critical industries”. Long story short: if you fall under the classification which seems to be limited right now to energy, transport, water, and health firms, you could be fined up to £17 million ($24 million in US Dollars at today’s exchange rate) in the event of a cyber attack taking you down. It was the WannaCry outages that precipitated the response (as an example, FedEx says WannaCry cost them about $300 million US Dollars). Remember this doozie from British Airways? Also covered under this new Network and Information Systems (NIS) Directive; it’s not just about security, but includes other things like power outages, hardware failure, and environmental hazards.

The NIS Directive is effective as of May 10, 2018, and is essentially based on this Consultation on the Security of Network and Information Systems Directive from August of 2017, and the outcome/latest is the document Security of Network and Information Systems: Analysis of responses to public consultation which was just published. I wouldn’t be surprised to see other places around the world adopt a similar stance. For some this may proverbially add insult to injury since everyone is already dealing with GDPR which goes into effect May of 2018 as well.

I’ve always talked about how security is a key component of availability. The UK government is literally putting their money where their mouth is. The NIS Directive isn’t meant to start with fines. The press release states the following:

Fines would be a last resort and will not apply to operators which have assessed the risks adequately, taken appropriate security measures and engaged with regulators but still suffered an attack.

That is actually good news – it’s not shoot, aim, fire. However, what that means is that you need to do the right steps to be prepared to avoid it if possible. That includes things like patching servers and having a strategy to do so in a timely manner is going to matter. Things like the recent Spectre/Meltown chip flaws (which I put everything you need to know as it relates to SQL Server in one place here) will not be a “kick the can down the road” exercise. To that point, I’m still seeing people saing they don’t need to worry about patching for Spectre and Meltdown. YOU DO. Yes, it sucks you may see a performance hit, but would you rather be down instead? I do not think so. I’d rather be slower and up than down and out.

It is always better to be proactive than reactive, and SQLHA can certainly help you assess where you are. We can help address and mitigate issues related to availablity and disaster recovery (which would help with things like accounting for power outages and hardware failure), but also devise realistic patching strategies that work. Max and I have done these types of things for some of the largest systems in the world over the course of our careers. It doesn’t matter if you are a small company or one of the biggest in the world – we’re happy to help! Just reach out.

January 4, 2018

The No Good, Terrible Processor Flaw and SQL Server Deployments – Nearly Everything You Need To Know

UPDATED JANUARY 18

If you haven’t been paying attention, a serious security flaw in nearly every processor made in the last ten years was discovered. Initially it was thought to be just Intel, but it appears it’s everyone. Official responses:

AMD (downplaying the issue)
ARM (great response)
Intel (oy)

There are two bugs which are known as Meltdown and Spectre. The Register has a great summarized writeup here – no need for me to regurgitate. This is a hardware issue – nothing short of new chips will eradicate it. That said, pretty much everyone who has written an OS, hypervisor, or software has (or will have) patches to hopefully eliminate this flaw. This blog post covers physical, virtualized, and cloud-based deployments of Windows, Linux, and SQL Server.

The fact every vendor is dealing with this swiftly is a good thing. The problem? Performance will most likely be impacted. No one knows the extent, especially with SQL Server workloads. You’re going to have to test and reset any expectations/performance SLAs. You’ll need new baselines and benchmarks. There is some irony here that it seems virtualized workloads will most likely take the biggest hit versus ones on physical deployments. Time will tell – no one knows yet.

What do you need to do? Don’t dawdle or bury your head in the sand thinking you don’t need to do anything and you are safe. If you have deployed anything in the past 10 – 15 years, it probably needs to be patched. Period. PATCH ALL THE THINGS! However, keep in mind that besides this massive scope, there’s pretty much a guarantee – even on Linux – you will have downtime associated with patching.

Below is a summarized list of the biggest players for SQL Server-related deployments covering physical, virtualized, and cloud. Finding all these links took some time, so I figured I should put them all in one convenient place for everyone. Each vendor and product has its own guidance and response, and there may be updates to what I’ve posted but this should get you started. What I did not list is all the hardware vendors. Check with Dell, HP, Hitachi, etc. to see if there are firmware/BIOS/UEFI updates as well.

If you want help with new baselines and benchmarks, or just assistance in sorting this out and coming up with a plan, contact us. If you are on an older, unsupported version of one of the things below that will not be patched, you should strongly consider accelerating your upgrade/migration plans. This is also something we can help with.

AWS

If you’re running workloads using Amazon Web Services, their response can be found here. It appears that their stuff has been patched, but if you’re running IaaS VMs with EC2, you’re going to have to patch your OSes and software in them.

Azure

Microsoft’s response for Azure customers can be found here. They also did a KB article (4073235) which can be found here. Like AWS, they’ve patched the underlying stuff. If you are running IaaS VMs, you’ll need to make sure they are patched properly unless you have automatic patching and running WIndows Server (see below).

GCP

If you’re using the Google Cloud for your workloads, their response is here. As with AWS and Azure, they took care of the base, but you’re responsible for your IaaS VMs/workloads.

Red Hat Enterprise Linux

Red Hat’s response can be found here which talks more about the impact and the performance. To understand the patching side of things, refer to this. SQL Server is supported on 7.3 or later, and those builds have patches available (although I didn’t see 7.4 listed as of the writing of this post, just 7.3). CentOS had its patches released on January 5th.

SQL Server

Microsoft did a great KB (4073225) article summarizing your options which you can read here. Microsoft is patching SQL Server 2008 and later, but reality is because SQL Server 2005 can technically run on Windows Server 2008 and 2008 R2, it would be affected but it’s out of support. I don’t see Microsoft doing anything for it. This would be a good time to consider when you are planning to upgrade or migrate. As of January 18th, patches are available for 2008, 2008 R2, 2012, 2014, 2016, and 2017.

Microsoft lists five scenarios in the KB. Please read them carefully and make the right choice(s), but the absolute wrong choice is to patch nothing.

SLES

If you’re using SLES for your SQL Server deployment, their information can be found here and here (KB). It appears they’ve patched 11 SP3-LTSS through 12 SP3. Although not officially supported for SQL Server, the OpenSUSE info can be found here.

Ubuntu

Here is Ubuntu’s high level response. Here is the link to where to get the patches. 16.04 is covered, which is important for SQL Server.

VMware

VMware posted a security announcement with regards to this issue as well as a blog post. So if you’re using ESXi as your hypervisor, you need to read it. As of the writing of this blog post, it looks like they patched ESXi 5.5, 6.0, and 6.5. It does not look like they are patching anything older than 5.5. There are two vulnerability alerts: VMSA-2018-002.1 and VMSA-2018-0004.2. VMware patched CVE-2017-5715 and CVE-2017-5753. VMware is not affected by CVE-2017-5754, so no patch exists for that.

If you are not on ESXi 5.5 or later, I strongly encourage you to upgrade as soon as possible, and you want that anyway since 6.0 is the first version of ESXi to support vMotion of clustered configurations of SQL Server.

Windows Server

Similar to SQL Server, Microsoft wrote a KB article (4072698) for this issue that can be found here. As of the writing of this blog post, Microsoft has released patches for Windows Server 2008 R2, 2012 R2, 2016, and RS3 (AKA 1709). Hopefully 2008 and 2012 will get patches soon (still the case as of 1/18). If you have automatic updating enabled, the fixes should be picked up by Windows Update. If not, apply them manually. If you’re still running Windows Server 2003/R2 or earlier, I don’t see Microsoft going back and patching. You’re on your own there. The mitigation would be to upgrade ASAP to something that is patched. If you’re running 2008 or 2012 and MS does not release a patch, I strongly urge you to consider upgrading/migrating your deployments to something that is patched.

More information about the January 3rd patch can be found in KB 4072699. Note that due to some anti-virus vendors, unless the registry is changed, you may not automatically see the patch.

XEN

If you’re using XEN as your hypervisor, they did a writeup as well. Things don’t look as rosy right there for now because they don’t seem to have patches for everything yet as of the time I’m writing this blog post. I’m sure that will change.

Other Stuff

Apple – If you’re running High Sierra, Sierra, or El Capitan, it looks like Apple took care of this back in December of 2017. See this for more infomation.

Browsers

Chrome – It looks like Google is going to release a patch for Chrome later in January. See this link for more information.
Firefox – Version 57 or later has the proper fixes. See this blog for more information, so patch away!
Edge and Internet Explorer – Microsoft has a blog post here. It looks like the January security update (KB4056890) takes care of that. So if you’re using either of these browsers, please update your OSes as soon as possible.

Hardware Vendors

This isn’t an exhaustive list, but will hopefully help some of you. A full list of vendors can be found here.

Cisco (thanks to the commenter below)
Dell Dell’s list of servers and storage is here. Here is a link for Dell’s Data Security product.
Hewlett Packard Enterprise HPE is continually updating this post with the various servers and such they sell with compliance and patch links.

Popular Posts

Categories