By: Allan Hirt on June 10, 2019 in Availability Groups, Kubernetes, SQL Server 2019 | No Comments
Happy Monday everyone.
I haven’t written much about them yet (key emphasis there …) but AGs now being supported for containers in SQL Server 2019 is a big deal. Recently, SQL Server 2019 CTP 3.0 was released, but there’s a slight problem: if you try to deploy an AG with Kubernetes, you may see the following errors when trying to deploy the pods with the YAML that contains their definition. The services (i.e. instances of SQL Server) get created, but the pods do not.
YAML execution error (click to make bigger)
The fix, however, is easy. For CTP 3.0, a few things changed for the SQL Server operator. You have to modify the operator.yaml file with two changes which are documented below.
The first new thing is the word “update” is added to the verbs list in the section below.
The second is a new section that now must exist after MSSQL_K8S_NAMESPACE.
The updated operator.yaml file can be downloaded from my repository.
Also, remember in the YAML file where you define the pods, update the following two lines:
Hope this helps some of you.
By: Allan Hirt on May 16, 2019 in Patching, Red Hat Enterprise Linux, Security, SQL Server, SUSE Linux Enterprise Server, Ubuntu, Windows Server | No Comments
This past week has been a doozy for security updates. There have been quite a few security alerts and patches. In this post, I’ll guide you through them as they relate to SQL Server and the operating systems that it runs on.
More Intel Processor Woes – Side-Channel Vulnerability/Microarchitectural Data Sampling (MDS)
After Spectre and Meltdown a few months back (which I cover in this blog post from January 4), another round of processor issues has hit the chipmaker. This one is for MDS (also known as a ZombieLoad) This one comprises the following security issues: CVE-2019-11091, CVE-2018-12126, CVE-2018-12127, and CVE-2018-12130. Whew! Fun fact: CVE stands for “Common Vulnerabilities and Exposures”.
As of now, this is only known to be an Intel, not AMD, issue. That is an important distinction here. The official Intel page on this issue can be found at this link. This issue does not exist in select 8th and 9th generation Intel Core processors as well as the 2nd generation Xeon Scalable processor family. (read: the latest stuff) This link in the Intel page is the detailed processor family list and what is/is not affected. If there are hardware-level fixes, those would come from your vendors such as Cisco, Dell, and HP. Many OSes as well as SQL Server have issued statements, guidance, and/or patches for this new Intel processor flaw.
Below is everything that relates to SQL Server or your underlying server configurations. Oh, and patch your systems RIGHT NOW.
This is the biggie. If you look at most of the mitigations beyond patching, nearly everyone is, depending on the circumstance, recommending disabling hyperthreading (HT) on the physical processors. Guidance will vary from vendor to vendor. Doing this will require downtime as you will need to boot into BIOS/UEFI to do this. It will most likely impact performance if you were relying on HT, so check your SQL Server workloads. Before disabling HT, check the guidance from your vendors.
The Spectre/Meltdown KB article (4073225) was updated to include this new flaw. SQL Server does not require any specific patches for MDS. Please see the Windows Server section and specifically ADV190013 (linked below) for Microsoft’s full view on this.
Red Hat Enterprise Linux (RHEL)
SQL Server is currently supported on RHEL 7.3, 7.4, and 7.6. Red Hat did an excellent page explaining things which even has a video on how the attack could work. Even if you don’t use RHEL, the video is a good watch. Their specific vulnerability page for this new flaw can be found here. Read it, but if you want to skip to the conclusion, switch to the Resolve tab.
SUSE Linux Enterprise Server (SLES)
SQL Server is supported on SLES 12 SP2.
Here are their specific pages for each of the vulnerabilities and how to deal with them:
SQL Server is supported on Ubuntu 16.04 which has a patch for MDS (the plain English stuff). The specificl Ubuntu Knowledge Base article for MDS can be found here which covers any updates that are needed.
VMware published a security advisory (VMSA-2019-0008) for the MDS issue. They cover all the versions affected and most importantly, which versions are the fixed ones. The main VMware Knowledge Base article linked you should be concerned with is KB67577. That article covers HT.
Microsoft updated their KB article on side-channel vulnerabilities (4072698) to include MDS. It has two very specific links: ADV190013 and KB4457591. Microsoft also has information for Azure IaaS VMs. I would read both carefully as they talk about the potential impact to performance. KB4457591 has a great section on whether or not you would need to disable HT, especially as it relates to Hyper-V.
From a fix perspective, the good news is that Microsoft shipped patches as part of the latest OS update released for Patch Tuesday covering Windows Server 2008, 2008 R2, 2012, 2016, and 2019. The link to the respective software is found at the bottom of ADV190013.
Amazon Web Services
AWS also had to deal with this flaw. Their information can be found in Security Bulletin AWS-2019-004. If you are using their IaaS services (EC2), you will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.
Google Cloud Platform
Like AWS and Azure, GCP has this problem, too. Similar to AWS (and Azure), the hosts were taken care of according to their posted notice. You will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.
Remode Code Execution Vulnerability in Remote Desktop Services (Terminal Services)
On May 14, Microsoft published security bulletin CVE-2019-0708 which details how a worm could happen with older versions of Windows desktop and server with Remote Desktop Services. The security bulletin describes how it would work, and what is – and is not – vulnerable. The good news is that if you are running Windows 8, 8.1, or 10 for the desktop or Windows Server 2012, 2012 R2, 2016, or 2019, you are safe. However, If your SQL Server instances are running on Windows 7 or Windows Server 2003, 2008, and 2008 R2 you are not. This one is bad enough that Microsoft patched Windows 7 and Windows Server 2003 which are out of support. While we do not see much Windows Server 2003 at our customers, there is a fair amount of Windows Server 2008 and 2008 R2 out there. The last version that was supported by Windows Server 2008 R2 was SQL Server 2014.
The security patches for Windows Server 2008 and 2008 R2 can be downloaded from this link. Note that this also affects Itanium processors if you still have those.
The security patch for WIndows Server 2003 can be downloaded from this link.
SQL Server 2017 – SQL Server Analysis Services
SQL Server released a rare security patch recently for SQL Server Analysis Services. According to KB4497700, There is a potential leak of restricted data that is not protected correctly by the Object-Level Security (OLS) system. This does not affect any other version of Analysis Services other than 2017, and is fixed in a GDR for RTM if you have not applied any CUs and a security update for CU14 (CU14 + GDR). CU14 is a re-release from March 25.
There have been fixes to other things such as Adobe Flash, Internet Explorer, and more. I’ve coverd the major things, but anything that ships as part of the underlying OS is important to patch if it’s installed.
Call To Action
Security flaws need to be patched right away. Yes, that means downtime you probably didn’t plan, but do you want to be the next company to land in the headlines because you didn’t patch? We would be happy to help you figure out the right patching strategy and help you migrate to later platforms/versions that are more secure. Contact us today.
We have an upcoming webinar on June 5 Why Upgrades Matter where we will discuss the reasons you should do your best to stay up to date with releases and patching since there can be benefits, too. Read the abstract here and register today.
By: Allan Hirt on April 30, 2019 in Consulting | No Comments
In Part I of this blog series, What You Can Learn from Hertz Part I: Hire the Right Consultants for the Right Reasons, I discussed why good consultants do not and suck and when it makes sense to use them. Hertz hired Accenture for the right reasons – it’s right there in the lawsuit:
Hertz did not have the internal expertise or resources to execute such a massive undertaking; it needed to partner with a world-class technology services firm.
Unfortunately it appears they did not deliver, which devolved into the lawsuit. If you want to read up on another famous epic consulting fail, look at Oracle and the State of Oregon.
An engagement needs to be structured and run to ensure fiascos are avoided. Contracts should make sense for both sides. At the same time, there are environmental, political, and budgetary aspects to why projects can get derailed and ultimately collapse. The customer/client has a lot to do with this, and they can be a big part – or even the sole reason – a project gets tanked. Competent consultants can’t compete with a customer getting in their own way. This blog post will cover what skin in the game the customer must have for successful consulting engagements.
Management must empower the team for success. Good upper management is transparent. Success, especially on large projects. usually happens because of top down support. Skunkworks projects also can succeed, but the risk – and sometimes the reward – is much higher. Both are valid approaches, and we’ve worked on many different kinds of projects over the years. When reaching out to a consulting firm, hopefully you’ll make it clear to them if you have management buy-in or not.
Make Resources Available
The consultant arrives at your office the first day of the engagement. Unfortunately, the key people he or she is supposed to work with are out at an offsite meeting for three days. Clearly that’s an issue. Now the company is going to pay for three days of paid time and the expenses of showing up with no real results. This is not the consultant’s fault. The start date should have been moved/rescheduled, but no one from the customer communicated this scheduling snafu to the consultant (or the company who sent them). Even when asked the week before if everything was fine, the company gave no indication it was not. Disconnects are expensive.
Good consultants are self sufficient, but we need information and feedback, too. This request time and input from key resources in your company to keep things moving. Projects – especially those with tight timelines – need to ensure that people are available for in person meetings or calls and answer e-mail (or IM) in a timely manner otherwise things grind to a hald.
Similarly, if a consultant needs access to systems, that process should happen sooner rather than later. That means once the ink is dry on the proverbial paperwork for the engagement, start whatever process it takes. Sometimes this takes a few days – or a few weeks – to set up if things like RSA tokens are involved. This eats time in an engagement and slows us down when we don’t have access to what we need to get the job done.
This is also a respect and common courtesy thing. We understand you have a day job and are not there to babysit us. Heck, I don’t want a babysitter! I know you have other stuff to do, but your company is also paying me to do a job and meet any scope and timeline that has been agreed upon. Respect a consultant’s time as they do yours.
Here’s a practical example: I’m writing a document as one of the deliverables on the project. In the contract, we agree on a review timeline once the document is submitted. The people who are reviewing the document need to be a) available to review when it is turned over b) able to deliver comments/revisions back to me on the agreed upon review cycle. Anything outside of that is most likely going to change the delivery date.
If you don’t have skin in the game to meet your deadlines, how can you expect us to meet them?
Establish Milestones and Timeline
Every project needs a plan, even if it is high level. A detailed one with dates and dependencies is much better for everyone. Whether the consultant/consultancy drives it or it is someone at the client, there needs to be someone making sure everything is on track. Customers are not fans of paying for project management since it seems like overhead, but good project management is fairly transparent and non-invasive. It also allows one person to be the point for any questions, and that person can get them to the right personnel (hired or in-house).
For Hertz, according to the lawsuit, they used Accenture for project management. Section 42 of the lawsuit calls out the failure of said project management. The example given is the removal of key resources from the project. Major changes like that should be communicated and the impact measured. We do not know why Accenture did that. This to me was an important phrase in the lawsuit:
Because Accenture failed to properly manage and perform the services, the go-live date was postponed twice, first until January 2018, and then until April 2018. By that point, Hertz no longer had any confidence that Accenture was capable ofcompleting the project, and Hertz terminated Accenture.
Forget the termination part – they missed the launch more than once. For a big public company like Hertz, this can be a public relations nightmare especially if you announce these things.
In the plan, key dates/events – milestones – must be set and agreed upon by both parties. If they need to be adjusted, again, everyone needs to agree to those changes. For fixed price engagements, milestones are absolutely necessary because for consultants, payments are usually tied to them. Even if payments are not tied to key dates, these should be big red flags. The nagging questions I have here are: why were tripwires were ignored and how did this not get nipped in the bud sooner?
I do not know why (or how) Hertz let $32M pass before they terminated things. There were two phases to the project, and the first obviously went well enough for Hertz to pony up for the second. Most customers I know would have cut the cord much sooner.
Ensure there is Testing, Acceptance Criteria, and Oversight
Besides milestones, most work should have a set of minimum documented criteria for success. You have carefully crafted scopes of work for a reason.
I do not know the details, but I am surprised by statements like the ones below in the lawsuit.
Without Hertz’s knowledge or approval, Accenture deliberately disregarded the extensibility requirement and wrote the code so that it was specific to the Hertz brand in North America and could not be used for the Hertz global brand or for the Dollar and Thrifty brands.
Going rogue on a client is never acceptable. Sometimes there are honest-to-goodness technical challenges or limtations that crop up and may make something impossible to do. I’ve been there. Those are hard conversations, but you find a way around them. Deciding to go another way just because? Nope, nope, nope.
In addition, Accenture failed to perform proper testing of the software that it developed. Accenture did not perform tests on many components of the system. When Accenture did perform tests, they were seriously inadequate, to the point of being misleading.
I have a QA background so testing is in my blood. People should test their work. However, where was Hertz in this? Did they leave all the testing to Accenture and had no resources touching the work along way? Did Accenture just do smoke and mirrors when showing “working” versions – if they did that at all? Maybe I missed that in the lawsuit, but this goes back to skin in the game. I’m a consultant, but if you’re paying me to do something, you should check in if we’re not doing so already. Have weekly status meetings. To let it get that far down the garden path is something that should have never happened. Clearly it does (see: lawsuit), but in theory, this could have been stopped sooner. Maybe it couldn’t. I was not there so I can’t say everything that went down on either side.
Avoid Scope Creep
One of the worst things that can happen on a project is scope creep. Scope creep is at some point during the project, the requirements change for whatever reason. Sometimes that is for valid reasons (for example, deploying to Azure and something changes on the backend and code needs to be refactored). Other times people just want to add new functionality in. It is more the latter than the former that is the problem, but the end result is the same: projects slip because of scope creep. Moving the ball but not the end date is not generally a good thing. All changes to scope need to be formally agreed to by both sides. The reason you put these things in contracts is that everybody knows what success looks like at the end. You deal with the unforseen, but avoid the ones that will change end dates just because someone now wants a widget to be purple.
Stick to the Budget
Changing the budget while a project is in progress is always a tricky proposition. More money for the right reasons is good, taking away money is bad. There are two aspects of this: affecting the overall funding of the project including the consulting dollars, and then just affecting other aspects of the project but the consulting bucket is untouched. The former is concerning for obvious reasons. If we agreed to X in the contract, that is the expectation. There’s generally no re-negotiating especially if you want us to do the same amount of work. However, that is not what I am talking about; it’s the latter scenario.
Let me give you an example. Years ago, I was working on a data center migration project under a big consultancy firm. I was leading the SQL Server team. We were supposed to get our own dedicated storage unit for the SQL Server implementations, and all our plans were based on this promise. A few months into the project, after I had delivered our storage requirements and verified that what the SQL team was getting would be suitable, I was told that we were not getting said storage. We would be sharing one unit with the file server and Exchange folks. That was not good news. Doing the “simple” math, the client’s SQL capacity (forget performance for a second) was in theory cut by 2/3 in the shared storage plan.
How did I handle it? We had already rolled up our sleeves to fully understand the environment and we had enough storage for launch. Due to changes in the application, we could not measure growth until we flipped the switch. Once we did, we predicted when storage capacity would run out. I then presented the findings and let them know their options. At that point, I was set to roll off the project. I was never told how they resolved the issue.
In that example, it was purely a financial decision made much higher up the food chain that impacted everyone. I know the Exchange and file folks were not thrilled, either. I can only speak to how things played out from my point of view.
In the initial stages of trying to work with a consultancy and putting a contract in place, they may ask about your budget. Be honest and transparent because we are with you. We understand you may be pitting us against someone else to see who will be a better fit for a variety of reasons including financial. That said, reasonable consultants will put together a good scope of work that will get you what you need and come in on or under budget.
As mentioned in Part I, expertise does not grow on trees. It’s better to hire the right consultants first than land up with a costlier project because you hired a square peg for a round hole. Sometimes spending a touch more than you thought will save you much more money down the road. Budget matters, no question. Good consultants get this loud and clear – I know I always have.
Remember to work out travel things, too. If you want someone onsite, understand what that means. If you request on a Friday to have someone there on Monday, the costs involved are often much higher than planning a trip in advance. It’s up to everyone involved to control costs where possible since the consultant (or their company) is laying out the money ahead of time, but be smart about things. Having a consultant fly across the world in the back row of a plane and be dead tired to work is counterproductive.
We always hear the big consulting horror stories because they land up in the news in one way or another. The fact that way more consulting engagements are successful and do not land up splashed in the headlines is a good thing. It’s the bad ones that make many think consultants are evil, but the companies hiring us need to ensure they’re doing their part to ensure successful projects. When it all works, everyone is happy. Don’t let news articles like this scare you. Hire us! 🙂
If Accenture’s side of the lawsuit pops up, I’ll most likely do some sort of Part III since I’d be curious to see their side of the story.
By: Allan Hirt on April 29, 2019 in SQLHAU, Training | No Comments
In honor of Star Wars Day (May 4th), all of SQLHAU’s training will be 40% off. That’s 40% off all classes – in person and online including the upcoming classes in Chicago this August. For a full list of classes, see the SQLHA Events page. Use discount code MAYTHE4TH.
Because the 4th falls on a Saturday, the flash sale will start Friday, May 3 and end at 11:59 PM on Sunday, May 5.
May the force of knowledge be with you!
By: Allan Hirt on April 26, 2019 in Consulting | No Comments
Stop me if you’ve heard this story before: company hires consultants for a “too big to fail” project, but the project goes awry. Deadlines are missed. Deliverables do not hit the mark. Lots of money is spent and to date, no real progress has been made. The consultants are fired even though they offered to fix the problem they caused … for more money. There is always risk in a “too big to fail” project. Risk can be managed and mitigated by people who know who what they are doing to deliver the project on time and (hopefully) on budget. That’s why you hired the experts at the consultancy, right?
This week, the Register reported that Hertz is suing Accenture for $32 million because Accenture allegedly could/did not deliver on what was contractually promised to Hertz. I suggest you also read the comments, as they inspired me to write this post and its follow up which will be coming in a few days.
Every time I hear a story like the Hertz one, I know deep down my job as a consultant (and a business owner) gets a little harder. I’ve been doing consulting in one form or another for about 20 years as my job title, but in reality, it’s been much longer. I’ve worked as part of small consultancies (such as SQLHA) and have been employed by some big ones you know. Consultants can get a bad reputation – sometimes for cause, other times for no rational reason. Here’s the painful truth: one bad egg often spoils it for the rest.
Existing or potential clients wants the best value and service for their consulting dollar/yen/pound/Euro. Working within budgets and keeping costs manageable is not optional. Consultants need to understand a customer’s price sensitivity/budgetary constraints. Do we want fair compensation? Yes. We’re not a charity and real expertise does not grow on trees. When consulting works, here is what happens: the consultant(s) do great work, deliver on time, fairly compensated, and customers are happy. Everybody wins.
Unfortunately, some “consultants” are better at business development and marketing than delivery. They are buzzword compliant, great schmoozers, and say all the right things. The customer gets all the right warm and fuzzy feelings to sign the contract, but as shown in the Hertz story and in some of those comments, things can and often do go awry when it’s time to work together, solve the hard problems, and deliver the solution.
Consultants Don’t Suck
More than once in my career, I have been hired to mop up the mess of another consultant and deliver what the customer should have gotten in the first place. They are skeptical of hiring anyone else because of how they got burned. Why? Clients who hire bad consultants fear that they will get very little return for the amount of money paid and no deliverables. From a people aspect, sometimes the admins and/or developers feel it is an us vs. them thing going on – the consultant is going to take their job, or worse, tell management they should be fired and are incompetent. That is a true story – I have heard that more than once where a consultant came in and said to management the staff was terrible when they were not.
To anyone who thinks that consultants are evil, understand this: there are good consultants!
A consultant’s ultimate goal is to make you successful. What success looks like will vary from project to project (examples: an assessment document, an architecure, an entire built solution, solving a perfomance or availability issue, upgrade to a new versions of <insert thing here>). The bottom line: when we ride off into the sunset, your company is better off, the administrators or developers are empowered, and goals were met or exceeded.
How to Effectively Use Consultants
When should you consider working with consultants? In my experience, here are five of the most common scenarios in our end of the world:
- Your staff has a lot on its plate. A new initiative needs to be spearheaded by folks wo have both the time and expertise to get it done by your deadline. Time is money and so is your company’s sanity—we’ll save both.
- To get a project done, there’s a skillset you do not have on staff. You need people who can augement and mentor your admins or developers not only deliver on time, but provide the knowledge transfer so that your everyone can manage things once the consultant leaves. REMEMBER, a need for knowledge transfer and training has nothing to do with how smart people are; new things take time to learn and master. Again, time is money. Sometimes it’s cheaper to hire than to have people bang around in the dark.
- You want to lean on people you trust when you have questions because they’ve been there, done that. You’ve worked with them so they know your systems, applications, and environment – , and the pitfalls thereof. Whether it is troubleshooting or just a sanity check, knowing you have people who are there for you provides peace of mind.
- In the overused buzzword bingo category, you’re looking to transform or modernize. For example, you have been mainly on premises but want to explore the public cloud or move towards hyperconverged. If your focus is SQL Server, you want people who understand SQL Server and those things (or different consultants that will work well together).
- You have upgrade/migration/patching pain and need the thorn pulled out. Sometimes it takes someone coming in from the outside to take a holistic look at what is going on.
Bringing in the right consultants is not only the difference between failure and success, but can be the spark to accelerate said success. This includes faster problem solving and enabling your team to manage as well as prevent future challenges. They understand the big picture (not just technical, but also the business and your company’s politics), yet can roll up their sleeves and get into the technical details. Good consultants have communication skills (technical and non-technical, written and verbal), mentor those they work with, bridge teams together, and may even be a good hang!
Here at SQLHA, we pride ourselves on being those consultants described in the last paragraph. Before we take on any engagement, we ensure it makes sense for both parties. We do not take on work we cannot deliver; that just leads to that dissatisfaction that Hertz experienced. We prefer to be those first call consultants who get it right and leave the customer happy from the start, but will also help remove the stress caused by someone else’s mess.
In my next post, I’ll talk about the role clients play on an engagement to ensure that the project is a success and delivers what they need. I’d love to hear your comments below since this is a hotbed topic. Do you think consultants are evil? What are your horror stories? What are some amazing things consultants have done for you or your company?
UPDATE – Here is a link to Part II.