By: Allan Hirt on June 11, 2019 in Advice | No Comments
I don’t always participate in T-SQL Tuesday, but this month’s topic appealed to me. Hosted by Mohammad Darab (blog | Twitter), his idea is to write a letter to my 20 year old self and give some advice. I hope you find it both useful and entertaining. Enjoy!
You’ve learned a lot in just over a quarter of a century – sometimes not the way you would have liked, but hey, welcome to life. It’ll be a journey, not a destination. Below are some tips that will serve you well.
Working Harder is Not a Life Goal
Besides work, you will average over 50,000 miles a year on planes and play in two big bands as well as a few different small ensembles (among other things you do). Your schedule still tires people out (something you’ll hear often). You’re just as busy – if not busier – today than you were at 20. However, you require more than an hour or two of sleep. Remember to eat and drink. Listen to your body. You’re not invincible. You’re no good to anyone if you’re tired, run down, or sick. You’ll never have perfect work/life balance but taking time for yourself is important. Sometimes a simple recharge – it could be a quick power nap or a week off at the right time – can make all the difference in the world.
Turn off the computer and have a life outside of work. That last bit of whatever will still be there in the AM to finish. Life is about moments – as you get older and time goes by quicker, this becomes more apparent. Spend time with people and in places that matter because they – or you – could be gone tomorrow. At 20, 40 or 50 seems old. It’s not. You will experience loss and pain. Celebrate the good times, don’t wallow in the bad ones. Memories are forever; don’t miss them because you were stuck in front of a screen.
Try to save a few shekels along the way from all that hard work. Don’t pass up that Roboto mask or seeing Rush in Los Angeles in 2015, though. However, all the money in the world will not buy happiness or health. The former comes from within, the latter only if you do what I say above.
You Can Say “No”
“No” is an acceptable answer to many questions and situations. Saying “yes” to everything makes you a doormat, even if well intentioned. You can’t be all things to all people, nor will you please everyone. Also related: exposure bucks don’t pay the bills. Your time has value, even if you choose to give it for free. If you don’t and people don’t like it, that’s their problem. You will always have haters no matter how much good you do or help.
Make major decisions with the bigger picture in mind. Be able to pivot if necessary. You have a great gut instinct. Trust it.
Choose People Wisely
While you are ultimately responsible for yourself and your actions, surround yourself with people (friends, family, colleagues) that support and love you but at the same time can be honest and kick your behind when needed. You need people in your life who won’t put you on a pedestal. Having people in your corner that have your back and can offer trusted advice is crucial.
Unfortunately, some people will disappoint and hurt you both personally and professionally. When your Spidey sense is tingling, listen. Don’t let the haters and those who hurt you harden you or control your narrative. Rise above the noise.
Side people note: every interaction you have with people – good and bad – matters. You’ll see the impact of this more and more in the years to come. Be confidient and humble. You don’t know it all. Definitely don’t worry what people think or say.
Go with the Flow
You may or may not still be known to be outspoken, opinionated, and passionate (stop chuckling, people). As you get older you realize that life has a funny way of not always working out as you thought it might. Loosen up and go with the flow where possible. You have a terrible poker face, though. People will make memes about you (you’ll find out what a meme is). Have a laugh.
Worry about what you can control, don’t fret (too much) over what you can’t. This will serve you well personally and professionally. Always push forward, don’t live with regrets, and don’t let the stress consume you. Playing “what if” or living in the past is not productive.
Perfection Is the Enemy of Good (or Done …)
You are and will always be your own harshest critic. The bar of quality you set for your work is impossibly high. This is a blessing … and a curse.
Remember how you thought it’d be fun and awesome to write books? Yeah, about that. They’re rewarding, but a LOT of work. You won’t make much money, so being an author is not a “retire early and live on the royalties” plan. With your attention to detail and level of perfection expected (among other things that will happen along the way …) the books you write are never simple to birth – think along the lines of having quintuplets. You will inevitably disappoint some people with how long it takes and feel bad about it. Get over yourself. Good and done is better than perfect and not done yet.
I’ll leave you with one last thought: Tokyo is every bit as amazing as you thought it would be, but if you can, get there about ten years earlier than I did. The CD shopping will be even more insane.
Your 47-Year-Old Self
By: Allan Hirt on June 10, 2019 in Availability Groups, Kubernetes, SQL Server 2019 | No Comments
Happy Monday everyone.
I haven’t written much about them yet (key emphasis there …) but AGs now being supported for containers in SQL Server 2019 is a big deal. Recently, SQL Server 2019 CTP 3.0 was released, but there’s a slight problem: if you try to deploy an AG with Kubernetes, you may see the following errors when trying to deploy the pods with the YAML that contains their definition. The services (i.e. instances of SQL Server) get created, but the pods do not.
YAML execution error (click to make bigger)
The fix, however, is easy. For CTP 3.0, a few things changed for the SQL Server operator. You have to modify the operator.yaml file with two changes which are documented below.
The first new thing is the word “update” is added to the verbs list in the section below.
The second is a new section that now must exist after MSSQL_K8S_NAMESPACE.
The updated operator.yaml file can be downloaded from my repository.
Also, remember in the YAML file where you define the pods, update the following two lines:
Hope this helps some of you.
By: Allan Hirt on May 16, 2019 in Patching, Red Hat Enterprise Linux, Security, SQL Server, SUSE Linux Enterprise Server, Ubuntu, Windows Server | No Comments
This past week has been a doozy for security updates. There have been quite a few security alerts and patches. In this post, I’ll guide you through them as they relate to SQL Server and the operating systems that it runs on.
More Intel Processor Woes – Side-Channel Vulnerability/Microarchitectural Data Sampling (MDS)
After Spectre and Meltdown a few months back (which I cover in this blog post from January 4), another round of processor issues has hit the chipmaker. This one is for MDS (also known as a ZombieLoad) This one comprises the following security issues: CVE-2019-11091, CVE-2018-12126, CVE-2018-12127, and CVE-2018-12130. Whew! Fun fact: CVE stands for “Common Vulnerabilities and Exposures”.
As of now, this is only known to be an Intel, not AMD, issue. That is an important distinction here. The official Intel page on this issue can be found at this link. This issue does not exist in select 8th and 9th generation Intel Core processors as well as the 2nd generation Xeon Scalable processor family. (read: the latest stuff) This link in the Intel page is the detailed processor family list and what is/is not affected. If there are hardware-level fixes, those would come from your vendors such as Cisco, Dell, and HP. Many OSes as well as SQL Server have issued statements, guidance, and/or patches for this new Intel processor flaw.
Below is everything that relates to SQL Server or your underlying server configurations. Oh, and patch your systems RIGHT NOW.
This is the biggie. If you look at most of the mitigations beyond patching, nearly everyone is, depending on the circumstance, recommending disabling hyperthreading (HT) on the physical processors. Guidance will vary from vendor to vendor. Doing this will require downtime as you will need to boot into BIOS/UEFI to do this. It will most likely impact performance if you were relying on HT, so check your SQL Server workloads. Before disabling HT, check the guidance from your vendors.
The Spectre/Meltdown KB article (4073225) was updated to include this new flaw. SQL Server does not require any specific patches for MDS. Please see the Windows Server section and specifically ADV190013 (linked below) for Microsoft’s full view on this.
Red Hat Enterprise Linux (RHEL)
SQL Server is currently supported on RHEL 7.3, 7.4, and 7.6. Red Hat did an excellent page explaining things which even has a video on how the attack could work. Even if you don’t use RHEL, the video is a good watch. Their specific vulnerability page for this new flaw can be found here. Read it, but if you want to skip to the conclusion, switch to the Resolve tab.
SUSE Linux Enterprise Server (SLES)
SQL Server is supported on SLES 12 SP2.
Here are their specific pages for each of the vulnerabilities and how to deal with them:
SQL Server is supported on Ubuntu 16.04 which has a patch for MDS (the plain English stuff). The specificl Ubuntu Knowledge Base article for MDS can be found here which covers any updates that are needed.
VMware published a security advisory (VMSA-2019-0008) for the MDS issue. They cover all the versions affected and most importantly, which versions are the fixed ones. The main VMware Knowledge Base article linked you should be concerned with is KB67577. That article covers HT.
Microsoft updated their KB article on side-channel vulnerabilities (4072698) to include MDS. It has two very specific links: ADV190013 and KB4457591. Microsoft also has information for Azure IaaS VMs. I would read both carefully as they talk about the potential impact to performance. KB4457591 has a great section on whether or not you would need to disable HT, especially as it relates to Hyper-V.
From a fix perspective, the good news is that Microsoft shipped patches as part of the latest OS update released for Patch Tuesday covering Windows Server 2008, 2008 R2, 2012, 2016, and 2019. The link to the respective software is found at the bottom of ADV190013.
Amazon Web Services
AWS also had to deal with this flaw. Their information can be found in Security Bulletin AWS-2019-004. If you are using their IaaS services (EC2), you will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.
Google Cloud Platform
Like AWS and Azure, GCP has this problem, too. Similar to AWS (and Azure), the hosts were taken care of according to their posted notice. You will have to check your IaaS VMs running in Google Compute Engine to make sure they are patched and that they are not running untrusted workloads.
Remode Code Execution Vulnerability in Remote Desktop Services (Terminal Services)
On May 14, Microsoft published security bulletin CVE-2019-0708 which details how a worm could happen with older versions of Windows desktop and server with Remote Desktop Services. The security bulletin describes how it would work, and what is – and is not – vulnerable. The good news is that if you are running Windows 8, 8.1, or 10 for the desktop or Windows Server 2012, 2012 R2, 2016, or 2019, you are safe. However, If your SQL Server instances are running on Windows 7 or Windows Server 2003, 2008, and 2008 R2 you are not. This one is bad enough that Microsoft patched Windows 7 and Windows Server 2003 which are out of support. While we do not see much Windows Server 2003 at our customers, there is a fair amount of Windows Server 2008 and 2008 R2 out there. The last version that was supported by Windows Server 2008 R2 was SQL Server 2014.
The security patches for Windows Server 2008 and 2008 R2 can be downloaded from this link. Note that this also affects Itanium processors if you still have those.
The security patch for WIndows Server 2003 can be downloaded from this link.
SQL Server 2017 – SQL Server Analysis Services
SQL Server released a rare security patch recently for SQL Server Analysis Services. According to KB4497700, There is a potential leak of restricted data that is not protected correctly by the Object-Level Security (OLS) system. This does not affect any other version of Analysis Services other than 2017, and is fixed in a GDR for RTM if you have not applied any CUs and a security update for CU14 (CU14 + GDR). CU14 is a re-release from March 25.
There have been fixes to other things such as Adobe Flash, Internet Explorer, and more. I’ve coverd the major things, but anything that ships as part of the underlying OS is important to patch if it’s installed.
Call To Action
Security flaws need to be patched right away. Yes, that means downtime you probably didn’t plan, but do you want to be the next company to land in the headlines because you didn’t patch? We would be happy to help you figure out the right patching strategy and help you migrate to later platforms/versions that are more secure. Contact us today.
We have an upcoming webinar on June 5 Why Upgrades Matter where we will discuss the reasons you should do your best to stay up to date with releases and patching since there can be benefits, too. Read the abstract here and register today.
By: Allan Hirt on April 30, 2019 in Consulting | No Comments
In Part I of this blog series, What You Can Learn from Hertz Part I: Hire the Right Consultants for the Right Reasons, I discussed why good consultants do not and suck and when it makes sense to use them. Hertz hired Accenture for the right reasons – it’s right there in the lawsuit:
Hertz did not have the internal expertise or resources to execute such a massive undertaking; it needed to partner with a world-class technology services firm.
Unfortunately it appears they did not deliver, which devolved into the lawsuit. If you want to read up on another famous epic consulting fail, look at Oracle and the State of Oregon.
An engagement needs to be structured and run to ensure fiascos are avoided. Contracts should make sense for both sides. At the same time, there are environmental, political, and budgetary aspects to why projects can get derailed and ultimately collapse. The customer/client has a lot to do with this, and they can be a big part – or even the sole reason – a project gets tanked. Competent consultants can’t compete with a customer getting in their own way. This blog post will cover what skin in the game the customer must have for successful consulting engagements.
Management must empower the team for success. Good upper management is transparent. Success, especially on large projects. usually happens because of top down support. Skunkworks projects also can succeed, but the risk – and sometimes the reward – is much higher. Both are valid approaches, and we’ve worked on many different kinds of projects over the years. When reaching out to a consulting firm, hopefully you’ll make it clear to them if you have management buy-in or not.
Make Resources Available
The consultant arrives at your office the first day of the engagement. Unfortunately, the key people he or she is supposed to work with are out at an offsite meeting for three days. Clearly that’s an issue. Now the company is going to pay for three days of paid time and the expenses of showing up with no real results. This is not the consultant’s fault. The start date should have been moved/rescheduled, but no one from the customer communicated this scheduling snafu to the consultant (or the company who sent them). Even when asked the week before if everything was fine, the company gave no indication it was not. Disconnects are expensive.
Good consultants are self sufficient, but we need information and feedback, too. This request time and input from key resources in your company to keep things moving. Projects – especially those with tight timelines – need to ensure that people are available for in person meetings or calls and answer e-mail (or IM) in a timely manner otherwise things grind to a hald.
Similarly, if a consultant needs access to systems, that process should happen sooner rather than later. That means once the ink is dry on the proverbial paperwork for the engagement, start whatever process it takes. Sometimes this takes a few days – or a few weeks – to set up if things like RSA tokens are involved. This eats time in an engagement and slows us down when we don’t have access to what we need to get the job done.
This is also a respect and common courtesy thing. We understand you have a day job and are not there to babysit us. Heck, I don’t want a babysitter! I know you have other stuff to do, but your company is also paying me to do a job and meet any scope and timeline that has been agreed upon. Respect a consultant’s time as they do yours.
Here’s a practical example: I’m writing a document as one of the deliverables on the project. In the contract, we agree on a review timeline once the document is submitted. The people who are reviewing the document need to be a) available to review when it is turned over b) able to deliver comments/revisions back to me on the agreed upon review cycle. Anything outside of that is most likely going to change the delivery date.
If you don’t have skin in the game to meet your deadlines, how can you expect us to meet them?
Establish Milestones and Timeline
Every project needs a plan, even if it is high level. A detailed one with dates and dependencies is much better for everyone. Whether the consultant/consultancy drives it or it is someone at the client, there needs to be someone making sure everything is on track. Customers are not fans of paying for project management since it seems like overhead, but good project management is fairly transparent and non-invasive. It also allows one person to be the point for any questions, and that person can get them to the right personnel (hired or in-house).
For Hertz, according to the lawsuit, they used Accenture for project management. Section 42 of the lawsuit calls out the failure of said project management. The example given is the removal of key resources from the project. Major changes like that should be communicated and the impact measured. We do not know why Accenture did that. This to me was an important phrase in the lawsuit:
Because Accenture failed to properly manage and perform the services, the go-live date was postponed twice, first until January 2018, and then until April 2018. By that point, Hertz no longer had any confidence that Accenture was capable ofcompleting the project, and Hertz terminated Accenture.
Forget the termination part – they missed the launch more than once. For a big public company like Hertz, this can be a public relations nightmare especially if you announce these things.
In the plan, key dates/events – milestones – must be set and agreed upon by both parties. If they need to be adjusted, again, everyone needs to agree to those changes. For fixed price engagements, milestones are absolutely necessary because for consultants, payments are usually tied to them. Even if payments are not tied to key dates, these should be big red flags. The nagging questions I have here are: why were tripwires were ignored and how did this not get nipped in the bud sooner?
I do not know why (or how) Hertz let $32M pass before they terminated things. There were two phases to the project, and the first obviously went well enough for Hertz to pony up for the second. Most customers I know would have cut the cord much sooner.
Ensure there is Testing, Acceptance Criteria, and Oversight
Besides milestones, most work should have a set of minimum documented criteria for success. You have carefully crafted scopes of work for a reason.
I do not know the details, but I am surprised by statements like the ones below in the lawsuit.
Without Hertz’s knowledge or approval, Accenture deliberately disregarded the extensibility requirement and wrote the code so that it was specific to the Hertz brand in North America and could not be used for the Hertz global brand or for the Dollar and Thrifty brands.
Going rogue on a client is never acceptable. Sometimes there are honest-to-goodness technical challenges or limtations that crop up and may make something impossible to do. I’ve been there. Those are hard conversations, but you find a way around them. Deciding to go another way just because? Nope, nope, nope.
In addition, Accenture failed to perform proper testing of the software that it developed. Accenture did not perform tests on many components of the system. When Accenture did perform tests, they were seriously inadequate, to the point of being misleading.
I have a QA background so testing is in my blood. People should test their work. However, where was Hertz in this? Did they leave all the testing to Accenture and had no resources touching the work along way? Did Accenture just do smoke and mirrors when showing “working” versions – if they did that at all? Maybe I missed that in the lawsuit, but this goes back to skin in the game. I’m a consultant, but if you’re paying me to do something, you should check in if we’re not doing so already. Have weekly status meetings. To let it get that far down the garden path is something that should have never happened. Clearly it does (see: lawsuit), but in theory, this could have been stopped sooner. Maybe it couldn’t. I was not there so I can’t say everything that went down on either side.
Avoid Scope Creep
One of the worst things that can happen on a project is scope creep. Scope creep is at some point during the project, the requirements change for whatever reason. Sometimes that is for valid reasons (for example, deploying to Azure and something changes on the backend and code needs to be refactored). Other times people just want to add new functionality in. It is more the latter than the former that is the problem, but the end result is the same: projects slip because of scope creep. Moving the ball but not the end date is not generally a good thing. All changes to scope need to be formally agreed to by both sides. The reason you put these things in contracts is that everybody knows what success looks like at the end. You deal with the unforseen, but avoid the ones that will change end dates just because someone now wants a widget to be purple.
Stick to the Budget
Changing the budget while a project is in progress is always a tricky proposition. More money for the right reasons is good, taking away money is bad. There are two aspects of this: affecting the overall funding of the project including the consulting dollars, and then just affecting other aspects of the project but the consulting bucket is untouched. The former is concerning for obvious reasons. If we agreed to X in the contract, that is the expectation. There’s generally no re-negotiating especially if you want us to do the same amount of work. However, that is not what I am talking about; it’s the latter scenario.
Let me give you an example. Years ago, I was working on a data center migration project under a big consultancy firm. I was leading the SQL Server team. We were supposed to get our own dedicated storage unit for the SQL Server implementations, and all our plans were based on this promise. A few months into the project, after I had delivered our storage requirements and verified that what the SQL team was getting would be suitable, I was told that we were not getting said storage. We would be sharing one unit with the file server and Exchange folks. That was not good news. Doing the “simple” math, the client’s SQL capacity (forget performance for a second) was in theory cut by 2/3 in the shared storage plan.
How did I handle it? We had already rolled up our sleeves to fully understand the environment and we had enough storage for launch. Due to changes in the application, we could not measure growth until we flipped the switch. Once we did, we predicted when storage capacity would run out. I then presented the findings and let them know their options. At that point, I was set to roll off the project. I was never told how they resolved the issue.
In that example, it was purely a financial decision made much higher up the food chain that impacted everyone. I know the Exchange and file folks were not thrilled, either. I can only speak to how things played out from my point of view.
In the initial stages of trying to work with a consultancy and putting a contract in place, they may ask about your budget. Be honest and transparent because we are with you. We understand you may be pitting us against someone else to see who will be a better fit for a variety of reasons including financial. That said, reasonable consultants will put together a good scope of work that will get you what you need and come in on or under budget.
As mentioned in Part I, expertise does not grow on trees. It’s better to hire the right consultants first than land up with a costlier project because you hired a square peg for a round hole. Sometimes spending a touch more than you thought will save you much more money down the road. Budget matters, no question. Good consultants get this loud and clear – I know I always have.
Remember to work out travel things, too. If you want someone onsite, understand what that means. If you request on a Friday to have someone there on Monday, the costs involved are often much higher than planning a trip in advance. It’s up to everyone involved to control costs where possible since the consultant (or their company) is laying out the money ahead of time, but be smart about things. Having a consultant fly across the world in the back row of a plane and be dead tired to work is counterproductive.
We always hear the big consulting horror stories because they land up in the news in one way or another. The fact that way more consulting engagements are successful and do not land up splashed in the headlines is a good thing. It’s the bad ones that make many think consultants are evil, but the companies hiring us need to ensure they’re doing their part to ensure successful projects. When it all works, everyone is happy. Don’t let news articles like this scare you. Hire us! 🙂
If Accenture’s side of the lawsuit pops up, I’ll most likely do some sort of Part III since I’d be curious to see their side of the story.
By: Allan Hirt on April 29, 2019 in SQLHAU, Training | No Comments
In honor of Star Wars Day (May 4th), all of SQLHAU’s training will be 40% off. That’s 40% off all classes – in person and online including the upcoming classes in Chicago this August. For a full list of classes, see the SQLHA Events page. Use discount code MAYTHE4TH.
Because the 4th falls on a Saturday, the flash sale will start Friday, May 3 and end at 11:59 PM on Sunday, May 5.
May the force of knowledge be with you!