advice Archives

January 12, 2023

Logs Do Not Care About Your Feelings or Emotions

I spend a lot of time looking at logs. Over the years, I analyze more Windows Server Failover Cluster (WSFC), Windows Event, and SQL Server logs than most.

That said, I want to address something I run into from time to time. For whatever reason, people seem to think what logs say is a personal attack on them, their skills, and/or something adjacent. Let me explain.

Log Fundamentals

I always say this: “If you are reading cluster logs, you are in a bad place.” One could argue reading any log, especially on a frequent basis, means you are not where you want to be. Logs are not light summer beach reading.

Proactive monitoring often uses logs to pop up events or information. Monitoring must be tailored to your day-to-day IT requirements. Trawling through multiple logs and correlating events to determine why your AG is down is a vastly different exercise than just seeing if your CPU spikes.

A log’s job is to document what is going on for a particular widget and often includes associated things. For example, a WSFC’s log contains information about networking as it relates to the WSFC. The WSFC log does not measure network performance from the various WMI counters but it may inspire you to go look there if an issue is seen.

Log Findings

Whenever I troubleshoot issues, often part of that exercise is to report findings. Sometimes that is a call, other times a summarized e-mail. For others, it is a detailed report that not only covers the “why”, but also how to mitigate or improve the situation. In other words, it depends.

As with most things, there are good and bad ways to convey a message. Unfortunately, the baby can be very ugly even in the best of scenarios. In those situations, there is no (easy) way to sugar coat that message. The core problem is that once human beings are involved, feelings and emotions come along for the ride. Deliver findings in a humane way – leading with “you suck at your job” is not going to help matters. If your findings are meant to hurt others, rethink your position because karma comes back around.

Information presented in a log has no opinion nor is said information alternate facts. Hopefully, the log output does not contain the words “you are a moron” or “you suck” (if so, find better vendors). Some AI aside that is probably in the works somewhere (hello robot overlords!), when a log documents that the server lost connectivity to the network, that’s all it is saying.

A problem documented in a log is not a referendum on your job as an administrator or your skills. It is most likely an indicator that something must be addressed. The widget and its log do not know why the network dropped out from underneath at a given moment. Knowing the network drop happened and its timeframe points you in a direction to resolve the issue. That may mean correlating other logs and their events. After that you may investigate if this was transient/one time or something that has happened multiple times. Let the information lead you on a treasure hunt.

If any of the above resonates with you, here’s how you can address this unique challenge.

What Can You Do?

First, don’t be defensive. Many companies have walls and divisions that exist between groups or organizations. This phenomenon often leads to finger pointing in times of crisis. Chances are if you are in a server down situation, multiple people/groups will need to work together to resolve the issue. Put egos and agendas aside. Never make others jump through hoops to prove “it’s not my problem”. Everyone is on the same team – Team <insert name of company here>.

Second, remember troubleshooting is a logical exercise. You start with logs or a set of assumptions and you either prove them true or false based on the data you have. Sometimes people know right away what an issue is. This is because they have either seen it before or they have years of experience that quickly rules out noise. Do not dismiss instinct; it is a very real thing.

Example One

DBAs are often the default blame acceptors since the problem will show up in SQL Server first even if it started somewhere else in the chain. If an Always On Availability Group (AG) goes down because the underlying network drops out, it’s not SQL Server nor the WSFC or Pacemaker’s fault. The network team will most likely need to investigate what is happening and resolve the situation.

Example Two

DBAs often must go out of their way to prove performance problems (physical or virtual) are not things like a query issue. Of course, performance problems manifest themselves in SQL Server, but what is the actual cause? For example, a noisy neighbor on a virtualization host can impact another VM. The problem shows up in SQL Server, but it is something the virtualization administrator will probably need to address in some way – not the DBA. Work together, not with two middle fingers held high by each camp.

Summary

When it’s time to look at what the logs tell us, have an open mind. Sometimes it’s not our problem and we can breathe a sigh of relief. Other times the spotlight is directly shining on you. Both are ok. Deal with it and move on. Making it personal is not a winning scenario.

Have you encountered situations like this? Chime in below.

February 16, 2021

Technical Debt – The (Not So) Silent Crisis

Here in the USA, many are experiencing unprecedented winter weather. For example, parts of Texas are without power and heat and experiencing blackouts. A big focus of what I do for customers – business continuity – has to account for things like the power going down. A few years ago California had rolling blackouts in the summertime. Austin tried that in this 2021 storm – and stop me if you’ve heard this before – the plan did not work since what happened isn’t what they expected. From this Austin-American Statesman article:

Austin Energy’s plan was to rotate the outages, meaning more neighborhoods would’ve shared the no-electricity burden for the entire city, for a period not to exceed 40 minutes. But the rotation was not possible, Sargent said, because it would have disrupted service for those critical operations.

There is a reason I strongly recommend testing all continuity plans. For IT folks, you do not just want your servers literally powering off. If that happens, pray you experience no data corruption. How are your backups? Test them recently?

Having said that, the Houston Chronicle wrote a damning article. Here’s the part I want to focus on:

“The ERCOT grid has collapsed in exactly the same manner as the old Soviet Union,” said Hirs. “It limped along on underinvestment and neglect until it finally broke under predictable circumstances.

“For more than a decade, generators have not been able to charge what it costs them to produce electricity,” said Hirs. “If you don’t make a return on your money, how can you keep it up? It’s like not taking care of your car. If you don’t change the oil and tires, you can’t expect your car to be ready to evacuate, let alone get you to work.”

Some of what is happening in Texas is not only due to Mother Nature but also partially because of technical debt. Old, aging infrastructure eventually buckles and sometimes fails. Technical debt is not just an IT problem. Real life examples such as this bring the concept front and center.

Kicking the proverbial can down the road and saying things like “We’ll worry about that later, it’s fine now” is often what makes the end results so painful. Neglect increases risk whether it is intentional or not. Max and I help customers so these big leaps incur less risk and pain including managing external dependencies.

Look at your company if you are a FTE or your customers if a consultant. How many applications and servers have not been even looked at because they “just work” and “it/they is/are fine”? Budget is always a concern, but to paraphrase the old adage, you will pay now or pay later. Paying later is often more expensive. Insert your own scenario or question here but answer this:

What is the actual cost if that system/application/server fails or is unavailable? I bet it is more than if things had been dealt with all along or right from the start. You can sometimes avoid a quadruple bypass.

For those of you affected by this weather, my heart goes out to you. Stay safe and warm. Even though we are in the midst of a pandemic, please check in with neighbors and loved ones.

We must be prepared both in life and in business to handle both the expected and unexpected which also means managing your technical debt. This is what we help our customers do every day at SQLHA, so if you want to ensure your business can not only be resilient but also in a good place to manage handle technical debt, contact us today.

February 1, 2021

Making Lemonade Out of Lemons

The last time I did any appreciable travel pre-COVID-19 was late January 2020. I went down to NYC for a customer meeting. My next trip was supposed to be SQLBits in the UK in March which never happened for obvious reasons. For the better part of the past 20 years, I’ve flown 50 – 100,000 miles per year. Some people I know fly more than that. There are two ways I could have handled the sudden and enforced change in my life: look at it as a problem and be miserable or lean into it and make something of it.

At first, I must be honest and say I was bummed and missed being out there. What I came to realize is that this past year has become a blessing in disguise. Let me explain.

Nearly 20 years on the road does things to your life – both good and bad. There were stretches I was home and sometimes travel came in spurts, but I didn’t earn lifetime status with American Airlines and Marriott because I am a homebody.

What I realized quickly in my forced “staycation” is that I was weary. All those years on the road which came to a sudden halt was like getting off a treadmill still running at full speed. I needed a reset both mentally and physically. Just as COVID was kicking in, I was dealing with a hand issue that was not only hampering my ability to type (affecting work) but also made it hard to play bass. I also realized I had a backlog of projects and other things personally and professionally that were long neglected for one reason or another.

For example, I’ve never worked in an office when at home. I not only set one up, but I figured out a lot of stuff which will influence some exciting things I will announce in the upcoming months. I just never had dedicated time to deal with setting one up when I was home a few days here, maybe a week or two there. Even when I was home, I was working all the time or trying to squeeze one more rehearsal in because I may not be able to play for weeks on end. I didn’t allow myself to slow down. I had to cram something into every moment. That’s normal until it isn’t.

A big silver lining is that 2020 was a busy year for SQLHA. I basically didn’t have downtime all year and just this past week was the first one I took a breath. I am VERY thankful for the work as I know some were not. We have great customers (why aren’t you one? contact us today ;)). I never take it for granted.

Even with being busy, I made myself a priority. I not only feel the best physically I have in years but also rejuvenated. Part of that rejuvenation is being inspired. For example, before these two recent blog posts, I realized I hadn’t written one in nearly six months. I just wasn’t feeling it.

Did 2020 suck in many ways? You bet. It was not all sunshine and roses. I’ll just say this: I certainly have more salt than pepper in my hair after 2020.

Realistically I probably won’t be hitting the road in any meaningful way until later in 2021 or early 2022 since vaccines are just rolling out. Most countries are closed for visitors. Even travel here in the US is iffy to me at the moment. I’m going to continue making lemonade out of lemons by checking items off the aforementioned list of backlogged projects. I know I can’t tackle them all at once, but it feels good to be making forward progress. I am also making sure I give myself some time, too. Self-care matters. You’re no good to anyone if you’re exhausted.

When the time comes, I’ll be grateful to be out and about again and am looking forward to seeing all of you outside of a screen. I know one thing for sure: I will appreciate it that much more. Until then, stay safe and healthy.

January 28, 2021

Closing in on 30 Years in the Database World

It was the fall of 1991. I applied for and landed a quality assurance internship at SQL Solutions in Burlington, MA which was located at 8 New England Executive Park right next to the Burlington Mall. SQL Solutions was purchased by Sybase not long after. I kept that internship through the end of graduation in 1994 and there’s no question it literally set up the rest of my professional career.

I always find it interesting how one thing, one event, one interaction can sometimes change the course of your life – good or bad. I’ve been on both sides of that.

I always went home in the summertime, but when back in MA for college, I was at Sybase twice a week (Tuesday and Thursday if memory serves me correctly). Two of the people I worked closely with and were my direct supervisors were women I still have the utmost respect for today. I’ve never seen gender, color of someone’s skin, religion, etc., as a problem then or now. No one should. I learned a lot in those two years, and I am forever thankful to everyone I worked with at Sybase. In fact, one of the people I met I wound up playing years later in various jazz ensembles (he’s a guitar player). Small world.

Part of me thought I’d land a permanent job at Sybase, but that wasn’t meant to be, and I believe things worked out how they should have. The skills I acquired in those two years gave me the foundation to land my first job post-college doing QA using FoxPro (then Visual FoxPro) for a company that did medical software. In reality, it set me up for life since I’m still tinkering with databases today.

In a harbinger of things to come, I once got in trouble in that job because I automated part of what I was supposed to do using Visual Basic and Excel macros. The results were predictable, so why would I eyeball that? Let a process tell me if it was good or bad. Was I DevOps before DevOps was cool? (kidding here, folks) That little “stunt” landed me on a PIP which I survived but it taught me a valuable lesson.

I can see why automating rubbed some the wrong way – here’s this young upstart at the tender age of maybe 23 trying to tell us how to do our job. I was just trying to free my time up to be more efficient and save my time for the stuff that needed my attention. It’s funny how automation and other things are commonplace now but were not then.

The events of the past year when we’ve all been stuck at home has given us time to reflect and see how we want or need to move forward and improve. I have always looked forward, but that doesn’t mean you can’t look to the past for inspiration. I look fondly on the early years of my career. In some ways I’ve come so far, but in others, I haven’t gone very far at all. I think I found my calling early if I wasn’t going to be a full-time musician but I marched to the beat of my own drummer and I’d like to think I still do to a degree. I’m certainly older. Wiser and smarter? I hope so. Better filter? Those who know me will have varying opinions here.

I’d like to think my success to date has outweighed my failures and shortcomings. I’m a work in progress. We all are. This is why even 30 years in, I try my best to learn and do new things knowing in some cases it will be difficult or I may be ahead of the curve.

A recent interview with Tomo Fujita by Mary Spender struck a huge chord with me. Paraphrasing what Tomo said at one point is this: teachers are not better than students and can learn from them even though they are still helping the student. This one quote stuck with me: “(A) teacher is just a great student.” Fast forward to about 8:37 if you want to see that interaction. It’s such a good interview where even if you are not a musician, you can apply a lot of what Tomo says to your life beyond that little bit.

I’m no longer that early 20-something but I learn something from all of my customers and students (including anyone who attends a webinar, etc.). I may not be the smartest person in the room and I’m totally fine with that but I’m at the proverbial table for a reason and it’s not just my boyish good looks and charm (laugh – it’s a joke). I’ll never try to be all things to all people; it’s impossible. I know my limitations. I can only attempt be the best me I can be at any given moment.

That 20-something also couldn’t have imagined one internship would one day allow him to travel the world and speak to, work with, and teach thousands of people. There are very few things as humbling when people tell you how you’ve inspired them or helped them and it made an impact.

Learn from your mistakes. Learn from every experience – good or bad. Learn from those around you. You never know what may shape and spark things to come … and remember to give back.

January 15, 2020

We’re Only Immortal for a Limited Time

Back in 2015, I wrote the blog post In the End – Life (and IT) Lessons from Rush after I had seen what wound up being their last live show at the Forum in Los Angeles. It’s still one of my more popular blog posts, and in my opinion, one of the better ones I’ve authored.

I’ve referenced Rush before in blogs (such as Fun With Naming Conventions). If you’ve ever seen me present, I always put music references (among other things) in my demos. I often have a three-node AG with node names of Geddy, Alex, and Neil. You get the picture. I was first turned on to Rush via MTV in 1981 or 82 with the videos from Moving Pictures, so they’ve been a part of my life pretty much from the moment I started playing bass. Exit Stage Left was the first Rush album I learned to play back to front. Rush in one way or another has been part of my life now for nearly 40 years. Very few bands have stuck with me like Rush has (there are a few, and most people know Styx is one; that’s a story for another time).

Rush was a phoenix. After Neil’s tragedies post-Test for Echo, there was a high probability Rush would not play again. Between losing his daughter Selina in a car accident and his wife Jackie to cancer just about a year later, would you blame Neil for walking away forever? When Vapor Trails came out, it spawned a triumphant second act for Rush that in some was was probably more satisfying. Rush earned their retirement and then some.

On January 7, Neil Peart passed away after battling Glioblastoma, brain cancer, for about three and a half years. The news was announced last Friday the 10th and needless to say, it hit me like a ton of bricks. The last time I was this affected was from the death of my friend Mike who I’ve blogged about fairly extensively here (two examples Life Is Fragile and Some Anniversaries Are Just Not Happy Days). There are two similarities between the two: their passing were unexpected and sudden and both were WAY too young. Sure, Neil was 67, but these days, that’s really not old. It’s funny how when you are in your teens or twenties 30- or 40-something seems ancient. When I first saw Rush in December of 1987 at the venerable Spectrum (RIP) in Philadelphia, Neil was just barely 35. 35! I’m almost 50 now. 35 is a “baby”.

After Mike’s death and now Neil’s, it really brings the point home you have to make the most of your time on this planet. In the past few years I’ve been doing a bit of a self-reassessment and addressing things if needed. I’ve written before about how self care is important (one example: A Letter to Myself at 20). It shouldn’t take someone passing to put things in focus; sadly it sometimes does. Most of us get caught up and then time just flies by. When we stop to take a breath, sometimes we realize we got off track. If we screw up (and heaven knows I have), all you can do is own it and try to fix it. Getting back on course takes time. The results might even be better.

In a way, it’s crazy to feel so sad and mourn for someone you never met. Yet Neil (along with Geddy and Alex) have touched my life in so many ways for so long. I’m not the only one given all the tributes which have been from all walks of life. Very few knew Neil was sick. Some have admitted they did after the fact (such as his longtime drum tech Lorne Wheaton). You’ve got a good support system when something that major does not leak in three and a half years. I can’t imagine how Geddy or Alex felt every time they were asked about a possible Rush reunion knowing what they knew. Looking back at their statements – especially Geddy – he says it without saying it. Neil stopped publishing on his website about the time it seems he got the diagnosis. The bread crumbs were there; we just didn’t know.

Neil dealt with his cancer and his ultimate passing in the way he lived life. He was an intensely private person but at the same time, a fighter. The prognosis for Glioblastoma is grim and he outlasted the odds. Arguably his main reason for walking away in 2015 was to be there for his daughter Olivia. He got a second chance at life with Carrie which makes this all the more cruel. The last song on Rush’s last album Clockwork Angels, “The Garden”, always reminded me in a way of Genesis’ “Fading Lights” from We Can’t Dance – a goodbye. It’s a very poignant song besides being one of the more beautiful ones they’ve written. You can bingoogle the full lyrics (Neil was the lyricist for Rush besides being the drummer if you didn’t know), but this one always sticks with me:

“The measure of a life is a measure of love and respect”.

I think Neil would downplay it, but he had that love and respect in spades. Just look at the outpouring of tributes that started on Friday. Rest in peace, Professor. I hope Jackie and Selina welcomed you with open arms. My condolences to Carrie and Olivia, Geddy and Alex, and Neil’s friends and family. I hope all the tributes to him and his enduring work will somehow comfort you and know that he will never be forgotten.

Neil’s life is a lesson which can be tied to this month’s T-SQL Tuesday topic, “Imposter Syndrome“. Neil was a lifelong student. Even by the mid-80s, many considered him to be one of the greatest rock drummers of all time. What did he do? He took drum lessons as time went on from Freddie Gruber and Peter Erskine. Your drummer’s favorite drummer took private classes! At the same time, he was a teacher. His instructional videos are some of the best selling ones for drums of all time. The two are not mutually exclusive. Don’t believe me? Read this page of tributes at Hudson Music, the publisher of Neil’s videos, especially from Mr. Erskine.

What I hate is when someone wants to play “stump the chump”. Simply put, it is when someone asks you a question to intentionally show they are “smarter” than you. Look, I remember a lot of things. I even joke I’ve forgotten more about clusters than some have even learned, but I’m also not a walking encyclopedia or Wiki, either. There’s a reason Books Online and documentation exists. If you got me to admit “I don’t know” – words everyone needs in their vocabulary – congrats, jerk. I’m human just like you. I put my pants on like everyone else. I remember sitting at PASS Summit a few years ago and behind me surprised I was right in front of them. Um, what? I’m just a normal person and not sitting on a throne. I turned around, introduced myself and we had a great chat. A bit like Neil, I would rather talk about pretty much anything else than SQL Server when I’m not formally having to talk about it. There’s more to life than WSFCs, Pacemaker, AGs, and FCIs.

Neil always wanted to improve. I think the same way. I never think I know all the answers and the older I get, sometimes I feel the less I know. Believe it or not, I’m even wrong sometimes! Own it. I do. I’m constantly learning new things and am in awe of those who do other things such as esoteric performance tuning just as I’ve had people say they don’t know how they can do what I do when it comes to HA and are amazed at my skills. It’s humbling when people say you’ve influenced them or come up and say one thing you said made a difference. Like Neil, I try to pass what I know on, including from mistakes and failure. Successful people fail all the time. Failure or not knowing something is not a sign of weakness; far from it.

This doesn’t mean successful people will not have a bit of an ego or strong opinions; far from it. They just are open to feedback, criticism, and improvement. Sometimes you need to reinvent yourself even if no one knows it like Neil did with his drumming in the 90s. I’ve done it myself in my career even if no one noticed. If I was the same person I was 20+ years ago, never absorbed feedback – good and bad – I’d be out of a job. Does that mean I’m perfect? No! To quote the title of one of Neil’s videos, I’m a “work in progress”. If that makes me an imposter, I’m proud to be the founding member of the club.

The moral of the story: be like Neil. Be humble. Be a student and learn from those around you. Ask questions. Listen more than you speak. If you’re not doing any of these things, you are the imposter.

12 3 4 Next ›Last »