August 2012 - SQLHA

August 30, 2012

Is an Outage Just an Outage?

I hear it all the time from customers that if they had unlimited budget like “the big guys”, they’d never have any issues or outages. Bzzzt! Thanks for playing – there will be some consolation prizes near the door for you.

First and foremost, even “the big guys” have budgets and other constraints. But that’s only part of the problem. Let me give you a great example. If you haven’t seen, United Airlines has had some, er, minor computer issues the past two days that has caused both passengers and the airline a lot of agita. We’re not talking a mom-and-pop corporation here; this is a major airline with hubs that deals with tens of thousands of passengers each day. The Flyertalk thread on this incident is a fun read. I just saw this:

Just a FYI for the anti-SHARES hysteria crowd. I just received word via an internal communication that the cause of the outage wasn’t SHARES, nor did it have anything to do with SHARES.

The cause was communication equipment in the data center that failed. This was a hardware problem, not software.

Now having said that, I am curious to know why there were no redundant systems in the data center or a quick (ie instant) method of re-routing network traffic via a different data center.

United has also said that it was a hardware problem. I loved this:

The problem was a piece of hardware in a data center that failed to communicate
properly with other computer equipment, said Megan McCarthy, a spokeswoman for
United Continental Holdings Inc. A backup system failed to take over for the
troubled hardware.

*cough* Wonder if they ever tested it? *cough* I’ll say it again for the 1,000,000,000,000,000th time – testing is important. Need I harp on this any more?

Hardware or software, what is in that linked article is telling:

United has been struggling with technology problems since March, when it
switched to a passenger information computer system that was previously used by
Continental. United and Continental merged in 2010. That system, called
“Shares,” has needed extensive reworking since March to make it easier for
workers to use.

That’s more than just a hardware problem that happened to occur. It’s a people, process, and technology issue. When two companies merge, problems like this occur all the time. You pick the software that will be used and the other will need to conform to it. Good example: Company A uses Lotus Notes (and yes, I’ve worked with customers still using Notes). Company B uses Microsoft Exchange. Company A buys Company B. Guess what the new corporate e-mail system is? Company B’s employees not only need to be trained and have Notes rolled out to them (a daunting set of tasks all by their lonesome), but all of their mail needs to be migrated to Notes, etc. That process is a potential disaster waiting to happen if done wrong. Unproductive and disgruntled workers is a side effect of bad decisions and process.

Clearly it seems like when CO and UA merged, the decision was made to use CO’s “stuff”. That’s all well and good. When you affect not only your internal folks but passengers – including delayed, cancelled, and otherwise affected flights where you are doing things like handwriting boarding passes, you’ve got serious, serious issues. I wonder if handwriting boarding passes was in their D/R plans?

I also talk about cost of downtime – clearly there was some here. On top of the normal stuff we all know about, an airline sometimes has to compensate passengers for things like hotel stays, free food in hotels – that’s not free to the airline. Add to that the disruption in travel for folks who may miss connections that have other downstream effects (example: missed vacations such as leaving for a cruise trip), and it’s a nightmare all around.

One of the big guys – United in this case – has more money than the mom-and-pop business down the street. But they struggle just like everyone else. Let this be a cautionary tale of availability and disaster recovery, and that there’s more to it than just technology.

August 29, 2012

Do Your @!(*%$^ Job

This past weekend I was down in the Big Apple (aka New York City for those of you not up on the hip lingo). I had a great time, but a few incidents – and not good ones – stick in my craw. Two were with cab drivers and one was at my hotel.

It was hot, so I hailed a cab going up to FAO Schwarz. The cab driver had no idea where that was.
The room was made up (late) one day and got a written note from the cleaning person apologizing for not providing a new washcloth – they ran out.
Hailed a cab going to Uniqlo on 34th street. The cabbie asks me what cross street.

Before you say I’m overreacting, let me elaborate. When people hire me or Ben for our expertise, they damn well expect we show up with our A game. It has nothing to do with our rates. You hire people to do a job – no more, no less. Expectations are set and hopefully met. For example, if you hire me to help you with your clusters, you expect that I guide you through it and that I can navigate you through any twists, turns, and pitfalls. Reasonable expectation, right?

With that first cab incident with FAO, here was my issue: we’re not talking some obscure NYC landmark. It’s FAO-freakin’-Schwarz. On the short list of known places in NYC, it’s there. A cab driver in NYC who can’t identify landmarks? Come on. The cabbie wound up taking someone else to destinations unknown and the next one I hailed knew exactly where it was.

The second cab incident galled me, though. Get in at the hotel, tell him where I’m going, and he then asks that $1,000,000 question: “What cross street?” So I politely said, “I don’t know.” Here’s where I pretty much was over it. The cab driver then proceeds to say, “Well, I’m not from here.” Look, NYC has lots of people of all shapes, sizes, religions, colors, and nationalities. But you are the one behind the wheel of the cab. I am coming out of a hotel. It’s YOUR job to either know or have the tools to be able to get me there. Period. Not being the shy and retiring type (and maybe I was wrong; but I’m sick of incompetence), I proceeded to say, “Well neither am I. I am staying in a hotel and you’re the cab driver who is supposed to be able to know how to get me there.” He then proceeds to whip out a GPS and enter the address. I was very annoyed.

The hotel stay was quite lovely overall and the washcloth thing really was a speck in the whole weekend, but it definitely falls under “don’t lie to me” and “do your job”. Like I said, the room was made up later in the day. After coming back to the room, there was the aforementioned note (something I’ve never seen in all the years I’ve been staying in hotels). I then bump into the cleaning person in the hall. She apologized again, said she hoped it was ok (it wasn’t), and made pleasant chit chat. I was perfectly nice to her. I went down to the front desk and asked if there was one clean washcloth somewhere in the hotel and if it could be sent up – it’s not like I was requesting 50 of them. When I get back upstairs I find the woman – a bit annoyed – who now says I should have asked her and she would have given me one. What? You wrote a note saying you didn’t have it and then asked me in the hall if it was ok. You set the impression that I couldn’t get one. So I nicely – and not nastily – went to the front desk and asked. I didn’t make a scene, call her dirty names, etc. If you knew you could get me one, why didn’t you do it in the first place?

Looking at the cab incidents, could the addresses have been looked up or cross streets found? Sure. Ironically, for FAO the address was looked up after the first incident. Second cabbie was given it, looked puzzled. Told him to go to FAO and he was like, “OK”. Point being, when I’m in a city that isn’t mine with no rental car, I’m reliant on cabs or public transportation. I hope whoever I get into a cab with either knows where I’m going or can look it up (be it on a map or GPS). That’s the skillset they bring to the table. If they can’t handle that, they shouldn’t be cab drivers. Harsh? Maybe. But I sure as hell wouldn’t be putting myself out there as someone who can help you implement HA solutions if I couldn’t do my job. I show up with the right tools to get it done. Otherwise why the hell are you hiring me? When I step into a cab, I expect the cabbie – whatever nationality, religion, color, or creed they are – to be able to do their job, too.

As for the hotel one, it’s a first world problem. I recognize this. I’m not one of these people who goes through a million towels. In fact, I often put my towel and such on the door but they replace it anyway; I try to do my part for the environment. Anyway, a hotel running out of linens or towels? Pretty much unheard of. I’ve never experienced that in all the years I’ve been on the road, nor do I remember it on trips with my folks as a kid. It’s just one of those things. When you’re paying a decent rate, there’s a minimum level of service that you expect. The same goes for what I do. Ben and I aren’t the cheapest kids on the block, but we’re not the most expensive, either. Our clients expect us to deliver at a high rate, and I hope we exceed it every time. So whether you charge $10 or millions for whatever service you provide to a customer, there’s a level of quality people expect. We know it.

What’s the point of all this? I find more and more that people are lazy or complacent in the workplace. I see it all the time or hear stories from friends and colleagues. I know plenty of hard working folks, and I’ve seen the slackers who those people need to compensate for. Don’t be the slacker. Have pride in yourself, your job, and the services you deliver even if you’re a full time employee. Half assing it to me is unacceptable. Expecting – or assuming – others should do your job for you is the wrong attitude to have. You’re not special or entitled even though you think you are. I work hard every day to put a roof over my head, pay my bills, and food in my mouth. I’m very lucky to have the job I do and I know I’m not going to throw that away. All jobs aren’t the best, but be the best wherever you work or find a way to get out (and yes, I know it’s not that cut and dry … I’m oversimplifying this one point here).

You may disagree with me, but people need to wake up. A college degree isn’t a guarantee of anything anymore, let alone a high paying job in IT or management. Walking with your hand out isn’t attractive. With the one cabbie who asked for the cross street, I almost got out and found another cab to prove my point – why should I honor bad behavior? Only after I gave him my line did he whip out the GPS. Like the cleaning person, it took me doing something to get the right thing done.

Where do you fall on this spectrum? Hard worker? Entitled? Know people who feel the world owes them? Curious to hear your thoughts.

August 28, 2012

Some Anniversaries Are Just Not Happy Days

Some moments in time are singed into your memory banks forever. I woke up the morning of August 29, 2011 and I saw an e-mail with the title “Mike Kenwood” and all it said in the body was “holy crap”. This was from a close college friend of mine who still lives in the Boston area and also knew Mike (but not as well as me, but knew how close I was with him). I was groggy, so the gravity didn’t quite hit me of the “holy crap” part. Anyone who knows me well knows that I am not a morning person – let alone on a Monday – no matter what time zone I’m in, but this day I did happen to be up before 9AM. I responded to him shortly after getting the mail and asked what happened and well, I’ve talked about the rest of the story – mostly here and here. If you want the condensed version and don’t want to read the links, my friend Mike Kenwood passed away a year ago today – August 28, 2011. A short time after that e-mail, I called his wife but also got a call from his wife’s sister shortly after that to tell me. She didn’t know that I knew already and apologized for not calling sooner, but clearly between the hurricane and the suddenness of it all (and as I found out later, how they got to NJ in the middle of a hurricane), they didn’t call me the night before when it happened. It’s not like they weren’t all trying to deal with a lot of stuff or anything …

A lot has happened in a year. I arguably had the worst fall ever (anyone surprised?), ran to Japan to clear my head in December, and then things have largely been on the upswing ever since which I am supremely thankful for. I’ve had a ton of wonderful opportunities present themselves, not the least of which is going to Australia in November to deliver two training classes. But don’t think for one minute that Mike doesn’t creep into my thoughts frequently or sometimes I wish that good or bad times, there are times I still want to reach out and pick up the phone (yes, use the phone as a phone … remember that?) to talk to him or send him an e-mail to share a laugh or get his opinion. Those are the toughest moments, but it gets easier as time goes on. Notice I didn’t say better, just easier.

Mike’s also had a helluva year. He was always a mensch, and he was posthumously recognized for his actions. (As an aside, why is it we don’t often recognize what people do when they’re alive? Just saying …). Here’s a list (and I’m sure I’ve missed some stuff):

He was honored at the National EMS Memorial Service
He received the Jefferson Award
A bench in his name in Princeton which will be dedicated on the anniversary of his death
A tribute concert this September (Sunday the 23rd at Princeton School Performing Arts Center) (which I strongly suggest you go to if you’re in the area; it’s being put on by Broadway Sings – the proceeds go to good causes)

Back in July, I went down to New Jersey for the unveiling of the headstone (here’s the Orthodox take on it if you’re curious what Jews do). It was a small ceremony – mainly family with a few non-family folks. Was it sad? Absolutely. Only a year before we were shoveling dirt on his coffin, and here we were placing small stones on his headstone with the grass neatly cut and cared for. The person officiating was the cantor from Beth and Michael’s synagogue. He did a wonderful job, and the thing is – as sad as it was, it became a celebration of Mike’s life. Moments were shared, some laughs (through the tears, of course) were had by all. It was very fitting.

Mike and I really had a lot of laughs; in keeping with the spirit of celebrating, not mourning, I recently looked at his Facebook page, and this was one of his last posts: “Just caught my 5th squirrel in the rabbit trap. It would be so bad if the rabbit wasn’t sitting next to it laughing.” Still makes me smile, because I can hear him saying that, and is a very typical quip from Mike. I miss that.

We also shared a love for Batman (and comic book stuff in general, but mainly Batman), so naturally he came to mind with the opening of The Dark Knight Rises. I would have loved to hear his take on it. I’m sure, like me, he would have seen it in IMAX (real, not fake). He and I did stuff like that together. I remember seeing one of the Spider Man flicks with him (Spider Man 2) as well as the Lion King (in IMAX at the Navy Pier) while he lived in Chicago. I think he also would have loved the Avengers movie.

Mike was about living and embracing life as I’ve talked about. He may have lived a little vicariously through me since I traveled all over the world (we once joked a little about me being a human Where’s Waldo?), but there was never any jealousy and he had as rich a life as anyone I know. He was happy, and good at so many things. I don’t think he was ever happier or more proud once he became a father. He loved every minute of being a Dad. They had just recently got done remodeling their kitchen (Mike liked to cook). He lived more in just under 40 years than some people do in a lifetime. What more could you ask for? (And yes, it’s still hard using past tense in relation to talking about him.)

I’m thankful for all the time I got to spend with him over the years, even if as we got older – as these things happen – you don’t get to see people as often as you used to (or liked) and had separate lives. I’m especially grateful Mike made the time to come down to see a Phillies game with me (and my Dad) at Citizens Bank Park in 2010. It was one of the last times he and I got to hang out. Again, some memories are written with indelible ink.

Recently I was watching the Live From Daryl’s House episode (#30), which was a tribute to T-Bone Wolk who had passed away shortly before that episode was recorded. I’ve always loved T-Bone because he was a fantastic bass player. The guys who honor him in this episode were his friends and bandmates. I love how they went to his family’s house, reminisce, and at the end, they have little individual tributes. What Daryl said at the end is really true: life won’t ever be the same, but it does go on somehow. At 1:38 in the last segment, Daryl says, “He’s an irreplaceable person.” So was Mike.

August 23, 2012

When Is It Time To Stop Using a Deprecated Feature?

Twitter conversations can sometimes lead to interesting things. Jonathan Kehayias (Blog | Twitter) and I were going back and forth about something, and in some cases, how we differ on how we teach (some of that was based on teaching or not teaching a deprecated feature). It was a very healthy discussion if you ask me. I really respect Jon and he’s one of the smartest folks on the planet when it comes to SQL Server in nearly every respect. I don’t say that lightly. Paul, Kim, and the rest of the crew over at SQLskills are very lucky to have him on their team.

Be it Windows, SQL, SharePoint, whatever – all of these products at some point release a new version and as part of the lifecycle of that product, some features are deprecated. A good example is that database mirroring has been deprecated as of SQL Server 2012. It’s still in the product, but its development is over and it will disappear from the product in a few versions. Another easy example for me to mention (and what sparked the debate between Jon and I) was the use of the aforementioned cluster.exe in Windows, which was deprecated in Windows Server 2008 R2 and is technically gone in Windows Server 2012 (disclaimer: you can add it back by enabling a certain feature, but it hasn’t seen any improvement or development since Windows Server 2008; more on cluster.exe in a bit). In the immortal words of Hall and Oates, “She’s Gone” (following the Daryl Hall theme from my last post; I was challenged to use “Rich Girl”, but this was more of a natural fit – sorry Rick!).

http://www.youtube.com/watch?v=bnVXIUyshng

That begs the question: when do you stop actively using/teaching/recommending/etc. a certain feature you either know has been deprecated or will be going away (but may not be public knowledge although you have good NDA information which you really can’t share)? The answer, not surprisingly, is “it depends”.

Let’s take the case of my good friend cluster.exe who I’ve lived with for the better part of 10+ years. At this point, I’m not seeing any new deployments of SQL Server 2008 (RTM branch or R2) on the original Windows Server 2008 (no R2); it’s all basically Windows Server 2008 R2 with SP1. W2K8 R2 introduced the first round of PowerShell cmdlets for Windows Server failover clustering (WSFC). When I wrote my SQL Server 2008 book, W2K8 R2 was just being wrapped up so I documented both. As time has marched on since 2008/9, not only have I personally used cluster.exe less, when I teach my classes or do presentations, I don’t even bother showing it anymore. Why? In 2012, to me it makes no sense to show something that is deprecated and outdated, and is gone in Windows Server 2012. If you’re using Windows Server 2008 R2, you really should be using PowerShell to administer your clusters (one man’s opinion, of course; feel free to disagree). However, there are still exceptions to that rule. For one specific thing in Windows Server 2008 R2, there is no PowerShell equivalent (it’s basically related to availability groups in SQL Server 2012). This is fixed in Windows Server 2012. Similarly, some things that have a PowerShell cmdlet (Test-Cluster), have no cluster.exe equivalent. The pros outweigh the cons for me of not showing or teaching cluster.exe these days. Would I if someone asked? Sure. I’m not so stuck on not showing it; I just think it’s better to learn the stuff you can use now AND in the future when you want to go to Windows Server 2012.

The story is not so cut and dry with a feature like database mirroring. Unlike Windows which really makes it clear cut you need to move on – and quickly, SQL Server keeps a feature around and intact for at least two versions before it goes away (another really good example was DB-Lib). That could possibly translate into almost 6 – 10 years of shelf life, giving you a false sense of security. Sometimes there’s a clear cut feature that replaces it. In the case of database mirroring, a lot of people use it now. That’s one factor, so it makes the announcement that much harder. But the real crux of the problem as I see it is a case of the haves and have-nots. If you are using Standard Edition of SQL Server, Microsoft is recommending log shipping as your replacement. For Enterprise (now Datacenter) Edition, it’s availability groups. AGs are a great evolution of what was started with DBM, but with the new dependence on WSFC for deploying AGs, it may be a non-starter for some. But people who want to upgrade to SQL Server 2012 or whatever vNext (or vNextNext) may be, they need to start planning a roadmap otherwise it makes it that eventual upgrade harder because your back will be up against the proverbial wall. There’s no easy answer here, but it’s a conversation I’m having with a lot of my customers.

When it comes to having NDA information on something to be announced in the future, I can never spill the beans. Sorry. If I do, that puts me at risk with whoever I have the NDA in place with (and that comes with all kinds of legal problems for me if I do). I love my customers, but it’s not worth putting me out of business by letting something slip. That said, I can still steer customers in the right direction without them even getting a hint that there may be an underlying and different reason I am doing so.

I’ll leave you with this one final thought: as much as deprecated features are important to consider in your planning efforts, don’t let it become an obsession, either. DBM is a great example – it’s not going away tomorrow. If you’re using it now, that’s OK. A deprecation announcement didn’t mean to stop using it this very minute; just don’t get complacent about making a decision, either. A few years goes by in the blink of an eye when it comes to IT (just look at the people still running SQL Server 2000 for whatever valid reason and are stuck supporting it). Don’t be that guy or gal keeping the lights on for the few instances requiring DBM for the next 15 years. Be smart, make the right choices, and stay on a supportability path.

August 22, 2012

Pick Any Two

It’s been said that you can do things right, cheap, or fast. Pick two.

I’ve heard people substitute other words into that first sentence (such as available, reliable, and fast), but you get the point: you generally won’t get it all when it comes to deploying a solution or working on a project … that is, unless you’re willing to pony up some serious money. Even then, it still may not be possible due to some other limitation. Truth be told, of that initial list, I might argue you only get one of three (right and fast are not always friendly bedmates), and even that may be a question mark. Why? This all underscores an important point in what we do in IT as well as in life: often times, it’s about tradeoffs. The right solution or decision is often the most difficult one to make or implement. No one likes to work anymore, right? 🙂

Whenever I see offerings from various companies offering you solutions that promise you the world (unlimited scalability, five nines of uptime, we manage it for you – nothing for you to worry about, etc.), I just chuckle. Either they did the right work on the backend to make that happen (not likely) or they’re selling you a bill of goods (maybe). The truth probably lies somewhere in between.

It takes effort, knowledge, skill, budget, and often, a bit of luck for the stars to align. How many times have you joined a new company with the illusion you’ll change the world or started a project completely idealistic, only to find things like a small budget spoils the plans and you’ve got to figure out how to “make it work”? This is also why in many cases, besides them not possibly applying to your scenario, best practices are often hard to implement. They’re not always real world. “But I must have RAID 10!!!!” Keep on living the dream when your SAN guy or gal gives you whatever they have and you have no choice in the matter.

It’s nice to be idealistic. Maybe you can join Daryl Hall in Dreamtime (and as evidenced in this clip, he still looks/sounds pretty damn good in 2012 at 65!)

http://www.youtube.com/watch?v=Re805lUUfvU

Here’s my reality: I live in the real world with customers that have resource, time, budget, and skill set issues (among many other constraints; every customer is different) which need to be taken into account when I deliver something. It’s not about being right (in some way it is, but standing on ceremony gets you nowhere) or saying “I told you so”. At the end of the day, you have to be realistic, and so does your management as well as the bean counters. You’re not getting 24x7x365 and great performance on things like limited budget, staff, and severe time limitations (a subject I addressed a little here). Something has to give. Sometimes it is a technical reason that something isn’t possible. Yes, we’ve come very far in the past 10 – 20 years, but everything isn’t possible. And Microsoft isn’t changing SQL Server or Windows just for you to make you work a certain way, either. Even if you offered to fund that development, I doubt it’d happen.

The worst offenders here I find are managers or C-levels. It’s not all of them, but many. They may have once been technical, but now they may be removed from the day-to-day of what it takes to get something done. I mean, don’t we worker-bee types all have magic fairy dust to create a /faster or /moreuptime switch? If I did, do you think I’d still be working? I’d be selling and/or licensing the hell out of it and retired.

What you need to do, and I do it all the time, is get everyone on the same page with the right information so you can move forward. Like I said, some things will have to give. That may mean you need more budget, time, etc., or hard cuts will have to be made to make other things happen. That’s life. When we can have the right conversation without anyone digging their heels in, the real work can begin.

Popular Posts

Categories