process Archives

February 27, 2013

Devs the New Superheroes and Rockstars? What About IT?

With the proliferation of smartphones and touch coming of age in the mobile and desktop world, we’re really starting to border on Minority Report territory. Applications are an end user’s interface into the online world, and they create the experience. Some are good, some are horrendous. We’ve all seen and used bad applications over the years. As a backend kind of guy who associates himself more in the IT space overall, it is all about the end user and/or the business. It’s something I have to remind some folks when I’m onsite doing work for them. We support them. Having said that, a few things lately have raise my dander – namely Code.org and this blog post (which I saw via Grant Fritchey’s blog post). Why do these bother me?

Let me say this up front: communication is a fundamental issue in many organizations and there’s enough blame pie to go around. Politics and other road blocks can slow any process down causing frustration everywhere. Poor communication and broken processes hurt more than help. However, process and communication should NOT be difficult. Implemented properly, they should be easy and transparent. Things like change management and source control are fundamental concepts in both IT and development, and should enhance things like high availablity. To me, they are cornerstones and core tenets of an overall availabilty strategy. However, how many of you DBAs use source control … go ahead … I’ll wait. In my experience, that number is very low. In a similar fashion, change control and things like testing are often viewed as a nuisance to developers and pushing direct to production can often lead to many headaches for DBAs. We annoy you, you annoy us; perfect recipe for a standoff.

I don’t disagree with the fundamental premise of Code.org. I remember getting started on a Commodore PET and then a 64 with things in BASIC like:

10 PRINT “HI”
20 GOTO 10

Even us IT guys need some programming skillz(tm). (See what I did there? I’m pandering to the hipster crowd.) Things like PowerShell or our very own Transact-SQL (T-SQL to those “in the know”) in SQL Server need you to understand at least fundamental programming concepts. Heck, there’s even WMI and SMO (among other things) if you really want to get adventurous in the Windows/SQL Server world. The longer video Code.org made is cute when the ask the kids what they want to be and when they asked them do they know what a computer programmer is. Most little kids want to be firemen, athletes, princesses, etc. That makes sense. Being a programmer is way too practical. I mean, you don’t see anyone at the age of 5 asking to be a sanitation worker or insurance broker, do you? I’m sure there are some kids, but they’d be a reaaaaaaaaaaaaaaaaaaaaally small percentage. Kids are allowed to be dreamers. Adults need to pay bills and live in reality most of the time.

But that got me thinking – why is it OK to glamorize just programming in the computer realm? Everything else is as, if not more, important in terms of the day-to-day running. Do you think Amazon.com runs on a single web server and only one database server? Heck no! Both the application they have and the backend are designed to scale and be available. If you’re going to teach programming, kids need fundamental IT basics, too. Things like understanding about backups would serve them well even in their daily lives. How may times to we have to hear things like, “I lost all of my photos on my external hard drive/phone and I didn’t have them anywhere else.” People’s lives are wrapped up in digital.

As I said, apps are the gateway for people. Be it a browser or one on your phone, it makes sense to say that programming and computers should be a fundamental part of one’s education curriculum earlier in life. That part I don’t disagree with. But what they are not telling you about is how to make it all work – the full view of the application lifecycle. It’s one thing to write an app. It’s another for it to work well and perform. Many of us who are consultants would have less opportunities if applications were written properly and scaled and supported what they needed to. I see way too often that applications are often the barrier for upgrades in many environments, leading to many different – and hard – problems to solve as time goes on. How many applications in your environment – third party or custom – support SQL Server 2008 or later? I bet not all do. Or if they did, your company won’t uprgade to it, forcing you to stay on an older version of SQL Server for other reasons like cost … and this puts not only your environment at higher risk as time goes on for issues, but your skills, too. SQL Server 2005 is now nearly 10 years old and four (4) major versions old. That’s like dog years in the technology world.

Developers also make false assumptions – like availability is only IT’s problem. It’s not. There’s stuff they need to do, too, yet they will blame us for their application woes when the application barfs after failover. Too bad, so sad. Now we all live with the pain of your stupid decisions and blinders that you need to be part of the solution from day one, not part of the problem. This is true whether you are using SQL Server, Oracle, or anything else. DBAs are often the last to know about an implementation – and thus, take a lot of heat and blame if/when things go wrong. DBAs need to be in on the planning from day one.

The thing that got me the most was the graphic in the NoDBA post – namely the bit where he has “Heroic Developers” and automatically associates things like bureaucracy and delays with data management. Huh? Sure, DBAs can be a pain in the tuchus. So can devs. Or network admins. Or storage admins (DBAs never have problems with storage folks, right?). You get the point. As I mention, process is a necessary evil for things like availability. In his post, Martin Fowler does acknowledge that one of the negatives of the dev to prod approach can be something like “bypassing DBA groups may also mean bypassing operations groups that know how to keep valuable data backed up and secure”. Amen. But I still won’t allow a dev in production if I can help it.

The bottom line: in the real world, we’re all part of the solution, and may even be part of the problem to someone else. There’s a reason in most cases devs should not just push code – let alone untested code – out to the world. But IT also needs to be more aglie than it has been. That’s one of the things we do here at SQLHA – we help get organizations up to speed and during implementation, do not implement crippling process that does not work for anyone. This is one of the reasons why virtualization and concepts like the private cloud are taking hold – it’s a more agile deployment than procuring hardware for every new deployment. It doesn’t make it right or wrong, but it changes the dynamics. Smartphones and apps have changed the dynamics of even rapid application development. IT unforunately hasn’t always caught up to meet that demand.

It’s time for everyone to grow up. Kids should learn about computers right along with math and other key things, devs and developing applications is important, but so is IT. You really can’t have one without the other, so let’s find a better way to work together and start earlier. That way we can have less finger pointing later on. Deal?

June 28, 2012

Maintaining Quorum During Windows Patching and Updating

I’ve been working with a customer to deploy their new Windows Server 2008 R2 SP1-based failover cluster that will have bot SQL Server 2008 R2 and SQL Server 2012 instances. One thing I always talk about in my classes as well as the presentations I focus on patching is this: you must maintain quorum during the patching and updating cycle. Quorum is one of the most important considerations in general for your clustered deployments (whether you’re implementing traditional failover clustering instances or availability groups), and it’s something that quite frankly, I find there’s a real lack of knowledge out there. It’s so important it’s most likely getting a whole chapter in my upcoming book Mission Critical SQL Server 2012.

I have talked to customers that have had SQL Server outages because their Windows guys when patching the underlying nodes for the Windows Server failover cluster (WSFC) rebooted too many at once, so there were not enough voters us, thus bringing the WSFC (and SQL) down when it wasn’t expected. This is NOT a scenario you want. While in theory it’s easy to control and keep track of when things reboot, this little gremlin in Figure 1 pops up after a little while reminding you it needs to be done:

Figure 1. Reboot needed warning

That’s all well and good, but the unskilled administrator who happens to see that message if they have to log onto the server may think nothing of it and just click Restart now. Yikes! The value of the reminder can be altered as shown in Figure 2, but you still need to worry about the reboot being done.

Figure 2. Changing the reminder value

There’s another dark underbelly I’ve seen these dialog boxes associated with: automatic updating of Windows via Windows Update. That is a much scarier situation because that means your servers could be updating 24×7 on their own, and some of those updates may even force reboots. Check your settings! I still see production servers at client sites that have automatic updating enabled. If they are, you need to have a serious discussion with your server admins as to why this is being done. Automatic updating is not in line with mission critical. Yes, it’s important to have a patch management strategy that keeps you up-to-date, but doing it smartly is the key. Not blindly.

August 5, 2009

Rome Wasn’t Built In 15 Minutes … Neither Is a SQL Server Solution

I hit on this topic a little in my previous blog post (Helping the Helpers), but I wanted to expand on it in a different way. Whether you’re posting to a newsgroup, at a conference in a 75 minute session, or even paying for an hour of help, very rarely – if ever – are you going to design a solution or get the whole story in a short amount of time. Is it impossible? No. Unlikely? You bet.

I’ve found people want absolutes, and I sound like a broken record, but nearly everything about SQL Server, and especially with things like high availability boil down to two words: it depends. I never take a one size fits all approach when I’m consulting or presenting a session and answering questions. What is right for one customer is not for another; there’s a difference between how something works, which can lead to black and white answers, and how something fits into an overall context. I like to think that I provide end-to-end, holistic, and realistic solutions when I work with customers; I don’t just “do clustering”. I can’t know everything about your situation by only talking to you for a few minutes; I can make some good guesses as to what you have going on based on experience, but I’m always hesitant to give anything more than more generic advice if I don’t know most of the facts. Here’s an example: people are suprised I don’t lead with failover clustering as a solution for everything considering it’s what I’m best known for. It’s just not appropriate in every single solution I help with, but more often than not, it can and does work.

Similarly, I’m often the focal point for what people don’t like or feedback about a feature – for example, I can’t tell you how many times I’m asked why failover clustering isn’t like Oracle’s RAC. I may have worked for Microsoft, but I didn’t design SQL Server. I don’t sit in meetings where decisions are made about features like failover clustering. A lot of the problems I think people have are just misunderstandings of how a feature or technology works versus how they think it should work. Whether you agree or disagree with how something works, it is what it is. No one knows that better than me.

It’s not always easy to ride the line between honesty about how a feature works versus sounding like you’re complaining. I try to do the former, and let people form their own opinions. Anyone who knows me well understands I’ll be honest about how I feel about something, but one of the things you learn as you get on in a career is when to self edit. If you’ve seen or used either Facebook or Twitter, you know what I mean. Some people put a LOT about themselves out there. My job as a speaker is to convey my experience, give some opinions, and to paint a realistic picture of how things work – good and bad. Nothing is perfect. It is NOT my job to bash Microsoft or a feature. Sometimes those opinions or facts of how a feature may come off as negative, but that’s not really the intent. Those in the audience won’t relate if I’m trying to sound like a salesman; they know the pain that can happen when you’re a DBA. I also realize that DBAs who come to see me speak just want some nuggets to do their job better; they’re so busy doing their day to day thing, when they’re at a conference they need to soak up as much as possible because they’ll be back to heads down in a day or two … if not already, since conferences are always working events for everyone.

I always joke, but it’s true – you often spend more time planning than you do the actual work. You can plan for months to do a data center move that when executed, only consumes 24 or 48 hours. It may seem odd, but it’s completely accurate. The technical bits are only part of the picture; failover clustering is just the end implementation of an overall high availability solution which is tied into people and process, SLAs, and the needs of the application. Those months are spent mitigating risk, gathering information, and testing.

So what’s the bottom line here? Don’t expect that what Company A does is right for you, and to get it right is more than talking to someone for a few minutes. Take what you get at a user group or conference, and apply it to your situation. There is no magic SQL fairy dust which gives you better availability that is completely transparent to every application known to mankind. Everything has its tradeoffs. Knowing how each feature works allows you to design applications and solutions which fit your needs.

May 21, 2009

Want a Great Example of Why People and Process (Including Testing) Matter for High Availability and Disaster Recovery?

I live in the Boston area. Today, there were massive delays on the T trains (public transportation) all due to a power outage. The power outage which lasted about 7 minutes, was inadvertently caused by a maintenance crew accidentally tripping a breaker at one of the worst possible times – rush hour. To get everything back up and running took about 30 minutes, but the outage brought the trains to a standstill and things took awhile to get back to normal.

http://www.boston.com/news/local/breaking_news/2009/05/power_outage_de.html

Imagine if someone did something similar to your IT infrastructure? This is why having clear processes, good plans, and fully testing everything BEFORE implementing in production is key.

Popular Posts

Categories