DBA Archives

February 27, 2013

Devs the New Superheroes and Rockstars? What About IT?

With the proliferation of smartphones and touch coming of age in the mobile and desktop world, we’re really starting to border on Minority Report territory. Applications are an end user’s interface into the online world, and they create the experience. Some are good, some are horrendous. We’ve all seen and used bad applications over the years. As a backend kind of guy who associates himself more in the IT space overall, it is all about the end user and/or the business. It’s something I have to remind some folks when I’m onsite doing work for them. We support them. Having said that, a few things lately have raise my dander – namely Code.org and this blog post (which I saw via Grant Fritchey’s blog post). Why do these bother me?

Let me say this up front: communication is a fundamental issue in many organizations and there’s enough blame pie to go around. Politics and other road blocks can slow any process down causing frustration everywhere. Poor communication and broken processes hurt more than help. However, process and communication should NOT be difficult. Implemented properly, they should be easy and transparent. Things like change management and source control are fundamental concepts in both IT and development, and should enhance things like high availablity. To me, they are cornerstones and core tenets of an overall availabilty strategy. However, how many of you DBAs use source control … go ahead … I’ll wait. In my experience, that number is very low. In a similar fashion, change control and things like testing are often viewed as a nuisance to developers and pushing direct to production can often lead to many headaches for DBAs. We annoy you, you annoy us; perfect recipe for a standoff.

I don’t disagree with the fundamental premise of Code.org. I remember getting started on a Commodore PET and then a 64 with things in BASIC like:

10 PRINT “HI”
20 GOTO 10

Even us IT guys need some programming skillz(tm). (See what I did there? I’m pandering to the hipster crowd.) Things like PowerShell or our very own Transact-SQL (T-SQL to those “in the know”) in SQL Server need you to understand at least fundamental programming concepts. Heck, there’s even WMI and SMO (among other things) if you really want to get adventurous in the Windows/SQL Server world. The longer video Code.org made is cute when the ask the kids what they want to be and when they asked them do they know what a computer programmer is. Most little kids want to be firemen, athletes, princesses, etc. That makes sense. Being a programmer is way too practical. I mean, you don’t see anyone at the age of 5 asking to be a sanitation worker or insurance broker, do you? I’m sure there are some kids, but they’d be a reaaaaaaaaaaaaaaaaaaaaally small percentage. Kids are allowed to be dreamers. Adults need to pay bills and live in reality most of the time.

But that got me thinking – why is it OK to glamorize just programming in the computer realm? Everything else is as, if not more, important in terms of the day-to-day running. Do you think Amazon.com runs on a single web server and only one database server? Heck no! Both the application they have and the backend are designed to scale and be available. If you’re going to teach programming, kids need fundamental IT basics, too. Things like understanding about backups would serve them well even in their daily lives. How may times to we have to hear things like, “I lost all of my photos on my external hard drive/phone and I didn’t have them anywhere else.” People’s lives are wrapped up in digital.

As I said, apps are the gateway for people. Be it a browser or one on your phone, it makes sense to say that programming and computers should be a fundamental part of one’s education curriculum earlier in life. That part I don’t disagree with. But what they are not telling you about is how to make it all work – the full view of the application lifecycle. It’s one thing to write an app. It’s another for it to work well and perform. Many of us who are consultants would have less opportunities if applications were written properly and scaled and supported what they needed to. I see way too often that applications are often the barrier for upgrades in many environments, leading to many different – and hard – problems to solve as time goes on. How many applications in your environment – third party or custom – support SQL Server 2008 or later? I bet not all do. Or if they did, your company won’t uprgade to it, forcing you to stay on an older version of SQL Server for other reasons like cost … and this puts not only your environment at higher risk as time goes on for issues, but your skills, too. SQL Server 2005 is now nearly 10 years old and four (4) major versions old. That’s like dog years in the technology world.

Developers also make false assumptions – like availability is only IT’s problem. It’s not. There’s stuff they need to do, too, yet they will blame us for their application woes when the application barfs after failover. Too bad, so sad. Now we all live with the pain of your stupid decisions and blinders that you need to be part of the solution from day one, not part of the problem. This is true whether you are using SQL Server, Oracle, or anything else. DBAs are often the last to know about an implementation – and thus, take a lot of heat and blame if/when things go wrong. DBAs need to be in on the planning from day one.

The thing that got me the most was the graphic in the NoDBA post – namely the bit where he has “Heroic Developers” and automatically associates things like bureaucracy and delays with data management. Huh? Sure, DBAs can be a pain in the tuchus. So can devs. Or network admins. Or storage admins (DBAs never have problems with storage folks, right?). You get the point. As I mention, process is a necessary evil for things like availability. In his post, Martin Fowler does acknowledge that one of the negatives of the dev to prod approach can be something like “bypassing DBA groups may also mean bypassing operations groups that know how to keep valuable data backed up and secure”. Amen. But I still won’t allow a dev in production if I can help it.

The bottom line: in the real world, we’re all part of the solution, and may even be part of the problem to someone else. There’s a reason in most cases devs should not just push code – let alone untested code – out to the world. But IT also needs to be more aglie than it has been. That’s one of the things we do here at SQLHA – we help get organizations up to speed and during implementation, do not implement crippling process that does not work for anyone. This is one of the reasons why virtualization and concepts like the private cloud are taking hold – it’s a more agile deployment than procuring hardware for every new deployment. It doesn’t make it right or wrong, but it changes the dynamics. Smartphones and apps have changed the dynamics of even rapid application development. IT unforunately hasn’t always caught up to meet that demand.

It’s time for everyone to grow up. Kids should learn about computers right along with math and other key things, devs and developing applications is important, but so is IT. You really can’t have one without the other, so let’s find a better way to work together and start earlier. That way we can have less finger pointing later on. Deal?

February 3, 2012

Just Because You Don’t Like The Answer Doesn’t Mean It’s Wrong

Many people, including myself, spend our own time answering questions on things like the #sqlhelp hash tag on Twitter and on various forums (TechNet/MSDN, SQL Server Central, etc.). This even happens during talks in Q&A sessions. Most people are nice and gracious, but every now and then, you have someone who just keeps chipping away at the stone to hopefully get the answer they really want to hear (even though it’s not possible). 10 observations:

One thing I see quite often is the lashback when a correct answer is provided but yet the arguments still continue. Here’s the deal: some technologies – clusters are a good example – work the way they work. I didn’t design the feature. For example, the requirement for shared storage for a clustered instance of SQL Server up through SQL Server 2008 R2 (this changes a little in 2012, but only in certain scenarios). You can’t get a Windows failover cluster or a SQL Server clustered instance to work any other way potentially without either unsupported (by MS) hacks or not clustering at all. Don’t hate us if you may have not thought out all aspects of your design, have run into an issue, and don’t like what we have to say. We’re really not trying to egg you on or irk you – we’re just giving facts.
Don’t expect us to design your massively complex solution for you for free based on 140 characters or a blog post in a free forum. That’s consulting and what most of us do for our day jobs. We’ve got bills to pay. We can point you in the right direction, but hopefully you’re going to base your design on more than a few line response. Those of us who answer questions love to give back, but realize that there’s always going to be some caveats to what you get for free in a smaller response.
We can’t make you experts in 140 characters or a series of replies. Hopefully you’re not expecting that.
Help us help you. This can’t be said enough. The more and accurate info we have, the better. I go through this with my clients all the time – it’s hard to make decisions about tomorrow when you don’t know what is going on today, what happened last week, or even last year. You need historical data to be able to make informed decisions.
You’re probably wasting your time if you ask something like “I need more disk performancebut I’ve got no budget. Will 5 SATA disks work?” You pretty much have already answered your own question. You can do things right, you can do things fast, or you can do things cheap. Those three things generally don’t all work together. Sometimes you can do them right and fast, but never cheap. Fast and cheap are often lumped together, but often leads to never being right. I think you get the point. There’s a tradeoff somewhere.
Capacity is not just size (memory, storage, etc.). There’s a whole performance aspect you need to consider. That’s why we ask questions like “How many I/Ops do you need?”
Your workload matters. Know what it is and be able to hopefully articulate that. What advice works for one implementation may be totally wrong for another.
RT(F)M. Before asking us a question, do some simple homework first. A simple BinGoogle search may yield your answer or point you to a BOL topic to allow you to ask a more directed question. That’s where we’ll provide you with (hopefully) some value.
Do NOT post any confidential info about your company, app, or data. This may include things like schema, object names, etc. Common sense, but is it really? Find a way to express your question while still keeping sensitive things under wraps. Easier said than done, of course. This is why I put NDAs in place with my customers when we work together. It allows me to do work and protects both them and me.
I subscribe to the motto that there are no stupid questions, and I truly believe that. Heck, we all started somewhere. It’s one reason I give back to the community. I didn’t get here on sheer will alone – I’ve had help along the way. But really look at #8 above – many questions can be answered that way. What is not easy, though, is trusting the information source it may come from. I’ve seen some sketchy advice put out there (no, I won’t say where), and it’s a free Internet. People can post what they want based on their experiences – just doesn’t make it technically sound advice. OK, I lied – one example which involves shrinking. We all know how shrinking is not a normal best practice. Don’t believe me? Just read some of Paul Randal’s thoughts. Point is, that first linked post gave OK technical instructions, but not necessarily the most sound advice. Learn to differentiate the two. In the case of that original link, I bet the DBA smartly sized the DB for longer term growth. We’ll never know.

Sorry if this sounds cranky, but I see this stuff over and over. We really do want to help you. Just don’t shoot the messenger if you don’t like the message.

October 31, 2011

So You Say You Want To Be Mission Critical? Hire a DBA

I don’t know what’s in the water in IT departments, and it’s definitely not the fact that today is Halloween. No tricks or treats in this blog post! I’ve had a lot of conversations with companies over the past year that are trying to be mission critical. By mission critical, I mean near 24×7 uptime (give or take). While I think that’s great and definitely an area I’m well suited to help them with from a consulting perspective, what I don’t like to hear is one of the following:

We don’t have any SQL Server DBAs
Our <insert worker already doing other things> is our part time DBA

I’m finding many companies are simply lacking a wholly owned, full time SQL Server DBA on their staff. I’m perplexed. Those two things – not having a DBA of any sort and being mission critical – are at odds. Let’s dispel some myths and deal with some realities.

SQL Server Manages Itself

I used to work at Microsoft and am lucky to still have a close relationship with them. Years ago Microsoft had a five nines campaign that launched around the same time we were doing a week long SQL Server 2000 high availability event. They were basically trying to show that Windows was ready for the enterprise and five nines (99.999% uptime – 5.24 minutes of downtime per year). The ad I remember most had the picture of a data center with a lone, empty rolling chair and balloons floating around insinuating everyone is out having fun and things managed themselves basically. They couldn’t have picked a worse time to send such, in my opinion, a wrong message. By the way, here is proof of that campaign. I can’t dig up the ad anywhere. If you have it or know where it is, please link below or e-mail it to me! Anyway, anyone who has done any production work knows that message in that campaign is wrong. Ever get paged at 2AM? ’nuff said.

SQL Server marketing has done an excellent job over the years of making it seem like SQL Server manages itself. To a point that is true with some of the stuff internally in the engine such as statistics (although there are cases where you may have to deal with those manually), etc. All IT shops that use SQL Server have the same basic care and feeding needs, but the scale is different in a small shop vs. a large enterprise. If you think you can effectively manage hundreds or thousands of databases and along with their accompanying instances (and all of the associated hardware) with no DBA or at least having someone on staff who knows SQL Server, I have a bridge to sell you in Brooklyn.

Sure, you can automate some tasks like monitoring and some tasks to take care of known issues, but you still need human beings to interpret what that information is saying. Some things are not obvious. A big part of monitoring is noticing trends – it’s not just for break/fix scenarios. A good DBA automates what he or she can, and deals with the rest. A good example is getting a report in the AM telling you what backups ran successfully or failed. You don’t want to connect to every instance to get that information. It’s time consuming. Spend your work hours dealing with the problems (like making sure the backups are done if failed) instead of figuring out of they ran or not.

Let’s talk about maintenance plans for a moment. I know they are popular and certainly have use. Heck, it’s easy to click some checkboxes, click Next a few times, and bam! Instant maintenance. In my estimation, 9 out of 10 times, it’s always better to code that stuff yourself and implement via SQL Server Agent jobs. That way it’s not a black box. Plus, you can easily alter things to meet your needs. For example, do you really need to rebuild ALL of your indexes at whatever interval you specify? Probably not. You most likely need to monitor what’s going on, anticipate the need based on history, and then have a job that is either manual or scheduled to fend off a problem based on that information. A maintenance plan isn’t going to work there.

Bottom line here is that over time, a DBA should be able transform a shop from being reactive to proactive. HUGE benefit and win for all. That’s why you need someone who knows what they’re doing. They make it look easy even when it’s not.

A Multi-Tasker is Better than Dedicated Staff

I think we’ve all been in roles at some point in our career where we’ve worn multiple hats. Knowing different disciplines can help any role, but you should have a primary focus. For example, a DBA should be able to understand Windows, storage, and networking, but they don’t need to be gurus. However, if you are the lone administrator responsible for everything from Active Directory and Exchange to SharePoint, SQL Server, and beyond, do you really think you’ll be efficient and a deep expert at any one of those – let alone all of them? I’m not saying it’s not possible … just improbable.

The argument, especially in very small shops, is that one person should be able to handle it all. I don’t necessarily disagree. In a small IT shop where there are not a lot of systems, the load is manageable, and nothing is very complex, a jack-of-all-trades may be perfect if they’ve got some experience under their belt. However, if you have complex deployments and numbers, that argument fades fast. For example, do you think your jack of all trades can handle multi-subnet clusters or SAN replication between sites? That’s some heady networking and storage infrastructure that needs proper planning and administration going forward. The wrong person doing the wrong thing could give you lots of downtime and problems.

Look at all of the dimensions of what a person brings to the table, not just the cost in terms of dollars.

Oh, and your developer – not such a good choice for a production SQL Server DBA for numerous reasons. I’ll leave that one alone in this blog post. Could make good fodder for another one.

Yes, Virginia – SQL Server is Mission Critical in Your Environment

That phrase is a spin on “Yes, Virginia – there is Santa Claus”. I’m never surprised anymore that some environments do not consider SQL Server essential. They’ll talk about Oracle or DB2. Yet when I go in and do some work – especially an assessment or consolidation – lo and behold we find out that there tons more SQL Server databases and instances (and the applications that use them) than they thought existed. They only knew about a subset since they were not under any kind of central administration (a different issue I’m not going to touch here). SQL Server may not be powering your Peoplesoft implementation (maybe it is; substitute your favorite application if you’d like), but it’s everywhere else under the sun and used by nearly everyone in one capacity or another. So I would say based on sheer numbers of DBs and instances alone (not even marrying that info to the apps which may be used 24×7), SQL Server is mission critical in your environment whether you like it or not. Welcome to 2011. SQL Server is ready for enterprise class stuff.

As such, it needs to be treated in the same way you do the other RDBMS choices you have. I bet you have Oracle DBAs – why not SQL Server?

The Economics

I also understand the math behind all of this. Hiring folks is not free, be it a FTE, a consultant, or an outsourced DBA service. How much is your data worth to you to ensure you have peace of mind? It’s arguably more important than your applications since your data is a big part of your IP. If you’ve got tight service level agreements (SLAs), recovery time objectives (RTOs), and recovery point objectives (RPOs), having that jack-of-all-trades-master-of-none may work in some cases, but what happens when it doesn’t? The cost to benefit ratio of having someone skilled is not a straightforward calculation that can be done by the bean counters behind a desk .

What I also see is that for whatever reason, SQL Server folks are valued lower than, say, their Oracle counterparts in enterprise IT shops. An enterprise DBA is an enterprise DBA, period. Even in terms of contract work, I sometimes chuckle at what gets sent my way. People want the sun, the moon, and the stars all in one person for very little money. I always tell whoever sends these requests that if they find that person for that rate, hang onto them.

The reality is that someone with a BI background may know very little about relational (outside of data warehouse stuff) and a relational person may know nothing about Analysis Services, Reporting Services, et al. Each one of those things is really a different role that just happens to fall under the banner of SQL Server because that is the branding. An operational DBA may do some of those other tasks, but their primary function is the care and and feeding of your SQL Server environment to ensure it remains up, healthy, and performing. Anything else in my mind is a bonus. SQL Server is a big product – understand that when you set out to find someone for your needs. You may need two or three people, not just one.

EDIT: After I posted, I saw this tweet on the #dba hashtag:

Oops!

Think this means less productivity? That’s a cost … and in this case, having the right person either may have prevented this OR would get them back up and running. Factor that in – you may cost others a lot of time and money with the wrong person, too.

Where Does This Leave Us?

I know this post is going to be a little controversial for some, and you may disagree with me (would love to see you post if so), but in the near 20 years I’ve been involved with SQL Server, I’ve never seen where having someone who really knows what they are doing is a detriment. It’s usually a benefit. 2011 is not different than 1990, 1995, 2000 or any other year – you need to know your stuff as best you can to do a good job. That person coming right out of college with no real world experience may be cheap in terms of cost, but would you trust your 24×7 system to them and no one else? Probably not. Ideally you would have a mix of senior and junior folks because both have value. The senior resources can help and mentor the junior ones.

At the end of the day, realize that if you want to be (or think you are) mission critical, you cannot ignore the simple fact that not having at least one qualified, dedicated SQL Server DBA is a surefire method that will most likely prevent you from ever achieving your goal.

June 2, 2011

Raise Your Game

You are under attack. No, I don’t mean some foreign country is going to invade your city. Your job as a DBA or IT admin is. Why do I say that? Well, you’ve got lots of threats depending on your environment: the cloud, virtualization, skill sets, budget, new technology, etc. There’s a lot you have to do on a daily basis to make sure you still have a job tomorrow and the next day. Fall too far behind and you become irrelevant. New releases keep coming, so you have to keep up even if your company is not using them or will get them in production any time soon to stay relevant. Ignore trends like virtualization and the cloud (and where you will fit into those pictures), and you will be left behind.

This blog post was inspired by a con call I had today. Some people on that call – all very smart and very good DBAs – did what a lot of DBAs I know do, and basically said, “I don’t want to be a Windows failover clustering expert.” Fair enough, but do you have any prayer of being successful with things like failover clustering (or the upcoming AlwaysOn availability groups feature in Denali) if you don’t undertstand the underlying platform which it is built upon? In some ways, you should be smarter than your Windows, networking, security, and storage guys as it relates to your implementations around this stuff because they probably don’t know SQL Server and what you don’t know about their shortcomings will hurt you long term. You must articulate what you need to be successful. I’m not just singling out failover clustering. This would hold true even if you are not using clusters or even a DBA and are just a plain ol’ IT admin. The only way to understand what you’re doing is to know how things work underneath the covers to a point you’re at least comfortable with it. By no means am I telling you to become a Cluster MVP.

If you’ve seen any of my talks, you know I’m a big supporter of PowerShell. Heck, way back in 2009, I wrote a blog post entitled “The Incredible Shrinking DBA Skillset Rant #1 – Scripting” and said this:
“I can’t say it enough at this point: learn PowerShell. And if you didn’t understand me, this should clear it up for you: L E A R N P O W E R S H E L L. You’d be foolish not to. Microsoft has often (and sometimes rightly so) been criticized for not standardizing toolsets between products. PowerShell is becoming that unifier, and you WILL be left behind as an administrator if you don’t learn it.”

On that call today, not only did the DBAs basically say “we don’t do that” with regards to PowerShell in general and for administering Windows failover clusters (do you ever patch your instances of SQL Server 2008/R2?), but one of them said even their Windows guys do not either. I was on a con call recently with another customer who said that where they use Windows Server Core (not for SQL Server – just qualifying), they have some issues with administering it because their guys are not command line or PowerShell knowledgable. So they generally don’t use Server Core because of the skills of their admins and their reliance on a GUI – it wasn’t a Windows issue. That floors me.

Look, I know we’ve all gotten used to some of the GUIs Microsoft and other vendors provide, but you may not get all options in a GUI. Some advanced things may only exist in PowerShell or some command prompt version of the GUI. Do not be scared of the keyboard. Maybe it’s because when I came up in the industry, I had to know command line for OSes like Solaris, HP-UX, Novell, and VAX/VMS (and no, I’m not 60!), but I can’t fathom NOT doing some command line stuff. Command line is sometimes faster, and to be dead honest with you, scripting is the only way to truly manage large environments for tasks to be repeatable. Running Maintenance Plan Wizards for 1000s of databases doesn’t really scale despite what you may believe.

I know not everyone out there is resisting scripting or burying their heads in the sand saying things like, “Dealing with the Windows cluster stuff isn’t my concern.” Kudos if you’re not that person. If you are – even a little – go ahead, continue to think that. You might as well shoot yourselves in the foot, too. If you are a masochist, you’ll be in your element. If you want to have an easier time, wake the heck up and grab a (proverbial) strong cup of coffee. Don’t hit snooze on the alarm.

Consider this your second tough love wakeup call. 2009 wasn’t enough with regards to scripting, but this “It’s not my problem” attitude has to go away. If you don’t understand the basics, how can you do anything advanced? I don’t think asking folks to know a bit of hardware and OS stuff is unreasonable for a DBA.

February 9, 2011

Transitioning for the DBA Crowd

Whether you’re leaving for a new opportunity, a fellow DBA is leaving (or going elsewhere in the company), or someone you deal with that touches the databases you administer (such as the application admin) is moving on, you need to be prepared to do your own exit interview. Chances are their collective knowledge has not been sucked out of their head, and it walks out the door when they leave – never to come back. As a DBA, there are things you will need in this situation, or if you’re the one moving on, what you should provide people with so you’re not the guy they are cursing two weeks from now when you’re no longer there.

The Dos

If you don’t do it already, start documenting your procedures and specific things you do which no one else may know. For example, when the SuXor application had performance issues and you had to apply a hotfix to correct the problem, what was it? That info will come in handy if they ever need to rebuild the server.
Ask questions. What you don’t know will hurt you. Chances are you didn’t set up the servers since you just got into your DBA role 30 days ago, but you damn well better know what you have and what they need. At the end of the day, it’s your butt on the line when the CxO can’t get their report and it’s due to a missing linked server you didn’t know about.
Get passwords … and change them. If the person leaving set up the apps and databases, chances are they used a password which may not be documented somewhere. Get it. Even though whoever is moving on may have been the nicest of people, if you have a single password used for ‘sa’ on all of your instances, it’s best to change it when the person leaves. Even if someone won’t have VPN access, you don’t want to take the chance.
Return everything. Sure, the company probably won’t miss that $50 USB stick they bought for you and has no asset tag. Give it back.
Be proactive.If you know you’re leaving, initiate the transition of your duties, information, etc. In a best case scenario you do such a good job at it you can leave earlier than planned to start your new opportunity. People will hopefully remember you fondly if you do this.
Set a reasonable time limit for people to contact you after you leave or ask if you can get that person’s contact info. Look, you don’t still want to get calls six months down the road, but assuming you left on good terms, there’s no harm in answering some questions quickly over the phone or e-mail for the first few weeks after you leave. Good karma, baby. Set a line in the sand (or ask where that is). At some point if they still need your help, you’ll have to cut the tap off or tell them they need to start paying you. Just as much as it’s good to have a nice transition out, you still need to onboard at a new position. Understand if it’s not you and you get the “no more” message. Goodwill only goes so far.
Realize that life goes on without you. Hard to believe, right? Every company I’ve left has managed to survive without me. They couldn’t always replace me, but they managed to get on with it. You’re not the be-all-end-all.

The Don’ts

Don’t burn a path on the way out the door with a flamethrower. Whether it’s you or someone else, people on the other end don’t appreciate the person who is leaving doing things that make enemies. The person leaving may be unhappy, but there’s a pretty good chance he or she will run into those people again in the future. Bad word of mouth gets around.
Don’t destroy anything which may be of use to others. I’m sure your share on the corporate network has lots of good stuff in it. Don’t delete it without thinking. If you’ve got things people may need, send them in e-mail or ask people where to copy that stuff to. Believe me, they’ll appreciate it. Don’t assume they have what you have.
Don’t poach other employees. Chances are you’ve got a clause in your employee agreement which prevents it, but even if you don’t, it’s pretty bad form to do it. At least if you’re going to do it, let the body go cold and have some discretion so it appears they left of their own free will. The last thing you want is a lawsuit. I bet the corporate lawyers have time and money on their side.

So that’s what I’ve got so far … what are your Dos and Don’ts?

Categories