Allan Hirt, Author at SQLHA

May 27, 2011

Life Lessons for IT Pros

I don’t know about you guys, but the longer I am in this industry, the more I realize there’s so much more to learn. I still have the same passion and desire, but a lot of the hubris and puffed out chest I had as a newbie right out of college (and has allowed me to paint myself into a corner when I didn’t give a whit whether right or wrong) is gone. I’ve matured (well, at least I hope so). Make no mistake about it – I’m just as opinionated and outspoken, but I’ve found better outlets and ways of handling myself.

When we start out in any new situation – IT or not- we are all naive. Sure, we did some homework but what did we really know? Book smarts can’t compensate completely for life smarts. In IT terms, what you do in a lab in perfect conditions doesn’t always translate out to how things go down in production. Theory and practice are two different things.

The inspiration for this blog post was that I was watching ex-SNL and current “Parks and Recreation” (NBC) star Amy Poehler’s speech to the Harvard Class of 2011, and it has some very poignant lines which many in IT can really take to heart. It’s funny at times, as you would expect. Here’s the link:
http://www.youtube.com/watch?v=T7N_L_pu74k

Three of the best ones I wanted to specifically call out:

“You can’t do it alone. As you navigate through the rest of your life, be open to collaboration. Other people and other people’s ideas are often better than your own. Find a group of people who challenge and inspire you, spend a lot of time with them, and it will change your life.”
“Try to keep your mind open to possibilities, and your mouth closed to matters you don’t know about. Limit your always and nevers.”
“Listen. Say yes. Live in the moment. Make sure you play with people who have your back. Make big choices early and often.”

All are oh-so-true, espeically in IT. With technology, there are some always and nevers – for example, what is supported or how something works is what it is. Let’s get that out of the way up front.

However, the rest of that you really need to take to heart. I used to think that I could do it all. I was smart. I obviously knew better than everyone else. I was invincible. I was in my 20s – what else would I think? Now, to be fair, sometimes I was right. But this is where being green behind the ears works against you. As I said to start, what I know now is vastly more than I did when graduating in 1994 and moving into the working world, but with the years of experience I know that no matter how much I learn, how much I know, how much I may be an expert at something:

There’s always someone smarter and better than you.
Admit what you don’t know and be honest about it. You’ll step into some massive potholes if you don’t and represent yourself as knowing something when you really don’t. Biting off more than you can chew can damage your professional reputation and people can lose faith in your abilities if you misrepresent yourself.
Since technology is an ever-changing target, never assume what you know about old tech is true.
The biggest skill I have developed is the one to listen. When I was younger, I did a lot of talking. Not a lot of listening. Often times saying less is more, be it an e-mail or a conversation. You learn much more by keeping your ears (and eyes) open. As a consultant, there is no better skill than the one to be able to listen to people to be able to respond properly.
I’m always learning new stuff.

I had the faith in myself just under 4 years ago to go independent and be my own boss. Boy, that had a lot of risk associated with it. You have to be bold in your career and manage it yourself. Don’t let others do it for you. Find people you can trust and will be great mentors over the years. The people who were my best mentors challeneged me, pushed me hard. That meant in that challenge, there was friction because I didn’t want to try or do something. But they saw the potential in me to give me that encouragement or push in the right direction. I wouldn’t be here today if it were not for people like that. Of course, there’s a lot of self-motivation and inquisitiveness (is that even a word?) on my part, but we all need a bit of kick in the rear every now and then.

Brent Ozar (blog | Twitter) tweeted this today (which ironically was about the same time I was watching Amy’s speech):
“Days like today, heading off for #SQLCruise, it really hits home: I don’t have a job, I love my coworkers, and I’ve never been happier.”

I couldn’t agree more. Having a workplace and co-workers you like and trust goes a long way. Look, there will be people we don’t get along with no matter where we are. But you don’t want to be miserable day in and day out, week after week, year after year. That’s not good for the psyche or the soul. Amy’s comment about people having your back is really important for IT and speaks to that trust aspect I mention. Do you want to be in a place that constantly throws you under the bus? (I could make a DBA joke here, because we always get thrown under the bus, right?)

There are times when what you’re doing will be a solo effort and that’s all you need. But I would argue some of my biggest successes in life and in my career were team efforts. My contributions were important, but so were everyone else’s. It’s not all about you. I know, sometimes that’s even difficult for me to admit about myself, but learn to give credit where credit is due.

The best learning experiences will be your failures or scenarios where things did not go smoothly. IT is definitely not all sunshine, roses, puppy dogs, and kittens. Always understand there’s a best case – and a worst case – scenario. You will have disappointements along your journey, but the best folks pick themselves up, dust themselves off, and start all over again (to paraphrase that famous song). Sure, some failures can be a career limiting move, especially in IT. I’d be naive myself if I thought otherwise (and I’ve seen that played out). Try to avoid those.

EDIT: Another great quote from Dennis DeYoung on success found as part of this podcast: “People who are ambitious and driven are trying to please somebody who cannot be pleased.” I can attest to that. I’m my own worst critic.

Your career is going to be a journey with peaks and valleys. You’ll have some very high highs and some lows you wonder why you even bothered to get up that day. But you can take a lot of life lessons you use in your daily tasks and apply them to IT. Remember, only you know what’s best for you. Never lose your passion, remain true to yourself, have high morals, be patient, and good things will happen to you.

May 16, 2011

That Darn Speaking Feedback Thing Again

I just got my comments and scores back from SQLbits 8 for the two sessions I did during the conference itself (I know how the precon went, and that was great). Overall I’m fairly happy with my scores, especially given the fact that I was really sick that entire trip. Over the course of three days I delivered close to 12 or so hours of content with a really bad ear infection that made my head feel like it was swimming. Not exactly ideal, and definitely not recommended.

I’m not going to give scores, but address some comments made.

My first session on Friday (“So Many High Availability Options, So Little Time”) was in a room that was being partitioned after the keynote and due to the changeover, started very late. Most of the numbers were 7, 8, or 9 (out of 10). However, what I do wish people would do when they dislike your talk (or aspects of it), give some constructive feedback on WHY you don’t like it. I know I’m not perfect, but I can’t improve if you don’t give me any kind of constructive feedback.

My second session on Saturday (“Multi-Site Failover Clusters with SQL Server 2008 and Denali”) overall went really well, people seemed to like it, but a few felt that the expectations were not met … but yet failed to tell me why in most cases. I’ll take that ding because clearly I missed the boat somewhere and probably in my description. My guess is that those who were dissatisfied were for this reason, which someone wrote “I was expecting more details about Always-On option.”

While a fair comment, here’s my session description:
“Planning a failover clustering implementation in a single data center is fairly straightforward, but what happens when you want to span data centers? While most of the considerations are the same, introducing distance complicates matters. The story is also more complex with SQL Server 2008 and 2008 R2 than it will be with Denali. This session will cover how to plan and deploy a multi-site SQL Server 2008 or Denali failover cluster using Windows Server 2008 R2.”

Nowhere in my description do I talk about AlwaysOn Availability Groups, which is the new feature of SQL Server Denali. So I’m not sure how I didn’t meet an expectation that I didn’t set. I guess throwing Denali in the title and abstract people assumed I’d talk about AGs. I didn’t really address it much in the session except briefly since it wasn’t what core I was talking about. I also have to say this: there was not much that can be said about Denali publicly past CTP1. So even if I could talk about AlwaysOn AGs, I couldn’t demo too much or talk to things that were not in CTP1. That’s one reason I didn’t really do much about AGs in any session. So I apologize if you thought I’d do a deep dive on Denali AlwaysOn AGs. That was never my intent.

I do want to address this specific comment: “Probably inadvertently, but by requiring the attendants to ask questions later, the presenter discouraged the audience to seek clarifications and/or contributing with it own challenges/opinions, which I think is a significant part of the value we get from these sessions.” (And for the record, the person did give me good scores.)

I think this could be a difference between the US and Europe or the UK. In a pretty small session (training, under 10 people where it’s really intimate) I sometimes don’t mind taking a question or two during a session, but when you’ve only got 60 or 75 minutes including Q&A to do your deck and demos, a talk can get easily sidetracked or derailed if you allow questions during the main part of the session because often one question begs another. Asking people to ask questions during a formal Q&A at the end really isn’t strange at all. I’m sorry it may have inhibited this person or others, but the thing I absolutely hate to do which cheats everyone is to start taking questions and the next thing you know, you’ve got 10 minutes left to do your slides and finish your demos. As a speaker you run the risk of comments like this. This is a ding I’ll happily accept. I’ve had a few sessions go to the wire where I didn’t get it all in. It’s bad for you and it’s bad for me.

Here’s another: “Allan is excellent, would have perhaps liked a little more demo but overall a valuable session.” I know the commenter, but let me ask this: how many of you are schlepping variations of multi-site clusters around? Go ahead. I’ll wait. <insert Jeopardy theme here>

Needless to say, not many people have the hardware to set up a demo multi-site cluster back home let alone on a laptop (and I had two setups under Hyper-V – SQL 2K8 R2 and Denali CTP1). I’m not sure what else I could have shown. I know it would have been more impressive to roll in a bunch of hardware, but that just ain’t gonna happen. I would be curious to hear from this person what he thinks I could have shown that I didn’t.

This comment I can address a bit more: “Was expecting more demos and details on Denali rather than 2008R2” Like I said above, Denali at the time was (and still is as of the time of the writing of the blog) still at CTP1 for folks to download and try. Even if some other build was available, it would not be something that could be demonstrated publicly without the approval of Microsoft. Because the UI may change, I deliberately did not show the Setup screens in CTP1 for configuring a Denali multi-site cluster. I did talk about what you need to do at the Windows level, which in my opinion, is really the important stuff. You need to get Windows working. SQL is easier in comparison in this case because most customers today do not span data centers for clusters. Since current tech is what most people will implement for the forseeable future, I did focus a bit more on that BUT nearly everything from SQL Server 2008/R2 is applicable (with some differences that I pointed out specifically at the Windows failover cluster level) to Denali. At its core, they are very similar in that way. I didn’t think I needed to bop people over the head with that lesson. Maybe I need to be more explicit.

Again, I’ll ask the same question of this other commenter: what demos would you like to have seen with regards to Denali that I didn’t show? Once the instance is installed and clustered, it’s just a clustered instance of SQL be it SQL Serer 2008 R2 or Denali (and one of the things I made sure to point out were differences between SQL 2K8/R2 and Denali as it will appear in Failover Cluster Manager).

This one is a bit hard for me: “I was expecting a lot more from this session. Having read Allan’s book on SQL2005 HA. I was a big fan of Allan’s, I think I was probably expecting a lot more and was a bit gutted as I had to decide between Allan and Klaus on the training day. I decided to go to Klaus’s training day session. So it may be that I was bitter that could not attend Allan’s training day session. Thats my fault. It didnt help that Allan was a bit under the weather”

I hate to disappoint anyone. I’m human. I know I’ve been to concerts where I walked in hoping for the most amazing experience and then at the end feeling underwhelmed. I’m sorry that happened to you – and that’s sincere. I woul dbe curious to know (as with the others) what more you were expecting. Clearly I missed something if it came up more than once.

On the flip side, I love these

“I will be telling my boss and directors about this session – it was that important”. Awesome! THAT’S why I do what I do.
“One of the best presentations of the conference for me, highlighted a lot of the pitfalls and what needs to be considered as well as the expected features for the next release of SQL. Has also had a profound impact on my career path. Excellent, excellent session.” Wow. I’m blown away – I don’t set out to impact someone’s career path, but just wow.
“Nice guy who knows his stuff. Would like to see more about clustering at future SQLBits events.” Hope to get back there! Definitely some interesting stuff coming in Denali for HA.
“Allan was another great speaker who I’m sure could speak at length for many complete days about his experience. After spending the Wednesday in the HA pre-con, there was still more to learn in this session. Very approachable and
knowledgeable.” Thanks! I tried to minimize overlap.

So that was my SQLbits feedback in a nutshell. I’ve got six months between now and PASS (assuming I get a session) to come up with some real whiz-bang Denali demos. Better get cracking!

I also hope to get back to SQLbits in the future and right some of these wrongs.

May 10, 2011

Patching Clustered SQL Server Instances with SQL Server 2008 and SQL Server 2008 R2

One of the things you will eventually have to do to an instance of SQL Server is apply one or more of the following: a community update (CU), service pack (SP), or hotfix. I recently wrote a whitepaper entitled “Applying Updates to a Clustered Instance of SQL Server 2008 or SQL Server 2008 R2” which can be downloaded from this link. It has all the background information you need, and includes how to do things like check for reboots, script thge installs, etc. The scripts for the whitepaper can be downloaded here.

That whitepaper describes the process on how to approach patching clustered instances of SQL Server and patching, but in its examples, covers two more difficult examples because things can be complex and those are not described anywhere else (hence me wanting to write the paper). Read the whitepaper for the background information, but I thought in this blog post I would present the most simple case: one instance on a two-node cluster. All changes will be denoted in red italics.

In this example, to start, the instance is currently owned by Node A and is at SQL Server 2008 SP1 and you are looking to update the instance to SQL Server 2008 Sp2. Both nodes can potentially own the instance at this point.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	Yes	Yes	SP1	No	Yes	SP1

Table 1. Initial cluster state

To start the process, you will first have to remove the node which does not own the instance as a possible owner of the network name resource for the instance. This process and why you need to do it is described in the whitepaper.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	Yes	Yes	SP1	No	No	SP1

Table 2. Cluster state after removing Node B as a possible owner

Install SP2 on the node which is not currently a possible owner of the instance. Once that it is done, it will look like Table 3.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	Yes	Yes	SP1	No	No	SP2

Table 3. Cluster state after patching the instance on Node B

Add Node B back as a possible owner of the network name resource. The cluster will now look like Table 4.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	Yes	Yes	SP1	No	Yes	SP2

Table 4. Cluster state after adding Node B back as a possible owner

You now need to fail the instance over to Node B to start the upgrade of the instance itself as shown in Table 5. This will be the first time you would have to schedule an outage. You should do this in a controlled fashion as to minimize the impact to end users and applications.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	No	Yes	SP1	Yes	Yes	SP2

Table 5. Failing the instance over to Node B

The instance itself and the databases that were online will now be upgraded. The time it takes to do this will be fairly short. Technically you could allow connections after the failover, but you may want to wait until you are able to complete the patching of Node A and test full failover. That is up to you.

As with Node B, remove Node A as a possible owner of the instance’s network name as shown in Table 6.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	No	No	SP1	Yes	Yes	SP2

Table 6. Removing Node A as a possible owner of the instance

Install SP2 on Node A as shown in Table 7.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	No	No	SP2	Yes	Yes	SP2

Table 7. Installing SP2 on Node A

Once Node A has been patched, add it back as a possible owner as shown in Table 8.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	No	Yes	SP2	Yes	Yes	SP2

Table 8. Adding Node A back as a possible owner of the instance

Finally, you should fail the instance back to Node A as shown in Table 8 not because you want it to “live” there (your instance should be able to run anywhere in the cluster assuming you did capacity planning correctly), but because you want to ensure that there will be no problems after the installations are done and that the instance can run on all nodes. If you have allowed connections into the instance already, you will have to schedule an outage or do it during an already scheduled outage.

	Node A			Node B
	Owner	Possible Owner	Version	Owner	Possible Owner	Version
Instance	Yes	Yes	SP2	No	Yes	SP2

Table 9. Final state

May 10, 2011

Vote for (My) Sessions at PASS Summit 2011: The Power Is In Your Hands

Pass Summit 2011 is not that far away. Unlike years past where all the sessions were picked by PASS, this year, you can take an active part in what sessions may ultimately be chosen to be presented. PASS is asking you to vote for your preferred sessions by May 20, 2011 to help them decide. I think this is a great move. Of course, I hope you vote for all of my sessions, but vote nonetheless. I submitted five sessions: one precon, one of the new half day sessions, and three normal ones. This year, I decided to go big or go home (so to speak) with the topics and abstracts I submitted.

A Deep Dive into AlwaysOn: Failover Clustering and Availability Groups [400]
Session Category: Pre-conference Session (7 hours)
Session Track: Enterprise Database Administration and Deployment
When it comes to the availability landscape, Denali changes the game for SQL Server deployments. There many new enhancements and for traditional failover clustering deployments for instances such as flexible failover policies and multi-site clusters that can and will change the way you plan, deploy, and administer. In addition to the changes to failover clustering, there is a whole new feature: availability groups. Availability groups may enable some scenarios that you have had trouble with in the past, such as using a standby database for reporting and some maintenance. This full day will take a deep dive into these two features and tell you everything you need to know about not only the changes to failover clustering, but the proper way to plan, deploy, and administer availability groups to be successful from day one with Denali. If you do not want a basic “this is failover clustering” or a quick introduction to availability groups, nor do you want just a day of theory but want to see everything live (including combining clustered instances with availability groups to increase availability even more), this day is exactly what you need.

Deploying Highly Available Multi-Site SQL Server 2008 R2 or Denali Using Windows Server 2008 R2 [300]
Session Category: Regular Session (75 minutes)
Session Track: Enterprise Database Administration and Deployment
Planning a failover clustering implementation in a single data center is fairly straightforward, but what happens when you want to span data centers? Introducing distance complicates matters. What you have to do depends on the version of SQL Server you are deploying. Using the combination of Windows Server 2008 R2 with SQL Server Denali can potentially make things much easier. This session will discuss how to plan and deploy either a multi-site SQL Server 2008 R2 or a Denali failover clustering instance, the differences between the approaches, and introduce some new concepts like asymmetric storage and why you need to know about it. In addition to traditional instances, this session will also cover how to plan for a multi-site Denali AlwaysOn Availability Group implementation since it is based on Windows failover clustering even if you are not using a failover clustering instance. A real multi-site cluster will be demonstrated.

Implementing SQL Server Denali Using Windows Server Core [400]
Session Category: Regular Session (75 minutes)
Session Track: Enterprise Database Administration and Deployment
One of the most exciting new features in SQL Server Denali is support for Windows Server Core. Server Core is a minimal installation of Windows which has no GUI. For those who want to reduce patching and increase security, Server Core is the way to go. This session will cover the tips, tricks, and best practices for what you need to do to get SQL Server Denali up and running both for clusters and standalone implementations without any user interface.

Is That A Failover Cluster On Your Laptop? [300]
Session Category: Regular Session (75 minutes)
Session Track: Enterprise Database Administration and Deployment
Over the years, one of the most common questions I am asked is, “How can I configure a cluster on my machine?” Ask no more! Maybe you do not have a cluster anywhere other than production. Production should never be the first place you do certain tasks such as installing a service pack against a clustered instance of SQL Server. Or maybe you are looking to set up a development or QA environment, use such a configuration for demos while speaking, or you just want learn about clustering. This session is for you. Using virutalization gives you the ability to create a cluster without needing lots of expensive physical hardware and is something every DBA should have in their arsenal. This session will demystify the “magic” behind using configuring a cluster on your laptop or desktop using either Vmware or Hyper-V and discuss best practices, tips, and tracks. Demonstrations will include how to configure the virutal hardware via the virtual machines themselves, the domain controller, and the iSCSI storage solution. Note that this session will not be showing how to install SQL Server itself: it is focused on the things you would need to do at base level to be able to cluster both Windows and SQL Server. The discussion will cover multiple OSes and SQL Server versions (including Denali), and touch on how to approach simulating a “multi-site” failover cluster.

Windows Failover Clustering for the DBA [300]
Session Category: 1/2 Day Sessions (3.5 hours)
Session Track: Enterprise Database Administration and Deployment
While there are numerous resources to show you how to run through SQL Server Setup to deploy a clustered SQL Server instance, the most important thing DBAs must understand about the clustering process is absent: how to plan, deploy, and administer the underlying Windows failover cluster. It may be some other person’s job in your company to do this task, but it can make or break your installations. Starting from the ground up, this session is designed to get DBAs up to speed on all of the underlying components and concepts that will affect the performance, security, availability, and stability of their clustered SQL Server deployments up through SQL Server Denali. In addition to understanding and planning, how to properly administer and troubleshoot a Windows failover cluster from a SQL Server perspective will be covered. Did you ever want to know how to properly use the PowerShell cmdlets in Windows Server 2008 R2? Have a question on how to properly deploy DTC with Windows Server 2008 or Windows Server 2008 R2? Do you want to know how to properly configure Kerberos for clustered instances? Want to learn how to approach troubleshooting? Do you want to know why mount points matter and why things like CNOs and VCOs matter (and how to deal with them in different OUs)? These are just some of the questions that will be answered to help you demystify the Windows portion of clustering. Most things discussed will be demonstrated live.

April 24, 2011

SQLU Post #2: Remote Hosting, Availability, and You: (Im)Perfect Together?

I know, I know: the week is technically up. But I got caught up in customer work. So I’ve got another post or two still coming. This one I felt compelled to finish first due to some recent events, most notably the Amazon EC2 cloud outage earlier this week that as of now is still affecting some websites. If you look at that article, some known sites were affected (such as Foursquare).

Know What You’re Protecting … and Its Cost
I often talk to technical-focused people such as DBAs or SAN administrators who really have no idea about what the applications or databases they are hosting and administering are for. On the surface, it shouldn’t matter – right? Wrong.

Whether you’re new to a company and stepping in as a DBA/IT guy to administer some systems or part of the team planning the architecture and infrastructure for a new application or solution, you have to know a bunch of things both technical and non-technical. Of course, there’s the standard stuff like capacity and your overall availability goals, service level agreements, and recovery objectives. However, none of that matters nor does what you put in place if it’s wrong for what you’re trying to protect.

For example: if you are deploying a system that is for a hospital and the information in it is needed 24×7 to be able to make life or death situations (for example, being able to see if a patient is allergic to any medicines), that’s a whole different ballgame than a 9 to 5 shop who has an accounting application that needs to only be up and running in that timeframe. Death is not a consequence you want on your head (nor something a hospital would either; someone would get sued if it really is wrongful death).

I find many companies want to have either “one size fits all” solutions or totally “packaged” offerings for everything. Need to deploy? Use <insert thing here>. This approach will work for most of your deployments, but realize there may be some that don’t fit the mold and no matter how much you try to shoehorn the thing in, it’s really a case of square peg, round hole. The hospital one is a good example of that.

Even if it isn’t loss of life, attach some monetary or other value (such as productivity) on downtime. If the cost of the solution is cheaper than the outage cost, it should be a no-brainer to implement barring any technical issues in your organization. Let me say this: I know cost is an issue – especially in the economy we’re having now. I’m not ignoring it or saying it doesn’t matter. It does. I’m just pointing out that it cannot and should not be the only driving factor.

Who Do You Trust?
Control is an issue when it comes to availability. When you own and control all the pieces of the puzzle such as the servers and data center, you live and die by your own sword.

Having said that, we’re in a day and age where it’s not always cost-effective to build and maintain your own data center. Many companies use a hosting facility, and often times the people there are responsible for a good portion of the maintenance. The cloud is just another abstraction of this if you really think about it. However, with the cloud you arguably giving up even more control and will have less control over many aspects of deployment and administration since there are more layers of abstraction. Some cloud providers even do all the patching. (Disclaimer: I’m not anti-cloud. But I don’t feel it’s quite ready for prime-time, mission critical applications for MOST customers. It may work for some, and I will say what I did with virtualization a few years ago: watch this space as it will affect you in one way or another at some point.)

The key thing when you outsource (and I don’t necessarily mean to some center halfway across the world, although what I’m going to say applies there, too) that you have to remember is that by placing your web server/database/whatever there, it becomes and extension of you. By that, I mean where it’s hosted has to meet the requirements documented as if you were controlling everything in your own data center. When you outsource any of the work or responsibility, you need to have a very comfortable level of trust that those responsible for meeting the availability YOU agreed to can follow through (in addition to ensuring things like security and performance; I’m just focusing on uptime and/or the lack of downtime). If they can’t, it’s your neck on the line.

Years ago I was working onsite at a customer with many others during a disaster they were experiencing. Their servers were at a location where some people had to fly to get there to be able to physically do some work on the servers. No big deal right? If only that were true. When they got there in the evening, the place was closed up tighter than a drum – no 24×7 access. That was not the time to find that out. So they came back in the morning only to find they could go in one at a time. Again, not a surprise you want to discover in server down situations. This example demonstrates what happens when you make assumptions and don’t ask the right questions of your service providers.

Another tip: make sure the company you are outsourcing to supports your core business hours whether they are 24×7 or 9 to 5. It doesn’t matter if they are next door or three continents away, if you can’t contact them and expect a response to meet your SLAs, what’s the point? I can remember transitioning some things to a remote support center half a world away on one engagement and whoever set that up never really told them they needed to work our hours. Huh? So let me get this straight: we were in the USA, but couldn’t get any support until after 8PM at night? I don’t think so.

Plan B – Have One
A lot of my customers over the years never really put disaster recovery in place. They put some primary form of availability along with some backups, and that was that. Disaster recovery is an additional expense (and insurance policy) that often gets cut due to things like time and budget. I always say it’s a matter if when, not if, something will happen. Hardware failure, natural disaster, user fat fingering data – whatever happens, there will be some scenario that can and will take you down. You may not have been able to plan for it nor did your primary form of availability cover it. Welcome to disaster recovery.

You can’t just have redundant systems in one location or possibly geographic area. If an earthquake hits Los Angeles and both of your data centers are there, do the math on that. Lose them both, possibly lose everything. Again, know what you’re protecting against and deal with it properly.

Even if your Plan B is just to have backups, do that. It’s better than nothing. For the really paranoid, you may even want to have Plans C, D, E, F, or more. No, I’m not kidding. It’s a lot of work, but if it saves your company’s proverbial bacon, who looks like a genius or hero? You.

Get To The Point, Will Ya?
So what inspired this post other than the SQLU HA/DR week? A Tweet I read pointed this forum posting for Amazon Web Services EC. I’ll just repost the first post:
Life of our patients is at stake – I am desperately asking you to contact
Posted on: Apr 22, 2011 11:20 PM

Sorry, I could not get through in any other way

We are a monitoring company and are monitoring hundreds of cardiac patients at home.
We were unable to see their ECG signals since 21st of April

Could you please contact us?
Our account number is: xxxx-xxxx-xxxx
Our servers IDs:

i-xxxxxxxx
i-xxxxxxxx
i-xxxxxxxx

Or please let me know how can I contact you more ditectly.
Thank you
————-

Oy vey! Where do I begin?

First and foremost: YOU NEVER WANT TO BE IN THIS GUY’S SHOES. EVER. Got it? Good. The guy (and I do feel bad for him and his customers; I truly hope no one died) took a proverbial beating in that thread and it was kinda justified. If you know that the outcome of you losing the ability to monitor could be someone’s death, have some sort of redundant provider and/or backup solution. Even if that solution is picking up the phone to check on those needing monitoring. Do something. I don’t even want to think about the legal implications of all of this in the unfortunate event someone did pass away (and I hope no one did) and as some brought up in that thread, compliance with whatever laws in your municipality/state/country/zone/whatever.

I’m not absolving Amazon completely here: they promised a service level the customer relied on and didn’t deliver. As a DBA or IT person, don’t do that either. It’s better to underpromise and overdeliver than the opposite. However, as I said above, even with Amazon’s promises, we’re talking life and death. We’re not talking about losing some fanboy’s music site to his favorite band here for a few days. Amazon I assume was trying to get things online, but lack of communication only makes things worse. If you want a good relationship with your customers, tell them the truth no matter how much it hurts. You may even take a ding or ten. We all have at some point in our career.

One quick thing I’d like to add here: if you are in that guy’s shoes, there’s a very real chance you may need to update your resume sooner rather than later with a blunder like this. I’m not unsympathetic towards that original poster at all; quite the contrary. But it’s major oversights like that which will get you fired.

This example underscores most of the points I made in this blog post:
1. Know why you’re implementing something. Possible death is a whole different level of planning that needs appropriate backup plans. You won’t think of every scenario (see: the Japanese nuclear plant situation), but if you cover the most common ones, you won’t be like this guy. They had no Plan B (or C or D). It was Plan A or bust. It may have seemed like a perfect idea to stick this monitoring application in the cloud, but boy did it bite them in the rump when it failed to deliver the uptime promised. Bet they get some redundancy after this.

2. Communication is key. If you don’t want the mob showing up at your doorstep with pitchforks and torches, be concise, be accurate, and don’t put people off. Bad news is better than no news for hours or days on end.

3. Ask the right questions of your hosting/services/cloud providers if you are going to take that leap of faith and not do the work yourself. If they do not meet your standards, even if what you’re looking at from a cost perspective is better, it is most likely NOT the right solution for you. Cost cannot be the only factor. Again, I only point you to this poor soul from the forum posting: things look great on paper and may even seem like a bargain, but are they really?

« First ‹ Prev 45 46 474849 50 51 Next ›Last »

Categories