SQL Server 2005 Archives

March 12, 2013

Creating a Clustered DTC for SQL Server Redux

Last year, I wrote a blog post entitled “How to Properly Configure DTC for Clustered Instances of SQL Server with Windows Server 2008 R2”. Today, I was helping a customer and found a few things. I’ve since updated that original post to reflect these findings and to fix a few things. I also took Windows Server 2008 R2 out of the title and the post’s name is now “How to Properly Configure DTC for Clustered Instances of SQL Server (Revised)” since it applies to Windows Server 2012 as well. The URL is the same as the old one, and that does reference W2K8 R2.

As was originally written, that article was assuming a new, greenfield installation. Not everyone is looking to create DTC with a brand new installation, and I realized that there were some gotchas I needed to outline.when you are already up in production. I also fixed a few things in the script.

1. In the script as it was written, one step added the resource as a dependency to SQL Server. That’s all well and good (more on that in a bit), but this had two issues with a non-greenfield implementation:

When you make sure that DTC has the right permissions (“Step Two – Enable Network Access for the Newly Created DTC”), it needs to take it offline. When you add DTC as a dependency to SQL Server, it will take it offline as well. This is fine for a new installation, but not for one that is already up and running. I revised the script to NOT do this and make it a separate thing later.
It doesn’t happen all the time, but we saw it on their installation and I also saw it once on my test WSFC (but not again – so it seems to be phantom). SQL Server Agent went offline. See Figure 1. That’s definitely not good.This is apparently a timing issue – DTC needs to be up before SQL Server. I explain all of this in a bit more detail in the revised post linked above.

Figure 1. SQL Server Agent down

If DTC goes down, it will cause a SQL Server failover by default. You may or may not want this behavior. I’ve revised the original blog post with more information on this.

2. I forgot to put one important line in the script – namely moving the disk resource into the group with SQL Server! The original script did have a gotcha line in there, but I made sure it was done in this version.

My customer was trying to fix a critical path issue that needed DTC up and running for their application, so they wound up putting DTC in its own resource group which is a valid option still in Windows Server 2008, 2008 R2, and 2012. They only have one instance installed in the WIndows Server failover cluster, so they can ensure DTC lives where the SQL Server instance is all the time to reduce network traffic. This saved them from another possible outage to ensure DTC was up before SQL.

Hope this helps some of you out there …

March 12, 2013

How to Properly Configure DTC for Clustered Instances of SQL Server (Revised)

Configuring the Microsoft Distributed Transaction Coordinator (DTC) for clustered SQL Server instances (FCIs) with Windows Server 2008 and later has been a confusing topic for many. The “definitive” word on the subject for awhile has been both of Cindy Gross’ blog posts ( “Do I need DTC for my SQL Server?” and “How to configure DTC for SQL Server in a Windows 2008 cluster”). In my opinion, with W2K8+, you really should create one DTC for each FCI and put it in with the SQL Server instance. This way it always lives on the node where the instance is currently running.

Having said that, what Cindy wrote doesn’t always work. In fact, the only truly reliable way some of us have found was talked about by my friend and fellow Cluster MVP (also a big SQL guy), Mike Steineke (blog | Twitter) in a post last week entitled “Clustered DTC and Multiple SQL Instances” where he shows how he had to configure DTC. Part of this is ensuring that even though DTC is in the group with SQL, it still gets its own name and IP address; using the SQL Server name can be problematic. To get this done, you really need to script it. I’ve done the work for you. Easy!

This updatated post stems from the blog post “Creating a Clustered DTC for SQL Server Redux“.

A sign that you need to use DTC for a given FCI can be found right in the SQL Server error log. If SQL Server cannot connect to a clustered DTC, you will see an error message like the following:

2013-03-12 10:23:51.670 spid1001 QueryInterface failed for “DTC_GET_TRANSACTION_MANAGER_EX::ITransactionDispenser”:0x8007138f(The cluster resource could not be found.).
2013-03-12 10:23:51.680 spid1001 QueryInterface failed for “ITransactionDispenser”: 0x8007138f(The cluster resource could not be found.).

This blog post now applies to Windows Server 2008, 2008 R2, and 2012 as well as SQL Server 2005, 2008, 2008 R2, and 2012 since depending on the OS you’re running, you may have a little of each …

Step One – Create the Clustered DTC in the Resource Group with the FCI (revised 3/12/2013)

The script below I actually created today and is based on my original blog post I wrote back in 2009 for Windows Server 2008 and takes into account what’s in Mike’s post. I actually used this script on a customer installation, so this isn’t something I wrote just for the hell of it. All you have to do is modify the variables up front and run the script. If you don’t know the name of some of the resources, you can use the PowerShell cmdlet Get-ClusterResource to show a list of everything.

For this update, instead of embedding the code, you can download the script from here. Much easier!

A sample execution is shown in Figure 1.

Figure 1. Running the DTC creation script

I also tested this script in Windows Server 2012 and it works just fine.

Step Two – Enable Network Access for the Newly Created DTC (revised 3/12/2013)

Once DTC is created, you must enable network access. This is where if you did things another way (i.e. not the way shown in this post), there’s a very high probability that when you went to go restart DTC (Step 11 below),the process may decide to step out for lunch and then stay away for a permanent vacation.

The steps below are all GUI. Nic Cain (Twitter | Blog) wrote up this blog post on how to configure this using PowerShell if you would prefer to script things.

From the Start Menu, select Administrative Tools, and then Component Services (or just run DcomCnfg).
Under the Console Root folder, expand Component Services.
Expand Computers.
Expand Distributed Transaction Coordinator.
Expand Clustered DTCs. You should now see something that looks like Figure 2.
Figure 2. Newly created DTC in Component Services
Right click the clustered DTC you just created and select Properties as shown in Figure 3
Figure 3. Opening the Properties of the DTC
Select the Security Tab
Under Security Settings, check the “Network DTC Access” box. Under Transaction Manager Communication, check both “Allow Inbound” and “Allow Outbound”. You should now see what is shown in Figure 4. If you need any other options (you know your app better than me), add them. Click OK when done.
Figure 4. DTC enabled for network access
You will now be prompted to restart DTC as shown in Figure 5. If you are doing this against a production instance of SQL Server and you have added the DTC resource as a dependency of SQL Server, it will take SQL Server offline as well. If you cannot do this, select No, but you cannot really use DTC properly until you restart it. To get around this, remove DTC as a dependency of SQL Sevrer, do the restart, and then add it back. Click Yes to restart.
Figure 5. Prompting for the restart of DTC
Click OK at the dialog shown in Figure 6 which denotes a successful restart.

Figure 6. Confirmation of DTC’s restart

Step Three – Testing DTC (added 3/12/2013)

Microsoft has a tool that will create a dummy transaction and can be found in KB 293799. While the best test is always your application, this will at least see if DTC is working. It requires an ODBC DSN to work.

Start Data Sources (ODBC) from Administrative Tools. Note that under Windows Server 2012, you will specifically have a 32-bit and 64-bit version. Run the 32-bit.
Select the User DSN tab.
Click Add.
On the Create New Data Source dialog, select SQL Server Native Client X, where X is the version of SQL Server. In Figure 7, it is SQL Server 2012, so it has a version of 11.0. Click Finish.
Figure 7. Selecting the version of SQL Server
On the Create a New Data Source to SQL Server dialog which will look similar to the one in Figure 8, enter a name for your DSN and for Server, enter the name of the clustered instance of SQL Server. Click Finish.
Figure 8. DSN creation
On the ODBC Microsoft SQL Server Setup dialog, click Test Data Source. An example is shown in Figure 9.
Figure 9. DSN created
On a dialog similar to the one shown in Figure 10, if the test was successful, you should see the right output. If you don’t, you will see an error. Click OK.
Figure 10. DSN works properly
Run dtctester using the syntax dtctester DSN username password, where DSN is the name of the DSN you just created, username is a valid user in SQL Server, and password is the password for that user. A sample execution is shown in Figure 11.
Figure 11. Runing dtctester

WARNING Do not create a System DSN. If you do, you may see an error similar to the one shown in Figure 12. This error stems from the fact that dtctester is a 32-bit program and the System DSN created is 64-bit.

Figure 12. dtctester error

Step Four – Optional Configuration Steps (added 3/12/2013)

The steps in this section are optional. Read to see if they apply to you.

xSetting the DTC Resources to Not Cause a SQL Server Failover

There are four resources associated with DTC: name, IP, disk, and of course, DTC. If you right click on any one of those and select the Policies tab, you will see what is in Figure 13.

Figure 13. Policies for the DTC Network Name

Notice the option “If restart is unsuccessful, fail over all resources in this service or application.” What this means is that if any one of those resource fails and cannot be restarted, it will cause SQL Server to fail and move to another node. If you are not sure if DTC is necessary but are creating it anyway OR you are OK with DTC failing but SQL still being up and living with whatever not working that DTC affects, deselect this option.

Adding DTC as a Dependency of SQL Server

In the testing my friend Mike did, the only way he could get things to work was to ensure that DTC was up before SQL Server. The only surefire way to do this is to add DTC as a dependency to SQL Server (see the previous section for possible implications of doing this). One of the problems that may occur and was noted in the blog post “” is that there is a chance that SQL Server Agent may go offline. We think this is possibly related to another bug that was fixed, but what is happening is that SQL Server Agent is trying to figure out the name of the instance it is using. If it gets confused, it will go offline. It may start, but it also may not work correctly. The only way it seems to work around this is to make sure that DTC is up before SQL Server starts, hence adding it as a dependency.

Once you get DTC up and running in the group with SQL Server, test your application against it and check in the SQL Server error log as noted above. If you see no DTC errors and everything looks fine in the app, you’re golden. You shouldn’t have to restart SQL Server to take advantage of the newly mapped DTC, but without an application right now, I can’t say this to be 100% true.

Adding a dependency to a resource in Windows Server 2008 and later does not require any downtime.

To add DTC as a dependency of the SQL Server resource in Failover Cluster Manager:

Right click on the SQL Server resource and select Properties.
Select the Dependencies tab.
Click Insert.
On the new line, from the dropdown select the DTC resource you created. An example is shown in Figure 14. If the line is at the bottom, make sure that the AND dependency is selected.
Figure 14. Adding DTC as a dependency
Click OK when done.

To do this in PowerShell, execute the following:

Add-ClusterResourceDependency SQLResourceName DTCResourceName -Cluster WSFCName

A sample execution is shown in Figure 15 along with verifying that it was done correctly.

Figure 15. Creating and verifying the dependency in PowerShell

Mapping DTC to a Specific FCI

Right now I do not have an application to test specifically the combo of DTC and the FCI in the same group that all is well without adding the DTC resource as a dependency. Based on what we know, it may not work. One thing which may solve that issue and/or if you want to truly be anal about things, DTC provides you with a way to specifically map an instance of DTC to only be used with a specific application or service such as a given SQL Server instance. This is done via command line. If you really want to ensure that DTC is mapped to the FCI, you use the tmMappingSet option of the command line for DTC.

NOTE This particular aspect/option most likely will NOT be necessary but I’m putting it in here for the sake of being complete.

Enter the command msdtc -tmMappingView * as shown in Figure X. If nothing is mapped, you will see the output in Figure 16.
Figure 16. No DTC mappings exist
To map the newly created DTC to a specific FCI, use the command: msdtc -tmMappingView -name DTCname -service SQLServerService -clusterResourceName SQLResourceInWSFC, where -service would be MSSQLSERVER for a default instance or MSSQL$NMDINS where NMDINS is the part after the slash and -ClusterResourceName is the name of the SQL Sevrer resource in the Windows Server failover cluster (WSFC). For a default instance it would just be SQL Server, and for a named instance SQL Sevrer (NMDINS). A sample execution for a named instance is shown in Figure 17 and you should see Figure 18 if it is done correctly.
Figure 17. Mapping DTC to an FCI

Figure 18. DTC mapped successfully
To verify things are correct, again enter msdtc -tmMappingView *. You shold now see something similar to Figure 19.
Figure 19. Successful mapping output

August 23, 2012

When Is It Time To Stop Using a Deprecated Feature?

Twitter conversations can sometimes lead to interesting things. Jonathan Kehayias (Blog | Twitter) and I were going back and forth about something, and in some cases, how we differ on how we teach (some of that was based on teaching or not teaching a deprecated feature). It was a very healthy discussion if you ask me. I really respect Jon and he’s one of the smartest folks on the planet when it comes to SQL Server in nearly every respect. I don’t say that lightly. Paul, Kim, and the rest of the crew over at SQLskills are very lucky to have him on their team.

Be it Windows, SQL, SharePoint, whatever – all of these products at some point release a new version and as part of the lifecycle of that product, some features are deprecated. A good example is that database mirroring has been deprecated as of SQL Server 2012. It’s still in the product, but its development is over and it will disappear from the product in a few versions. Another easy example for me to mention (and what sparked the debate between Jon and I) was the use of the aforementioned cluster.exe in Windows, which was deprecated in Windows Server 2008 R2 and is technically gone in Windows Server 2012 (disclaimer: you can add it back by enabling a certain feature, but it hasn’t seen any improvement or development since Windows Server 2008; more on cluster.exe in a bit). In the immortal words of Hall and Oates, “She’s Gone” (following the Daryl Hall theme from my last post; I was challenged to use “Rich Girl”, but this was more of a natural fit – sorry Rick!).

http://www.youtube.com/watch?v=bnVXIUyshng

That begs the question: when do you stop actively using/teaching/recommending/etc. a certain feature you either know has been deprecated or will be going away (but may not be public knowledge although you have good NDA information which you really can’t share)? The answer, not surprisingly, is “it depends”.

Let’s take the case of my good friend cluster.exe who I’ve lived with for the better part of 10+ years. At this point, I’m not seeing any new deployments of SQL Server 2008 (RTM branch or R2) on the original Windows Server 2008 (no R2); it’s all basically Windows Server 2008 R2 with SP1. W2K8 R2 introduced the first round of PowerShell cmdlets for Windows Server failover clustering (WSFC). When I wrote my SQL Server 2008 book, W2K8 R2 was just being wrapped up so I documented both. As time has marched on since 2008/9, not only have I personally used cluster.exe less, when I teach my classes or do presentations, I don’t even bother showing it anymore. Why? In 2012, to me it makes no sense to show something that is deprecated and outdated, and is gone in Windows Server 2012. If you’re using Windows Server 2008 R2, you really should be using PowerShell to administer your clusters (one man’s opinion, of course; feel free to disagree). However, there are still exceptions to that rule. For one specific thing in Windows Server 2008 R2, there is no PowerShell equivalent (it’s basically related to availability groups in SQL Server 2012). This is fixed in Windows Server 2012. Similarly, some things that have a PowerShell cmdlet (Test-Cluster), have no cluster.exe equivalent. The pros outweigh the cons for me of not showing or teaching cluster.exe these days. Would I if someone asked? Sure. I’m not so stuck on not showing it; I just think it’s better to learn the stuff you can use now AND in the future when you want to go to Windows Server 2012.

The story is not so cut and dry with a feature like database mirroring. Unlike Windows which really makes it clear cut you need to move on – and quickly, SQL Server keeps a feature around and intact for at least two versions before it goes away (another really good example was DB-Lib). That could possibly translate into almost 6 – 10 years of shelf life, giving you a false sense of security. Sometimes there’s a clear cut feature that replaces it. In the case of database mirroring, a lot of people use it now. That’s one factor, so it makes the announcement that much harder. But the real crux of the problem as I see it is a case of the haves and have-nots. If you are using Standard Edition of SQL Server, Microsoft is recommending log shipping as your replacement. For Enterprise (now Datacenter) Edition, it’s availability groups. AGs are a great evolution of what was started with DBM, but with the new dependence on WSFC for deploying AGs, it may be a non-starter for some. But people who want to upgrade to SQL Server 2012 or whatever vNext (or vNextNext) may be, they need to start planning a roadmap otherwise it makes it that eventual upgrade harder because your back will be up against the proverbial wall. There’s no easy answer here, but it’s a conversation I’m having with a lot of my customers.

When it comes to having NDA information on something to be announced in the future, I can never spill the beans. Sorry. If I do, that puts me at risk with whoever I have the NDA in place with (and that comes with all kinds of legal problems for me if I do). I love my customers, but it’s not worth putting me out of business by letting something slip. That said, I can still steer customers in the right direction without them even getting a hint that there may be an underlying and different reason I am doing so.

I’ll leave you with this one final thought: as much as deprecated features are important to consider in your planning efforts, don’t let it become an obsession, either. DBM is a great example – it’s not going away tomorrow. If you’re using it now, that’s OK. A deprecation announcement didn’t mean to stop using it this very minute; just don’t get complacent about making a decision, either. A few years goes by in the blink of an eye when it comes to IT (just look at the people still running SQL Server 2000 for whatever valid reason and are stuck supporting it). Don’t be that guy or gal keeping the lights on for the few instances requiring DBM for the next 15 years. Be smart, make the right choices, and stay on a supportability path.

July 11, 2012

Anti-Virus + Windows Failover Clustering + SQL FCIs = Bad

Recently, I helped a customer plan and configure a three node multiple instance failover cluster. The Windows failover cluster (WSFC) had four instances of SQL Server (3 x 2008 R2 with SP1 and 1 x 2012 with CU2). However, when they went to go enable their anti-virus on the cluster nodes (Symantec if you must know) and set the exclusions I document in my book, things went south. Pretty much everything went offline. They were able to get pretty much everything up but SQL Server Agent for one instance just wasn’t working. I just got off a troubleshooting call with them and I think we fixed that (I’ll know for sure later once they do a few other things). If you think this is an uncommon event, it’s not as evidenced by this quick Twitter exchange I had:

With most customers, anti-virus is a standard piece of kit to install on all servers no matter what. The base rationale makes sense. I mean, who wouldn’t want to protect your servers from potentially bad stuff, right? Here’s where I lose the plot a little bit. First and foremost, your production servers are not desktops. Joe and Jane Administrator should not be surfing the ‘net from there. Assuming that is the case, the risk of getting things like malware is probably really small. If they’re doing that, you’ve got to fix your processes and lock things down. I’m not saying it can’t happen, but it shouldn’t. If you’re on the server and need to use the Internet to go read a support KB, that’s perfectly acceptable usage. But to check their Hotmail/gmail/match.com/watch porn/personal stuff? Hell no. Most admin work should be done from workstations with remote tools – you shouldn’t have to log onto or RDP into the server for most tasks. That drives the risk up of something going wrong.

The other aspect of this is really file-based. Think about it – we get documents and download files from various places. That’s arguably how a lot of the whole malware thing started. A production SQL Server server (that always looks weird to me) should not be a file repository whether it is clustered or not. I will always say that in most cases, a production boxes with SQL Server on it should be dedicated for that purpose only. I know people want to maximize usage and you can create a clustered file share, but I’m assuming here that in most cases where you deploy FCIs, it’s on dedicated WSFCs just for SQL Server. If that is the case, why would you need anti-virus especially if you’re monitoring to ensure people are not doing rogue thins on the cluster. Again, it’s easy to minimize your risk exposure with having good processes. Imagine that.

If your company is still not convinced by these rational arguments, have a quick gander at these KB articles from Microsoft:

Pay particular attention to this bit in 250355:

You can run antivirus software on a SQL Server cluster. However, you must make sure that the antivirus software is cluster-aware. Contact your antivirus software vendor about cluster-aware versions and interoperability.

Not all anti-virus programs are created equal. So make sure if you need to install anti-virus, you get the right thing. If what you have is not cluster-aware, caveat emptor.

Let me add to this cautionary tale: back in the SQL Server 2000/2005 days on Windows Server 2003, I was working with another company. We got things up and running on the cluster with no issue. I told the team responsible for anti-virus how to configure it for the clustered instances of SQL; not much more I could do. Right after we went into production, a day or two in, my databases were marked as Suspect. I did a bit of digging and the smoking gun wasn’t hard to find: right in the Event Log, it was clear as day – a virus scan kicked off at about 11AM and minutes later my databases decided to take a dirt nap from which I had to restore them from backup. They were permanently corrupted due to the scan kicking off. The admins swore up and down it wasn’t them (even though I had the proof). Things were fine for one day and I bet you can guess what happened the next day at the same time. I was livid and took this up the chain. The problem eventually got fixed, but it should have never happened to begin with. That was largely a political issue at the end of the day, but it also speaks to the fact not everyone knows about clusters and SQL Server.

This is a major reason you set filters for things like the the .mdf, .ndf, and .ldf files even on standalone SQL Servers. This is magnified on clusters where if you have an instance that fails over to another node that now takes ownership of the disks associated with that instance, the anti-virus program is going to now see them and say, “Awesome! New files to scan.” That will not only hold up your instance starting, but see my previous paragraph – you may get more than you bargained for. Not only may you blow SLAs, you have the potential of hosing databases. Just what every DBA needs.

UPDATE: Also check out Allen Kinsel’s (Twitter) blog post SQL Server 2008 & IPV6 vs Symantec. It shows where Symantec does something even more fun: endpoint protection. As my friend Nic Cain (blog | Twitter) pointed out to me, endpoint protection can be even worse than anti-virus on a clustered configuration – especially since IPv6 is enabled by default on Windows Server 2008 (and up) and you shouldn’t disable it. Allen’s post is probably why my customer’s cluster went south to begin with, and SQL Server Agent for the one instance was just a casualty.

My recommendation to you: DON’T PUT ANTI-VIRUS ON SERVERS WITH SQL SERVER INSTALLED ON THEM AND ESPECIALLY IF THEY ARE CLUSTERS. You’ll be much, much happier.

June 28, 2012

Maintaining Quorum During Windows Patching and Updating

I’ve been working with a customer to deploy their new Windows Server 2008 R2 SP1-based failover cluster that will have bot SQL Server 2008 R2 and SQL Server 2012 instances. One thing I always talk about in my classes as well as the presentations I focus on patching is this: you must maintain quorum during the patching and updating cycle. Quorum is one of the most important considerations in general for your clustered deployments (whether you’re implementing traditional failover clustering instances or availability groups), and it’s something that quite frankly, I find there’s a real lack of knowledge out there. It’s so important it’s most likely getting a whole chapter in my upcoming book Mission Critical SQL Server 2012.

I have talked to customers that have had SQL Server outages because their Windows guys when patching the underlying nodes for the Windows Server failover cluster (WSFC) rebooted too many at once, so there were not enough voters us, thus bringing the WSFC (and SQL) down when it wasn’t expected. This is NOT a scenario you want. While in theory it’s easy to control and keep track of when things reboot, this little gremlin in Figure 1 pops up after a little while reminding you it needs to be done:

Figure 1. Reboot needed warning

That’s all well and good, but the unskilled administrator who happens to see that message if they have to log onto the server may think nothing of it and just click Restart now. Yikes! The value of the reminder can be altered as shown in Figure 2, but you still need to worry about the reboot being done.

Figure 2. Changing the reminder value

There’s another dark underbelly I’ve seen these dialog boxes associated with: automatic updating of Windows via Windows Update. That is a much scarier situation because that means your servers could be updating 24×7 on their own, and some of those updates may even force reboots. Check your settings! I still see production servers at client sites that have automatic updating enabled. If they are, you need to have a serious discussion with your server admins as to why this is being done. Automatic updating is not in line with mission critical. Yes, it’s important to have a patch management strategy that keeps you up-to-date, but doing it smartly is the key. Not blindly.

12 3

Popular Posts

Categories