By: Allan Hirt on April 15, 2016 in .NET Framework 3.51, .NET Framework 4.6, .NET Framework 4.6.1, Setup, SQL Server 2016 | No Comments
Hello everyone. I’ve been pretty heads down with a lot of customer work, teaching my Mission Critical SQL Server class (with Philly on the way in June), preparing for three upcoming conferences in Europe (including three pre- or post-conference sessions, one of which will have updated labs), and working on the book. Yes, I’ve really been working on it and you’ll see some things shortly. You may have noticed our site got hacked, and I’ll probably do a post about that at a later date, so that’s taken up some time as well. I did lose the post on Distributed Availability Groups which I will rewrite soon. Needless to say, blogging has not been my Number 1 priority. I would rather put that energy into my customers, the book ,and making sure my upcoming speaking appearances go off without a hitch.
Having said that, SQL Server 2016 RC3 was publicly released today and I felt this was important to put out there. There is a major change that actually came in with RC2 that no one seemed to notice. I already talked about how from a server perspective, SQL Server 2016 is 64-bit only and I had a lot to do with that. That was a massive change to SQL Server. This next one is also just as big. Many of us (not just me) have been asking to remove the reliance on the older version of .NET Framework (3.5x), and it’s finally happened. Starting with SQL Server 2016, it is no longer a requirement.
Let me give you a little bit of backstory. For every version of SQL Server from 2008 through 2014, SQL Server required that .NET Framework 3.5x was installed (or enabled as the case may be depending on your version of Windows Server). How to enable .NET Framework 3.5x on Window Server 2012 or later is not apparent to everyone. I even wrote two blog posts (one and two) on how to get this task done. There is also a lot of confusion on why the older .NET Framework version is needed if the later version in the 4.x tree is installed. The short answer is that 3.x (which is the same branch as 2.x and 1.x) is different than the 4.x branch. Parts of the engine still relied on the old .NET Framework so the SQL Server development team could not just rip it out easily.
Some of us knew this was coming, and when I pinged someone around the time of RC1, I was told it would definitely be in RTM. We got it a little bit earlier. This was some hard work on the dev team’s part. If you think this is a hoax, it is not. Figure 1 shows all of the SQL Server features selected but look at the highlighted section – the only dependency is .NET Framework 4.6.
Figure 1. All SQL Server features selected, no dependency
In Figure 2, the Command Prompt window is showing sqlcmd running a SELECT @@VERSION command with its output (RC3), and imposed on it, the output of showing that the older .NET Framework is not installed, yet SQL Server 2016 is running. Huzzah!
Figure 2. SQL Server RC3, no .NET Framework 3.5x
Couple that with the other changes to Setup introduced in the RCs by splitting out the tools install, installing a SQL Server instance is a more streamlined process … and much, much quicker. Getting an instance up and running is almost a pleasant experience. I have my issues with the new tools install, but I’ll take that as a first world problem.
By: Allan Hirt on December 28, 2015 in Advice, Mission Critical SQL Server | No Comments
As we wind down in 2015 and head towards 2016, a lot of us look back and take stock of both the good and bad. 2015 has been a pretty good year, but this is not “that” blog post.
In 2014, we got hit with a lot of snow in the Boston area. According to that report, in February alone we got nearly 65 inches. That is just about as tall as me (yes, I’m not very tall ha!). It seemed like it snowed every day for about two months. There was nowhere to put it. Now, those of you that know me know that I don’t mind cold weather or snow, but when you are getting hit day after day, at some point enough is enough. There is nowhere else to put it.
Anyway, what I didn’t talk about with many people is that I slipped twice on ice while shoveling, once landing myself in the emergency room after one of the February storms. That was the second fall. That second fall put a chain of events in place where basically I have not felt right for quite a long time. Things bothered me enough that I went to the doctor in the summer of 2014. I was dealing with the effects of falling until late spring/early summer of this year. It got to the point where I was in quite a bit of discomfort and pain. It was literally uncomfortable to stand, sit, or lie down; no position worked. To be honest, for the better part of 2015, I’ve been miserable.
Unfortunately, my schedule since going to the doctor in 2014 has been pretty crazy. I have been on the road a lot, and I did my best to not make it show, but I was suffering quite a bit. It was a challenge to be on planes for 6+ hours, and even worse, I basically could not sit still while working. When it really was exacerbated to the point of it affecting work, I knew it was time to take care of this problem. I’ve tried not to let it show, but I can say there have been private moments where my pain was excruciating to the point of tears. Now, imagine standing up and teaching a class for four days or doing a preconference session at PASS Summit. By the time Summit rolled around, there were times I had to sit. Summit Squared (PASS and MVP) was my final breaking point; I had to do something. It was hard to work this way, let alone do anything normal. Having to take things like Aleve just to get through the day is not my modus operandi. Some of you may have noticed me walking with a slight limp at one of the Summits. Now you know why.
This is the first time anywhere I’m talking about any of this outside of a small circle of friends and family. I’m not looking for sympathy. There’s a point to all of this. As an aside, those of you who have been asking about the book, here’s the major reason I didn’t get it done by Summit as much as I wanted to. The pain was too much. I’m honestly surprised I made it through 2015 and got as much done as I did looking back.
The good news is that post-Summit Squared, I was going to be off the road for a bit. I made a doctor’s appointment, and am now getting treatment. For the first time in nearly two years, I’m feeling better and closer to normal. I have a way of dealing with what’s wrong. I joke it’s suffering for my art, but I literally left this alone from the time I first saw the doctor in Summer 2014 to fall of 2015. In that time, clearly things got worse. I ignored my body. That was not the most brilliant thing I’ve ever done.
Now that things are back on track, don’t be like me. To be the best you, a big part of that is being healthy. There’s mental health (i.e. needing to take a vacation to recharge the batteries; I’ve blogged about this in the past) and there is your physical/general health. That’s what I’m talking about here. You’re no good to anyone if you are sick, run down, in pain, etc. Sure, you can put a brave face on it, but silently suffering doesn’t work long term. Take it from someone who has been there and is now doing what I should have a year ago.
As you look toward 2016 and your resolutions, take stock of where you are mentally and physically and make sure you carve out time to take care of yourself in the new year. It’s hard to do with all of our personal and professional obligations, but you have to do it. I’m glad I finally did. The difference really is night and day.
By: Allan Hirt on December 16, 2015 in Always On, AlwaysOn, Windows Server Failover Cluster | 15 Comments
If Microsoft and I had I relationship status, it would be “it’s complicated”. As a Microsoft MVP and someone who specializes in all things Windows and SQL Server (within reason … there’s way too much for one person to literally know everything), I make my bread and butter on their platforms. Sure, I’ve used Unix and other RDBMSes over the years, too – but not like SQL Server. I speak VMware very well (not just Hyper-V). While I don’t want to bite the hand that feeds me, anyone who knows me is aware that I’m outspoken. Some people love it, other people have an issue with it.
I try to be honest but not mean (despite what people may think – and you’d know when I’m being mean, trust me), but the reality is some people just can’t handle the truth so you pussyfoot around it. I come from a place where facts rule the day and I try not to let emotion cloud my judgement, but some of Microsoft’s recent decisions have me a bit puzzled, and quite frankly annoyed. What MS decides directly affects not only how I do my day job as a consultant and educator, but any one using Windows and/or SQL Server. In this post, I’m going to focus on two issues because I think Redmond needs some tough love and a dose of reality from someone who cares and is passionate about both Windows and SQL Server.
Issue One: Always On
Yes, you read that right – Always <space> On. It’s almost a joke now with me and AlwaysOn, but it ceased to be funny a long time ago. It’s funny in the “haha” sense, and you can see by my Twitter feed people troll me all the time. I’m used to it.
You may have seen my previous blog post “AlwaysOn Is the New Active/Passive and Active/Active“. If not, take a few minutes and read it because it sets this up. If you want the TL;DR approach here it is: AlwaysOn is not the AG feature. If that was the issue, I wouldn’t be writing a new blog post.
As noted in that blog post, the words “always” and “on” have a long history with SQL Server. Always On (with space) was the initial designation for the highly available storage for use with SQL Server. So availability has always been linked; that is not in question. To quote this Hitachi whitepaper:
The Microsoft SQL Server Storage Solution Review Program is a specific SQL Server program that enables storage solution providers to highlight those storage solutions and configurations, via the SQL Server “Always On” labeling, that they have successfully reviewed against core functional Microsoft SQL Server storage requirements. The core requirements defined herein must be met for reliable, highly available SQL Server storage systems.
Here are some other links from that era (there are more, but I think you’ll get the idea):
As I wrote about in my book Pro SQL Server 2005 High Availability (Apress, 2007) in Chapter 4 on page 122:
At TechEd 2006, Microsoft unveiled the SQL Server 2005 Always On program (http://www.microsoft.com/sql/alwayson/default.mspx) [note: this link does not work now; it did back then, so don’t try it], which is an umbrella that brings all the SQL Server availability technologies under one banner. Part of the Always On program is a partner program for storage solution partners. This specific portion of Always On is known SQL Server Storage Solution Review Program.
I write a bit more in that, but this was the first morphing of the Always On name, which now included clustered instances as well as database mirroring, log shipping, backup and restore, etc. By SQL Server 2008, the storage certification portion of Always On was killed and it was just the HA-related features (FCIs, DBM, and log shipping clearly mentioned) as evidenced in this SQL Server 2008 OLTP marketing PDF.
Enter the feature introduced in SQL Server 2012: availability groups (AGs). Yes, AlwaysOn was a name for the feature for about five minutes in its development cycle as noted in my other blog post, but ultimately marketing decided to name the feature to be AGs and now had a new umbrella moniker for only FCIs and AGs: AlwaysOn. So the official names for those features are/were (we’ll get to were in a minute) AlwaysOn Failover Cluster Instances and AlwaysOn Availability Groups. So AlwaysOn has no space in 2012. This can be seen here and a screen grab is below in Figure 1. So out are DBM, log shipping, etc from the AlwaysOn moniker. And for heaven’s sake, it’s not AOAG. It’s just AG. We don’t say AOFCI – it’s just FCI (which is clearly seen at the link and in the screen grab; click to make it bigger). Precedence, people. Follow it.
Figure 1. AlwaysOn defined in SQL Server 2012 Books Online
My frustration with people using AlwaysOn improperly as the AG feature is well documented in the other blog post, and that issue has been occurring now for the better part of five years at this point. I’m not going to lie – it’s hard getting people to refer to the feature properly. Microsoft and its employees are some of the worst offenders truth be told.
Fast forward to this year’s MVP Summit which was right after PASS Summit. I was tipped off by a few birdies there was a change coming to AlwaysOn I would not be happy about – Always On (with a space). At first I thought they were trolling me and having a laugh – but no, this was real. SQL Server marketing in their “infinite wisdom” has decided to go back to the original spelling from here on out, which means for SQL Server 2016 (and possibly everything in 2012 and 2014), forget the no space. Ugh.
The new and improved Always On (with space) made its first official appearance in the December 15th blog post “Enhanced Always On Availability Groups in SQL Server 2016“. Now that it’s out there, I can talk about it. I’ve been sitting on this one for over a month. As one Microsoft PM once said to me, “You can report the news, but you can’t make it.”
I’ll cut right to the chase: as if we didn’t already have an AlwaysOn problem, marketing just made it worse. I mean this in the nicest possible way: I really think their heads are up their posterior. That won’t win me any favors, but it has to be said. What the hell are/were they thinking here? This change neither clarifies things or makes them better. It makes them WORSE. We’re taking a time hop back nearly 10 years.
When a bunch of us who fight this constantly found this fact out last month, we were all pretty livid. Marketing apparently lives in a bubble where they don’t give a hoot that people like me have to live with these asinine and uninformed decisions. No, I don’t expect them to consult me personally when they do something related to the SQL Server availability feature set, but it’s like they asked no one who even uses this stuff or thought about the downstream effect. I have a great working relationship with SQL dev, but I’m calling this ugly spade what it is: ridiculous.
What they should have probably done for SQL Server 2016 is something I would actually advocate (no, really): call the feature currently known as AGs – what most do already – AlwaysOn (no space). Drop the umbrella banner and just have HA features. FCIs will just be FCIs. I have been adhering to the rules marketing set up for 2012 and 2014, and I get penalized for it because they do a piss poor job enforcing it. PMs on TAP calls and in presentations regularly call AGs AlwaysOn with nary a word said. MS should train their own employees properly to be honest; it would solve many of my issues. Now marketing pulls this rabbit out of the hat? Unbelievable. Well, it is believable; I just have a hard time believing they consciously thought this was a winner of an idea. This almost is the equivalent of Bizarro and Bizarro World in the Superman universe.
A good friend and fellow MVP, Mike Steineke (Twitter | Blog), said two things today with regards to this which made me laugh but are true:
- AlwaysOn naming is as random and complicated as SQL licensing
- “Lucas thought Jar Jar Binks was a good idea too… Meesa like AlwaysOn” – something one of his employees said and appropriate given the opening of Star Wars Episode VII The Force Awakens this week
I think those two quotes sum up my feelings pretty well.
EDIT: As has been pointed out by two comments, the real goal here is consistency. Failure to be consistent leads to confusion and problems (read the comments for more).
Issue Two: Services/Applications/Roles/Resource Groups and Windows Server Failover Clusters
SQL Server is not alone in its penchant to do things that are detrimental to customers and understanding the platform. Back in the day pre-Windows Server 2008, at the Windows level, we had the concept of resource groups. Simply put, they were basically containers for clustered resources. For example, if you installed a clustered instance of SQL Server 2005, its resources would be in its own resource group. In Cluster Administrator (the GUI-based management tool), they were referred to as groups. The command line cluster.exe also used the group nomenclature. So far, so good. Figure 2 shows what that looks like with a clustered default instance of SQL Server 2005 named BATMOBILE.
Figure 2. Cluster Administrator in Windows Server 2003
In Windows Server 2008, we got a new GUI administration tool – Failover Cluster Manager (FCM). Lo and behold, they changed the name for things in the GUI to be Services and Applications, not groups. But cluster.exe and in Windows Server 2008 R2, the PowerShell cmdlets (such as Get-ClusterGroup), refer to groups. For people who don’t know Windows Server Failover Clusters (WSFCs), this can be confusing (more on that in a minute), and if you come from 2003, it’s a problem as well. The reality is that underneath the covers it’s still a resource group. The concept never changed. There are other issues I won’t get into in this blog post. The groups –> Services and applications thing can be seen in Figure 3 which shows another default FCI named AJA.
Figure 3. Failover Cluster Manager in Windows Server 2008 R2
In Windows Server 2012, this was then changed in FCM to be Role(s), but again, any command line still refers to groups. Figure 4 shows a named clustered instance of SQL Server named MINI\DISC.
Figure 4. Failover Cluster Manager in Windows Server 2012
Why is the UI and the command line out of sync? There honestly isn’t a good reason other than it is what it is. As someone who trains and speaks all the time, I need to make that connection for people between what you do in PowerShell and why it is different than the UI management tool. That also usually means I have to show Windows Server 2003 to people – something I’m sure they do NOT want me doing. I’m sure we’ve all seen the 1,000,001 WINDOWS SERVER 2003 IS GOING OUT OF SUPPORT – UPGRADE NOW e-mails, ads, Tweets, etc. from various folks including Microsoft.
I’ve been talking to the Cluster PMs about this since Windows Server 2012, and with Windows Server 2016 since it wasn’t done, formally filed a bug which was recently closed as will not fix in this version – maybe in the future. Look, I get it. It’s basically a visual thing which does not affect functionality and it’s not broken, and I’m sure there are things that need time allocated to fix which are more important. However, it leads to confusion when showing PowerShell (“What’s a group?”). It’s bad enough dealing with the 2008 R2 to 2012 questions of the Service/Application change to Role. Three versions into this new Role cycle, I’m calling enough is enough. Usability is a concern.
So if they want me to continue to show Windows Server 2003, don’t fix this. Until they do, Windows Server 2003 will be shown in some of my presentations and classes because many people learn visually and need to see that progression. Either that or change the UI back to Group or Resource Group. I’d be OK with that, too.
The Bottom Line
As you can tell, I’m not a very happy camper. Windows and SQL Server have given themselves enough rope to hang themselves on these two issues. SQL Server’s issue is completely self inflicted and causing a lot of confusion out in the world, and has for the better part of five years. I can only hope they come to their senses before SQL Server 2016 RTMs. As for Windows, like I said, I get things that are truly broken need to be fixed. But at this point when you want to put 2003 to rest, you can’t do it half assed. There’s still time before Windows Server 2016 RTMs. I’m not asking for this change to roll back to 2012 or 2012 R2.
I can’t reiterate enough that I am writing this from a standpoint of passion and caring, but the truth hurts sometimes. No one likes hearing their baby (in this case, their work) is ugly. Unlike animals and humans, these problems I discussed here are not life and death, and can be fixed. The ball is in your court, Microsoft. Do the right thing. I would love to see comments below because I think Microsoft needs to hear it from more than just me. Many voices affect change.
By: Allan Hirt on December 10, 2015 in .NET Framework, .NET Framework 4.0, .NET Framework 4.5, .NET Framework 4.5.1, .NET Framework 4.5.2, .NET Framework 4.6, .NET Framework 4.6.1, SQL Server 2012, SQL Server 2014, SQL Server 2016, Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2 | No Comments
You may have seen one of the 1,000,001 Tweets, blog posts, and other warnings around SQL Server 2005 end of life, and prior to that, Windows Server 2003 end of life. But there is something else intrinsic to SQL Server that you also need to worry about making sure you are up to date which affects later versions of SQL Server you may be running: the .NET Framework.
If you’re using SQL Server 2012 or later, one of the required components is some version of .NET Framework 4.x. Windows Server 2008 R2 did not ship with .NET Framework 4.x, so you would need to install it on your own or have SQL Server do it. This is known as out of band. Windows Server 2012 shipped with .NET Framework 4.0 and Windows Server 2012 R2 shipped with 4.5. This is known as in band. The good news is that for installations of SQL Server 2012 or later, you would not have to install or enable .NET Framework 4.x; it’s just there.
Like any other component, the .NET Framework needs to be managed, maintained, and updated. Depending on how .NET Framework was installed/configured, either it will be managed as part of updating Windows (in band versions), or you will need to manually update if it was installed separately (out of band, such as .NET Framework 4.x on Windows Server 2008 R2 or if you installed .NET Framework 4.5 on Windows Server 2012).
Microsoft recently published an official .NET team blog post reiterating that .NET Framework versions 4, 4.5, and 4.5.1 will no longer receive security updates, support, or hotfixes as of January 12, 2016. This was first announced back in August of 2014, so it’s not like this is new news, but I can say from experience virtually no one is talking about it. MS’ new post talks more about the upgrade path. To sum it up, you need to install .NET Framework 4.5.2, 4.6, or 4.6.1 to be considered supported when it comes to your .NET Framework version. Security is a real issue for many, and those responsible may not know that your version of .NET Framework could be a possible attack vector. Is your security team aware of this impending problem? How will this affect your version matrices (you do have those, right?)?
The bigger question is this – have you even planned for this event in your environments for both servers and any clients or application servers that run or connect to SQL Server? Have you been keeping up with .NET Framework versions or did you just install SQL Server and forget about it? I would say that most people I encounter fall into the latter. Staying supported is a key component of being mission critical.
If you haven’t thought about updating/patching your .NET Framework version, now is the time to start planning the upgrade to your preferred, supported, and tested version of .NET Framework in your environments. If you do not test against your applications, you could possibly encounter problems. In other words, without testing you are rolling the dice. That is anti-mission critical and can lead to downtime.
Another question you have to get the answer to is this: who owns this upgrade? Is it the app owner? The DBAs? The sysadmins? The .NET Framework can be a bit of a grey area depending on the server and how it was installed.
Here’s a tip: if you’re considering 4.6, just go to 4.6.1 – you’ll not only have a longer shelf life of supportability, but they enhanced the behavior of MultiSubnetFailover (see this blog post for more information) which could benefit some Availability Groups implementations (I am trying to verify if it applies to FCIs; it should as well …). To help you get to green faster, here are some handy links:
- This is the KB article to get information on how to download .NET Framework 4.5.2 as an offline installer (including desktop OSes and all Windows Server OSes).
- This is the link article to get information on how to download .NET Framework 4.6 as an offline installer (including desktop OSes and all Windows Server OSes
- The blog post linked above for 4.6.1 has links for its download.
If you need help planning or implementing your upgrade strategies, let us know.
By: Allan Hirt on December 3, 2015 in Licensing, Windows Server 2016 | 1 Comment
Today, Microsoft published the FAQ (dated December 2015) for Windows Server 2016’s licensing. It is still early days and I have seen no official prices yet, but this document gives you an idea of what is coming. However, as this is still early and it has not RTMed yet, all of this is clearly subject to change so this blog post is based on what is known today. There are some important changes you should be aware of, and this is where my excitement for some of what is in the platform meets the reality of our customers at SQLHA.
Windows Server 2016 Will Have Different Features Per SKU
For the first time since Windows Server 2008 R2, there will be a difference in the feature sets. In Windows Server 2012 and 2012 R2, both Standard and Datacenter had the same stuff; the real difference was virtualization licensing. With Windows Server 2016, there will be a few differences, notably Storage Spaces Direct (S2D – a really cool feature I’ve demoed a few times with SQL Server) and as they put it, “an Azure-inspired networking stack.” This isn’t all doom and gloom; the basics are still in Standard, it just looks like the high end stuff, much like In-Memory (i.e. Hekaton) in SQL Server, is in Datacenter. While to me it is a bit disappointing they are diverging a little, I’m glad they are not seemingly hobbling Standard. The reality is that the use case for things like S2D will not be for everyone, so in a way, it makes sense to push it to Datacenter for now. Those higher end features generally require more expertise and cost to get running, so many smaller shops with limited time, budget, and resources would be hard pressed to take advantage of them out of the gate if I’m being honest.
Say Hello to Core-based Licensing for Windows Server
This is the one that may annoy most folks. Like SQL Server, Windows Server 2016 licensing will be core-based, including the Core Infrastructure Suite SKU. Historically, Windows pricing has been MUCH lower than SQL Server, and no prices have been announced. So before anyone has a conniption, let’s see what the core pricing will be based on the chart shown on page 2, there are cases where the cost may be the same as it is today.
Having said that, there are a few things that you should start thinking about.
- Some what is behind this change has to do with hybrid scenarios. The (Public) Cloud is a big part of licensing, so Microsoft had to address it in the in-box scenario at some point. For example, in October of this year, Microsoft allowed those with Software Assurance to start using their own Windows images in Azure and paying the standard compute rates, so this move aligns with that. Whether you agree or like that is a different story, but it makes sense. For more information on Windows Server license mobility with SA for Azure, see the official blog post here, and it will go into effect on Q1 of 2016 (calendar, not fiscal, year).
- Core-based licensing is based on physical cores. So if I’m interpreting that correctly, you do not pay extra if hyperthreading is enabled.
- The core-based licensing is slated to start coming into effect in Q3 of 2016, and this is the first document that points to the general availability of Windows Server which looks to be in the July – September timeframe. That should surprise no one, given Ignite is at the end of September of 2016.
- Core licenses will be sold in two-packs. Every physical processor will need a minimum of eight core licenses, or four packs. The math here is simple: even if you have a six-core box, you’re going to have to pay for eight. What does this mean? Stop buying processors with less than eight cores as you will be wasting money.
- Standard Edition allows for two VMs with licensing ; Datacenter, as it has in the past, allows for unlimited VMs assuming all processor cores are licensed.
- You still need CALs for users and devices accessing servers.
- Nested virtualization (I’ll be posting a blog on this in a few days) is covered with Datacenter.
Musings and a Call to Action
I’m not surprised by this move. Like SQL Server did with SQL Server 2012, they are aligning to how things are really purchased and deployed both on premises and in the cloud. As with SQL Server 2012, we had a great ride for a long time with processor-based licensing. Again, it’s important to keep in mind that historically Windows Server has not been SQL Server, so the costs will most likely be reasonable. But this will change the way we use and configure servers both on premises for physical and virtualization (regardless of whether or not it is part of a private cloud) and up in the public cloud if you’re bringing your own licenses.
Like I said, this is not doom and gloom. Rome is not burning yet and it may not for many customers depending on how things shake out. When we help our customers design, plan, and implement their architectures, a big part of it is maximizing their investments – and licensing is a huge component. So if you’re looking for help with a new project (on premises – physical or virtual, hybrid, or public cloud) where this may come into play at some point, or you’re looking to assess how this may impact your current deployments and want to minimize the impact of this announcement, contact us today. It’s always better to be proactive than reactive.
I’m curious as to your thoughts about this big change in Windows Server 2016. Will this drive you away from deploying Windows-based solutions, and as part of that, SQL Server? Is this a net-net of no change for you? Will it limit how much you deploy? Weigh in below.