backups Archives

May 18, 2020

Configure SQL Server Backup to URL Using Azure Cloud Shell

Happy Monday, everyone.

I had to do something this weekend that involved the SQL Server feature backup to URL which allows you to back up databases in a SQL Server instance directly to Azure blob storage. That PowerShell there is quite the adventure … and cumbersome. There’s a much easier way to get this done using Cloud Shell up in Azure. I used Bash, but I’m sure there’s an equivalent Azure CLI PowerShell way as well.

Backup to URL/Restore from URL works from Transact-SQL and the SQL Server PowerShell cmdlets. There is no UI option in SSMS. Figure 1 shows what happens when I tried to restore a valid backup stored in a URL in version 18.4. I hope this eventually gets fixed. For all graphics in this post, click on them to enlarge.

Figure 1 – Thanks for playing … don’t try again.

Storage Accounts and Access Keys

This post assumes you already have a storage account in Azure. To do nearly anything, you need one. You can create a special one just for backups if desired, but it’s not necessary. Work with your IT admins to see what the general direction is here if you already have a presence in Azure.

Whatever storage account you use, you’ll need three things: the name of the resource group that contains the storage account if using Cloud Shell, the storage account name, and the access key for the storage account. You can get the storage account name and the key from the Azure Portal. It can be found on the Access keys blade for the Storage Account as shown in Figure 2.

As an aside, the fact all of the menu things are not capitalized for the first letter completely drives me NUTS. Yes, I get that’s how things are today so get off my lawn. But if people stuck to it or used it consistently, I’d be fine – annoyed, but none worse the wear. Look at the screen shot below. Storage Explorer (which is a specific service in Azure) versus everything else (i.e. Access keys). Boy, that shift key is really, really hard for those extra words, no? Anyway …

Figure 2 – Access Keys in Azure.

If you don’t want to use bash to put the storage account key into a variable, you can just cut and paste from the Azure Portal. Stick it in a Notepad (or equivalent on your OS) document if you want that you don’t save; it’s just a file to keep open so you don’t have to keep navigating back to that blade.

To get the access key using in Bash and set it to a variable, execute the following command:

[code]storageaccountkey= $(az storage account keys list -n “storage_account_name” -g “resource_group_name” –output tsv –query “[?keyName == ‘key1’].value”)[/code]

Tip: you could also put the storage account name into a variable since you’re going to use it a few times, but if you have it handy, it’s just as easy to cut and paste. Don’t overthink things.

Containers and the Shared Access Signature

The backup file you generate in SQL Server or restore from is stored in a container as a blob.

If there is already a container you will use, you can skip this step. If a container is not created already or you want to use a different one, create a container in the storage account. Below is the syntax for Bash:

[code]az storage container create –name “container_name” –account-name “storage_account_name” –account-key $storageaccountkey –fail-on-exist[/code]

Now you have to generate the shared access signature (SAS). This is how you will set up the secure access to the blob up in Azure. -o tsv is the same as –output tsv which takes care of the double quotes at the beginning and end of the output. Remember to set the value for the expiration date (–expiry) well into the future. The date is in UTC format.

[code]az storage container generate-sas -n “container_name” –account-name “storage_account_name” –account-key $storageaccountkey –permissions “rwdl” –expiry “2020-05-31T00:00Z” -o tsv[/code]

A big long string of text like the one below is generated. After that happens, you use it in the Transact-SQL used to create the credential in SQL Server.

[code]se=2020-05-31T00%3A00Z&sp=rwdl&sv=2018-11-09&sr=c&sig=%2BRUJldVzQHFdh7lZAeynfRuZr5dUeUohfVwKHZZX3jE%3D[/code]

Create the SQL Server Credential

Creating the the credential in SQL Server is a simple Transact-SQL statement.

[code]IF NOT EXISTS
( SELECT * FROM sys.credentials
WHERE name = ‘https://storage_account_name.blob.core.windows.net/container_name’)
BEGIN
CREATE CREDENTIAL [https://storage_account_name.blob.core.windows.net/container_name]
WITH IDENTITY = ‘SHARED ACCESS SIGNATURE’,
SECRET = ‘se=2020-05-31T00%3A00Z&sp=rwdl&sv=2018-11-09&sr=c&sig=%2BRUJldVzQHFdh7lZAeynfRuZr5dUeUohfVwKHZZX3jE%3D’
END;
GO[/code]

That’s it. You should be able to back up to and restore from a blob in Azure. Figure 3 shows a successful database backup of WideWorldImporters to Azure using Transact-SQL.

Figure 3- It really works!

The backup file WideWorldImporters.bak can be seen in Figure 4.

Figure 4 – WideWorldImporters in Azure blob storage

The error message below appears if something is configured wrong.

[code]Msg 3201, Level 16, State 1, Line 33
Cannot open backup device ‘https://storage_account_name.blob.core.windows.net/container_name/WideWorldImporters.bak’. Operating system error 50(The request is not supported.).
Msg 3013, Level 16, State 1, Line 33
BACKUP DATABASE is terminating abnormally.[/code]

Chances are the problem was between the brain and the keyboard, such as fat fingering the name of the container or storage account. Some examples include extra characters at the beginning of the SAS such as a question mark (?) or having double quotes before and after the SAS.

Why Use Cloud Shell and Bash?

In my experience, it is many less lines of code. If you do not have to create the container and want to just cut and paste the storage account name and access key, it is literally one line of code you need to write to generate the SAS required for the credential in SQL Server.

Hope this helps some of you out there.

May 21, 2015

Mission Critical SQL Server Book Update – May, 2015

UPDATE – December, 2016 – see this blog post

These days, one of the things everyone asks me is related to the book – mainly “How is it coming?” – a nice way of asking “Is it done yet?” It’s been awhile since the last update, and I figured it is time for a new one. If you are subscribed to the pre-release content, look for new chapters starting the week of May 25 (next week). Everyone who is not subscribed will need to wait for the final, completed book. There is a chance I may decide to put a few preview chapters up, but only when some more hit the pre-release folks; it is only fair.

So why the delay? Writing a book and publishing it on your own is much harder. My previous books were all done through a publisher who had editors and other bits of infrastructure I do not have at my fingertips. For the most part, I am a single threaded, one man show on the production of this book.

When I put the book up for preorder in July of 2013, I had an idea what I was signing myself for because I had written books before. Writing and publishing – especially self-publishing – is a whole different ball game if you want to do it right and not just dump raw Word files into the world and call it a book.

The good news is that I recently found an editor so the book will have a nice level of spit and polish. I’ve worked with her in the past, and am looking forward to seeing what she delivers. Having said that, primary writing and reviews must be completed before I can turn it over to her.

The only thing I can equate this to is when I scored a friend’s musical in the mid-1990s. I had played in tons of pit bands going back to when I started playing bass in 1980, so I figured, “How hard could this be?” Oh, how naïve I was. Between learning the scoring program, writing and arranging for multiple instruments, rehearsals, corrections – you name it. This was not a one day or one week process. I learned a lot, but it was very painful. Arranging got a lot easier after that, and so, too, will any future projects I do in the self-publishing arena.

Add to that my normal day job (consulting, training, as well as running SQLHA with Max), presenting, eating, sleeping, and trying to live my life somewhere in there along with the writing – oy! There just is only so much time and energy I have. However, I do believe all of these things play into one reason I believe that my books have been well received in the past. I bring real world perspective and experience to the table. If I am not doing this stuff for real, it’s just lab work.

For example, Max and I spent the better part of last year working with a large customer on a virtualized SQL Server project which also leveraged quite a few new storage and networking technologies not usually used with SQL Server. What was gained there will be filtered into the book, so you will be getting a lot better content than if I was not also actively doing consulting projects.

In more good news, I have pretty much cleared the decks for the next month or two to get all of the primary writing completed. Chapters need to be reviewed, and then reviewer feedback and corrections incorporated. Once all of the chapters are done, they will be handed off to the editor. Last but not least, all of the final formatting needs to be finished on the edited chapters. This is the entire book process in a nutshell.

Below are some specific questions that I want to address now because I have been asked from time to time. I have also updated the FAQ accordingly.

What content/chapters will the book contain?

While it may change a little, this is the near-final chapter list with my working (read: not final) titles:

Basics of Availability
Data Center Principles
Brief Intro to the SQL Server HA Features
Windows
Networking Part 1
Networking Part 2
Storage Part 1
Storage Part 2
Security
WSFC
Quorum Part 1
Quorum Part 2
Planning FCIs
FCI 2008 R2
FCI 2012
FCI 2014
Scripting Installations
Planning AGs
Implementing AGs
Cluster Admin/Troubleshooting
Backups Part 1
Backups Part 2
Admin/Monitoring
Planning Highly Available Upgrades
Solutions
Building Your Own Virtualized Test Environment

As you can tell, this is quite a bit of content. Right now the book will weigh in somewhere between 1000 and 1500 pages in Word prior to any formatting. Many of these chapters are in process, so those of you that have seen the updates may have only seen a few chapters but a lot of these are in various states of completion. If you are subscribed to the pre-release content, you’ll see all of this content soon.

When will the book truly be done?

From a timing perspective, assuming no issues or problems, primary writing should be done by July, edits by late August/early September, and then onto formatting. It really should be released on or before PASS Summit 2015. That is what I am working towards, and I see no reason that cannot be achieved.

I know people have been waiting patiently for Mission Critical SQL Server. My mistake in hindsight was putting it up so soon for preorder and not anticipating some of the roadblocks that popped up since putting it up for sale. I know that now, and will never repeat that again. For that, I apologize and I do believe the wait will be worth it.

I paid forever and a day ago – where is my money?

No one has asked, but I’m being forthcoming here – it’s sitting in a bank account earmarked for paying book costs. It’s not long gone. I made sure this happened for a very good reason. I need to pay people like editors 🙂

I subscribed to get updates to the book once it is released – will I still get them?

Yes! Needless to say, I’ll make sure you are taken care of. See the next question for a touch more detail, too.

If you preorder with updates on or before May 31, 2015, look for an e-mail from me soon. Anyone preordering with the updates option post-June 1, 2015, will get updates for the specified term – one year post-book release.

Will you do anything for people who purchased Mission Critical SQL Server a long time ago and have been waiting patiently?

Yes! I’m figuring out what that will be. This is different than the post-release updates. But the same rules will generally apply: pre-May 31, 2015 and post-May 31, 2015. I do appreciate your patience, and trust me, no one wants this done more than me because it is a big thing on my plate. There are other things I’d like to do which cannot get done until the book is complete.

When you first announced the book, you said it would cover through SQL Server 2012. What about 2014 and 2016 now that they are out or announced? How about Windows Server 2016?

Here is even better news. With the book being delayed, SQL Server 2014 is now included in this version. I will touch on a few 2016 (SQL Server and Windows things) but the first updates to the book will arguably be SQL Server 2016 and Windows Server 2016. Once those are released, I’ll examine what needs to be updated or added, and go from there. It is too soon to tell.

Any current discounts?

Yes. 20% until May 31, 2015 off of anything in the store. Code to be entered at checkout: MAY31

February 19, 2015

Application Vendors and SQL Server – A Cautionary Tale

Earlier today I tweeted this: “Dear Third Party Vendors: Do us all a favor – if you don’t understand SQL Server, don’t make products for it.” (Link if you want to see the original tweet.) Some of the biggest complaints I hear from customers across the board all have to do with the third party applications and the vendors that make them. I previously touched on applications in an older blog post from 2009 entitled “Consolidation, Application Compatibility (or lack thereof), and SQL Server“. Support for a version of SQL Server is one problem, but blatantly having false information and trying to use outdated features is another.

Today I’m going to call out Dell‘s product named AppAssure, which is backup software. Let me say up front I like Dell overall – they make good server products and bought Quest who makes Lightspeed and Toad. Many of our customers use Dell for hardware, along with some of their storage products. They are one of the few certified Storage Spaces vendors. However, in my opinion, this AppAssure is all kinds of wrong when it comes to SQL Server. Looking at this video (script is below if you can’t watch the video) specifically talking about how AppAssure and SQL Server are a marriage made in backup heaven.

Dell says this:

There are actually three modes for back up. The two most important are simple and full. So let’s say, small organization, they don’t have a dedicated DBA. They just want to back up and be able to recover their SQL Server. Simple mode is the default mode. And it enables Microsoft VSS to snapshot and quiesce.

Full mode is what you would find in most larger shops that have dedicated DBAs. And what they’ll do is they will omit or not protect the database volumes. In this case, it would be F and G, F for the databases, G for the logs. They would not be protected because, in full mode, you don’t have VSS.

There is quite a bit to address from these short sentences.

I think Dell believes their customers are simple (pun intended) if they believe this. Any DBA or SQL Server professional worth a damn knows that the recovery model for all user databases is based on model, and that is Full. Simple is NOT the default recovery model for most databases, and it would be a conscious choice to make that change (unless model was altered).
The Simple = small businesses, Full = large companies is a crock of hooey. If you care about recoverability, use Full (or Bulk-logged, but few people do) which is always the recommended best practice for user and application databases. Many SQL Server features – like availability groups – require Full. Full allows you to recover to a granular point in time. Simple does not. You would only be as good as your latest full and/or differential backup. And you would think a backup vendor and, by extension, its product would know this! It’s SQL Server 101.
The Volume Shadow Copy service (VSS) in Windows that all hardware and software interfaces with to do backups does not require Simple; it is agnostic to SQL Server. VSS must talk to the SQL Server Virtual Device Interface (VDI) to ensure that a proper SQL Server backup for a database is generated. VDI is the only thing we as SQL Server folks should have to worry about here (and yes, AppAssure does use it; you can see the freeze/thaw of I/O in the SQL Server log) when using a third party backup utility that quiesces a drive. I have used other solutions (such as ones from EMC which were a combination of hardware and software) that had no issue with databases in Full.

Next up, Dell says:

The next thing they would also do is configure another volume. And they would use it for database dumps. They would be dumping the database and the transaction logs periodically throughout the day to that volume. And then AppAssure can come in and back that volume up.

The difference in your capabilities, in simple mode, AppAssure can back up as frequent as every five minutes. And that would be your RPO. In full mode, you get the ability to choose a literal point in time of an hour, a minute, and a second when you want to recover by either replaying the logs or rolling back the logs, the SQL logs.

These actually ties into the previous one and speak to a few things, not the least of which is your on-disk architecture for SQL Server and how backups should be done.

I am assuming based on my limited window into AppAssure at one client and this information presented that it is only capable of full and/or differential backups in SQL Server since a differential is the only granular recovery you will get with the Simple recovery model.
Simple recovery model makes sense if you don’t want to deal with the t-log because to do point in time recovery, you would a) need to be able to restore using WITH STANDBY or WITH NORECOVERY (I do not know if AppAssure can do that) b) be able to restore transaction logs c) quiesce in a way that you do the t-log and data at the same time to have a consistent snapshot of the database via VSS. You don’t need to be consistent if you’re ignoring the t-log.
There seems to be no way to just snapshot a single database, which would potentially impact how you lay databases out on disk. AppAssure looks like it just quiesces an entire drive at once. This is one reason why if you are going to use any third party utility (HW or SW) to back up SQL Server, you need to think about it before going into production and how it affects aspects of SQL Server like data and log file placement.
If you ‘re not going to deal with t-logs, AppAssure can make backups ’til the cows come home!
SQL Server Agent jobs can now go down to as frequent as 10 seconds – not that you would ever really do that in the real world for t-log backups. You get more granular if you want to in SQL Server by doing t-log backups if your recovery point objectives (RPOs) require it AND you can generate the backups fast enough.

Let’s talk terminology. In the golden years of SQL Server when I was just a baby in this industry, backing up SQL Server could be done via the DUMP Transact-SQL command and backups were called dumps. They are still using that phrasing which is now outdated.

However, the most egregious offender in that video was the following:

Truncating the log … or not

The option highlighted is “Truncate log after successful attachability check (simple recovery model only)”. Is this 1998? I want Sphinx (aka SQL Server 7.0) back if so. Prior to SQL Server 2000, there was the ability to truncate the t-log on checkpoint. With the introduction of recovery models in SQL Server 2000, this is no longer possible. Simple != truncate the t-log. So I have no idea what Dell is doing here because there is no corresponding feature I’m aware of in SQL Server today that even corresponds to what they are claiming. Simple (in essence) will reuse and basically wrap around, but there’s no truncation going on in the way truncate log on checkpoint worked in SQL Server 4.21a, 6.0, 6.5, and 7.0. This is a meaningless checkbox to nowhere.

All I can say is caveat emptor if you want to use AppAssure for your SQL Server backups! But this is a cautionary tale of what you do not know – or understand – can hurt you. Applications vendors should have a passion for doing things right for SQL Server. Dell is partially there – they got the VDI bit right – but everything else is just off to me. I wouldn’t have a problem with it coming and backing up a drive as they talk about, but not as my main backup for SQL Server databases.

Tom LaRock (Blog | Twitter) posted this question after my initial tweet: “Why stop at vendors and products? I’d throw in blogs/videos, too …”. I agree with that statement, but it is hard to police. I remember years ago talking with folks at PASS Summit about how some sessions were giving advice based on their experience – which is technically valid – but in some cases, worst practices were being told as if they were best practices! I will admit to sometimes being wrong, but I always strive to correct if I am. Some people or companies are ignorant and don’t give a damn. If it’s on the internet it must be true, right? Wrong. Ignorance is not bliss and can lead people who through no fault of their own down a path which will result in trouble.

PS – I would be more than happy to correct and update any part of this blog post if I am wrong, Dell.

July 11, 2013

Allan’s Mission Critical SQL Server Book Ordering Information and FAQ

(Updated 12/31/2016 – see this blog post)

Mission Critical SQL Server is available for preorder here. Below you’ll find all of the information you need about it. And if this doesn’t answer your questions, post a comment or contact us.

1. Why am I publishing the book on my own this time?

This is not because I had a falling out with Apress; we’re on good terms. We just didn’t agree on the vision for the content. I wanted a larger no-compromises book more like my Pro SQL Server 2005 High Availability one; they did not. It’s really that simple. With all of the changes in Windows and SQL Server just through their 2012 versions (as well as SQL Server 2014), this is not going to be anything approximating a 300 page book; that’s too tiny and would be too incomplete. You guys benefit from my being unwilling to compromise here, but I get to drive myself nuts in the process being the perfectionist I am. After this book I may do another for Apress again if this book doesn’t beat me!

2. What versions of Windows and SQL Server are covered in the First Edition?

See the product description on the store for details.

3. What electronic format(s) will Mission Critical SQL Server be available in?

Initially, the eBook and all associated content will be in the ubiquitous Adobe Portable Document Format (PDF) format. Most platforms have programs that can read this for free even if your preferred platform isn’t Windows. I hope to offer it in other formats like Kindle/Mobi, ePub, and maybe iBook in the future.

4. Will there be digital rights management (DRM) on the PDF file(s)?

This is a sticky wicket. DRM makes life hard and I respect people do not want it. Passwords make it difficult to put on reading devices. Non-DRMed files make it easier to consume. At the same time, I also don’t want to see my content thrown up for free on some torrent site, nor do I want people to be able to easily republish it. Each copy will essentially be watermarked for who buys it and restrictions as laid out in Mission Critical SQL Server Store and Content Terms apply.

5. Will I only be able to buy Mission Critical SQL Server at http://sqlha.contentshelf.com/shop?

For now, yes. I hope to offer the book elsewhere over time. As noted in #2, the goal is to convert the content to other popular formats such as ePub and Kindle/Mobi – maybe even iBooks. That would need to happen before I can offer the content elsewhere. With the number of graphic elements in the book, that may prove to be both challenging and expensive. If you are wanting another format, contact us. The more evidence I have, the more likely it will be to happen. Converting it to other formats will be costly, so if there is not enough interest, I may not be able to do it. I am being realistic here.

6. I live in <insert country here>. Can I buy the book?

Yes. As long as you can check out, it’s a digital download making it easy to get to you. All prices are in US Dollars.

7. I live in <insert country here> and do not read or speak English well. Will you be offering non-English versions of the book?

There are no plans for the book to be translated to other languages.

8. Will there be a physical/print version of the book?

I would love to be able to offer one. Once the book is complete and I know a page count, I may reconsider a print version if it is possible and the cost will not be astronomical. Stay tuned.

9. What is an updated content subscription and why are you offering one?

Books are like snapshots: they are a moment frozen in time. Updating them is difficult and usually there is a long time between releases. A lot can change in a few months, a year, two years.What I’ve decided to do to combat that is to offer a subscription where you will get all the updated content as it is finished. For example, once SQL Sever 2016 and Windows Server 2016 are released, those updates could be incorporated before doing a major edition of the book which would be a year or two out. This gets you newer content quicker – if you want it.

Also, since the release cadence is changing for the Microsoft server products, traditional publishing just may not be possible for specific versions. By the time you’re done writing it, it’s obsolete. I want to have a good foundation to build upon and just extend it. Hope that makes sense.

10. How long does a subscription last?

It’s a yearly subscription which you will manually have to resubscribe to.

UPDATE 5/21/2015: If you preordered the book with an update subscription prior to May 31, 2015, look for an e-mail from me soon. Post-May 31, it will be only one year from the date that this edition of the book is published. So if it is officially released, say, on 10/1/15, your subscription would end on 9/30/16 if not renewed.

11. How much is a subscription renewal?

It’s $25/year. And as pointed out in the previous question’s answer, you elect to renew or not. My goal is to provide high value content so that you want to.

12. Do I need to buy the book again when you publish a new edition and I already have an updated content subscription?

If you have an active subscription, no. The goal is to only have you buy one major edition of the book and then you get updates as they happen. Information changes over time, and I’m trying to be agile and deliver you content ASAP and not have to wait years between releases. If you only want to consume major versions, just buy the new edition of the eBook when it is published. Simple.

13. If I buy the book from anywhere else but your store, can I get a updated content subscription?

No. If I do convert the book to other formats and offer it for sale elsewhere such as Amazon, Barnes & Noble, etc., those stores will only get the major edition (i.e. First Edition, Second Edition, etc.). I have to give myself some sort of exclusive, right? I may change my mind on this once the book is available elsewhere and offer something. Time will tell, but do not hold your breath.

14. Can I get access to pre-release content like other publishers offer?

Yes. It’s not free, but it is pretty cheap considering you’ll get to see things months in advance. Two of the bundles offer this option. There is currently pre-release content with more coming. Remember that the pre-release content has not been reviewed or edited, so it’s going to be raw and could contain errors.

15. Why are things priced the way they are?

I can tell you from authoring books in the past I make pennies on the proverbial dollar taking into account all the work and hours I put into the final product. This time, since I’m self-publishing, I make even less PLUS I’ve got very real, very tangible costs I need to deal with such as editing as well as costs for hosting and selling the book (including fees associated with subscriptions). I am not rich. I can’t operate at a substantial loss. I also know that I can’t charge $100+ for an eBook. I’ve priced the main eBook competitively with other similarly priced books even though it will be larger than most of them. I think it represents good value for the price you’ll pay.

16. Do you offer discounts or coupons?

I run specials, so check the store.

17. Can I use the PDF(s) I get from you directly on any device I own?

Absolutely! See our Mission Critical SQL Server Store and Content Terms for more detail. There’s a reason I chose this format.

18. Can I buy multiple copies of the for multiple people such as a class, seminar, DBA department, etc.?

19. Can I share the PDF(s) of the content with people I know? They’re really nice people and need some information on these subjects. I vouch for them.

NO! The copy you purchase is only licensed for the person buying the book (hence the watermark). I didn’t put DRM or passwords on it. Respect me and ensure that everyone buys their own copy. And if you upload the content to some nefarious site, I’ll see the watermark. It’s things like this that would make me never do this again. I know I can’t stop it 100%, but you hurt me more than you help me.

20. What if I am unsure if I want to buy it? Will you have some sort of a sample for me to look at?

I am planning on providing this as things get finished and posted; nothing is ready yet. If you have read any of my other books or writings, though, you have an idea of what you’re in for. 🙂

[Updated 5/21/15] 21. It’s May 2015. Is the book done?
See this blog post for more details.

August 16, 2012

Drawing Inspiration and Data Aplenty

Hello everyone! I just came off a bit of a vacation where I went to the Disney D23 Destination D 75 Years of Animated Features event out in California. Of course I didn’t give myself much of a transition – I am finishing up a 3-day private training course today.

Anyway, I’ve talked about it before in a previous blog post, but growing up I wanted to be an animator for Disney. For me, Disney has always been in my life as far back as I can remember; I mean, I can vividly remember coming home in the afternoons and watching the old Mickey Mouse Club reruns on, if memory serves me correctly, Channel 48 in Philadelphia. Disney is liked for many reasons by different people, but for me it’s always been about the animation. So the opportunity to go to an event focused on the animation and while not talked about in that link above, there was a 3rd day that was limited to a certain number of people which allowed you to tour the Walt Disney Studio (but different than the normal D23 tour they do periodically) and see things like the Animation Research Library – no brainer. Going there has always been one of my holy grails. (Before you start rolling your eyes, I promise there will be IT/SQL related content in this post. No, really.)

That third day was simply amazing and worth every penny I paid. In the AM we got to tour the Roy E. Disney Animation building where the animators put together a whole program for us showing the animation process based on the upcoming Wreck It Ralph movie. They took us through the different processes, and not surprisingly, the one that interested me the most was recording the voices and such. Don’t get me wrong, using the real Disney animation tools in a lab to mess with Wreck It Ralph was fun, but I could have sat in that recording studio all day and absorbed info about the process and asked questions since I’m a music/audio guy.

In the afternoon, we got to do different things, one of which was visiting the aforementioned Animation Research Library – arguably the place if you care at all about Disney animation. It’s locked down tighter than Fort Knox. More than a holy grail, it’s proverbially sacred ground and something I thought I’d never see. 65 million pieces of artwork live there. They took a small selection out for us and we got to look inside a few of the vaults. Seeing things like actual pencil drawings from “Steamboat Willie” (1928), the original maquettes used by animators for various movies, the wooden Pinocchio marionette from 1940 seen here (lower right picture), and original Mary Blair artwork (among many other things) was a once-in-a-lifetime experience. Most Disney people can’t even get in, let alone see some of what we saw. I am still thinking about it days later.

Another standout thing was that we got various presentations, one of which was on the preservation and restoration of the Disney vault of films (animated and live action) and shorts. This was something right up my alley since I’m also a video guy (I still have and use my laserdisc player). It was clear I was the techie guy asking the questions and people were looking at me like I had two heads. It was definitely a magical day all around.

As much as I was on vacation and loving this experience, my IT/SQL mind started to creep in and come up to the foreground every now and then. When we were in the studio, some folks go to do some ADR work to see what that process was (and it was fun to observe), but I was a bit more interested in the gear and process. What was interesting to me is that besides the takes they recorded in one rig, they had a continuous session running in parallel on a different one to capture literally everything. One reason for that is simple: if something is missed, it is captured elsewhere (think backups and HA/DR – gotta love redundancy).

That begs the question: how much data is that? Well, potentially a lot for each project over its lifetime. So of course I asked the question – at what resolution do they record at? For you non-audio geeks, to put anything I’m going to say in perspective, a CD’s resolution has a sampling rate of 44.1 kHz at a depth of 16-bit (an 80 minute CD at 44/16 is roughly 700MB of data). In the audio world, higher sampling rates (48, 88.2, 96, 192, or 384 being the other common options) and more depth (either 24- or 32-bit) is generally a better thing. Not surprisingly, that all chews up a lot of space the more resolution and the higher the bit depth you go. They record at 48/24, and everything is stored on a server with a lot of space. Considering how much audio they must have for each project, that’s a lot to deal with (even at 48/24) not only in terms of needing it near term for the active movie project, but then to back it up and ultimately archive it for other uses down the road. This is not unlike a DBA needing to manage a total of giga-, tera-, or petabytes of information both in databases as well as the various copies (including backups). This is just the audio data – can you imagine all of the animation, too? Wow. Oh, and every bit of that audio needs to be meticulously cataloged, too. Much like you need to keep inventory of all of your servers, instances, and databases.

During the video restoration presentation, we (royal we here) got to talking shop a bit (probably to the dismay of the non-audio and video folks who were in my group) and I asked why they only scan in at 4K and not 8K, as well as discussed audio sample rates. The reason was similar: it’s a data problem and for some old film types, you’re theoretically not going to really get better with higher scans (MuppetVision 3D in Disneyland was scanned in at 6K if you care about such things, but that decision had a lot to do with the original format; this forum discusses this topic more if you want to see some opinions and other links). This puts a whole new spin on Big Data, eh? It may not be schemas, but it sure is a ton of information that is unwieldy to deal with.

What was also interesting about that presentation was the talk about the film stock and not only how flammable it could be (depending on its type), but no matter what they do, at some point, the originals will be completely unusable (much like how I related backups to audio which I blogged about awhile back). Digital, film, analog tape – stored properly it will have a long shelf life, but it may not … it all depends since there are so many factors that contribute to that. If you have important data that needs to live for a long time, preserve it properly and have adequate redundancy, too. Archiving is no joke. A company like Disney is built on its history, and they need to preserve it.

Shifting gears a bit, the Animation Research Library as I said houses 65,000,000 pieces of art. I didn’t ask, but I could tell it was all cataloged and scanned in (I thought I heard someone say 150 dpi which was good enough years ago; I believe they’re working on scanning in at either 600 or 1200dpi now that tech has moved on and storage has gotten cheaper). Anyway, point is, we’ve got another data thing going on here. This is data not only used by animators today that needs to be accessible (hello HA!), but quickly and organized. Side note: cool perk of working there is you can apparently choose pieces from the archive to put up in your office. Drool. This is Art on Demand on steroids! Where do I sign up at HR for framed pencil drawings of “The Band Concert“?

Would I still love to be an animator? Maybe, but some of what goes on at Disney isn’t too far off from what I do today. I help customers ensure their data is available when they need it, be it near line or offline. Your data is your IP … and legacy. So maybe I really did choose the right career path after all – just getting a glimpse of how Disney deals with things like archiving and redundancy gives me inspiration in what I do right now, and is more than just a doodle on a page or a ride in an amusement park.

Storage Accounts and Access Keys

Containers and the Shared Access Signature

Create the SQL Server Credential

Why Use Cloud Shell and Bash?

Popular Posts

Categories