Today is the 75^th anniversary of the liberation of Auschwitz. Auschwitz had more than one camp and this post is not about those details, the camp system itself, or things like that. It is also not a commentary on politics, corporations, religion, war, or any topic orbiting them. I’ve visited both Auschwitz and Dachau and blogged about my visit to Dachau in 2012. Auschwitz affected me in different ways and was its own distinct and sad experience for other reasons. It is profound and life changing.

Data drives the modern world – now more than ever. This is true in our personal as well as professional or business lives. Examples include how you invest money or how a company makes decisions. A lot of choices are based on information which is just another word for data. Some degree of luck and intuition can help, too.

Much of today’s information sits in databases (and not just in SQL Server-based ones). Queries and tools extract and present said data. It can be analyzed, filtered, and otherwise processed in numerous ways. Any data professional who has written queries knows that changing, say, a WHERE clause can make a huge difference in the result.

We tend to think of computers and mining data as recent innovations. They existed in more “primitive” ways before what we think of as modern computing . One early tool was the Hollerith Electric Tabulating System, first used for the 1890 US Census. It was later extended to use things like automatic card feeding as well as incorporated the first keypunch. The company which made those machines was combined with others in 1911 and renamed International Business Machines Corporation (IBM) in 1924. You may have heard of them.

Databases, relational or not, did not exist in the same way then as we think of them today. Those punch cards effectively made up what is similar to a database today. The reason I am specifically referencing the Hollerith is that those machines were used as part of the data-driven mechanism behind the tragedy in Europe which I first learned in Edwin Black’s excellent book IBM and the Holocaust. I highly recommend reading it.

I recently learned the US via the War Relocation Authority (WRA) also employed Hollerith machines during World War II. The WRA was responsible for the relocation and detainment of Japanese Americans. There’s no nice way to say that. IBM was subcontracted for that work. Accounting Information published an article about all of this in Volume 33 Number 1. The section “Big Brother Is Watching” discusses how the data was used, analyzed, and ultimately, reported. Here is a snippet:

His grand design for 1943 was a “locator file” in which would appear a Hollerith alphabetic punch card for each evacuee. These cards were to include standard demographic information about age, gender, education, occupation, family size, medical history, criminal record, and RC location. However, additional data categories about links to Japan were also maintained, such as years of residence in Japan and the extent of education received there.

Here is a what a Hollerith card looked like courtesy of Wikipedia.

Today many of us are annoyed by targeted ads, most of which stem from mined data about where you were looking online or by using data companies already have about you. American Express was recently able to identify fraudulent charges on my account due to data they have based on my history with them. Mining data is not inherently bad. How that data is used is where responsibility comes into play. There is a reason laws like HIPAA exist in the US.

As we commemorate this day which should be both celebrated for the aspect of freedom as well as respected for its remembrance of a terrible chapter in human history, I ask you to reflect on the use of data and both the responsibility and relationship to it. How do you deal with it in your job? In your personal life? Those are heady questions for a Monday, but it is an appropriate day to contemplate such weighty topics.