Cubegeek

Big Data Analytics vs Data Warehousing

Problem: Determine how many people are at the beach on the 4th of July.

Solution 1: Big Data Analytics - minded
Take a fleet of quadcopter drones and have them take thousands of snapshots of the people on the beach. Download many gigabytes of jpegs. Use fancy facial recognition software to identify individuals and take a best guess estimation by de-duplicating the result set. Give a number within a confidence interval.

Solution 2: Data Warehouse - minded
Block off all access to the beach. Don't let anyone in until they swipe their driver's license or photo ID. Query the database, get an exact count.

Big data analytics is what it is because it makes guesses from data that is not structured to answer specific questions. Data warehousing is what it is because you absolutely structure the domain for data collection according to a purpose. In an ideal world, all big data analysis guessing evolves to data warehouse structure. 

So 'big data analytics' essentially means inefficient unstructured data + smart guessing. All of the credit card transactions in the world are data warehouse structured, and have always been. But that's not 'small data'.

To get a longer explanation of some the details of data warehousing that would bore, but satisfy, an expert like me, read this article. But do so with the following understanding. Everything that 'big data analytics' is, will become a data management subset of future data warehousing. In other words, SQL has mastered all we need to know about set logic and transformations in database tech. It *is* the standard semantic level, as permanent as HTML is to web development. Everything the unstructured database tech now do in 'NoSQL' will be assimilated. Resistance is futile. At the core of the technology, what's too fast today will not be tomorrow. What's too voluminous today will not be in the future. 

At Full360, we already combine appropriate technologies in our DW + BI Elastic framework to bridge the gap. There's no data to fast, too wide or too big for us. 

Five Levels of Analytics

As 'Chief Iteration Officer' at Full360 my mantra is 'iterate to quality'. That means preserve what you know and get a little bit better every day. I have always been from the Deming School and don't often advocate the Hammer approach. I like John Boyd's OODA loop and I think it works wonders for business processes. As systems integrators, we don't always get greenfield projects. Most of the time we are given to take over systems that are broken, incomplete, poorly performing or out of date. Just having 'analytics' in place is not sufficient. The general idea is always good, but how well are you doing?

This is a guide to help describe the level of thoroughness of an business intelligence system.

Discovery - Level One

• I can use my system to know what information is available to me.

This is the broadest and most basic level. it means that as a user, you know which systems are available to you, how to access those systems and what you can expect to find in those systems. The cash flow reports are here. The current reports are here. The historical reports are there. The systems that identify operations are there. You can navigate to all of the systems relevant to your area of responsibility; you know their names, functions and update timeliness.

Clarity - Level Two

• The information available to me is clear and unambiguous.

This means that master data issues are resolved. Codes and spellings of descriptions are complete. There is no question that data is in the proper range. It is validated and audited so that there is little doubt that you are dealing with the right information at the right time. You are aware of the units of measurement and these units are converted into standardized, consistent, useable form.

Purpose - Level Three

• I understand how the information relates to the business at hand.

Purpose is achieved when data becomes information and you can describe the business in terms of that information. It might only describe the costs or staffing levels of the business, but it is specifically applicable to the operational function of the particular business unit. You may have unambiguous clarity about 'gallons of gas available' but it serves no purpose to the legal affairs office. What is measured should match what is known about the operations, whether or not the information is managed. Also, proportionality of the numbers themselves are understood at this level. One familiar with the business should know how various measures and metrics relate to each other, if at all.

Predictability - Level Four

• I can determine with precision what specific information describes business performance.

At this level of analytic depth, system users are capable of isolating that information that is a primary indicator of business performance as contrasted to those that might be trailing indicators or otherwise not driving the business. It is the difference between knowing when a small fluctuation or a big fluctuation is material to business performance. It might be something like a regulatory ratio that must be maintained, a quality control statistic for pass/fail of a manufactured lot of goods. It might be a stated performance objective like a tight budget or schedule variance. Industry standard metrics would be here.

Responsibility - Level Five

• I can accurately predict and plan business performance with my system.

The final level of analytic depth is responsibility. At this point reasonable budgets and forecasts are be made on key performance indicators. Variances are taken against actuals and not only business performance, but prediction performance is tracked. A well-managed business is data driven at this level of analysis and planning, and novel KPIs and metrics might be generated whose meaning and value are quickly evident.