Best BI

Q. 1. What's the "best" search/database/data warehouse software/system, in your opinion, that can handle larges amounts of data with ease and can handle asking "questions" to which you don't know the question? In other words, I have all of this sales data, but I don't know what the data can really tell me outside of my pre-canned "questions".

A. I'd probably overbuy in an enterprise scenario because I'd want all the capacity I could handle so when I mention brand names, you can certainly scale down from there. But there are a few components that would be absolutely necessary.

First and foremost (oddly) I'd get a metadata management system that has the ability to generate attributes based on dynamic business rules. That way, given any set of data, I can reclassify it according to things I find out about it later. My pick is the only one I'm familiar with which is Razza/Hyperion MDM.

Secondly, I'd get an industrial strength database for standard warehousing. I tend to go with DB2 because of its relative programmatic inflexibility as compared to Oracle and MSSQL. I don't want to do anything fancy in the relational layer and the proprietary extensions in T-SQL and PL-SQL make programmers get creative. I actually believe a lot of incredible things about Teradata and Sybase ASE, but I have no experience with either. I'd house this on a generic 6 or 8 way box with a huge SAN and use it to vacuum up all enterprise data for multidimensional staging.

Thirdly, I'd definitely go with Hyperion Essbase Analytic Services. The new Aggregate Storage Option on 64 bit can serve up thousands of 30 dimensional subsecond queries. Basically more than your brain can consider, faster than you can think it for hundreds of people concurrently. I'd actually spend more money on this interactive query platform than for the DW backend. I'm partial to HP's 64Bit hardware.

Finally, I'd use regular Excel as one of the front-ends and Tableau for the other. Anybody who is not capable of handling the complexities of reading multidimensional data in Excel is similarly incapable of understanding the implications of the analysis to the business. I have unsubscribed to the belief that computer systems will make things plain to those who don't understand the business.

You have to have some idea of the relations between profitability and cost of sales to put those two things into a two-way analysis, but then to put in employee benefits and freight costs as well and think of a management program that ties them together as befits your analysis is the level of thinking necessary. With this kind of system in place, you can rapidly build query spaces for that kind of analysis *and planning* and change your budgets and forecasts appropriately. By rapidly I mean in one day.

Given staffing and competence in the technologies it would take one day from transaction system to denormalized ODS to n-dimensional model to front-end. Both of these front-ends to Essbase are zero configuration, instant query mechanisms. which dynamically understand changes to the underlying master data. With MDM in place you can take results of your queries feed them back and restage the data every day. This is what I call a visual human-based data-mining paradigm. It presumes that the dimensionality of the incoming data is somewhere less than about 20, but its extensible.