Cubegeek

A History of BI: The 80s

BI is Business Intelligence. At this particular moment in history, BI is seen as the more technical aspect (tools & techniques) in Management Computing.

So let's start at the beginning.

Back in the mid 80s, before the advent of the Pentium class computer and the acceptance of pre-emptive multitasking operating systems in the business world, a cat named William Bowen (no relation) published a cover story on Fortune called 'The Puny Payoff from Office Computers'. He started off:

HAVE THE MILLIONS of computers purchased by U.S. businesses brought any overall improvement in productivity? Surprisingly, the best information available says no.

This collides with many people's beliefs regarding computers. Without computers, present-day credit card operations, check processing, and airline reservation systems would be unthinkable. But the figures indicate that, on a national scale, business's investment of hundreds of billions of dollars in computers and computer-aided communications has failed to bring about a discernible improvement in productivity. ''The puzzling thing,'' observes economist Martin Neil Baily, a senior fellow at the Brookings Institution think tank in Washington, D.C., ''is that the computer revolution has not yet paid off in productivity growth as did the earlier generations of innovation.''
So far productivity has grown more slowly in the computer age than it did before computers came into wide use.

The accepted wisdom was, at this time, that the typewriter business would remain profitable, business managers did not want to type and would not learn to use a mouse, and that any computer with more power than a pocket calculator could only be useful to engineers and scientists.

This contradicted the expectations of people like me, who were invested in the ideas of 'informating' the workplace. These ideas basically originated out of the theories expounded in The Rise of Managerial Computing, by John Rockart. Along with the other cats at CISR and the theories of John Seely Brown, there was a technical vanguard of folks who believed passionately that computing could bring efficiency to business decision making. Technically, there was a distinction made between Strong AI and Weak AI. Weak AI was also known as the 'Augmentation School' as proposed by Doug Englebart.

In my own career, fortunately cutting my teeth at Xerox, I was in direct contact with the tools and technologies that were enabling these theories. I played enough with tools like LOOPS and XDE (and even the HUMBLE Expert Shell) to recognize the great potential. I actually did some Viewpoint programming in Mesa back in those days.

As we at the Xerox Systems Group looked to make the 'office of the future' we experimented with any number of support strategies and tactics. The one that managed to stick was the idea of the 'IC', the Information Center. But one thing was immediately clear, if the IC was going to function properly, we had to address Bowen's complaint. What emerged from the work done in the mid-eighties was an interdisciplinary approach dedicated to improving business via the judicious use of computing technology. Principles I developed with these things in mind are still valuable to this day.

It bears repeating that in those days, most computing was done on mainframes via subserviant terminals. Most corporations understood this paradigm, but they did not understand the value of distributed peer networks. Napster is not the first group of upstarts to encounter resistance. At XSG, our calling card was decentralized, client server, object oriented, networked computing. Each of those qualifiers was fought tooth and nail, and our noses were rubbed into every failure. But we were still all about End-User Computing. How could we make it win? By breaking it down to cases, application by application. I won my first crystal trophy building a system adhering to my now golden Pledge.

We were determined to make productivity gains in the 'front office', among analysts, managers and those we called 'knowledge workers'. This followed Englebart's ideas about augmenting human intelligence with tools that aid thinking. Back office computing was dreary and boring to us - cutting payroll checks, keeping track of telephone bills, these were things that could be done on stupid mainframes with primative languages like COBOL and elementary data structures like VSAM. We were were trying to get people to work smarter through advance user interfaces. We had to think of data in more ways than just fields & sorts. We were engaging people at their desks in order to make people more productive.

The Cubegeek in 1988 at Xerox Systems Group in El Segundo. Note the 6085 workstation with the 19 inch monitor on the right.

A lot of time and effort was spent understanding how people approached their work, how they organized their physical work, how they would interact with the computer. We were engineering things in real-time. Over time all of these disciplines came to be known as Executive Information Systems.

At Xerox, and certainly most other corporations, the new front office productivity efforts centered around Finance organizations. Numbers were something we could do. Memos were something we could do. Reports were something we could do, and so we expected our first payoff would be in Finance and Accounting. In various ways, these corporate staffs already knew that computing could save them time and money. Backoffice computing was already calculating payroll, etc. Plus, these guys held the pursestrings. They could buy systems when they felt like it. Sure there were many advances in CAD/CAM during this same period. In particular at Xerox, we were undergoing a complete transformation of the management culture from top to bottom. Xerox' CEO David T. Kearns had hired McKinsey & Company to instigate a program called Leadership Through Quality. The most important thing about the Xerox reorg with regards to BI is that they implemented a practice of cascaded management objectives. The point was that employees throughout the organization should understand at every moment how their work contributed to the priorities of the company.

Without getting too deeply into it, Quality Improvement struggled through the 90s against something called Re-engineering. Among the management consultants, there were two schools of thought. The Deming School and the Hammer School. Coming from Xerox, I was part of the Deming School. With regards to a systems implementation philosophy a quality improvement perspective means learn from what you know and get better. It's about evolution rather than revolution.

So by 1990 I had a very clear picture of what I was doing in Managerial Computing. I was building EIS and EIS was about putting LAN enabled computers in front of managers, analysts, and other knowledge workers. It meant having a strong set of productivity benchmarks to insure that the investment would be worth it. It meant taking what people know already and improving upon that by augmenting their intelligence. Not making the computer think for them. Not putting simple rows and colums of numbers on paper (hell you could do that with greenbar). Not saying everything you know is wrong, here's how we're going to re-engineer your function. EIS was part of the inevitable evolution of computing from all the knowledge being centralized in the mainframe to taking the knowledge from the people of the organization and leveraging their ability to do the company's business.

So for me the theory and tasks were clear. Now it was all about communicating that, and waiting for the technology to grow up.

Understanding IBM's OLAP Strategy

Alex Woodie goes further than anyone else in explaining what's going on with IBM and Essbase. There was some mystery surrounding what was going on recently when IBM and Hyperion announced a split. According to this story it appears that Hyperion just gets to have IBM's customers.

As to why IBM ended the relationship in the first place, it really shouldn't be much of a mystery. After 10 years, IBM had about 320 customers running its DB2 OLAP Server version of Hyperion's Essbase software on distributed platforms, and about 80 running the OS/400 version based on the product from SPSS, which in January renewed its deal with Hyperion that gives it exclusive rights to port Essbase to OS/400 and sell the resulting product. Considering that Hyperion and SPSS got a cut from IBM's DB2 OLAP Server sales, and that the products were essentially the same ones that customers could get from the original developers, IBM really didn't have much of a story to tell in this area and could never match the margins of its partners. A lot of effort was being spent duplicating efforts and complicating the support process, and that is no way to run a profitable company in this day and age.

So I suppose you can say it's more like rearranging the partnership, which makes this news release make more sense.

Woodie's original story clarifies what had been muddy, at least to my ears. The question seems to be whether or not IBM's formulation of the market is influential or valuable beyond its own customer base. I think that depends on how big those customers are and who gets to service them on the implementation side. Are they relatively active customers? Are they demanding? When they are demanding, do they get what they expect from 'The Other IBM'? Hmm.

The Cubegeek has heard very good things about Cube Views from my IBM source and I swear that data management is one of the biggest issues that I have in implementations. If Cube Views does the right thing, cross-platform, we may have a winner. Who's using it and for what?

It's hard to believe that the technology has inched up a notch in complexity and sophistication in the past year or so, but it's true and I expect that implementors, even professionals like myself, will be headscratching a bit over the next six months as MS floods the marketplace with FUD. One thing is clear. IBM is out of the OLAP engine business.

My nickel says that they have more than a little buyer's remorse over Alphablox. There may be a good number of implementations out there, but there's no doubt that the success of Hyperion Planning is bigger than anything Alphablox has delivered since their inception. But IBM is undaunted:

"Like any other business, we're continually evaluating our strategy. I have the job of business intelligence and data warehousing strategy. In this area, I'm not expecting [any large changes]. We're achieving the growth numbers we're trying to get," Wozniak says.

Those growth numbers are coming from IBM's strategy of providing infrastructure tools, and having the applications themselves come from partners. "I think we're pretty sincere about that. We're trying to make that strategy real and make it work, and the feedback we get shows that customers like IBM as technology and infrastructure provider and like being able to buy applications from a vendor rooted in the industry they're in," Wozniak says. "I think customers are pretty savvy. There has to be a reason that OLAP vendors are the fastest growing software companies, with several approaching $1 billion. We're interested in partnering with them."

There's only one difficulty with building a market on BI infrastructure, and that's very few customers and very few software sales guys have the patience and knowhow to work those deals. Throwing hardware at the problem must work for the ROLAP vendors which is why IT directors must swear by them, but I don't believe that the majority of the market is working smart when it comes to integrated BI. There's still a lot of gelling to take place and technology is still moving relatively quickly in BI and the BPM space.

I'm dealing with this precise issue with 64bit technology. It's ready, but not a lot of BI customers are. I suspect that IBM understands this and wants to shepherd customers over to big iron. That's a good thing because the appetite for multidimensional data has only increased. Let's see how well their strategy works.

BI Basics: MDBMS

I've been hanging out at the comp.databases.olap group for a hot minute and there seems to be something of a need for some terminology. I think this is a good idea considering that I actually have used both MSAS and Essbase (And Pilot Timeserver, but let's not go there).

There's Wikipedia for a start, but I feel more comfy editorializing here. I'm always going to editorialize because I'm all about results, and I've gotten my best results over the past 17 years with Essbase and I see no sign of that changing. So you'll please forgive my bias as I state it clearly. I have been using Essbase for 9 years.

Anyway, let's start with Multidimensional Database. I know this is a controversial place to start because there are basically only a few vendors who make them even though there are lots of people in the OLAP and DW arena.

As far as I know, there are:

Hyperion Essbase

Microsoft Analysis Services

SAS

TM1

Oracle Express

Neteezza

The rest are off brands. There's also a specialty indexing offering from Sybase, but then again, that's Sybase - meaning if you don't already know about it, you're probably never going to buy it. It ain't cheap. Same thing with Teradata.

A multidimensional database, is by definition, proprietary. There are basically two standards and none of them is ODBC or JDBC. They are MDX and XMLA. MDX is rather like an SQL variant, and XMLA is something I don't really understand. What I do understand is that both Hyperion and Microsoft support both of those standards as does SAS. This industry is not about a lot of standards, primarily because there is only deep knowledge about it in a relatively few companies. Nobody just jumps into the BI arena, although there are some up and coming Open Source players.

Multidimensional queries are generally accepted as the best way for users to navigate through numerical data which allows them to come to some analytical understanding of their business. That may not be categorically true, however there are some tried and true methodologies that support the industry. It's not just theory, it works.

All of the above listed products can be thought of as Multidimensional Database Management Systems, more or less. SAS, Essbase & MSAS are the fully featured systems. The others are a bit more specialized and narrow. I didn't mention SPSS/Showcase or IBM's DB2 OLAP because both of them are OEMs of Hyperion Essbase.

..And I Helped

Information Age has named Essbase one of the top ten innovations of the past ten years.

Hyperion Essbase was ranked in the top 10 along with Netscape, Blackberry, Google, virtualisation, Voice Over IP (VOIP), Linux, XML, the Pentium processor and ADSL.

Writing in Information Age's 10th anniversary issue, editor Kenny MacIver said: "Hyperion Essbase was the multi-dimensional database technology that put online analytical processing on the business intelligence map. It has spurred the creation of scores of rival OLAP products -- and billions of OLAP cubes."

I don't know about billions, but there's gotta be a bunch of them out there.

A this juncture I've been thinking of two possibilities with regard to the future of BI. Clearly the OLAP paradigm is alive and will continue, but you've got to wonder what Google has up their sleeve. So let us imagine that there's a way to tweak Google appliances to watch intranet traffic in a way that's referenceable. So meta-BI can assist. I'd watch their lexical parsing stuff for new possibilities even though I don't see a practical demand for it.

Secondly, I think we are just about to reach the point at which data mining (first visually, then with sampling) is going to become a bit more mainstream. The reason for this is that Essbase snd MSAS are about to scale past a significant cognitive barrier - which is all the dimensions you can think about. Recall that in 1992, a 1GB datawarehouse was considered pretty big. At least it did for the DB2 mainframe guys. Now that corporate IT understands what it can do, requirements are coming from the grassroots which are more sophisticated. (Finally). We are clearly at the point where best practitioners are not afraid of these requests, and DW shouldn't fail as miserably as it has.

What's going to emerge is the *real* path which is Active Data Warehousing, the stuff that Teradata has been all over. From this, we're going to get some archtypical models for analysis in every industry. I myself am publishing some models within my practice next month. We're getting an infrastructure guy and we're going to have some nice boilerplate stuff to merge up with Hackett's benchmarks. It's going to be fairly awesome as we start belting out models in a variety of customers.

I'm not supposed to talk about the next generation platform from Hyperion, but I'll just say this: The OLAP wars are over. The next 18 months is going to be the end of it. Now it's all about TCO and plug & play. That's what I said, plug & play OLAP. Or as IBM would say OLAP 'On Demand'. The learning curve is about to go flat.

Tableau Keeps Momentum

It's been a while since I've cranked up my n-dimensional data generator, but I'm probably about to update it again. Now that I'm playing with the new Essbase ASO, I've run out of interesting old outlines to test for performance. I have to think of crazy stupid models. So the next one I'm going to try will have three really fat dimensions, a half million customers, and 50k products and 30k locations. Plus I may throw in some attributes. Fzample, in my fakeData repository I've got 900 colors - although now that I think about it, I could have easily mocked up RGB combinations.

FYI, here's a snippet of code:

if ($ndims==3) {
for ($i=0; $i<$top[0]; $i++) {
for ($j=0; $j<$top[1]; $j++) {
for ($k=0; $k<$top[2]; $k++) {
$num = int($df[0][$i]*$df[1][$j]*$df[2][$k]*$adj)+1;
$dat = $dates[int(rand 36)+1];
$record = "$topdim[0][$i]\t$topdim[1][$j]\t$topdim[2][$k]\t$dat\t$num\n";
print OUT $record;
}
}
}
close OUT;
print "done.\n";
}

Once that's done, I'm going to hook it up to my Hyperion Visual Explorer and see what I can see. The next step will be for me to play around and download publicly available Census data and try some of those datasets on for size.

Wouldn't it be interesting if Tableau were already looking at the next level of visualization? Not that they have to, they've already been scooping up awards. Still, I'm trying to think of ways to visualize the actual sparsity of datasets themselves. I'm sure more ideas will pop up as I get into some super dimensional models.

That reminds me. The biggest crazy model I ever had to deal with was a survey database for Coke. They looked at dimensions like 'Purchase Container' and 'Consumption Container'. They paid attention not only to where you bought the drink but if you transferred it out of the thing you bought it and if you drank it there or not. At the time that I built it, 10000 source records blew the limits of Essbase, so we had to use very smart sampling. Today I could probably knock this out in a heartbeat.

Anyway I'm looking forward to building new massively dimensional models (like 30+) in ASO for demographics and whatnot and do my mining visually. Plus I'm going to try to do some visual mining of more traditional retail affinities with fat dims and lots of attributes.

Inside Dirt: ASO vs MSAS

The Cubegeek has gotten some inside scoops that are big news for Essbase fans. The first is open information which I have no compunction from sharing, but it is significant when you think about it.

Item the First: Hyperion & MDX
George Spofford, who is widely acknowleged as the Joe Celko of MDX, has been hired by Hyperion. In that capacity, he is responsible (in some way) for the commitment to MDX in the Essbase product group. It is thus very likely, especially given Hyperion's past participation in open OLAP standards (APB-1, XMLA), that Essbase is going to implement the full set of MDX functions.

Spofford's new book is entitled 'MDX Solutions : with Microsoft SQL Server Analysis Services 2005 and Hyperion Essbase'.

Item the Second: ProClarity & Essbase
I have it on very reliable information that ProClarity is very seriously looking at adding Essbase support to their product line. This is fabulous news to me, having been certified back in 2003. ProClarity guys are top dogs in MDX programming. Back in 2003, they were acknowledged as the people who had made the best (and only) wizards that generated optimized MDX. More properly said, they were the company, along with Panorama, that debugged the Microsoft API for them. I have a good feeling that they're not going to have that kind of headache when dealing with Essbase. I hope Miss B. at Hyperion goes after them so that they feel welcome into the Essbase Ready family.

There have been many times when I would have loved to stick Essbase on the back-end of ProClarity, especially when it comes to iterative development of new KPIs. ProClarity is a sweet front-end and its server layer is very good about handling things other than query stuff - like documents and spreadsheets. It's a low-maintenance middle tier and handles books of reports very well. Anyway, I'll keep my ear to the ground on this one.

Item the Third: Performance
A document exists that shows some performance benchmarks of Essbase ASO vs MSAS. I've seen this document with my own eyes, and it has names on it of real people - some of whom I know personally. Now this document is older than 6 months, and who knows what has changed since then. But the bottom line is clear. ASO kicked MSAS butt.

The notable notes that I took from this was that unoptimized MSAS is fairly crappy. Who knows where it is now, but even after MSAS was optimized ASO queries processed 40 times faster. There were, as always, qualifications on this. When the Hyperion guys say that sub-second queries are the only acceptable queries, parsing 'better' goes beneath the radar of ordinary mortals. However when it comes to benchmarking, this is computer *science* after all, not computer politics.

Additionally, the results showed that MSAS used a lot less disk than the equivalent ASO models. On the other hand the results showed that ASO scaled much better for concurrent queries. So on the whole, for little applications with few users, it's a push which makes cheap (free) MSAS the winner. When you scale up, ASO wins. That means you have to define 'scalability'.

The great thing however is that the what most people will want from ASO is that it does what Essbase used to be unable to do.

Multi-Path for Hyperion Planning

A rather surprising development in the aftermarket for Hyperion Planning. Here is an idea that we've kicked around a bit and will probably implement on a large scale.

The basic idea that helps me get my mind around advanced complexity in Essbase and Planning apps is that of Multi-Path. The concept is simple. The sum total of data that you may need to make a decision may have varying levels of volatility and accesibility. So diagram their data paths separately and manage their merging into one application using multple cubes and partitions between them.

We're always talking about 'disparate data sources' and ending our paragraphs with 'one version of the truth'. That's the case, but let's look at a baseball analogy with Essbase as the catcher. Most of the time, you're getting the ball at high speed from the pitcher. You have some control over that and your gear has you prepared for that all the time. Call that the main stream of data. But occasionally, the batter will popup behind the plate. That ball is a very important one to catch but it doesn't come very often. What do you do? As a catcher, you stand up, flip up you mask and move to the spot where you can catch it. Different timing, different expectations, different path. Sometimes the batter bunts. Then you've got to run up on the ball and barehand it and throw it to first or catch the advancing runner. That's another path.

Back in the real world, what we've done is integrate an Actuals Path with a Budget Path with a Forecast Path. The budget path goes through multiple iterations before it's stable. That's low volume user originated information. Forecast data uses some actuals data and does an automatic calculation based on variance to plan and then populates another scenario. This is automated from within Essbase. A second path. Actuals are read only and come through the ETL system. In this case we use an ASO cube which is marked read-only and we view it through a partition.

Using a separate cube viewed through a partition allows Planning to manage, in relatively small cubes, the Budget and Forecast data. We avoid long calcs during the overhead work of Planning refreshes.

It takes a little bit of thought and extra work to synchronize the models and manage the partition, but it does increase the scalability of Planning apps and vastly improves performance. Think about it. Just manage each of the data paths according to their requirements and deliver the proper results on time to your hybrid Planning app. Sweet.

ASO Love

They say that necessity is the mother of invention. Necessity has been a mother for the past week and I have found myself face to face with some detail of Essbase Aggregate Storage Option. I have emerged with a clearer understanding and greater appreciation for it.

In the presentation we made to our prospect, we found that there were about three singular criteria that made ASO the preferred choice in the context of their application design. The group we met with were zealously focused on performance, having lived in hangtime hell for several years in another non-Essbase system. Since they required write-back to ASO, something a few of us know how to do with some acumen it changed the normal singular criterion to three.

Normally, there's only one question you need to ask about ASO. Is this a rack & stack application? If the answer is yes, meaning you don't need any complex or procedural calculations, than go with the flow and roll with ASO. What do you get? You get basically everything ROLAP vendors have been crowing about since day one: dimensional scalability. You get it in two ways. In one way you get millions of members in your dimension. In the other way you get to use more dimensions in a single model than are humanly comprehensible. You want 15 dimensions? Go for it. You want 12 million members in a single dim? No problem. What I've heard back from the rumor mill at the Hyperion Sales Kickoff is that with 64 bit, models can get crazy huge.

Since we have an encapsulated app, we were able to design, fairly simply, a writeback capability which is similar to what we percieve is MSAS' current ability. I won't give away the details but basically we can do it with the additional restrictions that we could have about a 10 second gap between interactions against the written-back cube, and that writes be single-threaded.

That's it. ASO writeback is doable in that context.

Quite frankly, I'm pleased enough at the fact that we can use most of the Essbase API for running ASO models. Having developed several apps over the past few years with MSAS, I can tell you that developing dimensions for those models was a real pain in the butt as compared to using Essbase App Manager. And although I have a gripe (that I believe that Eric S. remembers) that I sorely miss the ability to cut and paste hierarchies from Excel directly into the outline, EAS fits the bill just right. EAS + the Olap Underground's Outline Extractor is a great productivity combo that makes developing cubes easy and fun, no matter how complex the defintions become.

I must confess, however, that I have yet to find any front-end I find as nicely ubiquitous as MSSQL's Query Analyzer and Enterprise Manager. Since I can whip up an ODS on the spot (and I usually have to) with MSSQL, I have all the relational support I need. However it would be nice to have equivalent tools for MySQL. I'm going to make a point to do some searching this weekend. (suggestions are welcome).

So now ASO is a closer friend, I'm going to make more use of it. It still seems like magic.