Cubegeek

Information Thermodynamics - It's All About How

Ron Dimon asked me, why is 'How?' the most important question. I posted 'The most important question is 'how' on my Facebook page. So how and when am I going to publish what's going on behind that? I can't answer that directly because I've been admonished by my brother's advice which is not to tell people what you're going to do, but do something and then tell people what you did. So I'm telling you now how I responded to Ron.

--
Well, here's the context. I got notified by about.me that my profile was getting some hits. So marginally interested in that avenue of publicity, I went to check out the site. It turns out that they've added new functionality - something called 'Backstory' in which you basically turn about.me into some artistically satisfying version of LinkedIn. Redundant perhaps, but there's always another possible spin and audience. (And I'm upping my Twitter presence finally).

So I muddled around it and though I could at least put together a quote. But what I wanted to say, I haven't heard anyone quite say. It has to do with my theory of 'information thermodynamics' which goes a little something like this. "Knowledge, like matter, cannot be created or destroyed without the expenditure of energy. However like radioactive matter, knowledge does have a half-life which sputters away its substance." Not a very moving quote but pretty much what I have discovered. But the emphasis is on the energy required to sustain high levels of knowledge. IE just because a certain population knows something doesn't mean they will always retain and use that knowledge. All this is to counter the common concept of "The genie is out of the bottle". Yeah well the wind can blow the genie away. It always needs to be sheltered.

That line of thinking reminded me of the things I think are the most fascinating things existing in the world today. 1. Nuclear fusion & weapon systems, 2. Cryptography & Spycraft 3. Actuarial Economics & financial instruments. All of which are very knowledge intensive and difficult to understand. Which led me to the final matter of what is the most important thing to know about those three subject areas which have world shaping consequences. It seems quite evident to me that most everybody knows, what, who, when, where and why of all three of these - but only a select few know how.

Having spent many years in the BI world, I jumped over the fence from an Oracle loyalist (more or less) to DevOps, Cloud and Open Source about 4 years ago. We are way ahead of the market, but we're learning a lot of How. Once other people figure out their whys and whens, we'll still know the how, and that will keep us in business.

I've missed ODTUG since Monterrey and thought about when I might stop doing and start talking about what Ive been doing, but I'm still fascinated by the How. But I will be in Vegas for AWS Invent. The 'publishing' is just me talking. For now.

--

BTW If you see me in Vegas or anytime, ask me about VZ115. It's a great proof of what makes us at Full360 world-class.

The Answer for Apple Television

OK. I know what it is. Call me intuitive or foolish but here's my bet. My bet is that the revolution of television is going to be what the revolution of iTunes was. iTunes was basically digital distribution of every profitable song ever recorded that might sustain market interest today. iTunes for TV is the same thing. 

Get it?

It's all about getting the Jetsons and the Flintstones and every other property into the Apple distribution, and ultimately superdistribution. Tim Cook is going to come up with the market, soon to be joined by everybody else (meaning Amazon) that is going to allow Apple product users the ability to turn up the Wayback Machine and find their 32 favorite episodes of I Love Lucy. He's going to make an offer that the heirs and scions of Desilu Productions cannot refuse. It's a big data solution to the last holdout, which is the gigantic library of syndicated television. And he's going to show them how it has been done with music, including AC/DC and The Beatles and it has made those rights holders more money than they could have ever expected through a planet full of Tower Records stores. Apple has the platform, now Apple TV is all about managing the content. 

That's what Apple TV will be. 

The actual hardware is basically already done. There's not much that needs to be done to take that tiny thing that is the actual product called Apple TV right now and embed it into any set. In fact, Apple's probably spending lawyer time trying to keep Samsung from doing that right now. What Apple can do is negotiate the rights for the back ends.

Bucketmaster

Bucketmaster
M. Bowen
November, 1999
---

This is the first in a series of tools to enhance the data-management capabilities of essbase formed by a suite of backend tools which will enable the e-business.

The Bucketmaster will fit into the context of API based tools which work on essbase components. The should be developed outside of the core product as if they were 3rd party tools. They can later be integrated as options enabled by our license.

The purpose of this tool ist to perform a one-off or two-off analysis of a stream of dimensional data. It should allow dbas to quickly bucketize a flat dimension into either a dimensional hierarchy or a virtual attribute dimension. The source for this can be a flat file, or other stream directed to it via the rules editor, or an actual built essbase index, against which the sparsity of a single dimension can be judged. It should also be able to run directly against relational sources.

The overall goal is to give the multidimensional dba the opportunity to optimize design of the essbase database based on real data provided. This analysis should be able to be performed before or after loading data into Essbase.


Part of the Bucketizer is a Tokenizer. This tokenizer should be able to create a surrogate key against a single dimensional source which can be output back into a relational database or flat file. This will allow Essbase to quickly process odd files.

Outputs of the bucketizer will be:

Single Dimension Operations - Against Counts

1. an 'equal bucket' flat line distribution of source keys in the dimension with equivalent sized buckets in a 1 to n distribution.

2. Size constraint distribution of source keys in the dimension. Buckets would all be of equal

3. Bell Curve distribution of source keys in the dimension.


Single Dimension Operations - Against Measures

or the same thing against a second dimension.

2. A bell curve distribution in the form of a dimensional hierarchy or virtual attribute dimension.

3. An 80/20 (adjustable) distribution of dimensional data against a pre-defined or dynamic measure.

Dual Dimension Operations - (Sparsity Managment)

4. a graphical output of the distribution of data within an essbase cube for a two dimensional combination, or the same against a relational source. In this regard I am envisioning a 2dimensional field with red dots signifying the hits within the field. This needn't be done by measure but, color could represent a third dimension. As well it could be done in 3 dimensions which could add some value as a visualization tool.

Choices Choices

Python vs. Ruby: A Battle to The Death from Gary Bernhardt on Vimeo.

The Basics

This weekend I picked up a copy of Kernighan & Ritchie's Second Edition, remembering that I actually never read the first, although I should have. In my twisted career, I was the guy who was helping my friends study for the LSAT or doing their COBOL finals, but I never got to see the Sun workstation under my nose. That was college. And then I remember selling my AT&T/Olivetti machine to my buddy before I could get a C compiler. And then I mastered a few 4GLs and nobody was doing UNIX in the business world. So through a bizarre series of events and coincidences, I have never had a C compiler of my own, and thus no C programming experience. That changes now.

On first glance, the XCode compiler is massive yet deceptively simple. I remember when you stuck directives in source code to link different memory models for the build. Here there are completely different frameworks under which code is developed. For the short term, I'm working with the CLI model and going through K&R.

My goal is to be somewhat accomplished in a few years. I think that's achievable, and I'll be posting notes here along the way. I expect that my discipline for working in tiny cramped computing spaces for the past n years will make me a very efficient programmer, like a Russian. We'll see. Reading up on the iPhone developer's cookbook from Addison Wesley, I've been told that I have to work within 20MB of RAM. For me, given the state of the industry when I last did serious programming, that's a whole lot of space. I should have a ball.

I'm moving from being a data architect guy towards developing some API competency, and I have a feeling that I'll enjoy doing it in C more than in Java. There is nothing more aggravating to me than the stream of gibberish that ensues when a Java program crashes, but I may learn to deal with that too. 

Open Source Property Implications - Part 1

I've begun reading 'The Success of Open Source' as I try to get my head around the idea, or the fact, that open source databases have leapfrogged enterprise engineered databases in many important ways. It's a disorienting feeling I have, so I need to reconcile it with reality. That's the thrust, but I think there are important implications as well.

If there is one thing I've always liked about my industry and also hated it is the way in which programmers have been organized to work and create wealth has managed to be casual. I thought about this today while looking for a bathroom that was not being serviced in the spanking new building where I'm working. And so I'm curious to know some of the underlying principles, if there are any.

You see the building I'm currently working in has been architected for collaboration. Nobody owns a desk, anybody can use any 'cube' which is more like an open plan workstation, to dock their laptop and do whatever it is they do. There are lockers disbursed through the building so that if you have stuff, that you can park it securely as your itinerant work takes you wherever in the building. There are small private rooms with phones and glass doors so that you can do a conference call. Some are as small as a large telephone booth, others can seat three or four folks.

Now the unwritten rule of this architecture has to be written, and you will find spontaneously printed notes all around the joint reminding employees that camping is not allowed. In other words, it defies the spirit of the collaborative environment to claim one of these small private rooms for yourself. But enough people have camped for the signs to become necessary. I even noticed one printed in large red letters in the elavator this morning.

This environment comes complete, as one would expect in Silicon Valley, with an area with beanbag chairs a ping pong table and subsidized soft drink machines. It's hardly what one would think of as panoptic. Yet despite its thoroughly contemporary style you'll still find a catering truck pulling up to to the front of the building every day at lunch. I've been to a large number of companies in my time, and I would argue that this particular campus is on the more advanced edge of the IT high tech sort. In the building I visited yesterday, there are no corner offices. In the corners are conference rooms. Everything about the place says, young, smart, tansformative & green. Yes green. Outside the front door of my building is a solar powered trash can. Don't ask.

Is propriety at risk here? Yes, a little bit, but I cannot tell to what extent. My immediate reaction to the no camping sign is split. One could easily be chagrined that old habits die hard. On the other hand re-engineering is not necessarily quality improvement. To work in a building that encourages collaboration and play is a hallmark of the new style of business organization you find in Silicon Valley. But is hierarchical organization necessarily bad? Is the desire to be territorial wrong? Is it necessarily a good thing to have a rumpus room in a corporation?  I think it all depends at root on one's understanding of the ways and means people are organized around property and how that is perceived by workers and their management.

For example. If I understand the open source movement correctly, it produces a high quality product because the inner workings of products are subject to arbitrary and massive scrutiny. In architecting a building where people work around that same principle, you don't own your office because your work should be subject to arbitrary and massive scrutiny. Camping implies propriety, of keeping to yourself, of privacy.

--
The downside of proprietary engineering development that I am intimately familiar with originated with my experience at Xerox. Xerox fumbled the future because it slavishly attended to its customers - customers who were not future-oriented. Likewise when you make your money designing to spec in a bespoke manner for a limited set of customers you are likely to satisfy them very well but you will be inefficient for the masses. Open source takes advantage of the masses and a new arrangement between customer and provider. So I am aware how propriety inspires loyalty and trust and how loyalty and trust can be the enemies of innovation.

There are great implications about this new kind of property which is inherent in the open source movement, but the manifestations of the broader social implications are just beginning to appear.

Double DBA Insider

This week I got a call from one of my colleagues on the East Coast. Apparently some Federal agency needs a bit of assistance and is kicking off an interesting project. I actually like the East Coast but am working to build up a few things in my backyard. He asked me of my particular expertise, as we hadn't yet worked together and I filled him in. It wasn't until I got off the phone that I realized something. Now I'm an insider.

One of the primary reasons I have not expanded my relational DBA and architecture skills is because despite the fact that I have the background and basic capability, I have never been given the keys. I can recall numerous occasions in which people on projects would be stuck and I would be finished - 'and if we could only clone you...' (and get me for a lower rate) everything would be cool. I would twiddle my thumbs waiting for them to find the rare and wiley Informatica expert; when they finally got them on board, I would tell the guy what to build. I would have a logical model of exactly what I needed the feed from the data warehouse to be for my ODS and staging areas and I would twiddle waiting for their DBA to materialize and then materialize some views.

And so for a guy with as many years as I have in RDBMS, considering I did my first subselects and star schemas in 1991 on DB2, you'd think I'd know a lot more about VLDB, partitioning and various other fine points of physical database design. But I don't. But now I'm an insider, and I can pick all that up again and then some because the guys in our shop are original gangsters of Oracle. They knew Scott when he was a tiger (and there's some old school inside Oracle for you).

Life is interesting.

Snoop Poddy Pod

For Schneier's Movie Plot Contest, my first entry.

A recent study at the University of Wisconsin shows that 75% of minor children with iPods listen to inappropriate music. What's on your child's iPod? Think you know? Think again. Teens can easily alter the names of the songs on their iPod. Now thanks to a revolutionary new product, Snoop Poddy Pod, you can know exactly what kinds of messages your child is subjected to. Our Snoop Pod database analyzes the digital signature of thousands of offensive songs, updated daily, and checks to see which of those are synced to your child's iTunes. Snoop Pod parents are sent a weekly confidential email with the real scoop. Take back control with Snoop Poddy Pod.

Reversal of Fortune

It's official. I'm moving back to the Essbase world.

It has been about six weeks now since I have moved out of my old digs where I was VP of Services for a Microsoft partner. It was an invigorating experience, rather like a shower that ran uncontrollably hot and cold, alternately luxurious and chilling. I'm sure that I learned a few things, but I haven't bother much to decontexualize them. As folks have been speaking to me recently about my change, it has been difficult to express what has been good or bad without speaking out of school. Nevertheless I did have a good analogy that I used a bit while I was employed there regarding my disposition.

Half of my job was gardener, the other half of my job was catcher. So I wore two gloves, one for tending to flowers and fertilizing, the other for catching fastballs, curveballs and the occasional spitball. Occasionally I would use the wrong glove for the wrong task and a flower would get clobbered. More often it was the other way around and I would get seriously burned by the pepper. Builds character. Of course I wish all of my former employees and colleagues all the best, except for one, and that's all there is to say about that.

Today I'm completely decompressed and back to running on slightly modified versions of my old anti-Microsoft bias and gripes. There's a part of me that cannot wait to get back into Unix, and especially on the Macintosh platform. There's another part of me that cannot wait to get back to big hunkin' enterprise customers and their money and sophistication, and there's another part of me that is almost ready to get back to heads down, hands on work, but not really.

As it turns out I'll be doing that hands on in the short term and I'll be working on front-ends for a change. The thing that has really got my attention is, as I've said, multitouch and new visual paradigms for business intelligence. I've decided that 'glasstop' I will now call 'tabletop' computing and that's what I'll be working on as I come up with some theories and product ideas. They will grow to technology and technique ideas as I get my hands on a Mac later this year and start learning UI programming. This very simple application by CNN of Perceptive Pixel gives you an idea of what I think will be irresistible to BI customers in the next generation.

Immediately, however, I'm going to be working with Tim Tow of Applied OLAP on his new Dodeca product, which is a soup to nuts .NET smart client whose features I helped brainstorm a couple years ago.   It's a comprehensive development environment and the most sophisticated thing you can put on top of Essbase. Tim and his team are world class and I'm really proud to be working with him. In fact, I just bought 3/4 Terabyte drive just to celebrate the data mongering possibilities I will start to exploit as I get back into serious Essbase.

I have to say that in my career so far, the project that helped generate some of the Dodeca features was my most extreme and extraordinary. It was conceived to replace the cost estimating system for a global 100 company that allowed 17,000 Essbase models to be living all at once. That's was not centrally maintained and driven by a DW, that was with 5,000 cube modelers, with dimension building access who were generating their own data. It's the one that got me up with the HP folks in Cupertino running Essbase on the big 64bit Superdome where I did some screaming performance stuff with parallel java servers and the whole nine yards. When I tell you this front end is the greatest harness ever for Essbase, I kid you not. From what I've seen and know about its capabilities, it's everything that Alphablox and Arcplan promised to be, but better. Most importantly, It allows you to build custom workflows, custom forms and custom distributions of all that with more security granularity than any other front-end to Essbase. Again, all in a secure .NET framework. The guys at Applied OLAP have rethunk client distribution for the BI world, and everybody who is doing anything at all in Excel and VB really needs to check it out.

Only Dodeca enables this kind of Essbase farming. There's nothing else anywhere that scales as an application framework for Essbase. Did I say that it does distributed master data management too? Yeah. That's part of it. The really fabulous idea that makes this even more interesting is something of a future, but you know me, that's where my head goes. The fabulous idea is that you can begin to think about the move from unstructured data to multidimensional structured data because Dodeca is going to make it easier to do so on a different scale than people are accustomed to. But that's all I'm going to say for the moment.

It's wonderful to be as old as I am and still get excited about stuff.

ADP Screws the Pooch

Back in the summer of 1988, I went on vacation to Mexico with the woman who, unbeknown to me at the time, would eventually become the Spousal Unit. We hung out with a lot of folks from Texas, which was something of a surprise. it was my first vacation to Puerto Vallarta, and I had some strange notion that only Californians went anywhere in Mexico besides Cancun.  Well, one of the couples we hung out with was in some business whose details I forget. But our better halves both complained that it seemed like pulling teeth to get us away from work for a holiday. He explained that the toughest thing about being lead dog in a company was going to bed worrying if you were going to be able to meet payroll. I never forgot that.

Fast forward to a couple months ago. My company did payroll through Wells Fargo. Somehow Wells got involved with Intuit and then they did payroll. We decided to switch to ADP - the experts. Or so it seemed. Last pay period, they engaged when they weren't supposed to - so all of our employees got their paycheck doubled. This month was supposed to be their first payroll. Guess what? Nothing. All of their systems are down today, and none of the direct deposits went through.

We have been told that this doesn't just affect our company, but everybody that ADP services. Lovely. Here's to you ADP. Get it right please.