Cubegeek

Vault 107 Cloud Project

I have renamed The Terabyte to Vault 107 after my gaming experience with Fallout 3. And so into the next phase of my scientific madness, I am in the process of getting very chummy with cloud storage.

Obviously there are lots of ways to deal with these new paradigms of computing, but I'm trying to be incrementally logical about it, and get very familiar with one small chunk at a time. So for now the focus is on moving data, reorganizing it, optimizing its form, security and deduplication. With those things in mind, I will be experimenting with my terabyte or so of personal digital data that I have accumulated over the years and lives in various forms at the homestead.

On the back end I've got Amazon and Mosso/Rackspace as my vendors of choice. In the middle I have three intermediaries. They are Nirvanix, JungleDisk and Dropbox although Dropbox is something of a special case. On the front end, my sources, I have about 16 different hard drives, some spinning, some dormant. In addition I have about seven Case Logic folders full of DVDs and CDs. It's a nightmare. I should also add that I have Carbonite in the mix as well. It's backing up some 27-30 GB of my primary home computer. Plus I have photo archives at Picasaweb and Flickr to consider too. Needless to say I also have a family full of iPods some with DRM most without and about 800 CDs. I also have (shudder) about 150 DVD movies, another 100 VHS movies, 30 Hi8 videocassettes and 50 actual four track audio cassettes, some of which are over 20 years old. One includes some of my original DJ'ing. Yowsa.

My ultimate goal, as you might imagine, is to get all of that digital and digitizable material up into a cloud where it can be backed up, secured and made available to me wherever I am on a variety of devices. Chances are, you have a similar problem. At the very least, It would give me a reason to cut down on my insurance premium - because really who cares about cash if you've lost your digital stash? I want to make mine untouchable by disaster. It's a big investment.

Once upon a time, I thought that banks would get into this business which I considered back in 1996 when Windows 95 and Earthlink had basically made FTP available to the general public. When Jaz drives were all the rage, who would want to keep all of that stuff at home - maybe you had the same headache with Jaz as I did. Why not trust your digital assets to something like a bank? Even though we were all saying that 'disk is cheap' it still wasn't cheap enough and bandwidth was very expensive. Now it's all cheap. So let's do it.

What's not so obvious is that the great benefit of my doing this will put me ahead of the curve in the data management component of my day job. In fact, I'll be presenting some professional applications of this at Oracle Open World this fall, with any luck from the selection committee. But by then I will have had some experience with CouchDB and JSON and some other cool things. For now, I'll deal with dumb static data in simple formats and get some idea about what kind of muscle it takes to hump that up and down from clouds.

The first test is going to be of a product I own called Advance File Organizer. So far it has worked like a dream in dealing with the cataloging of every file on every disk. Of course I have been patchy in updating that master catalog. Through JungleDisk, I am able to map what appears to be an ordinary networked drive to an AWS S3 bucket. Once I get about 200-300 GB up into my bucket, I'm going to point AFO at it and see how it works. That will give me searchability in the cloud as if it were just another local volume. I'm next going to by a license of Tree Size Professional because the free version I have doesn't do network drives. And that will get me started.

Stay tuned.

Get In2Hyperion + Cloud Battles

There's a new blog on the block. Just a bit of horn-tooting. Please notice. It's called In2Hyperion. It's put together by the good, smart folks at Rolta TUSC and I expect that it will reward your attention. 

What's new with me? Well, I think it is inevitable that we're all going to get into big data, and big data is all going to live in the clouds. The coming battle, as I see it, is going to be the super IT folks vs the super cloud folks. It's a showdown that has been long brewing. There are going to be winners and losers, and the ultimate winners will be companies. 

The super cloud folks are those old heads who have had to work with massive scalability of the sort the most successful websites have had to use. The super IT folks are those old head who have had basically to manage outsourcing, capital budgets, traditional IT, and of course big ERP and big DW. If you ask me, they are two radically different perspectives with few skillsets that overlap. Here's an example.

This week on Cranky Geeks, the cranks assembled had the nerve to suggest that Oracle bought Sun merely to destroy MySQL, and that if the world has any justice then some anti-trust agency is going to force Oracle to give it up. I really, really think that's a laughable criticism. That's because I firmly believe that Oracle wonks have absolutely zero to fear from MySQL because it doesn't scale. Now I'm just going to accept that as axiomatic, because obviously some guys somewhere have figured out how to do some masterful multi-tiering perhaps with Hadoop or some-such. And yes there is this thing stuck in my head about hearing that a lot of web folks think that a million rows in MySQL is 'very large'. So I'll admit the prejudice. But here's what I understand very clearly about Oracle, they will not kill things that refuse to die especially when there's market share at stake.

On the other hand, it's difficult for me to believe that Facebook is running on Oracle behind the scenes. These web jockeys know *something*. Which is why there is a war coming for the enterprise dollar.

My interpretation of the Oracle Sun acquisition is that Oracle is going to have its own hardware and will get into the cloud business ultimately. That will leave HP to buy SAP or vice-versa. IBM will face its greatest challenge in five to seven years. There will be other players - likely Cisco if you ask me, but the great irony is that Scott McNeely's prediction (The Network is the Computer) will finally come true, except that he won't be running the company that realizes it. Intel will probably get into the cloud data center business fabbing up custom stuff rather like whomever used to make those big switches for AT&T way back in the day. BTW, there's a great scene of a switch in THX 1138 (minute 62). 

The advantage the cloud and web-guys have is that they know how to build for scale. The disadvantage is that most of their data is pretty dumb and they're not particularly interested in applying rules and discipline to it. That's the huge difference, from my way of looking at things, between corporate IT data and public web data. So the gap that needs to be filled, which may auger for a real evolution in enterprise computing is the judicious migration of critical applications to utility computing.

Imagine this. Oracle drops off a containerized data center at your company's data center. They hook up power and network and in a short time several of your applications have been ported over. The container hooks up to a truck and off it goes to Oracle's grid. You shut down your data center and never worry about capital expense again.

Agent Smith says it is inevitable.

An Argument Against Decentralization

I just joined a twitterfeed called bigdata. There is discussion afoot about IBM's new initiative to field a 4000 consultant army in Analytics. I think it's going to be a force to be reckoned with. In fact, it's something I've always hoped happened with Hyperion technology. But aside from that, a commenter named Hugo made the following point.

How many of the businesses who import into America use all these 'smart' systems? Very few. They have good management. Any American business that looks like they may need ERP (Enterprise Resource Planning) software, could ask them selves some questions: Have you tried splitting business units into units no bigger than 300 people? have you allowed those business units to make their own software choices, i.e a simple industry specific system price $100-$150 k, instead of $ millions? Have you checked the CEO is someone who understand the product, and is not a financial person? When the training budget is spent, is it mainly on the 5-10% in management, or are all employees included in some kind of training, including problem solving, idea generating? When these questions are answered, you probably won't need to waste money on expensive systems, and can invest the money in R&D, maintenance, training - and also renewable energy systems, so lowering break even.

I've seen what happens in big businesses that have decentralized systems in the pre-ERP era, and those who have avoided a centrally planned and disciplined systems approach. They are invariably hamstrung during economic times like these when the company needs to trim costs across the board. In other words, it doesn't help at all to have highly capable execs at the top of the company if they do not exert fiscal control down to the departments, and that is impossible without centralized, coordinated financial planning and reporting. I'll give you a concrete example.

One of the largest RFPs I've ever seen in this business called for the replacement of a system to manage 17,000 independent project budgets. Imagine a company so large that it ran that many independently funded projects every year. The opportunities for economies of scale may not exist at all on the systems delivery side. Surely as license agreements are concerned, vendors like IBM are going to be interested in premium pricing for their biggest offerings. Clearly there's a lot of consulting to pay for as well. But it has always been the advantages of economies of scale on the business side that have made these matters attractive to the buyers. Only a company can know how much savings they can realize through centralized planning and control. In the case of this RFP, we identified the potential for quickly making connections through that population of financial managers. Why? Because the corporation could always deploy an analytical group that could find opportunities throughout the system. In this case, buyers of titanium in 400 decentralized projects would never think to or be enabled to make a single coordinated buy, or do something as simple as multi-supplier comparison shopping with stovepiped specialty systems.

Such efficiencies became immediately obvious as we put together the design for the system - it was a home run built into the prospect of a single-vendor enterprise class solution. But such things are not always so obvious.

Here's a second example, one that I've told many times. I worked, back in 95, for a manufacturer that had a nationwide product and its own distribution network. I worked very closely with the CFO and his top analyst. He grew up in the business, she was an ice-sharp Ivy League MBA. In an afternoon they started to look at the books and try to squeeze costs out of the system. The analyst found a contradiction in the matter of the way discounts were booked in the ERP system. I confirmed it by looking at transaction types that were my feed to the dashboard. The CFO immediately got on the phone to several warehouse managers.

It turned out that there was a discount business rule tied to a customer attribute. If you were a good customer, good defined by total tons shipped per year, you got an automatic 10% discount on freight. But we discovered by looking at the aggregate numbers that something was amiss. In one afternoon of phone calls by the CFO, it was discovered that warehouse managers marked their discounts different ways. In particular, the most predominant way was that they took 10% off the fuel costs per delivery.

So here's where a smart manager who knows the business makes the right call. He named a customer and asked the warehouse manager if a shipment was going out that day to one of the big customers. Yes. How many pallets onthat truck? 1 and a half, was the answer. The truck could handle 10 pallets, but was taking these small shipments on demand and discounting the gas for every trip.

The CFO turned to me and said, I want you to build a system that shows the difference in our profitability when we send half-empty trucks out to customers and change the incentive to warehouse managers so they can see it come out of their bonus. Because today they all think that if they send a truck out every day or every other day it improves customer satisfaction. Yes it does, but it's killing us on fuel costs.

I had to shrug, because it was 1995 and nobody put EIS capable PCs in the hands of warehouse managers. Warehouses were not networked and doing so would require Citrix, very expensive in those days.

These days it is not very often that the lethal combination of line managers who grow up in the business work in coordination with hotshot analysts and soup to nuts developers that can deploy systems  into nationwide networks. I think those times are going to come back when cloud-resident systems become a reality. At that end of this evolution companies like IBM who field large consulting forces are going to be at a distinct disadvantage. In the short term, they will have the advantage.

Bumptop & Manual Labor

There's a new cool pen-centric GUI out there called Bumptop. I just learned about it half an hour ago and it reminds me of Glasstop, my BI centric 'surface computing' multitouch metaphor. 

But here's the real kicker. The more I think about Glasstop and reread it, the more I think about the possible conversion of computing to manual labor metaphors. Here's the short version of the thought. When the Wii came out, a bunch of people thought it would be a good idea to transform all of the gaming metaphor to that. I said no way. I now have a Wii and I don't like it but primarily because I haven't bothered getting a grownup game on it. But the real objection I had was that I like games that exercise fine motor control. There's something about a driving game or a first person shooter that allows me to be very precise with my thumbs and I like that. And of course if you talk to any PC gamer about consoles the first thing they will tell you is that there's no way they can exercise as much control over their avatars with their thumbs as with the combination of mouse and keyboard. 

But most of this is, when you think about it, a very small subset of what hands can be trained to do. Which is why I think about assembly line workers and musicians when I think about the future of computer GUIs. If and when we perfect some fine and large and quick motion capture for the hands - which I think will happen first with multitouch and multitouch plus pens, we are going to really revolutionize the kinds of things we can do, thinking more like musicians do in dealing with data.

For me it's about dealing with data, with diving through and manipulating huge datasets finding patterns, sorting, simulating, issuing commands, putting together sparse pieces of information from a broad selection. It's what I do now, but with very simple typewriter & television metaphors in browsers, spreadsheets and query tools. 

While I'm speculating about the future let me pull in one more idea which is something somebody said today about the problem with Google and generalized search. They keep trying to contextualize free text, and it simply is not going to work in the long run. In the end, we really want to be spoon fed by experts. Please keep that in mind. Google is a giant leap forward, but it's free for a reason. It can be. But debriefing a CIA agent will always be expensive and exclusive for the same reason sitting in a classroom with a real professor is more expensive than just getting the syllabus and the books. A lot of the computer assisted stuff is going to be free. 

Search is commodity. Human structured data in KM domains with specific UIs is the high end future. Ask Bloomberg.