Cubegeek

Stepping Out of the Elephant's Shadow

Hadoop.

Where do I begin to tell the story? How about we go from the epiphany backwards? This morning I read this from Dave Sisk. 

 Hadoop is not even in the same ballpark as any of the CDBMS's that have been mentioned. It's not even a database, for that matter...it's a giant MPP ETL process...which is a great thing IF you use it as a giant MPP ETL process instead of trying to use it as a database. I've examined HBase closely, and looked at a colleague's implementation...it has to be the only key/value store that I can think of that is a worse piece of crap than MongoDB. (At least MongoDB is better than something.) My colleague's company's 60-node implementation of HBase (on big honkin' enterprise hardware) struggles to insert 2000 rows of data second (I can insert at 10-20 times that into PostgreSQL running in a VM on my friggin' laptop), and reports run for hours (sometimes days). You can do the same work in a good columnar RDBMS 2-3 orders of magnitude faster...as in milliseconds or seconds (minutes at worst), instead of hour or days.

At a prior company, we used Vertica to consume hundreds of thousands of rows per second, and could return results from billion row high-reduction queries in a few hundred milliseconds (from about 5 hefty nodes)...Hadoop/Hive/HBase with hundreds of nodes could not come within 2 orders of magnitude of that kind of performance, no matter how much hardware you throw at it.

That is what I've been waiting to hear for years. So I wrote back.

Finally thank you for helping me understand what I didn't, that there is a massively huge performance gap between Vertica and Hadoop. I've been wondering why I never meet Hadoop folks in my space (Business Intelligence) and I've been listening to the guys at MAPR tell me theories of why their Amazon-approved version of EMR is going to be a world beater. I have assumed that all Hadoop is simply a very large scalable file system (and I've started to call it HDFS) + some clunky tools that lack a semantic layer. But I assumed that a reasonably competent Java programmer (who wants to be that?) could make it perform at a good clip. No matter what the performance, I expected that it was mostly ETL class tech. BUT I figured sooner or later somebody was going to build a semantic layer of SQL onto it and then it would be serious competition for columnar DBs, primarily because of mind and market share. 

However, if Hadoop is little more than a glorified giant file system and map reduce is to HDFS as regex grep sed & awk + perl is to ordinary file systems, then there's no way it will ever compete on performance and cost efficiency to columnar tech. 

I'm going to assume HDFS is a data lake from here on out, unsuitable for BI queries.

That assumption means a great deal in my world and it erases a class of insecurity I've been having over every discussion of 'big data' I've been a part of for several years. You see none of my customers have Hadoop or ask for Hadoop. All of them can form the syllables. They know about Hadoop. Hadoop *is* big data, as far as the layman's world (and another world that is not BI). So I haven't had a *reason* outside of great curiosity to build anything with Hadoop. 

First of all, I have S3. There's no bigger data lake I've ever needed. Terabytes are not a problem. I mean I've got terabytes at home. But under ElasticBI, I've got S3 whipped into shape and smartly integrated to every database I care about. (Vertica, Redshift, Essbase, VoltDB). So I never have to concern myself with running out of space for staging or moving massive amounts of data.  It's always about database optimization. That requires structure, structure requires purpose, and I know how to get that from my customers. Hadoop is about unstructured data.

Now when I start talking about unstructured data, I mean web data that has a volatile structure. And in that space I see tools like MongoDB, Cassandra and CouchDB with Couch as the winner. I've heard horror stories about Mongo, and that Cassandra is a big tease that never quite gets all of her drama together. There's also Riak which is heavyweight champion in the space, so I hear and believe. But I'm not building ecommerce websites and I don't need to manage volatile content and serve up XML to be rendered, so I have no need for that class of data management. Not right now anyway. I want to be the master of all data management, data store and database worlds, but I have to deal with one continent at a time, and I can accept that Riak and Couch are on another planet. Planet Unstructure. 

But when they say they're going to put on SQL clothing and take their big data + analytics into the Data Warehouse realm, that's war of the worlds. I don't like the prospects of that imminent invasion. Because, really, websites are the masters of handling 10,000 simultaneous concurrent users. I can't do anything like that. Those guys must know something. 

Well, so did LAMP stackers at one time. I'm going to take a gamble and commit HDFS and all that manipulation to the back porch. It's not competition in BI, and it never really was, but that was really in my mind a matter of market focus. Now I understand technically that there are hard limits to what all of that can ever do and strong technical reasons why what I do with my database tech is not threatened by these other systems.

I still want to know. I still want to spend some hands on time and eyeball some NoSQL tech in the context of their own systems. I wish I knew a guy, personally. But maybe our paths haven't crossed for good reasons. Either way, I am stepping out of the shadow of the elephant.  Hadoop is just a data lake fallback for collecting a lot of stuff that may or may not be map reduced into something coherent. It's a specialty ETL transform that we at Full360 will do with realtime streams.  So maybe if my Colorado River data pipeline fails, it will form a Hadoopified Salton Sea while we fix the levee. What it most definitely is not, is Fast Data. Meaning it has no place in the IoT future and is an artifact of what we will inevitably call something along the lines of JBOD. Lakes are meant to be drained. And now I've completely introduced heterogeneous metaphors into this. But that's kind of how epiphanies work, neh?

I'm out of the shadow of the elephant who drinks up lakes with that weird trunk of his. Yay circus tricks!

What IBM Should Do

If I were in charge of IBM this is what I would do. 

First thing to do is get rid of PWC. You have to admit that these guys were hired simply because IBM software and processes were too complex for customers to implement on their own. You have to ask yourself how did Facebook succeed without an army of consultants? It succeeded because the software made things obvious. That is what you can do with advanced web design. So what IBM has to do is change the way they think about software and systems implementation. Here's my idea. 

IBM cannot compete with Amazon,  Google or Microsoft Azure in the pure cloud space. They are already too late to attract new customers and all They can do is convert their installed base. And if that's not giving that installed base a price break then they cannot improve the revenue. IBM cannot compete with Intel; the PowerPC chipset is dead.

So what IBM has to do is go after the fractured midmarket and leapfrog all the small competitors in that arena and consolidate that business. This will be all about cutting off the high end from Apple and Microsoft and then eroding the market share of SAP and Oracle. But IBM also has something that midmarket software vendors do not have which is lots of capital. This means that IBM can put cells physically in markets where small competitors are at a loss. More on that in a minute.

The place where IBM's going to make cash is by competing in broadband. Google fiber is going to be a competitor, But I think an IBM Cisco partnership would make mince meat out of Time Warner cable Comcast and seriously put a hurting on Verizon. Broadband is a cash cow. IBM can makes sense of new net neutrality work rules and lock out any other smaller players in this market. 

So what does this mean? This means in any town with more than 50,000 people IBM will have a branch that branch will be like a WeWork / CoLoft presence. It will attract all the young programmers who don't want to go to places like PWC. It will give IBM first crack at rewriting all of the midmarket applications which would be no-brainers for these young ambitious programmers. Build next-generation software for the mid market and let them leapfrog all the legacy systems. This gives IBM new face, a cloud face, for all of their current AS/400 legacy systems. Mass migration. Lift and shift. IBM's first purchase? Workday. IBM's next purchase? RadioShack store fronts. IBM needs to keep its mainframe business for a time, but shed its other branded hardware server business faster than Dell and HP. Everything is apps. IBM has the advantage. 

Put IBM into the mainstream of the midmarket and take it over. Build next-generation cloud capable applications with more customer responsibility then Amazon or Google are ready to take. Run that section of the business life PeopleSoft used to be. Serve coffee in the IBM workcenters. Cohost meetups. Make it your friendly neighborhood IBM. Get into fast cycle agile development of mid market applications that scale. Apps for business. Same approach as AS/400 with next-generation technology and a new wave of developers, excuse me, 'devs'.  Here's the trick. Do an open source of IBM Research algorithms to the IBM cloud and a small license fee to off-IBM cloud implementations. Buy Raxspace if you have to and host a second kind of open-standards cloud. Two cloud models, one company. Only IBM could do that. 

There's a question about salesforce.com. And that question is whether their API is sufficiently robust to compete with the APIs of AWS and Google. Here I'd make a side-bet on Apple Swift. I would bet that Java is dead. Let Oracle have it. But I would also make bets on Python and Ruby. I would also make bets on GoLang because it's a good thing for IBM to be totally open, however, Java has too much legacy crap that existed before the era of clouds. Make a fresh start.

At some point let's say four years from now. The functionality gap between the new midmarket apps and the Legacy IBM mainframe apps will become clear. My bet would be on the 80/20 rule that scaling up the midmarket apps will be cost beneficial to current mainframe customers. The 20% that can't migrate now pay a premium. The rest they bring over and lock out the rest of the market with lower prices. This will put pressure on SAP and Oracle and revitalize the 'enterprise' space. The enterprise space is now a web-beautiful, neighborhood-centric dev's paradise, and those young hungry programmers get a crack in SAP and Oracle at Microsoft and Google and anybody else who tries to enter the business app space. And oh by the way IBM's broadband will now be at an appreciable market share, in the new world of net neutrality regulation. ROLM, anyone?

Digitalization & Transparency

McKinsey has several analysts who publish on the subject of 'digitalization'. The basic idea, to which I subscribe and find compelling, is that IT can be transforming the way that businesses interact with their customers and with the public. 

Generally when you say 'IT' however, you get this (old) idea in your mind about a staff of managers and programmers supporting the internal network, enterprise class servers, 'critical' applications to a largely internal, intranet captive audience of employees and contractors. 'Digital' incorporates social media and industry standard reporting systems, even beyond CRM, which is about as far as most companies go to tighten their relationship with declared, paying customers in the marketplace.

So my POV with regard to Digital means the holy grail of transparency.

 

Imagine the nightmare scenario. Your company is on the witness stand or testifying before Congress. What does the entire planet want to know? They will always ask "Who knew what and when?" That is the most difficult question any organization can face, and it generally requires a level of systems integration that very few organizations can muster. However, it is fundamentally true for the most high stakes information games on the planet, with commodities & securities trading systems, with military communications, with first responders and with national security intel systems. 

Knowing who knows what and exactly when in an auditable system, real-time, is the most advanced IT systems that exist - at least to my knowledge. Most businesses are light years away from that level of systems integration. McKinsey perceives that and knows that any business or organization that can advance their systems in that direction will have a huge advantage. 

I try to make my living with this kind of thing in mind, and yes I do analytics. Let me give you another example. Plane crash. Right now, the capability exists to stream flight recorder (and other sensor) information realtime and have enough eyes watching so that anomalies can generate alerts and produce corrective action. However the airlines have not decided to spend time and money on making those advances. There's nothing a traditional IT organization could do to cost justify that. However a forward thinking organization sees the advantage of digitalization transparency.

Programmers Are Not Quiet People

Back in the 80s when the world was new, I was a programmer. I would engage my professors and they would exhibit something I call programmer's dyslexia. This happens when you ask a good question that has many implications.

A good programmer will always think of the 0 case, the 1 case and the infinite case. Any calculation will fit into that and so we are always quantifying how much it takes to grind out an answer. But that also means we have to categorize our assumptions in order to come up with a reasonably proper answer. It was easier to ask such questions when the world was new, but now a lot of them have been asked and answered (I leave it to you as an exercise to qualify the query space of the web itself).

You should ascertain by this that the quietude of good programmers is born of respect. You are either asking a question with a trivially easy answer and so we are redirecting insults to our internal dev/null, or you are asking a question that needs greater specificity or you are asking a well-qualified question that requires significant thought. The second type of question can yield great amounts of humor.

For example, my wife has the habit of asking me questions like. "Honey would you like me to fix you a snack now or can you wait 45 minutes to dinner?" Now to me this is a simple yes or no question. But it requires a yes and a no. So I will say "No, yes", and then she will become confused. She wants me to say something inefficiently verbose, like "I'm kind of hungry right now but if you're fixing something really nice for dinner then I'll just grab myself a beer and wait." To a programmer, "No, yes" is the proper answer, to my wife and the rest of humanity, we are just being cryptic assholes. So sometimes we have to refrain from using our own unique sense of humor and send those responses down another path to the same mental bitbucket. That path often triggers a bit of emotional resentment.

Those of us who adapt well understand that and so we have developed fluffy verbose social skins over our lean, mean logical question parsers. But we often have to worldswap and figure out our contexts to check which skin to implement. Depending upon our environment, this takes a noticeable amount of time. Some of us even attempt to build human interpreters on the spot because the questions we get are so stochastic and meaningless that we simply don't have a map to the proper context. Sometimes it's easier to put on the 'asshole' skin (c. f. ehrlich@aviato).

Programmers are not quiet people. Our brains are always busy doing things, and we are always shouting at ourselves. What you are encountering is a very sophisticated set of filtering mechanisms that generate something approaching a simulation of civility, because most of the time your questions are not as interesting as the questions we ask ourselves, and you're interrupting us, dammit.

Be glad you don't know programmers without filters and social skins as we are wont to replace you with a very small bash script.