Cubegeek

A Curious, Skeptical & Analytical View of Climate Change

So I compartmentalize. Context is everything. Over at my other blog I blather about global climate change from the perspective of an extreme skeptic. But the best skeptics know exactly what is needed to change their biased minds. Evidence. So here's what I'd build. (apropos my first introduction to the dudes at Factual)

--

When I get really serious about questions of climate change, the most important question I have deals with how the world's food is produced and distributed, because truly the only thing that is going to make an important difference is whether or not what is good farmland and good fishery today is going to be good tomorrow. And indeed nobody's livlihood depends more on weather than farmers. Am I right? So take JR Simplot for egregious example. They are the world's king of potatoes. And if I had a metric I would do as Paolo Bacigalupi has done in his excellent 'The Windup Girl' and begin to measure total world food production in terms of calories and then divide that by how many people are in the world. Who grows how many calories and who eats how many calories, now what's the price of calories? How we know this about gasoline and oil and don't know it for food shows how unimportant the food problem is. Today.

So when do the weather watchers, who are also money minders, at JR Simplot say. Hmm? We may have to relocate out of Idaho because the climate has changed and Idaho just don't mean potatoes any more. Do the global warmists think JR Simplot are a bunch of dumb micks who will just die out in a potato blight? They surely think that the dudes at BP are bloody geniuses who will exploit the last drop of oil where ever on the globe it may be. So I think that sort of thinking should be applied to all of the seed bankers and mega-agriculturalists on the globe. Which is to say, they will figure out how to feed us all because it's in their interests, and furthermore the weather is not going to creep up on these guys. I am trying to imagine which group of eggheads has more access to more satellite time, feeds, maps and data - the scientists at Monsanto or the grad students at East Anglia? Duh.

So now you know my drift. I am much more interested in food scarcity as reflected in the business of the Chicago Mercantile Exchange and the likes of Duke & Duke. Nobody ever, ever, ever puts these two things together when talk about 'global climate change' is discussed. That's because they are politicizing the 'science' and ignoring the economics. Yeah I said 'they', and I don't bother to find out who they are. 

Here's the nut. Whatever the hockey stick is with climate change, farmers will move production, swap crops and otherwise be quicker than the weather or the climate. Say what you like about El Nino or the Ozone Hole, neither Katrina nor Sandy made one whit of difference to Iowa. When hurricanes get as far west as Ohio, maybe we can all take a hint. And what about you and me? You and I will eat whatever. Hell, if we can miss Twinkies and put up with non-alcholic beer, we really have no worries.

 

State of Big Data

Big Data is frustrating. Why? Because it has attracted too much attention and not quite enough discipline. And, like other complex matters, those who know don't talk and those who talk don't know. Of course there are degrees of knowing, and there are large mammals out there always seeking to develop us monkeys into chimps. Such a gathering of apes took place this morning over at the Entrepreneur's Forum at  CalTech where I spent several hours ingesting just the right amount of hormones in the milk.

The panel was excellent. Each of the speakers did a righteous job of presenting their POVs on matters of big data. The first that leaves me with a singular impression is Mike Carey of UCI who is putting together a partially open source fully parallel DBMS. Its aim is to corral and present the proper concepts that are now assembled with duct tape and spit by systems integrators who claim to be practitioners of Big Data. It's rather like Eigenbase, I suppose. But what got my attention was the slide he presented outlining all of the layers that comprise that asteroid belt of open source data management, not least including MongoDB, Hadoop and Cassandra. He got a great laugh out of me when he related that so much of the Hadoopery boiled down to guys sweating out writing Map this Reduce that and managing cascading jobs of MR. I kinda guessed that many years ago, but I have no excuse for not having actually learned to do it even though I like to pat myself on the back for not having sweated through that business. It just means I am subject to grand frameworks like Asterix, his project which is quite nice for publishing APIs at multiple levels of data abstraction. I like it. That's what I liked about Essbase too.  Carey knows what he's talking about, and it pleased me to hear that my approximation of the database industry is similar to his.

The real star of the show was Factual founder Gil Elbaz. I know that I am late to his party because I missed several meetups at Factual, including one last week or so. So color me impressed and rather surprised to hear him saying some things I haven't heard anyone say out loud outside of my head. That being that there need to be big data markets with standards, agreements, low costs and high quality. Yes I was thinking Infochimps too. 

Here's where we gel. Elbaz is a fan of structured data, and much of the hype around Hadoop and the other NoSQL key pair stores is around non-strucutred data. What I don't have is hands-on experience with a lot of data mining. I played with Crystal Ball, Clementine and another tool I forget once or twice a long time ago, but the game that pays is data people ask for specifically. In fact, the embarassing fact is that you and I know that Crystal Reports made millions of dollars because many thousands of customers knew exactly what they wanted in millions of data reports that were infinitesimal. Or as one of the panelist said, in the micro-terabyte range. I say that Big Data goes through data science to become reasonably structured data. So I'm on a hunt for data scientists to give me something new to chew on besides the same old DW. I'm not sure if that market is immature or if it's simply not so hot on going to the cloud, but the IBM guy and I agree - things are slower than they should be. My point is that you mine data when you don't particularly know what you're looking for, but once you do, then you no longer have an unstructured big data problem, you have a structured smaller data problem - often one that requires vizualization. A picture is worth 1000 TB? It must be, after all, there's Powerpoint. 

Elbaz sees his four keys to unlock data liquidity - an excellent economic metaphor with what's wrong with the data supply. D1 like M1? There's all kinds of data out there, but its value comes from multiply sourced contexts, and we can't do that because of those barriers to liquidity. My business is definitely working out some standards and cost issues, but we struggle with the contract constrictions and data quality issues of having industy standard data models.

I'm expecting, as you should of me, that our customers and partners will enable us to bridge those gaps and we will be able to present a new level of turnkey intelligence system. That is a strategic aim of Full360. For example, as Jaspersoft works through the XMLA standard compliance and adds Essbase to their Data Domains, we will be one step closer to having multi-platform multidimensional database data and presentation standards on the sorts of elastic stacks we design, host and manage in AWS.  Our customers are presently showing us new kinds of transaction data that has heretofore never made it into multidimensional models, and our partnership with Vertica enables that sort of system to be crafted. Enterprises ready to deploy onto mobile platforms will use the cloud, and we'll be ready.

So yes as one of the presenters said, the brilliant Mr. Lilley I believe, we should not look in terms of constrained data processing or in any scarcity based model, but in term of grasping all of the problem and coming up with smart ways to make all that is computable more liquid. The liquidity problem is more difficult than the computability problem, but whereas the complexity of ceratin problems are combinatorial - requiring science, the matter of data liquidity is technology and process. 

What truly excites me is the degree to which we can participate in developing defacto standards of multidimensional data representation in standards. I think this way intuitively, I'm like a wraith that feeds on ghosts of institutional memory - there's a Harry Dresden metaphor for you. Consider my formulation of an approach to find signals for global climate change. Oh wait. I haven't published that yet. Today.

There's more I could bore you with, but I've already got 1000 words. Maybe I'll follow up. Speaking of which, I didn't writeup two more adventures I recently had. Whew. I'm getting behind.. More later.

The Surrey Pigeon Problem

There's an interesting story in the UK about a man in Surrey who found the remains of a WW2 carrier pigeon in his fireplace. Attached to the dead bird's leg was a red canister that contained the following cipher. 

The first program I ever wrote in C was called 'askew' which was a text permutation algorithm. I never quite perfected it - ( I wanted to tie it to a dictionary) but I've always had some attraction to cryptography. So I'm going to try and solve this and keep an ear out for others who might have a clue. I doubt that it's a substitution cipher and it's probably in code too. We'll see.