Cubegeek

Waiting for Dot Gov

Somebody asked me: Could blockchain technology be used to implement Social Security or Universal Basic Income?

I sez:

Yes, it’s possible. It’s coming from a side angle though. Basically Congress is going to have to admit at some point that the credit card companies keep better track of individuals. And then the credit card companies will have to admit that the new crypto devs have a more secure way of keeping individuals in control of their own identity. And then somebody at Apple is going to make it easy to integrate anonymous identity with iCloud and ApplePay.

Basically, the interest around blockchain and cryptocurrency is going to drive the development of secure identity management and peer to peer payment tech.

If you look at something like Keybase you can get a feel for how a single ID can federate other well known IDs.

I forget the name of this other company, but they have the ability to take your one credit card and make it authorized for all your online purchases except you use a different ‘credit card number’ for each transaction. So your single, real credit card ID is never used online. It’s just used to authorize, from your desktop (or thumbprint as Apple has proven), an on-the-spot value card. This technology is similar to Amazon KMS and HashiCorp’s Vault. IE it’s already done and well understood.

This combination of technologies already exist. The question is basically who is going to mainstream it best. I bet Apple will. Now let’s say they do that next year or late this year. How many Congressmen do you have to buy in order for them to

1. Get it
2. Regulate it
3. Authorize it’s use for Government things.
4. Get bureaucrats across the US to understand it.

Yeah almost never. Which means somebody needs to hack the other two credit bureaus to prove what we already know. Current identification schemes that dominate US commerce are insecure and outdated. Then of course you’ll have to deal with millions of paranoids about the possibility of a National ID, seeing as it’s ‘racist’ to have people show a driver’s license when they vote.

Bottom line. Stop hoping for the masses. Do the math, figure it out, and do it with people who already get it now. And yes I did triple my money in Bitcoin (and in Amazon stock before that (and in Google before that (and in Netscape before that (and in Inktomi before that)))) and no I’m not hurt by the price collapse today.

On the other hand, there will be people who will slowly duplicate old tech with new tech, and maybe they’ll be able to sell this stuff. The old dis-spirited workers at the NSA and CIA are going to have to work somewhere.

 

Data Driven Business

I've been rather interested to see how the media has reported on 'working conditions' at Amazon, the notoriously data driven business. Like all partners with Amazon Web Services, I'm always glad to see them get more press. The inevitable benefits of cloud computing architecture are becoming more well known to new industries every year. At the core of the kind of work we do, Business Intelligence, these benefits include allowing downmarket businesses afford the kind of IT environment only available to multinationals just a few years ago. 

But the NYT article (behind a paywall of course) 'Inside Amazon: Wrestling Big Ideas in a Bruising Workplace' strikes the wrong tone. I'll simply sum it up like this, the article and many like it are anti-meritocracy. 

I am working with some of the smartest hard-working people ever in my career, and we are a small agile company. We keep thinking of ways to innovate in our particular business and our CTO is notorious for keeping us on our toes by introducing new technologies, fresh from open source development worldwide. And still, we can only keep about nine months ahead of what Amazon makes virtually infinitely scalable. This is unprecedented in computer science. At Amazon, information technology goes essentially from research to global industrialization in about two years. These are space race times. 

I'll tell you what Amazon's reputation is. It's that if you have a masters degree in computer science and 10 years experience you will make about 125K per year plus full benefits and you will work hard. In other words, you'll have a great job with an awesome company on the cutting edge of technology, but you'll never get rich. You're better off working in some small, young company where you'll be the smartest dude and closer to the top. That way you can do some fraction of what Amazon does, get stock options and make money on the exit strategy. In other words, the way to do it is to let VCs throw ridiculous financing at your hot, trendy Silicon Valley startup, take the money and run.  

Having been in this segment of the computing industry for nearly three decades, I can tell you how I have been continuously surprised by the inability of many companies to do what makes sense. If that weren't the case, Dilbert the cartoon would seem alien and non-sensical. Instead, Scott Adams has nailed the kind of mediocrity and dysfunction that plagues the best of our businesses not only in America but worldwide. You don't have to be a revolutionary to make improvements in business. In fact, you don't even have to be a 'data scientist' to enable data driven decision making. You simply need to know what to count, and build a system that counts it for you. Deliver those numbers to the decision makers and you're on your way.

It might seem odd, but the best businesses are already data driven, it's just that the data is not necessarily in digital form. If you're a bus driver, you have an idea if you are having a busy day on your route. When you stay in business, you know what you need to do to keep your head above water, you know where your nose is, but not exactly how many centimeters above the waves. Digitization of these metrics helps your organization under certain conditions, and this is what we identify together with our customers. We make a data model that follows the business model. We enable more people to see what the leadership sees. We socialize the metrics of the business.

Obviously, Amazon has been doing this and clearly the details are not shared with the editors of the New York Times. So it shouldn't be surprising that we hear from the outliers in their seat-of-the-pants description of the waves that overcame them. But that's human nature. We listen to outliers. Part of what we try to do as data experts is separate the signal from the noise, help identify if outliers are just loud squawking pests or the coal mine's actual canaries. 

We are always happy to discover with our clients what kinds of accurate instrumentation we can provide by harnessing the data at their disposal. Every business can be data driven, it's just a matter of disciplined collaboration with us at Full360. 

Contain Yourself

I'm going to repeat myself a little bit. Some of the best advice I ever heard came from my brother. He said not to tell people what you're going to do. Do something. Then tell people what you did.  Well, it wasn't all me at all, but I played some part in Full360's evolution of the ElasticBI Framework, which is starting to be something we can tell people about. Over here at Cubegeek, I'm doing a little amateur marketing and communications. 

So part of the whole ElasticBI PaaS Framework involves how we stand up application-specific stacks. And then we have to load balance them. And then we have to make them elastic. And then we have to secure them and all that stuff. We are building half a datacenter for every new customer. I put that work under the header of Virtualization & Orchestration.

VIrtualization
Virtualization is about creating the compute resources that are fungible to the cloud and to our dev workstations. It's the small grain of the cloud computing. Virtualization is about making boxes. 

Orchestration
Orchestration is about configuring stuff on the fly, keeping track of all  virtual resources in an application. It's the larger grain of cloud computing. Orchestration is about maintaining the size of the swarm of boxes. 

In the news this week, Google admits that we are right by following our lead. (Heh) You see for several months now, we've been using Docker as part of our V&O strategy. Along with Docker we are also using:

• Chef
• Packer
• Consul
• VirtualBox
• VMWare
• Vagrant

Now I could go into all of that, but it's actually not my department, and I don't want to tell you about what I will do before I do it. But I will tell you that we have in place a lot of docker hosts and containerize all of our applications, and by applications I mean Data Management & ETL. The databases themselves, we tend to run a bit closer to the metal. However, we have recently made very impressive strides in our orchestration chops. Stay tuned here and at Full360.com and elsewhere so that when actually blow our particular horns, you'll know about it. 

Discoveries in AWS

My brother Doc gave me some advice that I take to heart. Don't tell people what you're going to do. Do something, and then tell them what you've done. It's very good advice. But I wouldn't be me if I didn't speculate about the future and what I'm going to do next.

 

What I have done is learn Ruby. I'm right about now, good enough to be dangerous - which means I'm about 75% as proficient in Ruby as I am in Perl. So there's a subjective couple of statements. It turns out that Ruby is a lot more fun than Perl is, and that there is just about nothing that I would do in Perl that I can't do in Ruby. Those things where it might make the most difference, in the way Regex is interpreted, doesn't matter, because I've forgotten most of my Regex anyway. The advantages of Ruby are manifold. Firstly, all of the new generation APIs will speak Ruby. Secondly, it's object oriented - which gives me the confidence to put off my guilt about not learning Java after all of these years. And thirdly, it just makes sense to me as a language. I'm very comfortable with it.

 

What I have done secondly is learn Git. I know enough git to do 90% of what I'll ever do with it, and it too is a leapfrog technology that makes me glad I didn't bother with a lot of the other stuff I didn't bother to pick up. So I'm turning the corner, or turning back a page as it were, to become a real developer once again. I will be responsible for a lot of code and I will collaborate. Why? Because I'm building product - a different kind of product than I've ever worked on, but also something that the marketplace hasn't seen much of, all part of the Full360 mission, which is to deliver the next generation BI environment. It's something I've been talking about forever, that thing that would become 'mydatawarehouse.com'. That's where all of this is going.

 

I've also become stronger in Chef which is one major part of the glue, I've picked up the Rightscale AWS API, and I'm pretty comfortable with AWS itself. Next on the agenda is that new thing called Sinatra. All of this is coalescing into the Full360 architecture and environment for end to end full lifecycle design, development and service management of the next generation of BI. So let me talk about that for a minute.

 

Where we are going with all this is to enable what we call elasticBI and elasticPM. The practical concept is DevOps for EPM and BI. What is DevOps? It's the merging of development and operations. Practically speaking it is why Amazon.com can watch you buy a backpack today and market hiking shoes to you tomorrow (or later this afternoon). It's about being able to move new code into production on a daily basis, quickly and reliably at scale. Think about what that means in your environment. It means that as soon as somebody in finance comes up with a new KPI in their spreadsheet, we can integrate that into the production cube and have that KPI working for everyone the next day.

 

That sounds good in theory but there are 100 reasons that your IT department cannot do that in reality. That's the reality that we are working to change. We are going from speed of thought queries to speed of thought implementation. OK I exaggerate, but you get the picture. 

 

Inherent in tackling such a challenge involves things like new strategies for disaster recovery, new strategies for scaling data marts into parallel clusters and new ways to upsize and downsize environments. New ways to monitor performance and manage security are involved as well as a dozen other kinds of things like new batch processing tactics. This is where we are focused and where we intend to succeed where almost no EPM or BI providers have gone before. I want to go from 'disk is cheap' to 'cubes are cheap' to 'environments are cheap' to 'migration is cheap'. And so we're building a whole new language of BI and EPM specific concepts in infrastructure. 

 

This goes beyond a simple cloud 'play'. It's about rethinking the platform for BI and EPM given the advances that cloud scale eCommerce has demonstrated. There is no easy way to scale from the demands of 500 users, which is typical of a large Enterprise BI environment, to the demands of 50,000 users which is typical of a small eCommerce site. So quite frankly, enterprise IT has been leapfrogged. Our mission is to bring the methodology and technology of DevOps to the structured data space. Practically speaking, that means it should be no big deal to install a certified Hyperion Planning environment for 400 users in an afternoon. That's the kind of agility we are working to make practical, not only in pure cloud environments but in secured hybrid environments.

 

What I've been discovering by using the new tools I've gotten my hands on, is that this kind of operation is not just pie in the sky thinking, and I've got the scripts to prove it. Imagine the following scenario. You have been working with an ASO database for several years now, and you've been incrementally adding features that your users want. Query times have been slowing down, not too significantly but noticeably. As time has gone by, the crush time, end of the month close, has become unreasonable. But you have a very good idea of how to solve the problem - in fact you have a better performing cube in development right now. How much time and effort will it take you to test that new design in production, and what if it's not as good as you think it is? One of the aims of elasticBI and elasticPM are to allow you to experiment with that this afternoon. As quickly as you can dump the contents of your production cube and load that data into the new cube, you can begin testing in prod in parallel. 

 

As you can tell, I'm really enjoying my time at Full360, and guess what. We're going to do some expansion this year. So if this sounds as fascinating to you as it does to me, drop us a line. We might just have a spot for you on the coolest team in the space.

Wokcano Meetup

Yes Virginia there are interesting and smart people in Southern California. I just came back from my third Meetup, and I'm telling you that's a great tool. I met folks in the web business and they are doing some pretty interesting things. The flavor is definitely entrepreneurial and I've heard some exceptional stories. Maybe because they're new to me, but it's really refreshing to know that folks are making money themselves on the strength of new ideas and new technology.

William has got momentum now, and he had to change the venue from a smaller joint to Wokcano which is rather chic even though it costs 4 bucks for a Red Bull. He hipped me to RIAK by Basho, and although I don't quite have my head around how I might get BI knowledge out of massive document stores, it is interesting to hear that people frustrated with Hadoop are happy with RIAK. I seem to recall an Xtranormal movie joke about Hadoop cluster failures, and that's the thing that RIAK doesn't do. Surely Cloudera will come up with a smarter way to do system monitoring, but the legendary (now) Foursquare failure is a lesson that the big data community is trying to learn from.  Oh yeah and Joyent. I remember those guys. I guess they're the hotshots - they should know a whole lot by now.

So I'm expecting that what I might like to learn is something about the fuzzy area between unstructured and structured data. Given my understanding about analytic consumption and EPM feedback loops I'll add value. So this is where Derrick was telling me about Solr. It sounds like Solr is part of the missing link between structured and unstructured, or at the very least can assist in indexing massive data sets. While I've been warned away from Cassandra, there sounds to be something Solr can do to make up for its shortcomings - all of which is very interesting. He also tells me that I should check out Splunk. Hey wait a minute. I know that CEO.  Well, whaddya know? He was keying in on the phrase 'machine generated data'. Yep. Nice. The other term he used was 'faceting'.  That's a nice way to structure up stuff on a big data set if you don't know exactly what it might contain beforehand.

Let me think about that for a minute, adding to the minute I thought about what was interesting and weird about Qlikview. In Qlikview and in SAS, I seem to recall the ability to do what seemed to be a random kind of drilldown - not in the way that made sense from a multidimensional design standpoint. I'll call it something like faceted search, not knowing exactly what the proper definition is. But imagine that I index all of several fields in a data set, but don't organize them into hierarchichal dimensions. I can still drill down a path to narrow the data set without predetermining dimensions given the cardinality of remaining index counts as my key - the cardinality being an interesting part of the dataset itself. 

We commonly do this at BestBuy's website. You look at cameras, and then down the left banner, you get counts of all the brands of cameras available. You also have certain attribute ranges on the prices. 0-100, 101-250, etc, and counts on those. At the bottom of your drill down are particular documents. Nice. This kind of navigation is looking for a single item or maybe a single class of items - it doesn't make sense to aggregate all of the data up to the highest level, so this could be a preliminary search into interesting data - something sitting on top of cubes to go - a nice way to winnow down a huge data set.  Considering that free  Splunk will parse and index 500MB/day, that's a good way to get started. Thanks Derrick.

Last year when I went to the CalTech seminar on clouds, I met a couple guys in the lobby who were ripready to roar with their hosting. It looks like cloud hosting is hot and heavy here in LA. Dennis is already considering another property. It has been several years since I signed on at Dreamhost, so there's evidently a new generation going. 

I really enjoyed talking with Jad. He's got the right ideas for aggressively attacking important markets with targeted solutions.  Smart pricing model too. He reminds me of Levi whom I met last week over at McCabe's. One smart guy designing with a small team of engineers can take a big bite out of the marketshare of the larger software firms. Agility matters, especially when open source interoperability is so real. How real? That I don't know, but I do know that one size cannot fit all and the software vendors who come to recognize that their products can be rightsized to their customers are going to win big. Think about that for a minute.  Why pay for a legacy of features you don't use?

Shout outs to John from Boston and Ian. And to the guy from the VA, it's Charles Wyble.

 

The Cloud Money Quote

The cloud is nothing but the industrialization of any IT supported services.
-- Erich Clementi, General Manager, Enterprise Initiatives @ IBM

I just wanted to say that I see it the way that Clementi sees it. There are new database and messaging technologies that are cheap enough for masses to play in (read social software), but the mainstream of computing is now working towards commodity industrialization. This is rather like the mass production revolution, and the consequences for programmers are very much like they were for craftsmen.

We're Not Worthy

I'm almost completely around the bend for Google. 

I'm excited about the new stuff Google is kicking out this way to us consumers. I got my invitation to Google Wave, and even though I thought the demo was insipid, I'm getting some new interesting ideas about the value of flat organizations and collaborations out here in peasant territory. So Google is our friend because they deliver applications for infinite scale, which makes it so easy for us to expand control of our own properties - since Google gives us more than we might generally merit in a more vertically oriented world.

But what really has my knickers in a twist is Google's facial recognition that it has embedded in Picasa 3.5. And even though I'm signing up for Photoshop.com because it integrates Picasa, Facebook and Flickr, the first really useful thing I'm excited about is Google's face tagging. Seeing as I have some 60GB or so of snapshots, I love the idea that I can pick a few pictures of my kids and Google will go through and match up the other thousands for me. We'll see how well it works, but on a small set it has done pretty well. 

Well, since I started writing this four days ago, I've had a lot more experience with the new Picasa, and I have to say that it is successful to a creepy degree. My daughters are in a theater troupe and so we have lots of pictures of lots of kids at lots of performances, offstage etc. It's one thing to say I know three or four of the kids my kids hang out with, it's another to have software that has identified 80 snapshots of them. With a little help from my daughter peering over my shoulder, I've got the faces hooked up with the names. I feel like a CIA analyst. But now I know what a CIA analyst must have. It's obviously not as good as human face recognition, however, combined with a human confirming or denying its recommendations, Picasa is a very powerful tool. 

The only thing Google has done to tweak me wrongly is to discontinue the Google Notebook extension for Firefox, and the service itself for new subscribers. I found it a brilliant way to clip information into a very nice organizational form. They are keeping the service alive, but if they abandon it, I think they should GPL the code and leave it for somebody else. 

I say this because the other thing that I wanted so very much from Google was in fact a 'Google OS'. As soon as I heard about BigTable, I began following that thing which I believe will put my Oracle consulting business at a severe disadvantage, which would be not MySQL but MyDW. But Google was not interested in publishing the 'Google OS'. What we got was in fact better. We got what Cloudera has come to be, the Redhat of Hadoop. All open source stuff for building the ultimate file system.

Google's philosophy of thinking at scale has delivered an entire new kind of computing over the years that I never imagined when people first started talking about grid computing. I expected something like IBM Regattas and Veritas servers with all the approachability of supercomputers. It's nice to be wrong.

Tiptoeing Back From and Towards the Cloud

Well, right about now was the time when I expected to be speaking in tongues to the Oracle Open World. Unfortunately, my proposal didn't get a bite. It was high faluting sounding, but I think they saw through me. It was about Essbase in the Hybrid Cloud, and what it was basically about was using cloud-based storage to inch oh so gingerly into the cloud arena. Now that I've had a bit more time to think about it, I think I know better.

You have to jump in with both feet. At least, that's my attitude.

But first some geekery updates.

I'm backing off of S3 for my own personal use. Yes I have nice fat FIOS bandwidth pipes but I have determined that offsite storage is basically only good for backup & disaster recovery, not file sharing large media libraries with Samba or Windows file shares. So I'm coming down off my Jungle Disk high and transferring all my big stuff back to the new generation of Seagate external hard drives. Now there may still be a chance that I'll experiment with a PogoPlug at some point, but really, what do I want that for? I want it for music when I'm on the road. It's cheaper to buy the drive, or just use the damned iPhone. (or Orb, or Last.fm)

What has turned out to be much more useful than I ever imagined was Dropbox. It has basically put all of my flash drives out of commission. And all I ever really needed was 2GB. Whoda thunk?

I let my Carbonite subscription lapse although I hadn't intended it to. I gave my old machine over to my wife and it just wasn't worth keeping all that stuff there. I moved it over to my new big 1.5TB external and I'll back it up from there. How? It just turns out that I like Backblaze and they allow you to do just that, whereas I seem to recall that Carbonite didn't like mounted drives very much. For 5 bucks a month per computer, I'm digging the pricing model and I like the fact that they're open source.

In other geekery, I finally went the next step and cranked up an EC2 over at Amazon. It was fairly easy to do, but a lot more expensive than I thought it would be. But I'm starting to see the possibilities, which is why now more than ever I think that some of us service providers *are* the new next big thing. Which is to say, the first guys to master the art of hosting enterprise apps and demonstrate super scalability and truly rapid RAD wins big.

Suddenly, I feel like I just might become a hot commodity again because of what I did back in 2005 on that 32 way 64bit Superdome with 500 concurrent ASO cubes. However... I think between that and the 'cloud' are some very significant virtualization issues. Figuring those out is the next step.

You see, even though AWS was telling me my virtual box was 'Large' it certainly wasn't 'Fast'. Getting my favorite marketing database up into SQL 2005 took a lot longer than I expected it to, as in I seem to recall things moving faster on my laptop. So there is an I/O bottleneck getting things into the cloud if you're way down the pipe, as I am most of the time. There's got to be a better way - which is why I'm thinking about the trickly stuff that's built into the Carbonite and Backblaze products, made more transparent by Jungle Disk, and very likely available at a low level with Nirvanix (which I haven't been using at all).

So uptime is costly on EC2, whether or not your box is processing anything. It's the availability of the instance that you pay for. So my next trick is to learn how to park a big old virtual server image so it actually becomes something of an on-demand machine in the way I thought it was. There's a great big financial step from LAMP infrastructure providers like Dreamhost, whom I've been using for about 7 years, to Amazon's EC2. 

I'm also becoming convinced that a trickle server is necessary - just a dedicated machine to Jungle Disk or Nirvanix the required data up to S3 and then trigger something when that job is done. When I use it on a regular working PC, it just gets all bogged down, but part of my problem was that I was doing backups and file transfers intermittently, and I'd have the thing shutting down, sleeping or timing out. What you need, I think, is a machine that just never times out and does nothing but fill your pipe with upstream. An upload server. It shouldn't be so difficult, I should think, to link your S3 to your EC2 and then voila. That is an exercise for me.

--

In other news, I'm becoming basically competent in Planning after many years of barbed remarks. The thing actually works as advertised in 9.3.1. Of course it stands to reason that I'm pushed to the bleeding edge of the very latest feature set and as such will be getting a long luxuriating look at 11.1.1.3. Obviously there are little things that I'm discovering that people are saying 'Duh' to me. But I swear to God I have not had any customers that bothered with Smartview in X years, and I never got to be an expert. Now I see that it does all the things Joe Cajic said it would do back in 2004. Yay. 

Naturally, I jumped at the chance to play around with the OutlineLoader and dropped various parts of that into my Cubegeek's Toolkit. So I'll soon have scripts that generate scripts. And I suspect that I'll try to find ways to automate the dread parts of EMPA as well, time allowing.

Sometime next spring, and certainly by KScope next, I'll have mastered some small portion of the latest Planning scripty bits and present my modest results, board willing. It would be awfully nice to have one of those big ER diagram posters of the Planning repository somewhere wouldn't it? Heh. Listen to me, thinking I might do that HAB kind of development. Is that product still around?

--

Well that's all for now. I'm energized and optimistic.

Big Data at Structure 09

In which you will hear somebody say that relational databases don't scale.

Google Fusion Tables

Oh no. Not again.

I seem to recall the battle over decoupling Internet Explorer from Windows. Not like it was yesterday, but like it was the day before yesterday of last week. But what I do remember clearly was that Microsoft kept saying that somebody might someday put the entire desktop metaphor out of business through a sneaky browser, and I laughed. Maybe Sergey Brin was watching TV with an evil smirk on his face thinking 'one day...'

Well the bad news is that Google seems to have decided to get into the database business. Sorta. Surely there's a difference between these fusion tables and bigtable. But surely they will bridge that gap, just like MSFT bridged the gap between Access and SQL Server. And so now a new front opens.

A cat named Hellman worries that all database folks will go out of the window, but I say there's nothing like having a headstart when everything gets user friendly. I mean microwave ovens didn't kill the restaurant business.

I'm interested to see what kind of visualization appliances they put on the top of this bad boy. That will be the key to it doing something fairly cool. I'm not sanguine about the prospects of shotgun collaboration as I said about Google Wave and the 'appliancicity' of all sorts of simple technological machines as Google has made infinitely scalable. The magic about ERP systems, which in a simple way of speaking, are 'merely' tables and rules, are the brains behind the workflow between them. So visualization and linking between tables enables collaboration but it doesn't really enforce discipline and mark checkpoints along the way - and that's the entire difference between mere information dissemination and a process that could be called critical. Just like security isn't the mere act of putting a super incredible lock on your front door, what's extra cool about databases is not that you can slap them on something and have a system.

The real advance here is a species of cloudiness anyway which is speed of deployment. We now have the equivalent of an early version of Access - the Jet engine if you will, now in the hands of Google. This will not revolutionize anything more than Google Spreadsheets did - it just gives another sort of object in mashups. Nevertheless, this takes us one step closer to the entire unavoidable issue of the speed at which Google will be able to demonstrate that it can deploy artifacts like your company budget faster than your company's own IT department.

You have to take notice.