Cubegeek

Feedly for Me

IJWTS that as Google Reader dies its untimely death that I'm going with Feedly. I would have liked to have stayed with Flipboard but they just went too far with the winky blinkies and cluttered up my screen. They refused to support Evernote and so this is the end. So long, farewell and quite frankly good riddance. I can't figure out what the hell they were thinking when they decided to add that ridiculous floating sidebar, but that will no longer be my problem. Goodbye Flipboard, hello again Feedly.

Big Data - Part Two

I keep worrying about being formal vs being casual when I talk about what I'm doing professionally. So I have a whole lot of half-finished outlines describing the shift in my work over the past three years. But I suppose I can speak with enough authority to be casual now as I continue to talk about big data - and yes my biases will creep in.

Let's identify big data.

What makes big data big? The fact that you can no longer take operating with it for granted. It's that simple. There are several dimensions into which your data gets too big for the britches you bought it. And if you've heard this before, please forgive me, but consensus has to do with independent validation of what others are experiencing.. So my aim is to find the pain as I start diagnosing what kind of data headaches you are dealing with. 

In general, I find that people are much more specific and consistent when they talk about pain than when they talk about what they hope for or what they think otehr people might be thinking - so let's take the headache approach to defining specific types of big data. 

1. Velocity
Data gets out of hand when you have to process it faster. Think of those processing windows for your SLA.  What is real-time, near-real time? How much data do you need to see that is very recent? Is there a use case for the analysis of events that have taken place in the past two hours? 30 minutes? 5 minutes? Every database can process data in near real time, but how much do you need processed every minute? When this gets to be a headache, you are dealing with big data.

2. Veracity / Validity
When you have to process a billion records how many of them are perfect?  Meaning that if one field is bogus, you have to identify that cell within the mass of data you are processing, record by record. When it comes to database ingestion, chances are you can't just segregate out a million errors and then re-run them separately as a standard function of the database. So you have to engineer a process that will do so for you. When this gets to be a headache, you are dealing with big data.

3. Variety
In your current system, there is some data you have to deal with that makes you cringe because according to the current business rules you have to make six passes over the table in order to categorize it properly. Fortunately there are only 500,000 of those such records.  If that section of the data gets big, you have a problem. Or how about this. Some data you have requires 255 characters and there are no short descriptions. Or some data you have possesses two short keys and 275 floating point values. Big data means that these typical 'outliers' may now exist in significant sizes. If your weird data is getting larger, you are dealing with big data.

4. Virtual Hardware  
Obviously, some numbers just sound big. Since I am learning to think *really* big when I consider the capacities of Amazon Redshift things change. "More than I ever had to deal with" is actually quite big enough. But let's talk machine size for a minute.  Once upon a time, The Gap was tapping their feet waiting for an OLAP that could handle a million member dimension - for their marketing. They made available Sun's largest single machine, an E9000 or some such and re-niced our processes so we could have more of the box than any single program before. Once upon another time, HP made a 36 processor Superdome available for my team as we processed data for a design at Boeing. In those days, that was huge. Today both the Sun and the HP could fit inside of the processing space of a five node 8XL Redshift cluster with plenty room to spare. If you cannot conceive of compute power above single boxes, no matter how large those boxes may be, your data may soon become a headache.

Let me expand upon this point because it is the point over which some rather serious economics hinge, especially those economics about your job in the enterprise.

In business computing, there will always be unpredictable market conditions, evolving product performance data, changing skill and knowledge sets within the management and rank and file, and wildly varying customer information, not to mention regulatory compliance reports. That means to keep your business on top of the data being generated in your industry will always be beyond your ability - even if you have all the resources of Fort Meade. So what you have been allowed to build has always been constrained by budget, time, resources. 

The Big Data revolution is all about putting better tools and processes in place which allow you to do more with less. If new products and technology don't expand your capacity, then you are wasting time, money and effort. So the opportunity should be considered in terms of what you can do (more) with these new advances. 

If your organization's aims are to cut costs and keep the same functionality, your job is at risk anyway. It's not the cloud that threatens your position, it is your management's view the value of your work to the company. So if you feel threatened by the existence of cloud computing, what you need to do is start considering the unpredictable market conditions, evolving product performance data, changing skillsets and regulations as they relate to your unique understanding of what your company is experiencing. If you can't come up with ideas for improvement your management will by into, that says more about your company than it does about the cloud. The cloud is happening. It presents new economies of scale that your company does not have to pay for to research and innovate on. How your company takes advantage depends upon the culture of your company.  

Let's go back to basics. Why do we build systems and how do we judge their effectiveness? I have golden rules for end-user computing. We'll talk about those in the next part where we can identify where change is happening, understand if we need big data technologies to solve those big headaches and then demonstrate the value you can bring to your business.

 

Big Data - Part One

What is big data?

I want to eliminate some confusion by giving a history of big data from my perspective. I don't know how long this will take, but I'd guess about 5000 words. Let's get started.

Big Data is clumsy. Because it is big the habits you have with non-big data become impractical. Data becomes 'big' when it forces you, by its size, to adopt new processes, hardware or software to deal with it in a reasonable period of time.

Part One - Personal Data Growing Pains

I started thinking about big data a long time ago. I'd say it was somewhere around 2002. At the time, I used a laptop with about a 5GB drive. This was about as much as I would ever need to run my 32bit OS (Win 2K Pro which soon became Win XP Pro). But I had collected about 4000 names in my Palm Pilot contacts. As I had to move and sync this contact information with Outlook the system started to fail. It got slower and slower.  The same thing was happening with my email, which I had all in Eudora. Migrating this all important data from laptop to laptop began to be a serious problem. One day something went wrong. Somehow my address sync created duplicate records. Now I had 8000 contacts, Outlook slowed to a crawl and there was no way for me to deduplicate the records except by writing a custom progam (I didn't know how) or manually. It took months.

This simple mistake put me in a class of user that the builders of Outlook didn't consider mainstream. Even years later as I uploaded my contacts to Google's new mail service, GMail, it was very slow to lookup a contact. I could have paid $200 to purchase the popular contact manager Act!, but I didn't want to. The point was that my data was too big for Outlook - the way I had managed contacts all along. 

So the first notion I want to communicate is that no data is really too big. Somewhere there is a system that can handle your data no matter how big you think it is. The question is whether or not you must endure a paradigm shift to accomodate your outgrowing of your old system. Let us recall archive.org. The Internet Archive has been eating and saving the entire static WWW for a decade now. 

What really began to give me problems was my music collection. At some point I began using iTunes to rip my 600+ compact disks. This filled up my laptop in no time. I started buying external hard drives. At firstI got an expensive LaCie Porsche Design external USB drive. You may remember them from that Will Smith movie I Am Legend - in his laboratory basement... OK that's just me. Every six months my music, movies and pictures would outgrow my capacity and I bought a new one. Fortunately drive capacities were going up. 100, 200, 250GB. It was hard to imagine I would fill these up, but I did. Soon was dragging around the Seagate drives that eventually became the FreeAgent Pro series. A blue one for my business information and a red one for my music. Capacity finally but capricious.

Like clockwork every year for three years straight one of my major hard drives would fail. By 2007 I had a terabyte at home spinning in various disks attached to my Dell tower. By then I had become familiar with two vital tools. The first was a disk recovery system called simply Recover My Files, and the second was disk management program called File Tree Pro which had a deduplication facility as well as a heatmap visualizer. 

Disaster recovery became an issue. When your collection of data becomes larger, it becomes more precious because you realize you've been collecting it for years and you'll do anything to retain it. Over this period of failing drives, I realized nothing was foolproof so I had to make friends with redundancy. I kept a copy of all my iTunes music on my Dell tower and also on an external hard drive. And because of the way Recover My Files worked, I had up to half of my files named Recovered 039203.mp3 or some such. So some of my files were redundant and I knew it, some I didn't know. But now I had a class of files I couldn't identify but dare not throw away. I know that the next time a drive fails, I want to have a duplicate but now I have all these unidentified files in iTunes and that slows iTunes down. So literally for years I would see a song 'Recovered 034930.mp3' and I would have to listen to it, identify it and change the tags. Metadata management hell.

Recovery meant I ran out to Fry's on a Thursday, bought a new drive, plugged it in and dedicated 100% of that machine's processes to scour the drive. Maybe by Sunday I will have gotten through 200GB. Recovery is non-trivial. Deduplication is both necessary and risky. 

In the normal course of dealing with non-big data, having multiple versions makes sense. Deleting and restoring doesn't take much thought. Backup is simple, disk is cheap. Dealing with big data, deleting may not make sense, restoring takes time, yes disk is cheap but now it seems very slow. 

I've had websites for a long time. I've also had FTP. But I never considered FTPing anything but the most important files up to my website for safekeeping. Like most folks I used Google Docs. Then I got Flickr and Picasa to hold my pictures. It was a long time before a web-based music storage service became available. But I did start using Carbonite. But for all of these services, if I wasn't creating content at that web location, I always had to deal with copying data from my machine over the web. Painstaking and slow. The first real breakthrough came with Dropbox, and then with Backblaze. These two services (finally) let me keep my data locally and worked in the background without intervention from me to make copies of my data.  Now I had smart duplication, the kind of redundancy that didn't have the failings of my own recovery and backups. Files were duplicated in a separate place, I could identify all of them, and I didn't have to spend 100% of my processing time and attention making it so. 

When you are dealing with big data, you need to have this kind of backend process working in your favor. Hardware will eventually fail, networks are too slow, metadata management is an awful thing to do manually. 

--

Today I have the following personal setup.
I run everything from my MacBook Pro which I have attached:

OneBox - 1TB laptop hard drive
Bluebox - 320GB of business related files
Silverbox - 3TB of personal, non-media files
Whitebox -  2TB of media files
BBox - 1TB of backup of my main 1TB hard-drive
ZBox -  encrypted information

Backblaze - backup of all the above except for BBox.
DropBox - 10GB of documents including ZBox (redundant to OneBox)
BlueKey - External USB key including portions of DropBox & ZBox

iTunes Match - 25,000 duplicated music
PicasaWeb - some ungodly amount of duplicated pictures
Flickr - some unique fraction of same pictures + more
AmazonMusic Player - about 5000 MP3s
Audible - all of my audiobookx 
Amazon - all of my eBooks
OReilly -  the rest of my nonAmazon eBooks
Evernote - some 5000 documents 
iCloud - Contacts, notes, calendars (also on iPad, iPhone) 

So I have all of my data locally, most of it 1X redundant locally, the more important 2X locally redundant. I have all of my data remotely redundant, some  of my music  and some of my pictures 2X remotely redundant.  I manually initiate BBox backups which are incremental about 15 times a week (Apple TimeMachine)

My most important pictures in the world are on my keychain. 

Additionally I burn various USB sticks, DVD-ROMS and SD Cards with the more important fractions of important files and stash them around the house and in the garage, and physically offsite. Plus I have some stuff in Amazon S3 but I forget what. 

What I don't have redundantly is all my old software, which lie in Case Logic CD cases on my bookshelf. It turns out that almost nothing is as useless as old software. Even Eudora. 

I have come to trust the cloud for backup, but there isn't much that I have there that I do not keep locally, which basically means my blogs and what I've said on FB, Twitter and the dozens of webchat forums I have attended over the years. 

What I have learned is that at some point, you will outgrow your current systems, even if you didn't plan to. When you do there will be barriers you cross (for me it was 100GB collections) that force you to deal with new systems, new problems, new costs, but mostly new habits & new processes.  

Then guess what. It's just data again. Yeah, I have about 6-7TB of data, not counting redundant backups that are my personal data. I spend maybe 2 minutes a day making sure it gets backed up, and almost no time worrying about it.