Cubegeek

Seven Rules for Saving All Your Stuff

The most important thing about backing up important files is that you do it yourself, fairly often and redundantly in different media. That way, over time, you'll have opportunities to rescue data that is important to you, and then you'll have practical experience.

I have been doing this for about 20 years. I have used everything from magtape to 8 inch floppies, to zip and jaz drives, to cds, dvds, memory sticks, external hard drives and USB devices. I've used RAID NAS, and a couple generations of cloud services. 

The most important thing I've learned is that when it comes right down to it the things you value most, you probably don't even have digital copies. 

So I went through this exercise a long time ago (1989) when there was a fire two doors down from my apartment. I had already done the drill so I already had prioritized what I was prepared to lose in a fire, and what I could not bear to lose. My priority was my written diary and personal photos. I had it all in four milk crates and was ready to evacuate in under 10 minutes. Everything else, books, records, cds, clothes - were all replaceable. 

Rule #1 Go through all of your stuff and see what's irreplaceable. 

Pretend you have a $1,000,000 fire insurance policy and your house burns down. OK. Whatever you can buy a better new version of is basically disposable. Actually, insurance companies (and police departments) would like you to go through this very exercise. If you have lived through a home burglary or fire, you'll know this. Everybody should know exactly where that irreplaceable stuff is. My dad's college beer steins. My silver framed wedding picture. My autographed bass guitar. The cuff links from my wedding. 

Next, you would think official documents and stuff would be pretty valuable, but in fact it's mostly sentimental. Tax returns, death certificates, citizenship papers, passports, birth certificates, check stubs, medical receipts, degree certificates. All on record somewhere else. Disposable. Inconvenient if you don't have them, but really... that's only because you're in a hurry and you didn't plan. Admit it. And I have to say, if I had been divorced, or lost custody of my kids, those papers I probably wouldn't want around.

The big pain in the neck is scanning your important personal papers. Just do it. Once you've done it, print them out again and store them in boxes. Now you have originals, digital copies and physical copies. Store the physical copies at a relative's house. If you don't have any relatives, rent a big safe deposit box at a bank (preferably a bank downtown) and put the most personally valuable ones in their vault. 

Now with your digital copies, buy an external hard drive and dump them all over into them. Next buy a premium account with Evernote and send them all into Evernote. Obviously you want to use PDF. It's probably the most robust digital version of anything, although JPG is pretty damned good as well. Next backup your spinning hard drive with Backblaze or Carbonite. Finally put the most important, important sentimental docs also into Dropbox and Amazon S3. 

That's for documents. Next for pictures. 

It shouldn't take long to scan all of your prints. If you have color slides, they are the most troublesome. If you have super 8 or 16mm film, Kodak has an (expensive) processed to digitize all that. I would also search around for a local camera shop that converts Hi8 and VCR tapes to DVD. That's expensive but worth it. Movies of your kids are priceless, no matter how insipid the action. 

I've converted thousands of my own and my father's photographs. I have to say that unless they are 8x10 or larger prints, then once they are scanned, that's the best format. So don't bother getting reprints, you'll enjoy them more online anyway. For those exceptions, there is Shutterfly (I think). Remember redundancy. Make your favorite pictures gifts to other folks. We make picture calendars and Christmas pictures every year and send them to friends and family. Funny how those show up 15 years later. Right now Flickr is the best service for my money. Google + screwed up what was wonderful about Picasa, and Amazon is coming up with a new service. Apple photostream is a brilliant idea, well-executed, but that's just 1000 pix. 

Rule #2: Prioritize what is irreplaceable.

It will turn out that 'all of your digital stuff' will fall into the area that Taleb calls 'Mediocristan'. Basically there will be a predictable bell curve distribution for the number of digital assets you will find irreplaceable. For example, if you pick your favorite movies, chances are you'll have anywhere from 50 to 200. But you are very unlikely to have 2000 favorite movies. Similarly you are very unlikely to have 20,000 favorite songs or 10,000 favorite pictures. Your entire library might be that big, but not your favorite, irreplaceable ones.

The good news is that your favorite movies and songs are probably going to be other people's favorites as well. So movies and songs are likely to be disposable. So is software, except for the stuff you've written yourself. 

All that said, I've got some 50,000 photos all over the place redundantly, but about 2300 that I keep redundantly in Dropbox, Flickr, on DVD, on USB sticks and in S3. All of them are synched to my phone, and some fraction of them I've posted to Facebook and in my blog. I have it covered. 

Rule #3: Be Redundantly Redundant.

If you don't eat a Big Mac, you'll forget what a Big Mac tastes like. But you probably won't because there's a McDonalds everywhere. You should periodically visit your favorite digital assets just to make sure you can. I like looking at my resumes from college and pictures of my wife 30 pounds ago. I like listening to Beethoven's Moonlight Sonata and I could watch the Fifth Element another hundred times. (Multipass!) There's no point in stuff being digital unless you checkup on it. Go eyeball those CDs you burned 10 years ago. Mount that old Porsche Design 150MB hard drive. See what's on that old memory stick. 

Rule #4: Review and Recycle. 

Put old media into new media. Put new media into old media. Look at those old Grand Canyon pictures, are they really so precious. Your priorities change. You *should* overspend on your redundancy until you get to the point where you know you have too many copies of that one song and it's costing you too much money to store it. Then you reprioritize again. See if there's a better storage deal out there. 

Rule #5: Plan for the Disaster.

After the disaster happens, are you really going to care about all that stuff? Maybe you could admit that somebody else has a better collection than you? Have you really had a garage sale forced on you because you were desperate for money? Then you know that #3 wood you used to hole the Par 3 at the municipal course is actually only worth about $20 bucks. 

I learned these lessons because every year without fail for about five years straight, one or more of my hard drives would fail. I got really good at using recovery utilities to get data out, which is why I have thousands of songs titled like Recovered_00493.mp3. And oh by the way, the open source MP3 guys were right all along. So remember that the open source stuff lasts longer.  And also do NOT waste your money on anybody's NAS. If you think RAID5 will save your bacon, think again. (Up your's Iomega!). Just check around for what people charge for recovering hard drives. It's highway robbery. 

Rule #6: Do it yourself, plus the Rich get Richer.

The best kind of redundancy is the kind where you can depend on other people's self interests being aligned with your own. Find the big dog market leaders and place bets on their technology, but be smart about it. Pay attention to where other people are placing their bets. Amazon right now is a sure thing, and over the past 3 years have kicked Apple's ass when it comes to storing your music. But not long ago Apple's iMatch was boss. Flickr is great for pictures, but pro photogs like other services. Don't follow the crowds, follow the pros, but keep doing it your own way too. A little paranoia goes a long way when it comes to disaster planning. It might even change the way you look at everything. 

Rule #7: Expect Failure, and get over yourself.

If everything you did were so precious, people would line up to do this crap for you. Ask Bruce Willis if he has enough pictures of himself and his ex-wife. I know right? Every system fails. This is all about cost/benefit analysis. But in the end, we all die, and your surviving ex-wife is going to do things in your house that would set your teeth on edge. So recognize it's all small stuff, except for this chair. 

Barriers to Added Value in Data Analysis

There are also Five Primary Barriers to value add.

NR - Not Relevant
NA - Not Actionable
NS - Not Sufficient
NE - Not Efficient
NI - Not Interested

Imagine, for example, that your automobile has 100 sensors and each sensor creates a datastream of 1GB per mile. That's going to generate BIG data. But what do you really care about when it comes to your car? What you might care about is how many chicks you can pickup. In that case, none of the big data adds value. = NR

You might care about how quickly you can stop in the rain. You have brake sensors and you have wet weather sensors. So you can use your big data to calculate your braking distance in wet or sunny weather. The problem is that you are 500 miles away from anyplace (or anybody) that can install new brakes or tires. = NA

You might care about trunk capacity. You have sensors on all four wheels but they all combine into one number. You can't disambiguate between weight in the front seat from weight in the back seat or shifting weight in the car.  = NS

You might care about gas mileage on steep roads as compared to straight roads. However to get an adequate sample size, you have to drive so many steep roads (and you live in Kansas) that it costs more to find the steep roads and then calculate the savings, (and add sensor 101, an inclinometer), than you could possibly save. = NE.

Notice in all of these, I am talking about what you care about. If you don't care, then you certainly aren't going to pay somebody to care for you. If your answer to all of this is 'so what?', or 'maybe later', then you clearly are not going to add value. This happens more often than you might expect. A BI project might just be seen as something tangential to the business, or as a special gift to the dorks in accounting and IT. 'Buy him an iPad to shut him up'. Such systems do not contribute value. = NI

---
Understand the goals of the business. Find out who has responsibility for what that can be expressed in dollars. Everybody has to know *something* to make the organization work. Determine what the computer can know.

 

The Four Pillars of Business Intelligence

BI - What is it?

BI is business intelligence. It is the art and science of getting computers to know and think in ways humans do not do well, but need to know in order to better run their businesses. It involves the capture, management and presentation of structured data for use by decision makers. These data are usually organized into multidimensional hierarchies of key performance indicators. The aims of Business Intelligence systems are decision support and performance management.

Business Intelligence is my preferred term for this activity because it is the broadest, but various aspects of BI have gone as

• Data Warehousing
• Decision Support
• Analytics
• Big Data
• Data Science

The origins of these practices are found in the lessons of Operations Management in the growth of multinational corporations during the 1960s, 70s and 80s. Some thought leaders that led to the creation of this multidisciplinary field include the following:

• Peter Drucker
• Walter Wriston
• Ralph Kimball
• David Liddle

PECCC

The benchmarks for decision support and performance management are as follows:

• Productivity: Increase output.
• Efficiency: Reduce costs.
• Competence: Add capability.
• Confidence: Improve satisfaction.
• Communication: Deliver consistency.

Business Intelligence systems are all about improving one's ability to know, and helping one assess the impact of the changes made because of that knowledge. It is cyclical, so that new discoveries are added and the consequences of those results are measured as well. Business Intelligence systems enable a framework of iteration to quality and maintenance of compliance.

Application Scope

BI systems include a selection of IT tools, technologies and practices arranged for the delivery of applications. Each application should be implemented as an independent project which is determined by the aegis and impact of the key decision makers for the company's operational goal. One should ask, who is responsible for the execution of X, what do they need to know, how often do they need to know it? A proper BI application delivers actionable data. You cannot manage what you cannot measure. A proper BI application delivers the knowledge necessary to improve a specific area of the business.

The Four Pillars

There are four areas of expertise for a BI practitioner.

• Front End
• Back End
• Governance
• Inference

The Back End - Architecture & Process

Back end work is primarily about data management, and data modeling. There are two primary sort of tools one uses in the back end. The first is an ETL tool. The second is a database. ETL stands for extract, transform and load. It is a tool or a suite of technologies used to identify data sources, gather them up, translate them into a standard format and load them into a database. A database is a tool that has a store of data and gives it a particular structure so that it can be queried in part or in its entirety. The database makes the data 'live'. The way the data in the database is stored is called a model. The most important task a BI practitioner can have is designing a proper data model, because this determines the ease and speed with which the data becomes accessible for many years to come. When data stored in a database is not easily or readily available we call it a 'data jail'. All data that serves no purpose to humans is, relatively speaking in a data jail. The job of a BI practitioner is to free data from that jail so it can be part of productive society.

The Front End - Visualization & User Experience

Front end work is all about presentation. It is the arrangement and display of data to the end user in a compelling, accessible manner so that the meaning of the data can be readily communicated. A front end designer is responsible for the user's experience in navigating and interpreting the data. There are a wide number of graphical tools that are used in front ends. They are generally, tables and charts. But they are also alerts. They can be text, graphics, sounds and multimedia. They can be presented online via desktops and mobile devices or offline to paper reports. They can be delivered interactively or according to schedules. The front end designer must know his audience and insure that users do not become confused or frustrated when accessing data. Data presentation must be precise and crisp and encourage habitual consumption.

Governance - Security & Lifecycle

Governance work is about the meta process of business intelligence delivery. It starts with metadata management by considering the provenance and security of the data over the lifespan of the application. Who is authorized to use this data? Who is authorized to change this data? What are alternative sources of the data and how are they reconciled? How long will the data be live, when and where will it be archived? When and how must it be destroyed. What happens when something changes? Governance also can include resolving questions about the cost of maintaining the application and the scope of technologies used to deliver it with regards to upgrades and deprecations.

Inference - Statistics & Practices

Inference is the term I use to describe 'data science'. This is a new aspect of Business Intelligence that manages the interpretive aspects of data itself. It is necessary because the scale of BI application datasets have grown exponentially. So particular care needs be taken when imputations and complex composite metrics are made over these large sets. For certain metrics, direct interpretation is straightforward. How many cars? "159". For others more care must be taken. How much to people like those cars? "3.5". A BI practitioner in this specialization assures that the data is taken only as seriously as it can be, and helps balance decision makers' instincts and understanding against what the computer system is saying. They must understand the business goals and the application of statistical analysis to management. They must always draw the clear distinction between the map and the territory.

All of these areas of expertise must be engaged for a holistic integration of a BI system into a business process of continuous quality improvement.