Cubegeek

Data Analytics: What I Do All Day

What I do all day is put everything in boxes. I am a compulsive organizer. (Yes I subscribe to Things Organized Neatly) My organized things are data and the boxes are systems. I'm trying to figure out the best way to put all the data into all of the systems, but at the end of the day I'm a tour guide. I help people find the data they want in the systems they can afford. Sometimes the boxes they want are too big, most of the time they don't have enough time or interest, or they find the whole matter too confusing and would rather just guess.

What's interesting is not the techniques and methodologies of building data systems and the attendant analysis. What's most interesting is understanding how it is that people come to understand the data they feel is most important to them. What do people want to know badly enough so that they'll pay me money to organize it for them? You're a sailor, what does a sailor want to know? You're a security guard at the front desk of an office tower, what do you want to know? You're the buyer for oak barrels in a distillery in Kentucky, what do you need to know to do your job better? You're a doctor for a major airline trying to figure out the rate of respiratory infections for flight crews.  You're a car dealer that is thinking about fleet sales. These are all functions of running a business, or any operation that requires some stream of data that changes. A guy like me finds a way to turn a static pile of data consumed just once, into a reliable flow of  information that informs your work over time. 

It still comes down to counting things, figuring out percentages, risks, keeping track of assumptions, proving that numbers tie out in audits. It requires persistence, an obsession for perfection when people would rather be sloppy, a head for logic, and the ability to communicate what it all means. It means keeping track of a million moving parts and using compute & network tools to do so. It means being economical about which tools to use and why. It means listening patiently to people with half an idea. It also means never knowing enough because people are always unsatisfied. 

It's a field that will always be around as long as somebody is building the machines, because people will always want to know something and we've pretty much figured out how to make them trust machines. TV paved the way. People will want to know then they'll change their minds and want to know something else. Human curiosity is not infinite, but it's pretty large.

The Riverwood Story

This is an old story but particularly interesting. 

1995, I was working for a paper products company in Atlanta. I cannot even recall the project's details except to remember that most of the employees of the company ran PCs without TCP/IP stacks and all of their networking was done by Citrix. Still, the project was going slowly.

I got a chance to meet with the president, whose family was involved in the business and his top financial analyst who looked like Lucy Liu and was deadly with a spreadsheet. It was clear that these two were the brains of the operation. We were going over boring bills of lading, literally shipping documents when suddenly the president got an insight. He picked up one piece of paper as we sat in the conference room and called a warehouse manager.

"Ralph, this is Sam at HQ. You know our customer X?" 

"Yes it's our biggest customer"

"When did you last send them a shipment?" 

"Just yesterday"

"And how much did you send them?"

"1500 pounds. We send them twice a week, Tuesday & Thursday"

The president went on to discover that the warehouse manager used a flatbed stake truck that had a capacity for ten pallets and that 1500 pounds of product took up three. But the thing that caught his eye in the paperwork and that he knew was that it was customary to discount the freight cost when selling to your biggest buyers. This was a discount given at the discretion of the 85 warehouse managers nationwide. The freight cost was calculated per truck trip and tied to the cost of gasoline. Why make two trips with half a truckload and give the discount twice, if you could make one trip and give the discount once?

He had the financial analyst crunch the numbers and figure out how much the company could save by making fewer trips and/or not discounting freight. It was massive. I pulled the historical shipping records from the database. Within two hours we figured out how to save the company up to 1.2 million dollars per year. 

The president asked me how long it would take to build a system that would use the spreadsheet formula against the shipping records at each warehouse. Two months, I said. At the time, however, none of the warehouse managers had networked computers. 

Lessons Learned.
1. You need an executive with a keen insight to how the business actually operates. He knew how to read a shipping document, and he understood how warehouse managers work. Why they do things the way they do, how they perceive orders from HQ and what their financial incentives are. 

2. You need to be able to make accurate models of the actual costs involved and prove them out before you build any systems. You can't just build a system that captures 'everything' and expect that it's going to tell you something valuable.

3. You need a business culture where you can push down responsibility for costs and revenues to the people who actually do the work, and show them in dollar terms, how a change in behavior affects the bottom line. If your system only has a few end users at HQ, so what? There's a difference between knowing the right answer and doing something about it. Knowing only took us two hours.

4. You have to have compute infrastructure in place at a low enough cost so that the rare insights of cost savings are worth implementing a system for in the first place.

In the case of that company, number four killed the whole deal. If you counted up the warehouse managers, the cost of upgrading & networking them per warehouse, the time to build the system and the licensing cost per new user, it would have completely offset the cost savings. The company had to end up issuing a memo and policy change, like an order from God instead of building the system to let the warehouse managers see their efficiency rewarded. 

These days enterprise software takes a back seat to the cloud in terms of total cost of ownership. But like 1995, most companies are not forward enough in their thinking to rise to the leading edge of technology. Even so, technology is only one part of the equation in improving the business.