In my world, multidimensional design for analysis starts with knowing what the customer cares about. Ultimately I'm oriented towards solving a business problem - coming up with the right analysis for the kinds of problems the business is working on. When I have that kind of focus from the customer, time disappears. Most of the time, however, I don't have that focus, and so time is an issue. Thus, as a builder, I have to lock in schedules and project plans and all that. That's where agility comes into play. I'm talking about agile methodologies here. Agility leads to Focus.
Let's step back for a moment and try to recognize a few things.
- The Right Tools
- The Right People
- The Right Focus
If you have those three things working in sync, then is is just a short matter of time before a productive system is built. An agile methodology helps you get the right focus. What does that mean? It means you have a way of quickly determining what to deal with and what to throw away. That counts for code, it counts for requirements, it counts for data.
The keys to agile methodologies are informal communication, rapid cycling and close proximity. They are all inter-related. You basically need all three.
Informal Communication
This means that you do not create documents ahead of time. You create a lot of messages in real time. You talk a lot. You meet a lot. You repeat yourself. You ask a question as soon as possible. You get up and leave your desk to talk to the DBA when you get seven records instead of one. You call the AR guy to ask exactly what it means, this 'debit memo' thing. You whiteboard and you take pictures of your whiteboard. You have a conference call and you make a recording of the conference call with Audacity and then convert it into an MP3, burn it to disk and hand out copies. You blog your status reports. You have everybody's IM and you use it for stupid questions all the time.
When you work like this and people get used to it, you create a no-excuses environment. This is not a place for people who tend to drag their heels and be bureaucratic. This is not a place for passive-aggressive behavior. That shows up right away. When people don't want to communicate, when they hold things back, when they can't say what they really think, Informal Communication exposes them. When you have highly detailed documentation and requirements ahead of time, those people can hide, and they will sabotage your project. Why? Because if they are not 100% right the first time, getting from good to great becomes impossible. They will be too busy bowing at the temple they've created to see that Indiana Jones (you) have come to change things. Big Fat Documents become a stack of technical bibles, and when people swear on that stack of bibles, they build a Temple of Doom. That is the enemy of Agility.
Step Back a moment.
You've got to have the right tools and the right people. The tools have to offer ways to do things quickly so that fairly major changes can be made. Tools that encourage you to experiment are good. You want manual transmissions, not automatics. You want manual steering, you want to be able to turn the traction control off.
Now you have to have people who are willing to take the risk to use those tools. People who are ready to experiment, pop the clutch and see what that baby can do.
Back to Agility.
Rapid Cycling
The most important thing about an Agile methodology, in my view, is that it allows you to make mistakes quickly. A quick mistake is instructive, like when the Kung Fu master puts you to the mat in half a second, and then you get up and try again. Or like when Rafiki hit Simba with his walking stick. You learn quickly. The theory I'm getting to is the OODA Loop. How rapid is rapid? Well obviously you have to make enough mistakes in a short enough period so that you get it right by the time you need to. Your design for analysis has to be done rapidly enough so that it captures the end user's best guess as to what they need to analyze. I say best guess, because as you build, and they see new data for the first time, their understanding of the value of that data changes. You and your end user need to co-evolve that understanding. That means you will make mistakes.
On non-trivial tasks, you cannot control how many mistakes you're going to make. What you can control is how quickly you react to and correct them. This is key to developing a design which is going to be more adapted to your customer's evolving understanding of their own analytical needs. The bottom line is to make change your friend.
That is not to suggest that change doesn't have costs, but you first have to understand how quickly your design and development team can react and then what the pace of your customer is. Then you can communicate (remember?) what the incremental cost is for each change. What you don't want is the 'You want it when?' joke cartoon floating around. Get an understanding about how much can change, how quickly from your customer and then get inside his OODA Loop.
Every problem about IT's backlog is exactly this problem. The end user changes his mind faster than IT can develop products to support his mind. Unless the end-user's business is catching bullets in midair on Monday, milking cows on Tuesday and building nuclear powerplants on Wednesday, chances are that your IT's OODA Loop is too slow, not that the user changes his mind too quickly.
- Don't go into a skunk works mode.
- Don't build black boxes.
- Don't be afraid to tell your customer your secret sauce.
- Do reveal all your assumptions.
- Do ask stupid questions.
The more you cycle, the more you hone in on exactly what is important and why. This is an important learning experience for you and your customer. Even if you discover that you don't have the tools, or the people to actually realize the system.
Close Proximity
Almost nobody I started programming with liked working in a big open space with other people around. Not even in college did the best programmers work in study hall collaborating with other programmers. We all expect programmers to have extreme focus and that means isolation and silence. It doesn't work for Agility. Bullpens are best.
The programmer who can work in a bullpen, in a conference room with five different conversations going on is on his way to mastery. When the guy doing ETL is sitting right across from the Business Analyst who is rubbing elbows with the Front End designer, who is IMing the DBA, that's when the magic can happen. And these are exactly the same reasons why design work for analysis cannot be offshored. You need proximity. You need to be able to eyeball the work of other people and they need to be able to eyeball your work as well. End users need to be able to drive the system before UAT. What? Huh? Yes. Database architects need to be able to see the green screens of data entry personnel. Nobody can be too busy or too remote - these are not acceptable excuses. The more that everybody knows about what everybody else does, the more robust the system becomes.
Today we discovered that we need to make a combination key for invoices and vouchers for our data mart. Within 1 hour of that discovery we had three volunteers to do it in three different places, scrubbing before the ODS, modifying the SQL from the Cube to the ODS or changing the Essbase Load Rule. Now we have choices, That's Agility.
Agility Leads to Focus
When these kinds of tight, informal communication, that is rapidly cycling and in close proximity, meetings happen spontaneously, everybody in the room knows exactly where the kinks are in the system. The critical path reveals itself. This stuff tends to drive project managers crazy, especially the waterfall guys. Everything seems to get done, but nobody in particular is accountable. Everyone is accountable and the PM can't get around quickly enough to find out what's going on. This is an excellent situation, but it is one that is uncomfortable when you don't understand the dynamics. There is not much of a stepwise progression in this analysis, design, development loop. It's all happening at once. Lots of builds are getting done quickly, and people have to whiteboard where the new lines are going. And of course nobody wants to stop and write up a technical document.
Let's not forget however what the aim is. The aim is not to get rid of documentation or accountability. The aim is to gain focus. The aim is to insure that the system that gets built is exactly what the user wants without the problem of locking down specifications in a stack of bibles. The aim is to create and sustain a dynamism in an extended team that gets many different kinds of minds figuring out how to see in a way the company has never seen this piece of business before in an actualized analytical system.
That is the most exciting thing about designing and building analytical systems. That's why I almost never forget a good engagement.
Great analysis of what agility really means on the ground. I have often experienced the same but not able to express that well - the process stuff with an artificial sense of certainty seems more business like than the touchy-feely benefits of informal interactions, iterations and learning.
I have often found that a day's meeting in a room is worth more than 2 weeks of concalls with non-verbal clues, ad hoc drilling on topics, lunches, etc. Also the more 'failures' a system suffers in the early phases, the better explored and robust it becomes at the end.
Have you found any good article, book on this topic, most are too generic...
Posted by: MT | August 10, 2008 at 05:24 AM