Cubegeek

Big Implementations

These days I'm thinking about the particular dynamics of big implementations. As I've been reviewing some of my own history, I think it's fair to say that I've been in several implementations that would be considered big, and three in particular that I would be bold enough to say were 'giant'.

Interestingly enough, two of the giant implementations failed, one was scaled back to big and the other really never got off the ground. The successful one went through all sorts of gyrations before it settled down. There are several lessons I think are in the offing.

1. Never Forget the Grinchbag Theory
The Grinchbag is where all of the user and functional recommendations go. Unlike the cartoon Christmas show, they don't all get accumulated by one Grinch in one stealthy evening, but they do accumulate in spurts. Which is to say it's not every day that people get to look up from their desks and start contributing ideas on how systems should be improved. But when they do, the tendency is to take advantage of the political momentum. This often puts a lot of pressure on project managers to include more and more into the rollout. It encourages technologists to have whiteboard sessions which prove that they can do that. It encourages sales guys to sell more tools. It encourages functional managers to prove that they can improve the business. All of this energy is positive energy, but it needs to be throttled down.

There is, in my opinion, only one way to properly throttle down the energy level that you want and need to keep your project(s) manageable and all user requirements out of the single Grinchbag, and that is to adopt the Deming philosophy of continuous quality improvement. This means making partners out of all parties involved and managing expectations into a process rather than just into an initiative. Practically speaking it means multiple short iterations.

Everybody loves the hot breakthrough feature that they've been living without. In my view that takes two steps. One is stepping up to the platform and the second is exploiting the platform. Trying both at once works only in small shops. In big shops trying both at once is fraught with danger. 

Enterprise software implementations are just like blockbuster movies. They're only really a great success when you can't wait for the sequel. You can tell that when people start coming up with their own ideas for what should come next.

Now here's where the new paradigm shift of cloud computing is going to have a huge advantage. In the cloud and the network of clouds which comprise the next generation of enterprise virtual IT, the economies of scale are going to drive innovation out to the entire platform a lot faster than with the coordination nightmares that in-place IT managers have to deal with. But for the time being, smoothness of transition in versioning of inhouse applications is key.

2. Maintain Decision History

Whenever I meet people for the first time, I'm fairly quiet. I listen first. Then I get to a point where I cannot be shutup (well, not exactly). It's more like I'm the guy who says "Now in 1995 you said, and I quote.." God help me to not be passive-aggressive, but I am a servant and I work for you. I do what you ask. And so like Jeeves, I have to remind you of the reasons for which you have navigated me and my team into whatever predicament we're in. I only ask when we get started if you're really serious about building the best system possible. My reply will always be that I will guarantee that you get my best work, but I need a similar guarantee in return. 

One of my favorite tools is a decision matrix. It turned out that I was reading into some political stuff many years ago (during giant project #1) and I discovered that the way the US President does things is often through a 'finding'. In such cases, a set of options are presented given by the smartes guys in the room and a multiple choice signature line at the bottom of what should generally be a single page. 

Maam, we have three options at this point. We can break the architectural rule against using Perl scripts in the ETL path and solve the data integrity problem in stream #7 in 3 days, OR, we can go back to the ER approval process and redesign the Informatica code in that stream in 15 days, OR, we can modify the outline in Essbase so that the data shows up in the same bucket without the ability to disaggregate Plant #32 from the new Plant extension. Advantage, disadvantage. Impact on users, impact on developers, impact on project. Recommendation. Boom, boom, boom. We need a decision on Friday.

Every big project has a dozen of these. Every giant project has more with more serious impacts, and it's always when there are resource constraints, like how many experts there are in a six state radius who can do what's needed. Well, generally they are internal resources, like the time of the head DBA or the guy who certifies the extra RAID needed or the guy who deals with restarting EAR files in the Websphere cluster. It's always a headache, it's always an emergency.

But you always need glue members on the team 'on that wall' who can handle the truth given whatever political circumstances are on the ground who can answer the question 'What were they thinking?'. No technical solution is transparent, and therefore have useful longevity until that question can be answered in context. 'What were they thinking?' has to be answered, over and over. And you need somebody who can do that on all three sides, Functional, Technical, Project Operational. 

3. Managing Doldrums

This is an odd one, and it's a tricky one too. My theory goes a little something like this. No matter how fast your car, and now matter how wide the freeway, traffic always bunches up. There's some complex mathematics to explain that reality that I don't understand, but it's part of physics. If one person steps on the brakes way ahead of you even if he's just slightly slowing down, by the time the rate of his decelleration gets through the line of cars ahead of you all you know is that you have to slow down. But the precise amount of deceleration of the lead car is impossible to judge from your POV. It's like that game of Telephone where people whisper into the ear of the next person. Details are lost in translation, all you know is that you have to slow down.

The bigger the project, the more this phenomenon exhibits itself. Now there may be some way in computer science to synchronize parallel jobs of differing difficulties and dependencies and then coordinate them all out of the pipeline into one coherent whole but I'd say that's harder than rocket science. You may be even able to do it with massively parallel freeways so that there's always enough lanes so nobody has to brake. But even the most Agile of project managers can't do it with IT projects. When business requirements feedback and adjustments don't come from the functional group. When a glitch in the version of the toolkit makes one development piece take twice as long. You get traffic. 

Somebody, somewhere on some big project is always sitting still in traffic. It's inevitable. What do you do when it's a team of 3 developers?  The way you answer that question determines whether or not you are as efficient as possible.

Countdown to the Purge

Well, it just seems that the more I look at this stimulus package the more I find to dislike. But I am looking at the way it may affect my particular industry. I just discovered today a nice hunk of smelly protectionism, that will serve to benefit the American IT industry just fine - well, depending on how you look at it. 

The bottom line is that a big splash of cold water is going to be thrown on H1-B visas. What you have here is a political compromise that changes the rules of the game with some politically correct exceptions for cover. 

One thing we know in IT and especially in IT consulting is that labor markets move very quickly and talent is at a worldwide premium. That's why we have so many folks from all over the world employed in this business. Any old Joe won't do. I could get into the particulars about my segment of the industry and my opinions about outsourcing and staff augmentation. Suffice it to say that it's very easy to stereotype a lot of Indian workers. But the fact is that Indians who make up a visible bulk of guest workers are all over the map in terms of talent. That doesn't matter as we throw the baby out with the bathwater here. 

This is just another zero-sum, wealth transferring, centrally planned wrecking ball headed at the economy. Unintended consequences? Yeah right. 

RACF & Client Server Computing

I was reading Schneier this morning and one of the commenters on a thread about a breach at Monster mentioned RACF. It brought back memories. One notable one for me in particular. The reason I never became an expert on mainframes was because of the follow-up program to RACF which was ACF2.

Back when I was interning at Xerox in the early 80s, the second internship I got was working for a man by the name of Jack Starkey. Starkey was an old school consummate engineer. Buzz cut, white shirts, clubman glasses, slide rule, thin tie, crisp diction. Tough but fair, one of a dying breed. My assignment from Jack seems trivial by today's standards, but took some doing back in 1984. It was to establish a data dictionary for the Xerox PSD Support database. It ran in a tool called Datamanager on an IBM 360.

For this job I had to learn JCL. JCL is job control language and I soon discovered that there were a couple women in the department who worked there full-time making as much money as both of my parents put together just starting jobs and watching jobs written in JCL. It turned out that they weren't interested in helping any snot nosed college kid get the ins and outs of this language. Jack Starkey, quoth he 'RTFM', and so I hacked my way through half the summer.

For every Jack Starkey, there is a Jimbo Jones. I don't remember the name of my Jimbo Jones, but he was a long-haired Wozniak look-alike who was disrespectful and anti-establishment and smarter than anybody else in the building. It turned out that he used to be an almighty SysOp but either got demoted or bored. So it didn't take me long to find out from Jimbo that if I wanted to get certain tapes mounted and have my own tracksallocated on the DASD that there were shortcuts and tricks. You see my job was not only to figure out what the structure of the whole support database was, but to help in the migration process. That meant I had to move data from one data center to another. It still gives me headaches thinking about Bulk Data Transfer and how few people knew how to make it work, and how little help those who knew a little were.

At the time, some slicked back sales types descended down to Xerox to tell us about Focus 4. Focus was a new kind of database, a relational database. So we sat in the auditorium shooting paper clips at each other, Jimbo and I, while the sales creeps began to talk down to us, as if there was something magical about their new technology that none of us could understand. It was rather cool what they were showing, but they had this kind of arrogance that came from owning a technology that even IBM had yet to rollout into a product. After I displayed enough haughty nonchalance and reviewed the entire thing with Jimbo after the presentation, he confided in me. He had hacked RACF.

Hacking RACF was an accomplishment of extraordinary significance. That is because back in the 70s and early 80s, the most powerful computers were IBM mainframes (notwithstanding claims of Fujitsu and Amdahl) but every flop of their circuits was eyeballed and audited by SysOps. There was no time-sharing, there were jobs and job classes. And every job and its status was monitored at all times. The job of the SysOp was to put some business priority around the most important piece of software in the world, which was the heart of the IBM 360 operating system whose function was to allocate precious CPU cycles and even more precious Direct Access Storage Device (DASD), a glorified hard drive of maybe 500MB. I never quite learned how to convert the figures back in the day, but I remember how difficult it was for me to get one cylinder of storage, which is less than one MB. Basically a floppy's worth of data. 

One of the other pieces of madness you had to deal with was the job classes, as I mentioned. So I'm trying to get the Bulk Data Transfer to move database parts from one datacenter on one coast to another on another coast. Nobody will give me more than 1 MB of storage, tell me the proper syntax for bulk data transfer or give me a job class with enough priority so that I can try more than once or twice a day.

So I'm the intern who submits faulty JCL every day and waits for my job to abort. Big fun. I can't stand it.  I meet Jimbo the week before the company cuts over to ACF2, the new, unhackable security program, and he only tells me that he's hacked RACF after ACF2 is installed. Wonderful.

This became the one job I ever had where doing the documentation was more fun than doing the programming. For my documentation I used the Xerox Star Workstation, which, by the way, was hooked up to Fuji Xerox. I could send emails to scientists in Shibuya and even print documents on their printers, but could I get 1MB of data from Rochester, NY to El Segundo? Not on your life. So it was decided, I'm never going to give mainframe people the time of day.

One of my other buddies there who was also sick and tired of working with these slow tired systems left and worked for a company called Britton-Lee. He did pretty well for himself. I was so sick of IBM that I decided that no matter what, I was going to work with the workstations, client/server and object oriented programming. Unfortunately, it took another decade for business data to find its way over to peer networks. Thank God LU 6.2 was a colossal failure, because I'll be damned if I was going to take it from AS/400 folks. 

But I did have one more stop on the IBM train that moderated my hatred for them and big iron. It was called VM/CMS, and I actually loved it, but that's another story.

Staging Essbase Data with Hand Built ETL

The best fun in being a hacker and a crank is that you have this internal notion of what you should do that makes perfect sense, but you can't explain it. My new habit is to use Audacity and explain it to myself for the times I get back to needing that code several years later.

For example. I know that I have some code that I put on a timer to go check and see if the Hyperion Reports server is up and running. I forget the guy who wrote it or what modifications I made to it. I just know it's in a big pile somewhere in my new All Customers archive. But I think of it every time I need historical fx data because I know it was hacked out of a general purpose web snarfer.  I had that written and carried over from another customer. And interestingly enough I just remembered that part as I wrote that sentence.

Well I just finished unit testing a new Essbase back-end. This one is actually fairly cool. One of my favorite things to do on backends are 'fabs' - little perl programs that fabricate other scripts and then execute them. So this fab generates a calling program, a wrapper around a stored procedure with a couple parameters that I craft.

The back end is like this. I've got a weird proprietary HR database with a wide view of about 25 columns, about a half-dozen demographics, 3 or four geos, 4 dates, some titles and grades, position numbers, and salary. It's small, only about 500 employees in the table, but enough master data to generate 17 dimensions. The HR Source is on super lockdown, and therefore I can't query directly into the transaction system and the specified view even goes through another database with separate security. So basically my account has access down to an absolute minimum. All of that is on Server A. Transacation database, and a 'tunnelvision' database.

I get a push of the tunnelvision view data into my ODS which is on a slightly different form of lockdown. Bascially just an administrative account on the server plus one more account which belongs to SQL Server. The source is pushed into a table. Now I have a workplace to create a bunch of status flags which give various date fields multidimensional meaning. Plus I calculate a tenure metric for calculation of vacation days. And of course I have the facility to incorporate any new business rules necessary.

The tunnelvision is stateless so I have to stamp it - and that's the fun part. I hacked one of my old code archiving programs which gives me incremental version numbers and a date stamp into a new cut field in a historical table. I just pile on that whenever I initiate a manual refresh into the cube.

So I've got one wide table that updates 17 dimension tables, one narrow minimum fact table (there's just one fk that ties to most of the attribute dims) and one historical table. I have a fab that creates the calling SQL for a five step stored procedure and takes parameters from the same date facility that updates the substitution variables.

The coolest thing is that I have the ability to either let the batch program run this whole thing or I can initiate it from the user desktop. That was done via some .NET programming by a guy on staff here. Just double-clich on an icon and a .NET program initiates my batch file from a browser. The only problem with that is that the path changes when it runs from IIS instead of my Essbase batch subdirectory, so I can't use any relative pathing internal to my scripts. But now end users can be on the phone with HR, know that they are about to commit a change to their system and can initiate a refresh of the whole shebang end to end in under 5 minutes. 

Stored procedures have generally been a mystery to me precisely because DBAs are jealous of their positions. So being the multidimensional guy I never get access to do with SQL what I would generally want to do - I just provide the specs for the interface, they disappear for weeks and come back grumbling. But it turns out that they are super simple - at least they are in SQL 2K5 Express which is the freebie I use for my ODS.

The most difficult thing to do here was pass the security review. It's a huge help that MaXL now has the security hashing that encrypts your IDs and pws, but my server was closed enough to not require that. I really dig dealing with streams of data like this, mixing perl, SQL and Essbase calcs to each optimize that part of the final picture the end user is presented. It was a great discipline to learn how to manage those kinds of streams independently and asynchronously as they dropped into my world's inbucket. Now being able to do this on a standard batch schedule and on demand using the same code is the new cool thing at my customer.