Cubegeek

The Terse Programmer

A good programmer will always think of the 0 case, the 1 case and the infinite case. Any calculation will fit into that and so we are always quantifying how much it takes to grind out an answer. But that also means we have to categorize our assumptions in order to come up with a reasonably proper answer. It was easier to ask such questions when the world was new, but now a lot of them have been asked and answered (I leave it to you as an exercise to qualify the query space of Quora itself). 

You should ascertain by this that the quietude of good programmers is born of respect. You are either asking a question with a trivially easy answer and so we are redirecting insults to our internal dev/null, or you are asking a question that needs greater specificity or you are asking a well-qualified question that requires significant thought. The second type of question can yield great amounts of humor.

For example, my wife has the habit of asking me questions like. "Honey would you like me to fix you a snack now or can you wait 45 minutes to dinner?" Now to me this is a simple yes or no question. But it requires a yes and a no. So I will say "No, yes", and then she will become confused. She wants me to say something inefficiently verbose, like "I'm kind of hungry right now but if you're fixing something really nice for dinner then I'll just grab myself a beer and wait."  To a programmer, "No, yes" is the proper answer, to my wife and the rest of humanity, we are just being cryptic assholes. So sometimes we have to refrain from using our own unique sense of humor and send those responses down another path to the same mental bitbucket. That path often triggers a bit of emotional resentment.

Those of us who adapt well understand that and so we have developed fluffy verbose social skins over our lean, mean logical question parsers. But we often have to worldswap and figure out our contexts to check which skin to implement. Depending upon our environment, this takes a noticeable amount of time. Some of us even attempt to build human interpreters on the spot because the questions we get are so stochastic and meaningless that we simply don't have a map to the proper context. Sometimes it's easier to put on the 'asshole' skin (c. f. ehrlich@aviato).  

Programmers are not quiet people. Our brains are always busy doing things, and we are always shouting at ourselves. What you are encountering is a very sophisticated set of filtering mechanisms that generate something approaching a simulation of civility, because most of the time your questions are not as interesting as the questions we ask ourselves, and you're interrupting us, dammit. 

Be glad you don't know programmers without filters and social skins as we are wont to replace you with a very small bash script.

The Curator

We already have the concept of the Producer and the Consumer. Now we add the concept of the Curator. 

The purpose of this is to regularize the idea of dynamic resizing of the query space. The Curator would thus be responsible to know which data are permanent and unable to be further modified. Let's take the example of a financial data warehouse. Generally, we allow modification and restatements of data past the close of a fiscal period. At some point however, there needs to be a cutoff. This may be one period, two period or perhaps 6 or 12. The longer the period, the more transactions we would likely process in any refresh.  For the sake of our example, let us allow that cutoff to be three months, with a grain of one month. 

So every day we get a restatement of the current month. This is to be expected. But we may also get data which include transactions that are up to three months old. However on June 1, we would close off the trailing month of our three month window and accept no further updates to March. At this point, the Curator would engage the spinning database and push out the final version of March. This should be done in the most efficient manner for the sake of a database load. So the Curator has a kind of producer itself. This Final of March will go into its own S3 bucket and be registered with a Curator table.  Obviously this is a job that can vary in size with the desired grain of curation.  I would recommend a month or a quarter. Lets call this operation a Curator::SetFinal(2014-03). This makes a permanent copy of the last official update for that month of data and stores it out to the Curator's store.

Now depending on how we want to handle it, we can either spin up a new size database in parallel, or drop and rebuild the production database overnight to the new size. Lets say we want to do that operation to expand the history window from 3 years to 4 years. We could call that Curator::Exhibit(4 years), and that would pull the appropriate data from the Curator's store. Bottom line is that the Curator can dynamically resize the query space by using data certified by the application. 

Not bad eh? 

Thinking Change

Global logistics and supply chain management are the major force. Combined with information technology and improvements in mass communications, a global consumer class has been created that never has to deal with 'please allow 6 to 8 weeks for delivery' which was the common standard in the 1950s. These will continue to drive change - the internet hasn't changed 'everything' until the transportation rich delivery model is substantially replaced by e-commerce.

Today's model of the distribution of consumer goods remains largely as it was 50 years ago. Raw materials are collected one place, products are manufactured and assembled in another place, they are warehoused in distribution centers which feed retail establishments. People drive their automobiles to these establishments and shop. 

The advances in logistics and supply chain have made the creation of products so much more efficient that having container ships from multiple foreign countries is possible. For example, the typical t-shirt is made of cotton that is grown in China, shipped to and woven & dyed in Nigeria, shipped back and sewn and silk screened in China, then shipped to and warehoused in America to be sold in stores like JC Penney. 

People think that buying expensive 'green' automobiles will save the planet, but until the bulk of American shopping is delivered by the efficient logistics of UPS, Fedex or Amazon, the cost of product delivery to the end consumer will remain high. All you have to do is imagine what you would save by 'carpooling for groceries', that remains in the future. The very existence of shopping malls costs in transportation, and inflates the price of consumer goods.

However what we are beginning to see are faster and faster business, product development and delivery cycles that approach the speed of memes. 

It should also be said that with this ability to deliver consumer products more rapidly to broader populations is the increased demand for quicker and less disciplined avenues of financing. Global finance has radically changed with the diversity of funding sources for business. 50 years ago, there was no such thing as an MBA. The ability to make money as a capitalist has been 'democratized'. There remain many parts of the world where people do not understand money at all - but market economies and the associated social knowledge are flourishing, whereas in the 1960s millionaires could be thought of as a very distinct class of people. Today, hundreds of millions of people around the world have access to credit, which was unheard of.

I am always reminded of the famous shot of Hank Aaron's record breaking homerun. There is a large billboard ad in the stadium for BankAmeriCard, which at the time was the first time such things were offered to the general public. Today, you can buy all kinds of plastic money at American supermarkets.