Cubegeek

Wonking Through Consul Constellations

I first heard of Consul sitting over a wooden table in New Orleans three summers ago. I understood about 27.5% of what was being said, but the fact that it was said with such excitement was clearly impressive. I know more now of what I didn't know, but did appreciate then.

My task now is to roll my own fractional Cloudwatch. I am on-premise and have several clusters of sharded databases to look after, many of whose nodes can be flaky. So I've set up a datacenter in consul to keep multiple eyes on multiple services, including NFS. I have a combination of RHEL and Centos servers as well as a Mac and a couple virtual Ubuntus to help me out. It turns out that I am hesitating in writing everything in bash. But I  don't want to pull down a full kit of Ruby based containerized apps, even though it would help me digest working with json everywhere. So there's that. I'm also working with consul-alerts, a third-party Go app I've installed on my Mac. It's a bit much to learn the fundaments of Go, but now that I've stepped up my VM game, I'm not so afraid to experiment. My customers are staffing up to make moves into DevOps, so there is a growing coterie here of people who can appreciate the various fruits of the .io trees out there, but still many of them are forbidden. I've already done a simple Slack integration, but Slack is unknown to this enterprise. I tread slowly.

In the meantime, I am appreciating yet again, the other side of the business I've been engaging for decades. The duality of operations which, like war, consists of long periods of boredom punctuated by moments of sheer terror. Hedging against the terror is the 80/20 risk management, complicated by the political arcanities of the budgeting process. It's a different kind of sanity, I'll tell you that.

I'm learning how much I may actually come to appreciate HTTP APIs. Jury is still out, but I already love KV stores. More later.

 

First Ideas on Blockchain

So my first idea about blockchain is that you open up applications at the lowest level, or perhaps at agreed-upon aggregations, and spew it out for on-chain or off-chain management. My interest in blockchain is primarily about attaching proper off-chain big data as referenceable. So the first place I want to start is perhaps with something like public data and then mark that data to a standard, and then sit it in S3 or some likely public equivalent place, maybe like Sia. My aim is then execute (perhaps with Fugue), a simple infrastructure as code deployment of a pre-certified dataset into a standard Aurora, or massive KV store. The trick is to automate, with a simplified PITbull like architecture, a data warehouse of certified queryable data.

For the purposes of Reference Data, I don't think I need to reverse-engineer a FIPS compliant system, but that would be a decent start. After all, it's the integrity of the data that matters most. And if the dataset is signed, then you're in good shape. The question becomes for big data, how big a chunk should be certifiable.

The System is the Business

Some business prophet once said that 'The medium is the message'. I'm not quite sure I believed him. For decades even. I was all about the content. What a person knew was more important than the way they came to know it. Now I have learned how I was short-sighted and wrong. I didn't understand that elites would jam the doors of a few select places to learn what it is that anybody can know, but only the smart money and the big money and the fashionable money funds the elites. See? It doesn't matter that you can sign a high G so much that it matters that you sing it on the right stage in front of the right audience. That gets you out of the fat tails, and who wants to dwell there?

So it occurred to me last week, having attended an interesting meeting on blockchain that what SAP did with ERP, blockchain is going to do with everything else. Hear me when I tell you that everything you know about business, in your gut, is 'inefficient' and you will be replaced by 'efficient' software and systems. The world does not sustain enough OJT and universities to teach people the ropes. They will teach people how to work virtual ropes on their iPads, and that will be the business.

The COO will be the master of all systems. Period.

Now you know.

Recent Thinking about Bitcoin

The point which needs concession is that if Bitcoin weren't worth money, its infrastructure wouldn't be built so well. The warning is that just because a currency is metric, as opposed to Imperial, doesn't make it superior. All Bitcoin can do is gain market share; it cannot determine the shape of that market. What is there to predict? What there is to predict about Bitcoin is that it smells like a data haven, and seeks the complete and total isolation from 'evil' governments and authorities that makes it draw the untrustworthy. If that's all we may be, then Bitcoin may be our own hope. But I think in that limited view of humanity is a grave mistake about what commerce actually is and what it should be. 

In other words, Bitcoin establishes a 'trust nothing, validate everything' standard that inherently trusts algorithms and inherently distrusts humans. It's attractive and repulsive at the same time, because it must attract actors that trust no one. Are such actors trustworthy? The Bitcoin system assumes that it doesn't matter. Everything that contributes to the inviolability of the system must ultimately be accepted. In other words, it assumes a million monkeys will inevitably give rise to an emergent Torvalds, and that good things will follow. The problem is that nobody is quite sure how big such a swarm of hive minded machines running the algorithm must be. So there is a kind of overweening faith (yes I said faith) that Bitcoin or any such currency so dedicated and consecrated must be, by definition, too big to fail.

This seems to be the article of faith. Bitcoin is too big to hack, and too distributed to destroy. 

I suppose I wouldn't have a problem with that if the fans and evangelists of Bitcoin weren't aiming to be disruptive. But then again, Bitcoin is Bitcoin and currently inseparable in the popular imagination from the blockchain. But a blockchain is just an audit trail. Its trick is having a trustless blockchain, and the ultimate difficulty with a trustless blockchain is that it's slow. So the objective for me and people like me is to figure out how to define what's in between. That is to say, in order to sustain a market for distributed blockchain auditability, do you need a cryptocurrency?

Half the battle then is trying not to let your head explode when considering what we actually trust humans to do. That's a social difficulty right now for the generation and segments of society that take short attention span theatre seriously. It would be difficult to convince anyone in thrall to the trainwreck drama of our current presidential politics that there are large, strong and trustworthy institutions in this country that aren't petty and corrupt. That's a large problem because the public imagination is captured by that dysfunctional stream. But let us presume that there are a class of people, let's call them Swiss Bankers, who still can be trusted to manage semi-autonomous systems in a perfectly responsible manner. And I do me perfectly responsible. 

I am of two minds when it comes to the question of insurance. Insurance is easy to understand and accept. If you don't trust cryptocurrency and you're focused on who is dealing drugs and who is hacking miners, then you can add the side business of currency insurance. Something less than the full faith and credit of the US government is required for a market of something the size of global marijuana and heroin. So an economy of that scale can work. (Fill in blanks here with details). But such a market is not insured.

---

Well, this is all about the 'libertarian' aspect of blockchain tech liberation, and that is only of passing interest to me. What is of more interest is the ability that instant settlement and value chains bring to civilization. More about that next time. I've met some of the people who have done this thinking before, thank god. 

My main questions at this point around this ledger tech are the following:

• different trust models
• transaction speed
• asset tracking
• on chain vs off chain data
• identity management & complex ACLs
• off chain data mirrors
• reference data
• non-financial applications
• energy efficiencies in mining

 

Vertica Idempotent Set Transformational Denormalized Design Standards

 At Full360, we have developed a best practice around our design of greenfield and re-engineered DW applications. The following is a high level guide to how we accomplish this in Vertica. Vertica optimization is something we have pursued with vigor at Full360. There are several different levels at which this can be pursued. Implicit is the modularization of the applications so that the major functions of our data management philosophy can be expressed discretely. But let's get to the $10 words, shall we?

Idempotency
Both I and JDub could go on at to absurd lengths about how important this is to the modularizationf DW design. I will simply, which characteristic casualness, tell you that it makes all of our stuff idiot proof, in that it makes our data provision dependencies kind of go away. The basic idea is that in application units(which basically means the chunks at which the data to be consumed makes process and business sense)  you make your input streams discrete.  So when your input streams are discretely chunked, you can run your process over and over without concern about whether it has been done once, twice or never. You just run that independent data provision job and it creates the right sized bucket of data.

Set Transformational
VLDB folks are probably familiar with why you do ELT rather than ETL. The simple way of saying it is that database developers are more stingy and efficient with data than ETL developers. I developed a taste for hand-crafted 'ETL' back in the days when Informatica was a baby, and having my Unix biases, I always loved moving files around. At the time, my focus was on Essbase which had not ETL hooks, even though Arbor should have purchased an ETL company cheap. Interestingly tangential, Wall Street has never been very long in ETL companies. Anyway, I expect that Informatica and Talend will not like to hear that their days are numbered, but then neither did Carleton and DataStage, and they used to rule the world. The bottom line is that moving data from table to table using SQL rather than in a GUI that does not is going to be, in certain databases, much faster and more human readable than in third party tools. So we do set transformation, and even regex stuff, inside Vertica. One day we may even benchmark UDFs against external programs.

Denormalized
Vertica like Redshift is not a transaction database. It is columnar and it easily handles 600, 800, 1200 column tables. It was designed to. So there is no reason to do a lot of silly little joins in silly little tables to get juicy fat data. We make all that part of the ingestion process, which gives us what we want. Think about it for a minute. Consider the volatility of lookup tables and dimensions as compared to the volatility of atomic facts and transactions, aggregated or otherwise. The facts will be bigger and more fluid. So why spend join energy on query exhibits over the long haul when you can easily have all the columns you want? You don't have to. There are no table scans from hell, that's what columnar solves. So we go big.

--

Production
I'm not going to talk about the guts of Production other than to mention it briefly here. Production is some of the genius and we have a bag of tricks that is ever-expanding as we deal with realtime, near-realtime, and other odd types of data sources. Yes we lambda with streams and lakes, but we smart lambda. Again with whatever tech makes sense. Right now we're playing with Kinesis and Kafka and our own custom Actor models which we're sure will evolve over time. We're also looking at how to use Redis and other superfast KV stores. So I suspect we will grow many efficient tentacles as we Produce standardized data for ingestion into our columnar DW apps. Nuff said.

Ingestion

_SRC

We ingest data into source tables for each schema as they come to us. No matter how many fields, large or small, we will take them in using a COPY from produced files. Whether in Vertica or Redshift we standardize on UTF-8 with vertical bars delimited and a backslash escape. In some cases, if we've munged up variable length stuff from our own custom regex routines or other JSON, AVRO or semi-structured effluvia, we will have an additional pre-step using Vertica Flex Tables. We are coming up with best practices there too.

These source files should also retain the original names of the fields of the produced data when possible. This assists in debugging with the original developers.

_STG

All of the data that is to be used in the application should then be mapped to a view. This is the staged data. Staged data should be of the increment of ingestion (discretely chunked in application consumption units). That is to say that your _SRC and _STG will carry the same number of records although they are likely to carry a different number of fields once the ingestion is done.

Transformation

_CLN

The clean tables should be materialized tables with human-readable fields that are conceptually discrete. This is generally accomplished through a direct insert from one or more _STG views. Clean tables can be defined with blank fields for further rule application. Clean tables that contain all history for the query space should have the term _HIST_CLN, otherwise one should assume that a clean table is the same size as an ingestion increment. Clean tables should be optimized for the scope of the transactions that take place in their creation. When you are looking at the clean tables, you have all of the data you need from all of the sources presented in the way you as a developer think about the data. There should be very little ambiguity at this point, it's your best look at the details before they are aggregated and rationalized.

_LKP

Lookup tables are straightforward. They should be optimized for their transaction capacity with clean tables.

_IFT

In a complex model, clean tables can be made into Intermediate Fact tables. The intermediate fact tables are materialized tables with all of the necessary dimensions that support measures that can be made across the full query space. These may or may not be exhibited directly, but should be useful in a partial analysis of the particular measures they contain. An intermediate fact table should be the place where window functions are applied. They should be the place where obscure field conditions are made into explicit attribute and status fields. It is important to know that an application may have dependencies of different measures that seem to be dimensionally equivalent, but actually aren't. So by using IFTs we unburden ourselves of the very idea of a single massive star or snowflake that might have holes. Also, we can capture all of the attributes of a set of measures completely without concerning ourselves with the weight of them in the final presentation layer.

So think about this. A clean look of the data will probably not have sensible status fields they will have codes. There may be multiple ways to interpret a certain combination of fields. So whatever you need to support a the full scope of consumable data, even if it means synthetically remastering transactions, you can do with IFTs beforehand. Building these are the real guts of the DW application, and it's where the fun of working with sharp analysts come in, especially when it comes to data integration projects. I will apply all of the reasonable and semi-reasonable business rules here. This is where I have enough detail to point machine learning, because at this point the data is explicit and human readable. It will reveal the more interesting cases and outliers. And here it should be very rich - beyond human comprehension, so yeah maybe you don't go full wide with these tables, only add 4 of the 50 psychographic customer tags you have against their ID, because, security.

_XBT

Exhibit tables are materialized views in some cases but generally views that are presentable to the end user. These should be indexed and optimized for query retrieval. Security access rules are applied to user groups, etc only to exhibit tables. It should be assumed that none other than dbadmin processes will have access to any tables or views but the exhibit tables.

--

So there it is. This is rather the state of the art that I have internalized in five years of cloud based columnar data warehousing. Maybe I should write a book.