So I've finished my course in V&O 101 - Virtualization & Orchestration that is. Somehow on the working end of this stuff it doesn't seem so very exciting. Then again, we're going for the reliability. So here's the news, it works.
I now have a docker machine which is a vagrant based vm, some ubuntu precision 64 or some such, sitting on my Mac. I spin it up and pull down a git repo into its belly. It spits out a nice plump docker image weighing in around 1.5GB. I can then tag it properly and send it on its way to one of our docker registries, some sitting on customer sites, some in our own little cloud. Then I can pull it, almost like a git repo from one of those registries and boom. There's my application in its own self-contained environment, nicely synched up with the release version of the docker container and the application in concert. It takes about an hour end to end for me to fix a bug an move code flawlessly into production.
Now part of the coolness is the way we have wrapped these docker containers into our process. All of the logs that are generated by the application are encrypted and sent to S3. From there they can be downloaded and decrypted from a command line instruction. Also, they burp out AWS SNS notifications on cue, and other metadata about them are piped out to DynamoDB. So we have pretty much got our container act together.
In other news I just made an interesting sort of breakthrough. I figured out an implementation vulnerability with AES. Now it's not that I'm a genius, it's a fairly well known vulnerability, but I figured it out on my own and fixed it with an alternative. Now I still depend upon a cipher coded into OpenSSL, but I am that more confident and my customer's data is that more secure, and that makes me feel good.
What else is new is that we have extended our ability to automate dumps of massive files and work with full and incremental streams. This is a new feature of the ElasticBI Producer framework. Where we used to crap out our JVM around 80 or 90 MB per query, we are regularly doing 300 to 400MB per single query. I have it on good authority that we are essentially unlimited but we're not hitting any large such requirements yet.
We are expanding our ability to handle odd data. These days I'm parsing standard reports involving securities trading and generally boning up on my conditional logic. I.e if you read header type 4, then the next data line will be type 3 that you scrape to file X which gets put in database B, but the following line (if is stards with GNMA) is of data type 5 that you push to file Y that loads into data table R.
We're also implementing the new key value pair XML and JSON capabilities of Vertica. I will have done that up by the end of this month, and will report on that kind of fun stuff and as I do more integrations with DynamoDB I'll have some better rules of thumb about which data best goes which way.
That is all for now.