Ah. So many changes, so little time.
I think that when I look at all of the things I've learned about myself and technology over my career it is that there are two things I love the most. The first is having my hands on good technology and building a system everybody is proud of. The second is sitting down with project sponsors and proving to them that the technology I am selling absolutely does what I say it can do.
Before I do the second thing, I generally need to do the first thing. But the industry has changed. More of these components have goodies built for them or with them in mind to make all of that building so easy that it is almost unnecessary. On the other hand, there is so much more variety, size and shape of data to be integrated into new architectures that what was once simple implementation has become much more complex. This is especially the case with security and privacy concerns.
What I'm doing these days is taking what I have learned in DevOps and shrink wrapping it around my favorite database, which is Vertica. It has the most powerful engine for data warehousing. Data warehousing is still the king of structured data. And the toolset for structuring up unstructured data is improving all the time. So I'm going to go with structure. No surprise there. What it surprising is the extent to which the speed of data transformation can take place. I'm talking about Kafka, which is still a bit clumsy, but capable of great things.
Of course you know I have to talk about LLMs. Well all I'm going to say is the following. I find it difficult to believe that LLMs as a service will not pop up everywhere. For me, the trick is to find a way for individual companies, especially large ones, to enable private LLMs to become referential to every pool of data they own. In other words, this thing I'm coming to understand as Prompt Engineering will soon become the most extraordinary UI ever, and a lot of interactive prompting and responding with become the SQL of the future. So I'm looking at the probabilities of architecting NLP interactive stuff to bring back references to curated structured data.
Comments