At the heart of Kafka is real-time data. With data at the center of any Kafka environment, it should be the area that gets the most attention, but typically it gets the least. This happens because we see most organizations split their Kafka efforts into three areas: infrastructure, monitoring, and data operations.
Are you running your organization's Apache Kafka on-premise? If you are and you’re still reading this article, it’s more than likely that Kafka is or will be a keystone of your data infrastructure. But it’s also likely your teams are tired of the cost and complexity required to scale it, meaning your honeymoon with Kafka is coming to an end. So what does the imminent migration mean?
An old airport customer of mine (whilst I worked for another company) used to pop someone next to a busy runway with a stopwatch strapped round their neck. The unfortunate person had to manually log the time aircrafts spent on the runway to measure the runway occupancy. All very archaic. Even in those days.
We’re continuously releasing new capabilities to enable more people to be productive and compliant when working with Apache Kafka. Our engineers have been hard at work on a great Christmas present for our amazing community of Kafka users. Before unwrapping our latest and greatest release, here’s a sneak peak of what’s inside.
In this article we are going to explore lateral joins. "What is a lateral join?" you may ask. It's a new kind of join that allows to extract and work with the single elements found inside an array, as if the array was a normal table. Lenses 4.1 comes with a lot of new features that make your life easier when working with arrays: we introduced 6 new functions to work with arrays, better support for array literals, and lateral joins.
If you’re moving data in and out of Kafka, chances are you’re using Apache Kafka Connect. The strength of the Kafka community means we have a large catalog of available connectors for hundreds of different technologies, helping to accelerate the integration of Kafka with 3rd party systems.
Several years ago when I was working on a big data project, I saw something a data engineer shouldn’t see. Curious to understand the level of detail in a new credit score dataset we’d received in our data lake, I queried it. I was surprised at how easily and suddenly my screen was flooded with the mortgage history, overdraft limits and year-end financial statements of my colleagues, and I felt deeply uneasy.
Thanksgiving holiday is upon us. For many of our customers, this is one of the most important periods of the year, with more than 189.6 million U.S. shoppers buying up bargains from Thanksgiving day through Cyber Monday last year. For them and for us, it’s crucial that internal systems can handle high traffic volume without downtime or performance degradation.
In another life, I taught the Book of Genesis to high school students, including The Tower of Babel excerpt. It struck me ironic that God’s wrath strikes down the tower, cofounds the universal language and scatters humans around the globe to teach King Nimrod a lesson in hubris; meanwhile, the boys in my class were texting their girlfriends across the country and playing video games with friends in Europe and Asia.
Many in the community have been asking us to develop a new Kafka to S3 connector for some time. So we’re pleased to announce it's now available. It’s been designed to deliver a number of benefits over existing S3 connectors. Like our other Stream Reactors, the connector extends the standard connect config adding a parameter for a SQL command (Lenses Kafka Connect Query Language or “KCQL”). This defines how to map data from the source (in this case Kafka) to the target (S3).
