Systems | Development | Analytics | API | Testing

A year in at Confluent, Product Manager Surabhi Singh has learned a lot about data streaming—and even more about herself. In this fast-paced environment, Surabhi is highly motivated and committed to her work strategically planning, coordinating, and delivering product improvements for customers whose business operations depend on Confluent Platform.

The headless data architecture is the formalization of a data access layer at the center of your organization. Encompassing both streams and tables, it provides consistent data access for both operational and analytical use cases. Streams provide low-latency capabilities to enable timely reactions to events, while tables provide higher-latency but extremely batch-efficient querying capabilities. You simply choose the most relevant processing head for your requirements and plug it into the data.

The headless data architecture is an organic emergence of the separation of data storage, management, optimization, and access from the services that write, process, and query it. With this architecture, you can manage your data from a single logical location, including permissions, schema evolution, and table optimizations. And, to top it off, it makes regulatory compliance a lot simpler, because your data resides in one place, instead of being copied around to every processing engine that needs it.

Welcome back to the third installment of our blog series where we’re diving into the beautiful black box that is Apache Kafka to better understand how we interact with the cluster through producer and consumer clients. Earlier in the series, we took a look at the Kafka producer to see how the client works before following a produce request as it’s processed by the cluster.

Throughout my career in enterprise technology, I've witnessed numerous transformations play out across the Asia-Pacific (APAC) region. But the shift we're seeing now with data streaming is truly unprecedented. What was once a supportive technology is rapidly becoming the very foundation of modern business in our region.

We are now in the final chapter of Apache Kafka’s multi-year journey to remove Apache ZooKeeper and fully transition to self-managed metadata in KRaft. Many Kafka users and customers are beginning to migrate to KRaft and are eager to understand its performance characteristics in production environments.

Alright, I’m back. Time for part 2. In the first part, I covered how we handle bad data in batch processing. In particular, cutting out the bad data, replacing it, and running it again. But this strategy doesn’t work for immutable event streams as they are, well, immutable. You can’t cut out and replace bad data like you would in batch processed data sets.

Imagine getting into your car to head to work on a hot day. Your car already knows and sets the temperature, the ambient lighting, and the music you prefer. Not only that, it optimizes your route, and with Level 3 autonomy, it can even drive you there. But what does the automotive industry have to do on the backend in order to achieve this kind of personalization?