Systems | Development | Analytics | API | Testing

Software workloads that process large volumes of real-time data are becoming common. Decades working in this domain has taught me that building and operating reliable and maintainable real-time data products requires permissive access to the context of the environment. This article explains how to approach agentic engineering and apply it when building real-time data products inside an agentic software factory.

If you replicate data between two distinct Kafka clusters, you already know the payloads can match while the offsets might not. This post is about how K2K 2.0 now also keeps consumer committed offsets in sync between the source and the target so consumer groups can fail over in a Disaster Recovery situation, avoiding large re-reading of data or row skips. This offers the community more choice for DR than just MirrorMaker2 and Confluent solutions have until now.

Summary Kafka has become the backbone of the real-time enterprise. The streams it carries are not only time, but business critical: a fraud event isn't processed, a sales order not fulfilled, a trade not settled. Yet we heard a recurring theme from Kafka teams: their business is running critical streaming applications without proper Kafka resiliency.

*You wouldn't expose a REST API without a gateway. What about Kafka?* You would never expose a REST API without an API gateway in front of it. Authentication, rate limiting, observability, access control — these aren't optional extras. They're mandatory, and we've spent the last decade building the API gateway pattern to solve exactly this problem. So here's the question worth sitting with: would you expose Kafka or any event stream without a gateway?