Systems | Development | Analytics | API | Testing

Kafka cluster-to-cluster data replication is critical to many use cases: disaster recovery (DR), cloud or data center migration, testing applications with production-like data, and multi-region data distribution. Easy replication of data between clusters: The business case is clear, but the cost model is not. Some solutions appear free but impose heavy operational burden.

From weeks-long projects to self-service data distribution.

AWS MSK has become popular because it deploys Kafka easily and bills alongside other AWS services. Over the past few years, AWS announced Express Brokers, a new cluster type that offers unlimited storage and separates brokers from storage resources. This simplifies scaling and reduces the time needed to rebalance topics when adding or removing brokers.

You're three hours into debugging a stalled Kafka consumer. The lag is climbing. Customers are complaining. Your logging doesn't show anything useful, and changing the log level requires a deployment approval that won't come until tomorrow morning. Sound familiar? If you're operating Apache Kafka at scale, that sinking feeling when a consumer group stops progressing, and you're left playing detective with insufficient clues.

Devoxx Poland 2025 felt different. Not because of revolutionary new frameworks or another "this changes everything" moment, but because of what didn't happen. The conference had an unusual dose of pragmatism, skepticism, and – dare we say it – common sense. Maybe it's because developers are asking the right questions: "Does this solve a problem?" and "What happens when this inevitably breaks?" Here's what emerged from the sessions we watched, and the people we spoke to.

Let's talk about a problem that might sound simple but gets complex quickly: copying data from one Kafka cluster to another. As our Kafka usage grows, many of us find ourselves managing multiple clusters and needing to share data between them. Or worst still, sharing data to an external cluster. During a London meetup, we explored why this happens, what existing solutions offer, and why we decided to build our own Kafka replicator. Here's what we learned.

Current 2025 featured two days of engineers figuring out how streaming tech needs to evolve in an AI-driven world. Gone are the days when talks focused on basic Kafka setup. This year, everyone was tackling complex integrations, developer happiness, and practical AI implementation. Still, the event drew a range of people, with plenty of new faces stopping by the Lenses.io booth – clear evidence that Kafka and data streaming continue to attract newcomers.

Lenses 6.0 (code-named Panoptes after the Greek mythical character with many eyes) is generally available. It brings a multi-Kafka developer experience and complete autonomy to streaming data.

You didn’t become a developer to spend hours hunting down missing messages, or debugging consumer issues. Yet here we are. Valuable dev time evaporates as you wrestle with Apache Kafka, or wait for a central team to unblock you, when you should be finding, prepping, and shipping streaming data in minutes. Lenses Community Edition tackles these everyday frustrations.

New York City, US - February 12, 2025 - Lenses.io, a data streaming innovation leader whose software helps developers power the world’s largest businesses, today announces the development of an enterprise grade and vendor-agnostic Kafka-to-Kafka replicator. It will enable organizations to share streaming data across different domains, in order to keep up with real-time data demands as AI adoption grows.