Systems | Development | Analytics | API | Testing

Software workloads that process large volumes of real-time data are becoming common. Decades working in this domain has taught me that building and operating reliable and maintainable real-time data products requires permissive access to the context of the environment. This article explains how to approach agentic engineering and apply it when building real-time data products inside an agentic software factory.

Summary Kafka has become the backbone of the real-time enterprise. The streams it carries are not only time, but business critical: a fraud event isn't processed, a sales order not fulfilled, a trade not settled. Yet we heard a recurring theme from Kafka teams: their business is running critical streaming applications without proper Kafka resiliency.

If you replicate data between two distinct Kafka clusters, you already know the payloads can match while the offsets might not. This post is about how K2K 2.0 now also keeps consumer committed offsets in sync between the source and the target so consumer groups can fail over in a Disaster Recovery situation, avoiding large re-reading of data or row skips. This offers the community more choice for DR than just MirrorMaker2 and Confluent solutions have until now.

Engineering is in the middle of an almighty shift. Thanks to AI code-generation solutions, Engineers are being asked to take on a different and wider set of responsibilities in order to be more productive. It’s what’s increasingly being coined as Agentic Engineering - using AI agents to accelerate engineering & operations work while maintaining human oversight, quality and rigour.

You can now drive Lenses from Cursor, VS Code, IBM Bob or Claude Code without running any extra piece of infrastructure locally. Lenses MCP offers secure tools across topics, schemas, Kafka Connect, SQL processors, consumer groups, datasets and pod logs: everything an engineer would normally click through in the Lenses UI, now reachable from any MCP-compatible client over HTTP.

If you've written a line of code in the last 18 months, you already know this. Tools like Claude, Codex, Bob, Kiro and Cursor have made agentic software engineering the default. Most developers today are writing prompts as much as they are writing code. That shift changes what ‘developer experience’ means. Clean UIs, useful tools and good docs are still the foundation but the focus has shifted to ensuring a coding agent actually knows what it is doing, in the hands of a developer.

At Lenses, our goal has always been to help organizations get the most out of their streaming data. We started with visibility into the Apache Kafka, moving up to the part that drives value, the application layer and now the Agentic layer. Lenses 6 moved us into a multi-Kafka world, as increasing, our clients aren’t just running on one type of Kafka anymore, and as sovereign cloud becomes increasingly topical (no pun intended) this is only increasing.

Jonas Best & Patrick Polster Kafka migrations are one of the riskiest infrastructure projects a platform team can take on. Miss a dependency and a downstream app starts reprocessing events it already handled leading to breaking SLAs and eroding trust with application teams. Migrate without visibility and you risk a major production issue. The instinct is to reach for a replication tool and call it done. But replication is only one piece of the puzzle.

First in a 3-part series on self-service K2K replication. This post tackles how to give self-service access to deploy K2K without handing over the keys to your Kafka clusters. Lenses developed K2K (Kafka-to-Kafka) to solve two major problems: This includes making it as self-service as possible so developers can deploy without requiring a PhD in MirrorMaker2. One key design requirement: don’t force engineers to manage credentials to authenticate with Kafka.

In 2025, Model Context Protocol (MCP) captured the imagination of the AI engineering community. 2026 will be the year that everyone realises their prototypes need to run in production.