Systems | Development | Analytics | API | Testing

Agentic Engineering and the Agentic Software Factory for Real-Time Data Products

Software workloads that process large volumes of real-time data are becoming common. Decades working in this domain has taught me that building and operating reliable and maintainable real-time data products requires permissive access to the context of the environment. This article explains how to approach agentic engineering and apply it when building real-time data products inside an agentic software factory.

Introducing K2K 2.0: Enterprise Kafka DR - without vendor lock-in

Summary Kafka has become the backbone of the real-time enterprise. The streams it carries are not only time, but business critical: a fraud event isn't processed, a sales order not fulfilled, a trade not settled. Yet we heard a recurring theme from Kafka teams: their business is running critical streaming applications without proper Kafka resiliency.

Consumer offset mapping in Kafka-to-Kafka replication

If you replicate data between two distinct Kafka clusters, you already know the payloads can match while the offsets might not. This post is about how K2K 2.0 now also keeps consumer committed offsets in sync between the source and the target so consumer groups can fail over in a Disaster Recovery situation, avoiding large re-reading of data or row skips. This offers the community more choice for DR than just MirrorMaker2 and Confluent solutions have until now.

Preventing Kafka war rooms: with self-service troubleshooting

Most Kafka war rooms don't start at 2am in production. They start weeks earlier, when an engineer couldn’t explore a topic, see partition distribution, a quota or a config and shipped hoping for the best. That's the Platform Engineering waiting room: developers building apps waiting in a queue of tickets, with no self-service visibility into Kafka. That release triggers a P1 later.

Streaming highlights from Databricks Data + AI Summit

Join Tun Shwe and Jeremy Frenay as they stream live from the floor of the Databricks Data + AI Summit! They’ll break down the biggest announcements, key takeaways, and cutting-edge trends shaping the intersection of AI and data streaming. Register to get an insider look at the future of data AI streaming.

Turn Your Agents Into Kafka Experts with Skills. Live from Current London

Most AI agents are generalists that struggle with the nuances of streaming data and Kafka infrastructure, often leading you down a rabbit hole with many tokens spent. In this session, live from Current London, we’ll show you how to close the context gap with Skills for Kafka: structured files that are playbooks and level-ups for agents to handle complex multi-step tasks like reviewing schema changes, critiquing DLQ policies and auditing topics for best practice. We will walk through how these skills work, demonstrate the different ways to use them (including with MCP servers) in Cursor and Claude.

Lenses VS Code Plugin - multi-Kafka DevX & governance within the IDE

Engineering is in the middle of an almighty shift. Thanks to AI code-generation solutions, Engineers are being asked to take on a different and wider set of responsibilities in order to be more productive. It’s what’s increasingly being coined as Agentic Engineering - using AI agents to accelerate engineering & operations work while maintaining human oversight, quality and rigour.

Lenses MCP Server with OAuth 2.1

You can now drive Lenses from Cursor, VS Code, IBM Bob or Claude Code without running any extra piece of infrastructure locally. Lenses MCP offers secure tools across topics, schemas, Kafka Connect, SQL processors, consumer groups, datasets and pod logs: everything an engineer would normally click through in the Lenses UI, now reachable from any MCP-compatible client over HTTP.

Introducing Kafka Skills for AI Engineering Agents

If you've written a line of code in the last 18 months, you already know this. Tools like Claude, Codex, Bob, Kiro and Cursor have made agentic software engineering the default. Most developers today are writing prompts as much as they are writing code. That shift changes what ‘developer experience’ means. Clean UIs, useful tools and good docs are still the foundation but the focus has shifted to ensuring a coding agent actually knows what it is doing, in the hands of a developer.