Systems | Development | Analytics | API | Testing

February 2025

Building AI Agents and Copilots with Confluent, Airy, and Apache Flink

From automating routine tasks to providing real-time insights to inform complex decisions, AI agents and copilots are poised to become an integral part of enterprise operations. At least that’s true for the organizations that can figure out how to supply large language models (LLMs) with real-time, contextualized, and trustworthy data in a secure and scalable way.

A Distributed State of Mind: Event-Driven Multi-Agent Systems

This article was originally published on InfoWorld on Jan. 28, 2025 While large language models (LLMs) are useful, their real power emerges when they can act on insights, automating a broader range of problems. Reasoning agents have a long history in artificial intelligence (AI) research—they refer to a piece of software that can generalize what it has previously seen to apply in situations it hasn’t seen before.

Processing Without Pause: Continuous Stream Processing and Apache Flink | Life Is But A Stream

We’re diving even deeper into the fundamentals of data streaming to explore stream processing—what it is, the best tools and frameworks, and its real-world applications. Our guests, Anna McDonald, Distinguished Technical Voice of the Customer at Confluent, and Abhishek Walia, Staff Customer Success Technical Architect at Confluent, break down what stream processing is, how it differs from batch processing, and why tools like Flink are game changers.

Using Apache Flink for Model Inference: A Guide for Real-Time AI Applications

As real-time data processing becomes a cornerstone of modern applications, the ability to integrate machine learning model inference with Apache Flink offers developers a powerful tool for on-demand predictions in areas like fraud detection, customer personalization, predictive maintenance, and customer support. Flink enables developers to connect real-time data streams to external machine learning models through remote inference, where models are hosted on dedicated model servers and accessed via APIs.

How Real-Time Data Streaming with GenAI Accelerates Singapore's Smart Nation Vision

In today’s data-driven world, the ability to turn raw data into actionable insights is no longer a nice to have—it’s a necessity to power exemplary citizen service. Singapore’s Smart Nation initiative is built on the idea that data, when utilized effectively, can transform public services and improve lives.

Motivating Engineers to Solve Data Challenges with a Growth Mindset

With almost two years at Confluent under her belt, Suguna Ravanappa has taken impressive strides as a people manager. Her eight-person team of engineers in the Global Support organization helps customers tackle technical challenges in their data streaming environments. According to Suguna, leading this team and being part of Confluent’s unique company culture has helped her develop stronger skills as both a leader and a collaborator. Learn more about her experience.

Building High Throughput Apache Kafka Applications with Confluent and Provisioned Mode for AWS Lambda Event Source Mapping (ESM)

Confluent and AWS Lambda can be used to build scalable and real-time event-driven architectures (EDAs) that respond to specific business events. Confluent provides a streaming SaaS solution based on Apache Kafka and built on Kora: The Cloud-Native Engine for Apache Kafka, allowing you to focus on building event-driven applications without operating the underlying infrastructure.

Bridging the Data Divide: How Confluent and Databricks Are Unlocking Real-Time AI

We’re excited to announce an expanded partnership between Confluent and Databricks to dramatically simplify the integration between analytical and operational systems. This is particularly important as enterprises want to shorten the deployment time of AI and real-time data applications. This partnership enables those enterprises to spend less time fussing over siloed data and governance and more time creating value for their customers.

Build a Real-Time AI Chatbot for Social Media with Confluent & Databricks

Power AI-driven decision-making with Confluent’s Data Streaming Platform and Databricks’ Data Intelligence Platform. Watch this step-by-step demo to learn how to build a real-time social media AI chatbot that instantly engages with customer posts, like product reviews. New to Confluent? Experience unified Apache Kafka and Apache Flink with a free trial.

Automating Podcast Promotion with AI and Event-Driven Design

I host two podcasts, Software Engineering Daily and Software Huddle, and often appear as a guest on other shows. Promoting episodes—whether I’m hosting or featured—helps highlight the great conversations I have, but finding the time to craft a thoughtful LinkedIn post for each one is tough. Between hosting, work, and life, sitting down to craft a thoughtful LinkedIn post for every episode just doesn’t always happen.

Revolutionizing Failure Management in Apache Flink: Meet FLIP-304's Pluggable Failure Enrichers

FLIP-304 gives you the power to transform how failures are handled in Apache Flink. With custom failure enrichers, you can tag and classify errors, integrate enriched data with monitoring tools, and quickly identify root causes. Failures in distributed systems are inevitable, but managing them effectively makes all the difference. Enter FLIP-304: Pluggable Failure Enrichers, an upgrade that helps turn confusion into actionable insights.

Agentic AI: The Top 5 Challenges and How to Overcome Them

As generative AI continues its unprecedented surge in popularity, we are already seeing it evolve into the next generation of machine learning-driven technologies: agentic AI. With agentic AI, we are not just prompting models and receiving an answer in a simple one-step process. The AI is engaging in complex multi-step processes, often interacting with different systems to achieve a desired outcome.

Optimize SaaS Integration with Fully Managed HTTP Connectors V2 for Confluent Cloud

Integrations that fail to deliver consistent performance and suffer from long processing times inevitably fall short of the high standards set by today's dynamic business environment. Providing a seamless integration of Apache Kafka with various data sources and destinations is crucial in the fast-paced world of data streaming and real-time analytics.

Stream On: Unleashing Innovation with Data Streaming | Life is But a Stream

Real-time data streaming is shaking up everything we know about modern data systems. If you’re ready to dive in but unsure where to begin, no worries. That’s why we’re here. Our first episode breaks down the basics of data streaming—from what it is, to its pivotal role in processing and transferring data in a fast-paced digital environment. Your guide is Tim Berglund, VP of Developer Relations at Confluent, where he and his team work to make data streaming data and its emerging toolset accessible to all developers.

Unlock Cost Savings with Freight Clusters-Now in General Availability

Last year, we introduced Freight clusters, a new, cost-effective Confluent Cloud cluster type—purpose-built for high-throughput, latency-insensitive workloads—such as observability data, batch pipelines, and AI/ML data ingestion. Since then we've been working with early access customers to take their workloads to production, and in doing so, have helped them achieve 90% lower infrastructure costs, while maintaining the reliability we all know and expect from Confluent Cloud.