Systems | Development | Analytics | API | Testing

The Last (and Longest) Mile of Apache Kafka Migrations: Client Migrations With KCP and Confluent Cloud Gateway

In a previous blog post in this series, we introduced Kafka Copy Paste (KCP), an open source CLI tool that automates the discovery, provisioning, and data migration steps of moving your Apache Kafka environment to Confluent Cloud. We walked through how KCP and Cluster Linking work together to reduce a process that traditionally took weeks to a matter of hours. At the end of that post, we hinted that automated client migration was coming soon. That day has arrived.

Introducing Confluent Platform 8.2: Queues for Apache Kafka, Flink SQL, and More

Today, we're excited to announce the release of Confluent Platform 8.2, which builds on Apache Kafka 4.2! This release extends and simplifies what you can do with Apache Kafka and Apache Flink, whether that’s handling task queues natively with Apache Kafka 4.2, processing streams easily with Flink SQL, or managing cluster migration, upgrades, or disaster recovery without the usual operational pain. The release highlights are below, and additional details about the features are in the release notes.

From 1 to 1 Million: How Agent Taskflow Built a Scalable AI Future with AWS and Confluent

In the explosive new landscape of generative AI (GenAI), the difference between a proof of concept and a production-grade system is scale. For artificial intelligence (AI) infrastructure startup Agent Taskflow Inc. (ATF), this wasn't just a future goal; it was a foundational requirement. Founded in 2023, ATF provides a platform for rapid AI agent bootstrapping, multi-agent orchestration, and comprehensive observability.

The Rise of the Open Security Lake: Why CISOs Are Betting on Open Table Formats

As we head into the RSA Conference this year, the conversation on the show floor is going to be different. Yes, artificial intelligence (AI) will be everywhere. But if you listen closely to the C-suite discussions happening behind closed doors, the real buzz isn't just about the newest detection algorithm. It’s about data gravity and the unprecedented data explosion driven by AI-fueled bad actors.

Why ELT Can't Keep Up in the Era of High-Scale Data Engineering

While winning in artificial intelligence (AI) is critical to the future of business, old-school analytics—visualizations, dashboards, and infrequent reports—are still core to an organization's data needs. Behind the scenes, this analytics ecosystem remains heavily hydrated by batch-based ELT data integration. For a long time, this made perfect sense, as data sources were fewer, data volumes were manageable, and analytics consumers were limited.

How to Implement Your First ML Function in Streaming

The most effective way to adopt streaming machine learning (ML) is not by rebuilding your entire platform but by adding a single, high-value inference step to your existing data flow. This incremental approach allows you to transition from batch-based processing to real-time decision-making without the risk of a "big bang" migration, ensuring that your microservices architecture remains agile and responsive. What Is Streaming ML? ML in streaming is the practice of.

From Dumb Pipes to a Smart Data Plane: Introducing Schema IDs in Apache Kafka Headers

Apache Kafka powers massive, mission-critical data streams at enterprises worldwide. But in many organizations, those streams still behave like dumb pipes: raw JSON or bytes flowing between services, limited governance, weak contracts between teams, and data that’s hard to reuse for analytics or artificial intelligence (AI).

Confluent Cloud for Government Achieves FedRAMP Moderate: Mission-Ready Data Streaming for Federal Agencies

Federal agencies must perform a high-stakes balancing act: Modernize legacy systems, break down data silos, and deliver real-time citizen services—all while operating under strict security and compliance requirements with constrained budgets and staff. Today, we're announcing that Confluent Cloud for Government (CCG) is now available on the FedRAMP Marketplace, with FedRAMP Moderate authorization achieved through the competitive FedRAMP 20x Pilot program.

Sustainable Streaming Architectures: A GreenOps Guide to Efficient, Low-Carbon Data Systems

Data infrastructure growth has a direct, measurable relationship with energy consumption. As organizations ingest more events, retain more data, and deploy more always-on services, infrastructure energy use increases—often faster than business value. For streaming systems, this effect can be amplified by long-running clusters, peak-based sizing, and duplicated pipelines. Sustainability in this context is not about environmental reporting or corporate commitments.