This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes. This creates flexibility to handle real-time data feed volumes as they fluctuate.

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Administrators, developers, and data engineers who use Kafka clusters struggle to understand what is happening in their Kafka implementations.

This article explains how Ably complements Kafka to reach end-users on the public internet.

Do you need to process a lot of data in real time? Event streaming is a pattern that could help. David Sanchez walks us through how to do event streaming in Rails with Apache Kafka, the popular open-source event streaming platform.

Our everyday digital experiences are in the midst of a revolution. Customers increasingly expect their online experiences to be interactive, immersive, and realtime by default. The need to satisfy user expectations is driving the exponential growth of event-driven architectures in organizations of all shapes and sizes. And by enabling users to have realtime experiences whenever and wherever they want, 24/7, mobile drives this change further and faster.

The Ably Kafka Connector has a raft of new enhancements, and is now available in a full general availability (GA) release. Developers now have the option to use pattern-based mapping rules to enable streaming of data from many Kafka topics to many Ably channels - ideal for chat solutions, live sports updates, live streaming, broadcasting notifications and alerts.

Here at Ably, we're excited to announce our participation and Silver sponsorship of the Kafka Summit 2022, taking place between 25-26 April. The Kafka Summit is the only dedicated technical conference for the Apache Kafka® Community, and it's a great opportunity for anyone building large-scale event-driven systems to learn and share ideas. It's also the perfect event for us to launch the general availability of the Ably Kafka Connector.

Organizations today have access to a wide stream of data. Data is generated from recommendation engines, page clicks, internet searches, product orders, and more. It is necessary to have an infrastructure that would enable you to stream your data as it gets generated and carry out analytics on the go. To aid this objective, incorporating a data pipeline for moving data from Apache Kafka to BigQuery is a step in the right direction.

Until recently, teams were building a small handful of Kafka streaming applications. They were usually associated with Big Data workloads (analytics, data science etc.), and data serialization would typically be in AVRO or JSON. Now a wider set of engineering teams are building entire software products with microservices decoupled through Kafka. Many teams have adopted Google Protobuf as their serialization, partly due to its use in gRPC.