Cloudera DataFlow Designer: Kafka to Iceberg in Cloudera Data Warehouse
Learn how to use the new Cloudera DataFlow Designer to build a streaming data pipeline that ingests events from Kafka into Iceberg tables.
Systems | Development | Analytics | API | Testing
Modern Software Development For Today's Systems & Applications
From Kinesis to Kafka
At the beginning of 2021, a brand new data team was assembled to build a real-time data platform for Kong’s SaaS platform, Konnect. Our mission is to provide top-notch real-time API analytics features for Konnect customers. V1 platform architecture The initial backend consisted of three main components: an ingestion service to handle incoming telemetry requests, an intermediate message queue, and a real-time OLAP data storage.
Shorten the path to insights with Aiven for Apache Kafka and Google BigQuery
Unify data from diverse sources and formats using Aiven Kafka's open source streaming. Analyze with BigQuery for swift, accurate insights.
Qlik and Confluent Together Deliver Real-Time Data Streaming To Kafka
Qlik Data Integration enable you to automatically produce real-time transaction streams into Kafka. Take advantage of modern analytics and microservices, enabling streaming analytics as well as streaming ingestion into data lakes and data warehouse platforms. And unlock the potential of data from legacy systems with microservices environment integrations.
Using Kafka Connect Securely in the Cloudera Data Platform
In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform (CDP), allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role-based access control and sensitive information handling. If you are a developer moving data in or out of Kafka, an administrator, or a security expert this post is for you. But before I introduce the nitty-gritty first let’s start with the basics.
Scaling Kafka Brokers in Cloudera Data Hub
This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes. This creates flexibility to handle real-time data feed volumes as they fluctuate.
How to extend Kafka pipelines to users over the public internet
In the realms of event streaming, Kafka has become the number one choice for many organizations. It’s capable of handling and processing vast amounts of critical, time-sensitive event data. But Kafka isn’t designed for distributing data between internal systems and consumers on the public internet. Yet rich, live digital experiences are a must-have as far as users are concerned. They are increasingly becoming a key source of differentiation. In this webinar we will show you how you can harness the power of Kafka, to deliver engaging realtime application to users, using Ably.
Kafka best practices: Monitoring and optimizing the performance of Kafka applications
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Administrators, developers, and data engineers who use Kafka clusters struggle to understand what is happening in their Kafka implementations.
Ably extends your Kafka pipeline to end-users at the edge
This article explains how Ably complements Kafka to reach end-users on the public internet.
Event Streaming in Rails with Kafka
Do you need to process a lot of data in real time? Event streaming is a pattern that could help. David Sanchez walks us through how to do event streaming in Rails with Apache Kafka, the popular open-source event streaming platform.