See how to build a streaming data pipeline to a cloud data warehouse for real-time analytics. Leverage Confluent Data Streaming Platform to.

How to manage consumer offsets in #apachekafka.

The transactional outbox pattern leverages database transactions to update a microservice's state and an outbox table. Events in the outbox will be sent to an external messaging platform such as Apache Kafka. This technique is used to overcome the dual-write problem which occurs when you have to write data to two separate systems such as a database and Apache Kafka. The database transactions can be used to ensure atomic writes between the two tables. From there, a separate process can consume the outbox and update the external system as required.

Dive into streaming data quality with Gilles Philippart! Learn to keep bad data out with Confluent's Data Quality Rules.

As you scale up the number of teams and apps on #apachekafka , using schemas managed in a Schema Registry will save you endless problems. In this #Kafka Live Stream, learn the fundamentals of schemas for your applications and how you can best manage them.

The dual write problem occurs when you try to write to two separate systems and need them to be atomic. If one write fails, and the other succeeds, you can end up with inconsistent state. This is an easy trap to fall into, and it can be difficult to avoid. We'll explore what causes the dual-write problem and explore both valid and invalid solutions to it.

How to monitor your #kafka connectors, and automatically restart them when they fail.

Building scalable and resilient microservices requires an approach that eliminates the need to treat them as special. They should be treated as easily replaceable building blocks. This means eliminating bottlenecks and single points of failure but it can also mean changing from a pull-based approach to a push-based approach. CHAPTERS.