Working with Apache Kafka and real-time applications comes with challenges. Visibility into the deployed applications and their dependency on what we call the “data fabric” is one of them (For the sake of this blog, it means Kafka and all its state and configuration). If you’ve built a multi-tenant real-time data platform with Kafka, where teams are deploying applications outside your jurisdiction, this is where the pain is particularly acute. It goes something like this.

There is a great number of logging plugins for Kong, which might be enough for your needs. However, they have certain limitations: Most of them only work on HTTP/HTTPS traffic. They make sense in an API gateway scenario, with a single Kong cluster proxying traffic between consumers and services. Each log line will generally correspond to a request which is “independent” from the rest.

I ordered a ride share recently from a beach; the app struggled to find a car, so I had to make several requests. After the fourth or fifth attempt, my bank alerted me to possible fraudulent activity on my credit card via SMS. Each time I ordered a ride, the service put a pending charge on my card. After I texted back that it was just me, the bank reactivated my account. Though the process was annoying, I felt reassured my bank could detect possible fraudulence that quickly.

Here's why you need to double down on your DataOps before your vacation. In the past few months, everything has changed at work (or at home). Q1 plans were scrapped. Reset buttons were smashed. It was all about cost-cutting and keeping lights on. Many app and data teams sought quick solutions and developed workarounds to data challenges and operational problems as people prepared to work from home for the foreseeable future. And now, it’s time for a holiday.

Our CTO, Andrew Stevenson was interviewed by Alan Shimel for TechStrong TV. The discussion was all about hot data topics such as DataOps, DevOps and practices to successfully enable Kafka. Andrew narrates his journey from civil engineering to starting Lenses.io with Antonios, our CEO, to help organizations succeed with real-time data.