Systems | Development | Analytics | API | Testing

July 2020

Why our new Streaming SQL opens up your data platform

SQL has long been the universal language for working with data. In fact it’s more relevant today than it was 40 years ago. Many data technologies were born without it and inevitably ended up adopting it later on. Apache Kafka is one of these data technologies. At Lenses.io, we were the first in the market to develop a SQL layer for Kafka (yes, before KSQL) and integrate it in a few different areas of our product for different workloads.

Why SQL is your key to querying Kafka

If you’re an engineer exploring a streaming platform like Kafka, chances are you’ve spent some time trying to work out what’s going on with the data in there. But if you’re introducing Kafka to a team of data scientists or developers unfamiliar with its idiosyncrasies, you might have spent days, weeks, months trying to tack on self-service capabilities. We’ve been there.

Data dump to data catalog for Apache Kafka

From data stagnating in warehouses to a growing number of real-time applications, in this article we explain why we need a new class of Data Catalogs: this time for real-time data. The 2010s brought us organizations “doing big data”. Teams were encouraged to dump it into a data lake and leave it for others to harvest. But data lakes soon became data swamps.

Everything you want to know about Kafka monitoring

Apache Kafka is a popular and powerful component of modern data platforms. But it's complicated. Complicated to run, complex to manage and crucially - it's near impossible to drive Kafka adoption from the command line across your organization. So here's your how-to for seeing it through to production (... and possibly fame and fortune). We cover key principles for Kafka observability and monitoring.

It Takes Two to Kafka: AWS MSK + DataOps

I ordered a ride share recently from a beach; the app struggled to find a car, so I had to make several requests. After the fourth or fifth attempt, my bank alerted me to possible fraudulent activity on my credit card via SMS. Each time I ordered a ride, the service put a pending charge on my card. After I texted back that it was just me, the bank reactivated my account. Though the process was annoying, I felt reassured my bank could detect possible fraudulence that quickly.

Will your streaming data platform disturb your holiday?

Here's why you need to double down on your DataOps before your vacation. In the past few months, everything has changed at work (or at home). Q1 plans were scrapped. Reset buttons were smashed. It was all about cost-cutting and keeping lights on. Many app and data teams sought quick solutions and developed workarounds to data challenges and operational problems as people prepared to work from home for the foreseeable future. And now, it’s time for a holiday.

Future-proofing the supply chain with real-time data

Next to the healthcare system, COVID-19’s biggest infrastructural burden fell upon the supply chain. Fluctuations in supply and demand of essential goods, along with the oil surplus, led to a freight cliff in mid-April. Outbound tender volume and spot rates bottomed out, which highlighted a massive drop in demand. As the market rebounds, technological investments are key to the industry’s recovery.

The Data Modernization Imperative. The Emergence of Kafka & the Growing Role of DataOps - Session 1

Featuring: Matt Aslett, Research Director, Data, AI and Analytics, 451 Research Key trends in data modernization Matt Aslett, Research Director, 451 Research highlights data modernization strategies influenced by the Coronavirus Pandemic. In particular, he points to data-driven decision making and real-time technologies such as #ApacheKafka as enablers. Key initiatives, such as #DataOps are discussed, and the pivotable role it shares in driving data modernization.

The Data Modernization Imperative. The Emergence of Kafka & the Growing Role of DataOps - Session 2

Featuring: Matt Aslett, Research Director, Data, AI and Analytics, 451 Research Kafka adoption and the role of #DataOps Matt Aslett, Research Director, 451 Research discusses the emergence of #ApacheKafka, as a leading platform for real-time event-driven architectures. While adoption is strong, organizations are challenged by the complexity of #Kafka as their deployments expand and additional use cases are adopted.

The Data Modernization Imperative. The Emergence of Kafka & the Growing Role of DataOps - Session 3

Featuring: Matt Aslett, Research Director, Data, AI and Analytics, 451 Research The Kafka skills shortage Matt Aslett, Research Director, 451 Research shares the challenges created by the wide-scale adoption of #ApacheKafka. In many cases, Kafka is a victim of its own success. Listen in on strategies of how organizations building applications on #Kafka can scale their teams in spite of a skills shortage

Removing Kafka bottlenecks with DataOps

Our CTO, Andrew Stevenson was interviewed by Alan Shimel for TechStrong TV. The discussion was all about hot data topics such as DataOps, DevOps and practices to successfully enable Kafka. Andrew narrates his journey from civil engineering to starting Lenses.io with Antonios, our CEO, to help organizations succeed with real-time data.