Systems | Development | Analytics | API | Testing

Buy 10 sandwiches, get 1 free. Classic punch cards (and fishing for them in your wallet or occasionally misplacing one) have become a thing of the past, as today's digital landscape demands more innovative solutions. Today’s customer loyalty programs are increasingly sophisticated—evolving, proliferating, and diversifying across every industry from retail, travel, and hospitality to healthcare (e.g., a discount for paying within 30 days of a hospital visit).

More than two-thirds of companies are currently using Generative AI (GenAI) models, such as large language models (LLMs), which can understand and generate human-like text, images, video, music, and even code. However, the true power of these models lies in their ability to adapt to an enterprise’s unique context. By leveraging an organization’s proprietary data, GenAI models can produce highly relevant and customized outputs that align with the business’s specific needs and objectives.

Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. However, as applications and pipelines grow in complexity, understanding what’s happening beneath the surface becomes increasingly crucial. A lack of visibility hinders the development and maintenance of high-quality applications and pipelines, ultimately impacting customer experience.

Today’s data-driven world requires an agile approach. Modern data teams are constantly under pressure to deliver innovative solutions faster than ever before. Fragmented tooling across data engineering, application development and AI/ML development creates a significant bottleneck, hindering the speed of value delivery required to stay competitive. Disparate tools create a complex landscape for developers and data teams, hindering efficient pipeline development and deployment.

In part one of this series, we walked through how to use Streamlit, Apache Kafka, and Apache Flink to create a live data-driven user interface for a market data application to select a stock (e.g., SPY) and discussed the structure of the app at a high level. First, data with information on stock bid prices is moved via an Alpaca websocket, then, it’s produced to a Kafka topic in Confluent Cloud where it is also processed with Flink SQL.