Systems | Development | Analytics | API | Testing

At Confluent, we've been talking to customers about Streaming Agents—intelligent systems that work in the background responding to system events instead of to human chat inputs. The key question has been: What triggers these events? The signals have to be of high quality for an agent to act on since feeding raw events to an agent can be wasteful and overwhelming. A powerful pattern has emerged: agentic investigation and remediation of anomalies.

Historically, only a few AI experts within an organization could develop insights using machine learning (ML) and predictive analytics. Yet in this new wave of AI, democratizing ML to more data teams is crucial—and for Snowflake SQL users, it’s now a reality.

Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. Understanding memory management reduces the possibility of wasting your application's resources and the unexpected effects on performance. According to Sergey Kibish, Anomalies can be illustrated in a simple two-dimensional space.

“Data-driven” is the latest buzzword in organizations in which data-based decision making is directly connected to business success. According to Gartner’s Hype Cycle, more than 77% of the C-suite now say data science is critical to their organization meeting strategic objectives. For top organizations looking to adopt a data-driven culture to stay competitive, what does that mean?

Success in today’s high-velocity business environments means having the correct information to make the right decisions at the right time. As marketplaces grow more competitive and customer expectations continually rise, the “right time” is often real-time. Every transaction generates a plethora of data. Anomalies within your company’s data set can represent opportunities and threats to the business.

The global pandemic has changed B2C markets in many ways. In the U.S. market alone in 2020, consumers spent more than $860 billion with online retailers, driving up sales by 44% over the previous year.eCommerce sales are likely to remain high long after the pandemic subsides, as people have grown accustomed to the convenience of ordering online and having their goods – even groceries – delivered to their door.

When it comes to anomaly detection, one of the key challenges that many organizations face is that it can be difficult to know how to define what an anomaly is. How do you define and anticipate unusual network intrusions, manufacturing defects, or insurance fraud? If you have labeled data with known anomalies, then you can choose from a variety of supervised machine learning model types that are already supported in BigQuery ML.

Telecom companies monitor their network using a variety of monitoring tools. There are separate fault management and performance management platforms for different areas of the network (core, RAN, etc.), and infrastructure is monitored separately. Although these solutions monitor network functions and logic – something that would seem to make sense — in practice this strategy fails to produce accurate and effective monitoring or reduce time to detection of service experience issues.