Case study
"If we're waking up engineers at 3 am, we need to be confident that we're not reporting on false positives."
IT Operations Manager, Fortune 500 Retail
Introduction
A Fortune 500 Retail company with hundreds of stores and an ever-growing online customer base, leverages WhyLabs for constant visibility into the most critical key performance indicators (KPIs) of the business. While their KPIs provided insight into how the company is performing regarding sales goals, monitoring data in real-time for actionable insight was equally critical.
The Challenge: Unique data patterns and many sources
The company's Executive team had systems in place to monitor sales KPIs, but the team had limited visibility into the driving factors influencing them. Their computer vision systems counted the number of customers inside each of their stores, while other systems reported on things like the number of users connected to the company's wifi router, the number of transactions within a given store, and other data points for online purchases and the fulfillment process. Because data was collected from multiple sources, it was often too late to take action when an anomaly was detected.
Their goal was to identify driving factors for anomalous behavior and data quality issues. Even with existing systems in place, anomalies were treated as critical incidents leading to an excess of false positives. The data followed intricate patterns due to high seasonality since shopping patterns are strongly influenced by the day of the week. However, holidays and special sales disrupted these seasonal patterns, resulting in many more false positives.
With hundreds of stores independently reporting data from multiple sources to a centralized pipeline, monitoring for data quality issues was also a high priority. Missing data from certain stores or issues in the centralized data pipeline would impact their overall dataset. The company needed a monitoring solution tailored to the unique nature of their data and to avoid alerting engineers with false alarms.
The solution: Highly customizable monitoring with WhyLabs
It became clear that the flexibility of WhyLabs monitors were a great fit for this company's unique data patterns. Our data science team worked closely with their engineers to implement a solution that successfully accounted for the seasonal nature of the company's data while also accounting for predictable deviations during holidays and sales periods. We worked together to arrive at the right monitor sensitivity, alleviating the pain points around excessive false positives and continuous improvement of their anomaly detection algorithms.
Monitors were integrated into their pipeline to be notified immediately if any data issues were detected or if the expected profiles were not uploaded to the WhyLabs platform. Segments were defined so that subsets of data can be monitored independently for different delivery methods or phases of the fulfillment process. With WhyLabs notifications, they can ensure that PagerDuty incidents are automatically generated and assigned to the relevant engineers. To this day, they continue to customize their WhyLabs monitors to further reduce false positives and report on incidents earlier.
Outcome: A trustworthy solution to drive critical KPIs
As a result of the highly customizable nature of WhyLabs' monitors, their team now has a reliable solution for monitoring the factors that drive their most critical KPIs to take action as soon as incidents occur. With WhyLabs in place, they've increased their likelihood of meeting sales goals, and engineers can sleep better at night knowing they won't be woken up by false alarms!