Case study
"WhyLabs’ monitors are detecting issues, such as predicting wrong values, that we might have never known about. The downstream impact of catching these issues is that we’re able to operate our business more successfully"
Calvin Linford, Sr. Software Engineer, Airspace
Introduction
It’s not every day that we get to work with a company that literally saves lives by delivering human organs with precision and reliability. Airspace is on a mission to bring critical logistics into the 21st century and are creating the most trusted delivery network the world has ever seen. They solve the problems that plagued time-critical deliveries across the logistics lifecycle from quoting to tracking, serving customers across healthcare, aerospace and manufacturing. To do this, Airspace developed proprietary ML models to provide instant quotes, optimized routing, rapid driver dispatch, and to enable real-time communication on every shipment. Since the shipments are for extremely critical packages, there is no margin for error.
Problems and challenges
Airspace built a sophisticated deployment infrastructure which requires monitoring to ensure the performance of their proprietary models doesn’t degrade. With many business critical models in production, they explored building out a custom monitoring solution but decided that it would be hard to resource and a distraction from their core priorities.
As the performance of these proprietary models is pivotal for Airspace’s business, monitoring them became a top priority. For instance, one of these models predicts how frequently a shipment bid will be accepted. If this model estimates too low of a quote the shipping carrier might reject the offer when the actual quote comes back higher. If the estimate is too high, it can cause the carrier to walk away without even seeing the actual quote.
Issues such as feature drift, training-serving skew, and missing features were some of the biggest drivers of poor model performance. If data drift—or an upstream issue with the data pipeline—caused their models to produce erroneous predictions, it wouldn’t be identified until the downstream impact to the business was reported back to the ML team.
The team concluded that they needed an AI observability platform that will scale with them as their business grows. A platform that preserves their ML infrastructure flexibility while providing a turnkey solution that integrates with existing tools and alerting workflows. This is where WhyLabs comes in.
Why Airspace chose WhyLabs
WhyLabs provides a flexible monitoring solution compatible with the requirements of Airspace's sophisticated and unique deployment infrastructure. By using the whylogs open source package to log model data, Airspace was able to see the value in calculating statistical summaries to enable data quality validation and monitoring for their machine learning models. The integration path was simple, and the onboarding process was fast.
Another key aspect of the WhyLabs Platform is the simplified approach to monitor configuration and control. Having experimented with other tools, where configuring monitors for a single feature might take a day to manually set up, the WhyLabs zero configuration monitoring experience provided huge time savings. With WhyLabs, each input feature or model output has a suite of monitors that come configured out of the box. These include monitors for distribution drift and for common data quality metrics such as changes in number of missing values, unique value ratio, and changes in data type.
The experience of using the platform itself is intuitive and provides a number of task oriented dashboards for both observability and monitoring workflows. It’s easy to start using the platform and find it easy to add more models whenever needed.
If you, like Airspace, have models deployed in production and want to have visibility into their performance, try out WhyLabs! It’s free and easy to get started by signing up here.
Conclusion
The Airspace Machine Learning Team likes the scalable and cost effective way that WhyLabs provides AI observability and monitoring. Ultimately, with WhyLabs, they are able to operate models more efficiently, catch data issues proactively, and automate many manual steps required to keep models healthy. By reducing the time spent on ML operations, the Airspace team can focus on building more models to power their vision of building the most reliable delivery network in the world.
