blog bg left
Back to Blog

A Solution for Monitoring Image Data

As machine learning ecosystems become increasingly complex and data volumes grow at a dizzying rate, maintaining observability of your data and model health is more critical than ever. Data drift, concept drift, and data quality degradation can cause models to fail. In the worst cases, these are silent failures and can go undetected for months.

When working with tabular datasets, data quality can be monitored by capturing telemetry around missing values, cardinality, and data types for each of a dataset’s features. Data drift can be monitored by capturing descriptive statistics and approximate distributions of your data and computing statistical distances such as the Hellinger distance or KL Divergence. However, it’s not obvious how a similar approach can be applied to image data. As we will see, monitoring unstructured data such as images can be achieved by capturing telemetry which is structured and therefore, compatible with common statistical approaches.

Before moving forward, we must consider the types of problems that threaten computer vision systems. Once we’ve established this much, we can design a solution around these challenges.

Physical Factors

There are a variety of physical factors which can impact the consistency and quality of image data such as…

  • Device used
  • Device Settings
  • Changes in environment
  • Changes in the object(s) being detected

Hardware is an obvious example. Different devices (or device settings) can impact the size and resolution of images, along with many other properties. Suppose a healthcare company upgrades to a medical imaging device with a higher resolution. While these images may make diagnoses easier for human doctors, computer vision systems trained on images generated by the original device may actually perform worse.

Physical changes in the environment pose another challenge. For example, consider a computer vision system responsible for performing inspections of products on an assembly line. If the factory begins using a different type of light bulb, this could impact the images in a way that the machine learning model was not prepared to handle. While this is not likely to throw a hard error, the model’s performance can plummet, unbeknownst to those responsible for the model.

Aside from hardware and lighting, there are a countless number of potential challenges that can present themselves directly in the scene being imaged. There can be changes in the image background, the size or number of target objects (object detection), or target objects which are significantly different from anything encountered in the training set.

Images often have inconsistent lighting, backgrounds, image quality, object sizes, object counts, etc.

Data Pipeline Factors

Even if we can ensure that the raw images are consistently representative of our training data, these images often travel through a complex pipeline, which introduces many possible points of failure such as…

  • Swapped color channels
  • Inconsistent color spaces
  • Inconsistent scaling

Different image processing frameworks may read in color channels in a different order, causing a channel swap when migrating to a new tool. A newly introduced bug may result in inconsistencies in whether grayscale or colored images are passed to our model or in the way that pixel values are scaled. Suppose we train a model that expects pixel values ranging from 0-255. If we begin scaling these values from 0.0-1.0, this could result in a model effectively being fed black, featureless images.

Data pipeline issues may cause inconsistencies in pixel value scaling, channel ordering, or the number of channels

The Solution

In order to effectively monitor the issues described above, we need to compute metrics that are sensitive to these events. For example, calculating the mean pixel value of an image can serve as a measure of image brightness and can be leveraged to monitor for changes in image lighting. Most image processing tools can compute hue and saturation which provide information about the color palette of an image. These quantities can be used to monitor for things like changes in image backgrounds or issues such as the swapping of color channels in an image. Monitoring image height can be done trivially by capturing the shape of the tensor representing the raw image data. The number of channels in this tensor can be used to infer the colorspace of an image (RGB, CMYK, Grayscale).

In many cases, valuable metadata is available in the image file in the form of Exif data. Exif data often includes information such as the device make and model, the camera settings while taking the photo, as well as geolocation, date, and time associated with the image. Consider a model trained to identify plant species from images. Capturing geolocation from image exif data can help to inform ML engineers whether new images are expected to contain plant species that weren’t included in the training dataset.

Example of camera settings extracted from Exif data

Monitoring at Scale

Now that we know what kind of telemetry we wish to capture for images, how can we turn this into a full-fledged monitoring solution? The team at WhyLabs has developed an open source data logging library, whylogs, which was designed to capture valuable telemetry in an efficient and customizable way for any dataset. whylogs is designed with customizability as a priority, enabling users to integrate whylogs with any data pipeline, whether you’re working with image data, tabular data, text, or something else.

Furthermore, users can leverage powerful anomaly detection, informative visualizations, and automated notifications by uploading profiles to the WhyLabs AI Observatory for an end-to-end monitoring solution. Your first model is free!

A Solution for Monitoring Image Data” was originally published by Open Data Science.

Other posts

Re-imagine Data Monitoring with whylogs and Apache Spark

An overview of how the whylogs integration with Apache Spark achieves large scale data profiling, and how users can apply this integration into existing data and ML pipelines.

ML Monitoring in Under 5 Minutes

A quick guide to using whylogs and WhyLabs to monitor common issues with your ML models to surface data drift, concept drift, data quality, and performance issues.

AIShield and WhyLabs: Threat Detection and Monitoring for AI

The seamless integration of AIShield’s security insights on WhyLabs AI observability platform delivers comprehensive insights into ML workloads and brings security hardening to AI-powered enterprises.

Large Scale Data Profiling with whylogs and Fugue on Spark, Ray or Dask

Profiling large-scale data for use cases such as anomaly detection, drift detection, and data validation with Fugue on Spark, Ray or Dask.

Monitoring Image Data with whylogs v1

When operating computer vision systems, data quality and data drift issues always pose the risk of model performance degradation. Whylabs provides a simple yet highly customizable solution for maintaining observability into data to detect issues and take action sooner.

WhyLabs Private Beta: Real-time, No-code, Cloud Storage Data Profiling

We’re excited to announce our Private Beta release for a no-code integration option for WhyLabs, allowing users to bypass the need to integrate whylogs into their data pipeline.

Data and ML Monitoring is Easier with whylogs v1.1

The release of whylogs v1.1 brings many features to the whylogs data logging API, making it even easier to monitor your data and ML models!

Model Monitoring for Financial Fraud Classification

Model monitoring is helping the financial services industry avoid huge losses caused by performance degradation in their fraud transaction models.

Robust & Responsible AI Newsletter - Issue #3

Every quarter we send out a roundup of the hottest MLOps and Data-Centric AI news including industry highlights, what’s brewing at WhyLabs, and more.
pre footer decoration
pre footer decoration
pre footer decoration

Run AI With Certainty

Book a demo