blog bg left
Back to Blog

WhyLabs Private Beta: Real-time, No-code, Cloud Storage Data Profiling

Developers are busy enough as it is. For some, devoting development time to integrating DevOps tools and monitoring solutions can be enough of a deterrent that it ends up being perpetually kicked to the next sprint. We've talked to enough users at this point to understand that having no-code integration options is mandatory for certain use cases.

We've started working on a no-code integration option for WhyLabs that lets certain users bypass the need to integrate whylogs into their data pipeline. We already have a container based solution that enables you to black box the integration, but that involves hosting your own container. Some users find that more burdensome than just writing some code.

The first iteration of this solution is aimed at users who already have most of their data in cloud storage (starting with S3/AWS), don't want to invest any development time into profiling their data with whylogs, and don't mind permitting WhyLabs to ingest from their S3 bucket (without permanently storing any data). In the future, we'll support every cloud storage solution and give parts of this solution on-prem capabilities, so it can be deployed using tools like Terraform.

How it works

We're starting simple, essentially hosting our existing whylogs container ourselves and hooking it up to user S3 accounts via S3 events. The primary use case is real-time data monitoring; we won't be addressing historical data imports just yet.

The onboarding experience will consist of a few updates to your bucket:

  • Adding a policy to an S3 bucket that lets our AWS account download data.
  • Adding a special tag to your S3 bucket that we look out for so we know you own it.
  • Enabling S3 events directed at our SNS topic.

And then a few steps in our UI:

  • Specifying which column in your dataset represents time, or deciding to use the upload time instead.
  • Specifying the types of your columns if the inferred whylogs types aren't what you want.
  • Specifying the WhyLabs org and model id that you want to import the data into

At that point, you'll be able to upload csv and parquet files to your bucket and we'll automatically download and profile data. The image below is the rough sequence of events.

  1. Your application will periodically upload data to S3.
  2. AWS will trigger an event with a reference to that file in S3 to our SNS topic.
  3. SNS invokes our Lambda function.
  4. We validate the bucket tags and read any associated metadata in our back end (which org/model this belongs to, etc.).
  5. Pipe it over to Kinesis.
  6. Pipe it over into our whylogs container.
  7. Download the data from S3, profile it with whylogs, and delete it from the container.
  8. Upload the profile to WhyLabs for monitoring.

One of WhyLab's biggest strengths is that we don't need (or want) the raw data, we only need the profiles we generate with whylogs from the data. The only reason we're accessing data here is because we have a strong signal that some users would rather we "just do it" if it means they don't have to do any dev work. That said, we still have no interest in retaining the raw data and we drop it as soon as we profile it. We'll have better self hosting/on-prem options in the future for this system for people who can't share any data and who don't want to integrate whylogs into their architecture manually.

Wrapping up

We have several integration options depending on your needs as well. Integrating our open source whylogs library manually will always give the most flexibility, and our whylogs container is a happy middle ground between manual and no-code if you're willing to host it.

Early access and feedback

If you're interested in early access or have any feedback on the feature, reach out to us on Slack or email and mention this post, or fill out this google form. We'll be working on releasing this over the coming weeks and we'll follow up by adding support for Google Cloud and Azure eventually as well.

Other posts

Glassdoor Decreases Latency Overhead and Improves Data Monitoring with WhyLabs

The Glassdoor team describes their integration latency challenges and how they were able to decrease latency overhead and improve data monitoring with WhyLabs.

Understanding and Monitoring Embeddings in Amazon SageMaker with WhyLabs

WhyLabs and Amazon Web Services (AWS) explore the various ways embeddings are used, issues that can impact your ML models, how to identify those issues and set up monitors to prevent them in the future!

Data Drift Monitoring and Its Importance in MLOps

It's important to continuously monitor and manage ML models to ensure ML model performance. We explore the role of data drift management and why it's crucial in your MLOps pipeline.

Ensuring AI Success in Healthcare: The Vital Role of ML Monitoring

Discover how ML monitoring plays a crucial role in the Healthcare industry to ensure the reliability, compliance, and overall safety of AI-driven systems.

WhyLabs Recognized by CB Insights GenAI 50 among the Most Innovative Generative AI Startups

WhyLabs has been named on CB Insights’ first annual GenAI 50 list, named as one of the world’s top 50 most innovative companies developing generative AI applications and infrastructure across industries.

Hugging Face and LangKit: Your Solution for LLM Observability

See how easy it is to generate out-of-the-box text metrics for Hugging Face LLMs and monitor them in WhyLabs to identify how model performance and user interaction are changing over time.

7 Ways to Monitor Large Language Model Behavior

Discover seven ways to track and monitor Large Language Model behavior using metrics for ChatGPT’s responses for a fixed set of 200 prompts across 35 days.

Safeguarding and Monitoring Large Language Model (LLM) Applications

We explore the concept of observability and validation in the context of language models, and demonstrate how to effectively safeguard them using guardrails.

Robust & Responsible AI Newsletter - Issue #6

A quarterly roundup of the hottest LLM, ML and Data-Centric AI news, including industry highlights, what’s brewing at WhyLabs, and more.
pre footer decoration
pre footer decoration
pre footer decoration

Run AI With Certainty

Book a demo