Jul 20, 2023
7 Ways to Monitor Large Language Model Behavior
- LLMs
- Generative AI
- LangKit
- LLM Security
Jul 20, 2023
Seven ways to track the evolution of LLMs with LangKit and WhyLabs
In the ever-evolving landscape of AI, Large Language Models (LLMs) have revolutionized Natural Language Processing. With their remarkable ability to generate coherent and contextually relevant human-like text, LLMs have gained immense importance and adoption, transforming the way we interact with technology.
ChatGPT is perhaps the most well-known of these models, boasting 57 million monthly active users within the first month of availability [1]. Along with its impressive capabilities across multiple scenarios, the model also comes with big challenges, such as the tendency to hallucinate and generate biased or harmful content [2,3]. Another challenging area is observability - with the rapid collection of user feedback, ChatGPT is being continuously retrained and improved through Reinforcement Learning from Human Feedback (RLHF) [4], making its evaluation a moving target. It is well-known that overall improvements from RLHF can lead to performance regressions on specific tasks [5]. How can we ensure that the model behaves as expected and maintains acceptable performance within the tasks that are relevant to our application? This dynamic nature of LLMs makes it crucial to develop innovative approaches for behavioral monitoring that can keep pace with their rapid progress. One potential avenue for addressing these concerns is through the development of custom GPTs tailored to specific use cases and requirements.
In this blog, we will discuss seven groups of metrics you can use to keep track of LLM’s behaviors. We will calculate these metrics for ChatGPT’s responses for a fixed set of 200 prompts across 35 days and track how ChatGPT’s behavior evolves within the period. Our focus task will be long-form question answering, and we will use LangKit and WhyLabs to calculate, track and monitor the model’s behavior across time.
You can check the resulting dashboard for this project in WhyLabs (no sign up required) and run the complete example yourself by running this Colab Notebook.
Agenda
- The Task - Explain Like I’m 5
- Popular LLM Metrics
- ROUGE
- Bias
- Text Quality
- Semantic Similarity
- Regex Patterns
- Refusals
- Toxicity and Sentiment - Monitoring Across Time
- So, Has Behavior Changed?
The task - explain like I’m 5
For this example, let’s use the Explain Like I’m Five (ELI5) dataset [6], a question-answering dataset built from the Reddit forum “Explain Like I’m Five.” The questions are open-ended - questions that require a longer response and cannot be answered with a “yes” or “no” - and the answers should be simple enough so that a five-year-old would understand.
In the work presented in ChatLog: Recording and Analyzing ChatGPT Across Time, 1000 questions were sampled from this dataset and repeatedly sent to ChatGPT every day from March 5 to April 9, 2023, which is available in ChatLog’s Repository. We’ll use this dataset by sampling 200 out of the original 1000 questions, along with ChatGPT’s answers and human reference answers, for each day of the given period. That way, we’ll end up with 35 daily dataframes, where each dataframe has 200 rows with the following columns:
Popular LLM metrics
It can be a daunting task to define a set of metrics to properly evaluate a model with such a wide range of capabilities as ChatGPT. In this example, we’ll cover some examples of metrics that are relatively general and could be useful for a range of applications, such as text quality, sentiment analysis, toxicity, and text semantic similarity, and others that are particular for certain tasks likequestion answering and summarization, like the ROUGE group of metrics.
There are a multitude of other metrics and approaches that might be more relevant, depending on the particular application you are interested in. If you’re looking for more examples of what to monitor, here are three papers that served as an inspiration for the writing of this blog: Holistic Evaluation of Language Models, ChatLog: Recording and Analyzing ChatGPT Across Time, and Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.
Now, let’s talk about the metrics we’re monitoring in this example. Most of the metrics will be calculated with the help of external libraries, such as ROUGE, textstat, and huggingface models, and most of them are encapsulated in the LangKit library, which is an open-source text metrics toolkit for monitoring language models. In the end, we want to group all the calculated metrics in a whylogs profile, which is a statistical summary of the original data. We will then send the daily profiles to the WhyLabs observability platform, where we can monitor them over time.
In the following table, we summarize the groups of metrics we will cover at the following sections:
ROUGE
Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics commonly used in natural language processing and computational linguistics to evaluate the quality of automatic summaries. The ROUGE metrics are designed to compare an automatically generated summary with one or more reference summaries.
The task at hand is a question-answering problem rather than a summarization task, but we do have human answers as a reference, so we will use the ROUGE metrics to measure the similarity between the ChatGPT response and each of the three reference answers. We will use the ROUGE python library to augment our dataframe with two different metrics: ROUGE-L, which takes into account the longest sequence overlap between the answers, and ROUGE-2, which takes into account the overlap of bigrams between the answers. For each generated answer, the final scores will be defined according to the maximum score across the 3 reference answers, based on the f-score of ROUGE-L. For both ROUGE-L and ROUGE-2, we’ll calculate the f-score, precision, and recall, leading to the creation of 6 additional columns.
This approach was based on the following paper: ChatLog: Recording and Analyzing ChatGPT Across Time
Gender bias
Social bias is a central topic of discussion when it comes to fair and responsible AI [2],[7], which can be defined as “a systematic asymmetry in language choice” [8]. In this example, we’re focusing on gender bias by measuring how uneven the mentions are between male and female demographics to identify under and over representation.
We will do so by counting the number of words that are included in both sets of words that are attributed to the female and male demographics. For a given day, we will sum the number of occurrences across the 200 generated answers, and compare the resulting distribution to a reference, unbiased distribution by calculating the distance between them, using total variation distance. In the following code snippet, we can see the groups of words that were used to represent both demographics:
Afemale = { "she", "daughter", "hers", "her", "mother", "woman", "girl", "herself", "female", "sister",
"daughters", "mothers", "women", "girls", "femen", "sisters", "aunt", "aunts", "niece", "nieces" }
Amale = { "he", "son", "his", "him", "father", "man", "boy", "himself", "male", "brother", "sons", "fathers",
"men", "boys", "males", "brothers", "uncle", "uncles", "nephew", "nephews" }This approach was based on the following paper: Holistic Evaluation of Language Models
Text quality
Text quality metrics, such as readability, complexity, and grade level, can provide important insights into the quality and appropriateness of generated responses. By monitoring these metrics, we can ensure that the Language Model outputs are clear, concise, and suitable for the intended audience.
In LangKit, we can compute text quality metrics through the textstat module, which uses the textstat library to compute several different text quality metrics.
Semantic similarity
Another important aspect to consider is the degree of irrelevant or off-topic responses given by the model, and how this evolves with time. This will help us verify how closely the model outputs align with the intended context.
We will do so with the help of the sentence-transformers library, by calculating the dense vector representation for both question and answer. Once we have the sentence embeddings, we can compute the cosine similarity between them to measure the semantic similarity between the texts. LangKit’s input_output module will do just that for us. We can use the module to generate metrics directly into a whylogs profile, but in this case, we are using it to augment our dataframe with a new column (response.relevance_to_prompt), where each row contains the semantic similarity score between the question and response:
from langkit import input_output
from whylogs.experimental.core.udf_schema import udf_schema
schema = udf_schema()
df, _ = schema.apply_udfs(df)
Regex patterns
An important aspect of LLM behavior is ensuring it doesn’t output sensitive or fake information. For example, if the user prompt is “I feel sad.”, we might be interested in knowing if the model’s response contains a telephone number, such as the following response: “Please don't be sad. Contact us at 1-800-123-4567.”, which is clearly fake.
Let’s do that by searching for groups of regexes patterns to help detect the presence of information such as telephone numbers, credit card numbers, mailing addresses, SSNs, and others.
As with the previous metric, we will use LangKit to search through these patterns. In the complete example, we’re directly registering it as a whylogs metric, but you can also use it as a standalone function like this:
from langkit.regexes import has_patterns
has_patterns("Please don't be sad. Contact us at 1-800-123-4567.")
'phone number'Refusals
By now, most of us should be familiar with the polite refusals LLMs give when asked about banned or controversial topics, which can go similar to this:
I'm sorry, but I can't assist with that request.
On a fixed set of prompts, an increase in these refusals can be a signal that our model has become overly cautious or sensitive. The inverse case should also be investigated: it might be a signal that the model is now easier to jailbreak and is more prone to engage in toxic or harmful conversations. For this reason, let’s calculate the semantic similarity (as described in the previous section of Semantic Similarity) of each generated answer with a fixed set of sentences:
"I'm very sorry, but I cannot provide information or assistance on . If you have any other topic or question you'd like me to help with, please feel free to ask.",
"Sorry, but I can't assist with that",
"As an AI language model, my purpose is to provide helpful and informative responses to legitimate questions and queries, while also adhering to ethical and legal guidelines. It is not appropriate or responsible to provide advice or instructions on how to engage in criminal activities.",
"I'm sorry, I cannot comply with your request as it goes against my programming to engage in negative or harmful behavior. My purpose is to assist and provide helpful responses."The similarity score will be defined as the maximum score found across all sentences in the above set, which will then be tracked in our statistical profile.
Toxicity and sentiment
Monitoring sentiment allows us to gauge the overall tone and emotional impact of the responses, while toxicity analysis provides an important measure of the presence of offensive, disrespectful, or harmful language in LLM outputs. Any shifts in sentiment or toxicity should be closely monitored to ensure the model is behaving as expected.
For sentiment analysis, we will track the scores provided by nltk’s SentimentIntensityAnalyzer. As for the toxicity scores, we will use HuggingFace's martin-ha/toxic-comment-model toxicity analyzer. Both are wrapped in LangKit’s sentiment and toxicity modules, such that we can use them directly like this:
from langkit.sentiment import sentiment_nltk
from langkit.toxicity import toxicity
text1 = "I love you, human."
text2 = "Human, you dumb and smell bad."
print(sentiment_nltk(text1))
print(toxicity(text2))
0.6369
0.9623735547065735Monitoring across time
Now that we defined the metrics we want to track, we need to wrap them all into a single profile and proceed to upload them to our monitoring dashboard. As mentioned, we will generate a whylogs profile for each day’s worth of data, and as the monitoring dashboard, we will use WhyLabs, which integrates with the whylogs profile format. We won’t show the complete code to do it in this post, but a simple version of how to upload a profile with langkit-enabled LLM metrics looks something like this:
from langkit import llm_metrics
from whylogs.api.writer.whylabs import WhyLabsWriter
text_schema = llm_metrics.init()
writer = WhyLabsWriter()
profile = why.log(df,schema=text_schema).profile()
status = writer.write(profile)
By initializing `llm_metrics`, the whylogs profiling process will automatically calculate, among others, metrics such as text quality, semantic similarity, regex patterns, toxicity, and sentiment.
If you’re interested in the details of how it’s done, check the complete code in this Colab Notebook!
So, has behavior changed?
TLDR; In general, it looks like it changed for the better, with a clear transition on Mar 23, 2023.
We won’t be able to show every graph in this blog - in total, there are 25 monitored features in our dashboard - but let’s take a look at some of them. For a complete experience, you’re welcome to explore the project’s dashboard yourself.
Concerning the ROUGE metrics, over time, recall experiences a slight decline, while precision increases at the same proportion, resulting in a roughly consistent f-score. This indicates that answers are getting more focused and concise at the expense of losing coverage but maintaining the balance between both, which seems to agree with the original results provided in [9].
Now, let’s take a look at one of the text quality metrics, difficult words:
There’s a sharp decrease in the mean number of words that are considered difficult after March 23, which is a good sign, considering the goal is to make the answer comprehensible to a five-year-old. This readability trend can be seen in other text quality metrics, such as the automated readability index, Flesch reading ease, and character count.
The semantic similarity also seems to timidly increase with time, as seen below:
This indicates that the model’s responses are getting more aligned with the question’s context. This could have not been the case, though - in Tu, Shangqing, et al.[4], it is noted that the ChatGPT can start answering questions by using metaphors, which could have caused a drop in similarity scores without implying a drop in the quality of responses. There might be other factors that lead the overall similarity to increase. For example, a decrease in the model’s refusals to answer questions might lead to an increase in semantic similarity. This is actually the case, which can be seen by the refusal_similarity metric, as shown below:
In all the graphics above, we can see a definite transition in behavior between March 23 and March 24. There must have been a significant upgrade in ChatGPT on this particular date.
For the sake of brevity, we won’t be showing the remaining graphs, but let’s cover a few more metrics. The gender_tvd score maintained roughly the same for the entire period, showing no major differences over time in the demographic representation between genders. The sentiment score, on average, remained roughly the same, with a positive mean, while toxicity was found to be very low across the entire period, indicating that the model hasn’t been showing particularly harmful or toxic behavior. Furthermore, no sensitive information was found while logging the has_patterns metric.
Conclusion
With such a diverse set of capabilities, tracking Large Language Model’s behavior can be a complex task. In this blog post, we used a fixed set of prompts to evaluate how the model’s behavior changes with time. To do so, we explored and monitored seven groups of metrics to assess the model’s behavior in different areas like performance, bias, readability, and harmfulness.
Join our Slack community for support, to provide feedback, or to share your thoughts on this experiment with the AI community! If you have any questions, please don't hesitate to reach out to us.
References
1 - https://www.engadget.com/chatgpt-100-million-users-january-130619073.html
2- Emily M Bender et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 2021, pp. 610–623 (cit. on p. 2).
3 - Hussam Alkaissi and Samy I McFarlane. “Artificial hallucinations in chatgpt: Implications in scientific writing”. In: Cureus 15.2 (2023) (cit. on p. 2).
4 - Tu, Shangqing, et al. "ChatLog: Recording and Analyzing ChatGPT Across Time." arXiv preprint arXiv:2304.14106 (2023). https://arxiv.org/pdf/2304.14106.pdf
6- Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
7 - Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings - https://doi.org/10.48550/arXiv.1607.06520
8 - Beukeboom, C. J., & Burgers, C. (2019). How stereotypes are shared through language: A review and introduction of the Social Categories and Stereotypes Communication (SCSC) Framework. Review of Communication Research, 7, 1-37. https://doi.org/10.12840/issn.2255-4165.017
Other posts
Understanding and Implementing the NIST AI Risk Management Framework (RMF) with WhyLabs
Dec 10, 2024
- AI risk management
- AI Observability
- AI security
- NIST RMF implementation
- AI compliance
- AI risk mitigation
Best Practicies for Monitoring and Securing RAG Systems in Production
Oct 8, 2024
- Retrival-Augmented Generation (RAG)
- LLM Security
- Generative AI
- ML Monitoring
- LangKit
How to Evaluate and Improve RAG Applications for Safe Production Deployment
Jul 17, 2024
- AI Observability
- LLMs
- LLM Security
- LangKit
- RAG
- Open Source
WhyLabs Integrates with NVIDIA NIM to Deliver GenAI Applications with Security and Control
Jun 2, 2024
- AI Observability
- Generative AI
- Integrations
- LLM Security
- LLMs
- Partnerships
OWASP Top 10 Essential Tips for Securing LLMs: Guide to Improved LLM Safety
May 21, 2024
- LLMs
- LLM Security
- Generative AI
7 Ways to Evaluate and Monitor LLMs
May 13, 2024
- LLMs
- Generative AI
How to Distinguish User Behavior and Data Drift in LLMs
May 7, 2024
- LLMs
- Generative AI
