{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Detecting Issues in a Text Dataset with Datalab\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this 5-minute quickstart tutorial, we use Datalab to detect various issues in an intent classification dataset composed of (text) customer service requests at an online bank. We consider a subset of the [Banking77-OOS Dataset](https://arxiv.org/abs/2106.04564) containing 1,000 customer service requests which are classified into 10 categories based on their intent (you can run this same code on any text classification dataset). Cleanlab automatically identifies bad examples in our dataset, including mislabeled data, out-of-scope examples (outliers), or otherwise ambiguous examples. Consider filtering or correcting such bad examples\u00a0before you dive deep into modeling your data!\n", "\n", "**Overview of what we'll do in this tutorial:**\n", "\n", "- Use a pretrained transformer model to extract the text embeddings from the customer service requests\n", "\n", "- Train a simple Logistic Regression model on the text embeddings to compute out-of-sample predicted probabilities\n", "\n", "- Run cleanlab's `Datalab` audit with these predictions and embeddings in order to identify problems like: label issues, outliers, and near duplicates in the dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "Quickstart\n", "
\n", " \n", "Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you have some numeric `features` as well? Run the code below to find any potential label errors in your dataset.\n", "\n", "
\n", " \n", "```ipython3 \n", "from cleanlab import Datalab\n", "\n", "lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n", "lab.find_issues(pred_probs=your_pred_probs, features=your_features)\n", "\n", "lab.report()\n", "lab.get_issues()\n", "```\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Install required dependencies\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use `pip` to install all packages required for this tutorial as follows:\n", "\n", "```ipython3\n", "!pip install sentence-transformers\n", "!pip install \"cleanlab[datalab]\"\n", "# Make sure to install the version corresponding to this tutorial\n", "# E.g. if viewing master branch documentation:\n", "# !pip install git+https://github.com/cleanlab/cleanlab.git\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Package installation (hidden on docs.cleanlab.ai).\n", "# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n", "# Package versions we used:scikit-learn==1.2.0 sentence-transformers>=2.3.0\n", "\n", "dependencies = [\"cleanlab\", \"sentence_transformers\", \"datasets\"]\n", "\n", "if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n", " %pip install cleanlab # for colab\n", " cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n", " %pip install $cmd\n", "else:\n", " dependencies_test = [dependency.split('>')[0] if '>' in dependency \n", " else dependency.split('<')[0] if '<' in dependency \n", " else dependency.split('=')[0] for dependency in dependencies]\n", " missing_dependencies = []\n", " for dependency in dependencies_test:\n", " try:\n", " __import__(dependency)\n", " except ImportError:\n", " missing_dependencies.append(dependency)\n", "\n", " if len(missing_dependencies) > 0:\n", " print(\"Missing required dependencies:\")\n", " print(*missing_dependencies, sep=\", \")\n", " print(\"\\nPlease install them before running the rest of this notebook.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import re \n", "import string \n", "import pandas as pd \n", "from sklearn.metrics import accuracy_score, log_loss \n", "from sklearn.model_selection import cross_val_predict \n", "from sklearn.linear_model import LogisticRegression\n", "from sentence_transformers import SentenceTransformer\n", "\n", "from cleanlab import Datalab" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# This cell is hidden from docs.cleanlab.ai \n", "\n", "import random \n", "import numpy as np \n", "\n", "pd.set_option(\"display.max_colwidth\", None) \n", "\n", "SEED = 123456 # for reproducibility\n", "np.random.seed(SEED)\n", "random.seed(SEED)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Load and format the text dataset\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = pd.read_csv(\"https://s.cleanlab.ai/banking-intent-classification.csv\")\n", "data.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "raw_texts, labels = data[\"text\"].values, data[\"label\"].values\n", "num_classes = len(set(labels))\n", "\n", "print(f\"This dataset has {num_classes} classes.\")\n", "print(f\"Classes: {set(labels)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's view the i-th example in the dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "i = 1 # change this to view other examples from the dataset\n", "print(f\"Example Label: {labels[i]}\")\n", "print(f\"Example Text: {raw_texts[i]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data is stored as two numpy arrays:\n", "\n", "1. `raw_texts` stores the customer service requests utterances in text format\n", "2. `labels` stores the intent categories (labels) for each example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "Bringing Your Own Data (BYOD)?\n", "\n", "You can easily replace the above with your own text dataset, and continue with the rest of the tutorial.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next we convert the text strings into vectors better suited as inputs for our ML models. \n", "\n", "We will use numeric representations from a pretrained Transformer model as embeddings of our text. The [Sentence Transformers](https://huggingface.co/docs/hub/sentence-transformers) library offers simple methods to compute these embeddings for text data. Here, we load the pretrained `electra-small-discriminator` model, and then run our data through network to extract a vector embedding\u00a0of each example." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "transformer = SentenceTransformer('google/electra-small-discriminator')\n", "text_embeddings = transformer.encode(raw_texts)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our subsequent ML model will directly operate on elements of `text_embeddings` in order to classify the customer service requests." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Define a classification model and compute out-of-sample predicted probabilities" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted embeddings.\n", "\n", "To identify label issues, cleanlab requires a probabilistic prediction from your model for each datapoint. However these predictions will be _overfit_ (and thus unreliable) for datapoints the model was previously trained on. cleanlab is intended to only be used with **out-of-sample** predicted class probabilities, i.e. on datapoints held-out from the model during the training.\n", "\n", "Here we obtain out-of-sample predicted class probabilities for every example in our dataset using a Logistic Regression model with cross-validation.\n", "Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "model = LogisticRegression(max_iter=400)\n", "\n", "pred_probs = cross_val_predict(model, text_embeddings, labels, method=\"predict_proba\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Use cleanlab to find issues in your dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given feature embeddings and the (out-of-sample) predicted class probabilities obtained from any model\u00a0you have, cleanlab can quickly help you identify low-quality examples in your dataset.\n", "\n", "Here, we use cleanlab's `Datalab` to find issues in our data. Datalab offers several ways of loading the data; we\u2019ll simply wrap the training features and noisy labels in a dictionary. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_dict = {\"texts\": raw_texts, \"labels\": labels}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All that is need to audit your data is to call `find_issues()`. We pass in the predicted probabilities and the feature embeddings obtained above, but you do not necessarily need to provide all of this information depending on which types of issues you are interested in. The more inputs you provide, the more types of issues `Datalab` can detect in your data. Using a better model to produce these inputs will ensure cleanlab more accurately estimates issues." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "lab = Datalab(data_dict, label_name=\"labels\")\n", "lab.find_issues(pred_probs=pred_probs, features=text_embeddings)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After the audit is complete, review the findings using the `report` method:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "lab.report()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Label issues\n", "\n", "The report indicates that cleanlab identified many label issues in our dataset. We can see which examples are flagged as likely mislabeled and the label quality score for each example using the `get_issues` method, specifying `label` as an argument to focus on label issues in the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "label_issues = lab.get_issues(\"label\")\n", "label_issues.head() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 5 most likely mislabeled examples in our dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "identified_label_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n", "lowest_quality_labels = label_issues[\"label_score\"].argsort()[:5].to_numpy()\n", "\n", "print(\n", " f\"cleanlab found {len(identified_label_issues)} potential label errors in the dataset.\\n\"\n", " f\"Here are indices of the top 5 most likely errors: \\n {lowest_quality_labels}\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's review some of the most likely label errors. \n", "\n", "Here we display the top 5 examples identified as the most likely label errors in the dataset, together with their given (original) label and a suggested alternative label from cleanlab.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_with_suggested_labels = pd.DataFrame(\n", " {\"text\": raw_texts, \"given_label\": labels, \"suggested_label\": label_issues[\"predicted_label\"]}\n", ")\n", "data_with_suggested_labels.iloc[lowest_quality_labels]" ] }, { "cell_type": "markdown", "metadata": { "scrolled": true }, "source": [ "These are very clear label errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the intent of these requests, whoever produced this dataset made many mistakes that are important to address before modeling the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Outlier issues\n", "\n", "According to the report, our dataset contains some outliers.\n", "We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "outlier_issues = lab.get_issues(\"outlier\")\n", "outlier_issues.sort_values(\"outlier_score\").head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lowest_quality_outliers = outlier_issues[\"outlier_score\"].argsort()[:5]\n", "\n", "data.iloc[lowest_quality_outliers]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that cleanlab has identified entries in this dataset that do not appear to be proper customer requests. Outliers in this dataset appear to be out-of-scope customer requests and other nonsensical text which does not make sense for intent classification. Carefully consider whether such outliers may detrimentally affect your data modeling, and consider removing them from the dataset if so." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Near-duplicate issues\n", "\n", "According to the report, our dataset contains some sets of nearly duplicated examples.\n", "We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the text examples in our dataset that are most nearly duplicated." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "duplicate_issues = lab.get_issues(\"near_duplicate\")\n", "duplicate_issues.sort_values(\"near_duplicate_score\").head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see that example 160 and 148 are nearly duplicated, as are example 546 and 514.\n", "\n", "Let's view these examples to see how similar they are." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.iloc[[160, 148]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.iloc[[546, 514]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that these two sets of request are indeed very similar to one another! Including near duplicates in a dataset may have unintended effects on models, and be wary about splitting them across training/test sets. Learn more about handling near duplicates in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Non-IID issues (data drift)\n", "According to the report, our dataset does not appear to be Independent and Identically Distributed (IID). The overall non-iid score for the dataset (displayed below) corresponds to the `p-value` of a statistical test for whether the ordering of samples in the dataset appears related to the similarity between their feature values. A low `p-value` strongly suggests that the dataset violates the IID assumption, which is a key assumption required for conclusions (models) produced from the dataset to generalize to a larger population." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "p_value = lab.get_info('non_iid')['p-value']\n", "p_value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, our dataset was flagged as non-IID because the rows happened to be sorted by class label in the original data. This may be benign if we remember to shuffle rows before model training and data splitting. But if you don't know why your data was flagged as non-IID, then you should be worried about potential data drift or unexpected interactions between data points (their values may not be statistically independent). Think carefully about what future test data may look like (and whether your data is representative of the population you care about). You should not shuffle your data before the non-IID test runs (will invalidate its conclusions)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As demonstrated above, cleanlab can automatically shortlist the most likely issues in your dataset to help you better curate your dataset for subsequent modeling. With this shortlist, you can decide whether to fix these label issues or remove nonsensical or duplicated examples from your dataset to obtain a higher-quality dataset for training your next ML model. cleanlab's issue detection can be run with outputs from *any* type of model you initially trained.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n", "\n", "label_issue_indices = [981, 974, 982] # check these examples were found in label issues\n", "if not all(x in identified_label_issues.index for x in label_issue_indices):\n", " raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n", " \n", "identified_outlier_issues = outlier_issues[outlier_issues[\"is_outlier_issue\"] == True]\n", "outlier_issue_indices = [994, 989, 999] # check these examples were found in duplicates\n", "if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n", " raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n", "\n", "identified_duplicate_issues = duplicate_issues[duplicate_issues[\"is_near_duplicate_issue\"] == True]\n", "duplicate_issue_indices = [160, 148, 546, 514] # check these examples were found in duplicates\n", "if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n", " raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Text Classification with Datalab", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.7" } }, "nbformat": 4, "nbformat_minor": 4 }