wehub/cleanlab--cleanlab
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:49:22 +08:00
commit a41b2ab474
303 changed files with 64772 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/tutorials/.gitignore
+3
View File
@@ -0,0 +1,3 @@
*.tar.gz
pretrained_models/
spoken_digits/
docs/source/tutorials/clean_learning/index.rst
+8
View File
@@ -0,0 +1,8 @@
CleanLearning Tutorials
=======================
.. toctree::
:maxdepth: 1
Text Classification <text>
Tabular Classification (Numeric/Categorical) <tabular>
docs/source/tutorials/clean_learning/tabular.ipynb
+506
View File
@@ -0,0 +1,506 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Classification with Structured/Tabular Data and Noisy Labels\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Consider Using Datalab\n",
"<br/>\n",
"\n",
"If interested in detecting a wide variety of issues in your tabular data, check out the [Datalab tabular tutorial](https://docs.cleanlab.ai/stable/tutorials/datalab/tabular.html). Datalab can detect many other types of data issues beyond label issues, whereas CleanLearning is a convenience method to handle noisy labels with sklearn-compatible classification models.\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use cleanlab with scikit-learn models to find potential label errors in a classification dataset with tabular features (numeric/categorical columns). Tabular (or *structured*) data are typically organized in a row/column format and stored in a SQL database or file types like: CSV, Excel, or Parquet. Here we consider a Student Grades dataset, which contains over 900 individuals who have three exam grades and some optional notes, each being assigned a letter grade (their class label). cleanlab automatically identifies _hundreds_ of examples in this dataset that were mislabeled with the incorrect final grade (data entry mistakes). \n",
"\n",
"This tutorial shows how to handle noisy labels and produce more robust classification models for your own tabular datasets. cleanlab's `CleanLearning` class automatically detects and filters out such badly labeled data, in order to train a more robust version of any Machine Learning model. No change to your existing modeling code is required! \n",
"\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Train a classifier model (here scikit-learn's ExtraTreesClassifier, although any model could be used) and use this classifier to compute (out-of-sample) predicted class probabilities via cross-validation.\n",
"\n",
"- Identify potential label errors in the data with cleanlab's `find_label_issues` method.\n",
"\n",
"- Train a robust version of the same ExtraTrees model via cleanlab's `CleanLearning` wrapper.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have an sklearn compatible `model`, tabular `data` and given `labels`? Run the code below to train your `model` and get label issues.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.classification import CleanLearning\n",
"\n",
"cl = CleanLearning(model)\n",
"_ = cl.fit(train_data, labels)\n",
"label_issues = cl.get_label_issues()\n",
"preds = cl.predict(test_data) # predictions from a version of your model \n",
" # trained on auto-cleaned data\n",
"\n",
"\n",
"```\n",
" \n",
"</div>\n",
" \n",
"Is your model/data not compatible with `CleanLearning`? You can instead run cross-validation on your model to get out-of-sample `pred_probs`. Then run the code below to get label issue indices ranked by their inferred severity.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.filter import find_label_issues\n",
"\n",
"ranked_label_issues = find_label_issues(\n",
" labels,\n",
" pred_probs,\n",
" return_indices_ranked_by=\"self_confidence\",\n",
")\n",
" \n",
"\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install cleanlab\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numpy as np\n",
"import pandas as pd \n",
"from sklearn.preprocessing import StandardScaler, LabelEncoder\n",
"from sklearn.model_selection import cross_val_predict, train_test_split\n",
"from sklearn.metrics import accuracy_score\n",
"from sklearn.ensemble import ExtraTreesClassifier\n",
"\n",
"from cleanlab.filter import find_label_issues\n",
"from cleanlab.classification import CleanLearning\n",
"\n",
"SEED = 100 \n",
"\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and process the data\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We first load the data features and labels (which are possibly noisy).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"grades_data = pd.read_csv(\"https://s.cleanlab.ai/grades-tabular-demo-v2.csv\")\n",
"grades_data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_raw = grades_data[[\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]]\n",
"labels_raw = grades_data[\"letter_grade\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we preprocess the data. Here we apply one-hot encoding to features with categorical data, and standardize features with numeric data. We also perform label encoding on the labels, as cleanlab's functions require the labels for each example to be an interger integer in 0, 1, …, num_classes - 1. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"categorical_features = [\"notes\"]\n",
"X_encoded = pd.get_dummies(X_raw, columns=categorical_features, drop_first=True)\n",
"\n",
"numeric_features = [\"exam_1\", \"exam_2\", \"exam_3\"]\n",
"scaler = StandardScaler()\n",
"X_processed = X_encoded.copy()\n",
"X_processed[numeric_features] = scaler.fit_transform(X_encoded[numeric_features])\n",
"\n",
"encoder = LabelEncoder()\n",
"encoder.fit(labels_raw)\n",
"labels = encoder.transform(labels_raw)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"You can easily replace the above with your own tabular dataset, and continue with the rest of the tutorial.\n",
" \n",
"Your classes (and entries of `labels`) should be represented as integer indices 0, 1, ..., num_classes - 1. \n",
"For example, if your dataset has 7 examples from 3 classes, `labels` might look like: `np.array([2,0,0,1,2,0,1])`\n",
"\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Select a classification model and compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we use a simple ExtraTrees classifier that fits various randomized decision tress on our data, but you can choose any suitable scikit-learn model for this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"clf = ExtraTreesClassifier()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find potential labeling errors, cleanlab requires a probabilistic prediction from your model for every datapoint. However, these predictions will be _overfitted_ (and thus unreliable) for examples the model was previously trained on. For the best results, cleanlab should be applied with **out-of-sample** predicted class probabilities, i.e., on examples held out from the model during the training.\n",
"\n",
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. An additional benefit of cross-validation is that it provides a more reliable evaluation of our model than a single training/validation split. We can implement this via the `cross_val_predict` method from scikit-learn:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_crossval_folds = 5 \n",
"pred_probs = cross_val_predict(\n",
" clf,\n",
" X_processed,\n",
" labels,\n",
" cv=num_crossval_folds,\n",
" method=\"predict_proba\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the given labels and out-of-sample predicted probabilities, cleanlab can quickly help us identify poorly labeled instances in our data table. For a dataset with N examples from K classes, the labels should be a 1D array of length N and predicted probabilities should be a 2D (N x K) array. Here we request that the indices of the identified label issues be sorted by cleanlab's self-confidence score, which measures the quality of each given label via the probability assigned to it in our model's prediction."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ranked_label_issues = find_label_issues(\n",
" labels=labels, pred_probs=pred_probs, return_indices_ranked_by=\"self_confidence\"\n",
")\n",
"\n",
"print(f\"Cleanlab found {len(ranked_label_issues)} potential label errors.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's review some of the most likely label errors:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_raw.iloc[ranked_label_issues].assign(label=labels_raw.iloc[ranked_label_issues]).head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These final grades look suspicious and should definitely be carefully re-examined! This is a straightforward approach to visualize the rows in a data table that might be mislabeled."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Train a more robust model from noisy labels\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Following proper ML practice, let's split our data into train and test sets.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, labels_train, labels_test = train_test_split(\n",
" X_encoded,\n",
" labels,\n",
" test_size=0.2,\n",
" random_state=SEED,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We again standardize the numeric features, this time fitting the scaling parameters solely on the training set.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"scaler = StandardScaler()\n",
"X_train[numeric_features] = scaler.fit_transform(X_train[numeric_features])\n",
"X_test[numeric_features] = scaler.transform(X_test[numeric_features])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now train and evaluate the original ExtraTrees model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"clf.fit(X_train, labels_train)\n",
"acc_og = clf.score(X_test, labels_test)\n",
"print(f\"Test accuracy of original model: {acc_og}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"cleanlab provides a wrapper class that can be easily applied to any scikit-learn compatible model. Once wrapped, the resulting model can still be used in the exact same manner, but it will now train more robustly if the data have noisy labels.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"clf = ExtraTreesClassifier() # Note we first re-initialize clf\n",
"cl = CleanLearning(clf) # cl has same methods/attributes as clf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following operations take place when we train the cleanlab-wrapped model: The original model is trained in a cross-validated fashion to produce out-of-sample predicted probabilities. Then, these predicted probabilities are used to identify label issues, which are then removed from the dataset. Finally, the original model is trained on the remaining clean subset of the data once more.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"_ = cl.fit(X_train, labels_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get predictions from the resulting model and evaluate them, just like how we did it for the original scikit-learn model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"preds = cl.predict(X_test)\n",
"acc_cl = accuracy_score(labels_test, preds)\n",
"print(f\"Test accuracy of cleanlab-trained model: {acc_cl}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the test set accuracy slightly improved as a result of the data cleaning. Note that this will not always be the case, especially when we evaluate on test data that are themselves noisy. The best practice is to run cleanlab to identify potential label issues and then manually review them, before blindly trusting any accuracy metrics. In particular, the most effort should be made to ensure high-quality test data, which is supposed to reflect the expected performance of our model during deployment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"if acc_og >= acc_cl: # check cleanlab has improved prediction accuracy\n",
" raise Exception(\"Cleanlab training failed to improve model accuracy.\")\n",
" \n",
"# this file contains true and noisy labels\n",
"true_data = pd.read_csv(\"https://s.cleanlab.ai/student-grades-demo.csv\")\n",
"true_errors = np.where(true_data[\"letter_grade\"] != true_data[\"noisy_letter_grade\"])[0]\n",
"if not all(x in true_errors for x in ranked_label_issues[:5]): # check top errors are indeed errors\n",
" raise Exception(\"Some of the top listed errors are not actually label errors.\")"
]
}
],
"metadata": {
"interpreter": {
"hash": "cda20062bc42cfdcaa0f9720c0b28e880bba110e9dfce6c1689934eec9b595a1"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/clean_learning/text.ipynb
+584
View File
@@ -0,0 +1,584 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Text Classification with Noisy Labels\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Consider Using Datalab\n",
"<br/>\n",
"\n",
"If you are interested in detecting a wide variety of issues in your text dataset, check out the [Datalab text tutorial](https://docs.cleanlab.ai/stable/tutorials/datalab/text.html). Datalab can detect many other types of data issues beyond label issues, whereas CleanLearning is a convenience method to handle noisy labels with sklearn-compatible classification models.\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use cleanlab to find potential label errors in an intent classification dataset composed of (text) customer service requests at an online bank. We consider a subset of the [Banking77-OOS Dataset](https://arxiv.org/abs/2106.04564) containing 1,000 customer service requests which can be classified into 10 categories corresponding to the intent of the request. cleanlab will shortlist examples that confuse our ML model the most; many of which are potential label errors, out-of-scope examples, or otherwise ambiguous examples. cleanlab's `CleanLearning` class automatically detects and filters out such badly labeled data, in order to train a more robust version of any Machine Learning model. No change to your existing modeling code is required!\n",
"\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Define a ML model that can be trained on our dataset (here we use Logistic Regression applied to text embeddings from a pretrained Transformer network, you can use any text classifier model).\n",
"\n",
"- Use `CleanLearning` to wrap this ML model and compute out-of-sample predicted class probabilites, which allow us to identify potential label errors in the dataset.\n",
"\n",
"- Train a more robust version of the same ML model after dropping the detected label errors using `CleanLearning`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have an sklearn compatible `model`, `data` and given `labels`? Run the code below to train your `model` and get label issues using `CleanLearning`. \n",
" \n",
"You can subsequently use the same `CleanLearning` object to train a more robust model (only trained on the clean data) by calling the `.fit()` method and passing in the `label_issues` found earlier.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.classification import CleanLearning\n",
"\n",
"cl = CleanLearning(model)\n",
"label_issues = cl.find_label_issues(train_data, labels) # identify mislabeled examples \n",
" \n",
"cl.fit(train_data, labels, label_issues=label_issues)\n",
"preds = cl.predict(test_data) # predictions from a version of your model \n",
" # trained on auto-cleaned data\n",
"\n",
"\n",
"```\n",
" \n",
"</div>\n",
" \n",
"Is your model/data not compatible with `CleanLearning`? You can instead run cross-validation on your model to get out-of-sample `pred_probs`. Then run the code below to get label issue indices ranked by their inferred severity.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.filter import find_label_issues\n",
"\n",
"ranked_label_issues = find_label_issues(\n",
" labels,\n",
" pred_probs,\n",
" return_indices_ranked_by=\"self_confidence\",\n",
")\n",
" \n",
"\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install sentence-transformers\n",
"!pip install cleanlab\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs.cleanlab.ai).\n",
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
"# Package versions we used:scikit-learn==1.2.0 sentence-transformers>=2.3.0\n",
"\n",
"dependencies = [\"cleanlab\", \"sentence_transformers\"]\n",
"\n",
"# Supress outputs that may appear if tensorflow happens to be improperly installed: \n",
"import os \n",
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\" # disable parallelism to avoid deadlocks with huggingface\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import re \n",
"import string \n",
"import pandas as pd \n",
"from sklearn.metrics import accuracy_score\n",
"from sklearn.model_selection import train_test_split, cross_val_predict \n",
"from sklearn.preprocessing import LabelEncoder\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sentence_transformers import SentenceTransformer\n",
"\n",
"from cleanlab.classification import CleanLearning"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai \n",
"\n",
"import random \n",
"import numpy as np \n",
"\n",
"pd.set_option(\"display.max_colwidth\", None) \n",
"\n",
"SEED = 123456 # for reproducibility \n",
"\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and format the text dataset\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pd.read_csv(\"https://s.cleanlab.ai/banking-intent-classification.csv\")\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw_texts, raw_labels = data[\"text\"].values, data[\"label\"].values\n",
"\n",
"raw_train_texts, raw_test_texts, raw_train_labels, raw_test_labels = train_test_split(raw_texts, raw_labels, test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_classes = len(set(raw_train_labels))\n",
"\n",
"print(f\"This dataset has {num_classes} classes.\")\n",
"print(f\"Classes: {set(raw_train_labels)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's print the first example in the train set."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"i = 0\n",
"print(f\"Example Label: {raw_train_labels[i]}\")\n",
"print(f\"Example Text: {raw_train_texts[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data is stored as two numpy arrays for each the train and test set:\n",
"\n",
"1. `raw_train_texts` and `raw_test_texts` store the customer service requests utterances in text format\n",
"2. `raw_train_labels` and `raw_test_labels` store the intent categories (labels) for each example\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we need to perform label enconding on the labels, cleanlab's functions require the labels for each example to be an interger integer in 0, 1, \u2026, num_classes - 1. We will use sklearn's `LabelEncoder` to encode our labels.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"encoder = LabelEncoder()\n",
"encoder.fit(raw_train_labels)\n",
"\n",
"train_labels = encoder.transform(raw_train_labels)\n",
"test_labels = encoder.transform(raw_test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"You can easily replace the above with your own text dataset, and continue with the rest of the tutorial.\n",
"\n",
"Your classes (and entries of `train_labels` / `test_labels`) should be represented as integer indices 0, 1, ..., num_classes - 1.\n",
"For example, if your dataset has 7 examples from 3 classes, `train_labels` might be: `np.array([2,0,0,1,2,0,1])`\n",
"\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we convert the text strings into vectors better suited as inputs for our ML model. \n",
"\n",
"We will use numeric representations from a pretrained Transformer model as embeddings of our text. The [Sentence Transformers](https://huggingface.co/docs/hub/sentence-transformers) library offers simple methods to compute these embeddings for text data. Here, we load the pretrained `electra-small-discriminator` model, and then run our data through network to extract a vector embedding of each example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"transformer = SentenceTransformer('google/electra-small-discriminator')\n",
"\n",
"train_texts = transformer.encode(raw_train_texts)\n",
"test_texts = transformer.encode(raw_test_texts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our subsequent ML model will directly operate on elements of `train_texts` and `test_texts` in order to classify the customer service requests."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Define a classification model and use cleanlab to find potential label errors\n",
"\n",
"<a id=\"section3\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted embeddings."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression(max_iter=400)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can define the `CleanLearning` object with our Logistic Regression model and use `find_label_issues` to identify potential label errors.\n",
"\n",
"`CleanLearning` provides a wrapper class that can easily be applied to any scikit-learn compatible model, which can be used to find potential label issues and train a more robust model if the original data contains noisy labels."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cv_n_folds = 5 # for efficiency; values like 5 or 10 will generally work better\n",
"\n",
"cl = CleanLearning(model, cv_n_folds=cv_n_folds)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"label_issues = cl.find_label_issues(X=train_texts, labels=train_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `find_label_issues` method above will perform cross validation to compute out-of-sample predicted probabilites for each example, which is used to identify label issues.\n",
"\n",
"This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled). Note that the given and predicted labels here are encoded as intergers as that was the format expected by `cleanlab`, we will inverse transform them later in this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"label_issues.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 10 most likely mislabeled examples in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identified_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = label_issues[\"label_quality\"].argsort()[:10].to_numpy()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\n",
" f\"cleanlab found {len(identified_issues)} potential label errors in the dataset.\\n\"\n",
" f\"Here are indices of the top 10 most likely errors: \\n {lowest_quality_labels}\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's review some of the most likely label errors. To help us inspect these datapoints, we define a method to print any example from the dataset, together with its given (original) label and the suggested alternative label from cleanlab.\n",
"\n",
"We then display some of the top-ranked label issues identified by cleanlab:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def print_as_df(index):\n",
" return pd.DataFrame(\n",
" {\n",
" \"text\": raw_train_texts, \n",
" \"given_label\": raw_train_labels,\n",
" \"predicted_label\": encoder.inverse_transform(label_issues[\"predicted_label\"]),\n",
" },\n",
" ).iloc[index]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print_as_df(lowest_quality_labels[:5])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are very clear label errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the intent of these requests, whoever produced this dataset made many mistakes that are important to address before modeling the data.\n",
"\n",
"cleanlab has shortlisted the most likely label errors to speed up your data cleaning process. With this list, you can decide whether to fix these label issues or remove ambiguous examples from the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Train a more robust model from noisy labels\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Fixing the label issues manually may be time-consuming, but cleanlab can filter these noisy examples and train a model on the remaining clean data for you automatically.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To establish a baseline, let's first train and evaluate our original Logistic Regression model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"baseline_model = LogisticRegression(max_iter=400) # note we first re-instantiate the model\n",
"baseline_model.fit(X=train_texts, y=train_labels)\n",
"\n",
"preds = baseline_model.predict(test_texts)\n",
"acc_og = accuracy_score(test_labels, preds)\n",
"print(f\"\\n Test accuracy of original model: {acc_og}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have a baseline, let's check if using `CleanLearning` improves our test accuracy.\n",
"\n",
"`CleanLearning` provides a wrapper that can be applied to any scikit-learn compatible model. The resulting model object can be used in the same manner, but it will now train more robustly if the data has noisy labels.\n",
"\n",
"We can use the same `CleanLearning` object defined above, and pass the label issues we already computed into `.fit()` via the `label_issues` argument. This accelerates things; if we did not provide the label issues, then they would be recomputed via cross-validation. After that `CleanLearning` simply deletes the examples with label issues and retrains your model on the remaining data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"cl.fit(X=train_texts, labels=train_labels, label_issues=cl.get_label_issues())\n",
"\n",
"pred_labels = cl.predict(test_texts)\n",
"acc_cl = accuracy_score(test_labels, pred_labels)\n",
"print(f\"Test accuracy of cleanlab's model: {acc_cl}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the test set accuracy slightly improved as a result of the data cleaning. Note that this will not always be the case, especially when we are evaluating on test data that are themselves noisy. The best practice is to run cleanlab to identify potential label issues and then manually review them, before blindly trusting any accuracy metrics. In particular, the most effort should be made to ensure high-quality test data, which is supposed to reflect the expected performance of our model during deployment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"highlighted_indices = [646, 390, 628, 702] # check these examples were found in find_label_issues\n",
"if not all(x in identified_issues.index for x in highlighted_indices):\n",
" raise Exception(\"Some highlighted examples are missing from ranked_label_issues.\")\n",
"\n",
"# Also check that cleanlab has improved prediction accuracy\n",
"if acc_og >= acc_cl:\n",
" raise Exception(\"Cleanlab training failed to improve model accuracy.\")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Text Classification with CleanLearning",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/audio.ipynb
+791
View File
@@ -0,0 +1,791 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "eVufWTY3jRPx"
},
"source": [
"# Detecting Issues in an Audio Dataset with Datalab\n",
"\n",
"In this 5-minute quickstart tutorial, we use cleanlab to find label issues in the [Spoken Digit dataset](https://www.tensorflow.org/datasets/catalog/spoken_digit) (it's like MNIST for audio). The dataset contains 2,500 audio clips with English pronunciations of the digits 0 to 9 (these are the class labels to predict from the audio).\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Extract features from audio clips (.wav files) using a [pre-trained Pytorch model](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) from HuggingFace that was previously fit to the [VoxCeleb](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/) speech dataset.\n",
"\n",
"- Train a cross-validated linear model using the extracted features and generate out-of-sample predicted probabilities.\n",
"\n",
"- Apply cleanlab's `Datalab` audit to these predictions in order to identify which audio clips in the dataset are likely mislabeled.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have a `model`? Run cross-validation to get out-of-sample `pred_probs`, and then run the code below to audit your dataset and identify any potential issues.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, issue_types={\"label\":{}})\n",
"\n",
"lab.get_issues(\"label\")\n",
" \n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eqsqBq3PiUHA"
},
"source": [
"## 1. Install dependencies and import them\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7nT-U9qc8MS"
},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install huggingface_hub==0.17.0 speechbrain==0.5.13 \n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"# Package versions used: torch==2.1.2 torchaudio==2.1.2 speechbrain==0.5.13\n",
"\n",
"dependencies = [\"cleanlab\", \"huggingface_hub==0.17.0\", \"speechbrain==0.5.13\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "x-oboEbRdhf6"
},
"source": [
"Let's import some of the packages needed throughout this tutorial.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LaEiwXUiVHCS"
},
"outputs": [],
"source": [
"import os\n",
"import pandas as pd\n",
"import numpy as np\n",
"import random\n",
"import torch\n",
"import torchaudio\n",
"import torchaudio\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"SEED = 456 # ensure reproducibility"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This (optional) cell is hidden from docs.cleanlab.ai \n",
"\n",
"def set_seed(seed=0):\n",
" \"\"\"Ensure reproducibility.\"\"\"\n",
" np.random.seed(seed)\n",
" torch.manual_seed(seed)\n",
" torch.backends.cudnn.deterministic = True\n",
" torch.backends.cudnn.benchmark = False\n",
" torch.cuda.manual_seed_all(seed)\n",
"\n",
"\n",
"set_seed(SEED)\n",
"pd.options.display.max_colwidth = 500"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SOen_sxQidLC"
},
"source": [
"## 2. Load the data\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uHVskN2eeNj6"
},
"source": [
"We must first fetch the dataset. To run the below command, you'll need to have `wget` installed; alternatively you can manually navigate to the link in your browser and download from there.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GRDPEg7-VOQe",
"outputId": "cb886220-e86e-4a77-9f3a-d7844c37c3a6"
},
"outputs": [],
"source": [
"%%capture\n",
"\n",
"!wget https://github.com/Jakobovski/free-spoken-digit-dataset/archive/v1.0.9.tar.gz\n",
"!mkdir spoken_digits\n",
"!tar -xf v1.0.9.tar.gz -C spoken_digits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tRvNnyB0e_IE"
},
"source": [
"The audio data are .wav files in the `recordings/` folder. Note that the label for each audio clip (i.e. digit from 0 to 9) is indicated in the prefix of the file name (e.g. `6_nicolas_32.wav` has the label 6). If instead applying cleanlab to your own dataset, its classes should be represented as integer indices 0, 1, ..., num_classes - 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FDA5sGZwUSur",
"outputId": "0cedc509-63fd-4dc3-d32f-4b537dfe3895"
},
"outputs": [],
"source": [
"DATA_PATH = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/\"\n",
"\n",
"# Get list of .wav file names\n",
"# os.listdir order is nondeterministic, so for reproducibility,\n",
"# we sort first and then do a deterministic shuffle\n",
"file_names = sorted(i for i in os.listdir(DATA_PATH) if i.endswith(\".wav\"))\n",
"random.Random(SEED).shuffle(file_names)\n",
"\n",
"file_paths = [os.path.join(DATA_PATH, name) for name in file_names]\n",
"\n",
"# Check out first 3 files\n",
"file_paths[:3]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xi2592bVhSab"
},
"source": [
"Let's listen to some example audio clips from the dataset. We introduce a `display_example` function to process the .wav file so we can listen to it in this notebook (can skip these details)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the implementation of `display_example` **(click to expand)**</summary>\n",
"\n",
"```python\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import torch\n",
"import torchaudio\n",
"from pathlib import Path\n",
"from IPython import display\n",
"\n",
"# Utility function for loading audio files and making sure the sample rate is correct.\n",
"def load_wav_16k_mono(filename):\n",
" \"\"\"Load a WAV file, convert it to a float tensor, resample to 16 kHz single-channel audio.\"\"\"\n",
" # Load audio file with torchaudio\n",
" waveform, sample_rate = torchaudio.load(filename)\n",
" \n",
" # Convert to mono if stereo\n",
" if waveform.shape[0] > 1:\n",
" waveform = torch.mean(waveform, dim=0, keepdim=True)\n",
" \n",
" # Resample to 16kHz if needed\n",
" if sample_rate != 16000:\n",
" resampler = torchaudio.transforms.Resample(sample_rate, 16000)\n",
" waveform = resampler(waveform)\n",
" \n",
" return waveform.squeeze()\n",
"\n",
"\n",
"def display_example(wav_file_name, audio_rate=16000):\n",
" \"\"\"Allows us to listen to any wav file and displays its given label in the dataset.\"\"\"\n",
" wav_file_example = load_wav_16k_mono(wav_file_name)\n",
" label = Path(wav_file_name).parts[-1].split(\"_\")[0]\n",
" print(f\"Given label for this example: {label}\")\n",
" display.display(display.Audio(wav_file_example.numpy(), rate=audio_rate))\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import torch\n",
"import torchaudio\n",
"from pathlib import Path\n",
"from IPython import display\n",
"\n",
"# Utility function for loading audio files and making sure the sample rate is correct.\n",
"def load_wav_16k_mono(filename):\n",
" \"\"\"Load a WAV file, convert it to a float tensor, resample to 16 kHz single-channel audio.\"\"\"\n",
" # Load audio file with torchaudio\n",
" waveform, sample_rate = torchaudio.load(filename)\n",
" \n",
" # Convert to mono if stereo\n",
" if waveform.shape[0] > 1:\n",
" waveform = torch.mean(waveform, dim=0, keepdim=True)\n",
" \n",
" # Resample to 16kHz if needed\n",
" if sample_rate != 16000:\n",
" resampler = torchaudio.transforms.Resample(sample_rate, 16000)\n",
" waveform = resampler(waveform)\n",
" \n",
" return waveform.squeeze()\n",
"\n",
"\n",
"def display_example(wav_file_name, audio_rate=16000):\n",
" \"\"\"Allows us to listen to any wav file and displays its given label in the dataset.\"\"\"\n",
" wav_file_example = load_wav_16k_mono(wav_file_name)\n",
" label = Path(wav_file_name).parts[-1].split(\"_\")[0]\n",
" print(f\"Given label for this example: {label}\")\n",
" display.display(display.Audio(wav_file_example.numpy(), rate=audio_rate))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2bLlDRI6hzon"
},
"source": [
"Click the play button below to listen to this example .wav file. Feel free to change the `wav_file_name_example` variable below to listen to other audio clips in the dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "dLBvUZLlII5w",
"outputId": "c6a4917f-4a82-4a89-9193-415072e45550"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/7_jackson_43.wav\" # change this to hear other examples\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-QvbZA7yHwkh"
},
"source": [
"## 3. Use pre-trained SpeechBrain model to featurize audio\n",
"\n",
"The [SpeechBrain](https://github.com/speechbrain/speechbrain) package offers many Pytorch neural networks that have been pretrained for speech recognition tasks. Here we instantiate an audio feature extractor using SpeechBrain's `EncoderClassifier`. We'll use the \"spkrec-xvect-voxceleb\" network which has been pre-trained on the [VoxCeleb](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/) speech dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vL9lkiKsHvKr"
},
"outputs": [],
"source": [
"%%capture\n",
"\n",
"from speechbrain.pretrained import EncoderClassifier\n",
"\n",
"feature_extractor = EncoderClassifier.from_hparams(\n",
" \"speechbrain/spkrec-xvect-voxceleb\",\n",
" # run_opts={\"device\":\"cuda\"} # Uncomment this to run on GPU if you have one (optional)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vXlE6IK4ibcr"
},
"source": [
"Next, we run the audio clips through the pre-trained model to extract vector features (aka embeddings)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 143
},
"id": "obQYDKdLiUU6",
"outputId": "4e923d5c-2cf4-4a5c-827b-0a4fea9d87e4"
},
"outputs": [],
"source": [
"# Create dataframe with .wav file names\n",
"df = pd.DataFrame(file_paths, columns=[\"wav_audio_file_path\"])\n",
"df[\"label\"] = df.wav_audio_file_path.map(lambda x: int(Path(x).parts[-1].split(\"_\")[0]))\n",
"df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I8JqhOZgi94g"
},
"outputs": [],
"source": [
"def extract_audio_embeddings(model, wav_audio_file_path: str) -> tuple:\n",
" \"\"\"Feature extractor that embeds audio into a vector.\"\"\"\n",
" signal, fs = torchaudio.load(wav_audio_file_path) # Load audio signal as tensor\n",
" embeddings = model.encode_batch(\n",
" signal\n",
" ) # Pass tensor through pretrained neural net and extract representation\n",
" return embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2FSQ2GR9R_YA"
},
"outputs": [],
"source": [
"# Extract audio embeddings\n",
"embeddings_list = []\n",
"for i, file_name in enumerate(df.wav_audio_file_path): # for each .wav file name\n",
" embeddings = extract_audio_embeddings(feature_extractor, file_name)\n",
" embeddings_list.append(embeddings.cpu().numpy())\n",
"\n",
"embeddings_array = np.squeeze(np.array(embeddings_list))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dELkcdXgjTn_"
},
"source": [
"Now we have our features in a 2D numpy array. Each row in the array corresponds to an audio clip. We're now able to represent each audio clip as a 512-dimensional feature vector!\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kAkY31IVXyr8",
"outputId": "fd70d8d6-2f11-48d5-ae9c-a8c97d453632"
},
"outputs": [],
"source": [
"print(embeddings_array)\n",
"print(\"Shape of array: \", embeddings_array.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o4RBcaARmfVG"
},
"source": [
"## 4. Fit linear model and compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y9BIVyI9kHa4"
},
"source": [
"A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted network embeddings.\n",
"\n",
"To identify label issues, cleanlab requires a probabilistic prediction from your model for every datapoint that should be considered. However these predictions will be _overfit_ (and thus unreliable) for datapoints the model was previously trained on. cleanlab is intended to only be used with **out-of-sample** predicted probabilities, i.e. on datapoints held-out from the model during the training.\n",
"\n",
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset, by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. An additional benefit of cross-validation is that it provides more reliable evaluation of our model than a single training/validation split. We can obtain cross-validated out-of-sample predicted probabilities from any classifier via the [cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html) wrapper provided in scikit-learn.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "i_drkY9YOcw4"
},
"outputs": [],
"source": [
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"model = LogisticRegression(C=0.01, max_iter=1000, tol=1e-2, random_state=SEED)\n",
"\n",
"num_crossval_folds = 5 # can decrease this value to reduce runtime, or increase it to get better results\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=embeddings_array, y=df.label.values, cv=num_crossval_folds, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FW1yI9Ryrfkj"
},
"source": [
"For each audio clip, the corresponding predicted probabilities in `pred_probs` are produced by a copy of our `LogisticRegression` model that has never been trained on this audio clip. Hence we call these predictions _out-of-sample_. An additional benefit of cross-validation is that it provides more reliable evaluation of our model than a single training/validation split.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "_b-AQeoXOc7q",
"outputId": "15ae534a-f517-4906-b177-ca91931a8954"
},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score\n",
"\n",
"predicted_labels = pred_probs.argmax(axis=1)\n",
"cv_accuracy = accuracy_score(df.label.values, predicted_labels)\n",
"print(f\"Cross-validated estimate of accuracy on held-out data: {cv_accuracy}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SPz8WBwIlxUE"
},
"source": [
"## 5. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "laui-jXMm6qR"
},
"source": [
"Based on the given labels, out-of-sample predicted probabilities and features, cleanlab can quickly help us identify label issues in our dataset. For a dataset with N examples from K classes, the labels should be a 1D array of length N and predicted probabilities should be a 2D (N x K) array. \n",
"\n",
"Here, we use cleanlab to find potential label errors in our data. `Datalab` has several ways of loading the data. In this case, we can just pass the DataFrame created above to instantiate the object. We will then pass in the predicted probabilites to the `find_issues()` method so that Datalab can use them to find potential label errors in our data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(df, label_name=\"label\")\n",
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\":{}})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can view the results of running Datalab by calling the `report` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We observe from the report that cleanlab has found some label issues in our dataset. Let us investigate these examples further.\n",
"\n",
"We can view the more details about the label quality for each example using the `get_issues` method, specifying `label` as the issue type."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"label_issues.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled).\n",
"\n",
"We can then filter for the examples that have been identified as a label error:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identified_label_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = identified_label_issues.sort_values(\"label_score\").index\n",
"\n",
"print(f\"Here are indices of the most likely errors: \\n {lowest_quality_labels.values}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iI07jQ0BnTgt"
},
"source": [
"These examples flagged by cleanlab are those worth inspecting more closely."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 237
},
"id": "FQwRHgbclpsO",
"outputId": "fee5c335-c00e-4fcc-f22b-718705e93182"
},
"outputs": [],
"source": [
"df.iloc[lowest_quality_labels]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PsDmd5WDnZJG"
},
"source": [
"Let's listen to some audio clips below of label issues that were identified in this list.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p9jLn3Lp85rU"
},
"source": [
"In this example, the given label is **6** but it sounds like **8**.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "ff1NFVlDoysO",
"outputId": "8141a036-44c1-4349-c338-880432513e37"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_14.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HwokyN0bfVsn"
},
"source": [
"In the three examples below, the given label is **6** but they sound quite ambiguous.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "GZgovGkdiaiP",
"outputId": "d76b2ccf-8be2-4f3a-df4c-2c5c99150db7"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_36.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "lfa2eHbMwG8R",
"outputId": "6627ebe2-d439-4bf5-e2cb-44f6278ae86c"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_35.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_nicolas_8.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-rf8iSngtV83"
},
"source": [
"You can see that even widely-used datasets like Spoken Digit contain problematic labels. Never blindly trust your data! You should always check it for potential issues, many of which can be easily identified by cleanlab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"highlighted_indices = [1946, 516, 469, 2132] # verify these examples were found in find_label_issues\n",
"if not all(x in lowest_quality_labels for x in highlighted_indices):\n",
" raise Exception(\"Some highlighted examples are missing from label_issues_indices.\")"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "audio_quickstart_tutorial_deterministic.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/datalab_advanced.ipynb
+812
View File
@@ -0,0 +1,812 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datalab: Advanced workflows to audit your data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cleanlab offers a `Datalab` object to identify various issues in your machine learning datasets that may negatively impact models if not addressed. By default, `Datalab` can help you identify noisy labels, outliers, (near) duplicates, and other types of problems that commonly occur in real-world data.\n",
"\n",
"`Datalab` performs these checks by utilizing the (probabilistic) predictions from *any* ML model that has already been trained or its learned representations of the data. Underneath the hood, this class calls all the appropriate cleanlab methods for your dataset and provided model outputs, in order to best audit the data and alert you of important issues. This makes it easy to apply many functionalities of this library all within a single line of code. \n",
"\n",
"**This tutorial will demonstrate some advanced functionalities of Datalab including:**\n",
"\n",
"- Incremental issue search\n",
"- Specifying nondefault arguments to issue checks\n",
"- Save and load Datalab objects\n",
"- Adding a custom IssueManager\n",
"\n",
"If you are new to `Datalab`, check out this [quickstart tutorial](datalab_quickstart.html) for a 5-min introduction!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you have some `features` as well? Run the code below to examine your dataset for multiple types of issues.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(features=your_feature_matrix, pred_probs=your_pred_probs)\n",
"\n",
"lab.report()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Install and import required dependencies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` has additional dependencies that are not included in the standard installation of cleanlab.\n",
"\n",
"You can use pip to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib \n",
"!pip install \"cleanlab[datalab]\"\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"] # TODO: make sure this list is updated\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create and load the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll load a toy classification dataset for this tutorial. The dataset has two numerical features and a label column with three classes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code for data generation. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate = create_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We make a scatter plot of the features, with a color corresponding to the observed labels. Incorrect given labels are highlighted in red if they do not match the true label, outliers highlighted with an a black cross, and duplicates highlighted with a cyan cross."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(8, 6.5))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-3.5, 9.0)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.75, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.75, 0.45], title=\"Type of Issue\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In real-world scenarios, you won't know the true labels or the distribution of the features, so we won't use these in this tutorial, except for evaluation purposes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get out-of-sample predicted probabilities from a classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To detect certain types of issues in classification data (e.g. label errors), `Datalab` relies on predicted class probabilities from a trained model. Ideally, the prediction for each example should be out-of-sample (to avoid overfitting), coming from a copy of the model that was not trained on this example. \n",
"\n",
"This tutorial uses a simple logistic regression model \n",
"and the `cross_val_predict()` function from scikit-learn to generate out-of-sample predicted class probabilities for every example in the training set. You can replace this with *any* other classifier model and train it with cross-validation to get out-of-sample predictions.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=X_train, y=noisy_labels, cv=5, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Instantiate Datalab object"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we instantiate the Datalab object that will be used in the remainder in the tutorial by passing in the data created above.\n",
"\n",
"`Datalab` has several ways of loading the data. In this case, we'll simply wrap the training features and noisy labels in a dictionary so that we can pass it to `Datalab`.\n",
"\n",
"Other supported data formats for `Datalab` include: [HuggingFace Datasets](https://huggingface.co/docs/datasets/index) and [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). `Datalab` works across most data modalities (image, text, tabular, audio, etc). It is intended to find issues that commonly occur in datasets for which you have trained a supervised ML model, regardless of the type of data.\n",
"\n",
"Currently, pandas DataFrames that contain categorical columns might cause some issues when instantiating the `Datalab` object, so it is recommended to ensure that your DataFrame does not contain any categorical columns, or use other data formats (eg. python dictionary, HuggingFace Datasets) to pass in your data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_train, \"y\": noisy_labels}\n",
"\n",
"lab = Datalab(data, label_name=\"y\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 1**: Incremental issue search "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can call `find_issues` multiple times on a `Datalab` object to detect issues one type at a time.\n",
"\n",
"This is done via the `issue_types` argument which accepts a dictionary of issue types and any corresponding keyword arguments to specify nondefault keyword arguments to use for detecting each type of issues. In this first call, we only want to detect label issues, which are detected solely based on `pred_probs`, hence there is no need for us to pass in `features` here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\": {}}) \n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can check for additional types of issues with the same `Datalab`. Here, we would like to detect outliers and near duplicates which both utilize the features of the data.\n",
"\n",
"Notice that this second call to `find_issues()` updates the output of `report()`, we can see the existing label issues detected alongside the new issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(features=data[\"X\"], issue_types={\"outlier\": {}, \"near_duplicate\": {}})\n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 2**: Specifying nondefault arguments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also overwrite previously-executed checks for a type of issue. Here we re-run the detection of outliers, but specify that different non-default settings should be used (in this case, the number of neighbors `k` compared against to determine which datapoints are outliers). \n",
"The results from this new detection will replace the original outlier detection results in the updated `Datalab`. You could similarly specify non-default settings for other issue types in the first call to `Datalab.find_issues()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(features=data[\"X\"], issue_types={\"outlier\": {\"k\": 30}})\n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also increase the verbosity of the `report` to see additional information about the data issues and control how many top-ranked examples are shown for each issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.report(num_examples=10, verbosity=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how the number of flagged outlier issues has changed after specfying different settings to use for outlier detection."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 3**: Save and load Datalab objects"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A `Datalab` can be saved to a folder at a specified path. In a future Python process, this path can be used to load the `Datalab` from file back into memory. Your dataset is not saved as part of this process, so you'll need to save/load it separately to keep working with it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path = \"datalab-files\"\n",
"lab.save(path, force=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can load a `Datalab` object we have on file and view the previously detected issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"new_lab = Datalab.load(path)\n",
"new_lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 4**: Adding a custom IssueManager"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` detects pre-defined types of issues for you in one line of code: `find_issues()`. What if you want to check for other custom types of issues along with these pre-defined types, all within the same line of code?\n",
"\n",
"All issue types in `Datalab` are subclasses of cleanlab's `IssueManager` class.\n",
"To register a custom issue type for use with `Datalab`, simply also make it a subclass of `IssueManager`.\n",
"\n",
"The necessary members to implement in the subclass are:\n",
"\n",
"- A class variable called `issue_name` that acts as a unique identifier for the type of issue.\n",
"- An instance method called `find_issues` that:\n",
" - Computes a quality score for each example in the dataset (between 0-1), in terms of how *unlikely* it is to be an issue.\n",
" - Flags each example as an issue or not (may be based on thresholding the quality scores).\n",
" - Combine these in a dataframe that is assigned to an `issues` attribute of the `IssueManager`.\n",
" - Define a summary score for the overall quality of entire dataset, in terms of this type of issue. Set this score as part of the `summary` attribute of the `IssueManager`.\n",
" \n",
"To demonstrate this, we create an arbitrary issue type that checks the divisibility of an example's index in the dataset by 13."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.datalab.internal.issue_manager import IssueManager\n",
"from cleanlab.datalab.internal.issue_manager_factory import register\n",
"\n",
"\n",
"def scoring_function(idx: int, div: int = 13) -> float:\n",
" if idx == 0:\n",
" # Zero excluded from the divisibility check, gets the highest score\n",
" return 1\n",
" rem = idx % div\n",
" inv_scale = idx // div\n",
" if rem == 0:\n",
" return 0.5 * (1 - np.exp(-0.1*(inv_scale-1)))\n",
" else:\n",
" return 1 - 0.49 * (1 - np.exp(-inv_scale**0.5))*rem/div\n",
"\n",
"\n",
"@register # register this issue type for use with Datalab\n",
"class SuperstitionIssueManager(IssueManager):\n",
" \"\"\"A custom issue manager that keeps track of issue indices that\n",
" are divisible by 13.\n",
" \"\"\"\n",
" description: str = \"Examples with indices that are divisible by 13 may be unlucky.\" # Optional\n",
" issue_name: str = \"superstition\"\n",
"\n",
" def find_issues(self, div=13, **_) -> None:\n",
" ids = self.datalab.issues.index.to_series()\n",
" issues_mask = ids.apply(lambda idx: idx % div == 0 and idx != 0)\n",
" scores = ids.apply(lambda idx: scoring_function(idx, div))\n",
" self.issues = pd.DataFrame(\n",
" {\n",
" f\"is_{self.issue_name}_issue\": issues_mask,\n",
" self.issue_score_key: scores,\n",
" },\n",
" )\n",
" summary_score = 1 - sum(issues_mask) / len(issues_mask)\n",
" self.summary = self.make_summary(score = summary_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once registered, this `IssueManager` will perform custom issue checks when `find_issues` is called on a `Datalab` instance.\n",
"\n",
"As our `Datalab` instance here already has results from the outlier and near duplicate checks, we perform the custom issue check separately."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(issue_types={\"superstition\": {}})\n",
"lab.report()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"vscode": {
"interpreter": {
"hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/datalab_quickstart.ipynb
+843
View File
@@ -0,0 +1,843 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datalab: A unified audit to detect all kinds of issues in data and labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cleanlab offers a `Datalab` object that can identify various issues in your machine learning datasets, such as noisy labels, outliers, (near) duplicates, drift, and other types of problems common in real-world data. These data issues may negatively impact models if not addressed. `Datalab` utilizes *any* ML model you have already trained for your data to diagnose these issues, it only requires access to either: (probabilistic) predictions from your model or its learned representations of the data.\n",
"\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Compute out-of-sample predicted probabilities for a sample dataset using cross-validation.\n",
"- Use `Datalab` to identify issues such as noisy labels, outliers, (near) duplicates, and other types of problems \n",
"- View the issue summaries and other information about our sample dataset\n",
"\n",
"You can easily replace our demo dataset with your own image/text/tabular/audio/etc dataset, and then run the same code to discover what sort of issues lurk within it!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you also have some numeric `features` (or model embeddings of data)? Run the code below to examine your dataset for multiple types of issues.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(features=your_feature_matrix, pred_probs=your_pred_probs)\n",
"\n",
"lab.report()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install and import required dependencies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` has additional dependencies that are not included in the standard installation of cleanlab.\n",
"\n",
"You can use pip to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib\n",
"!pip install \"cleanlab[datalab]\"\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"] # TODO: make sure this list is updated\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Create and load the data (can skip these details)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll load a toy classification dataset for this tutorial. The dataset has two numerical features and a label column with three possible classes. Each example is classified as either: *low*, *mid* or *high*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code for data generation. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
" # Assign few datapoints to rare class\n",
" random_idx = np.random.randint(0, X_train.shape[0], 3)\n",
" noisy_labels[random_idx] = \"max\"\n",
" noisy_labels_idx[random_idx] = np.max(y_bin_idx) + 1\n",
" \n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate = create_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We make a scatter plot of the features, with a color corresponding to the observed labels. Incorrect given labels are highlighted in red if they do not match the true label, outliers highlighted with an a black cross, and duplicates highlighted with a cyan cross."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(8, 6.5))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-3.5, 9.0)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.75, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.75, 0.45], title=\"Type of Issue\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In real-world scenarios, you won't know the true labels or the distribution of the features, so we won't use these in this tutorial, except for evaluation purposes.\n",
"\n",
"\n",
"\n",
"`Datalab` has several ways of loading the data.\n",
"In this case, we'll simply wrap the training features and noisy labels in a dictionary so that we can pass it to `Datalab`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_train, \"y\": noisy_labels}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other supported data formats for `Datalab` include: [HuggingFace Datasets](https://huggingface.co/docs/datasets/index) and [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). `Datalab` works across most data modalities (image, text, tabular, audio, etc). It is intended to find issues that commonly occur in datasets for which you have trained a supervised ML model, regardless of the type of data.\n",
"\n",
"Currently, pandas DataFrames that contain categorical columns might cause some issues when instantiating the `Datalab` object, so it is recommended to ensure that your DataFrame does not contain any categorical columns, or use other data formats (eg. python dictionary, HuggingFace Datasets) to pass in your data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Get out-of-sample predicted probabilities from a classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To detect certain types of issues in classification data (e.g. label errors), `Datalab` relies on predicted class probabilities from a trained model. Ideally, the prediction for each example should be out-of-sample (to avoid overfitting), coming from a copy of the model that was not trained on this example. \n",
"\n",
"This tutorial uses a simple logistic regression model \n",
"and the `cross_val_predict()` function from scikit-learn to generate out-of-sample predicted class probabilities for every example in the training set. You can replace this with *any* other classifier model and train it with cross-validation to get out-of-sample predictions.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=data[\"X\"], y=data[\"y\"], cv=5, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use Datalab to find issues in the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We create a `Datalab` object from the dataset, also providing the name of the label column in the dataset. Only instantiate one `Datalab` object per dataset, and note that only classification datasets are supported for now.\n",
"\n",
"All that is need to audit your data is to call `find_issues()`.\n",
"This method accepts various inputs like: predicted class probabilities, numeric feature representations of the data. The more information you provide here, the more thoroughly `Datalab` will audit your data! Note that `features` should be some numeric representation of each example, either obtained through preprocessing transformation of your raw data or embeddings from a (pre)trained model. In this case, our data is already entirely numeric so we just provide the features directly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(data, label_name=\"y\")\n",
"lab.find_issues(pred_probs=pred_probs, features=data[\"X\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's review the results of this audit using `report()`.\n",
"This provides a high-level summary of each type of issue found in the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Learn more about the issues in your dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Datalab detects all sorts of issues in a dataset and what to do with the findings will vary case-by-case.\n",
"\n",
"To conceptually understand how each type of issue is defined and what it means if detected in your data, check out the [Issue Type Descriptions](../../cleanlab/datalab/guide/issue_type_description.html) page. The [Datalab Issue Types](https://docs.cleanlab.ai/stable/cleanlab/datalab/guide/issue_type_description.html) page also lists additional types of issues that `Datalab.find_issues()` can detect, as well as optional parameters you can specify for greater control over how your data are checked.\n",
"\n",
"Datalab offers several methods to understand more details about a particular issue in your dataset.\n",
"The `get_issue_summary()` method fetches summary statistics regarding how severe each type of issue is overall across the whole dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issue_summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the returned summary DataFrame: LOWER `score` values indicate types of issues that are MORE severe *overall* across the dataset (lower-quality data in terms of this issue), HIGHER `num_issues` values indicate types of issues that are MORE severe *overall* across the dataset (more datapoints appear to exhibit this issue).\n",
"\n",
"We can also only request the summary for a particular type of issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issue_summary(\"label\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `get_issues()` method returns information for each *individual example* in the dataset including: whether or not it is plagued by this issue (Boolean), as well as a *quality score* (numeric value betweeen 0 to 1) quantifying how severe this issue appears to be for this particular example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issues().head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each example receives a separate *quality score* for each issue type (eg. `outlier_score` is the *quality score* for the `outlier` issue type, quantifying *how typical* each datapoint appears to be). LOWER scores indicate MORE severe instances of the issue, so the most-concerning datapoints have the lowest quality scores. Sort by these scores to see the most-concerning examples in your dataset for each type of issue. The quality scores are directly comparable between examples/datasets, but not across different issue types.\n",
"\n",
"Similar to above, we can pass the type of issue as a argument to `get_issues()` to get the information for one particular type of issue.\n",
"As an example, let's see the examples identified as having the most severe *label* issues:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"examples_w_issue = (\n",
" lab.get_issues(\"label\")\n",
" .query(\"is_label_issue\")\n",
" .sort_values(\"label_score\")\n",
")\n",
"\n",
"examples_w_issue.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inspecting the labels for some of these top-ranked examples, we find their given label was indeed incorrect."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Get additional information \n",
"\n",
"Miscellaneous additional information (statistics, intermediate results, etc) related to a particular issue type can be accessed via `get_info(issue_name)`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"label_issues_info = lab.get_info(\"label\")\n",
"label_issues_info[\"classes_by_label_quality\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This portion of the info shows overall label quality summaries of all examples annotated as a particular class (e.g. the `Label Issues` column is the estimated number of examples labeled as this class that should actually have a different label).\n",
"To learn more about this, see the documentation for the [cleanlab.dataset.rank_classes_by_label_quality](../../cleanlab/dataset.html#cleanlab.dataset.rank_classes_by_label_quality)\n",
"method.\n",
"\n",
"You can view all sorts of information regarding your dataset using the `get_info()` method with no arguments passed. This is not printed here as it returns a huge dictionary but feel free to check it out yourself! Don't worry if you don't understand all of the miscellaneous information in this `info` dictionary, none of it is critical to diagnose the issues in your dataset. Understanding miscellaneous info may require reading the documentation of the miscellaneous cleanlab functions which computed it.\n",
"\n",
"#### Near duplicate issues \n",
"\n",
"Let's also inspect the examples flagged as (near) duplicates.\n",
"For each such example, the `near_duplicate_sets` column below indicates *which* other examples in the dataset are highly similar to it (this value is empty for examples not flagged as nearly duplicated). The `near_duplicate_score` quantifies *how similar* each example is to its nearest neighbor in the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issues(\"near_duplicate\").query(\"is_near_duplicate_issue\").sort_values(\"near_duplicate_score\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?). \n",
"\n",
"Other issues detected in this tutorial dataset include **outliers** and **class imbalance**, see the [Issue Type Descriptions](../../cleanlab/datalab/guide/issue_type_description.html) for more information. `Datalab` makes it very easy to check your datasets for all sorts of issues that are important to deal with for training robust models. The inputs it uses to detect issues can come from *any* model you have trained (the better your model, the more accurate the issue detection will be).\n",
"\n",
"To learn more, check out this [example notebook](https://github.com/cleanlab/examples/blob/master/datalab_image_classification/datalab.ipynb) (demonstrates Datalab applied to a real dataset) and the [advanced Datalab tutorial](datalab_advanced.html) (demonstrates configuration and customization options to exert greater control)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"def precision_at_k(predicted_indices, true_indices, k):\n",
" return len(set(predicted_indices[:k]).intersection(set(true_indices))) / k\n",
"\n",
"def recall_at_k(predicted_indices, true_indices, k):\n",
" return len(set(predicted_indices[:k]).intersection(set(true_indices))) / len(true_indices)\n",
"\n",
"def jaccard_similarity(l1, l2):\n",
" s1 = set(l1)\n",
" s2 = set(l2)\n",
" intersect_set = s1.intersection(s2)\n",
" union_set = s1.union(s2)\n",
" if len(intersect_set) == 0:\n",
" return 0\n",
" return len(intersect_set) / len(union_set)\n",
"\n",
"label_issues = lab.get_issues(\"label\")\n",
"predicted_label_issues_indices = (\n",
" label_issues.query(\"is_label_issue\").sort_values(\"label_score\").index.to_list()\n",
")\n",
"predicted_label_issues_indices_by_score = (\n",
" label_issues.sort_values(\"label_score\").index.to_list()\n",
")\n",
"label_issue_indices = np.where(y_train_idx != noisy_labels_idx)[0]\n",
"\n",
"label_quality_scores = label_issues[\"label_score\"].tolist()\n",
"Z = (y_train_idx == noisy_labels_idx).astype(float).tolist()\n",
"\n",
"predicted_outlier_issues_indices = (\n",
" lab.get_issues(\"outlier\").query(\"is_outlier_issue\").index.to_list()\n",
")\n",
"outlier_issue_indices = list(range(125, 130+1))\n",
"exact_duplicate_idx = [index for index, elem in enumerate(X_train) if (elem == X_duplicate).all()][0]\n",
"if exact_duplicate_idx >= 125: # if the random index selected to create a duplicate >= 125, then the last point is also an outlier\n",
" outlier_issue_indices.append(131)\n",
"\n",
"predicted_duplicate_issues_indices = (\n",
" lab.get_issues(\"near_duplicate\").query(\"is_near_duplicate_issue\").index.tolist()\n",
")\n",
"duplicate_issue_indices = [exact_duplicate_idx, 129, 130, 131]\n",
"\n",
"k = len(label_issue_indices)\n",
"assert precision_at_k(predicted_label_issues_indices, label_issue_indices, k) >= 0.75\n",
"assert recall_at_k(predicted_label_issues_indices, label_issue_indices, k) >= 0.75\n",
"assert precision_at_k(predicted_label_issues_indices_by_score, label_issue_indices, k) == 1.0\n",
"assert recall_at_k(predicted_label_issues_indices_by_score, label_issue_indices, k) == 1.0\n",
"assert roc_auc_score(Z, label_quality_scores) > 0.9\n",
"\n",
"assert jaccard_similarity(predicted_outlier_issues_indices, outlier_issue_indices) > 0.9\n",
"assert jaccard_similarity(predicted_duplicate_issues_indices, duplicate_issue_indices) > 0.9\n",
"\n",
"expected_issue_types = set([\"label\", \"outlier\", \"near_duplicate\", \"class_imbalance\"])\n",
"detected_issue_types = set(lab.get_issue_summary()[lab.get_issue_summary()[\"num_issues\"] > 0][\"issue_type\"])\n",
"assert detected_issue_types == expected_issue_types"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
},
"vscode": {
"interpreter": {
"hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/image.ipynb
+1319
View File
File diff suppressed because it is too large Load Diff
docs/source/tutorials/datalab/index.rst
+14
View File
@@ -0,0 +1,14 @@
Datalab Tutorials
=================
.. toctree::
:maxdepth: 1
Detecting Common Data Issues with Datalab <datalab_quickstart>
Advanced Data Auditing with Datalab <datalab_advanced>
Text Data <text>
Tabular Data (Numeric/Categorical) <tabular>
Image Data <image>
Audio Data <audio>
Guide to Datalab Issue Types </cleanlab/datalab/guide/issue_type_description>
Miscellaneous workflows with Datalab <workflows>
docs/source/tutorials/datalab/tabular.ipynb
+548
View File
@@ -0,0 +1,548 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Detecting Issues in Tabular Data (Numeric/Categorical columns) with Datalab\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use Datalab to detect various issues in a classification dataset with tabular (numeric/categorical) features. Tabular (or *structured*) data are typically organized in a row/column format and stored in a SQL database or file types like: CSV, Excel, or Parquet. Here we consider a Student Grades dataset, which contains over 900 individuals who have three exam grades and some optional notes, each being assigned a letter grade (their class label). cleanlab automatically identifies _hundreds_ of examples in this dataset that were mislabeled with the incorrect final grade selected. You can run the same code from this tutorial to detect incorrect information in your own tabular classification datasets.\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Train a classifier model (here scikit-learn's HistGradientBoostingClassifier, although any model could be used) and use this classifier to compute (out-of-sample) predicted class probabilities via cross-validation.\n",
"\n",
"- Create a K nearest neighbours (KNN) graph between the examples in the dataset.\n",
"\n",
"- Identify issues in the dataset with cleanlab's `Datalab` audit applied to the predictions and KNN graph.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on your original data labels? Have a `knn_graph` computed between dataset examples (reflecting similarity in their feature values)? Run the code below to find issues in your dataset.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, knn_graph=knn_graph)\n",
"\n",
"lab.get_issues()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.model_selection import cross_val_predict\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.ensemble import HistGradientBoostingClassifier\n",
"from sklearn.neighbors import NearestNeighbors\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"SEED = 100 # for reproducibility\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and process the data\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We first load the data features and labels (which are possibly noisy).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"grades_data = pd.read_csv(\"https://s.cleanlab.ai/grades-tabular-demo-v2.csv\")\n",
"grades_data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_raw = grades_data[[\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]]\n",
"labels = grades_data[\"letter_grade\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we preprocess the data. Here we apply one-hot encoding to columns with categorical values and standardize the values in numeric columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cat_features = [\"notes\"]\n",
"X_encoded = pd.get_dummies(X_raw, columns=cat_features, drop_first=True)\n",
"\n",
"numeric_features = [\"exam_1\", \"exam_2\", \"exam_3\"]\n",
"scaler = StandardScaler()\n",
"X_processed = X_encoded.copy()\n",
"X_processed[numeric_features] = scaler.fit_transform(X_encoded[numeric_features])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"Assign your data's features to variable `X` and its labels to variable `labels` instead.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Select a classification model and compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we use a simple histogram-based gradient boosting model (similar to XGBoost), but you can choose any suitable scikit-learn model for this tutorial.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"clf = HistGradientBoostingClassifier()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find potential labeling errors, cleanlab requires a probabilistic prediction from your model for every datapoint. However, these predictions will be _overfitted_ (and thus unreliable) for examples the model was previously trained on. For the best results, cleanlab should be applied with **out-of-sample** predicted class probabilities, i.e., on examples held out from the model during the training.\n",
"\n",
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n",
"We can implement this via the `cross_val_predict` method from scikit-learn.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_crossval_folds = 5 \n",
"pred_probs = cross_val_predict(\n",
" clf,\n",
" X_processed,\n",
" labels,\n",
" cv=num_crossval_folds,\n",
" method=\"predict_proba\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Construct K nearest neighbours graph"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The KNN graph reflects how close each example is when compared to other examples in our dataset (in the numerical space of preprocessed feature values). This similarity information is used by Datalab to identify issues like outliers in our data. For tabular data, think carefully about the most appropriate way to define the similarity between two examples.\n",
"\n",
"Here we use the `NearestNeighbors` class in sklearn to easily compute this graph (with similarity defined by the Euclidean distance between feature values). The graph should be represented as a sparse matrix with nonzero entries indicating nearest neighbors of each example and their distance."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"KNN = NearestNeighbors(metric='euclidean')\n",
"KNN.fit(X_processed.values)\n",
"\n",
"knn_graph = KNN.kneighbors_graph(mode=\"distance\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the given labels, predicted probabilities, and KNN graph, cleanlab can quickly help us identify suspicious values in our grades table.\n",
"\n",
"We use cleanlab's `Datalab` class which has several ways of loading the data. In this case, we’ll simply wrap the dataset (features and noisy labels) in a dictionary that is used instantiate a `Datalab` object such that it can audit our dataset for various types of issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_processed.values, \"y\": labels}\n",
"\n",
"lab = Datalab(data, label_name=\"y\")\n",
"lab.find_issues(pred_probs=pred_probs, knn_graph=knn_graph)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Label issues\n",
"\n",
"The above report shows that cleanlab identified many label issues in the data. We can see which examples are estimated to be mislabeled (as well as a numeric quality score quantifying how likely their label is correct) via the `get_issues` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"issue_results = lab.get_issues(\"label\")\n",
"issue_results.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To review the most severe label issues, sort the DataFrame above by the `label_score` column (a lower score represents that the label is less likely to be correct). \n",
"\n",
"Let's review some of the most likely label errors:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sorted_issues = issue_results.sort_values(\"label_score\").index\n",
"\n",
"X_raw.iloc[sorted_issues].assign(\n",
" given_label=labels.iloc[sorted_issues], \n",
" predicted_label=issue_results[\"predicted_label\"].iloc[sorted_issues]\n",
").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The dataframe above shows the original label (`given_label`) for examples that cleanlab finds most likely to be mislabeled, as well as an alternative `predicted_label` for each example.\n",
"\n",
"These examples have been labeled incorrectly and should be carefully re-examined - a student with grades of 89, 95 and 73 surely does not deserve a D! "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Outlier issues\n",
"\n",
"According to the report, our dataset contains some outliers. We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"outlier_results = lab.get_issues(\"outlier\")\n",
"sorted_outliers= outlier_results.sort_values(\"outlier_score\").index\n",
"\n",
"X_raw.iloc[sorted_outliers].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The student at index 3 has fractional exam scores, which is likely a error. We also see that the students at index 0 and 4 have numerical values in their notes section, which is also probably unintended. Lastly, we see that the student at index 8 has a html string in their notes section, definitely a mistake!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Near-duplicate issues\n",
"\n",
"According to the report, our dataset contains some sets of nearly duplicated examples.\n",
"We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the examples in our dataset that are most nearly duplicated."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"duplicate_results = lab.get_issues(\"near_duplicate\")\n",
"duplicate_results.sort_values(\"near_duplicate_score\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see some examples that cleanlab has flagged as being nearly duplicated. Let's view these examples to see how similar they are\n",
"\n",
"Using the one of the lowest-scoring examples, let's compare it against the identified near-duplicate sets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Identify the row with the lowest near_duplicate_score\n",
"lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].idxmin()\n",
"\n",
"# Extract the indices of the lowest scoring duplicate and its near duplicate sets\n",
"indices_to_display = [lowest_scoring_duplicate] + duplicate_results.loc[lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
"\n",
"# Display the relevant rows from the original dataset\n",
"X_raw.iloc[indices_to_display]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These examples are exact duplicates! Perhaps the same information was accidentally recorded multiple times in this data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, let's take a look at another example and the identified near-duplicate sets:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Identify the next row not in the previous near duplicate set\n",
"second_lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].drop(indices_to_display).idxmin()\n",
"\n",
"# Extract the indices of the second lowest scoring duplicate and its near duplicate sets\n",
"next_indices_to_display = [second_lowest_scoring_duplicate] + duplicate_results.loc[second_lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
"\n",
"# Display the relevant rows from the original dataset\n",
"X_raw.iloc[next_indices_to_display]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We identified another set of exact duplicates in our dataset! Including near/exact duplicates in a dataset may have unintended effects on models; be wary about splitting them across training/test sets. Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This tutorial highlighted a straightforward approach to detect potentially incorrect information in any tabular dataset. Just use Datalab with any ML model -- the better the model, the more accurate the data errors detected by Datalab will be!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"identified_label_issues = issue_results[issue_results[\"is_label_issue\"] == True]\n",
"label_issue_indices = [3, 723, 709, 886, 689] # check these examples were found in label issues\n",
"if not all(x in identified_label_issues.index for x in label_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n",
" \n",
"identified_outlier_issues = outlier_results[outlier_results[\"is_outlier_issue\"] == True]\n",
"outlier_issue_indices = [3, 7, 0, 4, 8] # check these examples were found in outlier issues\n",
"if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n",
" \n",
"identified_duplicate_issues = duplicate_results[duplicate_results[\"is_near_duplicate_issue\"] == True]\n",
"duplicate_issue_indices = [690, 246, 185, 582] # check these examples were found in duplicate issues\n",
"if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")\n",
" \n",
"# check that the near duplicates shown are actually flagged as near duplicate sets\n",
"if not duplicate_results.iloc[690][\"near_duplicate_sets\"] == 246:\n",
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
" \n",
"if not duplicate_results.iloc[185][\"near_duplicate_sets\"] == 582:\n",
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
"\n",
"# Function to check if all rows are identical\n",
"def are_rows_identical(df):\n",
" first_row = df.iloc[0]\n",
" return all(df.iloc[i].equals(first_row) for i in range(1, len(df)))\n",
"\n",
"# Test to ensure all displayed rows are identical\n",
"if not are_rows_identical(X_raw.iloc[indices_to_display]):\n",
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")\n",
"\n",
"# Repeat the test for the next set of indices\n",
"if not are_rows_identical(X_raw.iloc[next_indices_to_display]):\n",
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")"
]
}
],
"metadata": {
"interpreter": {
"hash": "cda20062bc42cfdcaa0f9720c0b28e880bba110e9dfce6c1689934eec9b595a1"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/text.ipynb
+591
View File
@@ -0,0 +1,591 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Detecting Issues in a Text Dataset with Datalab\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use Datalab to detect various issues in an intent classification dataset composed of (text) customer service requests at an online bank. We consider a subset of the [Banking77-OOS Dataset](https://arxiv.org/abs/2106.04564) containing 1,000 customer service requests which are classified into 10 categories based on their intent (you can run this same code on any text classification dataset). Cleanlab automatically identifies bad examples in our dataset, including mislabeled data, out-of-scope examples (outliers), or otherwise ambiguous examples. Consider filtering or correcting such bad examples\u00a0before you dive deep into modeling your data!\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Use a pretrained transformer model to extract the text embeddings from the customer service requests\n",
"\n",
"- Train a simple Logistic Regression model on the text embeddings to compute out-of-sample predicted probabilities\n",
"\n",
"- Run cleanlab's `Datalab` audit with these predictions and embeddings in order to identify problems like: label issues, outliers, and near duplicates in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you have some numeric `features` as well? Run the code below to find any potential label errors in your dataset.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, features=your_features)\n",
"\n",
"lab.report()\n",
"lab.get_issues()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install sentence-transformers\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs.cleanlab.ai).\n",
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
"# Package versions we used:scikit-learn==1.2.0 sentence-transformers>=2.3.0\n",
"\n",
"dependencies = [\"cleanlab\", \"sentence_transformers\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import re \n",
"import string \n",
"import pandas as pd \n",
"from sklearn.metrics import accuracy_score, log_loss \n",
"from sklearn.model_selection import cross_val_predict \n",
"from sklearn.linear_model import LogisticRegression\n",
"from sentence_transformers import SentenceTransformer\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai \n",
"\n",
"import random \n",
"import numpy as np \n",
"\n",
"pd.set_option(\"display.max_colwidth\", None) \n",
"\n",
"SEED = 123456 # for reproducibility\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and format the text dataset\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pd.read_csv(\"https://s.cleanlab.ai/banking-intent-classification.csv\")\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw_texts, labels = data[\"text\"].values, data[\"label\"].values\n",
"num_classes = len(set(labels))\n",
"\n",
"print(f\"This dataset has {num_classes} classes.\")\n",
"print(f\"Classes: {set(labels)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's view the i-th example in the dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"i = 1 # change this to view other examples from the dataset\n",
"print(f\"Example Label: {labels[i]}\")\n",
"print(f\"Example Text: {raw_texts[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data is stored as two numpy arrays:\n",
"\n",
"1. `raw_texts` stores the customer service requests utterances in text format\n",
"2. `labels` stores the intent categories (labels) for each example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"You can easily replace the above with your own text dataset, and continue with the rest of the tutorial.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we convert the text strings into vectors better suited as inputs for our ML models. \n",
"\n",
"We will use numeric representations from a pretrained Transformer model as embeddings of our text. The [Sentence Transformers](https://huggingface.co/docs/hub/sentence-transformers) library offers simple methods to compute these embeddings for text data. Here, we load the pretrained `electra-small-discriminator` model, and then run our data through network to extract a vector embedding\u00a0of each example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"transformer = SentenceTransformer('google/electra-small-discriminator')\n",
"text_embeddings = transformer.encode(raw_texts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our subsequent ML model will directly operate on elements of `text_embeddings` in order to classify the customer service requests."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Define a classification model and compute out-of-sample predicted probabilities"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted embeddings.\n",
"\n",
"To identify label issues, cleanlab requires a probabilistic prediction from your model for each datapoint. However these predictions will be _overfit_ (and thus unreliable) for datapoints the model was previously trained on. cleanlab is intended to only be used with **out-of-sample** predicted class probabilities, i.e. on datapoints held-out from the model during the training.\n",
"\n",
"Here we obtain out-of-sample predicted class probabilities for every example in our dataset using a Logistic Regression model with cross-validation.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"model = LogisticRegression(max_iter=400)\n",
"\n",
"pred_probs = cross_val_predict(model, text_embeddings, labels, method=\"predict_proba\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use cleanlab to find issues in your dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given feature embeddings and the (out-of-sample) predicted class probabilities obtained from any model\u00a0you have, cleanlab can quickly help you identify low-quality examples in your dataset.\n",
"\n",
"Here, we use cleanlab's `Datalab` to find issues in our data. Datalab offers several ways of loading the data; we\u2019ll simply wrap the training features and noisy labels in a dictionary. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_dict = {\"texts\": raw_texts, \"labels\": labels}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All that is need to audit your data is to call `find_issues()`. We pass in the predicted probabilities and the feature embeddings obtained above, but you do not necessarily need to provide all of this information depending on which types of issues you are interested in. The more inputs you provide, the more types of issues `Datalab` can detect in your data. Using a better model to produce these inputs will ensure cleanlab more accurately estimates issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab = Datalab(data_dict, label_name=\"labels\")\n",
"lab.find_issues(pred_probs=pred_probs, features=text_embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the audit is complete, review the findings using the `report` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Label issues\n",
"\n",
"The report indicates that cleanlab identified many label issues in our dataset. We can see which examples are flagged as likely mislabeled and the label quality score for each example using the `get_issues` method, specifying `label` as an argument to focus on label issues in the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"label_issues.head() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 5 most likely mislabeled examples in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identified_label_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = label_issues[\"label_score\"].argsort()[:5].to_numpy()\n",
"\n",
"print(\n",
" f\"cleanlab found {len(identified_label_issues)} potential label errors in the dataset.\\n\"\n",
" f\"Here are indices of the top 5 most likely errors: \\n {lowest_quality_labels}\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's review some of the most likely label errors. \n",
"\n",
"Here we display the top 5 examples identified as the most likely label errors in the dataset, together with their given (original) label and a suggested alternative label from cleanlab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_with_suggested_labels = pd.DataFrame(\n",
" {\"text\": raw_texts, \"given_label\": labels, \"suggested_label\": label_issues[\"predicted_label\"]}\n",
")\n",
"data_with_suggested_labels.iloc[lowest_quality_labels]"
]
},
{
"cell_type": "markdown",
"metadata": {
"scrolled": true
},
"source": [
"These are very clear label errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the intent of these requests, whoever produced this dataset made many mistakes that are important to address before modeling the data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Outlier issues\n",
"\n",
"According to the report, our dataset contains some outliers.\n",
"We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"outlier_issues = lab.get_issues(\"outlier\")\n",
"outlier_issues.sort_values(\"outlier_score\").head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lowest_quality_outliers = outlier_issues[\"outlier_score\"].argsort()[:5]\n",
"\n",
"data.iloc[lowest_quality_outliers]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that cleanlab has identified entries in this dataset that do not appear to be proper customer requests. Outliers in this dataset appear to be out-of-scope customer requests and other nonsensical text which does not make sense for intent classification. Carefully consider whether such outliers may detrimentally affect your data modeling, and consider removing them from the dataset if so."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Near-duplicate issues\n",
"\n",
"According to the report, our dataset contains some sets of nearly duplicated examples.\n",
"We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the text examples in our dataset that are most nearly duplicated."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"duplicate_issues = lab.get_issues(\"near_duplicate\")\n",
"duplicate_issues.sort_values(\"near_duplicate_score\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see that example 160 and 148 are nearly duplicated, as are example 546 and 514.\n",
"\n",
"Let's view these examples to see how similar they are."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.iloc[[160, 148]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.iloc[[546, 514]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that these two sets of request are indeed very similar to one another! Including near duplicates in a dataset may have unintended effects on models, and be wary about splitting them across training/test sets. Learn more about handling near duplicates in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Non-IID issues (data drift)\n",
"According to the report, our dataset does not appear to be Independent and Identically Distributed (IID). The overall non-iid score for the dataset (displayed below) corresponds to the `p-value` of a statistical test for whether the ordering of samples in the dataset appears related to the similarity between their feature values. A low `p-value` strongly suggests that the dataset violates the IID assumption, which is a key assumption required for conclusions (models) produced from the dataset to generalize to a larger population."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p_value = lab.get_info('non_iid')['p-value']\n",
"p_value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, our dataset was flagged as non-IID because the rows happened to be sorted by class label in the original data. This may be benign if we remember to shuffle rows before model training and data splitting. But if you don't know why your data was flagged as non-IID, then you should be worried about potential data drift or unexpected interactions between data points (their values may not be statistically independent). Think carefully about what future test data may look like (and whether your data is representative of the population you care about). You should not shuffle your data before the non-IID test runs (will invalidate its conclusions)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As demonstrated above, cleanlab can automatically shortlist the most likely issues in your dataset to help you better curate your dataset for subsequent modeling. With this shortlist, you can decide whether to fix these label issues or remove nonsensical or duplicated examples from your dataset to obtain a higher-quality dataset for training your next ML model. cleanlab's issue detection can be run with outputs from *any* type of model you initially trained.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"label_issue_indices = [981, 974, 982] # check these examples were found in label issues\n",
"if not all(x in identified_label_issues.index for x in label_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n",
" \n",
"identified_outlier_issues = outlier_issues[outlier_issues[\"is_outlier_issue\"] == True]\n",
"outlier_issue_indices = [994, 989, 999] # check these examples were found in duplicates\n",
"if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n",
"\n",
"identified_duplicate_issues = duplicate_issues[duplicate_issues[\"is_near_duplicate_issue\"] == True]\n",
"duplicate_issue_indices = [160, 148, 546, 514] # check these examples were found in duplicates\n",
"if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Text Classification with Datalab",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/workflows.ipynb
+1637
View File
File diff suppressed because it is too large Load Diff
docs/source/tutorials/dataset_health.ipynb
+313
View File
File diff suppressed because one or more lines are too long
docs/source/tutorials/faq.ipynb
+925
View File
@@ -0,0 +1,925 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ffe0d62e",
"metadata": {},
"source": [
"# FAQ\n",
"\n",
"Answers to frequently asked questions about the [cleanlab](https://github.com/cleanlab/cleanlab) open-source package.\n",
"\n",
"The code snippets in this FAQ come from a fully executable notebook you can run via Colab or locally by downloading it [here](https://github.com/cleanlab/cleanlab/blob/master/docs/source/tutorials/faq.ipynb).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2a4efdde",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden on docs.cleanlab.ai. Execute it to ensure all other cells below can be executed in your own notebook\n",
"\n",
"import os \n",
"import logging \n",
"import numpy as np \n",
"import sklearn \n",
"import cleanlab \n",
"\n",
"np.random.seed(123)\n",
"\n",
"# Toy dataset:\n",
"N = 50\n",
"K = 3\n",
"num_errors = 4\n",
"labels = np.random.randint(low=0, high=K, size=N)\n",
"pred_probs = np.random.random_sample(N*K).reshape((N,K))\n",
"pred_probs[np.arange(N),labels] += 4 # make pred_probs accurate\n",
"pred_probs = pred_probs/pred_probs.sum(axis=1)[:, np.newaxis]\n",
"data = np.array([[label+np.random.uniform(), label+np.random.uniform()] for label in labels])\n",
"# introduce label errors in last few examples:\n",
"og0_indices = labels[-num_errors:] == 0\n",
"labels[-num_errors:] = 0\n",
"labels[-num_errors:][og0_indices] = 1\n",
"\n",
"your_classifier=sklearn.linear_model.LogisticRegression() # toy classifier"
]
},
{
"cell_type": "markdown",
"id": "d504ec58",
"metadata": {},
"source": [
"### What data can cleanlab detect issues in?"
]
},
{
"cell_type": "markdown",
"id": "5e70efbc",
"metadata": {},
"source": [
"This package can be used to detect issues in any dataset for which you have trained a ML model. This includes datasets involving: multiple annotators per example (multi-annotator), or multiple labels per example (multi-label). This includes data from any modality such as: image, text, tabular, audio, etc. This package supports most common supervised learning tasks (entity recognition in text, image segmentation, object detection, tagging, regression, ...). If you have a particular task in mind, [let us know](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue)!"
]
},
{
"cell_type": "markdown",
"id": "eca36874",
"metadata": {},
"source": [
"### How do I format classification labels for cleanlab?"
]
},
{
"cell_type": "markdown",
"id": "38c50875",
"metadata": {},
"source": [
"**With Datalab**:\n",
"\n",
"Datalab simplifies label management by accepting both string and integer labels directly. Internally, unique labels are sorted alphanumerically and mapped to integers, facilitating seamless integration with lower-level cleanlab methods. Below are the supported label formats:\n",
"\n",
"- **List of strings or integers**: Directly pass labels as a list of strings or integers without manual encoding.\n",
"\n",
"- **Using** `datasets.Dataset` **with** `ClassLabel`: For advanced use cases, you can structure your dataset using HuggingFace's `datasets.Dataset` object, specifying label columns as `ClassLabel` feature objects for formatting the labels. Refer to the [datasets documentation](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.ClassLabel) for detailed guidance.\n",
"\n",
"```python\n",
"from cleanlab import Datalab\n",
"from datasets import Dataset, Features, Value, ClassLabel\n",
"\n",
"# Example 1: Labels as a list of strings\n",
"labels_str = ['cat', 'dog', 'cat', 'dog']\n",
"datalab_str = Datalab(data={\"text\": [\"a\", \"b\", \"c\", \"d\"], \"label\": labels_str}, label_name=\"label\")\n",
"print(\"String labels:\", datalab_str.labels)\n",
"\n",
"# Example 2: Labels as a list of integers\n",
"labels_int = [1, 2, 2, 1] # These will be remapped to [0, 1] internally\n",
"datalab_int = Datalab(data={\"text\": [\"a\", \"b\", \"c\", \"d\"], \"label\": labels_int}, label_name=\"label\")\n",
"print(\"Integer labels:\", datalab_int.labels)\n",
"\n",
"# Example 3: Advanced - Dataset with ClassLabel feature\n",
"my_dict = {\"pet_name\": [\"Spot\", \"Mittens\", \"Rover\", \"Rocky\", \"Pepper\", \"Socks\"], \"species\": [\"dog\", \"cat\", \"dog\", \"dog\", \"cat\", \"cat\"]}\n",
"features = Features({\"pet_name\": Value(\"string\"), \"species\": ClassLabel(names=[\"dog\", \"cat\"])})\n",
"dataset = Dataset.from_dict(my_dict, features=features)\n",
"datalab_dataset = Datalab(data=dataset, label_name=\"species\")\n",
"print(\"ClassLabel feature:\", datalab_dataset.labels)\n",
"```\n",
"\n",
"Using Datalab allows you to directly handle raw class name labels in your dataset while ensuring compatibility with label encoding requirements of lower-level cleanlab methods, which we'll cover in the next section.\n"
]
},
{
"cell_type": "markdown",
"id": "d5d0fbb3",
"metadata": {},
"source": [
"**Without Datalab**:\n",
"\n",
"Outside of Datalab, cleanlab offers various lower-level methods to directly operate on labels and diagnose issues. For instance: ``get_label_quality_scores()`` and ``find_label_issues()``. These lower-level methods only work with integer-encoded labels in the range `{0,1, ... K-1}` where `K = number_of_classes`. The `labels` array should only contain integer values in the range `{0, K-1}` and be of shape `(N,)` where `N = total_number_of_data_points`.\n",
"Do not pass in `labels` where some classes are entirely missing or are extremely rare, as cleanlab may not perform as expected. It is better to remove such classes entirely from the dataset first (also dropping the corresponding dimensions from `pred_probs` and then renormalizing it).\n",
"\n",
"**Text or string labels** should to be mapped to integers for each possible value. For example if your original data labels look like this: `[\"dog\", \"dog\", \"cat\", \"mouse\", \"cat\"]`, you should feed them to cleanlab like this: `labels = [1,1,0,2,0]` and keep track of which integer uniquely represents which class (classes were ordered alphabetically in this example). \n",
"\n",
"**One-hot encoded labels** should be integer-encoded by finding the argmax along the one-hot encoded axis. An example of what this might look like is shown below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "239d5ee7",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np \n",
"\n",
"# This example arr has 4 labels (one per data point) where \n",
"# each label can be one of 3 possible classes\n",
"\n",
"arr = np.array([[0,1,0],[1,0,0],[0,0,1],[1,0,0]])\n",
"labels_proper_format = np.argmax(arr, axis=1) # How labels should be formatted when passed into the model"
]
},
{
"cell_type": "markdown",
"id": "4181cac7",
"metadata": {},
"source": [
"### How do I infer the correct labels for examples cleanlab has flagged?"
]
},
{
"cell_type": "markdown",
"id": "6d4db5e1",
"metadata": {},
"source": [
"If you have a classifier that is compatible with [CleanLearning](../cleanlab/classification.html) (i.e. follows the sklearn API), here's an easy way to see predicted labels alongside the label issues:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28b324aa",
"metadata": {},
"outputs": [],
"source": [
"cl = cleanlab.classification.CleanLearning(your_classifier)\n",
"issues_dataframe = cl.find_label_issues(data, labels)"
]
},
{
"cell_type": "markdown",
"id": "6d4db5e2",
"metadata": {},
"source": [
"Alternatively if you have already computed out-of-sample predicted probabilities (`pred_probs`) from a classifier:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28b324ab",
"metadata": {},
"outputs": [],
"source": [
"cl = cleanlab.classification.CleanLearning()\n",
"issues_dataframe = cl.find_label_issues(X=None, labels=labels, pred_probs=pred_probs)"
]
},
{
"cell_type": "markdown",
"id": "b386dfc8",
"metadata": {},
"source": [
"Otherwise if you have already found issues via:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "90c10e18",
"metadata": {},
"outputs": [],
"source": [
"issues = cleanlab.filter.find_label_issues(labels, pred_probs)"
]
},
{
"cell_type": "markdown",
"id": "ad9ca03e",
"metadata": {},
"source": [
"then you can see your trained classifier's class prediction for each flagged example like this: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88839519",
"metadata": {},
"outputs": [],
"source": [
"class_predicted_for_flagged_examples = pred_probs[issues].argmax(axis=1)"
]
},
{
"cell_type": "markdown",
"id": "a668b74b",
"metadata": {},
"source": [
"Here you can see the classifier's class prediction for every example via:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "558490c2",
"metadata": {},
"outputs": [],
"source": [
"class_predicted_for_all_examples = pred_probs.argmax(axis=1)"
]
},
{
"cell_type": "markdown",
"id": "f9450eed",
"metadata": {},
"source": [
"We caution against just blindly taking the predicted label for granted, many of these suggestions may be wrong!"
]
},
{
"cell_type": "markdown",
"id": "bcc97591",
"metadata": {},
"source": [
"### How should I handle label errors in train vs. test data?\n",
"\n",
"If you do not address label errors in your test data, you may not even know when you have produced a better ML model because the evaluation is too noisy. For the best-trained models and most reliable evaluation of them, you should fix label errors in both training and testing data.\n",
"\n",
"To do this efficiently, first use cleanlab to automatically find label issues in both sets. You can simply merge these two sets into one larger dataset and run cross-validation training. On the merged dataset, you can do either of the following to detect label issues:\n",
"\n",
"\n",
"\n",
"**With Datalab**: Run `Datalab.find_issues()` on the merged dataset, then call `Datalab.report()` to see the label issues (and other types of data issues).\n",
"\n",
"```python\n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data = merged_dataset, label_name = \"label_column_name\")\n",
"\n",
"# Run proper cross-validation when computing predicted probabilities\n",
"lab.find_issues(pred_probs = pred_probs, issue_types = {\"label\": {}})\n",
"\n",
"lab.report()\n",
"```\n",
"\n",
"You can fetch the label issues DataFrame from the `Datalab` object by calling:\n",
"\n",
"```python\n",
"label_issues = lab.get_issues(\"label\")\n",
"```\n",
"\n",
"**Without Datalab**: Run cleanlab's lower-level `find_label_issues()` method on the merged datataset. Calling the [CleanLearning.find_label_issues()](../cleanlab/classification.html) method on your merged dataset both runs cross-validation training and finds label issues for you with any scikit-learn compatible classifier you choose.\n",
"\n",
"---\n",
"\n",
"After finding label issues, be **wary** about auto-correcting the labels for test examples. Instead manually fix the labels for your test data via careful review of the flagged issues.\n",
"\n",
"Auto-correcting labels for your training data is fair game, which should improve ML performance (if properly evaluated with clean test labels). You can boost ML performance further by manually fixing the training examples flagged with label issues, as demonstrated in this article:\n",
"\n",
"[**Handling Mislabeled Tabular Data to Improve Your XGBoost Model**](https://cleanlab.ai/blog/label-errors-tabular-datasets/)"
]
},
{
"cell_type": "markdown",
"id": "21f42f24",
"metadata": {},
"source": [
"### How can I find label issues in big datasets with limited memory? "
]
},
{
"cell_type": "markdown",
"id": "089f505e",
"metadata": {},
"source": [
"For a dataset with many rows and/or classes, there are more efficient methods in the `label_issues_batched` module. These methods read data in mini-batches and you can reduce the `batch_size` to control how much memory they require. Below is an example of how to use the `find_label_issues_batched()` method from this module, which can load mini-batches of data from `labels`, `pred_probs` saved as .npy files on disk. You can also run this method on Zarr arrays loaded from .zarr files. Try playing with the `n_jobs` argument for further multiprocessing speedups. If you need greater flexibility, check out the `LabelInspector` class from this module."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "41714b51",
"metadata": {},
"outputs": [],
"source": [
"# We'll assume your big arrays of labels, pred_probs have been saved to file like this:\n",
"from tempfile import mkdtemp\n",
"import os.path as path\n",
"\n",
"labels_file = path.join(mkdtemp(), \"labels.npy\")\n",
"pred_probs_file = path.join(mkdtemp(), \"pred_probs.npy\")\n",
"np.save(labels_file, labels)\n",
"np.save(pred_probs_file, pred_probs)\n",
"\n",
"# Code to find label issues by loading data from file in batches:\n",
"from cleanlab.experimental.label_issues_batched import find_label_issues_batched\n",
"\n",
"batch_size = 10000 # for efficiency, set this to as large of a value as your memory can handle\n",
"\n",
"# Indices of examples with label issues, sorted by label quality score (most severe to least severe):\n",
"indices_of_examples_with_issues = find_label_issues_batched(\n",
" labels_file=labels_file, pred_probs_file=pred_probs_file, batch_size=batch_size\n",
")"
]
},
{
"cell_type": "markdown",
"id": "13228a99-5d3f-47c0-87e5-2290d16461c4",
"metadata": {},
"source": [
"Methods that internally call `filter.find_label_issues()` can be sped up by specifying the argument `low_memory=True`, which will instead use `find_label_issues_batched()` internally. The following methods provide this option: \n",
"\n",
"1. [classification.CleanLearning](../cleanlab/classification.html#cleanlab.classification.CleanLearning)\n",
"2. [multilabel_classification.filter.find_label_issues](../cleanlab/multilabel_classification/filter.html#cleanlab.multilabel_classification.filter.find_label_issues)\n",
"3. [token_classification.filter.find_label_issues](../cleanlab/token_classification/filter.html?highlight=token#cleanlab.token_classification.filter.find_label_issues)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "20476c70",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
"\n",
"issue_indices = cleanlab.filter.find_label_issues(labels, pred_probs, filter_by = \"low_self_confidence\", return_indices_ranked_by=\"self_confidence\")\n",
"assert np.abs(len(issue_indices) - len(indices_of_examples_with_issues)) < 2, \"num issues differ in batched mode\"\n",
"set1 = set(issue_indices)\n",
"set2 = set(indices_of_examples_with_issues)\n",
"intersection = len(list(set1.intersection(set2)))\n",
"union = len(set1) + len(set2) - intersection\n",
"assert float(intersection) / union > 0.95, \"issue indices differ in batched mode\""
]
},
{
"cell_type": "markdown",
"id": "438b424d",
"metadata": {},
"source": [
"**To use less memory and get results faster if your dataset has many classes:** Try merging the rare classes into a single \"Other\" class before you find label issues. The resulting issues won't be affected much since cleanlab anyway does not have enough data to accurately diagnose label errors in classes that are rarely seen. To do this, you should aggregate all the probability assigned to the rare classes in `pred_probs` into a single new dimension of `pred_probs_merged` (where this new array no longer has columns for the rare classes). Here is a function that does this for you, which you can also modify as needed:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6983cdad",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden on docs.cleanlab.ai\n",
"# Add two rare additional classes to the dataset:\n",
"\n",
"num_rare_instances = 3\n",
"small_prob = 1e-4\n",
"pred_probs = np.hstack((pred_probs, np.ones((len(pred_probs),2))*small_prob))\n",
"pred_probs = pred_probs / np.sum(pred_probs, axis=1)[:, np.newaxis]\n",
"labels[:num_rare_instances] = 3\n",
"labels[num_rare_instances:(2*num_rare_instances)] = 4"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9092b8a0",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.internal.util import value_counts # use this to count how often each class occurs in labels\n",
"\n",
"def merge_rare_classes(labels, pred_probs, count_threshold = 10):\n",
" \"\"\" \n",
" Returns: labels, pred_probs after we merge all rare classes into a single 'Other' class.\n",
" Merged pred_probs has less columns. Rare classes are any occuring less than `count_threshold` times.\n",
" Also returns: `class_mapping_orig2new`, a dict to map new classes in merged labels back to classes \n",
" in original labels, useful for interpreting outputs from `dataset.heath_summary()` or `count.confident_joint()`.\n",
" \"\"\"\n",
" num_classes = pred_probs.shape[1]\n",
" num_examples_per_class = value_counts(labels, num_classes=num_classes)\n",
" rare_classes = [c for c in range(num_classes) if num_examples_per_class[c] < count_threshold]\n",
" if len(rare_classes) < 1:\n",
" raise ValueError(\"No rare classes found at the given `count_threshold`, merging is unnecessary unless you increase it.\")\n",
"\n",
" num_classes_merged = num_classes - len(rare_classes) + 1 # one extra class for all the merged ones\n",
" other_class = num_classes_merged - 1\n",
" labels_merged = labels.copy()\n",
" class_mapping_orig2new = {} # key = original class in `labels`, value = new class in `labels_merged`\n",
" new_c = 0\n",
" for c in range(num_classes):\n",
" if c in rare_classes:\n",
" class_mapping_orig2new[c] = other_class\n",
" else:\n",
" class_mapping_orig2new[c] = new_c\n",
" new_c += 1\n",
" labels_merged[labels == c] = class_mapping_orig2new[c]\n",
"\n",
" merged_prob = np.sum(pred_probs[:, rare_classes], axis=1, keepdims=True) # total probability over all merged classes for each example\n",
" pred_probs_merged = np.hstack((np.delete(pred_probs, rare_classes, axis=1), merged_prob)) # assumes new_class is as close to original_class in sorted order as is possible after removing the merged original classes\n",
" # check a few rows of probabilities after merging to verify they still sum to 1:\n",
" num_check = 1000 # only check a few rows for efficiency\n",
" ones_array_ref = np.ones(min(num_check,len(pred_probs)))\n",
" if np.isclose(np.sum(pred_probs[:num_check], axis=1), ones_array_ref).all() and (not np.isclose(np.sum(pred_probs_merged[:num_check], axis=1), ones_array_ref).all()):\n",
" raise ValueError(\"merged pred_probs do not sum to 1 in each row, check that merging was correctly done.\")\n",
" \n",
" return (labels_merged, pred_probs_merged, class_mapping_orig2new)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0a01109",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.filter import find_label_issues # can alternatively use find_label_issues_batched() shown above\n",
"\n",
"labels_merged, pred_probs_merged, class_mapping_orig2new = merge_rare_classes(labels, pred_probs, count_threshold=5)\n",
"examples_w_issues = find_label_issues(labels_merged, pred_probs_merged, return_indices_ranked_by=\"self_confidence\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8b1da032",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
"\n",
"rare_classes = [c for c in class_mapping_orig2new.keys() if class_mapping_orig2new[c] == pred_probs_merged.shape[1]-1]\n",
"og_examples_w_issues = find_label_issues(labels, pred_probs, return_indices_ranked_by=\"self_confidence\")\n",
"examples_of_interest = [x for x in examples_w_issues if labels[x] not in rare_classes]\n",
"og_examples_of_interest = [x for x in og_examples_w_issues if labels[x] not in rare_classes]\n",
"assert set(examples_of_interest) == set(og_examples_of_interest), \"merged label issues differ from non-merged label issues\""
]
},
{
"cell_type": "markdown",
"id": "3868ee8b",
"metadata": {},
"source": [
"### Why isn’t CleanLearning working for me?"
]
},
{
"cell_type": "markdown",
"id": "d13c9cd0",
"metadata": {},
"source": [
"At this time, CleanLearning only works with data formatted as numpy matrices or pd.DataFrames, \n",
"and with models that are compatible with the `sklearn` API \n",
"(check out [skorch](https://github.com/skorch-dev/skorch) for Pytorch compatibility). \n",
"You can still use cleanlab with other data formats though! Just separately obtain predicted probabilities (`pred_probs`) from your model via cross-validation and pass them as inputs. \n",
"\n",
"\n",
"If CleanLearning is running successfully but not improving predictive accuracy of your model, here are some tips:\n",
"\n",
"1. Use cleanlab to find label issues in your test data as well (we recommend pooling `labels` across both training and test data into one input for `find_label_issues()`). Then manually review and fix label issues identified in the test data to verify accuracy measurements are actually meaningful.\n",
"\n",
"2. Try different values for `filter_by`, `frac_noise`, and `min_examples_per_class` which can be set via the `find_label_issues_kwargs` argument in the initialization of `CleanLearning()`.\n",
"\n",
"3. Try to find a better model (eg. via hyperparameter tuning or changing to another classifier). `CleanLearning` can find better label issues by leveraging a better model, which allows it to produce better quality training data. This can form a virtuous cycle in which better models -> better issue detection -> better data -> even better models! \n",
"\n",
"4. Try jointly tuning both model hyperparameters and `find_label_issues_kwargs` values.\n",
"\n",
"5. Does your dataset have a *junk* (or *clutter*, *unknown*, *other*) class? If you have bad data, consider creating one (c.f. Caltech-256).\n",
"\n",
"6. Consider merging similar/overlapping classes found via ``cleanlab.dataset.find_overlapping_classes``.\n",
"\n",
"Other general tips to improve label error detection performance:\n",
"\n",
"1. Try creating more restrictive new filters by combining their intersections (e.g. `combined_boolean_mask = mask1 & mask2` where `mask1` and `mask2` are the boolean masks created by running `find_label_issues` with different values of the `filter_by` argument).\n",
"\n",
"2. If your `pred_probs` are obtained via a neural network, try averaging the `pred_probs` over the last K epochs of training instead of just using the final `pred_probs`. Similarly, you can try averaging `pred_probs` from several models (remember to re-normalize) or using ``cleanlab.rank.get_label_quality_ensemble_scores``."
]
},
{
"cell_type": "markdown",
"id": "9ae3899c",
"metadata": {},
"source": [
"### How can I use different models for data cleaning vs. final training in CleanLearning?"
]
},
{
"cell_type": "markdown",
"id": "a2ce1518",
"metadata": {},
"source": [
"The code below demonstrates CleanLearning with 2 different classifiers: `LogisticRegression()` and `GradientBoostingClassifier()`.\n",
"A `LogisticRegression` model is used to detect label issues (via cross-validation run inside CleanLearning) and a `GradientBoostingClassifier` model is finally trained on a clean subset of the data with issues removed.\n",
"This can be done with any two classifiers."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4c9e9030",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.classification import CleanLearning\n",
"import numpy as np\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.ensemble import GradientBoostingClassifier\n",
"\n",
"# Make example data\n",
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
"labels = np.array([0] * 100 + [1] * 100)\n",
"\n",
"# Introduce label errors\n",
"true_errors = [97, 98, 100, 101, 102, 104]\n",
"for idx in true_errors:\n",
" labels[idx] = 1 - labels[idx]\n",
"\n",
"# CleanLearning with 2 different classifiers: one classifier is used to detect label issues \n",
"# and a different classifier is subsequently trained on the clean subset of the data.\n",
"\n",
"model_to_find_errors = LogisticRegression() # this model will be trained many times via cross-validation\n",
"model_to_return = GradientBoostingClassifier() # this model will be trained once on clean subset of data\n",
"\n",
"cl0 = CleanLearning(model_to_find_errors)\n",
"issues = cl0.find_label_issues(data, labels)\n",
"\n",
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues)\n",
"pred_probs = cl.predict_proba(data) # predictions from GradientBoostingClassifier\n",
"\n",
"print(cl0.clf) # will be LogisticRegression()\n",
"print(cl.clf) # will be GradientBoostingClassifier()"
]
},
{
"cell_type": "markdown",
"id": "b71fef02",
"metadata": {},
"source": [
"### How do I hyperparameter tune only the final model trained (and not the one finding label issues) in CleanLearning?"
]
},
{
"cell_type": "markdown",
"id": "e7ec1956",
"metadata": {},
"source": [
"The code below demonstrates CleanLearning using a `GradientBoostingClassifier()` with no hyperparameter-tuning to find label issues but with hyperparameter-tuning via `RandomizedSearchCV(...)` for the final training of this model on the clean subset of the data.\n",
"This is a useful trick to avoid expensive hyperparameter-tuning for every fold of cross-validation (which is needed to find label issues)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8751619e",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from cleanlab.classification import CleanLearning\n",
"from sklearn.ensemble import GradientBoostingClassifier\n",
"from sklearn.model_selection import RandomizedSearchCV\n",
"\n",
"# Make example data\n",
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
"labels = np.array([0] * 100 + [1] * 100)\n",
"\n",
"# Introduce label errors\n",
"true_errors = [97, 98, 100, 101, 102, 104]\n",
"for idx in true_errors:\n",
" labels[idx] = 1 - labels[idx]\n",
"\n",
"# CleanLearning with no hyperparameter-tuning during expensive cross-validation to find label issues\n",
"# but hyperparameter-tuning for the final training of model on clean subset of the data:\n",
"\n",
"model_to_find_errors = GradientBoostingClassifier() # this model will be trained many times via cross-validation\n",
"model_to_return = RandomizedSearchCV(GradientBoostingClassifier(),\n",
" param_distributions = {\n",
" \"learning_rate\": [0.001, 0.05, 0.1, 0.2, 0.5],\n",
" \"max_depth\": [3, 5, 10],\n",
" }\n",
" ) # this model will be trained once on clean subset of data\n",
"\n",
"cl0 = CleanLearning(model_to_find_errors)\n",
"issues = cl0.find_label_issues(data, labels)\n",
"\n",
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues) # CleanLearning for hyperparameter final training\n",
"pred_probs = cl.predict_proba(data) # predictions from hyperparameter-tuned GradientBoostingClassifier\n",
"\n",
"print(cl0.clf) # will be GradientBoostingClassifier()\n",
"print(cl.clf) # will be RandomizedSearchCV(estimator=GradientBoostingClassifier(),...)"
]
},
{
"cell_type": "markdown",
"id": "d228decd",
"metadata": {},
"source": [
"### Why does regression.learn.CleanLearning take so long?"
]
},
{
"cell_type": "markdown",
"id": "de5c984b",
"metadata": {},
"source": [
"To effectively identify errors in a regression dataset, the methods in [regression.learn.CleanLearning](../../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning) estimate each datapoint's aleatoric uncertainty (by fitting a second copy of the regression model to predict the residuals’ magnitudes), as well as its epistemic uncertainty (by fitting multiple copies of the regression model with bootstrap resampling). These uncertainty estimates help provide a robust quality score that accounts for the model's imperfect predictions. \n",
"\n",
"These uncertainty estimates help produce better results but require longer runtimes. Here are a few options to speed up the runtime of these methods:\n",
"\n",
"- Reduce the number of bootstrap resampling rounds by decreasing the `n_boot` argument (default value is 5, set it to 0 to skip the epistemic uncertainty estimation entirely).\n",
"\n",
"- Set `include_aleatoric_uncertainty=False` to skip the aleatoric uncertainty estimation.\n",
"\n",
"- Include less elements in the `coarse_search_range` argument of [regression.learn.CleanLearning.find_label_issues](../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning.find_label_issues). This is overall set of values initially considered for estimating the fraction of data that have label issues.\n",
"\n",
"- Reduce the `fine_search_size` argument of [regression.learn.CleanLearning.find_label_issues](../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning.find_label_issues). A higher number represents a more thorough search to precisely estimate the fraction of data that have label issues.\n",
"\n",
"Below is sample code on how to pass in these arguments."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "623df36d",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.regression.learn import CleanLearning\n",
"\n",
"X = np.random.random(size=(30, 3))\n",
"coefficients = np.random.uniform(-1, 1, size=3)\n",
"y = np.dot(X, coefficients) + np.random.normal(scale=0.2, size=30)\n",
"\n",
"# passing optinal arguments to reduce runtime\n",
"cl = CleanLearning(n_boot=1, include_aleatoric_uncertainty=False)\n",
"cl.find_label_issues(X, y, coarse_search_range=[0.05, 0.1], fine_search_size=2)\n",
"\n",
"# you can also pass coarse_search_range and fine_search_size as kwargs to CleanLearning.fit\n",
"cl.fit(X, y, find_label_issues_kwargs={\"coarse_search_range\": [0.05, 0.1], \"fine_search_size\": 2})"
]
},
{
"cell_type": "markdown",
"id": "1677ba25",
"metadata": {},
"source": [
"**With Datalab**:\n",
"\n",
"Datalab runs CleanLearning under the hood when looking for label issues in regression datasets. Here's how you can achieve the same behavior as calling `CleanLearning.find_label_issues()` in the code above using Datalab:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "af3052ac",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data = {\"X\": X, \"y\": y}, label_name = \"y\", task=\"regression\")\n",
"\n",
"issue_types = {\n",
" \"label\": {\n",
" \"clean_learning_kwargs\": {\"n_boot\": 1, \"include_aleatoric_uncertainty\": False},\n",
" \"coarse_search_range\": [0.05, 0.1],\n",
" \"fine_search_size\": 2,\n",
" },\n",
"}\n",
"lab.find_issues(features=X, issue_types = issue_types)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How do I specify pre-computed data slices/clusters when detecting the Underperforming Group Issue?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The instructions for specifying pre-computed data slices/clusters when detecting underperforming groups in a dataset are now covered in detail in the Datalab workflows tutorial.\n",
"\n",
"- [Using clustering algorithms](./datalab/workflows.html#Find-Underperforming-Groups-in-a-Dataset).\n",
"- [Using categorical columns in a tabular dataset](./datalab/workflows.html#Predefining-Data-Slices-for-Detecting-Underperforming-Groups)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How to handle near-duplicate data identified by Datalab?\n",
"\n",
"cleanlab may identify near-duplicate examples in your dataset, these are examples that are very similar to each other and can potentially cause issues in model training and analytics. When near-duplicates are present, models may unexpectedly emphasize these examples, especially if they were accidentally duplicated. In such cases, it is crucial to remove the (near) duplicate copies from your dataset to ensure accurate and reliable results. A common strategy is to remove all but one of the duplicates from your dataset. Here's how you can achieve this with results from cleanlab's `Datalab` class:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from typing import Callable\n",
"import pandas as pd\n",
"\n",
"\n",
"def merge_duplicate_sets(df, merge_key: str):\n",
" \"\"\"Generate group keys for each row, then merge intersecting sets.\n",
" \n",
" :param df: DataFrame with columns 'is_near_duplicate_issue' and 'near_duplicate_sets'\n",
" :param merge_key: Name of the column to store the merged sets\n",
" \"\"\"\n",
"\n",
" df[merge_key] = df.apply(construct_group_key, axis=1)\n",
" merged_sets = consolidate_sets(df[merge_key].tolist())\n",
" df[merge_key] = df[merge_key].map(\n",
" lambda x: next(s for s in merged_sets if x.issubset(s))\n",
" )\n",
" return df\n",
"\n",
"def construct_group_key(row):\n",
" \"\"\"Convert near_duplicate_sets into a frozenset and include the row's own index.\"\"\"\n",
" return frozenset(row['near_duplicate_sets']).union({row.name})\n",
"\n",
"def consolidate_sets(sets_list):\n",
" \"\"\"Merge sets if they intersect.\"\"\"\n",
" \n",
" # Convert the input list of frozensets to a list of mutable sets\n",
" sets_list = [set(item) for item in sets_list]\n",
" \n",
" # A flag to keep track of whether any sets were merged in the current iteration\n",
" merged = True\n",
"\n",
" # Continue the merging process as long as we have merged some sets in the previous iteration\n",
" while merged:\n",
" merged = False\n",
" new_sets = []\n",
"\n",
" # Iterate through each set in our list\n",
" for current_set in sets_list:\n",
" # Skip empty sets\n",
" if not current_set:\n",
" continue\n",
"\n",
" # Find all sets that have an intersection with the current set\n",
" intersecting_sets = [s for s in sets_list if s & current_set]\n",
"\n",
" # If more than one set intersects, set the merged flag to True\n",
" if len(intersecting_sets) > 1:\n",
" merged = True\n",
"\n",
" # Merge all intersecting sets into one set\n",
" merged_set = set().union(*intersecting_sets)\n",
" new_sets.append(merged_set)\n",
"\n",
" # Empty the sets we've merged to prevent them from being processed again\n",
" for s in intersecting_sets:\n",
" sets_list[sets_list.index(s)] = set()\n",
"\n",
" # Replace the original sets list with the new list of merged sets\n",
" sets_list = new_sets\n",
"\n",
" # Convert the merged sets back to frozensets for the output\n",
" return [frozenset(item) for item in sets_list]\n",
"\n",
"def lowest_score_strategy(sub_df):\n",
" \"\"\"Keep the row with the lowest near_duplicate_score.\"\"\"\n",
" return sub_df['near_duplicate_score'].idxmin()\n",
"\n",
"\n",
"def filter_near_duplicates(data: pd.DataFrame, strategy_fn: Callable = lowest_score_strategy, **strategy_kwargs):\n",
" \"\"\"\n",
" Given a dataframe with columns 'is_near_duplicate_issue' and 'near_duplicate_sets',\n",
" return a series of boolean values where True indicates the rows to be removed.\n",
" The strategy_fn determines which rows to keep within each near_duplicate_set.\n",
"\n",
" :param data: DataFrame with is_near_duplicate_issue and near_duplicate_sets columns\n",
" :param strategy_fn: Function to determine which rows to keep within each near_duplicate_set\n",
" :return: Series of boolean values where True indicates rows to be removed.\n",
" \"\"\"\n",
" \n",
" # Filter out rows where 'is_near_duplicate_issue' is True to get potential duplicates\n",
" duplicate_rows = data.query(\"is_near_duplicate_issue\").copy()\n",
"\n",
" # Generate group keys for each row and merge intersecting sets\n",
" group_key = \"sets\"\n",
" duplicate_rows = merge_duplicate_sets(duplicate_rows, merge_key=group_key)\n",
"\n",
" # Use the strategy function to determine the indices of the rows to keep for each group\n",
" to_keep_indices = duplicate_rows.groupby(group_key).apply(strategy_fn, **strategy_kwargs).explode().values\n",
"\n",
" # Produce a boolean series indicating which rows should be removed\n",
" to_remove = ~data.index.isin(to_keep_indices)\n",
"\n",
" return to_remove"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The functions above collect sets of near-duplicate examples. Within each\n",
"collection, a single example is chosen to be kept in the dataset. The rest of the examples in the collection are removed.\n",
"Examples that are not near-duplicates of any other examples are kept in the dataset as well.\n",
"\n",
"The choice of which example to keep in each set of near-duplicate examples can be made in a variety of ways. Here, the example with the lowest near-duplicate score is chosen.\n",
"You can use any strategy that best suits your application by defining the strategy as a function and passing it as the `strategy_fn` argument to `filter_near_duplicates()`.\n",
"Below is an example of how this is applied to a dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from cleanlab import Datalab\n",
"import numpy as np\n",
"\n",
"# Assume you have a dataset with a set of 3 near-duplicate examples\n",
"features = np.random.random(size=(15, 3))\n",
"for neighbor in range(1, 3):\n",
" # Make examples 0, 1, and 2 near-duplicates of each other\n",
" features[neighbor] = features[0] + np.random.normal(scale=0.001, size=3)\n",
"\n",
"# Identify near-duplicate examples with Datalab\n",
"your_dataset = {\n",
" \"features\": features,\n",
"}\n",
"lab = Datalab(data=your_dataset)\n",
"lab.find_issues(features = features, issue_types={\"near_duplicate\": {}})\n",
"\n",
"# Pick out ids of near-duplicate examples to remove\n",
"near_duplicate_issues = (\n",
" lab.get_issues(\"near_duplicate\")\n",
" .query(\"is_near_duplicate_issue\")\n",
" .sort_values(\"near_duplicate_score\")\n",
")\n",
"ids_to_remove_series = filter_near_duplicates(near_duplicate_issues)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"Near-duplicate examples to keep:\", np.where(~ids_to_remove_series)[0].tolist())\n",
"\n",
"print(\"Near-duplicate examples to remove:\", np.where(ids_to_remove_series)[0].tolist())"
]
},
{
"cell_type": "markdown",
"id": "1520a93f",
"metadata": {},
"source": [
"### Can't find an answer to your question?\n",
"\n",
"If your question is not addressed in these tutorials and API documentation pages, then refer to the [Github issues](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue) or [Code Examples](https://github.com/cleanlab/examples).\n",
"\n",
"If you still have a question, open a [new Github issue](https://github.com/cleanlab/cleanlab/issues/new/choose)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/improving_ml_performance.ipynb
+1071
View File
File diff suppressed because it is too large Load Diff
docs/source/tutorials/indepth_overview.ipynb
+1156
View File
File diff suppressed because it is too large Load Diff
docs/source/tutorials/index.rst
+20
View File
@@ -0,0 +1,20 @@
Tutorials
=========
.. toctree::
:maxdepth: 1
datalab/
improving_ml_performance
clean_learning/
indepth_overview
dataset_health
outliers
multiannotator
multilabel_classification
regression
token_classification
segmentation
object_detection
pred_probs_cross_val
faq
docs/source/tutorials/multiannotator.ipynb
+789
View File
@@ -0,0 +1,789 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4c7436b8",
"metadata": {},
"source": [
"# Estimate Consensus and Annotator Quality for Data Labeled by Multiple Annotators"
]
},
{
"cell_type": "markdown",
"id": "4b432513",
"metadata": {},
"source": [
"This 5-minute quickstart tutorial shows how to use cleanlab for classification data that has been labeled by *multiple* annotators (where each example has been labeled by at least one annotator, but not every annotator has labeled every example). Compared to existing crowdsourcing tools, cleanlab helps you better analyze such data by leveraging a trained classifier model in addition to the raw annotations. With one line of code, you can automatically compute:\n",
"\n",
"- A **consensus label** for each example (i.e. *truth inference*) that aggregates the individual annotations (more accurately than algorithms from crowdsourcing like majority-vote, Dawid-Skene, or GLAD).\n",
"- A **quality score for each consensus label** which measures our confidence that this label is correct (via well-calibrated estimates that account for the: number of annotators which have labeled this example, overall quality of each annotator, and quality of our trained ML models).\n",
"- An analogous **label quality score** for each individual label chosen by one annotator for a particular example (to measure our confidence in alternate labels when annotators differ from the consensus).\n",
"- An **overall quality score for each annotator** which measures our confidence in the overall correctness of labels obtained from this annotator.\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Obtain initial consensus labels of multiannotator data using majority vote.\n",
"- Train a classifier model on the initial consensus labels and use it to obtain out-of-sample predicted class probabilities.\n",
"- Use cleanlab's `multiannotator.get_label_quality_multiannotator` function to get improved consensus labels that more accurately reflect the ground truth.\n",
"- View other information about your multiannotator dataset, such as consensus and annotator quality scores, agreement between annotators, detailed label quality scores and more!\n",
"\n",
"**Consensus labels** represent the best guess of the true label for each example and can be used for more reliable modeling/analytics. Cleanlab automatically produces enhanced estimates of consensus through the use of machine learning.\n",
"**Quality scores** help us determine how much trust we can place in each: consensus label, individual annotator, and particular label from a particular annotator. These quality scores can help you determine which annotators are best/worst overall, as well as which current consensus labels are least trustworthy and should perhaps be verified via additional annotation. \n",
"\n",
"This tutorial uses a toy *tabular* dataset labeled with multiple annotators but **these steps can easily be applied to image or text data**."
]
},
{
"cell_type": "markdown",
"id": "03385f84",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have `multiannotator_labels` and (out-of-sample) `pred_probs` from a model trained on an existing set of consensus labels? Run the code below to get improved consensus labels and more information about the quality of your labels and annotators.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab.multiannotator import get_label_quality_multiannotator\n",
"\n",
"get_label_quality_multiannotator(multiannotator_labels, pred_probs)\n",
"\n",
"```\n",
"\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "e6a48d31",
"metadata": {},
"source": [
"## 1. Install and import required dependencies"
]
},
{
"cell_type": "markdown",
"id": "6c6e5b15",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install cleanlab\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a3ddc95f",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "markdown",
"id": "dd0148e6",
"metadata": {},
"source": [
"Let’s import some of the packages needed throughout this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c4efd119",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab.multiannotator import get_label_quality_multiannotator, get_majority_vote_label"
]
},
{
"cell_type": "markdown",
"id": "345b6678",
"metadata": {},
"source": [
"## 2. Create the data (can skip these details)"
]
},
{
"cell_type": "markdown",
"id": "82aeedc8",
"metadata": {},
"source": [
"For this tutorial we will generate a toy dataset that has 50 annotators and 300 examples. There are three possible classes, `0`, `1` and `2`. \n",
"\n",
"Each annotator annotates approximately 10% of the examples. We also synthetically made the last 5 annotators in our toy dataset have much noisier labels than the rest of the annotators.\n",
"\n",
"Solely for evaluating cleanlab's consensus labels against other consensus methods, we here also generate the true labels for this example dataset. However, true labels are not required for any cleanlab multiannotator functions (and they usually are not available in real applications).\n",
"To generate our multiannotator data, we define a `make_data()` method (can skip these details)."
]
},
{
"cell_type": "markdown",
"id": "69b5ddaa",
"metadata": {},
"source": [
"<details><summary>See the code for data generation **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
" \n",
"from cleanlab.benchmarking.noise_generation import generate_noise_matrix_from_trace\n",
"from cleanlab.benchmarking.noise_generation import generate_noisy_labels\n",
"\n",
"SEED = 111 # set to None for non-reproducible randomness\n",
"np.random.seed(seed=SEED)\n",
"\n",
"def make_data(\n",
" means=[[3, 2], [7, 7], [0, 8]],\n",
" covs=[[[5, -1.5], [-1.5, 1]], [[1, 0.5], [0.5, 4]], [[5, 1], [1, 5]]],\n",
" sizes=[150, 75, 75],\n",
" num_annotators=50,\n",
"):\n",
" \n",
" m = len(means) # number of classes\n",
" n = sum(sizes)\n",
" local_data = []\n",
" labels = []\n",
"\n",
" for idx in range(m):\n",
" local_data.append(\n",
" np.random.multivariate_normal(mean=means[idx], cov=covs[idx], size=sizes[idx])\n",
" )\n",
" labels.append(np.array([idx for i in range(sizes[idx])]))\n",
" X_train = np.vstack(local_data)\n",
" true_labels_train = np.hstack(labels)\n",
"\n",
" # Compute p(true_label=k)\n",
" py = np.bincount(true_labels_train) / float(len(true_labels_train))\n",
" \n",
" noise_matrix_better = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.8 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
" \n",
" noise_matrix_worse = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.35 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" # Generate our noisy labels using the noise_matrix for specified number of annotators.\n",
" s = pd.DataFrame(\n",
" np.vstack(\n",
" [\n",
" generate_noisy_labels(true_labels_train, noise_matrix_better)\n",
" if i < num_annotators - 5\n",
" else generate_noisy_labels(true_labels_train, noise_matrix_worse)\n",
" for i in range(num_annotators)\n",
" ]\n",
" ).transpose()\n",
" )\n",
"\n",
" # Each annotator only labels approximately 10% of the dataset\n",
" # (unlabeled points represented with NaN)\n",
" s = s.apply(lambda x: x.mask(np.random.random(n) < 0.9)).astype(\"Int64\")\n",
" s.dropna(axis=1, how=\"all\", inplace=True)\n",
" s.columns = [\"A\" + str(i).zfill(4) for i in range(1, num_annotators+1)]\n",
"\n",
" row_NA_check = pd.notna(s).any(axis=1)\n",
"\n",
" return {\n",
" \"X_train\": X_train[row_NA_check],\n",
" \"true_labels_train\": true_labels_train[row_NA_check],\n",
" \"multiannotator_labels\": s[row_NA_check].reset_index(drop=True),\n",
" }\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c37c0a69",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from cleanlab.benchmarking.noise_generation import generate_noise_matrix_from_trace\n",
"from cleanlab.benchmarking.noise_generation import generate_noisy_labels\n",
"\n",
"SEED = 111 # set to None for non-reproducible randomness\n",
"np.random.seed(seed=SEED)\n",
"\n",
"def make_data(\n",
" means=[[3, 2], [7, 7], [0, 8]],\n",
" covs=[[[5, -1.5], [-1.5, 1]], [[1, 0.5], [0.5, 4]], [[5, 1], [1, 5]]],\n",
" sizes=[150, 75, 75],\n",
" num_annotators=50,\n",
"):\n",
" \n",
" m = len(means) # number of classes\n",
" n = sum(sizes)\n",
" local_data = []\n",
" labels = []\n",
"\n",
" for idx in range(m):\n",
" local_data.append(\n",
" np.random.multivariate_normal(mean=means[idx], cov=covs[idx], size=sizes[idx])\n",
" )\n",
" labels.append(np.array([idx for i in range(sizes[idx])]))\n",
" X_train = np.vstack(local_data)\n",
" true_labels_train = np.hstack(labels)\n",
"\n",
" # Compute p(true_label=k)\n",
" py = np.bincount(true_labels_train) / float(len(true_labels_train))\n",
" \n",
" noise_matrix_better = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.8 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
" \n",
" noise_matrix_worse = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.35 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" # Generate our noisy labels using the noise_matrix for specified number of annotators.\n",
" s = pd.DataFrame(\n",
" np.vstack(\n",
" [\n",
" generate_noisy_labels(true_labels_train, noise_matrix_better)\n",
" if i < num_annotators - 5\n",
" else generate_noisy_labels(true_labels_train, noise_matrix_worse)\n",
" for i in range(num_annotators)\n",
" ]\n",
" ).transpose()\n",
" )\n",
"\n",
" # Each annotator only labels approximately 10% of the dataset\n",
" # (unlabeled points represented with NaN)\n",
" s = s.apply(lambda x: x.mask(np.random.random(n) < 0.9)).astype(\"Int64\")\n",
" s.dropna(axis=1, how=\"all\", inplace=True)\n",
" s.columns = [\"A\" + str(i).zfill(4) for i in range(1, num_annotators+1)]\n",
"\n",
" row_NA_check = pd.notna(s).any(axis=1)\n",
"\n",
" return {\n",
" \"X_train\": X_train[row_NA_check],\n",
" \"true_labels_train\": true_labels_train[row_NA_check],\n",
" \"multiannotator_labels\": s[row_NA_check].reset_index(drop=True),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "99f69523",
"metadata": {},
"outputs": [],
"source": [
"data_dict = make_data()\n",
"\n",
"X = data_dict[\"X_train\"]\n",
"multiannotator_labels = data_dict[\"multiannotator_labels\"]\n",
"true_labels = data_dict[\"true_labels_train\"] # used for comparing the accuracy of consensus labels"
]
},
{
"cell_type": "markdown",
"id": "4a705e28",
"metadata": {},
"source": [
"Let's view the first few rows of the data used for this tutorial. Here are the labels selected by each annotator for the first few examples (rows) in the dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f241c16",
"metadata": {},
"outputs": [],
"source": [
"multiannotator_labels.head()"
]
},
{
"cell_type": "markdown",
"id": "4a705e29",
"metadata": {},
"source": [
"Here are the corresponding features for these examples:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f0819ba",
"metadata": {},
"outputs": [],
"source": [
"X[:5]"
]
},
{
"cell_type": "markdown",
"id": "0cb8131d",
"metadata": {},
"source": [
"`multiannotator_labels` contains the class label that each annotator chose for each example in the dataset, with examples that a particular annotator did not label represented using `np.nan`. \n",
"`X` contains the features for each example, which happen to be numeric in this tutorial but any feature modality can be used with ``cleanlab.multiannotator``."
]
},
{
"cell_type": "markdown",
"id": "946726ad",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"You can easily replace the above with your own multiannotator labels and features, then continue with the rest of the tutorial.\n",
" \n",
"`multiannotator_labels` should be a numpy array or pandas DataFrame with each column representing an annotator and each row representing an example. Your labels should be represented as integer indices 0, 1, ..., num_classes - 1, where examples that are not annotated by a particular annotator are represented using `np.nan` or `pd.NA`. If you have string labels or other labels that do not fit the required format, you can convert them to the proper format using `cleanlab.internal.multiannotator_utils.format_multiannotator_labels`. \n",
" \n",
"Your features can be represented however you like (since these are not inputs to `cleanlab.multiannotator` methods) as long as you are able to fit a classifer to them and obtain its predicted class probabilities! \n",
"\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"id": "51335def",
"metadata": {},
"source": [
"## 3. Get initial consensus labels via majority vote and compute out-of-sample predicted probabilities"
]
},
{
"cell_type": "markdown",
"id": "c1857cc7",
"metadata": {},
"source": [
"Before training a machine learning model, we must first obtain initial consensus labels from the data annotations representing a crude guess of the best label for each example. The most straight forward way to obtain an initial set of consensus labels is via simple majority vote."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d009f347",
"metadata": {},
"outputs": [],
"source": [
"majority_vote_label = get_majority_vote_label(multiannotator_labels)"
]
},
{
"cell_type": "markdown",
"id": "7287b733",
"metadata": {},
"source": [
"Majority vote consensus labels may not be very reliable, particularly for examples that were only labeled by one or a few annotators. To more reliably estimate consensus, we can account for the features associated with each example (based on which the annotations were derived in the first place). Fitting a classifier model serves as a natural way to account for these feature values, here we train a simple logistic regression model to get significantly more accurate estimates of consensus labels and associated quality scores.\n",
"\n",
"We fit the model with our initial consensus labels, and then get (out-of-sample) predicted class probabilities for each example in the dataset from the trained model. These predicted probabilities help us estimate the best consensus labels and associated confidence values in a statistically optimal manner that accounts for all the available information."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cbd1e415",
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"\n",
"num_crossval_folds = 5 \n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=X, y=majority_vote_label, cv=num_crossval_folds, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "4eab5188",
"metadata": {},
"source": [
"## 4. Use cleanlab to get better consensus labels and other statistics"
]
},
{
"cell_type": "markdown",
"id": "4d392ce5",
"metadata": {},
"source": [
"Using the annotators' labels and the (out-of-sample) predicted class probabilities from the model, cleanlab can estimate **improved consensus labels** for our data that are more accurate than our initial consensus labels were.\n",
"\n",
"Having accurate labels provides insight on each annotator's label quality and is key for boosting model accuracy and achieving dependable real-world results."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ca92617",
"metadata": {},
"outputs": [],
"source": [
"results = get_label_quality_multiannotator(multiannotator_labels, pred_probs, verbose=False)"
]
},
{
"cell_type": "markdown",
"id": "98042e7f",
"metadata": {},
"source": [
"Here, we use the `multiannotator.get_label_quality_multiannotator()` function which returns a dictionary containing three items:\n"
]
},
{
"cell_type": "markdown",
"id": "76d7c0e2",
"metadata": {},
"source": [
"1. `label_quality` which gives us the improved consensus labels using information from each of the annotators and the model. The DataFrame also contains information about the number of annotations, annotator agreement and consensus quality score for each example.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf945113",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"results[\"label_quality\"].head()"
]
},
{
"cell_type": "markdown",
"id": "984d65c4",
"metadata": {},
"source": [
"2. `detailed_label_quality` which returns the label quality score for each label given by every annotator"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14251ee0",
"metadata": {},
"outputs": [],
"source": [
"results[\"detailed_label_quality\"].head()"
]
},
{
"cell_type": "markdown",
"id": "db02e63d",
"metadata": {},
"source": [
"3. `annotator_stats` which gives us the annotator quality score for each annotator, alongisde other information such as the number of examples each annotator labeled, their agreement with the consensus labels and the class they perform the worst at. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "efe16638",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"results[\"annotator_stats\"].head(10)"
]
},
{
"cell_type": "markdown",
"id": "a0d09bfa",
"metadata": {},
"source": [
"The `annotator_stats` DataFrame is sorted by increasing `annotator_quality`, showing us the worst annotators first.\n",
"\n",
"Notice that in the above table annotators with ids A0046 to A0050 have the worst annotator quality score, which is expected because we made the last 5 annotators systematically worse than the rest."
]
},
{
"cell_type": "markdown",
"id": "20ca8dd2",
"metadata": {},
"source": [
"### Comparing improved consensus labels"
]
},
{
"cell_type": "markdown",
"id": "1b49657d",
"metadata": {},
"source": [
"We can get the improved consensus labels from the `label_quality` DataFrame shown above."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "abd0fb0b",
"metadata": {},
"outputs": [],
"source": [
"improved_consensus_label = results[\"label_quality\"][\"consensus_label\"].values"
]
},
{
"cell_type": "markdown",
"id": "1fd7a5fd",
"metadata": {},
"source": [
"Since our toy dataset is synthetically generated by adding noise to each annotator's labels, we know the ground truth labels for each example. Hence we can compare the accuracy of the consensus labels obtained using majority vote, and the improved consensus labels obtained using cleanlab."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cdf061df",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"majority_vote_accuracy = np.mean(true_labels == majority_vote_label)\n",
"cleanlab_label_accuracy = np.mean(true_labels == improved_consensus_label)\n",
"\n",
"print(f\"Accuracy of majority vote labels = {majority_vote_accuracy}\")\n",
"print(f\"Accuracy of cleanlab consensus labels = {cleanlab_label_accuracy}\")"
]
},
{
"cell_type": "markdown",
"id": "2c20b2c9",
"metadata": {},
"source": [
"We can see that the accuracy of the consensus labels improved as a result of using cleanlab, which not only takes the annotators' labels into account, but also a model to compute better consensus labels."
]
},
{
"cell_type": "markdown",
"id": "f82dd4d5",
"metadata": {},
"source": [
"### Inspecting consensus quality scores to find potential consensus label errors"
]
},
{
"cell_type": "markdown",
"id": "fddb5453",
"metadata": {},
"source": [
"We can get the consensus quality score from the `label_quality` DataFrame shown above."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "08949890",
"metadata": {},
"outputs": [],
"source": [
"consensus_quality_score = results[\"label_quality\"][\"consensus_quality_score\"]"
]
},
{
"cell_type": "markdown",
"id": "5f150a08",
"metadata": {},
"source": [
"Besides obtaining improved consensus labels, cleanlab also computes consensus quality scores for each example. The lower scores represent potential consensus label errors in the dataset.\n",
"\n",
"Here, we will extract 15 examples that have the lowest consensus quality score, and we can compare their average accuracy when compared to the true labels. We will also compute the average accuracy for the rest of the examples for comparison."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6948b073",
"metadata": {},
"outputs": [],
"source": [
"sorted_consensus_quality_score = consensus_quality_score.sort_values()\n",
"worst_quality = sorted_consensus_quality_score.index[:15]\n",
"better_quality = sorted_consensus_quality_score.index[15:]\n",
"\n",
"worst_quality_accuracy = np.mean(true_labels[worst_quality] == improved_consensus_label[worst_quality])\n",
"better_quality_accuracy = np.mean(true_labels[better_quality] == improved_consensus_label[better_quality])\n",
"\n",
"print(f\"Accuracy of 15 worst quality examples = {worst_quality_accuracy}\")\n",
"print(f\"Accuracy of better quality examples = {better_quality_accuracy}\")"
]
},
{
"cell_type": "markdown",
"id": "4fdf4d91",
"metadata": {},
"source": [
"We observe that the 15 worst-consensus-quality-score examples have a lower average accuracy compared to the rest of the examples. Cleanlab automatically determines which consensus labels are least trustworthy (perhaps want to have another annotator look at that data). Here we see these trustworthiness estimates really do correspond to the true quality of the consensus labels (which we know in this toy dataset because we have the true labels, unlike in your applications)"
]
},
{
"cell_type": "markdown",
"id": "06cae16a",
"metadata": {},
"source": [
"## 5. Retrain model using improved consensus labels"
]
},
{
"cell_type": "markdown",
"id": "8d4e31ab",
"metadata": {},
"source": [
"After obtaining the improved consensus labels, we can now retrain a better version of our machine learning model using these newly obtained labels. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6f8e6914",
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"\n",
"num_crossval_folds = 5 \n",
"improved_pred_probs = cross_val_predict(\n",
" estimator=model, X=X, y=improved_consensus_label, cv=num_crossval_folds, method=\"predict_proba\"\n",
")\n",
"\n",
"# alternatively, we can treat all the improved consensus labels as training labels to fit the model \n",
"# model.fit(X, improved_consensus_label)"
]
},
{
"cell_type": "markdown",
"id": "e59f7d4f",
"metadata": {},
"source": [
"## Further improvements \n",
"You can also repeat this process of getting better consensus labels using the model's out-of-sample predicted probabilities and then retraining the model with the improved labels to get even better predicted class probabilities in a virtuous cycle!\n",
"For details, see our [examples](https://github.com/cleanlab/examples) notebook on [Iterative use of Cleanlab to Improve Classification Models (and Consensus Labels) from Data Labeled by Multiple Annotators](https://github.com/cleanlab/examples/blob/master/multiannotator_cifar10/multiannotator_cifar10.ipynb).\n",
"\n",
"If possible, the best way to improve your model is to collect additional labels for both previously annotated data and extra not-yet-labeled examples (i.e. *active learning*). To decide which data is most informative to label next, use `cleanlab.multiannotator.get_active_learning_scores()` rather than the methods from this tutorial. This is demonstrated in our examples notebook on [Active Learning with Multiple Data Annotators via ActiveLab](https://github.com/cleanlab/examples/blob/master/active_learning_multiannotator/active_learning.ipynb).\n",
"\n",
"While this notebook focused on analzying the labels of your data, cleanlab can also check your data features for various issues. Learn how to do this by following our [Datalab tutorials](../tutorials/datalab/index.html), except you do not need to pass in `labels` now that you've already analyzed them with this notebook (or you can provide `labels` to Datalab as the consensus labels estimated here).\n",
"\n",
"\n",
"## How does cleanlab.multiannotator work?\n",
"\n",
"All estimates above are produced via the CROWDLAB algorithm, described in this paper that contains extensive benchmarks which show CROWDLAB can produce better estimates than popular methods like Dawid-Skene and GLAD:\n",
"\n",
"[CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://arxiv.org/abs/2210.06812)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b806d2ea",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"if majority_vote_accuracy >= cleanlab_label_accuracy: # check cleanlab has improved prediction accuracy\n",
" raise Exception(\"Cleanlab training failed to improve consensus label accuracy\")\n",
"\n",
"if worst_quality_accuracy > better_quality_accuracy: # check bad consensus quality score corresponds to bad consensus\n",
" raise Exception(\"Cleanlab consensus quality score failed to detect bad consensus labels\")\n",
" \n",
"annotator_stats = results[\"annotator_stats\"]\n",
"bad_annotator_idx = [\"A0046\", \"A0047\", \"A0048\", \"A0049\", \"A0050\"]\n",
"bad_annotator_mask = annotator_stats.index.isin(bad_annotator_idx)\n",
"\n",
"avg_annotator_quality_bad = np.mean(annotator_stats[bad_annotator_mask][\"annotator_quality\"])\n",
"avg_annotator_quality_good = np.mean(annotator_stats[~bad_annotator_mask][\"annotator_quality\"])\n",
"\n",
"if avg_annotator_quality_bad >= avg_annotator_quality_good: # check bad annotator get bad quality scores \n",
" raise Exception(\"Low quality annotators have higher quality scores than good quality annotators\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"vscode": {
"interpreter": {
"hash": "50292dbb1f747f7151d445135d392af3138fb3c65386d17d9510cb605222b10b"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/multilabel_classification.ipynb
+644
View File
@@ -0,0 +1,644 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "64053c0f-3582-465b-9e4c-a83da332da88",
"metadata": {},
"source": [
"# Find Label Errors in Multi-Label Classification Datasets\n",
"\n",
"This 5-minute quickstart tutorial demonstrates how to find potential label errors in multi-label classification datasets. In such datasets, each example is labeled as belonging to one *or more* classes (unlike in *multi-class classification* where each example can only belong to one class). For a particular example in such multi-label classification data, we say each class either applies or not. We may even have some examples where *no* classes apply. Common applications of this include image tagging (or document tagging), where multiple tags can be appropriate for a single image (or document). For example, a image tagging application could involve the following classes: [`copyrighted`, `advertisement`, `face`, `violence`, `nsfw`]"
]
},
{
"cell_type": "markdown",
"id": "adaefc8b-b639-4bdf-af0d-337519e37ffc",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"cleanlab finds data/label issues based on two inputs: `labels` formatted as a list of lists of integer class indices that apply to each example in your dataset, and `pred_probs` from a trained multi-label classification model (which do not need to sum to 1 since the classes are not mutually exclusive). Once you have these, run the code below to find issues in your multi-label dataset:\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"# Assuming your dataset has a label column named 'label'\n",
"lab = Datalab(dataset, label_name='label', task='multilabel')\n",
"# To detect more issue types, optionally supply `features` (numeric dataset values or model embeddings of the data)\n",
"lab.find_issues(pred_probs=pred_probs, features=features)\n",
"\n",
"lab.report()\n",
"```\n",
"\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "6a6261a3-6ea1-44a6-ac91-d375c8aa5535",
"metadata": {},
"source": [
"## 1. Install required dependencies and get dataset\n",
"\n",
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7383d024-8273-4039-bccd-aab3020d331f",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs.cleanlab.ai).\n",
"# Package versions we used: matplotlib==3.5.1\n",
"\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf9101d8-b1a9-4305-b853-45aaf3d67a69",
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numpy as np\n",
"import sklearn\n",
"from sklearn.multiclass import OneVsRestClassifier\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.model_selection import StratifiedKFold\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from cleanlab import Datalab\n",
"from cleanlab.internal.multilabel_utils import int2onehot, onehot2int"
]
},
{
"cell_type": "markdown",
"id": "6fe047ed",
"metadata": {},
"source": [
"Here we generate a small multi-label classification dataset for a quick demo. To see cleanlab applied to a real image tagging dataset, check out our [example](https://github.com/cleanlab/examples) notebook [\"Find Label Errors in Multi-Label Classification Data (CelebA Image Tagging)\"](https://github.com/cleanlab/examples/blob/master/multilabel_classification/image_tagging.ipynb)."
]
},
{
"cell_type": "markdown",
"id": "6b283ecc-ba52-4bd7-81d8-5397966b1621",
"metadata": {},
"source": [
"<details><summary>Code to generate dataset (can skip these details) **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
" \n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"def make_multilabel_data(\n",
" means=[[-5, 3.5], [0, 2], [-3, 6]],\n",
" covs=[[[3, -1.5], [-1.5, 1]], [[5, -1.5], [-1.5, 1]], [[3, -1.5], [-1.5, 1]]],\n",
" boxes_coordinates=[[-3.5, 0, -1.5, 1.7], [-1, 3, 2, 4], [-5, 2, -3, 4], [-3, 2, -1, 4]],\n",
" box_multilabels=[[0, 1], [1, 2], [0, 2], [0, 1, 2]],\n",
" sizes=[100, 80, 100],\n",
" avg_trace=0.9,\n",
" seed=1,\n",
"):\n",
" np.random.seed(seed=seed)\n",
" num_classes = len(means)\n",
" m = num_classes + len(\n",
" box_multilabels\n",
" ) # number of classes by treating each multilabel as 1 unique label\n",
" n = sum(sizes)\n",
" local_data = []\n",
" labels = []\n",
" test_data = []\n",
" test_labels = []\n",
" for i in range(0, len(means)):\n",
" local_data.append(np.random.multivariate_normal(mean=means[i], cov=covs[i], size=sizes[i]))\n",
" test_data.append(np.random.multivariate_normal(mean=means[i], cov=covs[i], size=sizes[i]))\n",
" test_labels += [[i]] * sizes[i]\n",
" labels += [[i]] * sizes[i]\n",
"\n",
" def make_multi(X, Y, bx1, by1, bx2, by2, label_list):\n",
" ll = np.array([bx1, by1]) # lower-left\n",
" ur = np.array([bx2, by2]) # upper-right\n",
"\n",
" inidx = np.all(np.logical_and(X.tolist() >= ll, X.tolist() <= ur), axis=1)\n",
" for i in range(0, len(Y)):\n",
" if inidx[i]:\n",
" Y[i] = label_list\n",
" return Y\n",
"\n",
" X_train = np.vstack(local_data)\n",
" X_test = np.vstack(test_data)\n",
"\n",
" for i in range(0, len(box_multilabels)):\n",
" bx1, by1, bx2, by2 = boxes_coordinates[i]\n",
" multi_label = box_multilabels[i]\n",
" labels = make_multi(X_train, labels, bx1, by1, bx2, by2, multi_label)\n",
" test_labels = make_multi(X_test, test_labels, bx1, by1, bx2, by2, multi_label)\n",
"\n",
" d = {}\n",
" for i in labels:\n",
" if str(i) not in d:\n",
" d[str(i)] = len(d)\n",
" inv_d = {v: k for k, v in d.items()}\n",
" labels_idx = [d[str(i)] for i in labels]\n",
" py = np.bincount(labels_idx) / float(len(labels_idx))\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=avg_trace * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=seed,\n",
" )\n",
" noisy_labels_idx = generate_noisy_labels(labels_idx, noise_matrix)\n",
" noisy_labels = [eval(inv_d[i]) for i in noisy_labels_idx]\n",
" return {\n",
" \"X_train\": X_train,\n",
" \"true_labels_train\": labels,\n",
" \"X_test\": X_test,\n",
" \"true_labels_test\": test_labels,\n",
" \"labels\": noisy_labels,\n",
" \"dict_unique_label\": d,\n",
" 'labels_idx': noisy_labels_idx,\n",
"\n",
" }\n",
"\n",
"def get_color_array(labels):\n",
" \"\"\"\n",
" This function returns a dictionary mapping multi-labels to unique colors\n",
" \"\"\"\n",
" dcolors ={'[0]': 'aa4400',\n",
" '[0, 2]': '55227f',\n",
" '[0, 1]': '55a100',\n",
" '[1]': '00ff00',\n",
" '[1, 2]': '007f7f',\n",
" '[0, 1, 2]': '386b55',\n",
" '[2]': '0000ff'}\n",
"\n",
" return [\"#\"+dcolors[str(i)] for i in labels]\n",
"\n",
"def plot_data(data, circles, title, alpha=1.0,colors = []):\n",
" plt.figure(figsize=(14, 5))\n",
" done = set()\n",
" for i in range(0,len(data)):\n",
" lab = str(labels[i])\n",
" if lab in done:\n",
" label = \"\"\n",
" else:\n",
" label = lab\n",
" done.add(lab)\n",
" plt.scatter(data[i, 0], data[i, 1], c=colors[i], s=30,alpha=0.6, label = label)\n",
" for i in circles:\n",
" plt.plot(\n",
" data[i][0],\n",
" data[i][1],\n",
" \"o\",\n",
" markerfacecolor=\"none\",\n",
" markeredgecolor=\"red\",\n",
" markersize=14,\n",
" markeredgewidth=2.5,\n",
" alpha=alpha\n",
" )\n",
" _ = plt.title(title, fontsize=25)\n",
" plt.legend()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e8ff5c2f-bd52-44aa-b307-b2b634147c68",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"def make_multilabel_data(\n",
" means=[[-5, 3.5], [0, 2], [-3, 6]],\n",
" covs=[[[3, -1.5], [-1.5, 1]], [[5, -1.5], [-1.5, 1]], [[3, -1.5], [-1.5, 1]]],\n",
" boxes_coordinates=[[-3.5, 0, -1.5, 1.7], [-1, 3, 2, 4], [-5, 2, -3, 4], [-3, 2, -1, 4]],\n",
" box_multilabels=[[0, 1], [1, 2], [0, 2], [0, 1, 2]],\n",
" sizes=[100, 80, 100],\n",
" avg_trace=0.9,\n",
" seed=1,\n",
"):\n",
" np.random.seed(seed=seed)\n",
" num_classes = len(means)\n",
" m = num_classes + len(\n",
" box_multilabels\n",
" ) # number of classes by treating each multilabel as 1 unique label\n",
" n = sum(sizes)\n",
" local_data = []\n",
" labels = []\n",
" test_data = []\n",
" test_labels = []\n",
" for i in range(0, len(means)):\n",
" local_data.append(np.random.multivariate_normal(mean=means[i], cov=covs[i], size=sizes[i]))\n",
" test_data.append(np.random.multivariate_normal(mean=means[i], cov=covs[i], size=sizes[i]))\n",
" test_labels += [[i]] * sizes[i]\n",
" labels += [[i]] * sizes[i]\n",
"\n",
" def make_multi(X, Y, bx1, by1, bx2, by2, label_list):\n",
" ll = np.array([bx1, by1]) # lower-left\n",
" ur = np.array([bx2, by2]) # upper-right\n",
"\n",
" inidx = np.all(np.logical_and(X.tolist() >= ll, X.tolist() <= ur), axis=1)\n",
" for i in range(0, len(Y)):\n",
" if inidx[i]:\n",
" Y[i] = label_list\n",
" return Y\n",
"\n",
" X_train = np.vstack(local_data)\n",
" X_test = np.vstack(test_data)\n",
"\n",
" for i in range(0, len(box_multilabels)):\n",
" bx1, by1, bx2, by2 = boxes_coordinates[i]\n",
" multi_label = box_multilabels[i]\n",
" labels = make_multi(X_train, labels, bx1, by1, bx2, by2, multi_label)\n",
" test_labels = make_multi(X_test, test_labels, bx1, by1, bx2, by2, multi_label)\n",
"\n",
" d = {}\n",
" for i in labels:\n",
" if str(i) not in d:\n",
" d[str(i)] = len(d)\n",
" inv_d = {v: k for k, v in d.items()}\n",
" labels_idx = [d[str(i)] for i in labels]\n",
" py = np.bincount(labels_idx) / float(len(labels_idx))\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=avg_trace * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=seed,\n",
" )\n",
" noisy_labels_idx = generate_noisy_labels(labels_idx, noise_matrix)\n",
" noisy_labels = [eval(inv_d[i]) for i in noisy_labels_idx]\n",
" return {\n",
" \"X_train\": X_train,\n",
" \"true_labels_train\": labels,\n",
" \"X_test\": X_test,\n",
" \"true_labels_test\": test_labels,\n",
" \"labels\": noisy_labels,\n",
" \"dict_unique_label\": d,\n",
" 'labels_idx': noisy_labels_idx,\n",
"\n",
" }\n",
"\n",
"def get_color_array(labels):\n",
" \"\"\"\n",
" This function returns a dictionary mapping multi-labels to unique colors\n",
" \"\"\"\n",
" dcolors ={'[0]': 'aa4400',\n",
" '[0, 2]': '55227f',\n",
" '[0, 1]': '55a100',\n",
" '[1]': '00ff00',\n",
" '[1, 2]': '007f7f',\n",
" '[0, 1, 2]': '386b55',\n",
" '[2]': '0000ff'}\n",
"\n",
" return [\"#\"+dcolors[str(i)] for i in labels]\n",
"\n",
"def plot_data(data, circles, title, alpha=1.0,colors = []):\n",
" plt.figure(figsize=(14, 5))\n",
" done = set()\n",
" for i in range(0,len(data)):\n",
" lab = str(labels[i])\n",
" if lab in done:\n",
" label = \"\"\n",
" else:\n",
" label = lab\n",
" done.add(lab)\n",
" plt.scatter(data[i, 0], data[i, 1], c=colors[i], s=30,alpha=0.6, label = label)\n",
" for i in circles:\n",
" plt.plot(\n",
" data[i][0],\n",
" data[i][1],\n",
" \"o\",\n",
" markerfacecolor=\"none\",\n",
" markeredgecolor=\"red\",\n",
" markersize=14,\n",
" markeredgewidth=2.5,\n",
" alpha=alpha\n",
" )\n",
" _ = plt.title(title, fontsize=25)\n",
" plt.legend()"
]
},
{
"cell_type": "markdown",
"id": "672bfc2a",
"metadata": {},
"source": [
"Some of the labels in our generated dataset purposely contain errors. The examples with label errors are circled in the plot below, which depicts the dataset. This dataset contains 3 classes, and any subset of these may be the given label for a particular example. We say this example has a label error if it is better described by an alternative subset of the classes than the given label."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dac65d3b-51e8-4682-b829-beab610b56d6",
"metadata": {},
"outputs": [],
"source": [
"num_class = 3\n",
"dataset = make_multilabel_data()\n",
"labels = dataset['labels']\n",
"true_errors = np.where(np.sum(int2onehot(dataset['true_labels_train'],3)!=int2onehot(dataset['labels'],3),axis=1)>=1)[0]\n",
"plot_data(dataset['X_train'], circles=true_errors, title=f\"True label errors in multi-label dataset with {num_class} classes\", colors = get_color_array(labels),alpha=0.5)"
]
},
{
"cell_type": "markdown",
"id": "144ad4c2-49bb-4147-a743-a83ed1656a11",
"metadata": {},
"source": [
"## 2. Format data, labels, and model predictions\n",
"\n",
"In multi-label classification, each example in the dataset is labeled as belonging to one **or more** of *K* possible classes (or none of the classes at all). To find label issues, cleanlab requires predicted class probabilities from a trained classifier. \n",
"Here we produce out-of-sample `pred_probs` by employing cross-validation to fit a multi-label **RandomForestClassifier** model via sklearn's [OneVsRestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html) framework. \n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n",
"`OneVsRestClassifier` offers an easy way to apply any multi-class classifier model from sklearn to multi-label classification tasks. It is done for simplicity here, but we advise against this approach as it does not properly model dependencies between classes.\n",
"\n",
"To instead train a state-of-the-art Pytorch neural network for multi-label classification and produce `pred_probs` on a real image dataset (that properly account for dependencies between classes), see our [example](https://github.com/cleanlab/examples) notebook [\"Train a neural network for multi-label classification on the CelebA dataset\"](https://github.com/cleanlab/examples/blob/master/multilabel_classification/pytorch_network_training.ipynb). "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b5fa99a9-2583-4cd0-9d40-015f698cdb23",
"metadata": {},
"outputs": [],
"source": [
"SEED = 0\n",
"random.seed(SEED)\n",
"y_onehot = int2onehot(labels, K=num_class) # labels in a binary format for sklearn OneVsRestClassifier\n",
"single_class_labels = [random.choice(i) for i in labels] # used only for stratifying the cross-validation split \n",
"clf = OneVsRestClassifier(RandomForestClassifier(random_state=SEED))\n",
"pred_probs = np.zeros(shape=(len(labels), num_class))\n",
"kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=SEED)\n",
"\n",
"for train_index, test_index in kf.split(X=dataset['X_train'], y=single_class_labels):\n",
" clf_cv = sklearn.base.clone(clf)\n",
" X_train_cv, X_test_cv = dataset['X_train'][train_index], dataset['X_train'][test_index]\n",
" y_train_cv, y_test_cv = y_onehot[train_index], y_onehot[test_index]\n",
" clf_cv.fit(X_train_cv, y_train_cv)\n",
" y_pred_cv = clf_cv.predict_proba(X_test_cv)\n",
" pred_probs[test_index] = y_pred_cv"
]
},
{
"cell_type": "markdown",
"id": "41c1efab",
"metadata": {},
"source": [
"`pred_probs` should be 2D array whose rows are length-*K* vectors for **each** example in the dataset, representing the model-estimated probability that this example belongs to each class. Since one example can belong to multiple classes in multi-label classification, these probabilities need not sum to 1. For the best label error detection performance, these `pred_probs` should be out-of-sample (from a copy of the model that never saw this example during training, e.g. produced via cross-validation).\n",
"\n",
"`labels` should be a list of lists, whose *i*-th entry is a list of (integer) class indices that apply to the *i*-th example in the dataset. If your classes are represented as string names, you should map these to integer indices. The label for an example that belongs to none of the classes should just be an empty list `[]`.\n",
"\n",
"Once you have `pred_probs` and `labels` appropriately formatted, you can find/analyze label issues in any multi-label dataset via `Datalab`!\n",
"\n",
"Here's what these look like for the first few examples in our synthetic multi-label dataset: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac1a60df",
"metadata": {},
"outputs": [],
"source": [
"num_to_display = 3 # increase this to see more examples\n",
"\n",
"print(f\"labels for first {num_to_display} examples in format expected by cleanlab:\")\n",
"print(labels[:num_to_display])\n",
"print(f\"pred_probs for first {num_to_display} examples in format expected by cleanlab:\")\n",
"print(pred_probs[:num_to_display])"
]
},
{
"cell_type": "markdown",
"id": "5a973506-c30e-4409-ac65-495537d13730",
"metadata": {},
"source": [
"## 3. Use cleanlab to find label issues \n",
"\n",
"Based on the given `labels` and `pred_probs` from a trained model, cleanlab can quickly help us find label errors in our dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d09115b6-ad44-474f-9c8a-85a459586439",
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(\n",
" data={\"labels\": labels},\n",
" label_name=\"labels\",\n",
" task=\"multilabel\",\n",
")\n",
"\n",
"lab.find_issues(\n",
" pred_probs=pred_probs,\n",
" issue_types={\"label\": {}}\n",
")"
]
},
{
"cell_type": "markdown",
"id": "439c003e",
"metadata": {},
"source": [
" Here we request that the indices of the examples identified with label issues be sorted by cleanlab’s self-confidence score, which is used to measure the quality of individual labels. The returned `issues` are a list of indices corresponding to the examples in your dataset that cleanlab finds most likely to be mislabeled. These indices are sorted by the *self-confidence* label quality score, with the lowest quality labels at the start."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c18dd83b",
"metadata": {},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"\n",
"issues = label_issues.query(\"is_label_issue\").sort_values(\"label_score\").index.values\n",
"\n",
"print(f\"Indices of examples with label issues:\\n{issues}\")"
]
},
{
"cell_type": "markdown",
"id": "d6af5833",
"metadata": {},
"source": [
"Let's look at the samples that cleanlab thinks are most likely to be mislabeled. You can see that cleanlab was able to identify most of `true_errors` in our small dataset (despite not having access to this variable, which you won't have in your own applications)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fffa88f6-84d7-45fe-8214-0e22079a06d1",
"metadata": {},
"outputs": [],
"source": [
"plot_data(dataset['X_train'], circles=issues, title=f\"Inferred label issues in multi-label dataset with {num_class} classes\", colors = get_color_array(labels), alpha = 1)"
]
},
{
"cell_type": "markdown",
"id": "32465521",
"metadata": {},
"source": [
"### Label quality scores\n",
"\n",
"The above code identifies which examples have label issues and sorts them by their label quality score. We can also take a look at this label quality score for each example in the dataset, which estimates our confidence that this example has been correctly labeled. These scores range between 0 and 1 with smaller values indicating examples whose label seems more suspect."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1198575",
"metadata": {},
"outputs": [],
"source": [
"scores = label_issues[\"label_score\"].values\n",
"\n",
"print(f\"Label quality scores of the first 10 examples in dataset:\\n{scores[:10]}\")"
]
},
{
"cell_type": "markdown",
"id": "d65af827-aeda-4b6b-9ae7-b1f0b84700d6",
"metadata": {},
"source": [
"### Data issues beyond mislabeling (outliers, duplicates, drift, ...)\n",
"\n",
"While this tutorial focused on label issues, cleanlab's `Datalab` object can automatically detect many other types of issues in your dataset (outliers, near duplicates, drift, etc).\n",
"Simply remove the `issue_types` argument from the above call to `Datalab.find_issues()` above and `Datalab` will more comprehensively audit your dataset.\n",
"Refer to our [Datalab quickstart tutorial](./datalab/datalab_quickstart.html) to learn how to interpret the results (the interpretation remains mostly the same across different types of ML tasks)."
]
},
{
"cell_type": "markdown",
"id": "d65af827-aeda-4b6b-9ae7-b1f0b84700d5",
"metadata": {},
"source": [
"### How to format labels given as a one-hot (multi-hot) binary matrix?\n",
"\n",
"For multi-label classification, cleanlab expects labels to be formatted as a list of lists, where each entry is an integer corresponding to a particular class. Here are some functions you can use to easily convert labels between this format and a binary matrix format commonly used to train multi-label classification models."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49161b19-7625-4fb7-add9-607d91a7eca1",
"metadata": {},
"outputs": [],
"source": [
"labels_binary_format = int2onehot(labels, K=num_class)\n",
"labels_list_format = onehot2int(labels_binary_format)"
]
},
{
"cell_type": "markdown",
"id": "a58200c8",
"metadata": {},
"source": [
"### Estimate label issues without Datalab \n",
"If you prefer to directly run the same lower-level mathematical functions Datalab uses to detect label issues, you can do so outside of Datalab via the methods in the `cleanlab.multilabel_classification` module such as: [multilabel_classification.filter.find_label_issues](../cleanlab/multilabel_classification/filter.html#cleanlab.multilabel_classification.filter.find_label_issues), [multilabel_classification.rank.get_label_quality_scores](../cleanlab/multilabel_classification/rank.html#cleanlab.multilabel_classification.rank.get_label_quality_scores) \n",
"\n",
"### Application to Real Data \n",
"\n",
"To see cleanlab applied to a real image tagging dataset, check out our [example](https://github.com/cleanlab/examples) notebook [\"Find Label Errors in Multi-Label Classification Data (CelebA Image Tagging)\"](https://github.com/cleanlab/examples/blob/master/multilabel_classification/image_tagging.ipynb). That example also demonstrates how to use a state-of-the-art Pytorch neural network for multi-label classification with image data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d1a2c008",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"A = set(issues)\n",
"B = set(true_errors)\n",
"jaccard = len(A.intersection(B)) / len(A.union(B))\n",
"if not jaccard > 0.7:\n",
" raise Exception(\"issues does not overlap much with the true errors\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/object_detection.ipynb
+758
View File
@@ -0,0 +1,758 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d299c1e8",
"metadata": {},
"source": [
"# Finding Label Errors in Object Detection Datasets\n",
"\n",
"This 5-minute quickstart tutorial demonstrates how to find potential label errors in object detection datasets. In object detection data, each image is annotated with multiple bounding boxes. Each bounding box surrounds a physical object within an image scene, and is annotated with a given class label. \n",
"\n",
"Using such labeled data, we train a model to predict the locations and classes of objects in an image. An example notebook to train the object detection model whose predictions we rely on in this tutorial is available [here](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training.ipynb). These predictions can subsequently be input to cleanlab in order to identify mislabeled images and a quality score quantifying our confidence in the overall annotations for each image. \n",
"\n",
"After correcting these label issues, **you can train an even better version of your model without changing your training code!**\n",
"\n",
"This tutorial uses a subset of the [COCO (Common Objects in Context)](https://cocodataset.org/#home) dataset which has images of everyday scenes and considers objects from the 5 most popular classes: car, chair, cup, person, traffic light.\n",
"\n",
"**Overview of what we we'll do in this tutorial**\n",
"\n",
"- Score images based on their overall label quality (i.e. our confidence each image is correctly labeled) using `cleanlab.object_detection.rank.get_label_quality_scores`\n",
"- Estimate which images have label issues using `cleanlab.object_detection.filter.find_label_issues`\n",
"- Visually review images + labels using `cleanlab.object_detection.summary.visualize`\n",
"\n",
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have `labels` and `predictions` in the proper format? Just run the code below to find label issues in your object detection dataset.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.object_detection.filter import find_label_issues\n",
"from cleanlab.object_detection.rank import get_label_quality_scores\n",
"\n",
"# To get boolean vector of label issues for all images\n",
"has_label_issue = find_label_issues(labels, predictions)\n",
"\n",
"# To get label quality scores for all images\n",
"label_quality_scores = get_label_quality_scores(labels, predictions)\n",
" \n",
" \n",
"```\n",
"\n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "8d552ab9",
"metadata": {},
"source": [
"## 1. Install required dependencies and download data\n",
"You can use `pip` to install all packages required for this tutorial as follows\n",
"```ipython\n",
"!pip install matplotlib\n",
"!pip install cleanlab\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0ba0dc70",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"huggingface_hub\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c90449c8",
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"from huggingface_hub import hf_hub_download\n",
"import zipfile\n",
"import os\n",
"\n",
"# Download files from HuggingFace Hub\n",
"labels_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'labels.pkl', repo_type=\"dataset\")\n",
"predictions_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'predictions.pkl', repo_type=\"dataset\")\n",
"images_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'example_images.zip', repo_type=\"dataset\")\n",
"\n",
"# Extract images to the correct location\n",
"# The zip file contains an example_images/ folder, so we extract it and then move contents\n",
"with zipfile.ZipFile(images_path, 'r') as zip_ref:\n",
" zip_ref.extractall('temp_extract/')\n",
"\n",
"# Move images from nested directory to target directory \n",
"import shutil\n",
"if os.path.exists('temp_extract/example_images/'):\n",
" if os.path.exists('example_images/'):\n",
" shutil.rmtree('example_images/')\n",
" shutil.move('temp_extract/example_images/', 'example_images/')\n",
" shutil.rmtree('temp_extract/')\n",
"else:\n",
" # Fallback: extract directly if structure is different\n",
" with zipfile.ZipFile(images_path, 'r') as zip_ref:\n",
" zip_ref.extractall('example_images/')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "df8be4c6",
"metadata": {},
"outputs": [],
"source": [
"import pickle\n",
"from cleanlab.object_detection.filter import find_label_issues\n",
"from cleanlab.object_detection.rank import (\n",
" _separate_label,\n",
" _separate_prediction,\n",
" get_label_quality_scores,\n",
" issues_from_scores,\n",
")\n",
"from cleanlab.object_detection.summary import visualize "
]
},
{
"cell_type": "markdown",
"id": "2506badc",
"metadata": {},
"source": [
"## 2. Format data, labels, and model predictions\n",
"\n",
"We begin by loading `labels` and `predictions` for our dataset, which are the only inputs required to find label issues with cleanlab. Note that the predictions should be **out-of-sample**, which can be obtained for every image in a dataset via K-fold cross-validation. \n",
"\n",
"In a separate [example](https://github.com/cleanlab/examples) notebook ([link](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training.ipynb)), we trained a Detectron2 object detection model and used it to obtain predictions on a held-out validation dataset\u00a0whose `labels` we audit here.\n",
"\n",
"**Note:** If you want to find all the mislabeled images across the entire COCO dataset, you can first execute our [other example notebook](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training-kfold.ipynb) that uses K-fold cross-validation to produce **out-of-sample** predictions for every image, then use those labels and predictions below."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2e9ffd6f",
"metadata": {},
"outputs": [],
"source": [
"IMAGE_PATH = './example_images/' # path to raw image files downloaded above\n",
"\n",
"# Load pickle files\n",
"with open(labels_path, 'rb') as f:\n",
" labels = pickle.load(f)\n",
"with open(predictions_path, 'rb') as f:\n",
" predictions = pickle.load(f)"
]
},
{
"cell_type": "markdown",
"id": "35d49e5d",
"metadata": {},
"source": [
"In object detection datasets, each given label is a made up of bounding box coordinates and a class label. A model prediction is also made up of a bounding box and predicted class label, as well as the model confidence (probability estimate) in its prediction. To detect label issues, cleanlab requires given labels for each image, and the corresponding model predictions for the image (but not the image itself).\n",
"\n",
"Here\u2019s what an example looks like in our dataset. We visualize the given and predicted labels (in red and blue) for this image using the `cleanlab.object_detection.summary.visualize` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "56705562",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import os\n",
"\n",
"image_to_visualize = 8 # change this to view other images\n",
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
"\n",
"# Check if the image file exists before trying to visualize\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=labels[image_to_visualize], prediction=predictions[image_to_visualize], overlay=False)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "ff36d97f",
"metadata": {},
"source": [
"The required format of these `labels` and `predictions` matches what popular object detection frameworks like [MMDetection](https://github.com/open-mmlab/mmdetection) and [Detectron2](https://github.com/facebookresearch/detectron2/) expect. Recall the 5 possible class labels in our dataset are: car, chair, cup, person, traffic light. These classes are represented as (zero-indexed) integers 0,1,...,4.\n",
"\n",
"`labels` is a list where for the i-th image in our dataset, `labels[i]` is a dictionary containing: key `labels` -- a list of class labels for each bounding box in this image and key `bboxes` -- a numpy array of the bounding boxes' coordinates. Each bounding box in `labels[i]['bboxes']` is in the format ``[x1,y1,x2,y2]`` format with respect to the image matrix where `(x1,y1)` corresponds to the top-left corner of the box and `(x2,y2)` the bottom-right (E.g. [XYXY in Keras](https://keras.io/api/keras_cv/bounding_box/formats/), [Detectron 2](https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box)).\n",
"\n",
"\n",
"Let's see what `labels[i]` looks like for our previous example image:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b08144d7",
"metadata": {},
"outputs": [],
"source": [
"labels[image_to_visualize]"
]
},
{
"cell_type": "markdown",
"id": "8f62da67",
"metadata": {},
"source": [
"`predictions` is a list where the predictions output by our model for the i-th image: `predictions[i]` is a list/array of shape `(K,)`. Here `K` is the number of classes in the dataset (same for every image) and `predictions[i][k]` is of shape `(M,5)`, where `M` is the number of bounding boxes predicted to contain objects of class `k` (in image i, differs between images). The five columns of `predictions[i][k]` correspond to ``[x1,y1,x2,y2,pred_prob]`` format with respect to the image matrix for each bounding box predicted by the model. Here `(x1,y1)` corresponds to the top-left corner of the box and `(x2,y2)` the bottom-right (E.g. [XYXY in Keras](https://keras.io/api/keras_cv/bounding_box/formats/), [Detectron 2](https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box)). The last column, `pred_prob` is the model confidence in its predicted label of class `k` for this box. Since our dataset has `K = 5` classes, we have: `predictions[i].shape = (5,)`.\n",
"\n",
"Let's see what `predictions[i]` looks like for our previous example image:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d70bec6",
"metadata": {},
"outputs": [],
"source": [
"predictions[image_to_visualize]"
]
},
{
"cell_type": "markdown",
"id": "cf95ea28",
"metadata": {},
"source": [
"\n",
"Once you have `labels` and `predictions` in the appropriate formats, you can **find label issues with cleanlab for any object detection dataset**!"
]
},
{
"cell_type": "markdown",
"id": "3daff923",
"metadata": {},
"source": [
"## 3. Use cleanlab to find label issues\n",
"Given `labels` and `predictions` from our trained model, cleanlab can automatically find mislabeled images in the dataset. In object detection, we consider an image mislabeled if **any** of its bounding boxes or their class labels are incorrect (including if the image contains any overlooked objects which should've been annotated with a box)\n",
"\n",
"Images may be mislabeled because annotators:\n",
"\n",
"- overlooked an object (forgot to annotate a bounding box around a depicted object)\n",
"- chose the wrong class label for an annotated box in the correct location\n",
"- imperfectly drew the bounding box such that its location is incorrect\n",
"\n",
"\n",
"Cleanlab is expected to flag images that exhibit **any** of these annotation errors as having label issues. More severe annotation errors are expected to produce lower cleanlab label quality scores closer to 0. Let's first estimate which images have label issues:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4caa635d",
"metadata": {},
"outputs": [],
"source": [
"label_issue_idx = find_label_issues(labels, predictions, return_indices_ranked_by_score=True)\n",
"\n",
"num_examples_to_show = 5 # view this many images flagged with the most severe label issues\n",
"label_issue_idx[:num_examples_to_show]"
]
},
{
"cell_type": "markdown",
"id": "66d5fae1",
"metadata": {},
"source": [
"The above code identifies *which* images have label issues, returning a list of their indices. This is because we specified the `return_indices_ranked_by_score` argument which sorts these indices by the estimated label quality of each image. Below we describe how to directly estimate the label quality scores of each image.\n",
"\n",
"**Note:** You can omit the `return_indices_ranked_by_score` argument for `find_label_issues()` to instead return a Boolean mask for the entire dataset (True entries in this mask correspond to images with label issues)"
]
},
{
"cell_type": "markdown",
"id": "5b501dc9",
"metadata": {},
"source": [
"### Get label quality scores\n",
"Cleanlab can also compute scores for each image to estimate our confidence that it has been correctly labeled. These label quality scores range between 0 and 1, with *smaller* values indicating examples whose annotation is *more* likely to be wrong in some way.\n",
"\n",
"Each image in the dataset receives a label quality score. These scores are useful for prioritizing which images to review; if you have too little time, first review the images with the lowest label quality scores."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a9b4c590",
"metadata": {},
"outputs": [],
"source": [
"scores = get_label_quality_scores(labels, predictions)\n",
"scores[:num_examples_to_show]"
]
},
{
"cell_type": "markdown",
"id": "349521e0",
"metadata": {},
"source": [
"We can also use the label quality scores to flag *which* images have label issues based on a threshold. Here we convert these per-image scores into an array of indices corresponding to images flagged with label issues, sorted by label quality score, in the same format returned by `find_label_issues()`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ffd9ebcc",
"metadata": {},
"outputs": [],
"source": [
"issue_idx = issues_from_scores(scores, threshold=0.5) # lower threshold will return fewer (but more confident) label issues\n",
"issue_idx[:num_examples_to_show], scores[issue_idx][:num_examples_to_show]"
]
},
{
"cell_type": "markdown",
"id": "5a3b8aa0",
"metadata": {},
"source": [
"## 4. Use ObjectLab to visualize label issues\n",
"Finally, we can visualize images with potential label errors via cleanlab's `visualize()` function. To enhance the visualization, you can supply a `class_names` dictionary to include as a legend and turn off `overlay` to see the given and predicted labels side by side."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4dd46d67",
"metadata": {},
"outputs": [],
"source": [
"issue_to_visualize = issue_idx[0] # change this to view other images\n",
"class_names = {\"0\": \"car\", \"1\": \"chair\", \"2\": \"cup\", \"3\":\"person\", \"4\": \"traffic light\"}\n",
"\n",
"label = labels[issue_to_visualize]\n",
"prediction = predictions[issue_to_visualize]\n",
"score = scores[issue_to_visualize]\n",
"image_path = IMAGE_PATH + label['seg_map']\n",
"\n",
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "de0d7205",
"metadata": {},
"source": [
"The visualization depicts the given label (original image annotation which cleanlab identified as problematic) in red on the left and the model-predicted label in blue on the right. Each bounding box contains a class-index number in the top corner indicating which object class that bounding box was annotated/predicted to contain.\n",
"\n",
"This image has a **low** label quality score and is marked as an error. On closer inspection we notice the annotator missed the reflection of the person in the mirror that the model identified. Additionally, the chairs visible in the reflection were not annotated.\n",
"\n",
"Notice examples where the predictions and labels are more similar have higher quality scores than those that are missmatched, and are less likeley to be marked as issues and the number of boxes is agnostic to the score.\n",
"\n",
"Better trained models will lead to better label error detection but you don't need a near perfect model to identify label issues.\n",
"\n",
"\n",
"### Different kinds of label issues identified by ObjectLab\n",
"Now lets view the first few images in our vaidation dataset that are clearly marked as issues and see what various inconsistencies between the `given` and `predicted` label we can spot. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ceec2394",
"metadata": {},
"outputs": [],
"source": [
"issue_to_visualize = issue_idx[1]\n",
"label = labels[issue_to_visualize]\n",
"prediction = predictions[issue_to_visualize]\n",
"score = scores[issue_to_visualize]\n",
"\n",
"image_path = IMAGE_PATH + label['seg_map']\n",
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "9b5c87fa",
"metadata": {},
"source": [
"Notice the armchair to the left of the TV is missing an annotation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "94f82b0d",
"metadata": {},
"outputs": [],
"source": [
"issue_to_visualize = issue_idx[9]\n",
"label = labels[issue_to_visualize]\n",
"prediction = predictions[issue_to_visualize]\n",
"score = scores[issue_to_visualize]\n",
"\n",
"image_path = IMAGE_PATH + label['seg_map']\n",
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "05610be0",
"metadata": {},
"source": [
"Similarly, the woman in a red jacket in the foreground is missing an annotation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1ea18c5d",
"metadata": {},
"outputs": [],
"source": [
"issue_to_visualize = issue_idx[2]\n",
"label = labels[issue_to_visualize]\n",
"prediction = predictions[issue_to_visualize]\n",
"score = scores[issue_to_visualize]\n",
"\n",
"image_path = IMAGE_PATH + label['seg_map']\n",
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "05c9229d",
"metadata": {},
"source": [
"The people in this image should have had individual bounding boxes around each persons (the COCO guidelines state only groups with 10+ objects of the same type can be a \\\"crowd\\\" bounded by a single box). Individuals in the back are missing annotations.\n",
"\n",
"All of these examples received low label quality scores reflecting their low annotation quality in the original dataset."
]
},
{
"cell_type": "markdown",
"id": "03d5a521",
"metadata": {},
"source": [
"### Other uses of visualize\n",
"The `visualize()` function can also depict non-issue images, labels or predictions alone, or just the image itself. Let's explore this with a few images in our dataset.\n",
"\n",
"We can save a visualization to file via the `save_path` argument. Note the label quality score is high for this example and it is marked as a non-issue. The given and predicted labels closely resemble each other contributing to the high score."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e770d23",
"metadata": {},
"outputs": [],
"source": [
"image_to_visualize = 0\n",
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=labels[image_to_visualize], prediction=predictions[image_to_visualize], class_names=class_names, save_path='./example_image.png')\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "6c9464e8",
"metadata": {},
"source": [
"For the next example, notice how we are only passing in the given labels to visualize. We can limit visualization to either labels, predictions, or neither."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "57e84a27",
"metadata": {},
"outputs": [],
"source": [
"image_to_visualize = 3\n",
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
"if os.path.exists(image_path):\n",
" visualize(image_path, label=labels[image_to_visualize], class_names=class_names)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "d8744ab9",
"metadata": {},
"source": [
"For completeness, let's just look at an image alone."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0302818a",
"metadata": {},
"outputs": [],
"source": [
"image_to_visualize = 2\n",
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
"if os.path.exists(image_path):\n",
" visualize(image_path)\n",
"else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "46d6282a-4601-4cc3-b8a8-187ea6d5f8bc",
"metadata": {},
"source": [
"## Exploratory data analysis\n",
"\n",
"This bonus section considers techniques to uncover annotation irregularities through exploratory data analysis. Specifically, we consider anomalies in object sizes, detect images with unusual object counts, and examine the distribution of class labels.\n",
"\n",
"Let's first consider the number of objects per image, and inspect the images with the largest values (which might reveal something off in our dataset):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5cacec81-2adf-46a8-82c5-7ec0185d4356",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from cleanlab.internal.object_detection_utils import calculate_bounding_box_areas\n",
"from cleanlab.object_detection.summary import (\n",
" bounding_box_size_distribution,\n",
" class_label_distribution,\n",
" object_counts_per_image,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3335b8a3-d0b4-415a-a97d-c203088a124e",
"metadata": {},
"outputs": [],
"source": [
"num_imgs_to_show = 3\n",
"lab_object_counts,pred_object_counts = object_counts_per_image(labels,predictions)\n",
"for image_to_visualize in np.argsort(lab_object_counts)[::-1][0:num_imgs_to_show]:\n",
" image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
" print(image_path, '| idx', image_to_visualize)\n",
" if os.path.exists(image_path):\n",
" visualize(image_path, label=labels[image_to_visualize], class_names=class_names)\n",
" else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")"
]
},
{
"cell_type": "markdown",
"id": "e5ddd4fe-4477-4b68-ba79-e5cbb62822eb",
"metadata": {},
"source": [
"Next let's study the distribution of class labels in the overall annotations, comparing the distribution in the given annotations vs. in the model predictions. This can sometimes reveal that something's off in our dataset or model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d4b7677-6ebd-447d-b0a1-76e094686628",
"metadata": {},
"outputs": [],
"source": [
"label_norm,pred_norm = class_label_distribution(labels,predictions)\n",
"print(\"Frequency of each class amongst annotated | predicted bounding boxes in the dataset:\\n\")\n",
"for i in label_norm:\n",
" print(f\"{class_names[str(i)]} : {label_norm[i]} | {pred_norm[i]}\")"
]
},
{
"cell_type": "markdown",
"id": "200cdebf-b24c-4c2b-8914-6a2fce218daf",
"metadata": {},
"source": [
"Finally, let's consider the distribution of bounding box sizes (aka object sizes) in the given annotations for each class label. The idea is to review any anomalies in bounding box areas for a given class (which might reveal problematic annotations or abnormal instances of this object class). The following code determines such anomalies by assessing each bounding box's area vs. the mean and standard deviation of areas for bounding boxes with the same class label."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "59d7ee39-3785-434b-8680-9133014851cd",
"metadata": {},
"outputs": [],
"source": [
"lab_area,pred_area = bounding_box_size_distribution(labels,predictions)\n",
"lab_area_mean = {i: np.mean(lab_area[i]) for i in lab_area.keys()}\n",
"lab_area_std = {i: np.std(lab_area[i]) for i in lab_area.keys()}\n",
"\n",
"max_deviation_values = []\n",
"max_deviation_classes = []\n",
"\n",
"for label in labels:\n",
" bounding_boxes, label_names = _separate_label(label)\n",
" areas = calculate_bounding_box_areas(bounding_boxes)\n",
" deviation_values = []\n",
" deviation_classes = []\n",
"\n",
" for class_name, mean_area, std_area in zip(lab_area_mean.keys(), lab_area_mean.values(), lab_area_std.values()):\n",
" class_areas = areas[label_names == class_name]\n",
" deviations_away = (class_areas - mean_area) / std_area\n",
" deviation_values.extend(list(deviations_away))\n",
" deviation_classes.extend([class_name] * len(class_areas))\n",
"\n",
" if deviation_values==[]:\n",
" max_deviation_values.append(0.0)\n",
" max_deviation_classes.append(-1)\n",
" else:\n",
" max_deviation_index = np.argmax(np.abs(deviation_values))\n",
" max_deviation_values.append(deviation_values[max_deviation_index])\n",
" max_deviation_classes.append(deviation_classes[max_deviation_index])\n",
"\n",
"max_deviation_classes, max_deviation_values = np.array(max_deviation_classes), np.array(max_deviation_values)"
]
},
{
"cell_type": "markdown",
"id": "b260142e-b760-490c-818e-c037fab5c6c8",
"metadata": {},
"source": [
"In our dataset here, this analysis reveals certain abnormally large bounding boxes that take up most of the image."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47b6a8ff-7a58-4a1f-baee-e6cfe7a85a6d",
"metadata": {},
"outputs": [],
"source": [
"num_imgs_to_show_per_class = 3\n",
"\n",
"for c in class_names.keys():\n",
" class_num = int(c)\n",
" sorted_indices = np.argsort(max_deviation_values)[::-1]\n",
" count = 0\n",
"\n",
" for i in sorted_indices:\n",
" if max_deviation_values[i] == 0 or max_deviation_classes[i] != class_num:\n",
" continue\n",
" image_path = IMAGE_PATH + labels[i]['seg_map']\n",
" print(image_path, '| idx', i, '| class', class_names[c])\n",
" if os.path.exists(image_path):\n",
" visualize(image_path, label=labels[i], class_names=class_names)\n",
" else:\n",
" print(f\"Image file not found: {image_path}\")\n",
" print(\"Skipping visualization - this is expected during documentation build.\")\n",
"\n",
" count += 1\n",
" if count == num_imgs_to_show_per_class:\n",
" break # Break the loop after visualizing the top 3 instances for the current class"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8ce74938",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"expected_values = {0: 50, 1: 16, 2: 31, 9: 62}\n",
"\n",
"for idx, value in expected_values.items():\n",
" assert value in issue_idx and issue_idx[idx] == value, f\"Assertion error at index {idx}: Expected {value}, got {issue_idx.get(idx, None)}\"\n",
"\n",
"assert all(i not in issue_idx for i in [0, 2, 3]), \"Unexpected values found in issue_idx\""
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/outliers.ipynb
+718
View File
@@ -0,0 +1,718 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1043b220",
"metadata": {},
"source": [
"# Detect Outliers with Cleanlab and PyTorch Image Models (timm)\n",
"\n",
"This quick tutorial shows how to detect outliers (out-of-distribution examples) in image data, using the [cifar10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset as an example. You can easily replace the image dataset + neural network used here with any other Pytorch dataset + neural network (e.g. to instead detect outliers in text data with minimal code changes). \n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"Detect outliers using `feature_embeddings`\n",
"\n",
"- Pre-process [cifar10](https://www.cs.toronto.edu/~kriz/cifar.html) into Pytorch datasets where `train_data` only contains images of animals and `test_data` contains images from all classes.\n",
"\n",
"- Use a pretrained neural network model from [timm](https://github.com/rwightman/pytorch-image-models) to extract feature embeddings of each image.\n",
"\n",
"- Use cleanlab to find naturally occurring outlier examples in the `train_data` (i.e. atypical images).\n",
"\n",
"- Find outlier examples in the `test_data` that do not stem from training data distribution (including out-of-distribution non-animal images).\n",
"\n",
"- Explore threshold selection for determining which images are outliers vs not.\n",
"\n",
"Detect outliers using `pred_probs` from a trained classifier\n",
"\n",
"- Adapt our [timm](https://github.com/rwightman/pytorch-image-models) network into a classifier by training an additional output layer using the (in-distribution) training data.\n",
"\n",
"- Use cleanlab to find out-of-distribution examples in the dataset based on the probabilistic predictions of this classifier, as an alternative to relying on feature embeddings."
]
},
{
"cell_type": "markdown",
"id": "70016f64",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have numeric **feature embeddings** for your data? Just run the code below to score how out-of-distribution each example is.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.outlier import OutOfDistribution\n",
" \n",
"ood = OutOfDistribution()\n",
"\n",
"# To get outlier scores for train_data using feature matrix train_feature_embeddings\n",
"ood_train_feature_scores = ood.fit_score(features=train_feature_embeddings)\n",
"\n",
"# To get outlier scores for additional test_data using feature matrix test_feature_embeddings\n",
"ood_test_feature_scores = ood.score(features=test_feature_embeddings)\n",
" \n",
" \n",
"```\n",
"\n",
"</div>\n",
" \n",
"Already have `pred_probs` and `labels` for your classification dataset? Just run the code below to to score how out-of-distribution each example is.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.outlier import OutOfDistribution\n",
" \n",
"ood = OutOfDistribution()\n",
"\n",
"# To get outlier scores for train_data using predicted class probabilities (from a trained classifier) and given class labels\n",
"ood_train_predictions_scores = ood.fit_score(pred_probs=train_pred_probs, labels=labels)\n",
"\n",
"# To get outlier scores for additional test_data using predicted class probabilities\n",
"ood_test_predictions_scores = ood.score(pred_probs=test_pred_probs)\n",
" \n",
" \n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "45cb0f90",
"metadata": {},
"source": [
"## 1. Install the required dependencies\n",
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib torch torchvision timm\n",
"!pip install cleanlab\n",
"...\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bbebfc8",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
"# Package versions we used: matplotlib==3.5.1, torch==2.1.2, torchvision==2.1.2, timm==0.6.12\n",
"\n",
"dependencies = [\"matplotlib\", \"torch\", \"torchvision\", \"timm\", \"cleanlab\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "markdown",
"id": "41733949",
"metadata": {},
"source": [
"Let's first import the required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4396f544",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from pylab import rcParams\n",
"import torch\n",
"import torchvision\n",
"import timm\n",
"from sklearn import preprocessing\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.ensemble import BaggingClassifier\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"from cleanlab.outlier import OutOfDistribution\n",
"from cleanlab.rank import find_top_issues"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3792f82e",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This (optional) cell is hidden from docs.cleanlab.ai \n",
"# Set some seeds for reproducibility. \n",
"\n",
"SEED = 42\n",
"np.random.seed(SEED)\n",
"torch.manual_seed(SEED)\n",
"torch.backends.cudnn.deterministic = True\n",
"torch.backends.cudnn.benchmark = False\n",
"torch.cuda.manual_seed_all(SEED)"
]
},
{
"cell_type": "markdown",
"id": "be38283d",
"metadata": {},
"source": [
"## 2. Pre-process the Cifar10 dataset\n",
"\n",
"Each image in the original [cifar10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) belongs to 1 of 10 classes: `[airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck]`. \n",
"After loading the data and processing the images, we manually remove some classes from the training dataset thereby making images from these classes outliers in the test dataset. Here we to remove all classes that are not an animal, such that test images from the following classes would be out-of-distribution: `[airplane, automobile, ship, truck]`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fd853a54",
"metadata": {},
"outputs": [],
"source": [
"# Load cifar10 images into tensors for training (rescales pixel values to [0,1] interval):\n",
"transform_normalize = torchvision.transforms.Compose(\n",
" [torchvision.transforms.ToTensor(),])\n",
"\n",
"train_data = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
" download=True, transform=transform_normalize)\n",
"test_data = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
" download=True, transform=transform_normalize)\n",
"\n",
"# Define in (animal) vs out (non-animal) of distribution labels\n",
"animal_classes = [2,3,4,5,6,7] # labels correspond to animal images\n",
"non_animal_classes = [0,1,8,9] # labels that correspond to non-animal images\n",
"\n",
"# Remove non-animal images from the training dataset\n",
"animal_idxs = np.where(np.isin(train_data.targets, animal_classes))[0]\n",
"\n",
"# Only work with small subset of each dataset to speedup tutorial\n",
"train_idxs = np.random.choice(animal_idxs, len(animal_idxs) // 6, replace=False)\n",
"test_idxs = np.random.choice(range(len(test_data)), len(test_data) // 10, replace=False)\n",
"\n",
"train_data = torch.utils.data.Subset(train_data, train_idxs) # select subset of animal images for train_data\n",
"test_data = torch.utils.data.Subset(test_data, test_idxs) # select subset of all images for test_data\n",
"print('train_data length: %s' % (len(train_data)))\n",
"print('test_data length: %s' % (len(test_data)))"
]
},
{
"cell_type": "markdown",
"id": "1be5ff2e",
"metadata": {},
"source": [
"#### Visualize some of the training and test examples"
]
},
{
"cell_type": "markdown",
"id": "47514fe7",
"metadata": {},
"source": [
"<details><summary>See the implementation of `plot_images` and `visualize_outliers` **(click to expand)**</summary>\n",
"\n",
"```python\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"txt_classes = {0: 'airplane', \n",
" 1: 'automobile', \n",
" 2: 'bird',\n",
" 3: 'cat', \n",
" 4: 'deer', \n",
" 5: 'dog', \n",
" 6: 'frog', \n",
" 7: 'horse', \n",
" 8:'ship', \n",
" 9:'truck'}\n",
"\n",
"def imshow(img):\n",
" npimg = img.numpy()\n",
" return np.transpose(npimg, (1, 2, 0))\n",
"\n",
"def plot_images(dataset, show_labels=False):\n",
" plt.rcParams[\"figure.figsize\"] = (9,7)\n",
" for i in range(15):\n",
" X,y = dataset[i]\n",
" ax = plt.subplot(3,5,i+1)\n",
" if show_labels:\n",
" ax.set_title(txt_classes[int(y)])\n",
" ax.imshow(imshow(X))\n",
" ax.axis('off')\n",
" plt.show()\n",
"\n",
"def visualize_outliers(idxs, data):\n",
" data_subset = torch.utils.data.Subset(data, idxs)\n",
" plot_images(data_subset)\n",
" \n",
"```\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9b64e0aa",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"txt_classes = {0: 'airplane', \n",
" 1: 'automobile', \n",
" 2: 'bird',\n",
" 3: 'cat', \n",
" 4: 'deer', \n",
" 5: 'dog', \n",
" 6: 'frog', \n",
" 7: 'horse', \n",
" 8:'ship', \n",
" 9:'truck'}\n",
"\n",
"def imshow(img):\n",
" npimg = img.numpy()\n",
" return np.transpose(npimg, (1, 2, 0))\n",
"\n",
"def plot_images(dataset, show_labels=False):\n",
" plt.rcParams[\"figure.figsize\"] = (9,7)\n",
" for i in range(15):\n",
" X,y = dataset[i]\n",
" ax = plt.subplot(3,5,i+1)\n",
" if show_labels:\n",
" ax.set_title(txt_classes[int(y)])\n",
" ax.imshow(imshow(X))\n",
" ax.axis('off')\n",
" plt.show()\n",
"\n",
"def visualize_outliers(idxs, data):\n",
" data_subset = torch.utils.data.Subset(data, idxs)\n",
" plot_images(data_subset)"
]
},
{
"cell_type": "markdown",
"id": "eb28f354",
"metadata": {},
"source": [
"Observe how there are only animals left in our `train_data`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a00aa3ed",
"metadata": {},
"outputs": [],
"source": [
"plot_images(train_data, show_labels=True)"
]
},
{
"cell_type": "markdown",
"id": "df819e85",
"metadata": {},
"source": [
"If we consider `train_data` to be representative of the typical data distribution, then non-animal images in `test_data` become outliers:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "41e5cb6b",
"metadata": {},
"outputs": [],
"source": [
"plot_images(test_data, show_labels=True)"
]
},
{
"cell_type": "markdown",
"id": "92caec8a",
"metadata": {},
"source": [
"## 3. Use cleanlab and feature embeddings to find outliers in the data\n",
"\n",
"\n",
"### Represent each image as a numeric feature embedding vector\n",
"\n",
"We can pass images through a neural network to generate vector embeddings via its hidden layer representation. Here we use a `resnet50` network from [timm](https://timm.fast.ai/), which has been pretrained on a large corpus of other images. Note that cleanlab's outlier detection can be applied to numeric feature embeddings generated from any model (or to the raw data features if they are already numeric vectors). Outlier detection works best with feature vectors whose values along each dimension are of a similar scale. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1cf25354",
"metadata": {},
"outputs": [],
"source": [
"# Generates 2048-dimensional feature embeddings from images\n",
"def embed_images(model, dataloader):\n",
" feature_embeddings = []\n",
" for data in dataloader:\n",
" images, labels = data\n",
" with torch.no_grad():\n",
" embeddings = model(images)\n",
" feature_embeddings.extend(embeddings.numpy())\n",
" feature_embeddings = np.array(feature_embeddings)\n",
" return feature_embeddings # each row corresponds to embedding of a different image"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "85a58d41",
"metadata": {},
"outputs": [],
"source": [
"# Load pretrained neural network\n",
"model = timm.create_model('resnet50', pretrained=True, num_classes=0) # this is a pytorch network\n",
"model.eval() # eval mode disables training-time operators (like batch normalization)\n",
"\n",
"# Use dataloaders to stream images through the network\n",
"batch_size = 50\n",
"trainloader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=False)\n",
"testloader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False)\n",
"\n",
"# Generate feature embeddings\n",
"train_feature_embeddings = embed_images(model, trainloader)\n",
"print(f'Train embeddings pooled shape: {train_feature_embeddings.shape}')\n",
"test_feature_embeddings = embed_images(model, testloader)\n",
"print(f'Test embeddings pooled shape: {test_feature_embeddings.shape}')"
]
},
{
"cell_type": "markdown",
"id": "ad857d69",
"metadata": {},
"source": [
"### Scoring outliers in a given dataset (training data)\n",
"\n",
"Fitting cleanlab's ``OutOfDistribution`` class on ``feature_embeddings`` will find any naturally occurring outliers in a given dataset. These examples are atypical images that look strange or different from the majority of examples in the dataset. In our case, these correspond to odd-looking images of animals that do not resemble typical animals depicted in **cifar10**. This method produces a score in [0,1] for each example, where lower values correspond to more atypical examples (more likely out-of-distribution)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "feb0f519",
"metadata": {},
"outputs": [],
"source": [
"ood = OutOfDistribution()\n",
"train_ood_features_scores = ood.fit_score(features=train_feature_embeddings)\n",
"\n",
"top_train_ood_features_idxs = find_top_issues(quality_scores=train_ood_features_scores, top=15)\n",
"visualize_outliers(top_train_ood_features_idxs, train_data)"
]
},
{
"cell_type": "markdown",
"id": "756333f7",
"metadata": {},
"source": [
"For fun, let's see what cleanlab considers the least likely outliers in the dataset! We can do this by calling `find_top_issues` on the negated outlier scores. These examples look quite homogeneous as each one is similar to many other training images."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "089d5860",
"metadata": {},
"outputs": [],
"source": [
"bottom_train_ood_features_idxs = find_top_issues(quality_scores=-train_ood_features_scores, top=15)\n",
"visualize_outliers(bottom_train_ood_features_idxs, train_data)"
]
},
{
"cell_type": "markdown",
"id": "2521aefb",
"metadata": {},
"source": [
"### Scoring outliers in additional test data\n",
"\n",
"Now suppose we want to find outlier images in some never before seen test data, in particular images unlikely to stem from the same distribution as the training data. We can use our already fitted `OutOfDistribution` estimator to score how typical each new test example would be under the training data distribution and visualize the most severe outliers in this additional data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "78b1951c",
"metadata": {},
"outputs": [],
"source": [
"test_ood_features_scores = ood.score(features=test_feature_embeddings)\n",
"\n",
"top_ood_features_idxs = find_top_issues(test_ood_features_scores, top=15)\n",
"visualize_outliers(top_ood_features_idxs, test_data)"
]
},
{
"cell_type": "markdown",
"id": "2c645c58",
"metadata": {},
"source": [
"Many outliers identified in `test_data` depict (non-animal) classes not present in the training set. These non-animal images have very different feature embeddings than the animal-only images in the training data."
]
},
{
"cell_type": "markdown",
"id": "0b5de6f6",
"metadata": {},
"source": [
"### Deciding which test examples are outliers\n",
"\n",
"Given outlier scores, how do we determine how many of the top-ranked examples in ``test_data`` should be marked as outliers? \n",
"\n",
"Inevitably this has some true positive / false positive trade-off, so let's suppose we want to ensure around at most 5% false positives. We can use the 5th percentile of the distribution of `train_ood_features_scores` (assuming the training data are in-distribution examples without outliers) as a hard score threshold below which to consider a test example an outlier.\n",
"\n",
"Let's plot the 5th percentile of the training outlier score distribution (shown as red line)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e9dff81b",
"metadata": {},
"outputs": [],
"source": [
"fifth_percentile = np.percentile(train_ood_features_scores, 5) # 5th percentile of the train_data distribution\n",
"\n",
"# Plot outlier_score distributions and the 5th percentile cutoff\n",
"fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))\n",
"plt_range = [min(train_ood_features_scores.min(),test_ood_features_scores.min()), \\\n",
" max(train_ood_features_scores.max(),test_ood_features_scores.max())]\n",
"axes[0].hist(train_ood_features_scores, range=plt_range, bins=50)\n",
"axes[0].set(title='train_outlier_scores distribution', ylabel='Frequency')\n",
"axes[0].axvline(x=fifth_percentile, color='red', linewidth=2)\n",
"axes[1].hist(test_ood_features_scores, range=plt_range, bins=50)\n",
"axes[1].set(title='test_outlier_scores distribution', ylabel='Frequency')\n",
"axes[1].axvline(x=fifth_percentile, color='red', linewidth=2)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "74c39ab1",
"metadata": {},
"source": [
"All test examples whose `test_ood_features_scores` fall left of the red line will be marked as an outlier.\n",
"\n",
"Let's plot the least-certain outliers of our `test_data` (i.e. 15 images with outlier scores right along the threshold). These are the images immediately to the left of that cutoff threshold (red line). The majority of them are still truly out-of-distribution non-animal images, but there are a few atypical-looking animals that are now erroneously identified as outliers as well."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "616769f8",
"metadata": {},
"outputs": [],
"source": [
"sorted_idxs = test_ood_features_scores.argsort()\n",
"ood_features_scores = test_ood_features_scores[sorted_idxs]\n",
"ood_features_indices = sorted_idxs[ood_features_scores < fifth_percentile] # Images in test data flagged as outliers\n",
"\n",
"visualize_outliers(ood_features_indices[::-1], test_data)"
]
},
{
"cell_type": "markdown",
"id": "cb4c0a06",
"metadata": {},
"source": [
"### How does cleanlab detect outliers from feature values?\n",
"\n",
"Outlier scores are defined relative to the average distance (computed over feature values) between each example and its K nearest neighbors in the training data. Such scores have been found to be particularly effective for out-of-distribution detection, see this paper for more details:\n",
"\n",
"[Back to the Basics: Revisiting Out-of-Distribution Detection Baselines](https://arxiv.org/abs/2207.03061)\n",
"\n",
"\n",
"Internally, cleanlab uses the `sklearn.neighbors.NearestNeighbor` class (with *cosine* distance) to find the K nearest neighbors, but you can easily use [another KNN estimator](https://github.com/cleanlab/examples/blob/master/outlier_detection_cifar10/outlier_detection_cifar10.ipynb) with cleanlab's `OutOfDistribution` class."
]
},
{
"cell_type": "markdown",
"id": "937c7e97",
"metadata": {},
"source": [
"## 4. Use cleanlab and `pred_probs` to find outliers in the data\n",
"\n",
"We sometimes wish to find outliers in classification datasets for which we do not have meaningful numeric feature representations. In this case, cleanlab can detect unusual examples in the data solely using predicted probabilities from a trained classifier.\n",
"\n",
"To get `pred_probs` here, a Logistic Regression classifier is fit on the already generated `train_feature_embeddings` (from our pretrained timm network) and the given label for each training image. We use a simple classifier here to quickly generate `pred_probs`, but in practice [fine-tuning the entire neural network for classification](https://github.com/cleanlab/examples/blob/master/outlier_detection_cifar10/outlier_detection_cifar10.ipynb) will be more effective (our approach here is equivalent to only training an extra output layer appended on top of the pretrained network)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "40fed4ef",
"metadata": {},
"outputs": [],
"source": [
"# Preprocess data\n",
"train_labels = np.array(train_data.dataset.targets)[train_data.indices]\n",
"train_labels = np.unique(train_labels, return_inverse=True)[1] # MAKE SURE to zero index training labels for sklearn\n",
"test_labels = np.array(test_data.dataset.targets)[test_data.indices]\n",
"\n",
"scaler = preprocessing.StandardScaler().fit(train_feature_embeddings)\n",
"train_feature_embeddings_scaled = scaler.transform(train_feature_embeddings)\n",
"test_feature_embeddings_scaled = scaler.transform(test_feature_embeddings)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "89f9db72",
"metadata": {},
"outputs": [],
"source": [
"# Our classifier employs bagging to better account for epistemic uncertainty \n",
"model = BaggingClassifier(LogisticRegression(max_iter=500), random_state=1, n_jobs=-1)\n",
"model.fit(train_feature_embeddings_scaled, train_labels)\n",
"\n",
"train_pred_probs = model.predict_proba(train_feature_embeddings_scaled)\n",
"train_pred_labels = train_pred_probs.argmax(1)\n",
"accuracy = np.mean(train_pred_labels == train_labels)\n",
"print(f\"Model accuracy on held-out train_data {accuracy}\")"
]
},
{
"cell_type": "markdown",
"id": "03e3f7b7",
"metadata": {},
"source": [
"We can use these `pred_probs` to again compute out-of-distribution scores for each image in our dataset using cleanlab's `OutOfDistribution` class."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "874c885a",
"metadata": {},
"outputs": [],
"source": [
"ood = OutOfDistribution()\n",
"train_ood_predictions_scores = ood.fit_score(pred_probs=train_pred_probs, labels=train_labels)"
]
},
{
"cell_type": "markdown",
"id": "dcff8e5a",
"metadata": {},
"source": [
"We can repeat this for additional test data, to identify test images that do not stem from the training data distribution."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e110fc4b",
"metadata": {},
"outputs": [],
"source": [
"test_pred_probs = model.predict_proba(test_feature_embeddings_scaled)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "85b60cbf",
"metadata": {},
"outputs": [],
"source": [
"test_ood_predictions_scores = ood.score(pred_probs=test_pred_probs)"
]
},
{
"cell_type": "markdown",
"id": "702aa162",
"metadata": {},
"source": [
"Detecting outliers based on feature embeddings can be done for arbitrary unlabeled datasets, but requires a meaningful numerical representation of the data. Detecting outliers based on predicted probabilities applies mainly for labeled classification datasets, but can be done with any effective classifier. The effectiveness of the latter approach depends on: how much auxiliary information captured in the feature values is lost in the predicted probabilities (determined by the particular set of labels in the classification task), the accuracy of our classifier, and how properly its predictions reflect epistemic uncertainty. Read more about it [here](https://pub.towardsai.net/a-simple-adjustment-improves-out-of-distribution-detection-for-any-classifier-5e96bbb2d627)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "17f96fa6",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"# Verify the top identified test outliers data are mostly non-animal images\n",
"top_ood_features_subset = torch.utils.data.Subset(test_data, top_ood_features_idxs)\n",
"num_animals = len([i for i in range(len(top_ood_features_subset)) if top_ood_features_subset[i][1] in animal_classes])\n",
"non_animal_frac = 1 - (num_animals / len(top_ood_features_subset))\n",
"if non_animal_frac < 0.81:\n",
" raise Exception(f\"Not enough non-animal images amongst top-ranked outliers in test_data, only: {non_animal_frac}\")\n",
"\n",
"top_ood_predictions_idxs = (test_ood_predictions_scores).argsort()[:15]\n",
"top_ood_predictions_subset = torch.utils.data.Subset(test_data, top_ood_predictions_idxs)\n",
"num_animals = len([i for i in range(len(top_ood_predictions_subset)) if top_ood_predictions_subset[i][1] in animal_classes])\n",
"non_animal_frac = 1 - (num_animals / len(top_ood_predictions_subset))\n",
"if non_animal_frac < 0.50:\n",
" raise Exception(f\"Not enough non-animal images amongst top-ranked ood datapoints in test_data, only: {non_animal_frac}\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/pred_probs_cross_val.rst
+60
View File
@@ -0,0 +1,60 @@
.. _pred_probs_cross_val:
Computing Out-of-Sample Predicted Probabilities with Cross-Validation
=====================================================================
Recall that cleanlab finds label issues in any dataset using some model's predicted class probabilities output. However, predicted probabilities from your model must be out-of-sample! You should never provide predictions on the same datapoints used to train the model, as these will be overfitted and unsuitable for finding label issues. It is ok if your model was trained on a separate dataset and you are only using cleanlab to evaluate labels in data that was previously held out (e.g., only searching for label issues in the test data).
To find label issues across all your data requires obtaining out-of-sample predicted probabilities for every datapoint in your dataset. This can be done via K-fold cross-validation as described below. Conventionally, `cross-validation <https://scikit-learn.org/stable/modules/cross_validation>`_ is used for model evaluation, but we'll use it to compute out-of-sample predicted probabilities for the entire dataset.
Out-of-sample predicted probabilities?
--------------------------------------
**Predicted probabilities** refer to a trained classification model's probabilistic estimate of the correct label for each datapoint. For example, a model trained to classify images of cats vs. dogs may predict that a new image is a cat with 90% confidence and a dog with 10% confidence --- these are the model's predicted probabilities for one datapoint. Whichever label with the highest predicted probability is often considered the model's class prediction (i.e., cat for the aforementioned hypothetical image).
**Out-of-sample** predicted probabilities refer to the model's probabilistic predictions made only on datapoints that were not shown to the model during training. In contrast, in-sample predicted probabilities on the model's training data will often be way overconfident and cannot be trusted. For example, in a traditional train-test split of the data, the train set will be shown to the model during its training, whereas the test set will only be used to evaluate the model's performance after training. Predicted probabilities generated for the test set can thus be considered as out-of-sample.
When using cleanlab, we will typically want to find label issues in all labeled data rather than just the test data. We can use K-fold cross-validation to generate out-of-sample predicted probabilities for every datapoint.
What is K-fold cross-validation?
--------------------------------
.. image:: https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/pred_probs_cross_val.png
:alt: Computing Out-of-Sample Predicted Probabilities from K-Fold Cross-Validation
The diagram above depicts K-fold cross-validation with K = 5. K-fold cross-validation partitions the entire dataset into *K* disjoint subsets of data called *folds*. *K* independent copies of our model are trained, where for each model copy, one fold of the data is held out from its training (the data in this fold may be viewed as a *validation set* for this copy of the model). Each copy of the model has a different validation set for which we can obtain out-of-sample predicted probabilities from this copy of the model. Since each datapoint is held-out from one copy of the model, this process allows us to get out-of-sample predictions for every datapoint! We recommend applying *stratified* cross-validation, which tries to ensure the proportions of data from each class match across different folds.
This method of producing out-of-sample predictions via cross-validation is also referred to as cross-validated prediction, out-of-folds predictions, and K-fold bagging. It can be easily applied to any `sklearn`-compatible model by invoking `cross_val_predict <https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html>`_. An additional benefit is that cross-validation produces `significantly superior estimates <https://medium.com/data-science/5-reasons-why-you-should-use-cross-validation-in-your-data-science-project-8163311a1e79>`_ of how the model will perform on new data.
Here is pseudocode for manually implementing K-fold cross-validation with K = 3:
.. code-block:: python
# Step 0
# Separate your data into three equal sized chunks (this is called 3-fold cross validation)
# Data = A B C
# Step 1 -- get out-of-sample pred probs for A
model = Model()
model.fit(data=B+C)
out_of_sample_pred_probs_for_A = model.pred_proba(data=A)
# Step 2 -- get out-of-sample pred probs for B
model = Model()
model.fit(data=A+C)
out_of_sample_pred_probs_for_B = model.pred_proba(data=B)
# Step 3 -- get out-of-sample pred probs for C
model = Model()
model.fit(data=A+B)
out_of_sample_pred_probs_for_C = model.pred_proba(data=C)
# Final step -- combine to get out-of-sample pred probs for entire dataset.
out_of_sample_pred_probs = concatenate([
out_of_sample_pred_probs_for_A,
out_of_sample_pred_probs_for_B,
out_of_sample_pred_probs_for_C,
])
docs/source/tutorials/regression.ipynb
+769
View File
@@ -0,0 +1,769 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ea0a577e",
"metadata": {},
"source": [
"# Find Noisy Labels in Regression Datasets"
]
},
{
"cell_type": "markdown",
"id": "e15b9f2f",
"metadata": {},
"source": [
"This 5-minute quickstart tutorial uses cleanlab to find potentially incorrect numeric values in a dataset column by means of a regression model. Unlike classification models, regression predicts numeric quantities such as price, income, age,... Response values in regression datasets may be corrupted due to: data entry or measurement errors, noise from sensors or other processes, or broken data pipelines. To find corrupted values in a numeric column, we treat it as the target value, i.e. label, to be predicted by a regression model and then use cleanlab to decide when the model predictions are trustworthy while deviating from the observed label value.\n",
"\n",
"In this tutorial, we consider a student grades dataset, which records three exam grades and some optional notes for over 900 students, each being assigned a final score. Combined with any regression model of your choosing, cleanlab automatically identifies examples in this dataset that have incorrect final scores.\n",
"\n",
"**Overview of what we’ll do in this tutorial:**\n",
"\n",
"- Fit a simple Gradient Boosting model (any other model could be used) on the exam-score and notes (covariates) in order to compute out-of-sample predictions of the final grade (the response variable in our regression).\n",
"- Use cleanlab's `CleanLearning.find_label_issues()` method to identify potentially incorrect final grade values based on outputs from this regression model.\n",
"- Train a more robust version of the same model after dropping the identified label errors using CleanLearning.\n",
"- Run an alternative workflow to detect errors via cleanlab's `Datalab` audit, which can simultaneously estimate **many other types of data issues**."
]
},
{
"cell_type": "markdown",
"id": "612a355a",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have an sklearn-compatible regression `model`, features/covariates `X`, and a label/target variable `y`? Run the code below to train your `model` and identify potentially incorrect `y` values in your dataset.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.regression.learn import CleanLearning\n",
"\n",
"cl = CleanLearning(model)\n",
"cl.fit(X, y)\n",
"label_issues = cl.get_label_issues()\n",
"preds = cl.predict(X_test) # predictions from a version of your model trained on auto-cleaned data\n",
"```\n",
" \n",
"</div>\n",
" \n",
"Is your model/data not compatible with `CleanLearning`? You can instead run cross-validation on your model to get out-of-sample `predictions`. With that, run the code below to find data and label issues in your regression dataset:\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"# Assuming your dataset has a label column named 'label'\n",
"lab = Datalab(dataset, label_name='label', task='regression')\n",
"# To detect more data issue types, optionally supply `features` (numeric dataset values or model embeddings of the data)\n",
"lab.find_issues(pred_probs=predictions, features=features)\n",
"\n",
"lab.report()\n",
" \n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "f9a290d6",
"metadata": {},
"source": [
"## 1. Install required dependencies"
]
},
{
"cell_type": "markdown",
"id": "8430ca39",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib\n",
"!pip install cleanlab[datalab]\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2e1af7d8",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"\n",
"dependencies = [\"cleanlab\", \"matplotlib>=3.6.0\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = \" \".join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4fb10b8f",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from sklearn.ensemble import HistGradientBoostingRegressor\n",
"from sklearn.model_selection import cross_val_predict\n",
"from sklearn.metrics import r2_score\n",
"\n",
"from cleanlab.regression.learn import CleanLearning"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "284dc264",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai \n",
"\n",
"import random \n",
"import numpy as np \n",
"\n",
"SEED = 111 # for reproducibility \n",
"\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"id": "2035042e",
"metadata": {},
"source": [
"## 2. Load and process the data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0f7450db",
"metadata": {},
"outputs": [],
"source": [
"train_data = pd.read_csv(\"https://s.cleanlab.ai/student_grades_r/train.csv\")\n",
"test_data = pd.read_csv(\"https://s.cleanlab.ai/student_grades_r/test.csv\")\n",
"train_data.head()"
]
},
{
"cell_type": "markdown",
"id": "aa0165ef",
"metadata": {},
"source": [
"In the DataFrame above, `final_score` represents the noisy scores and `true_final_score` represents the ground truth. Note that ground truth is usually not available in real-world datasets, and is just added in this tutorial dataset for demonstration purposes."
]
},
{
"cell_type": "markdown",
"id": "82285102",
"metadata": {},
"source": [
"We show a 3D scatter plot of the exam grades, with the color hue corresponding to the final score for each student. Incorrect datapoints are marked with an **X**."
]
},
{
"cell_type": "markdown",
"id": "c8173840",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
" \n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from mpl_toolkits.mplot3d import Axes3D\n",
"\n",
"def plot_data(train_data, errors_idx):\n",
" fig = plt.figure()\n",
" ax = fig.add_subplot(111, projection='3d')\n",
"\n",
" x, y, z = train_data[\"exam_1\"], train_data[\"exam_2\"], train_data[\"exam_3\"]\n",
" labels = train_data[\"final_score\"]\n",
"\n",
" img = ax.scatter(x, y, z, c=labels, cmap=\"jet\")\n",
" fig.colorbar(img)\n",
"\n",
" ax.plot(\n",
" x.iloc[errors_idx],\n",
" y.iloc[errors_idx],\n",
" z.iloc[errors_idx],\n",
" \"x\",\n",
" markeredgecolor=\"black\",\n",
" markersize=10,\n",
" markeredgewidth=2.5,\n",
" alpha=0.8,\n",
" label=\"Label Errors\"\n",
" )\n",
" ax.legend()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "55513fed",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from mpl_toolkits.mplot3d import Axes3D\n",
"\n",
"def plot_data(train_data, errors_idx):\n",
" fig = plt.figure()\n",
" ax = fig.add_subplot(111, projection='3d')\n",
"\n",
" x, y, z = train_data[\"exam_1\"], train_data[\"exam_2\"], train_data[\"exam_3\"]\n",
" labels = train_data[\"final_score\"]\n",
"\n",
" img = ax.scatter(x, y, z, c=labels, cmap=\"jet\")\n",
" fig.colorbar(img)\n",
"\n",
" ax.plot(\n",
" x.iloc[errors_idx],\n",
" y.iloc[errors_idx],\n",
" z.iloc[errors_idx],\n",
" \"x\",\n",
" markeredgecolor=\"black\",\n",
" markersize=10,\n",
" markeredgewidth=2.5,\n",
" alpha=0.8,\n",
" label=\"Label Errors\"\n",
" )\n",
" ax.legend()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "df5a0f59",
"metadata": {},
"outputs": [],
"source": [
"errors_mask = train_data[\"final_score\"] != train_data[\"true_final_score\"]\n",
"errors_idx = np.where(errors_mask == 1)\n",
"\n",
"plot_data(train_data, errors_idx)"
]
},
{
"cell_type": "markdown",
"id": "add939ae",
"metadata": {},
"source": [
"Next we preprocess the data by applying one-hot encoding to features with categorical data (this is optional if your regression model can work directly with categorical features)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7af78a8a",
"metadata": {},
"outputs": [],
"source": [
"feature_columns = [\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]\n",
"predicted_column = \"final_score\"\n",
"\n",
"X_train_raw, y_train = train_data[feature_columns], train_data[predicted_column]\n",
"X_test_raw, y_test = test_data[feature_columns], test_data[predicted_column]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9556c624",
"metadata": {},
"outputs": [],
"source": [
"categorical_features = [\"notes\"]\n",
"X_train = pd.get_dummies(X_train_raw, columns=categorical_features)\n",
"X_test = pd.get_dummies(X_test_raw, columns=categorical_features)"
]
},
{
"cell_type": "markdown",
"id": "1ce924cf",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"Assign your data's features to variable `X` and the target values to variable `y` instead, then continue with the rest of the tutorial.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "4b14309d",
"metadata": {},
"source": [
"## 3. Define a regression model and use cleanlab to find potential label errors"
]
},
{
"cell_type": "markdown",
"id": "81ee2349",
"metadata": {},
"source": [
"We'll first demonstrate regression with noisy labels via the `CleanLearning` class that can wrap any scikit-learn compatible regression model you have. `CleanLearning` uses your model to estimate label issues (i.e. noisy `y`-values) and train a more robust version of the same model when the original data contains noisy labels.\n",
"\n",
"Here we define a `CleanLearning` object with a histogram-based gradient boosting model (sklearn version of XGBoost) and use the `find_label_issues` method to find potential errors in our dataset's numeric label column. Any other sklearn-compatible regression model could be used, such as `LinearRegression` or `RandomForestRegressor` (or you can easily wrap arbitrary custom models to be compatible with the sklearn API)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c2f1ccc",
"metadata": {},
"outputs": [],
"source": [
"model = HistGradientBoostingRegressor()\n",
"cl = CleanLearning(model)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e1b7860",
"metadata": {},
"outputs": [],
"source": [
"label_issues = cl.find_label_issues(X_train, y_train)"
]
},
{
"cell_type": "markdown",
"id": "43bd6c7f",
"metadata": {},
"source": [
"`CleanLearning` internally fits multiple copies of our regression model via cross-validation and bootstrapping in order to compute predictions and uncertainty estimates for the dataset. These are used to identify label issues (i.e. likely corrupted `y`-values).\n",
"\n",
"This method returns a Dataframe containing a label quality score (between 0 and 1) for each example in your dataset. Lower scores indicate examples more likely to be mislabeled with an erroneous `y` value. The Dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating its `y`-value appears potentially corrupted). "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f407bd69",
"metadata": {},
"outputs": [],
"source": [
"label_issues.head()"
]
},
{
"cell_type": "markdown",
"id": "4ab5acf3",
"metadata": {},
"source": [
"We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 10 most likely mislabeled examples in our regression dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f7385336",
"metadata": {},
"outputs": [],
"source": [
"identified_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = label_issues[\"label_quality\"].argsort()[:10].to_numpy()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "59fc3091",
"metadata": {},
"outputs": [],
"source": [
"print(\n",
" f\"cleanlab found {len(identified_issues)} potential label errors in the dataset.\\n\"\n",
" f\"Here are indices of the top 10 most likely errors: \\n {lowest_quality_labels}\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "aa2c1fec",
"metadata": {},
"source": [
"Let’s review some of the values most likely to be erroneous. To help us inspect these datapoints, we define a method to print any example from the dataset, together with its given (original) label and the suggested alternative label predicted by your regression model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00949977",
"metadata": {},
"outputs": [],
"source": [
"def view_datapoint(index):\n",
" given_labels = label_issues[\"given_label\"]\n",
" predicted_labels = label_issues[\"predicted_label\"].round(1)\n",
" return pd.concat(\n",
" [X_train_raw, given_labels, predicted_labels], axis=1\n",
" ).iloc[index]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6c1ae3a",
"metadata": {},
"outputs": [],
"source": [
"view_datapoint(lowest_quality_labels[:5])"
]
},
{
"cell_type": "markdown",
"id": "f2be7a93",
"metadata": {},
"source": [
"These are very clear errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the final grade that these student should be getting. \n",
"\n",
"cleanlab has shortlisted the most likely label errors to speed up your data cleaning process. With this list, you can decide whether to fix these label issues or remove erroneous examples from the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9131d82d",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai \n",
"\n",
"label_issues_cl = label_issues.copy()"
]
},
{
"cell_type": "markdown",
"id": "e2761486",
"metadata": {},
"source": [
"## 4. Train a more robust model from noisy labels"
]
},
{
"cell_type": "markdown",
"id": "043bfb52",
"metadata": {},
"source": [
"Fixing the label issues manually may be time-consuming, but cleanlab can filter these noisy examples and train a model on the remaining clean data for you automatically.\n",
"\n",
"To establish a baseline, let’s first train and evaluate our original Gradient Boosting model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31c704e7",
"metadata": {},
"outputs": [],
"source": [
"baseline_model = HistGradientBoostingRegressor() \n",
"baseline_model.fit(X_train, y_train)\n",
"\n",
"preds_og = baseline_model.predict(X_test)\n",
"r2_og = r2_score(y_test, preds_og)\n",
"print(f\"r-squared score of original model: {r2_og:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "0d01f715",
"metadata": {},
"source": [
"Now that we have a baseline, let’s check if using `CleanLearning` improves our test accuracy.\n",
"\n",
"`CleanLearning` provides a wrapper that can be applied to any scikit-learn compatible model. The resulting model object can be used in the same manner, but it will now train more robustly if the data has noisy labels.\n",
"\n",
"We can use the same `CleanLearning` object defined above, and pass the label issues we already computed into `.fit()` via the `label_issues` argument. This accelerates things; if we did not provide the label issues, then they would be re-estimated via cross-validation. After the issues are estimated, `CleanLearning` simply removes the examples with label issues and retrains your model on the remaining clean data."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0bcc43db",
"metadata": {},
"outputs": [],
"source": [
"found_label_issues = cl.get_label_issues()\n",
"cl.fit(X_train, y_train, label_issues=found_label_issues)\n",
"\n",
"preds_cl = cl.predict(X_test)\n",
"r2_cl = r2_score(y_test, preds_cl)\n",
"print(f\"r-squared score of cleanlab's model: {r2_cl:.3f}\")"
]
},
{
"cell_type": "markdown",
"id": "3aea51da",
"metadata": {},
"source": [
"We can see that the coefficient of determination (r-squared score) of the test set improved as a result of the data cleaning. Note that this will not always be the case, especially when we are evaluating on test data that are themselves noisy. The best practice is to run cleanlab to identify potential label issues and then manually review them, before blindly trusting any evaluation metrics. In particular, the most effort should be made to ensure high-quality test data, which is supposed to reflect the expected performance of our model during deployment."
]
},
{
"cell_type": "markdown",
"id": "167fca90",
"metadata": {},
"source": [
"## 5. Other ways to find noisy labels in regression datasets"
]
},
{
"cell_type": "markdown",
"id": "5b4f8e14",
"metadata": {},
"source": [
"The `CleanLearning` workflow above requires a sklearn-compatible model. If your model or data format is not compatible with the requirements for using `CleanLearning`, you can instead run [cross-validation on your regression model to get out-of-sample predictions](https://docs.cleanlab.ai/stable/tutorials/pred_probs_cross_val.html), and then use the `Datalab` audit to estimate label quality scores for each example in your dataset.\n",
"\n",
"This approach requires two inputs:\n",
"\n",
"- `labels`: numpy array of given labels in the dataset. \n",
"- `predictions`: numpy array of predictions for each example in the dataset from your favorite model (these should be out-of-sample predictions to get the best results)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7021bd68",
"metadata": {},
"outputs": [],
"source": [
"# Get out-of-sample predictions using cross-validation:\n",
"model = HistGradientBoostingRegressor()\n",
"predictions = cross_val_predict(estimator=model, X=X_train, y=y_train)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d49c990b",
"metadata": {},
"outputs": [],
"source": [
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(\n",
" data=train_data.drop(columns=[\"true_final_score\"]),\n",
" label_name=\"final_score\",\n",
" task=\"regression\",\n",
")\n",
"\n",
"lab.find_issues(\n",
" pred_probs=predictions,\n",
" issue_types={\"label\": {}}, # specify we're only interested in label issues here \n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbab6fb3",
"metadata": {},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"\n",
"label_issues.sort_values(\"label_score\").head()"
]
},
{
"cell_type": "markdown",
"id": "3a0db9b2",
"metadata": {},
"source": [
"As before, these label quality scores are continuous values in the range [0,1] where 1 represents a clean label (given label appears correct) and 0 a represents dirty label (given label appears corrupted, i.e. the numeric value may be incorrect). You can sort examples by their label quality scores to inspect the most-likely corrupted datapoints.\n",
"\n",
"If possible, we recommend you use `CleanLearning` to wrap your regression model (over providing its pre-computed predictions) for the most accurate label error detection (that properly accounts for aleatoric/epistemic uncertainty in the regression model). To understand how these approaches work, refer to our paper: **[Detecting Errors in Numerical Data via any Regression Model](https://arxiv.org/abs/2305.16583)**"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5b39b8b5",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai\n",
"np.random.seed(SEED) # for reproducibility\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"id": "4366346a",
"metadata": {},
"source": [
"You can alternatively provide `features` to `Datalab` instead of pre-computed predictions. These are (preprocessed) numeric dataset covariates, aka independent variables to the regression model (such as neural network embeddings of your raw data). Internally, this is equivalent to using `CleanLearning` to find label issues if you also possible provide your sklearn-compatible regression model to `Datalab.find_issues`. But you can simultaneously detect many more types of issues in your dataset beyond mislabeling via Datalab (simply drop the `issue_types` argument below)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "df06525b",
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(\n",
" data=train_data.drop(columns=[\"true_final_score\"]),\n",
" label_name=\"final_score\",\n",
" task=\"regression\",\n",
")\n",
"\n",
"lab.find_issues(\n",
" features=X_train,\n",
" issue_types={ # Optional drop this to simultaneously detect many types of data/label issues \n",
" \"label\": {\n",
" # Optional: Specify which type of sklearn-compatible regression model is used to find label errors\n",
" \"clean_learning_kwargs\": {\"model\": HistGradientBoostingRegressor()}\n",
" }\n",
" },\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "05282559",
"metadata": {},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"\n",
"label_issues.sort_values(\"label_score\").head()"
]
},
{
"cell_type": "markdown",
"id": "c1353758",
"metadata": {},
"source": [
"While this tutorial focused on label issues, cleanlab's `Datalab` object can automatically detect many other types of issues in your dataset (outliers, near duplicates, etc).\n",
"Simply remove the `issue_types` argument from the above call to `Datalab.find_issues()` above and `Datalab` will more comprehensively audit your dataset (a default regression model will be used if you don't specify the model type).\n",
"Refer to our [Datalab quickstart tutorial](./datalab/datalab_quickstart.html) to learn how to interpret the results (the interpretation remains mostly the same across different types of ML tasks).\n",
"\n",
"**Summary:** To detect many types of issues in your regression dataset, we recommend using `Datalab` with provided `features` plus the best regression model you know for your data. If your goal is to train a robust regression model with noisy data rather than detect data/label issues, then use `CleanLearning`. Alternatively, if you don't have a sklearn-compatible regression model or already have pre-computed predictions from the model you'd like to rely on, you can pass these predictions into `Datalab` directly to find issues based on them instead of providing a regression model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "95531cda",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"from sklearn.metrics import roc_auc_score\n",
"from cleanlab.regression.rank import get_label_quality_scores\n",
"\n",
"if r2_cl <= r2_og:\n",
" raise ValueError(\"CleanLearning did not improve r2 score\")\n",
"\n",
"label_quality_score_cl = label_issues_cl[\"label_quality\"]\n",
"label_quality_scores_residual = get_label_quality_scores(labels=y_train, predictions=predictions, method=\"residual\")\n",
"\n",
"label_quality_scores = get_label_quality_scores(labels=y_train, predictions=predictions)\n",
"\n",
"auc_outre = roc_auc_score(errors_mask, 1 - label_quality_scores)\n",
"auc_cl = roc_auc_score(errors_mask, 1 - label_quality_score_cl)\n",
"auc_residual = roc_auc_score(errors_mask, 1 - label_quality_scores_residual)\n",
"\n",
"if auc_outre <= 0.5 or auc_cl <= 0.5:\n",
" raise ValueError(\"Label quality scores did not perform well enough\")\n",
"\n",
"if auc_outre <= auc_residual:\n",
" raise ValueError(\"Outre label quality scores did not outperform alternative scores\")\n",
" \n",
"if auc_cl <= auc_residual:\n",
" raise ValueError(\"CL label quality scores did not outperform alternative scores\")\n",
"\n",
"# Test that CleanLearning label issues and Datalab label issues match\n",
"pd.testing.assert_frame_equal(\n",
" # CleanLearning DataFrame\n",
" label_issues_cl.rename(columns={\"label_quality\": \"label_score\"}), \n",
" # Datalab DataFrame\n",
" label_issues,\n",
")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorials/segmentation.ipynb
+486
View File
File diff suppressed because one or more lines are too long
docs/source/tutorials/token_classification.ipynb
+549
View File
@@ -0,0 +1,549 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d0d2e007",
"metadata": {},
"source": [
"# Find Label Errors in Token Classification (Text) Datasets\n",
"\n",
"This 5-minute quickstart tutorial shows how you can use cleanlab to find potential label errors in text datasets for token classification. In token-classification, our data consists of a bunch of sentences (aka documents) in which every token (aka word) is labeled with one of K classes, and we train models to predict the class of each token in a new sentence. Example applications in NLP include part-of-speech-tagging or entity recognition, which is the focus on this tutorial. Here we use the [CoNLL-2003 named entity recognition](https://deepai.org/dataset/conll-2003-english) dataset which contains around 20,000 sentences with 300,000 individual tokens. Each token is labeled with one of the following classes:\n",
"\n",
"- LOC (location entity)\n",
"- PER (person entity)\n",
"- ORG (organization entity)\n",
"- MISC (miscellaneous other type of entity)\n",
"- O (other type of word that does not correspond to an entity)\n",
"\n",
"**Overview of what we'll do in this tutorial:** \n",
"\n",
"- Find tokens with label issues using `cleanlab.token_classification.filter.find_label_issues`. \n",
"- Rank sentences based on their overall label quality using `cleanlab.token_classification.rank.get_label_quality_scores`."
]
},
{
"cell_type": "markdown",
"id": "07936a54",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"cleanlab uses three inputs to handle token classification data:\n",
"\n",
"- `tokens`: List whose `i`-th element is a list of strings/words corresponding to tokenized version of the `i`-th sentence in dataset. \n",
" Example: `[..., [\"I\", \"love\", \"cleanlab\"], ...]`\n",
"- `labels`: List whose `i`-th element is a list of integers corresponding to class labels of each token in the `i`-th sentence. Example: `[..., [0, 0, 1], ...]`\n",
"- `pred_probs`: List whose `i`-th element is a np.ndarray of shape `(N_i, K)` corresponding to predicted class probabilities for each token in the `i`-th sentence (assuming this sentence contains `N_i` tokens and dataset has `K` possible classes). These should be out-of-sample `pred_probs` obtained from a token classification model via cross-validation. \n",
" Example: `[..., np.array([[0.8,0.2], [0.9,0.1], [0.3,0.7]]), ...]`\n",
"\n",
"Using these, you can find/display label issues with this code: \n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab.token_classification.filter import find_label_issues \n",
"from cleanlab.token_classification.summary import display_issues\n",
" \n",
"issues = find_label_issues(labels, pred_probs)\n",
"display_issues(issues, tokens, pred_probs=pred_probs, labels=labels,\n",
" class_names=OPTIONAL_LIST_OF_ORDERED_CLASS_NAMES)\n",
"\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "1da020bc",
"metadata": {},
"source": [
"## 1. Install required dependencies and download data\n",
"\n",
"You can use `pip` to install all packages required for this tutorial as follows: \n",
"\n",
" !pip install cleanlab "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ae8a08e0",
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"from huggingface_hub import hf_hub_download\n",
"\n",
"!wget -nc https://data.deepai.org/conll2003.zip && mkdir -p data \n",
"!unzip -o conll2003.zip -d data/ && rm conll2003.zip \n",
"\n",
"# Download from HuggingFace Hub\n",
"pred_probs_path = hf_hub_download('Cleanlab/token-classification-tutorial', 'pred_probs.npz', repo_type=\"dataset\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "439b0305",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"\n",
"dependencies = [\"cleanlab\", \"huggingface_hub\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a1349304",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from cleanlab.token_classification.filter import find_label_issues \n",
"from cleanlab.token_classification.rank import get_label_quality_scores, issues_from_scores \n",
"from cleanlab.internal.token_classification_utils import get_sentence, filter_sentence, mapping \n",
"from cleanlab.token_classification.summary import display_issues, common_label_issues, filter_by_token \n",
"\n",
"np.set_printoptions(suppress=True)"
]
},
{
"cell_type": "markdown",
"id": "9ad75b45",
"metadata": {},
"source": [
"## 2. Get data, labels, and pred_probs\n",
"\n",
"In token classification tasks, each token in the dataset is labeled with one of *K* possible classes.\n",
"To find label issues, cleanlab requires predicted class probabilities from a trained classifier. These `pred_probs` contain a length-*K* vector for **each** token in the dataset (which sums to 1 for each token). Here we use `pred_probs` which are out-of-sample predicted class probabilities for the full CoNLL-2003 dataset (merging training, development, and testing splits), obtained from a BERT Transformer fit via cross-validation. Our example notebook [\"Training Entity Recognition Model for Token Classification\"](https://github.com/cleanlab/examples/blob/master/entity_recognition/entity_recognition_training.ipynb) contains the code to produce such `pred_probs` and save them in a `.npz` file, which we simply load here via a `read_npz` function (can skip these details)."
]
},
{
"cell_type": "markdown",
"id": "6cc832fd",
"metadata": {},
"source": [
"<details><summary>See the code for reading the `.npz` file **(click to expand)**</summary> \n",
"\n",
"```python\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"def read_npz(filepath): \n",
" data = dict(np.load(filepath)) \n",
" data = [data[str(i)] for i in range(len(data))] \n",
" return data \n",
"\n",
"```\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ab9d59a0",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"def read_npz(filepath): \n",
" data = dict(np.load(filepath)) \n",
" data = [data[str(i)] for i in range(len(data))] \n",
" return data "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "519cb80c",
"metadata": {},
"outputs": [],
"source": [
"# Load predicted probabilities\n",
"pred_probs = read_npz(pred_probs_path)"
]
},
{
"cell_type": "markdown",
"id": "a8136f37",
"metadata": {},
"source": [
"`pred_probs` is a list of numpy arrays, which we'll describe later. Let's first also load the dataset and its labels. We collect sentences from the original text files defining: \n",
"\n",
"- `tokens` as a nested list where `tokens[i]` is a list of strings corrsesponding to a (word-level) tokenized version of the `i`-th sentence\n",
"- `given_labels` as a nested list of the given labels in the dataset where `given_labels[i]` is a list of labels for each token in the `i`-th sentence. \n",
"\n",
"This version of CoNLL-2003 uses IOB2-formatting for tagging, where `B-` and `I-` prefixes in the class labels indicate whether the tokens are at the start of an entity or in the middle. We ignore these distinctions in this tutorial (as label errors that confuse `B-` and `I-` are less interesting), and thus have two sets of entities: \n",
"\n",
"- `given_entities` = ['O', 'B-MISC', 'I-MISC', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'] \n",
"- `entities` = ['O', 'MISC', 'PER', 'ORG', 'LOC']. These are our classes of interest for the token classification task.\n",
"\n",
"We use some helper methods to load the CoNLL data (can skip these details)."
]
},
{
"cell_type": "markdown",
"id": "43a87745",
"metadata": {},
"source": [
"<details><summary>See the code for reading the CoNLL data files **(click to expand)**</summary>\n",
"\n",
"```python\n",
"\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"given_entities = ['O', 'B-MISC', 'I-MISC', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC']\n",
"entities = ['O', 'MISC', 'PER', 'ORG', 'LOC'] \n",
"entity_map = {entity: i for i, entity in enumerate(given_entities)} \n",
"\n",
"def readfile(filepath, sep=' '): \n",
" lines = open(filepath)\n",
" data, sentence, label = [], [], []\n",
" for line in lines:\n",
" if len(line) == 0 or line.startswith('-DOCSTART') or line[0] == '\\n':\n",
" if len(sentence) > 0:\n",
" data.append((sentence, label))\n",
" sentence, label = [], []\n",
" continue\n",
" splits = line.split(sep) \n",
" word = splits[0]\n",
" if len(word) > 0 and word[0].isalpha() and word.isupper():\n",
" word = word[0] + word[1:].lower()\n",
" sentence.append(word)\n",
" label.append(entity_map[splits[-1][:-1]])\n",
"\n",
" if len(sentence) > 0:\n",
" data.append((sentence, label))\n",
"\n",
" tokens = [d[0] for d in data] \n",
" given_labels = [d[1] for d in data]\n",
" return tokens, given_labels\n",
"\n",
"```\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "202f1526",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"given_entities = ['O', 'B-MISC', 'I-MISC', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC']\n",
"entities = ['O', 'MISC', 'PER', 'ORG', 'LOC'] \n",
"entity_map = {entity: i for i, entity in enumerate(given_entities)} \n",
"\n",
"def readfile(filepath, sep=' '): \n",
" lines = open(filepath)\n",
" data, sentence, label = [], [], []\n",
" for line in lines:\n",
" if len(line) == 0 or line.startswith('-DOCSTART') or line[0] == '\\n':\n",
" if len(sentence) > 0:\n",
" data.append((sentence, label))\n",
" sentence, label = [], []\n",
" continue\n",
" splits = line.split(sep) \n",
" word = splits[0]\n",
" if len(word) > 0 and word[0].isalpha() and word.isupper():\n",
" word = word[0] + word[1:].lower()\n",
" sentence.append(word)\n",
" label.append(entity_map[splits[-1][:-1]])\n",
"\n",
" if len(sentence) > 0:\n",
" data.append((sentence, label))\n",
" \n",
" tokens = [d[0] for d in data] \n",
" given_labels = [d[1] for d in data] \n",
" return tokens, given_labels "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4381f03",
"metadata": {},
"outputs": [],
"source": [
"filepaths = ['data/train.txt', 'data/valid.txt', 'data/test.txt'] \n",
"tokens, given_labels = [], [] \n",
"\n",
"for filepath in filepaths: \n",
" words, label = readfile(filepath) \n",
" tokens.extend(words) \n",
" given_labels.extend(label)\n",
" \n",
"sentences = list(map(get_sentence, tokens)) \n",
"\n",
"sentences, mask = filter_sentence(sentences) \n",
"tokens = [words for m, words in zip(mask, tokens) if m] \n",
"given_labels = [labels for m, labels in zip(mask, given_labels) if m] \n",
"\n",
"maps = [0, 1, 1, 2, 2, 3, 3, 4, 4] \n",
"labels = [mapping(labels, maps) for labels in given_labels] "
]
},
{
"cell_type": "markdown",
"id": "46cb7c93",
"metadata": {},
"source": [
"To find label issues in token classification data, cleanlab requires `labels` and `pred_probs`, which should look as follows: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7842e4a3",
"metadata": {},
"outputs": [],
"source": [
"indices_to_preview = 3 # increase this to view more examples\n",
"for i in range(indices_to_preview):\n",
" print('\\nsentences[%d]:\\t' % i + str(sentences[i])) \n",
" print('labels[%d]:\\t' % i + str(labels[i])) \n",
" print('pred_probs[%d]:\\n' % i + str(pred_probs[i])) "
]
},
{
"cell_type": "markdown",
"id": "9b71eb4a",
"metadata": {},
"source": [
"Note that these correspond to the sentences in the dataset, where each sentence is treated as an individual training example (could be document instead of sentence). If using your own dataset, both `pred_probs` and `labels` should each be formatted as a nested-list where: \n",
"\n",
"- `pred_probs` is a list whose `i`-th element is a np.ndarray of shape `(N_i, K)` corresponding to predicted class probabilities for each token in the `i`-th sentence (assuming this sentence contains `N_i` tokens and dataset has `K` possible classes). Each row of one np.ndarray corresponds to a token `t` and contains a model's predicted probability that `t` belongs to each possible class, for each of the K classes. The columns must be ordered such that the probabilities correspond to class 0, 1, ..., K-1. These should be out-of-sample `pred_probs` obtained from a token classification model via cross-validation. \n",
"\n",
"- `labels` is a list whose `i`-th element is a list of integers corresponding to class label of each token in the `i`-th sentence. For dataset with K classes, labels must take values in 0, 1, ..., K-1. "
]
},
{
"cell_type": "markdown",
"id": "1dc3150f",
"metadata": {},
"source": [
"## 3. Use cleanlab to find label issues \n",
"\n",
"Based on the given labels and out-of-sample predicted probabilities, cleanlab can quickly help us identify label issues in our dataset. Here we request that the indices of the identified label issues be sorted by cleanlab\u2019s self-confidence score, which measures the quality of each given label via the probability assigned to it in our model\u2019s prediction. The returned `issues` are a list of tuples `(i, j)`, which corresponds to the `j`th token of the `i`-th sentence in the dataset. These are the tokens cleanlab thinks may be badly labeled in your dataset. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c2ad9ad",
"metadata": {},
"outputs": [],
"source": [
"issues = find_label_issues(labels, pred_probs) "
]
},
{
"cell_type": "markdown",
"id": "7221c12b",
"metadata": {},
"source": [
"Let's look at the top 20 tokens that cleanlab thinks are most likely mislabeled. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "95dc7268",
"metadata": {},
"outputs": [],
"source": [
"top = 20 # increase this value to view more identified issues\n",
"print('Cleanlab found %d potential label issues. ' % len(issues)) \n",
"print('The top %d most likely label errors:' % top) \n",
"print(issues[:top]) "
]
},
{
"cell_type": "markdown",
"id": "65421a2d",
"metadata": {},
"source": [
"We can better decide how to handle these issues by viewing the original sentences containing these tokens.\n",
"Given that `O` and `MISC` classes (corresponding to integers 0 and 1 in our class ordering) can sometimes be ambiguous, they are excluded from our visualization below. This is achieved via the `exclude` argument, a list of tuples `(i, j)` such that tokens predicted as `entities[j]` but labeled as `entities[i]` are ignored."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e13de188",
"metadata": {},
"outputs": [],
"source": [
"display_issues(issues, tokens, pred_probs=pred_probs, labels=labels, \n",
" exclude=[(0, 1), (1, 0)], class_names=entities) "
]
},
{
"cell_type": "markdown",
"id": "96d04902",
"metadata": {},
"source": [
"More than half of the potential label issues correspond to tokens that are incorrectly labeled. As shown above, some examples are ambigious and may require more thoughful handling. cleanlab has also discovered some edge cases such as tokens which are simply punctuations such as `/` and `(`. "
]
},
{
"cell_type": "markdown",
"id": "d213b2b2",
"metadata": {},
"source": [
"### Most common word-level token mislabels \n",
"\n",
"We may also wish to understand which tokens tend to be most commonly mislabeled throughout the entire dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4a006bd",
"metadata": {},
"outputs": [],
"source": [
"info = common_label_issues(issues, tokens, \n",
" labels=labels, \n",
" pred_probs=pred_probs, \n",
" class_names=entities, \n",
" exclude=[(0, 1), (1, 0)]) "
]
},
{
"cell_type": "markdown",
"id": "9c417061",
"metadata": {},
"source": [
"The printed information\u00a0above is also stored in pd.DataFrame `info`."
]
},
{
"cell_type": "markdown",
"id": "a35ef843",
"metadata": {},
"source": [
"### Find sentences containing a particular mislabeled word \n",
"\n",
"You can also only focus on the subset of potentially problematic sentences where a particular token may have been mislabeled."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c8f4e163",
"metadata": {},
"outputs": [],
"source": [
"token_issues = filter_by_token('United', issues, tokens)\n",
"\n",
"display_issues(token_issues, tokens, pred_probs=pred_probs, labels=labels, \n",
" exclude=[(0, 1), (1, 0)], class_names=entities) "
]
},
{
"cell_type": "markdown",
"id": "1759108b",
"metadata": {},
"source": [
"### Sentence label quality score \n",
"\n",
"For best reviewing label issues in a token classification dataset, you want to look at sentences one at a time. Here sentences more likely to contain a label error should be ranked earlier. Cleanlab can provide an overall label quality score for each sentence (ranging from 0 to 1) such that lower scores indicate sentences more likely to contain some mislabeled token. We can also obtain label quality scores for each individual token and manually decide which of these are label issues by thresholding them. For automatically estimating which tokens are mislabeled (and the number of label errors), you should use `find_label_issues()` instead. `get_label_quality_scores()` is useful if you only have time to review a few sentences and want to prioritize which, or if you're specifically aiming to detect label errors with high precision (or high recall) rather than overall estimation of the set of mislabeled tokens."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "db0b5179",
"metadata": {},
"outputs": [],
"source": [
"sentence_scores, token_scores = get_label_quality_scores(labels, pred_probs)\n",
"issues = issues_from_scores(sentence_scores, token_scores=token_scores) \n",
"display_issues(issues, tokens, pred_probs=pred_probs, labels=labels, \n",
" exclude=[(0, 1), (1, 0)], class_names=entities) "
]
},
{
"cell_type": "markdown",
"id": "1759108c",
"metadata": {},
"source": [
"## How does cleanlab.token_classification work?\n",
"\n",
"The underlying algorithms used to produce these scores are described in [this paper](https://arxiv.org/abs/2210.03920)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a18795eb",
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"highlighted_indices = [(2907, 0), (19392, 0), (9962, 4), (8904, 30), (19303, 0), \n",
" (12918, 0), (9256, 0), (11855, 20), (18392, 4), (20426, 28), \n",
" (19402, 21), (14744, 15), (19371, 0), (4645, 2), (83, 9), \n",
" (10331, 3), (9430, 10), (6143, 25), (18367, 0), (12914, 3)] \n",
"\n",
"if not all(x in issues for x in highlighted_indices):\n",
" raise Exception(\"Some highlighted examples are missing from ranked_label_issues.\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}