wehub/cleanlab--cleanlab
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:49:22 +08:00
commit a41b2ab474
303 changed files with 64772 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/tutorials/datalab/audio.ipynb
+791
View File
@@ -0,0 +1,791 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "eVufWTY3jRPx"
},
"source": [
"# Detecting Issues in an Audio Dataset with Datalab\n",
"\n",
"In this 5-minute quickstart tutorial, we use cleanlab to find label issues in the [Spoken Digit dataset](https://www.tensorflow.org/datasets/catalog/spoken_digit) (it's like MNIST for audio). The dataset contains 2,500 audio clips with English pronunciations of the digits 0 to 9 (these are the class labels to predict from the audio).\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Extract features from audio clips (.wav files) using a [pre-trained Pytorch model](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) from HuggingFace that was previously fit to the [VoxCeleb](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/) speech dataset.\n",
"\n",
"- Train a cross-validated linear model using the extracted features and generate out-of-sample predicted probabilities.\n",
"\n",
"- Apply cleanlab's `Datalab` audit to these predictions in order to identify which audio clips in the dataset are likely mislabeled.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have a `model`? Run cross-validation to get out-of-sample `pred_probs`, and then run the code below to audit your dataset and identify any potential issues.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, issue_types={\"label\":{}})\n",
"\n",
"lab.get_issues(\"label\")\n",
" \n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eqsqBq3PiUHA"
},
"source": [
"## 1. Install dependencies and import them\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7nT-U9qc8MS"
},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install huggingface_hub==0.17.0 speechbrain==0.5.13 \n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"# Package versions used: torch==2.1.2 torchaudio==2.1.2 speechbrain==0.5.13\n",
"\n",
"dependencies = [\"cleanlab\", \"huggingface_hub==0.17.0\", \"speechbrain==0.5.13\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "x-oboEbRdhf6"
},
"source": [
"Let's import some of the packages needed throughout this tutorial.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LaEiwXUiVHCS"
},
"outputs": [],
"source": [
"import os\n",
"import pandas as pd\n",
"import numpy as np\n",
"import random\n",
"import torch\n",
"import torchaudio\n",
"import torchaudio\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"SEED = 456 # ensure reproducibility"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This (optional) cell is hidden from docs.cleanlab.ai \n",
"\n",
"def set_seed(seed=0):\n",
" \"\"\"Ensure reproducibility.\"\"\"\n",
" np.random.seed(seed)\n",
" torch.manual_seed(seed)\n",
" torch.backends.cudnn.deterministic = True\n",
" torch.backends.cudnn.benchmark = False\n",
" torch.cuda.manual_seed_all(seed)\n",
"\n",
"\n",
"set_seed(SEED)\n",
"pd.options.display.max_colwidth = 500"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SOen_sxQidLC"
},
"source": [
"## 2. Load the data\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uHVskN2eeNj6"
},
"source": [
"We must first fetch the dataset. To run the below command, you'll need to have `wget` installed; alternatively you can manually navigate to the link in your browser and download from there.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GRDPEg7-VOQe",
"outputId": "cb886220-e86e-4a77-9f3a-d7844c37c3a6"
},
"outputs": [],
"source": [
"%%capture\n",
"\n",
"!wget https://github.com/Jakobovski/free-spoken-digit-dataset/archive/v1.0.9.tar.gz\n",
"!mkdir spoken_digits\n",
"!tar -xf v1.0.9.tar.gz -C spoken_digits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tRvNnyB0e_IE"
},
"source": [
"The audio data are .wav files in the `recordings/` folder. Note that the label for each audio clip (i.e. digit from 0 to 9) is indicated in the prefix of the file name (e.g. `6_nicolas_32.wav` has the label 6). If instead applying cleanlab to your own dataset, its classes should be represented as integer indices 0, 1, ..., num_classes - 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FDA5sGZwUSur",
"outputId": "0cedc509-63fd-4dc3-d32f-4b537dfe3895"
},
"outputs": [],
"source": [
"DATA_PATH = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/\"\n",
"\n",
"# Get list of .wav file names\n",
"# os.listdir order is nondeterministic, so for reproducibility,\n",
"# we sort first and then do a deterministic shuffle\n",
"file_names = sorted(i for i in os.listdir(DATA_PATH) if i.endswith(\".wav\"))\n",
"random.Random(SEED).shuffle(file_names)\n",
"\n",
"file_paths = [os.path.join(DATA_PATH, name) for name in file_names]\n",
"\n",
"# Check out first 3 files\n",
"file_paths[:3]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xi2592bVhSab"
},
"source": [
"Let's listen to some example audio clips from the dataset. We introduce a `display_example` function to process the .wav file so we can listen to it in this notebook (can skip these details)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the implementation of `display_example` **(click to expand)**</summary>\n",
"\n",
"```python\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import torch\n",
"import torchaudio\n",
"from pathlib import Path\n",
"from IPython import display\n",
"\n",
"# Utility function for loading audio files and making sure the sample rate is correct.\n",
"def load_wav_16k_mono(filename):\n",
" \"\"\"Load a WAV file, convert it to a float tensor, resample to 16 kHz single-channel audio.\"\"\"\n",
" # Load audio file with torchaudio\n",
" waveform, sample_rate = torchaudio.load(filename)\n",
" \n",
" # Convert to mono if stereo\n",
" if waveform.shape[0] > 1:\n",
" waveform = torch.mean(waveform, dim=0, keepdim=True)\n",
" \n",
" # Resample to 16kHz if needed\n",
" if sample_rate != 16000:\n",
" resampler = torchaudio.transforms.Resample(sample_rate, 16000)\n",
" waveform = resampler(waveform)\n",
" \n",
" return waveform.squeeze()\n",
"\n",
"\n",
"def display_example(wav_file_name, audio_rate=16000):\n",
" \"\"\"Allows us to listen to any wav file and displays its given label in the dataset.\"\"\"\n",
" wav_file_example = load_wav_16k_mono(wav_file_name)\n",
" label = Path(wav_file_name).parts[-1].split(\"_\")[0]\n",
" print(f\"Given label for this example: {label}\")\n",
" display.display(display.Audio(wav_file_example.numpy(), rate=audio_rate))\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import torch\n",
"import torchaudio\n",
"from pathlib import Path\n",
"from IPython import display\n",
"\n",
"# Utility function for loading audio files and making sure the sample rate is correct.\n",
"def load_wav_16k_mono(filename):\n",
" \"\"\"Load a WAV file, convert it to a float tensor, resample to 16 kHz single-channel audio.\"\"\"\n",
" # Load audio file with torchaudio\n",
" waveform, sample_rate = torchaudio.load(filename)\n",
" \n",
" # Convert to mono if stereo\n",
" if waveform.shape[0] > 1:\n",
" waveform = torch.mean(waveform, dim=0, keepdim=True)\n",
" \n",
" # Resample to 16kHz if needed\n",
" if sample_rate != 16000:\n",
" resampler = torchaudio.transforms.Resample(sample_rate, 16000)\n",
" waveform = resampler(waveform)\n",
" \n",
" return waveform.squeeze()\n",
"\n",
"\n",
"def display_example(wav_file_name, audio_rate=16000):\n",
" \"\"\"Allows us to listen to any wav file and displays its given label in the dataset.\"\"\"\n",
" wav_file_example = load_wav_16k_mono(wav_file_name)\n",
" label = Path(wav_file_name).parts[-1].split(\"_\")[0]\n",
" print(f\"Given label for this example: {label}\")\n",
" display.display(display.Audio(wav_file_example.numpy(), rate=audio_rate))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2bLlDRI6hzon"
},
"source": [
"Click the play button below to listen to this example .wav file. Feel free to change the `wav_file_name_example` variable below to listen to other audio clips in the dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "dLBvUZLlII5w",
"outputId": "c6a4917f-4a82-4a89-9193-415072e45550"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/7_jackson_43.wav\" # change this to hear other examples\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-QvbZA7yHwkh"
},
"source": [
"## 3. Use pre-trained SpeechBrain model to featurize audio\n",
"\n",
"The [SpeechBrain](https://github.com/speechbrain/speechbrain) package offers many Pytorch neural networks that have been pretrained for speech recognition tasks. Here we instantiate an audio feature extractor using SpeechBrain's `EncoderClassifier`. We'll use the \"spkrec-xvect-voxceleb\" network which has been pre-trained on the [VoxCeleb](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/) speech dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vL9lkiKsHvKr"
},
"outputs": [],
"source": [
"%%capture\n",
"\n",
"from speechbrain.pretrained import EncoderClassifier\n",
"\n",
"feature_extractor = EncoderClassifier.from_hparams(\n",
" \"speechbrain/spkrec-xvect-voxceleb\",\n",
" # run_opts={\"device\":\"cuda\"} # Uncomment this to run on GPU if you have one (optional)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vXlE6IK4ibcr"
},
"source": [
"Next, we run the audio clips through the pre-trained model to extract vector features (aka embeddings)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 143
},
"id": "obQYDKdLiUU6",
"outputId": "4e923d5c-2cf4-4a5c-827b-0a4fea9d87e4"
},
"outputs": [],
"source": [
"# Create dataframe with .wav file names\n",
"df = pd.DataFrame(file_paths, columns=[\"wav_audio_file_path\"])\n",
"df[\"label\"] = df.wav_audio_file_path.map(lambda x: int(Path(x).parts[-1].split(\"_\")[0]))\n",
"df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I8JqhOZgi94g"
},
"outputs": [],
"source": [
"def extract_audio_embeddings(model, wav_audio_file_path: str) -> tuple:\n",
" \"\"\"Feature extractor that embeds audio into a vector.\"\"\"\n",
" signal, fs = torchaudio.load(wav_audio_file_path) # Load audio signal as tensor\n",
" embeddings = model.encode_batch(\n",
" signal\n",
" ) # Pass tensor through pretrained neural net and extract representation\n",
" return embeddings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2FSQ2GR9R_YA"
},
"outputs": [],
"source": [
"# Extract audio embeddings\n",
"embeddings_list = []\n",
"for i, file_name in enumerate(df.wav_audio_file_path): # for each .wav file name\n",
" embeddings = extract_audio_embeddings(feature_extractor, file_name)\n",
" embeddings_list.append(embeddings.cpu().numpy())\n",
"\n",
"embeddings_array = np.squeeze(np.array(embeddings_list))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dELkcdXgjTn_"
},
"source": [
"Now we have our features in a 2D numpy array. Each row in the array corresponds to an audio clip. We're now able to represent each audio clip as a 512-dimensional feature vector!\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kAkY31IVXyr8",
"outputId": "fd70d8d6-2f11-48d5-ae9c-a8c97d453632"
},
"outputs": [],
"source": [
"print(embeddings_array)\n",
"print(\"Shape of array: \", embeddings_array.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o4RBcaARmfVG"
},
"source": [
"## 4. Fit linear model and compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y9BIVyI9kHa4"
},
"source": [
"A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted network embeddings.\n",
"\n",
"To identify label issues, cleanlab requires a probabilistic prediction from your model for every datapoint that should be considered. However these predictions will be _overfit_ (and thus unreliable) for datapoints the model was previously trained on. cleanlab is intended to only be used with **out-of-sample** predicted probabilities, i.e. on datapoints held-out from the model during the training.\n",
"\n",
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset, by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. An additional benefit of cross-validation is that it provides more reliable evaluation of our model than a single training/validation split. We can obtain cross-validated out-of-sample predicted probabilities from any classifier via the [cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html) wrapper provided in scikit-learn.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "i_drkY9YOcw4"
},
"outputs": [],
"source": [
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"model = LogisticRegression(C=0.01, max_iter=1000, tol=1e-2, random_state=SEED)\n",
"\n",
"num_crossval_folds = 5 # can decrease this value to reduce runtime, or increase it to get better results\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=embeddings_array, y=df.label.values, cv=num_crossval_folds, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FW1yI9Ryrfkj"
},
"source": [
"For each audio clip, the corresponding predicted probabilities in `pred_probs` are produced by a copy of our `LogisticRegression` model that has never been trained on this audio clip. Hence we call these predictions _out-of-sample_. An additional benefit of cross-validation is that it provides more reliable evaluation of our model than a single training/validation split.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "_b-AQeoXOc7q",
"outputId": "15ae534a-f517-4906-b177-ca91931a8954"
},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score\n",
"\n",
"predicted_labels = pred_probs.argmax(axis=1)\n",
"cv_accuracy = accuracy_score(df.label.values, predicted_labels)\n",
"print(f\"Cross-validated estimate of accuracy on held-out data: {cv_accuracy}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SPz8WBwIlxUE"
},
"source": [
"## 5. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "laui-jXMm6qR"
},
"source": [
"Based on the given labels, out-of-sample predicted probabilities and features, cleanlab can quickly help us identify label issues in our dataset. For a dataset with N examples from K classes, the labels should be a 1D array of length N and predicted probabilities should be a 2D (N x K) array. \n",
"\n",
"Here, we use cleanlab to find potential label errors in our data. `Datalab` has several ways of loading the data. In this case, we can just pass the DataFrame created above to instantiate the object. We will then pass in the predicted probabilites to the `find_issues()` method so that Datalab can use them to find potential label errors in our data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(df, label_name=\"label\")\n",
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\":{}})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can view the results of running Datalab by calling the `report` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We observe from the report that cleanlab has found some label issues in our dataset. Let us investigate these examples further.\n",
"\n",
"We can view the more details about the label quality for each example using the `get_issues` method, specifying `label` as the issue type."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"label_issues.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled).\n",
"\n",
"We can then filter for the examples that have been identified as a label error:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identified_label_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = identified_label_issues.sort_values(\"label_score\").index\n",
"\n",
"print(f\"Here are indices of the most likely errors: \\n {lowest_quality_labels.values}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iI07jQ0BnTgt"
},
"source": [
"These examples flagged by cleanlab are those worth inspecting more closely."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 237
},
"id": "FQwRHgbclpsO",
"outputId": "fee5c335-c00e-4fcc-f22b-718705e93182"
},
"outputs": [],
"source": [
"df.iloc[lowest_quality_labels]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PsDmd5WDnZJG"
},
"source": [
"Let's listen to some audio clips below of label issues that were identified in this list.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p9jLn3Lp85rU"
},
"source": [
"In this example, the given label is **6** but it sounds like **8**.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "ff1NFVlDoysO",
"outputId": "8141a036-44c1-4349-c338-880432513e37"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_14.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HwokyN0bfVsn"
},
"source": [
"In the three examples below, the given label is **6** but they sound quite ambiguous.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "GZgovGkdiaiP",
"outputId": "d76b2ccf-8be2-4f3a-df4c-2c5c99150db7"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_36.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"id": "lfa2eHbMwG8R",
"outputId": "6627ebe2-d439-4bf5-e2cb-44f6278ae86c"
},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_yweweler_35.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wav_file_name_example = \"spoken_digits/free-spoken-digit-dataset-1.0.9/recordings/6_nicolas_8.wav\"\n",
"display_example(wav_file_name_example)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-rf8iSngtV83"
},
"source": [
"You can see that even widely-used datasets like Spoken Digit contain problematic labels. Never blindly trust your data! You should always check it for potential issues, many of which can be easily identified by cleanlab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"highlighted_indices = [1946, 516, 469, 2132] # verify these examples were found in find_label_issues\n",
"if not all(x in lowest_quality_labels for x in highlighted_indices):\n",
" raise Exception(\"Some highlighted examples are missing from label_issues_indices.\")"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"name": "audio_quickstart_tutorial_deterministic.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/datalab_advanced.ipynb
+812
View File
@@ -0,0 +1,812 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datalab: Advanced workflows to audit your data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cleanlab offers a `Datalab` object to identify various issues in your machine learning datasets that may negatively impact models if not addressed. By default, `Datalab` can help you identify noisy labels, outliers, (near) duplicates, and other types of problems that commonly occur in real-world data.\n",
"\n",
"`Datalab` performs these checks by utilizing the (probabilistic) predictions from *any* ML model that has already been trained or its learned representations of the data. Underneath the hood, this class calls all the appropriate cleanlab methods for your dataset and provided model outputs, in order to best audit the data and alert you of important issues. This makes it easy to apply many functionalities of this library all within a single line of code. \n",
"\n",
"**This tutorial will demonstrate some advanced functionalities of Datalab including:**\n",
"\n",
"- Incremental issue search\n",
"- Specifying nondefault arguments to issue checks\n",
"- Save and load Datalab objects\n",
"- Adding a custom IssueManager\n",
"\n",
"If you are new to `Datalab`, check out this [quickstart tutorial](datalab_quickstart.html) for a 5-min introduction!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you have some `features` as well? Run the code below to examine your dataset for multiple types of issues.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(features=your_feature_matrix, pred_probs=your_pred_probs)\n",
"\n",
"lab.report()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Install and import required dependencies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` has additional dependencies that are not included in the standard installation of cleanlab.\n",
"\n",
"You can use pip to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib \n",
"!pip install \"cleanlab[datalab]\"\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"] # TODO: make sure this list is updated\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create and load the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll load a toy classification dataset for this tutorial. The dataset has two numerical features and a label column with three classes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code for data generation. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate = create_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We make a scatter plot of the features, with a color corresponding to the observed labels. Incorrect given labels are highlighted in red if they do not match the true label, outliers highlighted with an a black cross, and duplicates highlighted with a cyan cross."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(8, 6.5))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-3.5, 9.0)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.75, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.75, 0.45], title=\"Type of Issue\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In real-world scenarios, you won't know the true labels or the distribution of the features, so we won't use these in this tutorial, except for evaluation purposes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get out-of-sample predicted probabilities from a classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To detect certain types of issues in classification data (e.g. label errors), `Datalab` relies on predicted class probabilities from a trained model. Ideally, the prediction for each example should be out-of-sample (to avoid overfitting), coming from a copy of the model that was not trained on this example. \n",
"\n",
"This tutorial uses a simple logistic regression model \n",
"and the `cross_val_predict()` function from scikit-learn to generate out-of-sample predicted class probabilities for every example in the training set. You can replace this with *any* other classifier model and train it with cross-validation to get out-of-sample predictions.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=X_train, y=noisy_labels, cv=5, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Instantiate Datalab object"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we instantiate the Datalab object that will be used in the remainder in the tutorial by passing in the data created above.\n",
"\n",
"`Datalab` has several ways of loading the data. In this case, we'll simply wrap the training features and noisy labels in a dictionary so that we can pass it to `Datalab`.\n",
"\n",
"Other supported data formats for `Datalab` include: [HuggingFace Datasets](https://huggingface.co/docs/datasets/index) and [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). `Datalab` works across most data modalities (image, text, tabular, audio, etc). It is intended to find issues that commonly occur in datasets for which you have trained a supervised ML model, regardless of the type of data.\n",
"\n",
"Currently, pandas DataFrames that contain categorical columns might cause some issues when instantiating the `Datalab` object, so it is recommended to ensure that your DataFrame does not contain any categorical columns, or use other data formats (eg. python dictionary, HuggingFace Datasets) to pass in your data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_train, \"y\": noisy_labels}\n",
"\n",
"lab = Datalab(data, label_name=\"y\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 1**: Incremental issue search "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can call `find_issues` multiple times on a `Datalab` object to detect issues one type at a time.\n",
"\n",
"This is done via the `issue_types` argument which accepts a dictionary of issue types and any corresponding keyword arguments to specify nondefault keyword arguments to use for detecting each type of issues. In this first call, we only want to detect label issues, which are detected solely based on `pred_probs`, hence there is no need for us to pass in `features` here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\": {}}) \n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can check for additional types of issues with the same `Datalab`. Here, we would like to detect outliers and near duplicates which both utilize the features of the data.\n",
"\n",
"Notice that this second call to `find_issues()` updates the output of `report()`, we can see the existing label issues detected alongside the new issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(features=data[\"X\"], issue_types={\"outlier\": {}, \"near_duplicate\": {}})\n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 2**: Specifying nondefault arguments"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also overwrite previously-executed checks for a type of issue. Here we re-run the detection of outliers, but specify that different non-default settings should be used (in this case, the number of neighbors `k` compared against to determine which datapoints are outliers). \n",
"The results from this new detection will replace the original outlier detection results in the updated `Datalab`. You could similarly specify non-default settings for other issue types in the first call to `Datalab.find_issues()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(features=data[\"X\"], issue_types={\"outlier\": {\"k\": 30}})\n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also increase the verbosity of the `report` to see additional information about the data issues and control how many top-ranked examples are shown for each issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.report(num_examples=10, verbosity=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how the number of flagged outlier issues has changed after specfying different settings to use for outlier detection."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 3**: Save and load Datalab objects"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A `Datalab` can be saved to a folder at a specified path. In a future Python process, this path can be used to load the `Datalab` from file back into memory. Your dataset is not saved as part of this process, so you'll need to save/load it separately to keep working with it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"path = \"datalab-files\"\n",
"lab.save(path, force=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can load a `Datalab` object we have on file and view the previously detected issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"new_lab = Datalab.load(path)\n",
"new_lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## **Functionality 4**: Adding a custom IssueManager"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` detects pre-defined types of issues for you in one line of code: `find_issues()`. What if you want to check for other custom types of issues along with these pre-defined types, all within the same line of code?\n",
"\n",
"All issue types in `Datalab` are subclasses of cleanlab's `IssueManager` class.\n",
"To register a custom issue type for use with `Datalab`, simply also make it a subclass of `IssueManager`.\n",
"\n",
"The necessary members to implement in the subclass are:\n",
"\n",
"- A class variable called `issue_name` that acts as a unique identifier for the type of issue.\n",
"- An instance method called `find_issues` that:\n",
" - Computes a quality score for each example in the dataset (between 0-1), in terms of how *unlikely* it is to be an issue.\n",
" - Flags each example as an issue or not (may be based on thresholding the quality scores).\n",
" - Combine these in a dataframe that is assigned to an `issues` attribute of the `IssueManager`.\n",
" - Define a summary score for the overall quality of entire dataset, in terms of this type of issue. Set this score as part of the `summary` attribute of the `IssueManager`.\n",
" \n",
"To demonstrate this, we create an arbitrary issue type that checks the divisibility of an example's index in the dataset by 13."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from cleanlab.datalab.internal.issue_manager import IssueManager\n",
"from cleanlab.datalab.internal.issue_manager_factory import register\n",
"\n",
"\n",
"def scoring_function(idx: int, div: int = 13) -> float:\n",
" if idx == 0:\n",
" # Zero excluded from the divisibility check, gets the highest score\n",
" return 1\n",
" rem = idx % div\n",
" inv_scale = idx // div\n",
" if rem == 0:\n",
" return 0.5 * (1 - np.exp(-0.1*(inv_scale-1)))\n",
" else:\n",
" return 1 - 0.49 * (1 - np.exp(-inv_scale**0.5))*rem/div\n",
"\n",
"\n",
"@register # register this issue type for use with Datalab\n",
"class SuperstitionIssueManager(IssueManager):\n",
" \"\"\"A custom issue manager that keeps track of issue indices that\n",
" are divisible by 13.\n",
" \"\"\"\n",
" description: str = \"Examples with indices that are divisible by 13 may be unlucky.\" # Optional\n",
" issue_name: str = \"superstition\"\n",
"\n",
" def find_issues(self, div=13, **_) -> None:\n",
" ids = self.datalab.issues.index.to_series()\n",
" issues_mask = ids.apply(lambda idx: idx % div == 0 and idx != 0)\n",
" scores = ids.apply(lambda idx: scoring_function(idx, div))\n",
" self.issues = pd.DataFrame(\n",
" {\n",
" f\"is_{self.issue_name}_issue\": issues_mask,\n",
" self.issue_score_key: scores,\n",
" },\n",
" )\n",
" summary_score = 1 - sum(issues_mask) / len(issues_mask)\n",
" self.summary = self.make_summary(score = summary_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once registered, this `IssueManager` will perform custom issue checks when `find_issues` is called on a `Datalab` instance.\n",
"\n",
"As our `Datalab` instance here already has results from the outlier and near duplicate checks, we perform the custom issue check separately."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.find_issues(issue_types={\"superstition\": {}})\n",
"lab.report()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"vscode": {
"interpreter": {
"hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/datalab_quickstart.ipynb
+843
View File
@@ -0,0 +1,843 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datalab: A unified audit to detect all kinds of issues in data and labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cleanlab offers a `Datalab` object that can identify various issues in your machine learning datasets, such as noisy labels, outliers, (near) duplicates, drift, and other types of problems common in real-world data. These data issues may negatively impact models if not addressed. `Datalab` utilizes *any* ML model you have already trained for your data to diagnose these issues, it only requires access to either: (probabilistic) predictions from your model or its learned representations of the data.\n",
"\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Compute out-of-sample predicted probabilities for a sample dataset using cross-validation.\n",
"- Use `Datalab` to identify issues such as noisy labels, outliers, (near) duplicates, and other types of problems \n",
"- View the issue summaries and other information about our sample dataset\n",
"\n",
"You can easily replace our demo dataset with your own image/text/tabular/audio/etc dataset, and then run the same code to discover what sort of issues lurk within it!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you also have some numeric `features` (or model embeddings of data)? Run the code below to examine your dataset for multiple types of issues.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(features=your_feature_matrix, pred_probs=your_pred_probs)\n",
"\n",
"lab.report()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install and import required dependencies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Datalab` has additional dependencies that are not included in the standard installation of cleanlab.\n",
"\n",
"You can use pip to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib\n",
"!pip install \"cleanlab[datalab]\"\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"] # TODO: make sure this list is updated\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Create and load the data (can skip these details)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll load a toy classification dataset for this tutorial. The dataset has two numerical features and a label column with three possible classes. Each example is classified as either: *low*, *mid* or *high*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code for data generation. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(250, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.5, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
" # Assign few datapoints to rare class\n",
" random_idx = np.random.randint(0, X_train.shape[0], 3)\n",
" noisy_labels[random_idx] = \"max\"\n",
" noisy_labels_idx[random_idx] = np.max(y_bin_idx) + 1\n",
" \n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate = create_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We make a scatter plot of the features, with a color corresponding to the observed labels. Incorrect given labels are highlighted in red if they do not match the true label, outliers highlighted with an a black cross, and duplicates highlighted with a cyan cross."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(8, 6.5))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-3.5, 9.0)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.75, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.75, 0.45], title=\"Type of Issue\", alignment=\"left\", title_fontproperties={\"weight\":\"semibold\"})\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
"\n",
"\n",
" outlier = ax.scatter(X_out[:, 0], X_out[:, 1], color=\"k\", marker=\"x\", s=100, linewidth=2, label=\"Outlier\")\n",
"\n",
" # Plot the exact duplicate\n",
" dups = ax.scatter(\n",
" X_duplicate[:, 0],\n",
" X_duplicate[:, 1],\n",
" color=\"c\",\n",
" marker=\"x\",\n",
" s=100,\n",
" linewidth=2,\n",
" label=\"Duplicates\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err, outlier, dups], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_data(X_train, y_train_idx, noisy_labels_idx, X_out, X_duplicate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In real-world scenarios, you won't know the true labels or the distribution of the features, so we won't use these in this tutorial, except for evaluation purposes.\n",
"\n",
"\n",
"\n",
"`Datalab` has several ways of loading the data.\n",
"In this case, we'll simply wrap the training features and noisy labels in a dictionary so that we can pass it to `Datalab`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_train, \"y\": noisy_labels}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Other supported data formats for `Datalab` include: [HuggingFace Datasets](https://huggingface.co/docs/datasets/index) and [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). `Datalab` works across most data modalities (image, text, tabular, audio, etc). It is intended to find issues that commonly occur in datasets for which you have trained a supervised ML model, regardless of the type of data.\n",
"\n",
"Currently, pandas DataFrames that contain categorical columns might cause some issues when instantiating the `Datalab` object, so it is recommended to ensure that your DataFrame does not contain any categorical columns, or use other data formats (eg. python dictionary, HuggingFace Datasets) to pass in your data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Get out-of-sample predicted probabilities from a classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To detect certain types of issues in classification data (e.g. label errors), `Datalab` relies on predicted class probabilities from a trained model. Ideally, the prediction for each example should be out-of-sample (to avoid overfitting), coming from a copy of the model that was not trained on this example. \n",
"\n",
"This tutorial uses a simple logistic regression model \n",
"and the `cross_val_predict()` function from scikit-learn to generate out-of-sample predicted class probabilities for every example in the training set. You can replace this with *any* other classifier model and train it with cross-validation to get out-of-sample predictions.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=data[\"X\"], y=data[\"y\"], cv=5, method=\"predict_proba\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use Datalab to find issues in the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We create a `Datalab` object from the dataset, also providing the name of the label column in the dataset. Only instantiate one `Datalab` object per dataset, and note that only classification datasets are supported for now.\n",
"\n",
"All that is need to audit your data is to call `find_issues()`.\n",
"This method accepts various inputs like: predicted class probabilities, numeric feature representations of the data. The more information you provide here, the more thoroughly `Datalab` will audit your data! Note that `features` should be some numeric representation of each example, either obtained through preprocessing transformation of your raw data or embeddings from a (pre)trained model. In this case, our data is already entirely numeric so we just provide the features directly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(data, label_name=\"y\")\n",
"lab.find_issues(pred_probs=pred_probs, features=data[\"X\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's review the results of this audit using `report()`.\n",
"This provides a high-level summary of each type of issue found in the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Learn more about the issues in your dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Datalab detects all sorts of issues in a dataset and what to do with the findings will vary case-by-case.\n",
"\n",
"To conceptually understand how each type of issue is defined and what it means if detected in your data, check out the [Issue Type Descriptions](../../cleanlab/datalab/guide/issue_type_description.html) page. The [Datalab Issue Types](https://docs.cleanlab.ai/stable/cleanlab/datalab/guide/issue_type_description.html) page also lists additional types of issues that `Datalab.find_issues()` can detect, as well as optional parameters you can specify for greater control over how your data are checked.\n",
"\n",
"Datalab offers several methods to understand more details about a particular issue in your dataset.\n",
"The `get_issue_summary()` method fetches summary statistics regarding how severe each type of issue is overall across the whole dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issue_summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the returned summary DataFrame: LOWER `score` values indicate types of issues that are MORE severe *overall* across the dataset (lower-quality data in terms of this issue), HIGHER `num_issues` values indicate types of issues that are MORE severe *overall* across the dataset (more datapoints appear to exhibit this issue).\n",
"\n",
"We can also only request the summary for a particular type of issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issue_summary(\"label\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `get_issues()` method returns information for each *individual example* in the dataset including: whether or not it is plagued by this issue (Boolean), as well as a *quality score* (numeric value betweeen 0 to 1) quantifying how severe this issue appears to be for this particular example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issues().head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each example receives a separate *quality score* for each issue type (eg. `outlier_score` is the *quality score* for the `outlier` issue type, quantifying *how typical* each datapoint appears to be). LOWER scores indicate MORE severe instances of the issue, so the most-concerning datapoints have the lowest quality scores. Sort by these scores to see the most-concerning examples in your dataset for each type of issue. The quality scores are directly comparable between examples/datasets, but not across different issue types.\n",
"\n",
"Similar to above, we can pass the type of issue as a argument to `get_issues()` to get the information for one particular type of issue.\n",
"As an example, let's see the examples identified as having the most severe *label* issues:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"examples_w_issue = (\n",
" lab.get_issues(\"label\")\n",
" .query(\"is_label_issue\")\n",
" .sort_values(\"label_score\")\n",
")\n",
"\n",
"examples_w_issue.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inspecting the labels for some of these top-ranked examples, we find their given label was indeed incorrect."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Get additional information \n",
"\n",
"Miscellaneous additional information (statistics, intermediate results, etc) related to a particular issue type can be accessed via `get_info(issue_name)`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"label_issues_info = lab.get_info(\"label\")\n",
"label_issues_info[\"classes_by_label_quality\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This portion of the info shows overall label quality summaries of all examples annotated as a particular class (e.g. the `Label Issues` column is the estimated number of examples labeled as this class that should actually have a different label).\n",
"To learn more about this, see the documentation for the [cleanlab.dataset.rank_classes_by_label_quality](../../cleanlab/dataset.html#cleanlab.dataset.rank_classes_by_label_quality)\n",
"method.\n",
"\n",
"You can view all sorts of information regarding your dataset using the `get_info()` method with no arguments passed. This is not printed here as it returns a huge dictionary but feel free to check it out yourself! Don't worry if you don't understand all of the miscellaneous information in this `info` dictionary, none of it is critical to diagnose the issues in your dataset. Understanding miscellaneous info may require reading the documentation of the miscellaneous cleanlab functions which computed it.\n",
"\n",
"#### Near duplicate issues \n",
"\n",
"Let's also inspect the examples flagged as (near) duplicates.\n",
"For each such example, the `near_duplicate_sets` column below indicates *which* other examples in the dataset are highly similar to it (this value is empty for examples not flagged as nearly duplicated). The `near_duplicate_score` quantifies *how similar* each example is to its nearest neighbor in the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab.get_issues(\"near_duplicate\").query(\"is_near_duplicate_issue\").sort_values(\"near_duplicate_score\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?). \n",
"\n",
"Other issues detected in this tutorial dataset include **outliers** and **class imbalance**, see the [Issue Type Descriptions](../../cleanlab/datalab/guide/issue_type_description.html) for more information. `Datalab` makes it very easy to check your datasets for all sorts of issues that are important to deal with for training robust models. The inputs it uses to detect issues can come from *any* model you have trained (the better your model, the more accurate the issue detection will be).\n",
"\n",
"To learn more, check out this [example notebook](https://github.com/cleanlab/examples/blob/master/datalab_image_classification/datalab.ipynb) (demonstrates Datalab applied to a real dataset) and the [advanced Datalab tutorial](datalab_advanced.html) (demonstrates configuration and customization options to exert greater control)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"from sklearn.metrics import roc_auc_score\n",
"\n",
"def precision_at_k(predicted_indices, true_indices, k):\n",
" return len(set(predicted_indices[:k]).intersection(set(true_indices))) / k\n",
"\n",
"def recall_at_k(predicted_indices, true_indices, k):\n",
" return len(set(predicted_indices[:k]).intersection(set(true_indices))) / len(true_indices)\n",
"\n",
"def jaccard_similarity(l1, l2):\n",
" s1 = set(l1)\n",
" s2 = set(l2)\n",
" intersect_set = s1.intersection(s2)\n",
" union_set = s1.union(s2)\n",
" if len(intersect_set) == 0:\n",
" return 0\n",
" return len(intersect_set) / len(union_set)\n",
"\n",
"label_issues = lab.get_issues(\"label\")\n",
"predicted_label_issues_indices = (\n",
" label_issues.query(\"is_label_issue\").sort_values(\"label_score\").index.to_list()\n",
")\n",
"predicted_label_issues_indices_by_score = (\n",
" label_issues.sort_values(\"label_score\").index.to_list()\n",
")\n",
"label_issue_indices = np.where(y_train_idx != noisy_labels_idx)[0]\n",
"\n",
"label_quality_scores = label_issues[\"label_score\"].tolist()\n",
"Z = (y_train_idx == noisy_labels_idx).astype(float).tolist()\n",
"\n",
"predicted_outlier_issues_indices = (\n",
" lab.get_issues(\"outlier\").query(\"is_outlier_issue\").index.to_list()\n",
")\n",
"outlier_issue_indices = list(range(125, 130+1))\n",
"exact_duplicate_idx = [index for index, elem in enumerate(X_train) if (elem == X_duplicate).all()][0]\n",
"if exact_duplicate_idx >= 125: # if the random index selected to create a duplicate >= 125, then the last point is also an outlier\n",
" outlier_issue_indices.append(131)\n",
"\n",
"predicted_duplicate_issues_indices = (\n",
" lab.get_issues(\"near_duplicate\").query(\"is_near_duplicate_issue\").index.tolist()\n",
")\n",
"duplicate_issue_indices = [exact_duplicate_idx, 129, 130, 131]\n",
"\n",
"k = len(label_issue_indices)\n",
"assert precision_at_k(predicted_label_issues_indices, label_issue_indices, k) >= 0.75\n",
"assert recall_at_k(predicted_label_issues_indices, label_issue_indices, k) >= 0.75\n",
"assert precision_at_k(predicted_label_issues_indices_by_score, label_issue_indices, k) == 1.0\n",
"assert recall_at_k(predicted_label_issues_indices_by_score, label_issue_indices, k) == 1.0\n",
"assert roc_auc_score(Z, label_quality_scores) > 0.9\n",
"\n",
"assert jaccard_similarity(predicted_outlier_issues_indices, outlier_issue_indices) > 0.9\n",
"assert jaccard_similarity(predicted_duplicate_issues_indices, duplicate_issue_indices) > 0.9\n",
"\n",
"expected_issue_types = set([\"label\", \"outlier\", \"near_duplicate\", \"class_imbalance\"])\n",
"detected_issue_types = set(lab.get_issue_summary()[lab.get_issue_summary()[\"num_issues\"] > 0][\"issue_type\"])\n",
"assert detected_issue_types == expected_issue_types"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
},
"vscode": {
"interpreter": {
"hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/image.ipynb
+1319
View File
File diff suppressed because it is too large Load Diff
docs/source/tutorials/datalab/index.rst
+14
View File
@@ -0,0 +1,14 @@
Datalab Tutorials
=================
.. toctree::
:maxdepth: 1
Detecting Common Data Issues with Datalab <datalab_quickstart>
Advanced Data Auditing with Datalab <datalab_advanced>
Text Data <text>
Tabular Data (Numeric/Categorical) <tabular>
Image Data <image>
Audio Data <audio>
Guide to Datalab Issue Types </cleanlab/datalab/guide/issue_type_description>
Miscellaneous workflows with Datalab <workflows>
docs/source/tutorials/datalab/tabular.ipynb
+548
View File
@@ -0,0 +1,548 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Detecting Issues in Tabular Data (Numeric/Categorical columns) with Datalab\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use Datalab to detect various issues in a classification dataset with tabular (numeric/categorical) features. Tabular (or *structured*) data are typically organized in a row/column format and stored in a SQL database or file types like: CSV, Excel, or Parquet. Here we consider a Student Grades dataset, which contains over 900 individuals who have three exam grades and some optional notes, each being assigned a letter grade (their class label). cleanlab automatically identifies _hundreds_ of examples in this dataset that were mislabeled with the incorrect final grade selected. You can run the same code from this tutorial to detect incorrect information in your own tabular classification datasets.\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Train a classifier model (here scikit-learn's HistGradientBoostingClassifier, although any model could be used) and use this classifier to compute (out-of-sample) predicted class probabilities via cross-validation.\n",
"\n",
"- Create a K nearest neighbours (KNN) graph between the examples in the dataset.\n",
"\n",
"- Identify issues in the dataset with cleanlab's `Datalab` audit applied to the predictions and KNN graph.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on your original data labels? Have a `knn_graph` computed between dataset examples (reflecting similarity in their feature values)? Run the code below to find issues in your dataset.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, knn_graph=knn_graph)\n",
"\n",
"lab.get_issues()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.model_selection import cross_val_predict\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.ensemble import HistGradientBoostingClassifier\n",
"from sklearn.neighbors import NearestNeighbors\n",
"\n",
"from cleanlab import Datalab\n",
"\n",
"SEED = 100 # for reproducibility\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and process the data\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We first load the data features and labels (which are possibly noisy).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"grades_data = pd.read_csv(\"https://s.cleanlab.ai/grades-tabular-demo-v2.csv\")\n",
"grades_data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_raw = grades_data[[\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]]\n",
"labels = grades_data[\"letter_grade\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we preprocess the data. Here we apply one-hot encoding to columns with categorical values and standardize the values in numeric columns."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cat_features = [\"notes\"]\n",
"X_encoded = pd.get_dummies(X_raw, columns=cat_features, drop_first=True)\n",
"\n",
"numeric_features = [\"exam_1\", \"exam_2\", \"exam_3\"]\n",
"scaler = StandardScaler()\n",
"X_processed = X_encoded.copy()\n",
"X_processed[numeric_features] = scaler.fit_transform(X_encoded[numeric_features])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"Assign your data's features to variable `X` and its labels to variable `labels` instead.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Select a classification model and compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we use a simple histogram-based gradient boosting model (similar to XGBoost), but you can choose any suitable scikit-learn model for this tutorial.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"clf = HistGradientBoostingClassifier()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find potential labeling errors, cleanlab requires a probabilistic prediction from your model for every datapoint. However, these predictions will be _overfitted_ (and thus unreliable) for examples the model was previously trained on. For the best results, cleanlab should be applied with **out-of-sample** predicted class probabilities, i.e., on examples held out from the model during the training.\n",
"\n",
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n",
"We can implement this via the `cross_val_predict` method from scikit-learn.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_crossval_folds = 5 \n",
"pred_probs = cross_val_predict(\n",
" clf,\n",
" X_processed,\n",
" labels,\n",
" cv=num_crossval_folds,\n",
" method=\"predict_proba\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Construct K nearest neighbours graph"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The KNN graph reflects how close each example is when compared to other examples in our dataset (in the numerical space of preprocessed feature values). This similarity information is used by Datalab to identify issues like outliers in our data. For tabular data, think carefully about the most appropriate way to define the similarity between two examples.\n",
"\n",
"Here we use the `NearestNeighbors` class in sklearn to easily compute this graph (with similarity defined by the Euclidean distance between feature values). The graph should be represented as a sparse matrix with nonzero entries indicating nearest neighbors of each example and their distance."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"KNN = NearestNeighbors(metric='euclidean')\n",
"KNN.fit(X_processed.values)\n",
"\n",
"knn_graph = KNN.kneighbors_graph(mode=\"distance\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the given labels, predicted probabilities, and KNN graph, cleanlab can quickly help us identify suspicious values in our grades table.\n",
"\n",
"We use cleanlab's `Datalab` class which has several ways of loading the data. In this case, we’ll simply wrap the dataset (features and noisy labels) in a dictionary that is used instantiate a `Datalab` object such that it can audit our dataset for various types of issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = {\"X\": X_processed.values, \"y\": labels}\n",
"\n",
"lab = Datalab(data, label_name=\"y\")\n",
"lab.find_issues(pred_probs=pred_probs, knn_graph=knn_graph)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Label issues\n",
"\n",
"The above report shows that cleanlab identified many label issues in the data. We can see which examples are estimated to be mislabeled (as well as a numeric quality score quantifying how likely their label is correct) via the `get_issues` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"issue_results = lab.get_issues(\"label\")\n",
"issue_results.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To review the most severe label issues, sort the DataFrame above by the `label_score` column (a lower score represents that the label is less likely to be correct). \n",
"\n",
"Let's review some of the most likely label errors:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sorted_issues = issue_results.sort_values(\"label_score\").index\n",
"\n",
"X_raw.iloc[sorted_issues].assign(\n",
" given_label=labels.iloc[sorted_issues], \n",
" predicted_label=issue_results[\"predicted_label\"].iloc[sorted_issues]\n",
").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The dataframe above shows the original label (`given_label`) for examples that cleanlab finds most likely to be mislabeled, as well as an alternative `predicted_label` for each example.\n",
"\n",
"These examples have been labeled incorrectly and should be carefully re-examined - a student with grades of 89, 95 and 73 surely does not deserve a D! "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Outlier issues\n",
"\n",
"According to the report, our dataset contains some outliers. We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"outlier_results = lab.get_issues(\"outlier\")\n",
"sorted_outliers= outlier_results.sort_values(\"outlier_score\").index\n",
"\n",
"X_raw.iloc[sorted_outliers].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The student at index 3 has fractional exam scores, which is likely a error. We also see that the students at index 0 and 4 have numerical values in their notes section, which is also probably unintended. Lastly, we see that the student at index 8 has a html string in their notes section, definitely a mistake!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Near-duplicate issues\n",
"\n",
"According to the report, our dataset contains some sets of nearly duplicated examples.\n",
"We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the examples in our dataset that are most nearly duplicated."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"duplicate_results = lab.get_issues(\"near_duplicate\")\n",
"duplicate_results.sort_values(\"near_duplicate_score\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see some examples that cleanlab has flagged as being nearly duplicated. Let's view these examples to see how similar they are\n",
"\n",
"Using the one of the lowest-scoring examples, let's compare it against the identified near-duplicate sets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Identify the row with the lowest near_duplicate_score\n",
"lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].idxmin()\n",
"\n",
"# Extract the indices of the lowest scoring duplicate and its near duplicate sets\n",
"indices_to_display = [lowest_scoring_duplicate] + duplicate_results.loc[lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
"\n",
"# Display the relevant rows from the original dataset\n",
"X_raw.iloc[indices_to_display]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These examples are exact duplicates! Perhaps the same information was accidentally recorded multiple times in this data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, let's take a look at another example and the identified near-duplicate sets:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Identify the next row not in the previous near duplicate set\n",
"second_lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].drop(indices_to_display).idxmin()\n",
"\n",
"# Extract the indices of the second lowest scoring duplicate and its near duplicate sets\n",
"next_indices_to_display = [second_lowest_scoring_duplicate] + duplicate_results.loc[second_lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
"\n",
"# Display the relevant rows from the original dataset\n",
"X_raw.iloc[next_indices_to_display]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We identified another set of exact duplicates in our dataset! Including near/exact duplicates in a dataset may have unintended effects on models; be wary about splitting them across training/test sets. Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This tutorial highlighted a straightforward approach to detect potentially incorrect information in any tabular dataset. Just use Datalab with any ML model -- the better the model, the more accurate the data errors detected by Datalab will be!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"identified_label_issues = issue_results[issue_results[\"is_label_issue\"] == True]\n",
"label_issue_indices = [3, 723, 709, 886, 689] # check these examples were found in label issues\n",
"if not all(x in identified_label_issues.index for x in label_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n",
" \n",
"identified_outlier_issues = outlier_results[outlier_results[\"is_outlier_issue\"] == True]\n",
"outlier_issue_indices = [3, 7, 0, 4, 8] # check these examples were found in outlier issues\n",
"if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n",
" \n",
"identified_duplicate_issues = duplicate_results[duplicate_results[\"is_near_duplicate_issue\"] == True]\n",
"duplicate_issue_indices = [690, 246, 185, 582] # check these examples were found in duplicate issues\n",
"if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")\n",
" \n",
"# check that the near duplicates shown are actually flagged as near duplicate sets\n",
"if not duplicate_results.iloc[690][\"near_duplicate_sets\"] == 246:\n",
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
" \n",
"if not duplicate_results.iloc[185][\"near_duplicate_sets\"] == 582:\n",
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
"\n",
"# Function to check if all rows are identical\n",
"def are_rows_identical(df):\n",
" first_row = df.iloc[0]\n",
" return all(df.iloc[i].equals(first_row) for i in range(1, len(df)))\n",
"\n",
"# Test to ensure all displayed rows are identical\n",
"if not are_rows_identical(X_raw.iloc[indices_to_display]):\n",
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")\n",
"\n",
"# Repeat the test for the next set of indices\n",
"if not are_rows_identical(X_raw.iloc[next_indices_to_display]):\n",
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")"
]
}
],
"metadata": {
"interpreter": {
"hash": "cda20062bc42cfdcaa0f9720c0b28e880bba110e9dfce6c1689934eec9b595a1"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/text.ipynb
+591
View File
@@ -0,0 +1,591 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Detecting Issues in a Text Dataset with Datalab\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this 5-minute quickstart tutorial, we use Datalab to detect various issues in an intent classification dataset composed of (text) customer service requests at an online bank. We consider a subset of the [Banking77-OOS Dataset](https://arxiv.org/abs/2106.04564) containing 1,000 customer service requests which are classified into 10 categories based on their intent (you can run this same code on any text classification dataset). Cleanlab automatically identifies bad examples in our dataset, including mislabeled data, out-of-scope examples (outliers), or otherwise ambiguous examples. Consider filtering or correcting such bad examples\u00a0before you dive deep into modeling your data!\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Use a pretrained transformer model to extract the text embeddings from the customer service requests\n",
"\n",
"- Train a simple Logistic Regression model on the text embeddings to compute out-of-sample predicted probabilities\n",
"\n",
"- Run cleanlab's `Datalab` audit with these predictions and embeddings in order to identify problems like: label issues, outliers, and near duplicates in the dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have (out-of-sample) `pred_probs` from a model trained on an existing set of labels? Maybe you have some numeric `features` as well? Run the code below to find any potential label errors in your dataset.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, features=your_features)\n",
"\n",
"lab.report()\n",
"lab.get_issues()\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install sentence-transformers\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (hidden on docs.cleanlab.ai).\n",
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
"# Package versions we used:scikit-learn==1.2.0 sentence-transformers>=2.3.0\n",
"\n",
"dependencies = [\"cleanlab\", \"sentence_transformers\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import re \n",
"import string \n",
"import pandas as pd \n",
"from sklearn.metrics import accuracy_score, log_loss \n",
"from sklearn.model_selection import cross_val_predict \n",
"from sklearn.linear_model import LogisticRegression\n",
"from sentence_transformers import SentenceTransformer\n",
"\n",
"from cleanlab import Datalab"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This cell is hidden from docs.cleanlab.ai \n",
"\n",
"import random \n",
"import numpy as np \n",
"\n",
"pd.set_option(\"display.max_colwidth\", None) \n",
"\n",
"SEED = 123456 # for reproducibility\n",
"np.random.seed(SEED)\n",
"random.seed(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load and format the text dataset\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = pd.read_csv(\"https://s.cleanlab.ai/banking-intent-classification.csv\")\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"raw_texts, labels = data[\"text\"].values, data[\"label\"].values\n",
"num_classes = len(set(labels))\n",
"\n",
"print(f\"This dataset has {num_classes} classes.\")\n",
"print(f\"Classes: {set(labels)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's view the i-th example in the dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"i = 1 # change this to view other examples from the dataset\n",
"print(f\"Example Label: {labels[i]}\")\n",
"print(f\"Example Text: {raw_texts[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data is stored as two numpy arrays:\n",
"\n",
"1. `raw_texts` stores the customer service requests utterances in text format\n",
"2. `labels` stores the intent categories (labels) for each example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"You can easily replace the above with your own text dataset, and continue with the rest of the tutorial.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we convert the text strings into vectors better suited as inputs for our ML models. \n",
"\n",
"We will use numeric representations from a pretrained Transformer model as embeddings of our text. The [Sentence Transformers](https://huggingface.co/docs/hub/sentence-transformers) library offers simple methods to compute these embeddings for text data. Here, we load the pretrained `electra-small-discriminator` model, and then run our data through network to extract a vector embedding\u00a0of each example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"transformer = SentenceTransformer('google/electra-small-discriminator')\n",
"text_embeddings = transformer.encode(raw_texts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our subsequent ML model will directly operate on elements of `text_embeddings` in order to classify the customer service requests."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Define a classification model and compute out-of-sample predicted probabilities"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A typical way to leverage pretrained networks for a particular classification task is to add a linear output layer and fine-tune the network parameters on the new data. However this can be computationally intensive. Alternatively, we can freeze the pretrained weights of the network and only train the output layer without having to rely on GPU(s). Here we do this conveniently by fitting a scikit-learn linear model on top of the extracted embeddings.\n",
"\n",
"To identify label issues, cleanlab requires a probabilistic prediction from your model for each datapoint. However these predictions will be _overfit_ (and thus unreliable) for datapoints the model was previously trained on. cleanlab is intended to only be used with **out-of-sample** predicted class probabilities, i.e. on datapoints held-out from the model during the training.\n",
"\n",
"Here we obtain out-of-sample predicted class probabilities for every example in our dataset using a Logistic Regression model with cross-validation.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"model = LogisticRegression(max_iter=400)\n",
"\n",
"pred_probs = cross_val_predict(model, text_embeddings, labels, method=\"predict_proba\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use cleanlab to find issues in your dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given feature embeddings and the (out-of-sample) predicted class probabilities obtained from any model\u00a0you have, cleanlab can quickly help you identify low-quality examples in your dataset.\n",
"\n",
"Here, we use cleanlab's `Datalab` to find issues in our data. Datalab offers several ways of loading the data; we\u2019ll simply wrap the training features and noisy labels in a dictionary. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_dict = {\"texts\": raw_texts, \"labels\": labels}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All that is need to audit your data is to call `find_issues()`. We pass in the predicted probabilities and the feature embeddings obtained above, but you do not necessarily need to provide all of this information depending on which types of issues you are interested in. The more inputs you provide, the more types of issues `Datalab` can detect in your data. Using a better model to produce these inputs will ensure cleanlab more accurately estimates issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab = Datalab(data_dict, label_name=\"labels\")\n",
"lab.find_issues(pred_probs=pred_probs, features=text_embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the audit is complete, review the findings using the `report` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Label issues\n",
"\n",
"The report indicates that cleanlab identified many label issues in our dataset. We can see which examples are flagged as likely mislabeled and the label quality score for each example using the `get_issues` method, specifying `label` as an argument to focus on label issues in the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"label_issues = lab.get_issues(\"label\")\n",
"label_issues.head() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This method returns a dataframe containing a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. The dataframe also contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get the subset of examples flagged with label issues, and also sort by label quality score to find the indices of the 5 most likely mislabeled examples in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"identified_label_issues = label_issues[label_issues[\"is_label_issue\"] == True]\n",
"lowest_quality_labels = label_issues[\"label_score\"].argsort()[:5].to_numpy()\n",
"\n",
"print(\n",
" f\"cleanlab found {len(identified_label_issues)} potential label errors in the dataset.\\n\"\n",
" f\"Here are indices of the top 5 most likely errors: \\n {lowest_quality_labels}\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's review some of the most likely label errors. \n",
"\n",
"Here we display the top 5 examples identified as the most likely label errors in the dataset, together with their given (original) label and a suggested alternative label from cleanlab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data_with_suggested_labels = pd.DataFrame(\n",
" {\"text\": raw_texts, \"given_label\": labels, \"suggested_label\": label_issues[\"predicted_label\"]}\n",
")\n",
"data_with_suggested_labels.iloc[lowest_quality_labels]"
]
},
{
"cell_type": "markdown",
"metadata": {
"scrolled": true
},
"source": [
"These are very clear label errors that cleanlab has identified in this data! Note that the `given_label` does not correctly reflect the intent of these requests, whoever produced this dataset made many mistakes that are important to address before modeling the data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Outlier issues\n",
"\n",
"According to the report, our dataset contains some outliers.\n",
"We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"outlier_issues = lab.get_issues(\"outlier\")\n",
"outlier_issues.sort_values(\"outlier_score\").head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lowest_quality_outliers = outlier_issues[\"outlier_score\"].argsort()[:5]\n",
"\n",
"data.iloc[lowest_quality_outliers]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that cleanlab has identified entries in this dataset that do not appear to be proper customer requests. Outliers in this dataset appear to be out-of-scope customer requests and other nonsensical text which does not make sense for intent classification. Carefully consider whether such outliers may detrimentally affect your data modeling, and consider removing them from the dataset if so."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Near-duplicate issues\n",
"\n",
"According to the report, our dataset contains some sets of nearly duplicated examples.\n",
"We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the text examples in our dataset that are most nearly duplicated."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"duplicate_issues = lab.get_issues(\"near_duplicate\")\n",
"duplicate_issues.sort_values(\"near_duplicate_score\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see that example 160 and 148 are nearly duplicated, as are example 546 and 514.\n",
"\n",
"Let's view these examples to see how similar they are."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.iloc[[160, 148]]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data.iloc[[546, 514]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that these two sets of request are indeed very similar to one another! Including near duplicates in a dataset may have unintended effects on models, and be wary about splitting them across training/test sets. Learn more about handling near duplicates in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Non-IID issues (data drift)\n",
"According to the report, our dataset does not appear to be Independent and Identically Distributed (IID). The overall non-iid score for the dataset (displayed below) corresponds to the `p-value` of a statistical test for whether the ordering of samples in the dataset appears related to the similarity between their feature values. A low `p-value` strongly suggests that the dataset violates the IID assumption, which is a key assumption required for conclusions (models) produced from the dataset to generalize to a larger population."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"p_value = lab.get_info('non_iid')['p-value']\n",
"p_value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, our dataset was flagged as non-IID because the rows happened to be sorted by class label in the original data. This may be benign if we remember to shuffle rows before model training and data splitting. But if you don't know why your data was flagged as non-IID, then you should be worried about potential data drift or unexpected interactions between data points (their values may not be statistically independent). Think carefully about what future test data may look like (and whether your data is representative of the population you care about). You should not shuffle your data before the non-IID test runs (will invalidate its conclusions)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As demonstrated above, cleanlab can automatically shortlist the most likely issues in your dataset to help you better curate your dataset for subsequent modeling. With this shortlist, you can decide whether to fix these label issues or remove nonsensical or duplicated examples from your dataset to obtain a higher-quality dataset for training your next ML model. cleanlab's issue detection can be run with outputs from *any* type of model you initially trained.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"label_issue_indices = [981, 974, 982] # check these examples were found in label issues\n",
"if not all(x in identified_label_issues.index for x in label_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n",
" \n",
"identified_outlier_issues = outlier_issues[outlier_issues[\"is_outlier_issue\"] == True]\n",
"outlier_issue_indices = [994, 989, 999] # check these examples were found in duplicates\n",
"if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n",
"\n",
"identified_duplicate_issues = duplicate_issues[duplicate_issues[\"is_near_duplicate_issue\"] == True]\n",
"duplicate_issue_indices = [160, 148, 546, 514] # check these examples were found in duplicates\n",
"if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n",
" raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Text Classification with Datalab",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/source/tutorials/datalab/workflows.ipynb
+1637
View File
File diff suppressed because it is too large Load Diff