wehub-resource-sync
1320 lines
48 KiB
Plaintext
1320 lines
48 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Detecting Issues in an Image Dataset with Datalab\n",
|
||
"\n",
|
||
"This quickstart tutorial demonstrates how to find issues in image classification data. Here we use the Fashion-MNIST dataset (60,000 images of fashion products from 10 categories), but you can replace this with your own image classification dataset and still follow the same tutorial.\n",
|
||
"\n",
|
||
"**Overview of what we'll do in this tutorial:**\n",
|
||
"\n",
|
||
"- Build a simple [PyTorch](https://pytorch.org/) neural net.\n",
|
||
"\n",
|
||
"- Use cross-validation to compute out-of-sample predicted probabilities (`pred_probs`) and feature embeddings (`features`) for each image in the dataset.\n",
|
||
"\n",
|
||
"- Utilize these `pred_probs` and `features` to identify potential issues within the dataset using the `Datalab` class from cleanlab. The issues found by cleanlab include mislabeled examples, near duplicates, outliers, and image-specific problems such as excessively dark or low information images."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<div class=\"alert alert-info\">\n",
|
||
"Quickstart\n",
|
||
"<br/>\n",
|
||
" \n",
|
||
"Already have a ML model? Run cross-validation to get out-of-sample `pred_probs` and provide `features` (embeddings of the data). Then use the code below to find any potential issues in your dataset (you can also run this code with one of `pred_probs` or `features` instead of both, but less issue types will be considered).\n",
|
||
"\n",
|
||
"\n",
|
||
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
|
||
" \n",
|
||
"```python\n",
|
||
"from cleanlab import Datalab\n",
|
||
"\n",
|
||
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\") # include `image_key` to detect low-quality images\n",
|
||
"lab.find_issues(pred_probs=pred_probs, features=features)\n",
|
||
"\n",
|
||
"lab.report()\n",
|
||
"```\n",
|
||
" \n",
|
||
"</div>\n",
|
||
"</div>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. Install and import required dependencies"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"You can use `pip` to install all packages required for this tutorial as follows:\n",
|
||
"\n",
|
||
"```ipython3\n",
|
||
"!pip install matplotlib torch torchvision datasets>=2.19.0\n",
|
||
"!pip install \"cleanlab[image]\"\n",
|
||
"# We install cleanlab with extra dependencies for image data\n",
|
||
"# Make sure to install the version corresponding to this tutorial\n",
|
||
"# E.g. if viewing master branch documentation:\n",
|
||
"# !pip install \"cleanlab[image] @ git+https://github.com/cleanlab/cleanlab.git\"\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Package installation (this cell is hidden from docs.cleanlab.ai).\n",
|
||
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
|
||
"\n",
|
||
"dependencies = [\"cleanlab\", \"matplotlib\", \"torch\", \"torchvision\", \"datasets\", \"cleanvision\"]\n",
|
||
"\n",
|
||
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
|
||
" %pip install \"cleanlab[image]\" # for colab\n",
|
||
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
|
||
" %pip install $cmd\n",
|
||
"else:\n",
|
||
" missing_dependencies = []\n",
|
||
" for dependency in dependencies:\n",
|
||
" try:\n",
|
||
" __import__(dependency)\n",
|
||
" except ImportError:\n",
|
||
" missing_dependencies.append(dependency)\n",
|
||
"\n",
|
||
" if len(missing_dependencies) > 0:\n",
|
||
" print(\"Missing required dependencies:\")\n",
|
||
" print(*missing_dependencies, sep=\", \")\n",
|
||
" print(\"\\nPlease install them before running the rest of this notebook.\")\n",
|
||
"\n",
|
||
"# Suppress benign warnings: \n",
|
||
"import warnings \n",
|
||
"warnings.filterwarnings(\"ignore\", \"Lazy modules are a new feature.*\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from torch.utils.data import DataLoader, TensorDataset, Subset\n",
|
||
"import torch\n",
|
||
"import torch.nn as nn\n",
|
||
"import torch.optim as optim\n",
|
||
"\n",
|
||
"from sklearn.model_selection import StratifiedKFold\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"\n",
|
||
"from tqdm.autonotebook import tqdm\n",
|
||
"import math\n",
|
||
"import time\n",
|
||
"import multiprocessing\n",
|
||
"\n",
|
||
"from cleanlab import Datalab\n",
|
||
"from datasets import load_dataset"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Fetch and normalize the Fashion-MNIST dataset\n",
|
||
"\n",
|
||
"Load train split of the fashion_mnist dataset and view the number of rows and columns in the dataset"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"dataset = load_dataset(\"fashion_mnist\", split=\"train\")\n",
|
||
"dataset"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Get number of classes in the dataset"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_classes = len(dataset.features[\"label\"].names)\n",
|
||
"num_classes"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Convert PIL image to torch tensors\n",
|
||
"transformed_dataset = dataset.with_format(\"torch\")\n",
|
||
"\n",
|
||
"\n",
|
||
"# Apply transformations\n",
|
||
"def normalize(example):\n",
|
||
" example[\"image\"] = (example[\"image\"] / 255.0) # each pixel value was originally between 0 and 255 \n",
|
||
" return example\n",
|
||
"\n",
|
||
"\n",
|
||
"transformed_dataset = transformed_dataset.map(normalize, num_proc=multiprocessing.cpu_count())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Convert the transformed dataset to a torch dataset. Torch datasets are more efficient with dataloading in practice."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"torch_dataset = TensorDataset(transformed_dataset[\"image\"], transformed_dataset[\"label\"])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<div class=\"alert alert-info\">\n",
|
||
"Bringing Your Own Data (BYOD)?\n",
|
||
"\n",
|
||
"Load any huggingface dataset or your local image folder dataset, apply relevant transformations, and continue with the rest of the tutorial.\n",
|
||
"\n",
|
||
"</div>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Define a classification model\n",
|
||
"Here, we define a simple neural network with PyTorch. Note this is just a toy model to ensure quick runtimes for the tutorial, you can replace it with any other (larger) PyTorch network."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class Net(nn.Module):\n",
|
||
" def __init__(self):\n",
|
||
" super().__init__()\n",
|
||
" self.cnn = nn.Sequential(\n",
|
||
" nn.Conv2d(1, 6, 5),\n",
|
||
" nn.ReLU(),\n",
|
||
" nn.BatchNorm2d(6),\n",
|
||
" nn.MaxPool2d(2, 2),\n",
|
||
" nn.Conv2d(6, 16, 5, bias=False),\n",
|
||
" nn.ReLU(),\n",
|
||
" nn.BatchNorm2d(16),\n",
|
||
" nn.MaxPool2d(2, 2),\n",
|
||
" )\n",
|
||
" self.linear = nn.Sequential(nn.LazyLinear(128), nn.ReLU())\n",
|
||
" self.output = nn.Sequential(nn.Linear(128, num_classes))\n",
|
||
"\n",
|
||
" def forward(self, x):\n",
|
||
" x = self.embeddings(x)\n",
|
||
" x = self.output(x)\n",
|
||
" return x\n",
|
||
"\n",
|
||
" def embeddings(self, x):\n",
|
||
" x = self.cnn(x)\n",
|
||
" x = torch.flatten(x, 1) # flatten all dimensions except batch\n",
|
||
" x = self.linear(x)\n",
|
||
" return x"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# This (optional) cell is hidden from docs.cleanlab.ai\n",
|
||
"\n",
|
||
"SEED = 123 # for reproducibility\n",
|
||
"np.random.seed(SEED)\n",
|
||
"torch.manual_seed(SEED)\n",
|
||
"torch.backends.cudnn.deterministic = True\n",
|
||
"torch.backends.cudnn.benchmark = True\n",
|
||
"torch.cuda.manual_seed_all(SEED)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<details><summary>Helper methods for cross validation **(click to expand)**</summary>\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"# Set device\n",
|
||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method to calculate validation accuracy in each epoch\n",
|
||
"def get_test_accuracy(net, testloader):\n",
|
||
" net.eval()\n",
|
||
" accuracy = 0.0\n",
|
||
" total = 0.0\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in testloader:\n",
|
||
" images, labels = data[\"image\"].to(device), data[\"label\"].to(device)\n",
|
||
"\n",
|
||
" # run the model on the test set to predict labels\n",
|
||
" outputs = net(images)\n",
|
||
"\n",
|
||
" # the label with the highest energy will be our prediction\n",
|
||
" _, predicted = torch.max(outputs.data, 1)\n",
|
||
" total += labels.size(0)\n",
|
||
" accuracy += (predicted == labels).sum().item()\n",
|
||
"\n",
|
||
" # compute the accuracy over all test images\n",
|
||
" accuracy = 100 * accuracy / total\n",
|
||
" return accuracy\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for training the model\n",
|
||
"def train(trainloader, testloader, n_epochs, patience):\n",
|
||
" model = Net()\n",
|
||
"\n",
|
||
" criterion = nn.CrossEntropyLoss()\n",
|
||
" optimizer = optim.AdamW(model.parameters())\n",
|
||
"\n",
|
||
" model = model.to(device)\n",
|
||
"\n",
|
||
" best_test_accuracy = 0.0\n",
|
||
"\n",
|
||
" for epoch in range(n_epochs): # loop over the dataset multiple times\n",
|
||
" start_epoch = time.time()\n",
|
||
" running_loss = 0.0\n",
|
||
"\n",
|
||
" for _, data in enumerate(trainloader):\n",
|
||
" # get the inputs; data is a dict of {\"image\": images, \"label\": labels}\n",
|
||
"\n",
|
||
" inputs, labels = data[\"image\"].to(device), data[\"label\"].to(device)\n",
|
||
"\n",
|
||
" # zero the parameter gradients\n",
|
||
" optimizer.zero_grad()\n",
|
||
"\n",
|
||
" # forward + backward + optimize\n",
|
||
" outputs = model(inputs)\n",
|
||
" loss = criterion(outputs, labels)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
"\n",
|
||
" running_loss += loss.detach().cpu().item()\n",
|
||
"\n",
|
||
" # Get accuracy on the test set\n",
|
||
" accuracy = get_test_accuracy(model, testloader)\n",
|
||
"\n",
|
||
" if accuracy > best_test_accuracy:\n",
|
||
" best_epoch = epoch\n",
|
||
"\n",
|
||
" # Condition for early stopping\n",
|
||
" if epoch - best_epoch > patience:\n",
|
||
" print(f\"Early stopping at epoch {epoch + 1}\")\n",
|
||
" break\n",
|
||
"\n",
|
||
" end_epoch = time.time()\n",
|
||
"\n",
|
||
" print(\n",
|
||
" f\"epoch: {epoch + 1} loss: {running_loss / len(trainloader):.3f} test acc: {accuracy:.3f} time_taken: {end_epoch - start_epoch:.3f}\"\n",
|
||
" )\n",
|
||
" return model\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for computing out-of-sample embeddings\n",
|
||
"def compute_embeddings(model, testloader):\n",
|
||
" embeddings_list = []\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in tqdm(testloader):\n",
|
||
" images, labels = data[\"image\"].to(device), data[\"label\"].to(device)\n",
|
||
"\n",
|
||
" embeddings = model.embeddings(images)\n",
|
||
" embeddings_list.append(embeddings.cpu())\n",
|
||
"\n",
|
||
" return torch.vstack(embeddings_list)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for computing out-of-sample predicted probabilities\n",
|
||
"def compute_pred_probs(model, testloader):\n",
|
||
" pred_probs_list = []\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in tqdm(testloader):\n",
|
||
" images, labels = data[\"image\"].to(device), data[\"label\"].to(device)\n",
|
||
"\n",
|
||
" outputs = model(images)\n",
|
||
" pred_probs_list.append(outputs.cpu())\n",
|
||
"\n",
|
||
" return torch.vstack(pred_probs_list)\n",
|
||
"```\n",
|
||
"</details>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Set device\n",
|
||
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method to calculate validation accuracy in each epoch\n",
|
||
"def get_test_accuracy(net, testloader):\n",
|
||
" net.eval()\n",
|
||
" accuracy = 0.0\n",
|
||
" total = 0.0\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in testloader:\n",
|
||
" images, labels = data[0].to(device), data[1].to(device)\n",
|
||
"\n",
|
||
" # run the model on the test set to predict labels\n",
|
||
" outputs = net(images)\n",
|
||
"\n",
|
||
" # the label with the highest energy will be our prediction\n",
|
||
" _, predicted = torch.max(outputs.data, 1)\n",
|
||
" total += labels.size(0)\n",
|
||
" accuracy += (predicted == labels).sum().item()\n",
|
||
"\n",
|
||
" # compute the accuracy over all test images\n",
|
||
" accuracy = 100 * accuracy / total\n",
|
||
" return accuracy\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for training the model\n",
|
||
"def train(trainloader, testloader, n_epochs, patience):\n",
|
||
" model = Net()\n",
|
||
"\n",
|
||
" criterion = nn.CrossEntropyLoss()\n",
|
||
" optimizer = optim.AdamW(model.parameters())\n",
|
||
"\n",
|
||
" model = model.to(device)\n",
|
||
"\n",
|
||
" best_test_accuracy = 0.0\n",
|
||
"\n",
|
||
" for epoch in range(n_epochs): # loop over the dataset multiple times\n",
|
||
" start_epoch = time.time()\n",
|
||
" running_loss = 0.0\n",
|
||
"\n",
|
||
" for _, data in enumerate(trainloader):\n",
|
||
" # get the inputs; data is a dict of {\"image\": images, \"label\": labels}\n",
|
||
"\n",
|
||
" inputs, labels = data[0].to(device), data[1].to(device)\n",
|
||
"\n",
|
||
" # zero the parameter gradients\n",
|
||
" optimizer.zero_grad()\n",
|
||
"\n",
|
||
" # forward + backward + optimize\n",
|
||
" outputs = model(inputs)\n",
|
||
" loss = criterion(outputs, labels)\n",
|
||
" loss.backward()\n",
|
||
" optimizer.step()\n",
|
||
"\n",
|
||
" running_loss += loss.detach().cpu().item()\n",
|
||
"\n",
|
||
" # Get accuracy on the test set\n",
|
||
" accuracy = get_test_accuracy(model, testloader)\n",
|
||
"\n",
|
||
" if accuracy > best_test_accuracy:\n",
|
||
" best_epoch = epoch\n",
|
||
"\n",
|
||
" # Condition for early stopping\n",
|
||
" if epoch - best_epoch > patience:\n",
|
||
" print(f\"Early stopping at epoch {epoch + 1}\")\n",
|
||
" break\n",
|
||
"\n",
|
||
" end_epoch = time.time()\n",
|
||
"\n",
|
||
" print(\n",
|
||
" f\"epoch: {epoch + 1} loss: {running_loss / len(trainloader):.3f} test acc: {accuracy:.3f} time_taken: {end_epoch - start_epoch:.3f}\"\n",
|
||
" )\n",
|
||
" return model\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for computing out-of-sample embeddings\n",
|
||
"def compute_embeddings(model, testloader):\n",
|
||
" embeddings_list = []\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in tqdm(testloader):\n",
|
||
" images, labels = data[0].to(device), data[1].to(device)\n",
|
||
"\n",
|
||
" embeddings = model.embeddings(images)\n",
|
||
" embeddings_list.append(embeddings.cpu())\n",
|
||
"\n",
|
||
" return torch.vstack(embeddings_list)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Method for computing out-of-sample predicted probabilities\n",
|
||
"def compute_pred_probs(model, testloader):\n",
|
||
" pred_probs_list = []\n",
|
||
"\n",
|
||
" with torch.no_grad():\n",
|
||
" for data in tqdm(testloader):\n",
|
||
" images, labels = data[0].to(device), data[1].to(device)\n",
|
||
"\n",
|
||
" outputs = model(images)\n",
|
||
" pred_probs_list.append(outputs.cpu())\n",
|
||
"\n",
|
||
" return torch.vstack(pred_probs_list)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. Prepare the dataset for K-fold cross-validation \n",
|
||
"\n",
|
||
"To find label issues based on `pred_probs`, we recommend out-of-sample predictions, which can be produced [via K-fold cross-validation](https://docs.cleanlab.ai/stable/tutorials/pred_probs_cross_val.html). To ensure this tutorial runs quickly, we set K and other important neural network training hyperparameters to small values here. Use larger values to get good results in practice!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"K = 3 # Number of cross-validation folds. Set to small value here to ensure quick runtimes, we recommend 5 or 10 in practice for more accurate estimates.\n",
|
||
"n_epochs = 2 # Number of epochs to train model for. Set to a small value here for quick runtime, you should use a larger value in practice.\n",
|
||
"patience = 2 # Parameter for early stopping. If the validation accuracy does not improve for this many epochs, training will stop.\n",
|
||
"train_batch_size = 64 # Batch size for training\n",
|
||
"test_batch_size = 512 # Batch size for testing\n",
|
||
"num_workers = multiprocessing.cpu_count() # Number of workers for data loaders\n",
|
||
"\n",
|
||
"# Create k splits of the dataset\n",
|
||
"kfold = StratifiedKFold(n_splits=K, shuffle=True, random_state=0)\n",
|
||
"splits = kfold.split(transformed_dataset, transformed_dataset[\"label\"])\n",
|
||
"\n",
|
||
"train_id_list, test_id_list = [], []\n",
|
||
"\n",
|
||
"for fold, (train_ids, test_ids) in enumerate(splits):\n",
|
||
" train_id_list.append(train_ids)\n",
|
||
" test_id_list.append(test_ids)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5. Compute out-of-sample predicted probabilities and feature embeddings\n",
|
||
"\n",
|
||
"We use cross-validation to compute out-of-sample predicted probabilities separately for each dataset fold. However, we use only one model to generate embeddings for all the images across the full dataset. This ensures all feature embeddings lie in the same representation space for more accurate detection of data issues. Here we embed all the data using our model trained in the first cross-validation fold, but you could also train a separate embedding model on the full dataset."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"pred_probs_list, embeddings_list = [], []\n",
|
||
"embeddings_model = None\n",
|
||
"\n",
|
||
"for i in range(K):\n",
|
||
" print(f\"\\nTraining on fold: {i+1} ...\")\n",
|
||
"\n",
|
||
" # Create train and test sets and corresponding dataloaders\n",
|
||
" trainset = Subset(torch_dataset, train_id_list[i])\n",
|
||
" testset = Subset(torch_dataset, test_id_list[i])\n",
|
||
"\n",
|
||
" trainloader = DataLoader(\n",
|
||
" trainset,\n",
|
||
" batch_size=train_batch_size,\n",
|
||
" shuffle=False,\n",
|
||
" num_workers=num_workers,\n",
|
||
" pin_memory=True,\n",
|
||
" )\n",
|
||
" testloader = DataLoader(\n",
|
||
" testset, batch_size=test_batch_size, shuffle=False, num_workers=num_workers, pin_memory=True\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Train model\n",
|
||
" model = train(trainloader, testloader, n_epochs, patience)\n",
|
||
" if embeddings_model is None:\n",
|
||
" embeddings_model = model\n",
|
||
"\n",
|
||
" # Compute out-of-sample embeddings\n",
|
||
" print(\"Computing feature embeddings ...\")\n",
|
||
" fold_embeddings = compute_embeddings(embeddings_model, testloader)\n",
|
||
" embeddings_list.append(fold_embeddings)\n",
|
||
"\n",
|
||
" print(\"Computing predicted probabilities ...\")\n",
|
||
" # Compute out-of-sample predicted probabilities\n",
|
||
" fold_pred_probs = compute_pred_probs(model, testloader)\n",
|
||
" pred_probs_list.append(fold_pred_probs)\n",
|
||
"\n",
|
||
"print(\"Finished Training\")\n",
|
||
"\n",
|
||
"\n",
|
||
"# Combine embeddings and predicted probabilities from each fold\n",
|
||
"features = torch.vstack(embeddings_list).numpy()\n",
|
||
"\n",
|
||
"logits = torch.vstack(pred_probs_list)\n",
|
||
"pred_probs = nn.Softmax(dim=1)(logits).numpy()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Reorder rows of the dataset based on row order in `features` and `pred_probs`. **Carefully ensure your ordering of the dataset matches these objects!**\n",
|
||
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"indices = np.hstack(test_id_list)\n",
|
||
"dataset = dataset.select(indices)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 7. Use cleanlab to find issues\n",
|
||
"\n",
|
||
"Based on the out-of-sample predicted probabilities and feature embeddings from our ML model, cleanlab can automatically detect issues in our labeled dataset. \n",
|
||
"\n",
|
||
"Here we use cleanlab's `Datalab` class to find issues in our data. `Datalab` supports several data formats, in this tutorial we have a Hugging Face Dataset. `Datalab` takes in two optional dataset arguments: `label_name`, which corresponds to the column containing labels (if your dataset is labeled), and `image_key`, corresponding to the name of a key in your vision dataset to access the raw images. When you provide these optional arguments, `Datalab` will audit your dataset for more types of issues than it would by default."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"lab = Datalab(data=dataset, label_name=\"label\", image_key=\"image\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The `find_issues` method can automatically infer the types of issues to be checked for based on the provided arguments. Here, we provide `features` and `pred_probs` as arguments. If you want to check for a specific issue type, you can do so using the `issue_types` argument. Check the [documentation](https://docs.cleanlab.ai/stable/cleanlab/datalab/datalab.html#cleanlab.datalab.datalab.Datalab.find_issues) for a more comprehensive guide on `find_issues` method."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"lab.find_issues(features=features, pred_probs=pred_probs)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### View report\n",
|
||
"\n",
|
||
"After the audit is complete, we can view a high-level report of detected data issues."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"lab.report()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Label issues\n",
|
||
"\n",
|
||
"Let's first inspect mislabeled examples in the dataset. Such errors occur when the given label for an image is incorrect, usually due to mistakes made by data annotators. Cleanlab automatically detects mislabeled data that you can correct to improve your dataset.\n",
|
||
"\n",
|
||
"For each type of issue that Cleanlab detects, you can use the `get_issues` method to see which examples in the dataset exhibit this type of issue (and how severely). Let's see which images in our dataset are estimated to be mislabeled:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"label_issues = lab.get_issues(\"label\")\n",
|
||
"label_issues.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The above dataframe contains a `label_score` for each example in the dataset. These numeric quality scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. It contains a boolean column `is_label_issue` specifying whether or not each example appears to have a label issue (indicating it is likely mislabeled).\n",
|
||
"\n",
|
||
"Filter the `label_issues` DataFrame to see which examples have label issues, and sort by `label_score`(in ascending order) to see the most likely mislabeled examples first."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"label_issues_df = label_issues.query(\"is_label_issue\").sort_values(\"label_score\")\n",
|
||
"label_issues_df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<details><summary>We define a helper method plot_label_issue_examples to visualize results. **(click to expand)**</summary>\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"def plot_label_issue_examples(label_issues_df, num_examples=15):\n",
|
||
" ncols = 5\n",
|
||
" nrows = int(math.ceil(num_examples / ncols))\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" label_issue_indices = label_issues_df.index.values\n",
|
||
"\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i >= num_examples:\n",
|
||
" ax.axis(\"off\")\n",
|
||
" continue\n",
|
||
" idx = int(label_issue_indices[i])\n",
|
||
" row = label_issues.loc[idx]\n",
|
||
" ax.set_title(\n",
|
||
" f\"id: {idx}\\n GL: {row.given_label}\\n SL: {row.predicted_label}\",\n",
|
||
" fontdict={\"fontsize\": 8},\n",
|
||
" )\n",
|
||
" ax.imshow(dataset[idx][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()\n",
|
||
"```\n",
|
||
"</details>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def plot_label_issue_examples(label_issues_df, num_examples=15):\n",
|
||
" ncols = 5\n",
|
||
" nrows = int(math.ceil(num_examples / ncols))\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" label_issue_indices = label_issues_df.index.values\n",
|
||
"\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i >= num_examples:\n",
|
||
" ax.axis(\"off\")\n",
|
||
" continue\n",
|
||
" idx = int(label_issue_indices[i])\n",
|
||
" row = label_issues.loc[idx]\n",
|
||
" ax.set_title(\n",
|
||
" f\"id: {idx}\\n GL: {row.given_label}\\n SL: {row.predicted_label}\",\n",
|
||
" fontdict={\"fontsize\": 8},\n",
|
||
" )\n",
|
||
" ax.imshow(dataset[idx][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### View most likely examples with label errors\n",
|
||
"\n",
|
||
"Here we define\n",
|
||
"`GL` : given label in the original dataset\n",
|
||
"`SL` : suggested alternative label by cleanlab"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"plot_label_issue_examples(label_issues_df, num_examples=15)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Outlier issues\n",
|
||
"\n",
|
||
"Datalab also detects atypical images lurking in our dataset. Such outliers are significantly different from the majority of the dataset and may have an outsized impact on how models fit to this data.\n",
|
||
"\n",
|
||
"Similarly to the previous section, we filter the `outlier_issues` DataFrame to find examples that are considered to be outliers. We then sort the filtered results by their outlier quality score, where examples with the lowest scores are those that appear least typical relative to the rest of the dataset."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"outlier_issues_df = lab.get_issues(\"outlier\")\n",
|
||
"outlier_issues_df = outlier_issues_df.query(\"is_outlier_issue\").sort_values(\"outlier_score\")\n",
|
||
"outlier_issues_df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### View most severe outliers\n",
|
||
"\n",
|
||
"In this visualization, the first image in every row shows the potential outlier, while the remaining images in the same row depict typical instances from the corresponding class."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<details><summary>We define a helper method plot_outlier_issues_examples to visualize results. **(click to expand)**</summary>\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"def plot_outlier_issues_examples(outlier_issues_df, num_examples):\n",
|
||
" ncols = 4\n",
|
||
" nrows = num_examples\n",
|
||
" N_comparison_images = ncols - 1\n",
|
||
"\n",
|
||
" def sample_from_class(label, number_of_samples, index):\n",
|
||
" index = int(index)\n",
|
||
"\n",
|
||
" non_outlier_indices = (\n",
|
||
" label_issues.join(outlier_issues_df)\n",
|
||
" .query(\"given_label == @label and is_outlier_issue.isnull()\")\n",
|
||
" .index\n",
|
||
" )\n",
|
||
" non_outlier_indices_excluding_current = non_outlier_indices[non_outlier_indices != index]\n",
|
||
"\n",
|
||
" sampled_indices = np.random.choice(\n",
|
||
" non_outlier_indices_excluding_current, number_of_samples, replace=False\n",
|
||
" )\n",
|
||
"\n",
|
||
" label_scores_of_sampled = label_issues.loc[sampled_indices][\"label_score\"]\n",
|
||
"\n",
|
||
" top_score_indices = np.argsort(label_scores_of_sampled.values)[::-1][:N_comparison_images]\n",
|
||
"\n",
|
||
" top_label_indices = sampled_indices[top_score_indices]\n",
|
||
"\n",
|
||
" sampled_images = [dataset[int(i)][\"image\"] for i in top_label_indices]\n",
|
||
"\n",
|
||
" return sampled_images\n",
|
||
"\n",
|
||
" def get_image_given_label_and_samples(idx):\n",
|
||
" image_from_dataset = dataset[idx][\"image\"]\n",
|
||
" corresponding_label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" comparison_images = sample_from_class(corresponding_label, 30, idx)[:N_comparison_images]\n",
|
||
"\n",
|
||
" return image_from_dataset, corresponding_label, comparison_images\n",
|
||
"\n",
|
||
" count = 0\n",
|
||
" images_to_plot = []\n",
|
||
" labels = []\n",
|
||
" idlist = []\n",
|
||
" for idx, row in outlier_issues_df.iterrows():\n",
|
||
" idx = row.name\n",
|
||
" image, label, comparison_images = get_image_given_label_and_samples(idx)\n",
|
||
" labels.append(label)\n",
|
||
" idlist.append(idx)\n",
|
||
" images_to_plot.append(image)\n",
|
||
" images_to_plot.extend(comparison_images)\n",
|
||
" count += 1\n",
|
||
" if count >= nrows:\n",
|
||
" break\n",
|
||
"\n",
|
||
" ncols = 1 + N_comparison_images\n",
|
||
" fig, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i % ncols == 0:\n",
|
||
" ax.set_title(f\"id: {idlist[i // ncols]}\\n GL: {labels[i // ncols]}\", fontdict={\"fontsize\": 8})\n",
|
||
" ax.imshow(images_to_plot[i], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()\n",
|
||
"```\n",
|
||
"</details>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def plot_outlier_issues_examples(outlier_issues_df, num_examples):\n",
|
||
" ncols = 4\n",
|
||
" nrows = num_examples\n",
|
||
" N_comparison_images = ncols - 1\n",
|
||
"\n",
|
||
" def sample_from_class(label, number_of_samples, index):\n",
|
||
" index = int(index)\n",
|
||
"\n",
|
||
" non_outlier_indices = (\n",
|
||
" label_issues.join(outlier_issues_df)\n",
|
||
" .query(\"given_label == @label and is_outlier_issue.isnull()\")\n",
|
||
" .index\n",
|
||
" )\n",
|
||
" non_outlier_indices_excluding_current = non_outlier_indices[non_outlier_indices != index]\n",
|
||
"\n",
|
||
" sampled_indices = np.random.choice(\n",
|
||
" non_outlier_indices_excluding_current, number_of_samples, replace=False\n",
|
||
" )\n",
|
||
"\n",
|
||
" label_scores_of_sampled = label_issues.loc[sampled_indices][\"label_score\"]\n",
|
||
"\n",
|
||
" top_score_indices = np.argsort(label_scores_of_sampled.values)[::-1][:N_comparison_images]\n",
|
||
"\n",
|
||
" top_label_indices = sampled_indices[top_score_indices]\n",
|
||
"\n",
|
||
" sampled_images = [dataset[int(i)][\"image\"] for i in top_label_indices]\n",
|
||
"\n",
|
||
" return sampled_images\n",
|
||
"\n",
|
||
" def get_image_given_label_and_samples(idx):\n",
|
||
" image_from_dataset = dataset[idx][\"image\"]\n",
|
||
" corresponding_label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" comparison_images = sample_from_class(corresponding_label, 30, idx)[:N_comparison_images]\n",
|
||
"\n",
|
||
" return image_from_dataset, corresponding_label, comparison_images\n",
|
||
"\n",
|
||
" count = 0\n",
|
||
" images_to_plot = []\n",
|
||
" labels = []\n",
|
||
" idlist = []\n",
|
||
" for idx, row in outlier_issues_df.iterrows():\n",
|
||
" idx = row.name\n",
|
||
" image, label, comparison_images = get_image_given_label_and_samples(idx)\n",
|
||
" labels.append(label)\n",
|
||
" idlist.append(idx)\n",
|
||
" images_to_plot.append(image)\n",
|
||
" images_to_plot.extend(comparison_images)\n",
|
||
" count += 1\n",
|
||
" if count >= nrows:\n",
|
||
" break\n",
|
||
"\n",
|
||
" ncols = 1 + N_comparison_images\n",
|
||
" fig, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i % ncols == 0:\n",
|
||
" ax.set_title(\n",
|
||
" f\"id: {idlist[i // ncols]}\\n GL: {labels[i // ncols]}\", fontdict={\"fontsize\": 8}\n",
|
||
" )\n",
|
||
" ax.imshow(images_to_plot[i], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"plot_outlier_issues_examples(outlier_issues_df, num_examples=5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Near duplicate issues\n",
|
||
"\n",
|
||
"Datalab also detects which examples are (near) duplicates of other examples in the dataset. Near duplicate images in a dataset can lead to model overfitting and have an outsized impact on evaluation metrics (especially when you have duplicates between training and test splits).\n",
|
||
"\n",
|
||
"The `near_duplicate_issues` DataFrame tells us which examples are considered to be nearly duplicated in the dataset (including exact duplicates as well). We can sort all images via the `near_duplicate_score` which quantifies how severe this issue is for each image (lower values indicate more severe instances of a type of issue, in this case, how similar the image is to its closest neighbor in the dataset).\n",
|
||
"\n",
|
||
"This allows us to visualize examples in the dataset that are considered nearly duplicated, along with their highly similar counterparts."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"near_duplicate_issues_df = lab.get_issues(\"near_duplicate\")\n",
|
||
"near_duplicate_issues_df = near_duplicate_issues_df.query(\"is_near_duplicate_issue\").sort_values(\n",
|
||
" \"near_duplicate_score\"\n",
|
||
")\n",
|
||
"near_duplicate_issues_df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### View sets of near duplicate images"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<details><summary>We define a helper method plot_near_duplicate_issue_examples to visualize results. **(click to expand)**</summary>\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"def plot_near_duplicate_issue_examples(near_duplicate_issues_df, num_examples=3):\n",
|
||
" nrows = num_examples\n",
|
||
" seen_id_pairs = set()\n",
|
||
"\n",
|
||
" def get_image_and_given_label_and_predicted_label(idx):\n",
|
||
" image = dataset[idx][\"image\"]\n",
|
||
" label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" predicted_label = label_issues.loc[idx][\"predicted_label\"]\n",
|
||
" return image, label, predicted_label\n",
|
||
"\n",
|
||
" count = 0\n",
|
||
" for idx, row in near_duplicate_issues_df.iterrows():\n",
|
||
" image, label, predicted_label = get_image_and_given_label_and_predicted_label(idx)\n",
|
||
" duplicate_images = row.near_duplicate_sets\n",
|
||
" nd_set = set([int(i) for i in duplicate_images])\n",
|
||
" nd_set.add(int(idx))\n",
|
||
"\n",
|
||
" if nd_set & seen_id_pairs:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(1, len(nd_set), figsize=(len(nd_set), 3))\n",
|
||
" for i, ax in zip(list(nd_set), axes):\n",
|
||
" label = label_issues.loc[i][\"given_label\"]\n",
|
||
" ax.set_title(f\"id: {i}\\n GL: {label}\", fontdict={\"fontsize\": 8})\n",
|
||
" ax.imshow(dataset[i][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" seen_id_pairs.update(nd_set)\n",
|
||
" count += 1\n",
|
||
" if count >= nrows:\n",
|
||
" break\n",
|
||
"\n",
|
||
" plt.show()\n",
|
||
"```\n",
|
||
"</details>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def plot_near_duplicate_issue_examples(near_duplicate_issues_df, num_examples=3):\n",
|
||
" nrows = num_examples\n",
|
||
" seen_id_pairs = set()\n",
|
||
"\n",
|
||
" def get_image_and_given_label_and_predicted_label(idx):\n",
|
||
" image = dataset[idx][\"image\"]\n",
|
||
" label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" predicted_label = label_issues.loc[idx][\"predicted_label\"]\n",
|
||
" return image, label, predicted_label\n",
|
||
"\n",
|
||
" count = 0\n",
|
||
" for idx, row in near_duplicate_issues_df.iterrows():\n",
|
||
" image, label, predicted_label = get_image_and_given_label_and_predicted_label(idx)\n",
|
||
" duplicate_images = row.near_duplicate_sets\n",
|
||
" nd_set = set([int(i) for i in duplicate_images])\n",
|
||
" nd_set.add(int(idx))\n",
|
||
"\n",
|
||
" if nd_set & seen_id_pairs:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(1, len(nd_set), figsize=(len(nd_set), 3))\n",
|
||
" for i, ax in zip(list(nd_set), axes):\n",
|
||
" label = label_issues.loc[i][\"given_label\"]\n",
|
||
" ax.set_title(f\"id: {i}\\n GL: {label}\", fontdict={\"fontsize\": 8})\n",
|
||
" ax.imshow(dataset[i][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
" seen_id_pairs.update(nd_set)\n",
|
||
" count += 1\n",
|
||
" if count >= nrows:\n",
|
||
" break\n",
|
||
"\n",
|
||
" plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"plot_near_duplicate_issue_examples(near_duplicate_issues_df, num_examples=5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Dark images\n",
|
||
"\n",
|
||
"Datalab can also detect low-quality images in the dataset, such as those that are abnormally dark. It can be challenging for both annotators and models to assign a proper class label for low-quality data, which can hamper model training and testing.\n",
|
||
"\n",
|
||
"The `dark_issues` DataFrame reveals which examples are considered to be abnormally dark. We can sort them via the `dark_score` which quantifies how severe this issue is for each image (lower values indicate more severe instances of a type of issue). This allows us to visualize images in the dataset considered to be too dark (you might consider omitting such low-quality examples from a training dataset)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"dark_issues = lab.get_issues(\"dark\")\n",
|
||
"dark_issues_df = dark_issues.query(\"is_dark_issue\").sort_values(\"dark_score\")\n",
|
||
"dark_issues_df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### View top examples of dark images"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<details><summary>We define a helper method plot_image_issue_examples to visualize results. **(click to expand)**</summary>\n",
|
||
"\n",
|
||
"```python\n",
|
||
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"def plot_image_issue_examples(issues_df, num_examples=15):\n",
|
||
" ncols = 5\n",
|
||
" nrows = int(math.ceil(num_examples / ncols))\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" issue_indices = issues_df.index.values\n",
|
||
"\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i >= num_examples:\n",
|
||
" ax.axis(\"off\")\n",
|
||
" continue\n",
|
||
" idx = int(issue_indices[i])\n",
|
||
" label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" predicted_label = label_issues.loc[idx][\"predicted_label\"]\n",
|
||
" ax.set_title(\n",
|
||
" f\"id: {idx}\\n GL: {label}\\n SL: {predicted_label}\",\n",
|
||
" fontdict={\"fontsize\": 8},\n",
|
||
" )\n",
|
||
" ax.imshow(dataset[idx][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
"\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()\n",
|
||
"```\n",
|
||
"</details>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def plot_image_issue_examples(issues_df, num_examples=15):\n",
|
||
" ncols = 5\n",
|
||
" nrows = int(math.ceil(num_examples / ncols))\n",
|
||
"\n",
|
||
" _, axes = plt.subplots(nrows=nrows, ncols=ncols, figsize=(1.5 * ncols, 1.5 * nrows))\n",
|
||
" axes_list = axes.flatten()\n",
|
||
" issue_indices = issues_df.index.values\n",
|
||
"\n",
|
||
" for i, ax in enumerate(axes_list):\n",
|
||
" if i >= num_examples:\n",
|
||
" ax.axis(\"off\")\n",
|
||
" continue\n",
|
||
" idx = int(issue_indices[i])\n",
|
||
" label = label_issues.loc[idx][\"given_label\"]\n",
|
||
" predicted_label = label_issues.loc[idx][\"predicted_label\"]\n",
|
||
" ax.set_title(\n",
|
||
" f\"id: {idx}\\n GL: {label}\\n SL: {predicted_label}\",\n",
|
||
" fontdict={\"fontsize\": 8},\n",
|
||
" )\n",
|
||
" ax.imshow(dataset[idx][\"image\"], cmap=\"gray\")\n",
|
||
" ax.axis(\"off\")\n",
|
||
"\n",
|
||
" plt.subplots_adjust(hspace=0.7)\n",
|
||
" plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"plot_image_issue_examples(dark_issues_df, num_examples=5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can see from above examples that too dark images can also lead to label errors as it is difficult to see the contents of the image clearly."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Low information images\n",
|
||
"\n",
|
||
"Other types of low-quality images that Datalab can automatically detect include images whose information content is low. Low information images can hamper model generalization if they are present disproportionately in some classes.\n",
|
||
"\n",
|
||
"The `lowinfo_issues` DataFrame reveals which images are considered to be low information. We can sort them via the `low_information_score` which quantifies how severe this issue is for each image (lower values indicate more severe instances of a type of issue). This allows us to visualize the images in our dataset containing the least amount of information (you might consider omitting such low-quality examples from a training dataset)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"lowinfo_issues = lab.get_issues(\"low_information\")\n",
|
||
"lowinfo_issues_df = lowinfo_issues.query(\"is_low_information_issue\").sort_values(\n",
|
||
" \"low_information_score\"\n",
|
||
")\n",
|
||
"lowinfo_issues_df.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"plot_image_issue_examples(lowinfo_issues_df, num_examples=5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here we can see a lot of low information images belong to the Sandal class."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"cleanlab is most effective when you run this code with a good ML model. Try to produce the best ML model you can for your data (instead of the toy model from this tutorial)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"assert set([53050, 40875, 9594, 34825, 37530]).issubset(lowinfo_issues_df.index.values.tolist())\n",
|
||
"assert set([34848, 50270, 3936, 733, 8094]).issubset(dark_issues_df.index.values.tolist())\n",
|
||
"assert set([47824, 3370, 3952, 37119]).issubset(near_duplicate_issues_df.index.values.tolist())\n",
|
||
"assert set([38093, 22628, 44031, 25316, 40329]).issubset(outlier_issues_df.index.values.tolist())\n",
|
||
"assert set([45561, 11262, 54078, 53564]).issubset(label_issues_df.index.values.tolist())"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.7"
|
||
},
|
||
"orig_nbformat": 4
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|