wehub-resource-sync
758 lines
33 KiB
Plaintext
758 lines
33 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d299c1e8",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Finding Label Errors in Object Detection Datasets\n",
|
|
"\n",
|
|
"This 5-minute quickstart tutorial demonstrates how to find potential label errors in object detection datasets. In object detection data, each image is annotated with multiple bounding boxes. Each bounding box surrounds a physical object within an image scene, and is annotated with a given class label. \n",
|
|
"\n",
|
|
"Using such labeled data, we train a model to predict the locations and classes of objects in an image. An example notebook to train the object detection model whose predictions we rely on in this tutorial is available [here](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training.ipynb). These predictions can subsequently be input to cleanlab in order to identify mislabeled images and a quality score quantifying our confidence in the overall annotations for each image. \n",
|
|
"\n",
|
|
"After correcting these label issues, **you can train an even better version of your model without changing your training code!**\n",
|
|
"\n",
|
|
"This tutorial uses a subset of the [COCO (Common Objects in Context)](https://cocodataset.org/#home) dataset which has images of everyday scenes and considers objects from the 5 most popular classes: car, chair, cup, person, traffic light.\n",
|
|
"\n",
|
|
"**Overview of what we we'll do in this tutorial**\n",
|
|
"\n",
|
|
"- Score images based on their overall label quality (i.e. our confidence each image is correctly labeled) using `cleanlab.object_detection.rank.get_label_quality_scores`\n",
|
|
"- Estimate which images have label issues using `cleanlab.object_detection.filter.find_label_issues`\n",
|
|
"- Visually review images + labels using `cleanlab.object_detection.summary.visualize`\n",
|
|
"\n",
|
|
"<div class=\"alert alert-info\">\n",
|
|
"Quickstart\n",
|
|
"<br/>\n",
|
|
" \n",
|
|
"Already have `labels` and `predictions` in the proper format? Just run the code below to find label issues in your object detection dataset.\n",
|
|
"\n",
|
|
"\n",
|
|
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
|
|
" \n",
|
|
"```python\n",
|
|
"\n",
|
|
"from cleanlab.object_detection.filter import find_label_issues\n",
|
|
"from cleanlab.object_detection.rank import get_label_quality_scores\n",
|
|
"\n",
|
|
"# To get boolean vector of label issues for all images\n",
|
|
"has_label_issue = find_label_issues(labels, predictions)\n",
|
|
"\n",
|
|
"# To get label quality scores for all images\n",
|
|
"label_quality_scores = get_label_quality_scores(labels, predictions)\n",
|
|
" \n",
|
|
" \n",
|
|
"```\n",
|
|
"\n",
|
|
"</div>\n",
|
|
"</div>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8d552ab9",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Install required dependencies and download data\n",
|
|
"You can use `pip` to install all packages required for this tutorial as follows\n",
|
|
"```ipython\n",
|
|
"!pip install matplotlib\n",
|
|
"!pip install cleanlab\n",
|
|
"# Make sure to install the version corresponding to this tutorial\n",
|
|
"# E.g. if viewing master branch documentation:\n",
|
|
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0ba0dc70",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Package installation (hidden on docs website).\n",
|
|
"dependencies = [\"cleanlab\", \"matplotlib\", \"huggingface_hub\"]\n",
|
|
"\n",
|
|
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
|
|
" %pip install cleanlab # for colab\n",
|
|
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
|
|
" %pip install $cmd\n",
|
|
"else:\n",
|
|
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
|
|
" else dependency.split('<')[0] if '<' in dependency \n",
|
|
" else dependency.split('=')[0] for dependency in dependencies]\n",
|
|
" missing_dependencies = []\n",
|
|
" for dependency in dependencies_test:\n",
|
|
" try:\n",
|
|
" __import__(dependency)\n",
|
|
" except ImportError:\n",
|
|
" missing_dependencies.append(dependency)\n",
|
|
"\n",
|
|
" if len(missing_dependencies) > 0:\n",
|
|
" print(\"Missing required dependencies:\")\n",
|
|
" print(*missing_dependencies, sep=\", \")\n",
|
|
" print(\"\\nPlease install them before running the rest of this notebook.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c90449c8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%capture\n",
|
|
"from huggingface_hub import hf_hub_download\n",
|
|
"import zipfile\n",
|
|
"import os\n",
|
|
"\n",
|
|
"# Download files from HuggingFace Hub\n",
|
|
"labels_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'labels.pkl', repo_type=\"dataset\")\n",
|
|
"predictions_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'predictions.pkl', repo_type=\"dataset\")\n",
|
|
"images_path = hf_hub_download('Cleanlab/object-detection-tutorial', 'example_images.zip', repo_type=\"dataset\")\n",
|
|
"\n",
|
|
"# Extract images to the correct location\n",
|
|
"# The zip file contains an example_images/ folder, so we extract it and then move contents\n",
|
|
"with zipfile.ZipFile(images_path, 'r') as zip_ref:\n",
|
|
" zip_ref.extractall('temp_extract/')\n",
|
|
"\n",
|
|
"# Move images from nested directory to target directory \n",
|
|
"import shutil\n",
|
|
"if os.path.exists('temp_extract/example_images/'):\n",
|
|
" if os.path.exists('example_images/'):\n",
|
|
" shutil.rmtree('example_images/')\n",
|
|
" shutil.move('temp_extract/example_images/', 'example_images/')\n",
|
|
" shutil.rmtree('temp_extract/')\n",
|
|
"else:\n",
|
|
" # Fallback: extract directly if structure is different\n",
|
|
" with zipfile.ZipFile(images_path, 'r') as zip_ref:\n",
|
|
" zip_ref.extractall('example_images/')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "df8be4c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pickle\n",
|
|
"from cleanlab.object_detection.filter import find_label_issues\n",
|
|
"from cleanlab.object_detection.rank import (\n",
|
|
" _separate_label,\n",
|
|
" _separate_prediction,\n",
|
|
" get_label_quality_scores,\n",
|
|
" issues_from_scores,\n",
|
|
")\n",
|
|
"from cleanlab.object_detection.summary import visualize "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2506badc",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Format data, labels, and model predictions\n",
|
|
"\n",
|
|
"We begin by loading `labels` and `predictions` for our dataset, which are the only inputs required to find label issues with cleanlab. Note that the predictions should be **out-of-sample**, which can be obtained for every image in a dataset via K-fold cross-validation. \n",
|
|
"\n",
|
|
"In a separate [example](https://github.com/cleanlab/examples) notebook ([link](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training.ipynb)), we trained a Detectron2 object detection model and used it to obtain predictions on a held-out validation dataset\u00a0whose `labels` we audit here.\n",
|
|
"\n",
|
|
"**Note:** If you want to find all the mislabeled images across the entire COCO dataset, you can first execute our [other example notebook](https://github.com/cleanlab/examples/blob/master/object_detection/detectron2_training-kfold.ipynb) that uses K-fold cross-validation to produce **out-of-sample** predictions for every image, then use those labels and predictions below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2e9ffd6f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"IMAGE_PATH = './example_images/' # path to raw image files downloaded above\n",
|
|
"\n",
|
|
"# Load pickle files\n",
|
|
"with open(labels_path, 'rb') as f:\n",
|
|
" labels = pickle.load(f)\n",
|
|
"with open(predictions_path, 'rb') as f:\n",
|
|
" predictions = pickle.load(f)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35d49e5d",
|
|
"metadata": {},
|
|
"source": [
|
|
"In object detection datasets, each given label is a made up of bounding box coordinates and a class label. A model prediction is also made up of a bounding box and predicted class label, as well as the model confidence (probability estimate) in its prediction. To detect label issues, cleanlab requires given labels for each image, and the corresponding model predictions for the image (but not the image itself).\n",
|
|
"\n",
|
|
"Here\u2019s what an example looks like in our dataset. We visualize the given and predicted labels (in red and blue) for this image using the `cleanlab.object_detection.summary.visualize` method."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "56705562",
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"image_to_visualize = 8 # change this to view other images\n",
|
|
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
|
|
"\n",
|
|
"# Check if the image file exists before trying to visualize\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=labels[image_to_visualize], prediction=predictions[image_to_visualize], overlay=False)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff36d97f",
|
|
"metadata": {},
|
|
"source": [
|
|
"The required format of these `labels` and `predictions` matches what popular object detection frameworks like [MMDetection](https://github.com/open-mmlab/mmdetection) and [Detectron2](https://github.com/facebookresearch/detectron2/) expect. Recall the 5 possible class labels in our dataset are: car, chair, cup, person, traffic light. These classes are represented as (zero-indexed) integers 0,1,...,4.\n",
|
|
"\n",
|
|
"`labels` is a list where for the i-th image in our dataset, `labels[i]` is a dictionary containing: key `labels` -- a list of class labels for each bounding box in this image and key `bboxes` -- a numpy array of the bounding boxes' coordinates. Each bounding box in `labels[i]['bboxes']` is in the format ``[x1,y1,x2,y2]`` format with respect to the image matrix where `(x1,y1)` corresponds to the top-left corner of the box and `(x2,y2)` the bottom-right (E.g. [XYXY in Keras](https://keras.io/api/keras_cv/bounding_box/formats/), [Detectron 2](https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box)).\n",
|
|
"\n",
|
|
"\n",
|
|
"Let's see what `labels[i]` looks like for our previous example image:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b08144d7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"labels[image_to_visualize]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8f62da67",
|
|
"metadata": {},
|
|
"source": [
|
|
"`predictions` is a list where the predictions output by our model for the i-th image: `predictions[i]` is a list/array of shape `(K,)`. Here `K` is the number of classes in the dataset (same for every image) and `predictions[i][k]` is of shape `(M,5)`, where `M` is the number of bounding boxes predicted to contain objects of class `k` (in image i, differs between images). The five columns of `predictions[i][k]` correspond to ``[x1,y1,x2,y2,pred_prob]`` format with respect to the image matrix for each bounding box predicted by the model. Here `(x1,y1)` corresponds to the top-left corner of the box and `(x2,y2)` the bottom-right (E.g. [XYXY in Keras](https://keras.io/api/keras_cv/bounding_box/formats/), [Detectron 2](https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box)). The last column, `pred_prob` is the model confidence in its predicted label of class `k` for this box. Since our dataset has `K = 5` classes, we have: `predictions[i].shape = (5,)`.\n",
|
|
"\n",
|
|
"Let's see what `predictions[i]` looks like for our previous example image:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3d70bec6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"predictions[image_to_visualize]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cf95ea28",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"Once you have `labels` and `predictions` in the appropriate formats, you can **find label issues with cleanlab for any object detection dataset**!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3daff923",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Use cleanlab to find label issues\n",
|
|
"Given `labels` and `predictions` from our trained model, cleanlab can automatically find mislabeled images in the dataset. In object detection, we consider an image mislabeled if **any** of its bounding boxes or their class labels are incorrect (including if the image contains any overlooked objects which should've been annotated with a box)\n",
|
|
"\n",
|
|
"Images may be mislabeled because annotators:\n",
|
|
"\n",
|
|
"- overlooked an object (forgot to annotate a bounding box around a depicted object)\n",
|
|
"- chose the wrong class label for an annotated box in the correct location\n",
|
|
"- imperfectly drew the bounding box such that its location is incorrect\n",
|
|
"\n",
|
|
"\n",
|
|
"Cleanlab is expected to flag images that exhibit **any** of these annotation errors as having label issues. More severe annotation errors are expected to produce lower cleanlab label quality scores closer to 0. Let's first estimate which images have label issues:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4caa635d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"label_issue_idx = find_label_issues(labels, predictions, return_indices_ranked_by_score=True)\n",
|
|
"\n",
|
|
"num_examples_to_show = 5 # view this many images flagged with the most severe label issues\n",
|
|
"label_issue_idx[:num_examples_to_show]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "66d5fae1",
|
|
"metadata": {},
|
|
"source": [
|
|
"The above code identifies *which* images have label issues, returning a list of their indices. This is because we specified the `return_indices_ranked_by_score` argument which sorts these indices by the estimated label quality of each image. Below we describe how to directly estimate the label quality scores of each image.\n",
|
|
"\n",
|
|
"**Note:** You can omit the `return_indices_ranked_by_score` argument for `find_label_issues()` to instead return a Boolean mask for the entire dataset (True entries in this mask correspond to images with label issues)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5b501dc9",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Get label quality scores\n",
|
|
"Cleanlab can also compute scores for each image to estimate our confidence that it has been correctly labeled. These label quality scores range between 0 and 1, with *smaller* values indicating examples whose annotation is *more* likely to be wrong in some way.\n",
|
|
"\n",
|
|
"Each image in the dataset receives a label quality score. These scores are useful for prioritizing which images to review; if you have too little time, first review the images with the lowest label quality scores."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a9b4c590",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"scores = get_label_quality_scores(labels, predictions)\n",
|
|
"scores[:num_examples_to_show]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "349521e0",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can also use the label quality scores to flag *which* images have label issues based on a threshold. Here we convert these per-image scores into an array of indices corresponding to images flagged with label issues, sorted by label quality score, in the same format returned by `find_label_issues()`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ffd9ebcc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issue_idx = issues_from_scores(scores, threshold=0.5) # lower threshold will return fewer (but more confident) label issues\n",
|
|
"issue_idx[:num_examples_to_show], scores[issue_idx][:num_examples_to_show]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a3b8aa0",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Use ObjectLab to visualize label issues\n",
|
|
"Finally, we can visualize images with potential label errors via cleanlab's `visualize()` function. To enhance the visualization, you can supply a `class_names` dictionary to include as a legend and turn off `overlay` to see the given and predicted labels side by side."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4dd46d67",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issue_to_visualize = issue_idx[0] # change this to view other images\n",
|
|
"class_names = {\"0\": \"car\", \"1\": \"chair\", \"2\": \"cup\", \"3\":\"person\", \"4\": \"traffic light\"}\n",
|
|
"\n",
|
|
"label = labels[issue_to_visualize]\n",
|
|
"prediction = predictions[issue_to_visualize]\n",
|
|
"score = scores[issue_to_visualize]\n",
|
|
"image_path = IMAGE_PATH + label['seg_map']\n",
|
|
"\n",
|
|
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "de0d7205",
|
|
"metadata": {},
|
|
"source": [
|
|
"The visualization depicts the given label (original image annotation which cleanlab identified as problematic) in red on the left and the model-predicted label in blue on the right. Each bounding box contains a class-index number in the top corner indicating which object class that bounding box was annotated/predicted to contain.\n",
|
|
"\n",
|
|
"This image has a **low** label quality score and is marked as an error. On closer inspection we notice the annotator missed the reflection of the person in the mirror that the model identified. Additionally, the chairs visible in the reflection were not annotated.\n",
|
|
"\n",
|
|
"Notice examples where the predictions and labels are more similar have higher quality scores than those that are missmatched, and are less likeley to be marked as issues and the number of boxes is agnostic to the score.\n",
|
|
"\n",
|
|
"Better trained models will lead to better label error detection but you don't need a near perfect model to identify label issues.\n",
|
|
"\n",
|
|
"\n",
|
|
"### Different kinds of label issues identified by ObjectLab\n",
|
|
"Now lets view the first few images in our vaidation dataset that are clearly marked as issues and see what various inconsistencies between the `given` and `predicted` label we can spot. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ceec2394",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issue_to_visualize = issue_idx[1]\n",
|
|
"label = labels[issue_to_visualize]\n",
|
|
"prediction = predictions[issue_to_visualize]\n",
|
|
"score = scores[issue_to_visualize]\n",
|
|
"\n",
|
|
"image_path = IMAGE_PATH + label['seg_map']\n",
|
|
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9b5c87fa",
|
|
"metadata": {},
|
|
"source": [
|
|
"Notice the armchair to the left of the TV is missing an annotation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "94f82b0d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issue_to_visualize = issue_idx[9]\n",
|
|
"label = labels[issue_to_visualize]\n",
|
|
"prediction = predictions[issue_to_visualize]\n",
|
|
"score = scores[issue_to_visualize]\n",
|
|
"\n",
|
|
"image_path = IMAGE_PATH + label['seg_map']\n",
|
|
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "05610be0",
|
|
"metadata": {},
|
|
"source": [
|
|
"Similarly, the woman in a red jacket in the foreground is missing an annotation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1ea18c5d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issue_to_visualize = issue_idx[2]\n",
|
|
"label = labels[issue_to_visualize]\n",
|
|
"prediction = predictions[issue_to_visualize]\n",
|
|
"score = scores[issue_to_visualize]\n",
|
|
"\n",
|
|
"image_path = IMAGE_PATH + label['seg_map']\n",
|
|
"print(image_path, '| idx', issue_to_visualize , '| label quality score:', score, '| is issue: True')\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=label, prediction=prediction, class_names=class_names, overlay=False)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "05c9229d",
|
|
"metadata": {},
|
|
"source": [
|
|
"The people in this image should have had individual bounding boxes around each persons (the COCO guidelines state only groups with 10+ objects of the same type can be a \\\"crowd\\\" bounded by a single box). Individuals in the back are missing annotations.\n",
|
|
"\n",
|
|
"All of these examples received low label quality scores reflecting their low annotation quality in the original dataset."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "03d5a521",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Other uses of visualize\n",
|
|
"The `visualize()` function can also depict non-issue images, labels or predictions alone, or just the image itself. Let's explore this with a few images in our dataset.\n",
|
|
"\n",
|
|
"We can save a visualization to file via the `save_path` argument. Note the label quality score is high for this example and it is marked as a non-issue. The given and predicted labels closely resemble each other contributing to the high score."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7e770d23",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_to_visualize = 0\n",
|
|
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
|
|
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=labels[image_to_visualize], prediction=predictions[image_to_visualize], class_names=class_names, save_path='./example_image.png')\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6c9464e8",
|
|
"metadata": {},
|
|
"source": [
|
|
"For the next example, notice how we are only passing in the given labels to visualize. We can limit visualization to either labels, predictions, or neither."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "57e84a27",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_to_visualize = 3\n",
|
|
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
|
|
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=labels[image_to_visualize], class_names=class_names)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d8744ab9",
|
|
"metadata": {},
|
|
"source": [
|
|
"For completeness, let's just look at an image alone."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0302818a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_to_visualize = 2\n",
|
|
"image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
|
|
"print(image_path, '| idx', image_to_visualize , '| label quality score:', scores[image_to_visualize], '| is issue:', image_to_visualize in issue_idx)\n",
|
|
"if os.path.exists(image_path):\n",
|
|
" visualize(image_path)\n",
|
|
"else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "46d6282a-4601-4cc3-b8a8-187ea6d5f8bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Exploratory data analysis\n",
|
|
"\n",
|
|
"This bonus section considers techniques to uncover annotation irregularities through exploratory data analysis. Specifically, we consider anomalies in object sizes, detect images with unusual object counts, and examine the distribution of class labels.\n",
|
|
"\n",
|
|
"Let's first consider the number of objects per image, and inspect the images with the largest values (which might reveal something off in our dataset):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5cacec81-2adf-46a8-82c5-7ec0185d4356",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"from cleanlab.internal.object_detection_utils import calculate_bounding_box_areas\n",
|
|
"from cleanlab.object_detection.summary import (\n",
|
|
" bounding_box_size_distribution,\n",
|
|
" class_label_distribution,\n",
|
|
" object_counts_per_image,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3335b8a3-d0b4-415a-a97d-c203088a124e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"num_imgs_to_show = 3\n",
|
|
"lab_object_counts,pred_object_counts = object_counts_per_image(labels,predictions)\n",
|
|
"for image_to_visualize in np.argsort(lab_object_counts)[::-1][0:num_imgs_to_show]:\n",
|
|
" image_path = IMAGE_PATH + labels[image_to_visualize]['seg_map']\n",
|
|
" print(image_path, '| idx', image_to_visualize)\n",
|
|
" if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=labels[image_to_visualize], class_names=class_names)\n",
|
|
" else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e5ddd4fe-4477-4b68-ba79-e5cbb62822eb",
|
|
"metadata": {},
|
|
"source": [
|
|
"Next let's study the distribution of class labels in the overall annotations, comparing the distribution in the given annotations vs. in the model predictions. This can sometimes reveal that something's off in our dataset or model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9d4b7677-6ebd-447d-b0a1-76e094686628",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"label_norm,pred_norm = class_label_distribution(labels,predictions)\n",
|
|
"print(\"Frequency of each class amongst annotated | predicted bounding boxes in the dataset:\\n\")\n",
|
|
"for i in label_norm:\n",
|
|
" print(f\"{class_names[str(i)]} : {label_norm[i]} | {pred_norm[i]}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "200cdebf-b24c-4c2b-8914-6a2fce218daf",
|
|
"metadata": {},
|
|
"source": [
|
|
"Finally, let's consider the distribution of bounding box sizes (aka object sizes) in the given annotations for each class label. The idea is to review any anomalies in bounding box areas for a given class (which might reveal problematic annotations or abnormal instances of this object class). The following code determines such anomalies by assessing each bounding box's area vs. the mean and standard deviation of areas for bounding boxes with the same class label."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "59d7ee39-3785-434b-8680-9133014851cd",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"lab_area,pred_area = bounding_box_size_distribution(labels,predictions)\n",
|
|
"lab_area_mean = {i: np.mean(lab_area[i]) for i in lab_area.keys()}\n",
|
|
"lab_area_std = {i: np.std(lab_area[i]) for i in lab_area.keys()}\n",
|
|
"\n",
|
|
"max_deviation_values = []\n",
|
|
"max_deviation_classes = []\n",
|
|
"\n",
|
|
"for label in labels:\n",
|
|
" bounding_boxes, label_names = _separate_label(label)\n",
|
|
" areas = calculate_bounding_box_areas(bounding_boxes)\n",
|
|
" deviation_values = []\n",
|
|
" deviation_classes = []\n",
|
|
"\n",
|
|
" for class_name, mean_area, std_area in zip(lab_area_mean.keys(), lab_area_mean.values(), lab_area_std.values()):\n",
|
|
" class_areas = areas[label_names == class_name]\n",
|
|
" deviations_away = (class_areas - mean_area) / std_area\n",
|
|
" deviation_values.extend(list(deviations_away))\n",
|
|
" deviation_classes.extend([class_name] * len(class_areas))\n",
|
|
"\n",
|
|
" if deviation_values==[]:\n",
|
|
" max_deviation_values.append(0.0)\n",
|
|
" max_deviation_classes.append(-1)\n",
|
|
" else:\n",
|
|
" max_deviation_index = np.argmax(np.abs(deviation_values))\n",
|
|
" max_deviation_values.append(deviation_values[max_deviation_index])\n",
|
|
" max_deviation_classes.append(deviation_classes[max_deviation_index])\n",
|
|
"\n",
|
|
"max_deviation_classes, max_deviation_values = np.array(max_deviation_classes), np.array(max_deviation_values)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b260142e-b760-490c-818e-c037fab5c6c8",
|
|
"metadata": {},
|
|
"source": [
|
|
"In our dataset here, this analysis reveals certain abnormally large bounding boxes that take up most of the image."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "47b6a8ff-7a58-4a1f-baee-e6cfe7a85a6d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"num_imgs_to_show_per_class = 3\n",
|
|
"\n",
|
|
"for c in class_names.keys():\n",
|
|
" class_num = int(c)\n",
|
|
" sorted_indices = np.argsort(max_deviation_values)[::-1]\n",
|
|
" count = 0\n",
|
|
"\n",
|
|
" for i in sorted_indices:\n",
|
|
" if max_deviation_values[i] == 0 or max_deviation_classes[i] != class_num:\n",
|
|
" continue\n",
|
|
" image_path = IMAGE_PATH + labels[i]['seg_map']\n",
|
|
" print(image_path, '| idx', i, '| class', class_names[c])\n",
|
|
" if os.path.exists(image_path):\n",
|
|
" visualize(image_path, label=labels[i], class_names=class_names)\n",
|
|
" else:\n",
|
|
" print(f\"Image file not found: {image_path}\")\n",
|
|
" print(\"Skipping visualization - this is expected during documentation build.\")\n",
|
|
"\n",
|
|
" count += 1\n",
|
|
" if count == num_imgs_to_show_per_class:\n",
|
|
" break # Break the loop after visualizing the top 3 instances for the current class"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8ce74938",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
|
"\n",
|
|
"expected_values = {0: 50, 1: 16, 2: 31, 9: 62}\n",
|
|
"\n",
|
|
"for idx, value in expected_values.items():\n",
|
|
" assert value in issue_idx and issue_idx[idx] == value, f\"Assertion error at index {idx}: Expected {value}, got {issue_idx.get(idx, None)}\"\n",
|
|
"\n",
|
|
"assert all(i not in issue_idx for i in [0, 2, 3]), \"Unexpected values found in issue_idx\""
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |