wehub-resource-sync
925 lines
42 KiB
Plaintext
925 lines
42 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ffe0d62e",
|
||
"metadata": {},
|
||
"source": [
|
||
"# FAQ\n",
|
||
"\n",
|
||
"Answers to frequently asked questions about the [cleanlab](https://github.com/cleanlab/cleanlab) open-source package.\n",
|
||
"\n",
|
||
"The code snippets in this FAQ come from a fully executable notebook you can run via Colab or locally by downloading it [here](https://github.com/cleanlab/cleanlab/blob/master/docs/source/tutorials/faq.ipynb).\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2a4efdde",
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# This cell is hidden on docs.cleanlab.ai. Execute it to ensure all other cells below can be executed in your own notebook\n",
|
||
"\n",
|
||
"import os \n",
|
||
"import logging \n",
|
||
"import numpy as np \n",
|
||
"import sklearn \n",
|
||
"import cleanlab \n",
|
||
"\n",
|
||
"np.random.seed(123)\n",
|
||
"\n",
|
||
"# Toy dataset:\n",
|
||
"N = 50\n",
|
||
"K = 3\n",
|
||
"num_errors = 4\n",
|
||
"labels = np.random.randint(low=0, high=K, size=N)\n",
|
||
"pred_probs = np.random.random_sample(N*K).reshape((N,K))\n",
|
||
"pred_probs[np.arange(N),labels] += 4 # make pred_probs accurate\n",
|
||
"pred_probs = pred_probs/pred_probs.sum(axis=1)[:, np.newaxis]\n",
|
||
"data = np.array([[label+np.random.uniform(), label+np.random.uniform()] for label in labels])\n",
|
||
"# introduce label errors in last few examples:\n",
|
||
"og0_indices = labels[-num_errors:] == 0\n",
|
||
"labels[-num_errors:] = 0\n",
|
||
"labels[-num_errors:][og0_indices] = 1\n",
|
||
"\n",
|
||
"your_classifier=sklearn.linear_model.LogisticRegression() # toy classifier"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d504ec58",
|
||
"metadata": {},
|
||
"source": [
|
||
"### What data can cleanlab detect issues in?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5e70efbc",
|
||
"metadata": {},
|
||
"source": [
|
||
"This package can be used to detect issues in any dataset for which you have trained a ML model. This includes datasets involving: multiple annotators per example (multi-annotator), or multiple labels per example (multi-label). This includes data from any modality such as: image, text, tabular, audio, etc. This package supports most common supervised learning tasks (entity recognition in text, image segmentation, object detection, tagging, regression, ...). If you have a particular task in mind, [let us know](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue)!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "eca36874",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How do I format classification labels for cleanlab?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "38c50875",
|
||
"metadata": {},
|
||
"source": [
|
||
"**With Datalab**:\n",
|
||
"\n",
|
||
"Datalab simplifies label management by accepting both string and integer labels directly. Internally, unique labels are sorted alphanumerically and mapped to integers, facilitating seamless integration with lower-level cleanlab methods. Below are the supported label formats:\n",
|
||
"\n",
|
||
"- **List of strings or integers**: Directly pass labels as a list of strings or integers without manual encoding.\n",
|
||
"\n",
|
||
"- **Using** `datasets.Dataset` **with** `ClassLabel`: For advanced use cases, you can structure your dataset using HuggingFace's `datasets.Dataset` object, specifying label columns as `ClassLabel` feature objects for formatting the labels. Refer to the [datasets documentation](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.ClassLabel) for detailed guidance.\n",
|
||
"\n",
|
||
"```python\n",
|
||
"from cleanlab import Datalab\n",
|
||
"from datasets import Dataset, Features, Value, ClassLabel\n",
|
||
"\n",
|
||
"# Example 1: Labels as a list of strings\n",
|
||
"labels_str = ['cat', 'dog', 'cat', 'dog']\n",
|
||
"datalab_str = Datalab(data={\"text\": [\"a\", \"b\", \"c\", \"d\"], \"label\": labels_str}, label_name=\"label\")\n",
|
||
"print(\"String labels:\", datalab_str.labels)\n",
|
||
"\n",
|
||
"# Example 2: Labels as a list of integers\n",
|
||
"labels_int = [1, 2, 2, 1] # These will be remapped to [0, 1] internally\n",
|
||
"datalab_int = Datalab(data={\"text\": [\"a\", \"b\", \"c\", \"d\"], \"label\": labels_int}, label_name=\"label\")\n",
|
||
"print(\"Integer labels:\", datalab_int.labels)\n",
|
||
"\n",
|
||
"# Example 3: Advanced - Dataset with ClassLabel feature\n",
|
||
"my_dict = {\"pet_name\": [\"Spot\", \"Mittens\", \"Rover\", \"Rocky\", \"Pepper\", \"Socks\"], \"species\": [\"dog\", \"cat\", \"dog\", \"dog\", \"cat\", \"cat\"]}\n",
|
||
"features = Features({\"pet_name\": Value(\"string\"), \"species\": ClassLabel(names=[\"dog\", \"cat\"])})\n",
|
||
"dataset = Dataset.from_dict(my_dict, features=features)\n",
|
||
"datalab_dataset = Datalab(data=dataset, label_name=\"species\")\n",
|
||
"print(\"ClassLabel feature:\", datalab_dataset.labels)\n",
|
||
"```\n",
|
||
"\n",
|
||
"Using Datalab allows you to directly handle raw class name labels in your dataset while ensuring compatibility with label encoding requirements of lower-level cleanlab methods, which we'll cover in the next section.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d5d0fbb3",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Without Datalab**:\n",
|
||
"\n",
|
||
"Outside of Datalab, cleanlab offers various lower-level methods to directly operate on labels and diagnose issues. For instance: ``get_label_quality_scores()`` and ``find_label_issues()``. These lower-level methods only work with integer-encoded labels in the range `{0,1, ... K-1}` where `K = number_of_classes`. The `labels` array should only contain integer values in the range `{0, K-1}` and be of shape `(N,)` where `N = total_number_of_data_points`.\n",
|
||
"Do not pass in `labels` where some classes are entirely missing or are extremely rare, as cleanlab may not perform as expected. It is better to remove such classes entirely from the dataset first (also dropping the corresponding dimensions from `pred_probs` and then renormalizing it).\n",
|
||
"\n",
|
||
"**Text or string labels** should to be mapped to integers for each possible value. For example if your original data labels look like this: `[\"dog\", \"dog\", \"cat\", \"mouse\", \"cat\"]`, you should feed them to cleanlab like this: `labels = [1,1,0,2,0]` and keep track of which integer uniquely represents which class (classes were ordered alphabetically in this example). \n",
|
||
"\n",
|
||
"**One-hot encoded labels** should be integer-encoded by finding the argmax along the one-hot encoded axis. An example of what this might look like is shown below."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "239d5ee7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np \n",
|
||
"\n",
|
||
"# This example arr has 4 labels (one per data point) where \n",
|
||
"# each label can be one of 3 possible classes\n",
|
||
"\n",
|
||
"arr = np.array([[0,1,0],[1,0,0],[0,0,1],[1,0,0]])\n",
|
||
"labels_proper_format = np.argmax(arr, axis=1) # How labels should be formatted when passed into the model"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4181cac7",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How do I infer the correct labels for examples cleanlab has flagged?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6d4db5e1",
|
||
"metadata": {},
|
||
"source": [
|
||
"If you have a classifier that is compatible with [CleanLearning](../cleanlab/classification.html) (i.e. follows the sklearn API), here's an easy way to see predicted labels alongside the label issues:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "28b324aa",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"cl = cleanlab.classification.CleanLearning(your_classifier)\n",
|
||
"issues_dataframe = cl.find_label_issues(data, labels)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6d4db5e2",
|
||
"metadata": {},
|
||
"source": [
|
||
"Alternatively if you have already computed out-of-sample predicted probabilities (`pred_probs`) from a classifier:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "28b324ab",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"cl = cleanlab.classification.CleanLearning()\n",
|
||
"issues_dataframe = cl.find_label_issues(X=None, labels=labels, pred_probs=pred_probs)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b386dfc8",
|
||
"metadata": {},
|
||
"source": [
|
||
"Otherwise if you have already found issues via:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "90c10e18",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"issues = cleanlab.filter.find_label_issues(labels, pred_probs)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ad9ca03e",
|
||
"metadata": {},
|
||
"source": [
|
||
"then you can see your trained classifier's class prediction for each flagged example like this: "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "88839519",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class_predicted_for_flagged_examples = pred_probs[issues].argmax(axis=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a668b74b",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here you can see the classifier's class prediction for every example via:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "558490c2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"class_predicted_for_all_examples = pred_probs.argmax(axis=1)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f9450eed",
|
||
"metadata": {},
|
||
"source": [
|
||
"We caution against just blindly taking the predicted label for granted, many of these suggestions may be wrong!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bcc97591",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How should I handle label errors in train vs. test data?\n",
|
||
"\n",
|
||
"If you do not address label errors in your test data, you may not even know when you have produced a better ML model because the evaluation is too noisy. For the best-trained models and most reliable evaluation of them, you should fix label errors in both training and testing data.\n",
|
||
"\n",
|
||
"To do this efficiently, first use cleanlab to automatically find label issues in both sets. You can simply merge these two sets into one larger dataset and run cross-validation training. On the merged dataset, you can do either of the following to detect label issues:\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"**With Datalab**: Run `Datalab.find_issues()` on the merged dataset, then call `Datalab.report()` to see the label issues (and other types of data issues).\n",
|
||
"\n",
|
||
"```python\n",
|
||
"from cleanlab import Datalab\n",
|
||
"\n",
|
||
"lab = Datalab(data = merged_dataset, label_name = \"label_column_name\")\n",
|
||
"\n",
|
||
"# Run proper cross-validation when computing predicted probabilities\n",
|
||
"lab.find_issues(pred_probs = pred_probs, issue_types = {\"label\": {}})\n",
|
||
"\n",
|
||
"lab.report()\n",
|
||
"```\n",
|
||
"\n",
|
||
"You can fetch the label issues DataFrame from the `Datalab` object by calling:\n",
|
||
"\n",
|
||
"```python\n",
|
||
"label_issues = lab.get_issues(\"label\")\n",
|
||
"```\n",
|
||
"\n",
|
||
"**Without Datalab**: Run cleanlab's lower-level `find_label_issues()` method on the merged datataset. Calling the [CleanLearning.find_label_issues()](../cleanlab/classification.html) method on your merged dataset both runs cross-validation training and finds label issues for you with any scikit-learn compatible classifier you choose.\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"After finding label issues, be **wary** about auto-correcting the labels for test examples. Instead manually fix the labels for your test data via careful review of the flagged issues.\n",
|
||
"\n",
|
||
"Auto-correcting labels for your training data is fair game, which should improve ML performance (if properly evaluated with clean test labels). You can boost ML performance further by manually fixing the training examples flagged with label issues, as demonstrated in this article:\n",
|
||
"\n",
|
||
"[**Handling Mislabeled Tabular Data to Improve Your XGBoost Model**](https://cleanlab.ai/blog/label-errors-tabular-datasets/)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "21f42f24",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How can I find label issues in big datasets with limited memory? "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "089f505e",
|
||
"metadata": {},
|
||
"source": [
|
||
"For a dataset with many rows and/or classes, there are more efficient methods in the `label_issues_batched` module. These methods read data in mini-batches and you can reduce the `batch_size` to control how much memory they require. Below is an example of how to use the `find_label_issues_batched()` method from this module, which can load mini-batches of data from `labels`, `pred_probs` saved as .npy files on disk. You can also run this method on Zarr arrays loaded from .zarr files. Try playing with the `n_jobs` argument for further multiprocessing speedups. If you need greater flexibility, check out the `LabelInspector` class from this module."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "41714b51",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# We'll assume your big arrays of labels, pred_probs have been saved to file like this:\n",
|
||
"from tempfile import mkdtemp\n",
|
||
"import os.path as path\n",
|
||
"\n",
|
||
"labels_file = path.join(mkdtemp(), \"labels.npy\")\n",
|
||
"pred_probs_file = path.join(mkdtemp(), \"pred_probs.npy\")\n",
|
||
"np.save(labels_file, labels)\n",
|
||
"np.save(pred_probs_file, pred_probs)\n",
|
||
"\n",
|
||
"# Code to find label issues by loading data from file in batches:\n",
|
||
"from cleanlab.experimental.label_issues_batched import find_label_issues_batched\n",
|
||
"\n",
|
||
"batch_size = 10000 # for efficiency, set this to as large of a value as your memory can handle\n",
|
||
"\n",
|
||
"# Indices of examples with label issues, sorted by label quality score (most severe to least severe):\n",
|
||
"indices_of_examples_with_issues = find_label_issues_batched(\n",
|
||
" labels_file=labels_file, pred_probs_file=pred_probs_file, batch_size=batch_size\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "13228a99-5d3f-47c0-87e5-2290d16461c4",
|
||
"metadata": {},
|
||
"source": [
|
||
"Methods that internally call `filter.find_label_issues()` can be sped up by specifying the argument `low_memory=True`, which will instead use `find_label_issues_batched()` internally. The following methods provide this option: \n",
|
||
"\n",
|
||
"1. [classification.CleanLearning](../cleanlab/classification.html#cleanlab.classification.CleanLearning)\n",
|
||
"2. [multilabel_classification.filter.find_label_issues](../cleanlab/multilabel_classification/filter.html#cleanlab.multilabel_classification.filter.find_label_issues)\n",
|
||
"3. [token_classification.filter.find_label_issues](../cleanlab/token_classification/filter.html?highlight=token#cleanlab.token_classification.filter.find_label_issues)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "20476c70",
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
|
||
"\n",
|
||
"issue_indices = cleanlab.filter.find_label_issues(labels, pred_probs, filter_by = \"low_self_confidence\", return_indices_ranked_by=\"self_confidence\")\n",
|
||
"assert np.abs(len(issue_indices) - len(indices_of_examples_with_issues)) < 2, \"num issues differ in batched mode\"\n",
|
||
"set1 = set(issue_indices)\n",
|
||
"set2 = set(indices_of_examples_with_issues)\n",
|
||
"intersection = len(list(set1.intersection(set2)))\n",
|
||
"union = len(set1) + len(set2) - intersection\n",
|
||
"assert float(intersection) / union > 0.95, \"issue indices differ in batched mode\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "438b424d",
|
||
"metadata": {},
|
||
"source": [
|
||
"**To use less memory and get results faster if your dataset has many classes:** Try merging the rare classes into a single \"Other\" class before you find label issues. The resulting issues won't be affected much since cleanlab anyway does not have enough data to accurately diagnose label errors in classes that are rarely seen. To do this, you should aggregate all the probability assigned to the rare classes in `pred_probs` into a single new dimension of `pred_probs_merged` (where this new array no longer has columns for the rare classes). Here is a function that does this for you, which you can also modify as needed:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "6983cdad",
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# This cell is hidden on docs.cleanlab.ai\n",
|
||
"# Add two rare additional classes to the dataset:\n",
|
||
"\n",
|
||
"num_rare_instances = 3\n",
|
||
"small_prob = 1e-4\n",
|
||
"pred_probs = np.hstack((pred_probs, np.ones((len(pred_probs),2))*small_prob))\n",
|
||
"pred_probs = pred_probs / np.sum(pred_probs, axis=1)[:, np.newaxis]\n",
|
||
"labels[:num_rare_instances] = 3\n",
|
||
"labels[num_rare_instances:(2*num_rare_instances)] = 4"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9092b8a0",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab.internal.util import value_counts # use this to count how often each class occurs in labels\n",
|
||
"\n",
|
||
"def merge_rare_classes(labels, pred_probs, count_threshold = 10):\n",
|
||
" \"\"\" \n",
|
||
" Returns: labels, pred_probs after we merge all rare classes into a single 'Other' class.\n",
|
||
" Merged pred_probs has less columns. Rare classes are any occuring less than `count_threshold` times.\n",
|
||
" Also returns: `class_mapping_orig2new`, a dict to map new classes in merged labels back to classes \n",
|
||
" in original labels, useful for interpreting outputs from `dataset.heath_summary()` or `count.confident_joint()`.\n",
|
||
" \"\"\"\n",
|
||
" num_classes = pred_probs.shape[1]\n",
|
||
" num_examples_per_class = value_counts(labels, num_classes=num_classes)\n",
|
||
" rare_classes = [c for c in range(num_classes) if num_examples_per_class[c] < count_threshold]\n",
|
||
" if len(rare_classes) < 1:\n",
|
||
" raise ValueError(\"No rare classes found at the given `count_threshold`, merging is unnecessary unless you increase it.\")\n",
|
||
"\n",
|
||
" num_classes_merged = num_classes - len(rare_classes) + 1 # one extra class for all the merged ones\n",
|
||
" other_class = num_classes_merged - 1\n",
|
||
" labels_merged = labels.copy()\n",
|
||
" class_mapping_orig2new = {} # key = original class in `labels`, value = new class in `labels_merged`\n",
|
||
" new_c = 0\n",
|
||
" for c in range(num_classes):\n",
|
||
" if c in rare_classes:\n",
|
||
" class_mapping_orig2new[c] = other_class\n",
|
||
" else:\n",
|
||
" class_mapping_orig2new[c] = new_c\n",
|
||
" new_c += 1\n",
|
||
" labels_merged[labels == c] = class_mapping_orig2new[c]\n",
|
||
"\n",
|
||
" merged_prob = np.sum(pred_probs[:, rare_classes], axis=1, keepdims=True) # total probability over all merged classes for each example\n",
|
||
" pred_probs_merged = np.hstack((np.delete(pred_probs, rare_classes, axis=1), merged_prob)) # assumes new_class is as close to original_class in sorted order as is possible after removing the merged original classes\n",
|
||
" # check a few rows of probabilities after merging to verify they still sum to 1:\n",
|
||
" num_check = 1000 # only check a few rows for efficiency\n",
|
||
" ones_array_ref = np.ones(min(num_check,len(pred_probs)))\n",
|
||
" if np.isclose(np.sum(pred_probs[:num_check], axis=1), ones_array_ref).all() and (not np.isclose(np.sum(pred_probs_merged[:num_check], axis=1), ones_array_ref).all()):\n",
|
||
" raise ValueError(\"merged pred_probs do not sum to 1 in each row, check that merging was correctly done.\")\n",
|
||
" \n",
|
||
" return (labels_merged, pred_probs_merged, class_mapping_orig2new)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b0a01109",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab.filter import find_label_issues # can alternatively use find_label_issues_batched() shown above\n",
|
||
"\n",
|
||
"labels_merged, pred_probs_merged, class_mapping_orig2new = merge_rare_classes(labels, pred_probs, count_threshold=5)\n",
|
||
"examples_w_issues = find_label_issues(labels_merged, pred_probs_merged, return_indices_ranked_by=\"self_confidence\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "8b1da032",
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
|
||
"\n",
|
||
"rare_classes = [c for c in class_mapping_orig2new.keys() if class_mapping_orig2new[c] == pred_probs_merged.shape[1]-1]\n",
|
||
"og_examples_w_issues = find_label_issues(labels, pred_probs, return_indices_ranked_by=\"self_confidence\")\n",
|
||
"examples_of_interest = [x for x in examples_w_issues if labels[x] not in rare_classes]\n",
|
||
"og_examples_of_interest = [x for x in og_examples_w_issues if labels[x] not in rare_classes]\n",
|
||
"assert set(examples_of_interest) == set(og_examples_of_interest), \"merged label issues differ from non-merged label issues\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3868ee8b",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Why isn’t CleanLearning working for me?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d13c9cd0",
|
||
"metadata": {},
|
||
"source": [
|
||
"At this time, CleanLearning only works with data formatted as numpy matrices or pd.DataFrames, \n",
|
||
"and with models that are compatible with the `sklearn` API \n",
|
||
"(check out [skorch](https://github.com/skorch-dev/skorch) for Pytorch compatibility). \n",
|
||
"You can still use cleanlab with other data formats though! Just separately obtain predicted probabilities (`pred_probs`) from your model via cross-validation and pass them as inputs. \n",
|
||
"\n",
|
||
"\n",
|
||
"If CleanLearning is running successfully but not improving predictive accuracy of your model, here are some tips:\n",
|
||
"\n",
|
||
"1. Use cleanlab to find label issues in your test data as well (we recommend pooling `labels` across both training and test data into one input for `find_label_issues()`). Then manually review and fix label issues identified in the test data to verify accuracy measurements are actually meaningful.\n",
|
||
"\n",
|
||
"2. Try different values for `filter_by`, `frac_noise`, and `min_examples_per_class` which can be set via the `find_label_issues_kwargs` argument in the initialization of `CleanLearning()`.\n",
|
||
"\n",
|
||
"3. Try to find a better model (eg. via hyperparameter tuning or changing to another classifier). `CleanLearning` can find better label issues by leveraging a better model, which allows it to produce better quality training data. This can form a virtuous cycle in which better models -> better issue detection -> better data -> even better models! \n",
|
||
"\n",
|
||
"4. Try jointly tuning both model hyperparameters and `find_label_issues_kwargs` values.\n",
|
||
"\n",
|
||
"5. Does your dataset have a *junk* (or *clutter*, *unknown*, *other*) class? If you have bad data, consider creating one (c.f. Caltech-256).\n",
|
||
"\n",
|
||
"6. Consider merging similar/overlapping classes found via ``cleanlab.dataset.find_overlapping_classes``.\n",
|
||
"\n",
|
||
"Other general tips to improve label error detection performance:\n",
|
||
"\n",
|
||
"1. Try creating more restrictive new filters by combining their intersections (e.g. `combined_boolean_mask = mask1 & mask2` where `mask1` and `mask2` are the boolean masks created by running `find_label_issues` with different values of the `filter_by` argument).\n",
|
||
"\n",
|
||
"2. If your `pred_probs` are obtained via a neural network, try averaging the `pred_probs` over the last K epochs of training instead of just using the final `pred_probs`. Similarly, you can try averaging `pred_probs` from several models (remember to re-normalize) or using ``cleanlab.rank.get_label_quality_ensemble_scores``."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9ae3899c",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How can I use different models for data cleaning vs. final training in CleanLearning?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a2ce1518",
|
||
"metadata": {},
|
||
"source": [
|
||
"The code below demonstrates CleanLearning with 2 different classifiers: `LogisticRegression()` and `GradientBoostingClassifier()`.\n",
|
||
"A `LogisticRegression` model is used to detect label issues (via cross-validation run inside CleanLearning) and a `GradientBoostingClassifier` model is finally trained on a clean subset of the data with issues removed.\n",
|
||
"This can be done with any two classifiers."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "4c9e9030",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab.classification import CleanLearning\n",
|
||
"import numpy as np\n",
|
||
"from sklearn.linear_model import LogisticRegression\n",
|
||
"from sklearn.ensemble import GradientBoostingClassifier\n",
|
||
"\n",
|
||
"# Make example data\n",
|
||
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
|
||
"labels = np.array([0] * 100 + [1] * 100)\n",
|
||
"\n",
|
||
"# Introduce label errors\n",
|
||
"true_errors = [97, 98, 100, 101, 102, 104]\n",
|
||
"for idx in true_errors:\n",
|
||
" labels[idx] = 1 - labels[idx]\n",
|
||
"\n",
|
||
"# CleanLearning with 2 different classifiers: one classifier is used to detect label issues \n",
|
||
"# and a different classifier is subsequently trained on the clean subset of the data.\n",
|
||
"\n",
|
||
"model_to_find_errors = LogisticRegression() # this model will be trained many times via cross-validation\n",
|
||
"model_to_return = GradientBoostingClassifier() # this model will be trained once on clean subset of data\n",
|
||
"\n",
|
||
"cl0 = CleanLearning(model_to_find_errors)\n",
|
||
"issues = cl0.find_label_issues(data, labels)\n",
|
||
"\n",
|
||
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues)\n",
|
||
"pred_probs = cl.predict_proba(data) # predictions from GradientBoostingClassifier\n",
|
||
"\n",
|
||
"print(cl0.clf) # will be LogisticRegression()\n",
|
||
"print(cl.clf) # will be GradientBoostingClassifier()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b71fef02",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How do I hyperparameter tune only the final model trained (and not the one finding label issues) in CleanLearning?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e7ec1956",
|
||
"metadata": {},
|
||
"source": [
|
||
"The code below demonstrates CleanLearning using a `GradientBoostingClassifier()` with no hyperparameter-tuning to find label issues but with hyperparameter-tuning via `RandomizedSearchCV(...)` for the final training of this model on the clean subset of the data.\n",
|
||
"This is a useful trick to avoid expensive hyperparameter-tuning for every fold of cross-validation (which is needed to find label issues)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "8751619e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"from cleanlab.classification import CleanLearning\n",
|
||
"from sklearn.ensemble import GradientBoostingClassifier\n",
|
||
"from sklearn.model_selection import RandomizedSearchCV\n",
|
||
"\n",
|
||
"# Make example data\n",
|
||
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
|
||
"labels = np.array([0] * 100 + [1] * 100)\n",
|
||
"\n",
|
||
"# Introduce label errors\n",
|
||
"true_errors = [97, 98, 100, 101, 102, 104]\n",
|
||
"for idx in true_errors:\n",
|
||
" labels[idx] = 1 - labels[idx]\n",
|
||
"\n",
|
||
"# CleanLearning with no hyperparameter-tuning during expensive cross-validation to find label issues\n",
|
||
"# but hyperparameter-tuning for the final training of model on clean subset of the data:\n",
|
||
"\n",
|
||
"model_to_find_errors = GradientBoostingClassifier() # this model will be trained many times via cross-validation\n",
|
||
"model_to_return = RandomizedSearchCV(GradientBoostingClassifier(),\n",
|
||
" param_distributions = {\n",
|
||
" \"learning_rate\": [0.001, 0.05, 0.1, 0.2, 0.5],\n",
|
||
" \"max_depth\": [3, 5, 10],\n",
|
||
" }\n",
|
||
" ) # this model will be trained once on clean subset of data\n",
|
||
"\n",
|
||
"cl0 = CleanLearning(model_to_find_errors)\n",
|
||
"issues = cl0.find_label_issues(data, labels)\n",
|
||
"\n",
|
||
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues) # CleanLearning for hyperparameter final training\n",
|
||
"pred_probs = cl.predict_proba(data) # predictions from hyperparameter-tuned GradientBoostingClassifier\n",
|
||
"\n",
|
||
"print(cl0.clf) # will be GradientBoostingClassifier()\n",
|
||
"print(cl.clf) # will be RandomizedSearchCV(estimator=GradientBoostingClassifier(),...)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d228decd",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Why does regression.learn.CleanLearning take so long?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "de5c984b",
|
||
"metadata": {},
|
||
"source": [
|
||
"To effectively identify errors in a regression dataset, the methods in [regression.learn.CleanLearning](../../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning) estimate each datapoint's aleatoric uncertainty (by fitting a second copy of the regression model to predict the residuals’ magnitudes), as well as its epistemic uncertainty (by fitting multiple copies of the regression model with bootstrap resampling). These uncertainty estimates help provide a robust quality score that accounts for the model's imperfect predictions. \n",
|
||
"\n",
|
||
"These uncertainty estimates help produce better results but require longer runtimes. Here are a few options to speed up the runtime of these methods:\n",
|
||
"\n",
|
||
"- Reduce the number of bootstrap resampling rounds by decreasing the `n_boot` argument (default value is 5, set it to 0 to skip the epistemic uncertainty estimation entirely).\n",
|
||
"\n",
|
||
"- Set `include_aleatoric_uncertainty=False` to skip the aleatoric uncertainty estimation.\n",
|
||
"\n",
|
||
"- Include less elements in the `coarse_search_range` argument of [regression.learn.CleanLearning.find_label_issues](../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning.find_label_issues). This is overall set of values initially considered for estimating the fraction of data that have label issues.\n",
|
||
"\n",
|
||
"- Reduce the `fine_search_size` argument of [regression.learn.CleanLearning.find_label_issues](../cleanlab/regression/learn.html#cleanlab.regression.learn.CleanLearning.find_label_issues). A higher number represents a more thorough search to precisely estimate the fraction of data that have label issues.\n",
|
||
"\n",
|
||
"Below is sample code on how to pass in these arguments."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "623df36d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab.regression.learn import CleanLearning\n",
|
||
"\n",
|
||
"X = np.random.random(size=(30, 3))\n",
|
||
"coefficients = np.random.uniform(-1, 1, size=3)\n",
|
||
"y = np.dot(X, coefficients) + np.random.normal(scale=0.2, size=30)\n",
|
||
"\n",
|
||
"# passing optinal arguments to reduce runtime\n",
|
||
"cl = CleanLearning(n_boot=1, include_aleatoric_uncertainty=False)\n",
|
||
"cl.find_label_issues(X, y, coarse_search_range=[0.05, 0.1], fine_search_size=2)\n",
|
||
"\n",
|
||
"# you can also pass coarse_search_range and fine_search_size as kwargs to CleanLearning.fit\n",
|
||
"cl.fit(X, y, find_label_issues_kwargs={\"coarse_search_range\": [0.05, 0.1], \"fine_search_size\": 2})"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1677ba25",
|
||
"metadata": {},
|
||
"source": [
|
||
"**With Datalab**:\n",
|
||
"\n",
|
||
"Datalab runs CleanLearning under the hood when looking for label issues in regression datasets. Here's how you can achieve the same behavior as calling `CleanLearning.find_label_issues()` in the code above using Datalab:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "af3052ac",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab import Datalab\n",
|
||
"\n",
|
||
"lab = Datalab(data = {\"X\": X, \"y\": y}, label_name = \"y\", task=\"regression\")\n",
|
||
"\n",
|
||
"issue_types = {\n",
|
||
" \"label\": {\n",
|
||
" \"clean_learning_kwargs\": {\"n_boot\": 1, \"include_aleatoric_uncertainty\": False},\n",
|
||
" \"coarse_search_range\": [0.05, 0.1],\n",
|
||
" \"fine_search_size\": 2,\n",
|
||
" },\n",
|
||
"}\n",
|
||
"lab.find_issues(features=X, issue_types = issue_types)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How do I specify pre-computed data slices/clusters when detecting the Underperforming Group Issue?"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The instructions for specifying pre-computed data slices/clusters when detecting underperforming groups in a dataset are now covered in detail in the Datalab workflows tutorial.\n",
|
||
"\n",
|
||
"- [Using clustering algorithms](./datalab/workflows.html#Find-Underperforming-Groups-in-a-Dataset).\n",
|
||
"- [Using categorical columns in a tabular dataset](./datalab/workflows.html#Predefining-Data-Slices-for-Detecting-Underperforming-Groups)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### How to handle near-duplicate data identified by Datalab?\n",
|
||
"\n",
|
||
"cleanlab may identify near-duplicate examples in your dataset, these are examples that are very similar to each other and can potentially cause issues in model training and analytics. When near-duplicates are present, models may unexpectedly emphasize these examples, especially if they were accidentally duplicated. In such cases, it is crucial to remove the (near) duplicate copies from your dataset to ensure accurate and reliable results. A common strategy is to remove all but one of the duplicates from your dataset. Here's how you can achieve this with results from cleanlab's `Datalab` class:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from typing import Callable\n",
|
||
"import pandas as pd\n",
|
||
"\n",
|
||
"\n",
|
||
"def merge_duplicate_sets(df, merge_key: str):\n",
|
||
" \"\"\"Generate group keys for each row, then merge intersecting sets.\n",
|
||
" \n",
|
||
" :param df: DataFrame with columns 'is_near_duplicate_issue' and 'near_duplicate_sets'\n",
|
||
" :param merge_key: Name of the column to store the merged sets\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" df[merge_key] = df.apply(construct_group_key, axis=1)\n",
|
||
" merged_sets = consolidate_sets(df[merge_key].tolist())\n",
|
||
" df[merge_key] = df[merge_key].map(\n",
|
||
" lambda x: next(s for s in merged_sets if x.issubset(s))\n",
|
||
" )\n",
|
||
" return df\n",
|
||
"\n",
|
||
"def construct_group_key(row):\n",
|
||
" \"\"\"Convert near_duplicate_sets into a frozenset and include the row's own index.\"\"\"\n",
|
||
" return frozenset(row['near_duplicate_sets']).union({row.name})\n",
|
||
"\n",
|
||
"def consolidate_sets(sets_list):\n",
|
||
" \"\"\"Merge sets if they intersect.\"\"\"\n",
|
||
" \n",
|
||
" # Convert the input list of frozensets to a list of mutable sets\n",
|
||
" sets_list = [set(item) for item in sets_list]\n",
|
||
" \n",
|
||
" # A flag to keep track of whether any sets were merged in the current iteration\n",
|
||
" merged = True\n",
|
||
"\n",
|
||
" # Continue the merging process as long as we have merged some sets in the previous iteration\n",
|
||
" while merged:\n",
|
||
" merged = False\n",
|
||
" new_sets = []\n",
|
||
"\n",
|
||
" # Iterate through each set in our list\n",
|
||
" for current_set in sets_list:\n",
|
||
" # Skip empty sets\n",
|
||
" if not current_set:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" # Find all sets that have an intersection with the current set\n",
|
||
" intersecting_sets = [s for s in sets_list if s & current_set]\n",
|
||
"\n",
|
||
" # If more than one set intersects, set the merged flag to True\n",
|
||
" if len(intersecting_sets) > 1:\n",
|
||
" merged = True\n",
|
||
"\n",
|
||
" # Merge all intersecting sets into one set\n",
|
||
" merged_set = set().union(*intersecting_sets)\n",
|
||
" new_sets.append(merged_set)\n",
|
||
"\n",
|
||
" # Empty the sets we've merged to prevent them from being processed again\n",
|
||
" for s in intersecting_sets:\n",
|
||
" sets_list[sets_list.index(s)] = set()\n",
|
||
"\n",
|
||
" # Replace the original sets list with the new list of merged sets\n",
|
||
" sets_list = new_sets\n",
|
||
"\n",
|
||
" # Convert the merged sets back to frozensets for the output\n",
|
||
" return [frozenset(item) for item in sets_list]\n",
|
||
"\n",
|
||
"def lowest_score_strategy(sub_df):\n",
|
||
" \"\"\"Keep the row with the lowest near_duplicate_score.\"\"\"\n",
|
||
" return sub_df['near_duplicate_score'].idxmin()\n",
|
||
"\n",
|
||
"\n",
|
||
"def filter_near_duplicates(data: pd.DataFrame, strategy_fn: Callable = lowest_score_strategy, **strategy_kwargs):\n",
|
||
" \"\"\"\n",
|
||
" Given a dataframe with columns 'is_near_duplicate_issue' and 'near_duplicate_sets',\n",
|
||
" return a series of boolean values where True indicates the rows to be removed.\n",
|
||
" The strategy_fn determines which rows to keep within each near_duplicate_set.\n",
|
||
"\n",
|
||
" :param data: DataFrame with is_near_duplicate_issue and near_duplicate_sets columns\n",
|
||
" :param strategy_fn: Function to determine which rows to keep within each near_duplicate_set\n",
|
||
" :return: Series of boolean values where True indicates rows to be removed.\n",
|
||
" \"\"\"\n",
|
||
" \n",
|
||
" # Filter out rows where 'is_near_duplicate_issue' is True to get potential duplicates\n",
|
||
" duplicate_rows = data.query(\"is_near_duplicate_issue\").copy()\n",
|
||
"\n",
|
||
" # Generate group keys for each row and merge intersecting sets\n",
|
||
" group_key = \"sets\"\n",
|
||
" duplicate_rows = merge_duplicate_sets(duplicate_rows, merge_key=group_key)\n",
|
||
"\n",
|
||
" # Use the strategy function to determine the indices of the rows to keep for each group\n",
|
||
" to_keep_indices = duplicate_rows.groupby(group_key).apply(strategy_fn, **strategy_kwargs).explode().values\n",
|
||
"\n",
|
||
" # Produce a boolean series indicating which rows should be removed\n",
|
||
" to_remove = ~data.index.isin(to_keep_indices)\n",
|
||
"\n",
|
||
" return to_remove"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The functions above collect sets of near-duplicate examples. Within each\n",
|
||
"collection, a single example is chosen to be kept in the dataset. The rest of the examples in the collection are removed.\n",
|
||
"Examples that are not near-duplicates of any other examples are kept in the dataset as well.\n",
|
||
"\n",
|
||
"The choice of which example to keep in each set of near-duplicate examples can be made in a variety of ways. Here, the example with the lowest near-duplicate score is chosen.\n",
|
||
"You can use any strategy that best suits your application by defining the strategy as a function and passing it as the `strategy_fn` argument to `filter_near_duplicates()`.\n",
|
||
"Below is an example of how this is applied to a dataset.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from cleanlab import Datalab\n",
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"# Assume you have a dataset with a set of 3 near-duplicate examples\n",
|
||
"features = np.random.random(size=(15, 3))\n",
|
||
"for neighbor in range(1, 3):\n",
|
||
" # Make examples 0, 1, and 2 near-duplicates of each other\n",
|
||
" features[neighbor] = features[0] + np.random.normal(scale=0.001, size=3)\n",
|
||
"\n",
|
||
"# Identify near-duplicate examples with Datalab\n",
|
||
"your_dataset = {\n",
|
||
" \"features\": features,\n",
|
||
"}\n",
|
||
"lab = Datalab(data=your_dataset)\n",
|
||
"lab.find_issues(features = features, issue_types={\"near_duplicate\": {}})\n",
|
||
"\n",
|
||
"# Pick out ids of near-duplicate examples to remove\n",
|
||
"near_duplicate_issues = (\n",
|
||
" lab.get_issues(\"near_duplicate\")\n",
|
||
" .query(\"is_near_duplicate_issue\")\n",
|
||
" .sort_values(\"near_duplicate_score\")\n",
|
||
")\n",
|
||
"ids_to_remove_series = filter_near_duplicates(near_duplicate_issues)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Near-duplicate examples to keep:\", np.where(~ids_to_remove_series)[0].tolist())\n",
|
||
"\n",
|
||
"print(\"Near-duplicate examples to remove:\", np.where(ids_to_remove_series)[0].tolist())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1520a93f",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Can't find an answer to your question?\n",
|
||
"\n",
|
||
"If your question is not addressed in these tutorials and API documentation pages, then refer to the [Github issues](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue) or [Code Examples](https://github.com/cleanlab/examples).\n",
|
||
"\n",
|
||
"If you still have a question, open a [new Github issue](https://github.com/cleanlab/cleanlab/issues/new/choose)."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.7"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
} |