wehub-resource-sync
549 lines
20 KiB
Plaintext
549 lines
20 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Detecting Issues in Tabular Data (Numeric/Categorical columns) with Datalab\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"In this 5-minute quickstart tutorial, we use Datalab to detect various issues in a classification dataset with tabular (numeric/categorical) features. Tabular (or *structured*) data are typically organized in a row/column format and stored in a SQL database or file types like: CSV, Excel, or Parquet. Here we consider a Student Grades dataset, which contains over 900 individuals who have three exam grades and some optional notes, each being assigned a letter grade (their class label). cleanlab automatically identifies _hundreds_ of examples in this dataset that were mislabeled with the incorrect final grade selected. You can run the same code from this tutorial to detect incorrect information in your own tabular classification datasets.\n",
|
||
"\n",
|
||
"**Overview of what we'll do in this tutorial:**\n",
|
||
"\n",
|
||
"- Train a classifier model (here scikit-learn's HistGradientBoostingClassifier, although any model could be used) and use this classifier to compute (out-of-sample) predicted class probabilities via cross-validation.\n",
|
||
"\n",
|
||
"- Create a K nearest neighbours (KNN) graph between the examples in the dataset.\n",
|
||
"\n",
|
||
"- Identify issues in the dataset with cleanlab's `Datalab` audit applied to the predictions and KNN graph.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<div class=\"alert alert-info\">\n",
|
||
"Quickstart\n",
|
||
"<br/>\n",
|
||
" \n",
|
||
"Already have (out-of-sample) `pred_probs` from a model trained on your original data labels? Have a `knn_graph` computed between dataset examples (reflecting similarity in their feature values)? Run the code below to find issues in your dataset.\n",
|
||
"\n",
|
||
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
|
||
" \n",
|
||
"```ipython3 \n",
|
||
"from cleanlab import Datalab\n",
|
||
"\n",
|
||
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
|
||
"lab.find_issues(pred_probs=your_pred_probs, knn_graph=knn_graph)\n",
|
||
"\n",
|
||
"lab.get_issues()\n",
|
||
"```\n",
|
||
" \n",
|
||
"</div>\n",
|
||
"</div>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. Install required dependencies\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"You can use `pip` to install all packages required for this tutorial as follows:\n",
|
||
"\n",
|
||
"```ipython3\n",
|
||
"!pip install \"cleanlab[datalab]\"\n",
|
||
"# Make sure to install the version corresponding to this tutorial\n",
|
||
"# E.g. if viewing master branch documentation:\n",
|
||
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Package installation (hidden on docs website).\n",
|
||
"dependencies = [\"cleanlab\", \"datasets\"]\n",
|
||
"\n",
|
||
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
|
||
" %pip install cleanlab # for colab\n",
|
||
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
|
||
" %pip install $cmd\n",
|
||
"else:\n",
|
||
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
|
||
" else dependency.split('<')[0] if '<' in dependency \n",
|
||
" else dependency.split('=')[0] for dependency in dependencies]\n",
|
||
" missing_dependencies = []\n",
|
||
" for dependency in dependencies_test:\n",
|
||
" try:\n",
|
||
" __import__(dependency)\n",
|
||
" except ImportError:\n",
|
||
" missing_dependencies.append(dependency)\n",
|
||
"\n",
|
||
" if len(missing_dependencies) > 0:\n",
|
||
" print(\"Missing required dependencies:\")\n",
|
||
" print(*missing_dependencies, sep=\", \")\n",
|
||
" print(\"\\nPlease install them before running the rest of this notebook.\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import random\n",
|
||
"import numpy as np\n",
|
||
"import pandas as pd\n",
|
||
"\n",
|
||
"from sklearn.model_selection import cross_val_predict\n",
|
||
"from sklearn.preprocessing import StandardScaler\n",
|
||
"from sklearn.ensemble import HistGradientBoostingClassifier\n",
|
||
"from sklearn.neighbors import NearestNeighbors\n",
|
||
"\n",
|
||
"from cleanlab import Datalab\n",
|
||
"\n",
|
||
"SEED = 100 # for reproducibility\n",
|
||
"np.random.seed(SEED)\n",
|
||
"random.seed(SEED)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Load and process the data\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We first load the data features and labels (which are possibly noisy).\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"grades_data = pd.read_csv(\"https://s.cleanlab.ai/grades-tabular-demo-v2.csv\")\n",
|
||
"grades_data.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"X_raw = grades_data[[\"exam_1\", \"exam_2\", \"exam_3\", \"notes\"]]\n",
|
||
"labels = grades_data[\"letter_grade\"]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next we preprocess the data. Here we apply one-hot encoding to columns with categorical values and standardize the values in numeric columns."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"cat_features = [\"notes\"]\n",
|
||
"X_encoded = pd.get_dummies(X_raw, columns=cat_features, drop_first=True)\n",
|
||
"\n",
|
||
"numeric_features = [\"exam_1\", \"exam_2\", \"exam_3\"]\n",
|
||
"scaler = StandardScaler()\n",
|
||
"X_processed = X_encoded.copy()\n",
|
||
"X_processed[numeric_features] = scaler.fit_transform(X_encoded[numeric_features])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<div class=\"alert alert-info\">\n",
|
||
"Bringing Your Own Data (BYOD)?\n",
|
||
"\n",
|
||
"Assign your data's features to variable `X` and its labels to variable `labels` instead.\n",
|
||
"\n",
|
||
"</div>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Select a classification model and compute out-of-sample predicted probabilities\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here we use a simple histogram-based gradient boosting model (similar to XGBoost), but you can choose any suitable scikit-learn model for this tutorial.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"clf = HistGradientBoostingClassifier()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"To find potential labeling errors, cleanlab requires a probabilistic prediction from your model for every datapoint. However, these predictions will be _overfitted_ (and thus unreliable) for examples the model was previously trained on. For the best results, cleanlab should be applied with **out-of-sample** predicted class probabilities, i.e., on examples held out from the model during the training.\n",
|
||
"\n",
|
||
"K-fold cross-validation is a straightforward way to produce out-of-sample predicted probabilities for every datapoint in the dataset by training K copies of our model on different data subsets and using each copy to predict on the subset of data it did not see during training. Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.\n",
|
||
"We can implement this via the `cross_val_predict` method from scikit-learn.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"num_crossval_folds = 5 \n",
|
||
"pred_probs = cross_val_predict(\n",
|
||
" clf,\n",
|
||
" X_processed,\n",
|
||
" labels,\n",
|
||
" cv=num_crossval_folds,\n",
|
||
" method=\"predict_proba\",\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. Construct K nearest neighbours graph"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The KNN graph reflects how close each example is when compared to other examples in our dataset (in the numerical space of preprocessed feature values). This similarity information is used by Datalab to identify issues like outliers in our data. For tabular data, think carefully about the most appropriate way to define the similarity between two examples.\n",
|
||
"\n",
|
||
"Here we use the `NearestNeighbors` class in sklearn to easily compute this graph (with similarity defined by the Euclidean distance between feature values). The graph should be represented as a sparse matrix with nonzero entries indicating nearest neighbors of each example and their distance."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"KNN = NearestNeighbors(metric='euclidean')\n",
|
||
"KNN.fit(X_processed.values)\n",
|
||
"\n",
|
||
"knn_graph = KNN.kneighbors_graph(mode=\"distance\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5. Use cleanlab to find label issues\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Based on the given labels, predicted probabilities, and KNN graph, cleanlab can quickly help us identify suspicious values in our grades table.\n",
|
||
"\n",
|
||
"We use cleanlab's `Datalab` class which has several ways of loading the data. In this case, we’ll simply wrap the dataset (features and noisy labels) in a dictionary that is used instantiate a `Datalab` object such that it can audit our dataset for various types of issues."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"data = {\"X\": X_processed.values, \"y\": labels}\n",
|
||
"\n",
|
||
"lab = Datalab(data, label_name=\"y\")\n",
|
||
"lab.find_issues(pred_probs=pred_probs, knn_graph=knn_graph)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"scrolled": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"lab.report()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Label issues\n",
|
||
"\n",
|
||
"The above report shows that cleanlab identified many label issues in the data. We can see which examples are estimated to be mislabeled (as well as a numeric quality score quantifying how likely their label is correct) via the `get_issues` method."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"issue_results = lab.get_issues(\"label\")\n",
|
||
"issue_results.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"To review the most severe label issues, sort the DataFrame above by the `label_score` column (a lower score represents that the label is less likely to be correct). \n",
|
||
"\n",
|
||
"Let's review some of the most likely label errors:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"sorted_issues = issue_results.sort_values(\"label_score\").index\n",
|
||
"\n",
|
||
"X_raw.iloc[sorted_issues].assign(\n",
|
||
" given_label=labels.iloc[sorted_issues], \n",
|
||
" predicted_label=issue_results[\"predicted_label\"].iloc[sorted_issues]\n",
|
||
").head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The dataframe above shows the original label (`given_label`) for examples that cleanlab finds most likely to be mislabeled, as well as an alternative `predicted_label` for each example.\n",
|
||
"\n",
|
||
"These examples have been labeled incorrectly and should be carefully re-examined - a student with grades of 89, 95 and 73 surely does not deserve a D! "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Outlier issues\n",
|
||
"\n",
|
||
"According to the report, our dataset contains some outliers. We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via `get_issues`. We sort the resulting DataFrame by cleanlab's outlier quality score to see the most severe outliers in our dataset."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"outlier_results = lab.get_issues(\"outlier\")\n",
|
||
"sorted_outliers= outlier_results.sort_values(\"outlier_score\").index\n",
|
||
"\n",
|
||
"X_raw.iloc[sorted_outliers].head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The student at index 3 has fractional exam scores, which is likely a error. We also see that the students at index 0 and 4 have numerical values in their notes section, which is also probably unintended. Lastly, we see that the student at index 8 has a html string in their notes section, definitely a mistake!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Near-duplicate issues\n",
|
||
"\n",
|
||
"According to the report, our dataset contains some sets of nearly duplicated examples.\n",
|
||
"We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by cleanlab's near-duplicate quality score to see the examples in our dataset that are most nearly duplicated."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"duplicate_results = lab.get_issues(\"near_duplicate\")\n",
|
||
"duplicate_results.sort_values(\"near_duplicate_score\").head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The results above show which examples cleanlab considers nearly duplicated (rows where `is_near_duplicate_issue == True`). Here, we see some examples that cleanlab has flagged as being nearly duplicated. Let's view these examples to see how similar they are\n",
|
||
"\n",
|
||
"Using the one of the lowest-scoring examples, let's compare it against the identified near-duplicate sets."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Identify the row with the lowest near_duplicate_score\n",
|
||
"lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].idxmin()\n",
|
||
"\n",
|
||
"# Extract the indices of the lowest scoring duplicate and its near duplicate sets\n",
|
||
"indices_to_display = [lowest_scoring_duplicate] + duplicate_results.loc[lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
|
||
"\n",
|
||
"# Display the relevant rows from the original dataset\n",
|
||
"X_raw.iloc[indices_to_display]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"These examples are exact duplicates! Perhaps the same information was accidentally recorded multiple times in this data."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Similarly, let's take a look at another example and the identified near-duplicate sets:\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Identify the next row not in the previous near duplicate set\n",
|
||
"second_lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].drop(indices_to_display).idxmin()\n",
|
||
"\n",
|
||
"# Extract the indices of the second lowest scoring duplicate and its near duplicate sets\n",
|
||
"next_indices_to_display = [second_lowest_scoring_duplicate] + duplicate_results.loc[second_lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
|
||
"\n",
|
||
"# Display the relevant rows from the original dataset\n",
|
||
"X_raw.iloc[next_indices_to_display]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We identified another set of exact duplicates in our dataset! Including near/exact duplicates in a dataset may have unintended effects on models; be wary about splitting them across training/test sets. Learn more about handling near duplicates detected in a dataset from [the FAQ](../faq.html#How-to-handle-near-duplicate-data-identified-by-cleanlab?)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"This tutorial highlighted a straightforward approach to detect potentially incorrect information in any tabular dataset. Just use Datalab with any ML model -- the better the model, the more accurate the data errors detected by Datalab will be!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"nbsphinx": "hidden"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
|
||
"\n",
|
||
"identified_label_issues = issue_results[issue_results[\"is_label_issue\"] == True]\n",
|
||
"label_issue_indices = [3, 723, 709, 886, 689] # check these examples were found in label issues\n",
|
||
"if not all(x in identified_label_issues.index for x in label_issue_indices):\n",
|
||
" raise Exception(\"Some highlighted examples are missing from identified_label_issues.\")\n",
|
||
" \n",
|
||
"identified_outlier_issues = outlier_results[outlier_results[\"is_outlier_issue\"] == True]\n",
|
||
"outlier_issue_indices = [3, 7, 0, 4, 8] # check these examples were found in outlier issues\n",
|
||
"if not all(x in identified_outlier_issues.index for x in outlier_issue_indices):\n",
|
||
" raise Exception(\"Some highlighted examples are missing from identified_outlier_issues.\")\n",
|
||
" \n",
|
||
"identified_duplicate_issues = duplicate_results[duplicate_results[\"is_near_duplicate_issue\"] == True]\n",
|
||
"duplicate_issue_indices = [690, 246, 185, 582] # check these examples were found in duplicate issues\n",
|
||
"if not all(x in identified_duplicate_issues.index for x in duplicate_issue_indices):\n",
|
||
" raise Exception(\"Some highlighted examples are missing from identified_duplicate_issues.\")\n",
|
||
" \n",
|
||
"# check that the near duplicates shown are actually flagged as near duplicate sets\n",
|
||
"if not duplicate_results.iloc[690][\"near_duplicate_sets\"] == 246:\n",
|
||
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
|
||
" \n",
|
||
"if not duplicate_results.iloc[185][\"near_duplicate_sets\"] == 582:\n",
|
||
" raise Exception(\"These examples are not in the same near duplicate set\")\n",
|
||
"\n",
|
||
"# Function to check if all rows are identical\n",
|
||
"def are_rows_identical(df):\n",
|
||
" first_row = df.iloc[0]\n",
|
||
" return all(df.iloc[i].equals(first_row) for i in range(1, len(df)))\n",
|
||
"\n",
|
||
"# Test to ensure all displayed rows are identical\n",
|
||
"if not are_rows_identical(X_raw.iloc[indices_to_display]):\n",
|
||
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")\n",
|
||
"\n",
|
||
"# Repeat the test for the next set of indices\n",
|
||
"if not are_rows_identical(X_raw.iloc[next_indices_to_display]):\n",
|
||
" raise Exception(\"Not all rows are identical! These examples should belong to the same EXACT duplicate set\")"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"interpreter": {
|
||
"hash": "cda20062bc42cfdcaa0f9720c0b28e880bba110e9dfce6c1689934eec9b595a1"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.10.12"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|