wehub/cleanlab--cleanlab
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:49:22 +08:00
commit a41b2ab474
303 changed files with 64772 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.editorconfig
+15
View File
@@ -0,0 +1,15 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
trim_trailing_whitespace = true
indent_size = 4
[*.ipynb]
insert_final_newline = false
[*.{yml,yaml}]
indent_size = 2
.github/ISSUE_TEMPLATE/1-bug-report.md
+32
View File
@@ -0,0 +1,32 @@
---
name: Bug Report
about: Use this template to report a bug.
title: "[Short summary of the bug]"
labels: needs triage
assignees: ''
---
<!-- Briefly summarize the issue. INCLUDE the exact code you are trying to run! Ideally in a self-contained script so we can reproduce your bug. -->
# Stack trace
<!-- If applicable, please include a full stack trace here. If you need to omit
the bottom of the stack trace (e.g. it includes stack frames from your private
code), that is okay. Try to include all cleanlab stack frames. -->
# Steps to reproduce
<!-- Be as detailed as possible here. If possible, include a self-contained
runnable example that demonstrates the issue. Remember to supply any data
necessary to run your example, or construct your example with synthetic data.
This is not strictly required, but the more detailed your bug report, the more
quickly we can help you and fix the bug. -->
# Additional information
- **Cleanlab version**: <!-- `cleanlab.__version__`, or the git commit hash if you're using an unreleased version -->
- **Operating system**: <!-- e.g. macOS 12.1, Ubuntu 20.04, Windows 10 -->
- **Python version**: <!-- you can find this with `python --version` -->
<!-- Please include any other information that could be helpful for debugging. -->
.github/ISSUE_TEMPLATE/2-feature-request.md
+26
View File
@@ -0,0 +1,26 @@
---
name: Feature Request
about: Use this template to ask for new functionality in cleanlab.
title: "[Short summary of the feature]"
labels: needs triage
assignees: ''
---
<!-- Briefly summarize the proposed feature. -->
# Details
<!--
Describe your proposed feature in more detail. INCLUDE the exact code you are trying to run or would like to see, if your question is related to code. Answer any of the following
questions that you can:
* What is the problem you're trying to solve?
* What tasks or workflows would be enabled by having support for your
proposed feature in cleanlab?
* Can you share code snippets or pseudocode describing uses of your feature?
* Can you share any datasets that can help us assess the usefulness of the
proposed feature?
* Have you considered any alternatives to your proposed feature/design?
* How are you working around the lack of native support for your proposed
feature?
-->
.github/ISSUE_TEMPLATE/3-help.md
+17
View File
@@ -0,0 +1,17 @@
---
name: Help
about: Use this template to ask for help.
title: "[Short summary of the question]"
labels: question
assignees: ''
---
<!--
Please be as detailed as possible in your question. INCLUDE the exact code you are trying to run, if your question is related to code.
We will answer questions posted here, but you will likely get an answer faster
by posting in our Slack Community:
https://cleanlab.ai/slack
-->
.github/ISSUE_TEMPLATE/4-other.md
+10
View File
@@ -0,0 +1,10 @@
---
name: Other
about: Use this template when no other template applies.
title: "[Short summary of the issue]"
labels: needs triage
assignees: ''
---
<!-- Please be as detailed as possible in your issue. INCLUDE the exact code you are trying to run, if your question is related to code. -->
.github/get_min_dependencies.py
+16
View File
@@ -0,0 +1,16 @@
"""This script fetches minimum dependencies of cleanlab package and writes them to the file requirements-min.txt"""
import json
if __name__ == "__main__":
with open("./deps.json", "r") as f:
deps = json.load(f)
for package in deps:
if package["package"]["package_name"] == "cleanlab":
for dep in package["dependencies"]:
req_version = dep["required_version"]
with open("requirements-min.txt", "a") as f:
if req_version.startswith(">="):
f.write(f"{dep['package_name']}=={req_version[2:]}\n")
.github/pull_request_template.md
+82
View File
@@ -0,0 +1,82 @@
## Summary
> 🎯 **Purpose**: Describe the objective of your changes in this Pull-Request.
>
> 📜 **Example Usage**: If adding new functionality, strive to provide a minimal working example (MWE) that demonstrates end-to-end usage and output of your new methods. If creating a MWE is not feasible, a conceptual code snippet showcasing the usage is also appreciated. Ensure that the code provided can be copied, pasted, and run without modifications.
**[ ✏️ Write your summary here. ]**
```python
# Example code snippet
# Necessary Imports:
from my_module import my_new_function
# Setup if needed (e.g., data loading, model instantiation):
a = 20
b = 22
# Usage and Output:
result = my_new_function(a, b)
# 42
# ---- Your code below ----
# [Replace the example above with your own code, ensuring it is executable and demonstrates your feature or fix.]
[your_uncommented_runnable_code_here]
# [your_output_here_as_a_comment]
```
## Impact
> 🌐 Areas Affected: Enumerate modules, functions, or documentation impacted by your code.
>
> 👥 Who’s Affected: Mention who might be impacted (if applicable).
**Screenshots**
> 📸 If your changes modify documentation or a tutorial notebook, please include screenshots or GIFs showing the rendered outputs for a quicker initial review, highlighting the main changes.
## Testing
> 🔍 Testing Done: Outline what kinds of tests you performed.
>
> 🔗 Test Case Link: Directly link to the most end-to-end check of your new functionality.
**Unaddressed Cases**
> It's ok if your unit tests are not yet
> comprehensive when you first open the PR,
> we can revisit them later!
>
> ⚠️ Mention any aspects of testing that have *not* been covered, and why.
-
## Links to Relevant Issues or Conversations
> 🔗 What Git or Slack items (Issues, threads, etc) that are specifically related to
> this work? Please link them here.
-
## References
> 📚 Include any extra information that may be helpful to reviewers of your
> Pull-Request here (e.g. relevant URLs or Wiki pages that could help give
> background information).
>
> Share links, docs, or sources that facilitated your changes.
>
> If relevant, please include additional code snippets
> (and their outputs/return values) showing alternative ways to use your newly added methods.
```python
[additional code snippets and explanations]
```
-
## Reviewer Notes
> 💡 Include any specific points for the reviewer to consider during their review.
.github/workflows/bleeding-edge.yml
+35
View File
@@ -0,0 +1,35 @@
name: Bleeding Edge Dependencies Test
on:
schedule:
- cron: '0 6 1 * *' # First day of each month at 6:00 UTC
workflow_dispatch: # Allow manual triggering
jobs:
test-bleeding-edge:
name: "Bleeding Edge Dependencies: Python Latest"
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v6
- uses: snok/latest-python-versions@v1
id: python-versions
with:
min-version: '3.14'
include-prereleases: false
- uses: actions/setup-python@v6
with:
python-version: ${{ fromJson(steps.python-versions.outputs.latest-python-versions)[0] }}
- name: Download cleanvision test data
run: |
git clone https://github.com/cleanlab/assets.git
mv assets/cleanlab_test_data ./tests/datalab/data
- name: Install bleeding-edge dependencies
run: |
python -m pip install --upgrade pip
pip install --pre --upgrade numpy scikit-learn pandas tqdm termcolor
pip install --pre torch torchvision || pip install torch torchvision
pip install -e ".[all]"
- name: Install test dependencies
run: pip install --no-cache-dir -r requirements-test-core.txt
- name: Test with bleeding-edge deps
run: pytest --verbose --maxfail=10 -m "not slow" --ignore=tests/test_frameworks.py --ignore=tests/test_model_pytorch_cnn.py
.github/workflows/ci.yml
+165
View File
@@ -0,0 +1,165 @@
name: CI
on:
push:
pull_request:
schedule:
- cron: '0 12 * * 1' # Monday at 12:00 UTC
jobs:
test-core:
name: "Core Tests: Python ${{ matrix.python }} on ${{ matrix.os }}"
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-latest
- macos-latest
- windows-latest
python:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "3.14"
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python }}
- name: Download cleanvision test data
run: |
git clone https://github.com/cleanlab/assets.git
mv assets/cleanlab_test_data ./tests/datalab/data
- name: Install cleanlab
run: |
python -m pip install --upgrade pip
pip install -e ".[all]"
- name: Install core test dependencies
run: pip install --no-cache-dir -r requirements-test-core.txt
- name: Audit dependencies for vulnerabilities
uses: pypa/gh-action-pip-audit@v1.1.0
with:
inputs: requirements-test-core.txt
- name: Run core tests (excluding ML framework tests)
run: pytest --verbose --maxfail=30 --order-tests --cov=cleanlab/ --cov-config .coveragerc --cov-report=xml -m "not slow" --ignore=tests/test_frameworks.py --ignore=tests/test_model_pytorch_cnn.py
- uses: codecov/codecov-action@v5
with:
version: "v0.7.3"
test-ml-frameworks:
name: "ML Framework Tests: Python ${{ matrix.python }} on ubuntu-latest"
runs-on: ubuntu-latest
strategy:
matrix:
python:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "3.14"
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python }}
- name: Download cleanvision test data
run: |
git clone https://github.com/cleanlab/assets.git
mv assets/cleanlab_test_data ./tests/datalab/data
- name: Install cleanlab
run: |
python -m pip install --upgrade pip
pip install -e ".[all]"
- name: Install ML framework dependencies
run: pip install --no-cache-dir -r requirements-dev.txt
- name: Audit dependencies for vulnerabilities
uses: pypa/gh-action-pip-audit@v1.1.0
with:
inputs: requirements-dev.txt
- name: Run ML framework tests
run: pytest --verbose --maxfail=30 tests/test_frameworks.py tests/test_model_pytorch_cnn.py
test-without-extras-min-versions:
name: Test without optional dependencies and with minimum compatible versions of dependencies
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: '3.10'
- name: Install cleanlab
run: |
python -m pip install --upgrade pip
pip install .
- name: Install test dependencies
run: |
pip install pytest pipdeptree hypothesis
pipdeptree -j > deps.json
- name: Install minimum versions
run: |
python ./.github/get_min_dependencies.py
pip install -r requirements-min.txt
- name: Run tests
run: |
pytest tests/test_multilabel_classification.py tests/test_multiannotator.py tests/test_filter_count.py
test-max-versions:
name: "Maximum Dependency Versions: Python 3.14"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Download cleanvision test data
run: |
git clone https://github.com/cleanlab/assets.git
mv assets/cleanlab_test_data ./tests/datalab/data
- name: Install latest stable versions
run: |
python -m pip install --upgrade pip
pip install --upgrade numpy scikit-learn pandas tqdm termcolor
pip install --upgrade torch torchvision
pip install -e ".[all]"
- name: Install test dependencies
run: pip install --no-cache-dir -r requirements-test-core.txt
- name: Test with latest stable versions
run: pytest --verbose --maxfail=10 -m "not slow" --ignore=tests/test_frameworks.py --ignore=tests/test_model_pytorch_cnn.py
typecheck:
name: Type check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: '3.14'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install ".[all]" # install dependencies
pip install --no-cache-dir -r requirements-test-core.txt # install core dependencies and type stubs
- name: Type check
run: mypy --install-types --non-interactive cleanlab
fmt:
name: Format
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: psf/black@stable
flake8:
name: Check for unused/wildcard imports
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: '3.14'
- name: Install flake8
run: pip install flake8
- name: Lint with flake8
run: flake8 cleanlab tests
nblint:
name: Lint Notebooks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: cleanlab/nblint-action@v1
with:
directory: 'docs'
.github/workflows/gh-pages.yaml
+210
View File
@@ -0,0 +1,210 @@
name: GitHub Pages
on:
push:
branches:
- "**"
release:
types: [published]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-24.04
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Install Pandoc
# pin Pandoc version
run: |
cd /tmp
wget https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-linux-amd64.tar.gz
sudo tar xzvf pandoc-2.19.2-linux-amd64.tar.gz --strip-components 1 -C /usr/local
- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: "3.11"
- name: Setup Node
uses: actions/setup-node@v6
with:
node-version: "22"
- name: Install FFmpeg
run: |
sudo apt-get update
sudo apt-get install -y ffmpeg
- name: Upgrade pip
run: python3 -m pip install --upgrade pip
# Caching removed to resolve disk space issues
# Heavy ML packages (TensorFlow, PyTorch, HuggingFace) will be downloaded fresh each run
# but this avoids the 3.7GB cache that causes "No space left on device" errors
- name: Install cleanlab
run: pip install -e ".[all]"
- name: Install dependencies
run: python3 -m pip install -r docs/requirements.txt
- name: Install katex
run: npm install -g katex
- name: Find and replace "%pip install cleanlab" with master branch in .ipynb files
if: ${{ github.ref_type == 'branch' }}
uses: jacobtomlinson/gha-find-replace@v2
with:
find: "%pip install cleanlab ${{ '#' }} for colab"
replace: "%pip install git+https://github.com/${{ github.repository }}.git@${{ github.sha }}"
include: "docs/source**.ipynb"
- name: Find and replace "%pip install cleanlab" with release tag in .ipynb files
if: ${{ github.ref_type == 'tag' }}
uses: jacobtomlinson/gha-find-replace@v2
with:
find: "%pip install cleanlab ${{ '#' }} for colab"
replace: "%pip install cleanlab==${{ github.ref_name }}"
include: "docs/source**.ipynb"
- name: Add and Commit changes to .ipynb files
run: |
git config --global user.email "actions@github.com"
git config --global user.name "GitHub Actions"
git add -A && git commit -m "Change cleanlab ver in .ipynb files"
- name: Change tag to the latest commit
if: ${{ github.ref_type == 'tag' }}
run: |
git tag -f $GITHUB_REF_NAME
- name: Run Sphinx with sphinx-multiversion wrapper - branch trigger
if: ${{ github.ref_type == 'branch' }}
env:
TF_CPP_MIN_LOG_LEVEL: 3
HF_HOME: /tmp/hf_cache
HF_HUB_DISABLE_SYMLINKS_WARNING: 1
TRANSFORMERS_CACHE: /tmp/transformers
TORCH_HOME: /tmp/torch
run: sphinx-multiversion docs/source cleanlab-docs -D smv_branch_whitelist=${{ github.ref_name }} -D smv_tag_whitelist=None
- name: Run Sphinx with sphinx-multiversion wrapper - tag trigger
if: ${{ github.ref_type == 'tag' }}
env:
TF_CPP_MIN_LOG_LEVEL: 3
HF_HOME: /tmp/hf_cache
HF_HUB_DISABLE_SYMLINKS_WARNING: 1
TRANSFORMERS_CACHE: /tmp/transformers
TORCH_HOME: /tmp/torch
run: sphinx-multiversion docs/source cleanlab-docs -D smv_branch_whitelist=None -D smv_tag_whitelist=${{ github.ref_name }}
- name: Debug temporary cache usage (not saved)
run: |
echo "=== Temporary Cache Usage ==="
echo "These caches are created in /tmp and cleaned up after build:"
echo " /tmp/hf_cache: $(du -sh /tmp/hf_cache 2>/dev/null | cut -f1 || echo 'not found')"
echo " /tmp/transformers: $(du -sh /tmp/transformers 2>/dev/null | cut -f1 || echo 'not found')"
echo " /tmp/torch: $(du -sh /tmp/torch 2>/dev/null | cut -f1 || echo 'not found')"
echo "Pip cache directory: $(pip cache dir)"
echo "Pip cache size: $(du -sh $(pip cache dir) 2>/dev/null | cut -f1 || echo 'not found')"
echo "========================"
- name: Get latest stable release
id: stable_release
continue-on-error: true
uses: pozetroninc/github-action-get-latest-release@master
with:
owner: ${{ github.repository_owner }}
repo: cleanlab
excludes: prerelease, draft
- name: Create latest_release.txt file
if: ${{ steps.stable_release.outcome == 'success' }}
run: echo ${{ steps.stable_release.outputs.release }} > cleanlab-docs/latest_release.txt
- name: Find and replace "placeholder_version_number" in versioning.js
if: ${{ steps.stable_release.outcome == 'success' }}
uses: jacobtomlinson/gha-find-replace@v2
with:
find: "placeholder_version_number"
replace: ${{ steps.stable_release.outputs.release }}
include: "docs/source/_templates/versioning.js"
- name: Find and replace "placeholder_commit_hash" in versioning.js
uses: jacobtomlinson/gha-find-replace@v2
with:
find: "placeholder_commit_hash"
replace: ${{ github.sha }}
include: "docs/source/_templates/versioning.js"
- name: Copy versioning JS file to cleanlab-docs/
run: |
cp docs/source/_templates/versioning.js cleanlab-docs/versioning.js
- name: Find and replace "stable_url" in redirect-to-stable.html
if: ${{ github.ref_type == 'tag' && steps.stable_release.outcome == 'success' }}
uses: jacobtomlinson/gha-find-replace@v2
with:
find: "stable_url"
replace: ${{ env.DOCS_SITE_URL }}stable/index.html
include: "docs/source/_templates/redirect-to-stable.html"
- name: Copy redirecting HTML file to cleanlab-docs/
if: ${{ github.ref_type == 'tag' }}
run: cp docs/source/_templates/redirect-to-stable.html cleanlab-docs/index.html
- name: Update OpenGraph URLs in HTML files
# This code fixes the URLs in the HTML files with OpenGraph tags.
# It uses a regular expression to capture two parts of the URL:
# $1: The first part of the URL, including the protocol and domain.
# $2: The rest of the URL, including the path and query string.
# The replacement string combines the captured parts with the ref name.
# This ensures that the ref name is inserted into the URL correctly.
env:
REF_URL_SEGMENT: ${{ (github.ref_type == 'tag' && steps.stable_release.outcome == 'success') && 'stable' || github.ref_name }}
uses: jacobtomlinson/gha-find-replace@v2
with:
find: '(<meta property="og:url" content="https?://[^/]+)/([^"]+")'
replace: '$1/${{ env.REF_URL_SEGMENT }}/$2'
include: "cleanlab-docs/**/*.html"
regex: true
- name: Deploy
if: ${{ (github.ref == 'refs/heads/master') || (github.ref_type == 'tag') }}
uses: peaceiris/actions-gh-pages@v4
with:
deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
external_repository: ${{ github.repository_owner }}/cleanlab-docs
publish_branch: master
publish_dir: cleanlab-docs
keep_files: true
exclude_assets: ""
- uses: actions/upload-artifact@v5
with:
name: docs-html
path: cleanlab-docs
retention-days: 1 # don't need this except in link checking step below
links:
needs: deploy
runs-on: ubuntu-24.04
steps:
- uses: actions/download-artifact@v5
with:
name: docs-html
path: cleanlab-docs
- uses: anishathalye/proof-html@v1
with:
directory: ./cleanlab-docs
external_only: true
url_ignore_re: |
^https://pradyunsg.me/
^https://keras.io/
.github/workflows/links.yml
+35
View File
@@ -0,0 +1,35 @@
name: GitHub Markdown Links
on:
push:
pull_request:
schedule:
- cron: "0 8 * * 6"
jobs:
test:
runs-on: ubuntu-latest
steps:
- run: sudo apt-get update -y
- run: >-
sudo apt-get install -y
pandoc
- uses: actions/checkout@v6
- run: |
find . -name '*.html' -delete
- run: |
find . -name '*.md' -exec pandoc -i {} -o {}.html \;
- uses: anishathalye/proof-html@v2
with:
directory: .
check_html: false
check_favicon: false
ignore_missing_alt: true
tokens: |
{"https://github.com": "${{ secrets.GITHUB_TOKEN }}"}
ignore_url: |
https://jair.org/index.php/jair/article/view/12125
https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe
ignore_url_re: |
https://www.gnu.org/software/wget
swap_urls: |
{"^(\\..*)\\.md(#?.*)$": "\\1.md.html\\2",
"^(https://github\\.com/.*)#.*$": "\\1"}
.github/workflows/manual-testpypi.yml
+154
View File
@@ -0,0 +1,154 @@
name: Manual TestPyPI Release
on:
workflow_dispatch: # Manual trigger only
inputs:
reason:
description: 'Reason for this test release'
required: false
default: 'Testing changes before official release'
jobs:
extract_project_name:
runs-on: ubuntu-latest
outputs:
cleanlab_package_name: ${{ steps.extract_name.outputs.value }}
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: TOML Reader
uses: SebRollen/toml-action@v1.2.0
id: extract_name
with:
file: "pyproject.toml"
field: "project.name"
build:
needs: extract_project_name
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Remove tests directory
run: |
TEST_DIR=$(find . -type d -name "tests")
if [ -n "$TEST_DIR" ]; then
rm -rf $TEST_DIR
else
echo "No tests directory found! This is unexpected."
exit 1
fi
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Modify version for TestPyPI
run: |
# Add .dev suffix with run number for TestPyPI
sed -i "s/__version__ = \"\(.*\)\"/__version__ = \"\1.dev${{ github.run_number }}\"/" cleanlab/version.py
echo "Modified version for TestPyPI:"
grep "__version__" cleanlab/version.py
- name: Install dependencies
run: pip install --upgrade "setuptools>=65.0,<70.0" wheel twine==5.0.0 pkginfo
- name: Clean build artifacts
run: |
rm -rf build/ dist/ *.egg-info cleanlab.egg-info/
- name: Perform some tests on the metadata of the package
run: |
python setup.py check --metadata --strict
- name: Build package
run: |
python setup.py sdist bdist_wheel
- name: Test package
run: |
twine check dist/*
- name: Display package info
run: |
echo "📦 Built packages:"
ls -lh dist/
echo ""
echo "📝 Reason: ${{ github.event.inputs.reason }}"
- name: Store build artifacts
uses: actions/upload-artifact@v5
with:
name: package
path: dist/*
publish-testpypi:
needs: build
runs-on: ubuntu-latest
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
steps:
- name: Fetch build artifacts
uses: actions/download-artifact@v5
with:
name: package
path: dist/
- name: Publish to TestPyPI
uses: pypa/gh-action-pypi-publish@e53eb8b103ffcb59469888563dc324e3c8ba6f06
with:
repository-url: https://test.pypi.org/legacy/
verify-metadata: true
test-basic-import:
needs: [publish-testpypi, extract_project_name]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package from TestPyPI
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ $CLEANLAB_PACKAGE_NAME
- name: Test importing package
run: |
python -c "import cleanlab
from cleanlab.outlier import OutOfDistribution
ood = OutOfDistribution()
print(ood)
print('✅ Basic import test passed!')
print('Version:', cleanlab.__version__)"
test-extras-import:
needs: [publish-testpypi, extract_project_name]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package with extras
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ $CLEANLAB_PACKAGE_NAME[all]
- name: Test importing package with extras
run: |
python -c "import cleanlab
from cleanlab import Datalab
lab = Datalab({'a': [1, 2, 3, 4, 5], 'b': ['a', 'b', 'c', 'b', 'a']})
print(lab)
print('✅ Extras import test passed!')
print('Version:', cleanlab.__version__)"
.github/workflows/release-build-publish.yml
+260
View File
@@ -0,0 +1,260 @@
name: Release Build Publish
on:
release:
types: [ published ]
jobs:
extract_project_name:
runs-on: ubuntu-latest
outputs:
cleanlab_package_name: ${{ steps.extract_name.outputs.value }}
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: TOML Reader
uses: SebRollen/toml-action@v1.2.0
id: extract_name
with:
file: "pyproject.toml"
field: "project.name"
store_published_version:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.compare.outputs.version }}
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package
run: pip install .
- name: Compare the published version with the package version
id: compare
run: |
TAG_NAME="${{ github.ref_name }}"
# TAG_NAME must be in the form "v1.2.3[...]", error if it doesn't match
if [[ ! $TAG_NAME =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
echo "Tag name '$TAG_NAME' does not match the expected pattern 'vX.Y.Z'"
exit 1
fi
# Strip the 'v' prefix from the tag, and store the version in an output variable
VERSION=${TAG_NAME:1} # Strip the 'v' prefix from the tag
echo "version=$VERSION" >> $GITHUB_OUTPUT
# Check that the package version matches the tag version
PACKAGE_VERSION=$(python -c "import cleanlab; print(cleanlab.__version__)")
if [ "$PACKAGE_VERSION" != "$VERSION" ]; then
echo "Package version $PACKAGE_VERSION does not match tag $VERSION"
exit 1
fi
build:
needs: [store_published_version, extract_project_name]
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Remove tests directory
run: |
# Remove the tests directory from the package
TEST_DIR=$(find . -type d -name "tests")
if [ -n "$TEST_DIR" ]; then
rm -rf $TEST_DIR
else
echo "No tests directory found! This is unexpected."
exit 1
fi
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install dependencies
run: pip install --upgrade "setuptools>=65.0,<70.0" wheel twine==5.0.0 pkginfo
- name: Clean build artifacts
run: |
rm -rf build/ dist/ *.egg-info cleanlab.egg-info/
- name: Perform some tests on the metadata of the package
run: |
python setup.py check --metadata --strict
- name: Build package
run: |
python setup.py sdist bdist_wheel
- name: Test package
run: |
twine check dist/*
- name: Verify contents of the tar.gz
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: |
# Extract the tar.gz to a temporary directory
TMPDIR=$(mktemp -d)
tar -xzf dist/*.tar.gz -C $TMPDIR
PACKAGE_DIR=$TMPDIR/$(ls $TMPDIR) # Assuming there's only one directory extracted
# Define expected files and directories
EXPECTED_FILES="LICENSE PKG-INFO MANIFEST.in DEVELOPMENT.md CONTRIBUTING.md CODE_OF_CONDUCT.md README.md pyproject.toml setup.cfg setup.py"
EXPECTED_DIRS="cleanlab $CLEANLAB_PACKAGE_NAME.egg-info"
# Collect actual top-level files and directories
ACTUAL_CONTENTS=$(ls $PACKAGE_DIR)
ACTUAL_FILES=$(find $PACKAGE_DIR -maxdepth 1 -type f -exec basename {} \;)
ACTUAL_DIRS=$(find $PACKAGE_DIR -maxdepth 1 -type d -exec basename {} \; | sed "1d") # Remove the package directory itself from the list
# Function to check for unexpected or missing files and directories
check_contents() {
local missing=0
# Check for expected files
for expected_file in $EXPECTED_FILES; do
echo $ACTUAL_FILES | grep -qw $expected_file || { echo "Missing expected file: $expected_file"; missing=1; }
done
# Check for expected directories
for expected_dir in $EXPECTED_DIRS; do
echo $ACTUAL_DIRS | grep -qw $expected_dir || { echo "Missing expected directory: $expected_dir"; missing=1; }
done
# Check for unexpected files and directories
for actual in $ACTUAL_CONTENTS; do
echo $EXPECTED_FILES $EXPECTED_DIRS | grep -qw $actual || { echo "Unexpected item in package: $actual"; missing=1; }
done
# Exit if anything unexpected or missing
if [ $missing -ne 0 ]; then
echo "Package content verification failed."
exit 1
fi
}
# Execute checks
check_contents
- name: Store build artifacts
uses: actions/upload-artifact@v5
with:
name: package
path: dist/*
publish-testpypi:
needs: build
runs-on: ubuntu-latest
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
steps:
- name: Fetch build artifacts
uses: actions/download-artifact@v5
with:
name: package
path: dist/
- name: Publish to TestPyPI
uses: pypa/gh-action-pypi-publish@e53eb8b103ffcb59469888563dc324e3c8ba6f06
with:
repository-url: https://test.pypi.org/legacy/
verify-metadata: true
verify-version:
needs: [publish-testpypi, store_published_version, extract_project_name]
runs-on: ubuntu-latest
environment: testpypi
steps:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package from TestPyPI
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ $CLEANLAB_PACKAGE_NAME
- name: Verify the published version
env:
VERSION: ${{ needs.store_published_version.outputs.version }}
run: |
TEST_IMPORT_VERSION=$(python -c "import cleanlab; print(cleanlab.__version__)")
if [ "$TEST_IMPORT_VERSION" != "$VERSION" ]; then
echo "Imported version $TEST_IMPORT_VERSION does not match tag $VERSION"
exit 1
fi
test-basic-import:
needs: [verify-version, extract_project_name]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package from TestPyPI
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ $CLEANLAB_PACKAGE_NAME
- name: Test importing package
run: |
python -c "import cleanlab
from cleanlab.outlier import OutOfDistribution
ood = OutOfDistribution()
print(ood)"
test-extras-import:
needs: [verify-version, extract_project_name]
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.11
- name: Install package with extras
env:
CLEANLAB_PACKAGE_NAME: ${{ needs.extract_project_name.outputs.cleanlab_package_name }}
run: pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ $CLEANLAB_PACKAGE_NAME[all]
- name: Test importing package with extras
run: |
python -c "import cleanlab
from cleanlab import Datalab
lab = Datalab({'a': [1, 2, 3, 4, 5], 'b': ['a', 'b', 'c', 'b', 'a']})
print(lab)"
publish-pypi:
needs: [test-basic-import, test-extras-import]
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
steps:
- name: Fetch build artifacts
uses: actions/download-artifact@v5
with:
name: package
path: dist/
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@e53eb8b103ffcb59469888563dc324e3c8ba6f06
with:
verify-metadata: true
.gitignore
+159
View File
@@ -0,0 +1,159 @@
cleanlab/*.ipynb
.DS_Store
*/.DS_Store
tests/fasttext_data/
data/
docs/_build/
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
.coverage*
*.py,cover
.hypothesis/**
.pytest_cache/**
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
logs/
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
build
_build
cleanlab-docs
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
.direnv
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
# Downloaded datasets for docs
/docs/source/notebooks/aclImdb
/docs/source/notebooks/data
/docs/source/notebooks/*.gz
/docs/source/notebooks/spoken_digits
/docs/source/notebooks/pretrained_models
/docs/source/tutorials/datalab/datalab-files/
# Editor files
.vscode/
.idea/
# jupyter notebooks
.ipynb_checkpoints/
# Misc
results/
image_files*
# datasets
cifar*
# IPython
profile_default/
ipython_config.py
__pypackages__/
# Data files
**.zip
**-ubyte
**.gz
**.tar
**.7Z
**.BZ2
**.rar
**.zarr
**.npy
.mypy.ini
+32
View File
@@ -0,0 +1,32 @@
[mypy]
allow_redefinition = True
[mypy-sklearn.*]
ignore_missing_imports = True
[mypy-scikeras.*]
ignore_missing_imports = True
[mypy-tensorflow.*]
ignore_missing_imports = True
[mypy-IPython.display.*]
ignore_missing_imports = True
[mypy-torch.*]
ignore_missing_imports = True
[mypy-torchvision.*]
ignore_missing_imports = True
[mypy-tqdm.*]
ignore_missing_imports = True
[mypy-matplotlib.*]
ignore_missing_imports = True
[mypy-datasets.*]
ignore_missing_imports = True
[mypy-scipy.*]
ignore_missing_imports = True
.pre-commit-config.yaml
+5
View File
@@ -0,0 +1,5 @@
repos:
- repo: https://github.com/psf/black
rev: 25.12.0
hooks:
- id: black
CODE_OF_CONDUCT.md
+131
View File
@@ -0,0 +1,131 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Providing specific details to ensure your comments are easy to understand
* Valuing other people's time
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Failing to properly attribute original sources
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
team@cleanlab.ai.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
CONTRIBUTING.md
+59
View File
@@ -0,0 +1,59 @@
# Contributing to cleanlab
All kinds of contributions to cleanlab are greatly appreciated. If you're not looking to write code, submitting a [feature request](#feature-requests) or
[bug report](#bug-reports) is a great way to contribute. If you want to get
your hands dirty, you can submit [Pull Requests](#pull-requests), either working on your
own ideas or addressing [existing issues][issues].
If you are unsure or confused about anything, please go ahead and submit your
issue or pull request anyways! We appreciate all contributions, and we'll do
our best to incorporate your feedback or code into cleanlab.
For some [ideas on useful contributions](https://github.com/cleanlab/cleanlab/wiki#ideas-for-contributing-to-cleanlab), look for [Issues](https://github.com/cleanlab/cleanlab/issues) or [Projects](https://github.com/cleanlab/cleanlab/projects) with the "good first issue" tag. Join our [Slack Community](https://cleanlab.ai/slack) to discuss other ideas!
Detailed contributing instructions can be found in the [Development Guide](DEVELOPMENT.md), please read this carefully!
## Feature Requests
Do you have an idea for an awesome new feature for cleanab? Let us know by
[submitting a feature request][issue].
If you are inclined to do so, you're welcome to [fork][fork] cleanlab, work on
implementing the feature yourself, and submit a patch. In this case, it's
*highly recommended* that you first [open an issue][issue] describing your
enhancement to get early feedback on the new feature that you are implementing.
This will help avoid wasted efforts and ensure that your work is incorporated
into the cleanlab code base.
## Bug Reports
Did something go wrong with cleanlab? Sorry about that! Bug reports are greatly
appreciated!
When you [submit a bug report][issue], please include as much context as you
can. This includes information like Python version, cleanlab version, an error
message or stack trace, and detailed steps to reproduce the bug (if possible, including toy data that reproduces the error). The more information you can include, the better.
## Pull Requests
Want to write code to help improve cleanlab? Awesome!
If there are [open issues][issues], you're more than welcome to work on those (a good place to start is those tagged "good first issue"). If you have your own ideas, that's great too! In that case, before working on substantial changes to the code base, it is *highly recommended* that you first
[open an issue][issue] describing what you intend to work on.
To contribute your code to the library, you'll want to [fork the repo](https://docs.github.com/en/get-started/quickstart/fork-a-repo), push your changes to your fork, and finally create a new [Pull Request][pr].
Detailed development instructions, such as how to run the tests, are available
in [DEVELOPMENT.md](DEVELOPMENT.md).
---
If you have any questions about contributing to cleanlab, feel free to
[ask][discussions]!
[issue]: https://github.com/cleanlab/cleanlab/issues/new
[issues]: https://github.com/cleanlab/cleanlab/issues
[fork]: https://github.com/cleanlab/cleanlab/fork
[pr]: https://github.com/cleanlab/cleanlab/pulls
[discussions]: https://github.com/cleanlab/cleanlab/discussions
DEVELOPMENT.md
+426
View File
@@ -0,0 +1,426 @@
# Development
This document explains how to set up a development environment for
[contributing](CONTRIBUTING.md) to cleanlab.
## Setting up a virtual environment
While this is not required, we recommend that you do development and testing in
a virtual environment. There are a number of tools to do this, including
[uv](https://docs.astral.sh/uv/), [virtualenv](https://virtualenv.pypa.io/),
[pipenv](https://pipenv.pypa.io/), and [venv](https://docs.python.org/3/library/venv.html).
You can [compare](https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe)
the tools and choose what is right for you.
### Using uv (fast)
```shell
uv venv ./ENV # create a new virtual environment in the directory ENV
source ./ENV/bin/activate # switch to using the virtual environment
```
### Using venv (built-in)
```shell
python3 -m venv ./ENV # create a new virtual environment in the directory ENV
source ./ENV/bin/activate # switch to using the virtual environment
```
You only need to create the virtual environment once, but you will need to
activate it every time you start a new shell. Once the virtual environment is
activated, the install commands below will install dependencies into the
virtual environment rather than your system Python installation.
## Installing dependencies and cleanlab
Run the following commands in the repository's root directory.
### Using uv
```shell
uv pip install -r requirements-dev.txt # install development requirements
uv pip install -e . # install cleanlab as an editable package
```
### Using pip
```shell
pip install -r requirements-dev.txt # install development requirements
pip install -e . # install cleanlab as an editable package
```
For Macs with Apple silicon: replace `tensorflow` in requirements-dev.txt with: `tensorflow-macos==2.9.2` and `tensorflow-metal==0.5.1`
### Handling optional dependencies
When designing a class that relies on an optional, domain-specific runtime dependency, it is better to use lazy-importing to avoid forcing users to install the dependency if they do not need it.
Depending on the coupling of your class to the dependency, you may want to consider importing it at the module-level or as an instance variable of the class or a function that uses the dependency.
If the dependency is used by many methods in the module or other classes, it is better to import it at the module-level.
On the other hand, if the dependency is only used by a handful of methods, then it's better to import it inside the method. If the dependency is not installed, an ImportError should be raised when the method is called, along with instructions on how to install the dependency.
Here is an example of a class that lazily imports CuPy and has a sum method (element-wise) that can be used on both CPU and GPU devices.
Unless an alternative implementations of the sum method is available, an `ImportError` should be raised when the method is called with instructions on how to install the dependency.
<details> <summary>Example code</summary>
```python
def lazy_import_cupy():
try:
import cupy
except ImportError as error:
# If the dependency is required for the class to work,
# replace this block with a raised ImportError containing instructions
print("Warning: cupy is not installed. Please install it with `pip install cupy`.")
cupy = None
return cupy
class Summation:
def __init__(self):
self.cupy = lazy_import_cupy()
def sum(self, x) -> float:
if self.cupy is None:
return sum(x)
return self.cupy.sum(x)
```
</details>
For the build system to recognize the optional dependency, you should add it to the `EXTRAS_REQUIRE` constant in **setup.py**:
<details> <summary>Example code</summary>
```python
EXTRAS_REQUIRE = {
...
"gpu": [
# Explain why the dependency below is needed,
# e.g. "for performing summation on GPU"
"cupy",
],
}
```
Or assign to a separate variable and add it to `EXTRAS_REQUIRE`
```python
GPU_REQUIRES = [
# Explanation ...
"cupy",
]
EXTAS_REQUIRE = {
...
"gpu": GPU_REQUIRES,
}
```
</details>
The package can be installed with the optional dependency (here called `gpu`) via:
### PyPI installation
```shell
uv pip install "cleanlab[gpu]"
```
### Editable installation
```shell
uv pip install -e ".[gpu]"
```
## Testing
**Download test data**
The test data for cleanlab resides in the assets repository. Use the following commands to download test data before running tests.
```shell
git clone https://github.com/cleanlab/assets.git
mv assets/cleanlab_test_data cleanlab/tests/datalab/data
```
**Run all the tests:**
```shell
pytest
```
**Run a specific file or test:**
```shell
pytest -k <filename or filter expression>
```
**Run with verbose output:**
```shell
pytest --verbose
```
**Run with code coverage:**
```shell
pytest --cov=cleanlab/ --cov-config .coveragerc --cov-report=html
```
The coverage report will be available in `coverage_html_report/index.html`,
which you can open with your web browser.
### Type checking
Cleanlab uses [mypy](https://mypy.readthedocs.io/en/stable/) typing. Type checking happens automatically during CI but can be run locally.
**Check typing in all files:**
```shell
mypy cleanlab
```
The above is just a simplified command for demonstration, do NOT run this for testing your own type annotations!
Our CI adds a few additional flags to the `mypy` command it uses in the file:
**.github/workflows/ci.yml**.
To exactly match the `mypy` command that is executed in CI, copy these flags, and also ensure your version of `mypy` and related packages like `pandas-stubs` match the latest released versions (used in our CI).
### Examples
You can check that the [examples](https://github.com/cleanlab/examples) still
work with changes you make to cleanlab by manually running the notebooks.
You can also run all example notebooks as follows:
```shell
git clone https://github.com/cleanlab/examples.git
```
Then specify your local version of cleanlab source in the first line of: **examples/requirements.txt**.
E.g. you can edit this line to point to your local version of cleanlab as a relative path such as `../cleanlab` if the `cleanlab` and `examples` repos are sibling directories on your computer.
Finally execute the bash script:
```shell
examples/run_all_notebooks.sh
```
## How to style new code contributions
cleanlab follows the [Black](https://black.readthedocs.io/) code style (see [pyproject.toml](pyproject.toml)). This is
enforced by CI, so please format your code by invoking `black` before submitting a pull request.
Generally aim to follow the [PEP-8 coding style](https://peps.python.org/pep-0008/).
Please do not use wildcard `import *` in any files, instead you should always import the specific functions that you need from a module.
All cleanlab code should have a maximum line length of 100 characters.
### Pre-commit hook
This repo uses the [pre-commit framework](https://pre-commit.com/) to easily
set up code style checks that run automatically whenever you make a commit.
You can install the git hook scripts with:
```shell
pre-commit install
```
### EditorConfig
This repo uses [EditorConfig](https://editorconfig.org/) to keep code style
consistent across editors and IDEs. You can install a plugin for your editor,
and then your editor will automatically ensure that indentation and line
endings match the project style.
## Adding new modules into the source code
You should go through the following checklist if you intend to add new functionality to the package in a separate module.
- [x] Add brief description of the module’s purpose in a comment at the top of file and docstrings for every function.
- [x] Import the module `my_module.py` into main [``__init__.py``](cleanlab/__init__.py)
- [x] Create detailed unit tests (typically in a new file `tests/test_my_module.py`)
- [x] Add new module to docs index pages (**docs/source/index.rst**) and create .rst file in **docs/source/cleanlab/** (so that module appears on [docs.cleanlab.ai](https://docs.cleanlab.ai/stable/index.html) -- please verify its documentation also looks good there)
- [x] Create a QuickStart (**docs/source/tutorials**) notebook that runs main module functionality in 5min or less and add it to index pages (**docs/source/tutorials/index.rst**, **docs/source/index.rst**). Clear cell output before pushing.
- [x] Create an [examples](https://github.com/cleanlab/examples) notebook that runs more advanced module functionality with a more real-world application (can have a longer run time). Push with printed cell output.
## Contributing new issue types to Datalab
To contribute a new type of issue that Datalab can automatically detect in any dataset, refer to our guide on [Creating Your Own Issues Manager](https://docs.cleanlab.ai/master/cleanlab/datalab/guide/custom_issue_manager.html).
Do not add your new issue type to the set of issues that Datalab detects by default, our team can add it to this default set later on once it's utility has been thoroughly validated.
Don't forget to update the [issue type descriptions guide](https://github.com/cleanlab/cleanlab/blob/master/docs/source/cleanlab/datalab/guide/issue_type_description.rst) with a brief description of your new issue type.
It is ideal to stick to a format that maintains consistency and readability.
Generally, the format includes a title, explanation of the issue, required arguments, then any additional information.
It would be helpful to include a tip for users on how to detect the issue using Datalab.
Try to add tests for this new issue type. It's a good idea to start with some tests in a separate module in the [issue manager test directory](https://github.com/cleanlab/cleanlab/tree/master/tests/datalab/issue_manager).
## Documentation
You can build the docs from your local cleanlab version by following [these
instructions](./docs/README.md#build-the-cleanlab-docs-locally).
If editing existing docs or adding new tutorials, please first read through our [guidelines](./docs/README.md#tips-for-editing-docstutorials).
## Documentation style
cleanlab uses [NumPy
style](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard)
docstrings
([example](https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html)).
Aspects that are not covered in the NumPy style or that are different from the
NumPy style are documented below:
- **Referring to the cleanlab package**: we refer to cleanlab without any
special formatting, so no `cleanlab`, just cleanlab.
- **Cross-referencing**: when mentioning functions/classes/methods, always
[cross-reference](https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects)
them to create a clickable link. Cross-referencing code from Jupyter
notebooks is not currently supported.
- **Variable, module, function, and class names**: when not cross-references,
should be written between single back-ticks, like `` `pred_probs` ``. Such
names in Jupyter notebooks (Markdown) can be written between single
back-ticks as well.
- **Math**: We support [LaTeX
math](https://sphinxcontrib-katex.readthedocs.io/en/v0.8.6/examples.html)
with the inline `` :math:`x+y` `` or the block:
```
.. math::
\sum_{0}^{n} 2n+1
```
- **Pseudocode vs math**: Prefer pseudocode in double backticks over LaTeX math.
- **Bold vs italics**: Use italics when defining a term, and use bold sparingly
for extra emphasis.
- **Shapes**: Do not include shapes in the type of parameters, instead use
`np.array` or `array_like` as the type and specify allowed shapes in the
description. See, for example, the documentation for
`cleanlab.classification.CleanLearning.fit()`. Format for 1D shape: `(N,1)`
- **Optional arguments**: for the most part, just put `, optional` in the type.
- **Type unions**: if a parameter or return type is something like "a numpy
array or None", you can use "or" to separate types, e.g. `np.array or None`,
and it'll be parsed correctly.
- **Parameterized types**: Use [standard Python type
hints](https://docs.python.org/3/library/typing.html) for referring to
parameters and parameterized types in docs, e.g. `Iterable[int]` or
`list[float]`.
### Common variable names / terminology used throughout codebase
- `N` - the number of examples/datapoints in a dataset.
- `num_examples` may also be used when additional clarity is needed.
- `K` - the number of classes (unique labels) for a dataset.
- `num_classes` may also be used when additional clarity is needed.
- `labels` - a label for each example, length should be N (sample-size of dataset)
- `classes` - set of possible labels for any one example, length should be K (number of possible categories in classification problem)
Try to adhere to this standardized terminology unless you have good reason not to!
### Relative Link Formatting Instructions
Use relative linking to connect information between docs and jupyter notebooks, and make sure links will remain valid in the future as new cleanlab versions are released! Sphinx/html works with relative paths so try to specify relative paths if necessary. For specific situations:
- Link another function or class from within a source code docstring:
- If you just want to specify the function/class name (ie. the function/class is unique throughout our library): `` `~cleanlab.file.function_or_class_name` ``.
This uses the [Sphinx's](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-default_role) `default_role = "py:obj"` setting, so the leading tilde shortens the link to only display `function_or_class_name`.
- If you want to additionally specify the module which the function belongs to:
- `` :py:func:`file.function_name <cleanlab.file.function_name>` `` for functions
- ``:py:class:`file.class_name <cleanlab.file.class_name>` `` for classes
Here you have more control over the text that is displayed to display the module name. When referring to a function that is alternatively defined in other modules as well, always use this option to be more explicit about which module you are referencing.
- Link a tutorial (rst file) from within a source code docstring or rst file: ``:ref:`tutorial_name <tutorial_name>` ``
- Link a tutorial notebook (ipynb file) from within a source code docstring or rst file: `` `notebook_name <tutorials/notebook_name.ipynb>`_ `` . (If the notebook is not the in the same folder as the source code, use a relative path)
- Link a function from within a tutorial notebook: `[function_name](../cleanlab/file.html#cleanlab.file.function_name)`
Links from master branch tutorials will reference master branch functions, similarly links from tutorials in stable branch will reference stable branch functions since we are using relative paths.
- Link a specific section of a notebook from within the notebook: `[section title](#section-title)`
- Link a different tutorial notebook from within a tutorial notebook: `[another notebook](another_notebook.html)`. (Note this only works when the other notebook is in same folder as this notebook, otherwise may need to try relative path)
- Link another specific section of different notebook from within a tutorial notebook: `[another notebook section title](another_notebook.html#another-notebook-section-title)`
- Linking examples notebooks from inside tutorial notebooks can be simply done by linking global url of the example notebook in master branch of github.com/cleanlab/examples/
## Packaging and releasing
The release process is automated using GitHub Actions. When a release is published on the main [cleanlab](https://github.com/cleanlab/cleanlab) repository, the following workflows are triggered:
1. Docs are built and pushed to the `cleanlab-docs` depository within the same organization, which handles the deployment to [docs.cleanlab.ai](https://docs.cleanlab.ai/stable/index.html).
2. A new release is created on PyPI with the same version number as the release on GitHub.
There are other workflows that need to be handled manually in other repositories, but that is outside the score of this section.
This section will focus on the PyPI release process.
### Developing the PyPI release process
It's important to test the release process on a separate PyPI project before releasing to the main [cleanlab project](https://pypi.org/project/cleanlab/). For the remainder of this section, we will refer to the test project as `test-cleanlab-<username>`, where `<username>` is your GitHub username. This name should be unique to avoid conflicts with other users' test projects.
#### Prerequisites
##### PyPI Prerequisites
- Create separate user accounts on [PyPI](https://pypi.org/) and [Test PyPI](https://test.pypi.org/).
- [Register here on PyPI](https://pypi.org/account/register/).
- [Register here on Test PyPI](https://test.pypi.org/account/register/).
- Ideally, these accounts should have the same username, but this is not strictly necessary.
- Add a "[Trusted Publisher](https://docs.pypi.org/trusted-publishers/)" on both PyPI accounts (i.e. Publishing with OpenID Connect).
- This will allow you to publish packages to PyPI and Test PyPI using GitHub Actions, without needing to store your PyPI credentials in the repository.
- Walk through the steps in ["Creating a PyPI project with a trusted publisher"](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/) for both PyPI and Test PyPI.
- The PyPI Project name is: `test-cleanlab-<username>`.
- The owner is: `<username>` (your GitHub username).
- The repository name is: `cleanlab` (your fork of the cleanlab repository).
- The [workflow file](.github/workflows/release-build-publish.yml) name is: `release-build-publish.yml`
- ATTENTION: The environment name should be left empty in the Test PyPI project, and set to `pypi` in the PyPI project.
- See discussion on the environment in the [GitHub Prerequisites](#github-prerequisites) section.
##### GitHub Prerequisites
- [Fork the cleanlab repository](https://github.com/cleanlab/cleanlab/fork) to your GitHub account.
- This will allow you to test the release process on your fork, on a separate PyPI project.
- On your fork of the cleanlab repository, create two environments called `testpypi` and `pypi` in the "Environments" tab, under the repository "Settings".
- For the `testpypi` environment, add a wait timer of 1 minute to allow TestPyPI uploads to complete processing before testing installation.
- For the `pypi` environment, add a protection rule for requiring a review from a maintainer. For extra security, you may disallow a self-review so that a second maintainer must approve the release.
- You may wish to limit which tags can trigger a release in these environment, to avoid accidental releases.
- Github Docs provides instructions on [how to create a new environment](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment#creating-an-environment) with these considerations.
##### Local Prerequisites
- In `pyproject.toml` replace the `project.name` configuration with the value `test-cleanlab-<username>`, where `<username>` is your GitHub username.
#### Testing the release process
1. Push a commit with an updated version number in `cleanlab/version.py`.
- Ideally, this should be a patch version bump, e.g. `0.1.0` to `0.1.1` or a minor version bump, e.g. `0.1.X` to `0.2.0`.
- This is typically done via a standalone PR to the cleanlab/cleanlab repository.
```diff
# Bump the version number in cleanlab/version.py
- __version__ = "2.6.0"
+ __version__ = "2.6.1"
```
2. On the repository's GitHub page, navigate to the "Releases" page and click "Draft a new release".
- Choose a tag version that matches the version number in `cleanlab/version.py`, it should follow the format `vX.Y.Z`, e.g. `v2.6.1` or `v2.7.0`.
- This kind of format will be automatically checked by the release workflow.
- This tag may not exist yet, but Github allows you to create it upon publishing the release.
- Target the `master` branch.
- Select the previous tag to compare against, if it exists. Usually this is the previous release tag, e.g. the previous patch version.
- GitHub should allow you to generate release notes based on this information.
- When you've finalized the release notes and are ready to publish the release, click "Publish release".
- This will kick off the release workflow, which will build and publish the package to Test PyPI, test the package installation, and then publish the package to PyPI.
3. Open up the "Actions" tab on your fork of the cleanlab repository and monitor the progress of the release workflow.
- A "Release Build Publish" workflow should be triggered by the release, and you can monitor its progress there.
- It will check for the project name (for uploading to the proper PyPI project), and validate the version name/tag.
- When these steps pass, it will build the distribution and check the contents.
- Passing the build step, it will upload the distribution to Test PyPI.
- After the upload, it will kick off several test jobs to install the package from Test PyPI and run various tests.
- Adding more kinds of tests at this stage in the workflow is a good idea, to ensure the package is working as expected. Just create a new job that `needs` the `verify-version` job, and runs the tests you want to add.
- After all the tests pass, it will trigger the final job. However, the environment should be configure to require a review from a maintainer before the final job can be run.
- View the deployment and approve it it everything looks good so far. This will trigger the final job to publish the package to PyPI.
LICENSE
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
MANIFEST.in
+6
View File
@@ -0,0 +1,6 @@
include LICENSE README.md CONTRIBUTING.md DEVELOPMENT.md CODE_OF_CONDUCT.md
# Exclude development and documentation files from distribution
exclude requirements-dev.txt
exclude requirements-test-core.txt
recursive-exclude docs *
README.md
+286
View File
@@ -0,0 +1,286 @@
<div align="center">
<img src="https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/cleanlab_logo_open_source.png" width=60%>
</div>
<div align="center">
<a href="https://pypi.org/pypi/cleanlab/" target="_blank"><img src="https://img.shields.io/pypi/v/cleanlab.svg" alt="pypi_versions"></a>
<a href="https://pypi.org/pypi/cleanlab/" target="_blank"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="py_versions"></a>
<a href="https://app.codecov.io/gh/cleanlab/cleanlab" target="_blank"><img src="https://codecov.io/gh/cleanlab/cleanlab/branch/master/graph/badge.svg" alt="coverage"></a>
<a href="https://github.com/cleanlab/cleanlab/stargazers/" target="_blank"><img src="https://img.shields.io/github/stars/cleanlab/cleanlab?style=social&maxAge=2592000" alt="Github Stars"></a>
<a href="https://twitter.com/CleanlabAI" target="_blank"><img src="https://img.shields.io/twitter/follow/CleanlabAI?style=social" alt="Twitter"></a>
</div>
<h4 align="center">
<p>
<a href="https://docs.cleanlab.ai/">Documentation</a> |
<a href="https://github.com/cleanlab/examples">Examples</a> |
<a href="https://cleanlab.ai/blog/learn/">Blog</a> |
<a href="#citation-and-related-publications">Research</a>
<p>
</h4>
Cleanlab’s open-source library helps you **clean** data and **lab**els by automatically detecting issues in a ML dataset. To facilitate **machine learning with messy, real-world data**, this data-centric AI package uses your *existing* models to estimate dataset problems that can be fixed to train even *better* models.
<p align="center">
<img src="https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/datalab_issues.png" width=74%>
</p>
<p align="center">
Examples of various issues in Cat/Dog dataset <b>automatically detected</b> by cleanlab via this code:
</p>
```python
lab = cleanlab.Datalab(data=dataset, label="column_name_for_labels")
# Fit any ML model, get its feature_embeddings & pred_probs for your data
lab.find_issues(features=feature_embeddings, pred_probs=pred_probs)
lab.report()
```
- Use cleanlab to automatically check every: [text](https://docs.cleanlab.ai/stable/tutorials/datalab/text.html), [audio](https://docs.cleanlab.ai/stable/tutorials/datalab/audio.html), [image](https://docs.cleanlab.ai/stable/tutorials/datalab/image.html), or [tabular](https://docs.cleanlab.ai/stable/tutorials/datalab/tabular.html) dataset.
- Use cleanlab to automatically: [detect data issues (outliers, duplicates, label errors, etc)](https://docs.cleanlab.ai/stable/tutorials/datalab/datalab_quickstart.html), [train robust models](https://docs.cleanlab.ai/stable/tutorials/indepth_overview.html), [infer consensus + annotator-quality for multi-annotator data](https://docs.cleanlab.ai/stable/tutorials/multiannotator.html), [suggest data to (re)label next (active learning)](https://github.com/cleanlab/examples/blob/master/active_learning_multiannotator/active_learning.ipynb).
---
## Run cleanlab open-source
This cleanlab package runs on Python 3.10+ and supports Linux, macOS, as well as Windows.
- Get started [here](https://docs.cleanlab.ai/)! Install via `uv`, `pip`, or `conda`.
- Developers who install the bleeding-edge from source should refer to [this master branch documentation](https://docs.cleanlab.ai/master/index.html).
**Practicing data-centric AI can look like this:**
1. Train initial ML model on original dataset.
2. Utilize this model to diagnose data issues (via cleanlab methods) and improve the dataset.
3. Train the same model on the improved dataset.
4. Try various modeling techniques to further improve performance.
Most folks jump from Step 1 → 4, but you may achieve big gains without *any* change to your modeling code by using cleanlab!
Continuously boost performance by iterating Steps 2 → 4 (and try to evaluate with *cleaned* data).
![](https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/flowchart.png)
## Use cleanlab with any model and in most ML tasks
All features of cleanlab work with **any dataset** and **any model**. Yes, any model: PyTorch, Tensorflow, Keras, JAX, HuggingFace, OpenAI, XGBoost, scikit-learn, etc.
cleanlab is useful across a wide variety of Machine Learning tasks. Specific tasks this data-centric AI package offers dedicated functionality for include:
1. [Binary and multi-class classification](https://docs.cleanlab.ai/stable/tutorials/indepth_overview.html)
2. [Multi-label classification](https://docs.cleanlab.ai/stable/tutorials/multilabel_classification.html) (e.g. image/document tagging)
3. [Token classification](https://docs.cleanlab.ai/stable/tutorials/token_classification.html) (e.g. entity recognition in text)
4. [Regression](https://docs.cleanlab.ai/stable/tutorials/regression.html) (predicting numerical column in a dataset)
5. [Image segmentation](https://docs.cleanlab.ai/stable/tutorials/segmentation.html) (images with per-pixel annotations)
6. [Object detection](https://docs.cleanlab.ai/stable/tutorials/object_detection.html) (images with bounding box annotations)
7. [Classification with data labeled by multiple annotators](https://docs.cleanlab.ai/stable/tutorials/multiannotator.html)
8. [Active learning with multiple annotators](https://github.com/cleanlab/examples/blob/master/active_learning_multiannotator/active_learning.ipynb) (suggest which data to label or re-label to improve model most)
9. [Outlier detection](https://docs.cleanlab.ai/stable/tutorials/outliers.html) (identify atypical data that appears out of distribution)
For other ML tasks, cleanlab can still help you improve your dataset if appropriately applied.
See our [Example Notebooks](https://github.com/cleanlab/examples) and [Blog](https://cleanlab.ai/blog/learn/).
## So fresh, so cleanlab
Beyond automatically catching [all sorts of issues](https://docs.cleanlab.ai/stable/cleanlab/datalab/guide/issue_type_description.html) lurking in your data, this data-centric AI package helps you deal with **noisy labels** and train more **robust ML models**.
Here's an example:
```python
# cleanlab works with **any classifier**. Yup, you can use PyTorch/TensorFlow/OpenAI/XGBoost/etc.
cl = cleanlab.classification.CleanLearning(sklearn.YourFavoriteClassifier())
# cleanlab finds data and label issues in **any dataset**... in ONE line of code!
label_issues = cl.find_label_issues(data, labels)
# cleanlab trains a robust version of your model that works more reliably with noisy data.
cl.fit(data, labels)
# cleanlab estimates the predictions you would have gotten if you had trained with *no* label issues.
cl.predict(test_data)
# A universal data-centric AI tool, cleanlab quantifies class-level issues and overall data quality, for any dataset.
cleanlab.dataset.health_summary(labels, confident_joint=cl.confident_joint)
```
cleanlab **clean**s your data's **lab**els via state-of-the-art *confident learning* algorithms, published in this [paper](https://jair.org/index.php/jair/article/view/12125) and [blog](https://l7.curtisnorthcutt.com/confident-learning). See some of the datasets cleaned with cleanlab at [labelerrors.com](https://labelerrors.com).
cleanlab is:
1. **backed by theory** -- with [provable guarantees](https://arxiv.org/abs/1911.00068) of exact label noise estimation, even with imperfect models.
2. **fast** -- code is parallelized and scalable.
4. **easy to use** -- one line of code to find mislabeled data, bad annotators, outliers, or train noise-robust models.
6. **general** -- works with **[any dataset](https://labelerrors.com/)** (text, image, tabular, audio,...) + **any model** (PyTorch, OpenAI, XGBoost,...)
<br/>
![](https://raw.githubusercontent.com/cleanlab/assets/master/cleanlab/label-errors-examples.png)
<p align="center">
Examples of incorrect given labels in various image datasets <a href="https://l7.curtisnorthcutt.com/label-errors">found and corrected</a> using cleanlab.
While these examples are from image datasets, this also works for text, audio, tabular data.
</p>
## Citation and related publications
cleanlab is based on peer-reviewed research. Here are relevant papers to cite if you use this package:
<details><summary><a href="https://arxiv.org/abs/1911.00068">Confident Learning (JAIR '21)</a> (<b>click to show bibtex</b>) </summary>
@article{northcutt2021confidentlearning,
title={Confident Learning: Estimating Uncertainty in Dataset Labels},
author={Curtis G. Northcutt and Lu Jiang and Isaac L. Chuang},
journal={Journal of Artificial Intelligence Research (JAIR)},
volume={70},
pages={1373--1411},
year={2021}
}
</details>
<details><summary><a href="https://arxiv.org/abs/1705.01936">Rank Pruning (UAI '17)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{northcutt2017rankpruning,
author={Northcutt, Curtis G. and Wu, Tailin and Chuang, Isaac L.},
title={Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels},
booktitle = {Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence},
series = {UAI'17},
year = {2017},
location = {Sydney, Australia},
numpages = {10},
url = {http://auai.org/uai2017/proceedings/papers/35.pdf},
publisher = {AUAI Press},
}
</details>
<details><summary><a href="https://jonasmueller.org/info/LabelQuality_icml.pdf"> Label Quality Scoring (ICML '22)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{kuan2022labelquality,
title={Model-agnostic label quality scoring to detect real-world label errors},
author={Kuan, Johnson and Mueller, Jonas},
booktitle={ICML DataPerf Workshop},
year={2022}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2210.03920"> Label Errors in Token Classification / Entity Recognition (NeurIPS '22)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{wang2022tokenerrors,
title={Detecting label errors in token classification data},
author={Wang, Wei-Chen and Mueller, Jonas},
booktitle={NeurIPS Workshop on Interactive Learning for Natural Language Processing (InterNLP)},
year={2022}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2211.13895"> Label Errors in Multi-Label Classification (ICLR '23)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{thyagarajan2023multilabel,
title={Identifying Incorrect Annotations in Multi-Label Classification Data},
author={Thyagarajan, Aditya and Snorrason, Elías and Northcutt, Curtis and Mueller, Jonas},
booktitle={ICLR Workshop on Trustworthy ML},
year={2023}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2309.00832"> Label Errors in Object Detection (ICML '23)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{tkachenko2023objectlab,
title={ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data},
author={Tkachenko, Ulyana and Thyagarajan, Aditya and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2307.05080"> Label Errors in Image Segmentation (ICML '23)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{lad2023segmentation,
title={Estimating label quality and errors in semantic segmentation data via any model},
author={Lad, Vedang and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2305.16583"> Detecting Errors in Numerical Data (DMLR '24)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{zhou2023errors,
title={Detecting Errors in a Numerical Response via any Regression Model},
author={Zhou, Hang and Mueller, Jonas and Kumar, Mayank and Wang, Jane-Ling and Lei, Jing},
booktitle={Journal of Data-centric Machine Learning Research},
year={2024}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2207.03061"> Out-of-Distribution Detection (ICML '22)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{kuan2022ood,
title={Back to the Basics: Revisiting Out-of-Distribution Detection Baselines},
author={Kuan, Johnson and Mueller, Jonas},
booktitle={ICML Workshop on Principles of Distribution Shift},
year={2022}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2210.06812"> CROWDLAB for Data with Multiple Annotators (NeurIPS '22)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{goh2022crowdlab,
title={CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators},
author={Goh, Hui Wen and Tkachenko, Ulyana and Mueller, Jonas},
booktitle={NeurIPS Human in the Loop Learning Workshop},
year={2022}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2301.11856"> ActiveLab: Active learning with data re-labeling (ICLR '23)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{goh2023activelab,
title={ActiveLab: Active Learning with Re-Labeling by Multiple Annotators},
author={Goh, Hui Wen and Mueller, Jonas},
booktitle={ICLR Workshop on Trustworthy ML},
year={2023}
}
</details>
<details><summary><a href="https://arxiv.org/abs/2305.15696"> Detecting Dataset Drift and Non-IID Sampling (ICML '23)</a> (<b>click to show bibtex</b>) </summary>
@inproceedings{cummings2023drift,
title={Detecting Dataset Drift and Non-IID Sampling via k-Nearest Neighbors},
author={Cummings, Jesse and Snorrason, Elías and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
</details>
To understand/cite other cleanlab functionality not described above, check out our [Blog](https://cleanlab.ai/blog/learn).
## Other resources
- [Example Notebooks demonstrating practical applications of this package](https://github.com/cleanlab/examples)
- [Cleanlab Blog](https://cleanlab.ai/blog/learn)
- [Blog post: Introduction to Confident Learning](https://l7.curtisnorthcutt.com/confident-learning)
- [NeurIPS 2021 paper: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://arxiv.org/abs/2103.14749)
- [Introduction to Data-centric AI (MIT IAP Course)](https://dcai.csail.mit.edu/)
- [Release notes for past versions](https://github.com/cleanlab/cleanlab/releases)
**Interested in contributing?** See the [contributing guide](CONTRIBUTING.md), [development guide](DEVELOPMENT.md), and [ideas on useful contributions](https://github.com/cleanlab/cleanlab/wiki#ideas-for-contributing-to-cleanlab).
**Have questions?** Check out [our FAQ](https://docs.cleanlab.ai/stable/tutorials/faq.html) and [Github Issues](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue).
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`cleanlab/cleanlab`
- 原始仓库：https://github.com/cleanlab/cleanlab
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
cleanlab/__init__.py
+57
View File
@@ -0,0 +1,57 @@
from .version import __version__
from . import classification
from . import count
from . import rank
from . import filter
from . import benchmarking
from . import dataset
from . import multiannotator
from . import outlier
from . import token_classification
from . import multilabel_classification
from . import object_detection
from . import regression
from . import segmentation
class DatalabUnavailable:
def __init__(self, message):
self.message = message
def __getattr__(self, name):
message = self.message + f" (raised when trying to access {name})"
raise ImportError(message)
def __call__(self, *args, **kwargs):
message = (
self.message + f" (raised when trying to call with args: {args}, kwargs: {kwargs})"
)
raise ImportError(message)
def _datalab_import_factory():
try:
from .datalab.datalab import Datalab as _Datalab
return _Datalab
except ImportError:
return DatalabUnavailable(
"Datalab is not available due to missing dependencies. "
"To install Datalab, run `pip install 'cleanlab[datalab]'`."
)
def _issue_manager_import_factory():
try:
from .datalab.internal.issue_manager import IssueManager as _IssueManager
return _IssueManager
except ImportError:
return DatalabUnavailable(
"IssueManager is not available due to missing dependencies for Datalab. "
"To install Datalab, run `pip install 'cleanlab[datalab]'`."
)
Datalab = _datalab_import_factory()
IssueManager = _issue_manager_import_factory()
cleanlab/benchmarking/__init__.py
+1
View File
@@ -0,0 +1 @@
from . import noise_generation
cleanlab/benchmarking/noise_generation.py
+486
View File
@@ -0,0 +1,486 @@
"""
Helper methods that are useful for benchmarking cleanlab’s core algorithms.
These methods introduce synthetic noise into the labels of a classification dataset.
Specifically, this module provides methods for generating valid noise matrices (for which learning with noise is possible),
generating noisy labels given a noise matrix, generating valid noise matrices with a specific trace value, and more.
"""
from typing import Optional
import numpy as np
from cleanlab.internal.util import value_counts
from cleanlab.internal.constants import FLOATING_POINT_COMPARISON
def noise_matrix_is_valid(noise_matrix, py, *, verbose=False) -> bool:
"""Given a prior `py` representing ``p(true_label=k)``, checks if the given `noise_matrix` is a
learnable matrix. Learnability means that it is possible to achieve
better than random performance, on average, for the amount of noise in
`noise_matrix`.
Parameters
----------
noise_matrix : np.ndarray
An array of shape ``(K, K)`` representing the conditional probability
matrix ``P(label=k_s|true_label=k_y)`` containing the fraction of
examples in every class, labeled as every other class. Assumes columns of
`noise_matrix` sum to 1.
py : np.ndarray
An array of shape ``(K,)`` representing the fraction (prior probability)
of each true class label, ``P(true_label = k)``.
Returns
-------
is_valid : bool
Whether the noise matrix is a learnable matrix.
"""
# Number of classes
K = len(py)
# let's assume some number of training examples for code readability,
# but it doesn't matter what we choose as it's not actually used.
N = float(10000)
ps = np.dot(noise_matrix, py) # P(true_label=k)
# P(label=k, true_label=k')
joint_noise = np.multiply(noise_matrix, py) # / float(N)
# Check that joint_probs is valid probability matrix
if not (abs(joint_noise.sum() - 1.0) < FLOATING_POINT_COMPARISON):
return False
# Check that noise_matrix is a valid matrix
# i.e. check p(label=k)*p(true_label=k) < p(label=k, true_label=k)
for i in range(K):
C = N * joint_noise[i][i]
E1 = N * joint_noise[i].sum() - C
E2 = N * joint_noise.T[i].sum() - C
O = N - E1 - E2 - C
if verbose:
print(
"E1E2/C",
round(E1 * E2 / C),
"E1",
round(E1),
"E2",
round(E2),
"C",
round(C),
"|",
round(E1 * E2 / C + E1 + E2 + C),
"|",
round(E1 * E2 / C),
"<",
round(O),
)
print(
round(ps[i] * py[i]),
"<",
round(joint_noise[i][i]),
":",
ps[i] * py[i] < joint_noise[i][i],
)
if not (ps[i] * py[i] < joint_noise[i][i]):
return False
return True
def generate_noisy_labels(true_labels, noise_matrix) -> np.ndarray:
"""Generates noisy `labels` from perfect labels `true_labels`,
"exactly" yielding the provided `noise_matrix` between `labels` and `true_labels`.
Below we provide a for loop implementation of what this function does.
We do not use this implementation as it is not a fast algorithm, but
it explains as Python pseudocode what is happening in this function.
Parameters
----------
true_labels : np.ndarray
An array of shape ``(N,)`` representing perfect labels, without any
noise. Contains K distinct natural number classes, 0, 1, ..., K-1.
noise_matrix : np.ndarray
An array of shape ``(K, K)`` representing the conditional probability
matrix ``P(label=k_s|true_label=k_y)`` containing the fraction of
examples in every class, labeled as every other class. Assumes columns of
`noise_matrix` sum to 1.
Returns
-------
labels : np.ndarray
An array of shape ``(N,)`` of noisy labels.
Examples
--------
.. code:: python
# Generate labels
count_joint = (noise_matrix * py * len(y)).round().astype(int)
labels = np.ndarray(y)
for k_s in range(K):
for k_y in range(K):
if k_s != k_y:
idx_flip = np.where((labels==k_y)&(true_label==k_y))[0]
if len(idx_flip): # pragma: no cover
labels[np.random.choice(
idx_flip,
count_joint[k_s][k_y],
replace=False,
)] = k_s
"""
# Make y a numpy array, if it is not
true_labels = np.asarray(true_labels)
# Number of classes
K = len(noise_matrix)
# Compute p(true_label=k)
py = value_counts(true_labels) / float(len(true_labels))
# Counts of pairs (labels, y)
count_joint = (noise_matrix * py * len(true_labels)).astype(int)
# Remove diagonal entries as they do not involve flipping of labels.
np.fill_diagonal(count_joint, 0)
# Generate labels
labels = np.array(true_labels)
for k in range(K): # Iterate over true_label == k
# Get the noisy labels that have non-zero counts
labels_per_class = np.where(count_joint[:, k] != 0)[0]
# Find out how many of each noisy label we need to flip to
label_counts = count_joint[labels_per_class, k]
# Create a list of the new noisy labels
noise = [labels_per_class[i] for i, c in enumerate(label_counts) for z in range(c)]
# Randomly choose y labels for class k and set them to the noisy labels.
idx_flip = np.where((labels == k) & (true_labels == k))[0]
if len(idx_flip) and len(noise) and len(idx_flip) >= len(noise): # pragma: no cover
labels[np.random.choice(idx_flip, len(noise), replace=False)] = noise
# Validate that labels indeed produces the correct noise_matrix (or close to it)
# Compute the actual noise matrix induced by labels
# counts = confusion_matrix(labels, true_labels).astype(float)
# new_noise_matrix = counts / counts.sum(axis=0)
# assert(np.linalg.norm(noise_matrix - new_noise_matrix) <= 2)
return labels
def generate_noise_matrix_from_trace(
K,
trace,
*,
max_trace_prob=1.0,
min_trace_prob=1e-5,
max_noise_rate=1 - 1e-5,
min_noise_rate=0.0,
valid_noise_matrix=True,
py=None,
frac_zero_noise_rates=0.0,
seed=0,
max_iter=10000,
) -> Optional[np.ndarray]:
"""Generates a ``K x K`` noise matrix ``P(label=k_s|true_label=k_y)`` with
``np.sum(np.diagonal(noise_matrix))`` equal to the given `trace`.
Parameters
----------
K : int
Creates a noise matrix of shape ``(K, K)``. Implies there are
K classes for learning with noisy labels.
trace : float
Sum of diagonal entries of array of random probabilities returned.
max_trace_prob : float
Maximum probability of any entry in the trace of the return matrix.
min_trace_prob : float
Minimum probability of any entry in the trace of the return matrix.
max_noise_rate : float
Maximum noise_rate (non-diagonal entry) in the returned np.ndarray.
min_noise_rate : float
Minimum noise_rate (non-diagonal entry) in the returned np.ndarray.
valid_noise_matrix : bool, default=True
If ``True``, returns a matrix having all necessary conditions for
learning with noisy labels. In particular, ``p(true_label=k)p(label=k) < p(true_label=k,label=k)``
is satisfied. This requires that ``trace > 1``.
py : np.ndarray
An array of shape ``(K,)`` representing the fraction (prior probability) of each true class label, ``P(true_label = k)``.
This argument is **required** when ``valid_noise_matrix=True``.
frac_zero_noise_rates : float
The fraction of the ``n*(n-1)`` noise rates
that will be set to 0. Note that if you set a high trace, it may be
impossible to also have a low fraction of zero noise rates without
forcing all non-1 diagonal values. Instead, when this happens we only
guarantee to produce a noise matrix with `frac_zero_noise_rates` *or
higher*. The opposite occurs with a small trace.
seed : int
Seeds the random number generator for numpy.
max_iter : int, default=10000
The max number of tries to produce a valid matrix before returning ``None``.
Returns
-------
noise_matrix : np.ndarray or None
An array of shape ``(K, K)`` representing the noise matrix ``P(label=k_s|true_label=k_y)`` with `trace`
equal to ``np.sum(np.diagonal(noise_matrix))``. This a conditional probability matrix and a
left stochastic matrix. Returns ``None`` if `max_iter` is exceeded.
"""
if valid_noise_matrix and trace <= 1:
raise ValueError(
"trace = {}. trace > 1 is necessary for a".format(trace)
+ " valid noise matrix to be returned (valid_noise_matrix == True)"
)
if valid_noise_matrix and py is None and K > 2:
raise ValueError(
"py must be provided (not None) if the input parameter" + " valid_noise_matrix == True"
)
if K <= 1:
raise ValueError("K must be >= 2, but K = {}.".format(K))
if max_iter < 1:
return None
np.random.seed(seed)
# Special (highly constrained) case with faster solution.
# Every 2 x 2 noise matrix with trace > 1 is valid because p(y) is not used
if K == 2:
if frac_zero_noise_rates >= 0.5: # Include a single zero noise rate
noise_mat = np.array(
[
[1.0, 1 - (trace - 1.0)],
[0.0, trace - 1.0],
]
)
return noise_mat if np.random.rand() > 0.5 else np.rot90(noise_mat, k=2)
else: # No zero noise rates
diag = generate_n_rand_probabilities_that_sum_to_m(2, trace)
noise_matrix = np.array(
[
[diag[0], 1 - diag[1]],
[1 - diag[0], diag[1]],
]
)
return noise_matrix
# K > 2
for z in range(max_iter):
noise_matrix = np.zeros(shape=(K, K))
# Randomly generate noise_matrix diagonal.
nm_diagonal = generate_n_rand_probabilities_that_sum_to_m(
n=K,
m=trace,
max_prob=max_trace_prob,
min_prob=min_trace_prob,
)
np.fill_diagonal(noise_matrix, nm_diagonal)
# Randomly distribute number of zero-noise-rates across columns
num_col_with_noise = K - np.count_nonzero(1 == nm_diagonal)
num_zero_noise_rates = int(K * (K - 1) * frac_zero_noise_rates)
# Remove zeros already in [1,0,..,0] columns
num_zero_noise_rates -= (K - num_col_with_noise) * (K - 1)
num_zero_noise_rates = np.maximum(num_zero_noise_rates, 0) # Prevent negative
num_zero_noise_rates_per_col = (
randomly_distribute_N_balls_into_K_bins(
N=num_zero_noise_rates,
K=num_col_with_noise,
max_balls_per_bin=K - 2,
# 2 = one for diagonal, and one to sum to 1
min_balls_per_bin=0,
)
if K > 2
else np.array([0, 0])
) # Special case when K == 2
stack_nonzero_noise_rates_per_col = list(K - 1 - num_zero_noise_rates_per_col)[::-1]
# Randomly generate noise rates for columns with noise.
for col in np.arange(K)[nm_diagonal != 1]:
num_noise = stack_nonzero_noise_rates_per_col.pop()
# Generate num_noise noise_rates for the given column.
noise_rates_col = list(
generate_n_rand_probabilities_that_sum_to_m(
n=num_noise,
m=1 - nm_diagonal[col],
max_prob=max_noise_rate,
min_prob=min_noise_rate,
)
)
# Randomly select which rows of the noisy column to assign the
# random noise rates
rows = np.random.choice(
[row for row in range(K) if row != col], num_noise, replace=False
)
for row in rows:
noise_matrix[row][col] = noise_rates_col.pop()
if not valid_noise_matrix or noise_matrix_is_valid(noise_matrix, py):
return noise_matrix
return None
def generate_n_rand_probabilities_that_sum_to_m(
n,
m,
*,
max_prob=1.0,
min_prob=0.0,
) -> np.ndarray:
"""
Generates `n` random probabilities that sum to `m`.
When ``min_prob=0`` and ``max_prob = 1.0``, use
``np.random.dirichlet(np.ones(n))*m`` instead.
Parameters
----------
n : int
Length of array of random probabilities to be returned.
m : float
Sum of array of random probabilities that is returned.
max_prob : float, default=1.0
Maximum probability of any entry in the returned array. Must be between 0 and 1.
min_prob : float, default=0.0
Minimum probability of any entry in the returned array. Must be between 0 and 1.
Returns
-------
probabilities : np.ndarray
An array of probabilities.
"""
if n == 0:
return np.array([])
if (max_prob + FLOATING_POINT_COMPARISON) < m / float(n):
raise ValueError(
"max_prob must be greater or equal to m / n, but "
+ "max_prob = "
+ str(max_prob)
+ ", m = "
+ str(m)
+ ", n = "
+ str(n)
+ ", m / n = "
+ str(m / float(n))
)
if min_prob > (m + FLOATING_POINT_COMPARISON) / float(n):
raise ValueError(
"min_prob must be less or equal to m / n, but "
+ "max_prob = "
+ str(max_prob)
+ ", m = "
+ str(m)
+ ", n = "
+ str(n)
+ ", m / n = "
+ str(m / float(n))
)
# When max_prob = 1, min_prob = 0, the next two lines are equivalent to:
# intermediate = np.sort(np.append(np.random.uniform(0, 1, n-1), [0, 1]))
# result = (intermediate[1:] - intermediate[:-1]) * m
result = np.random.dirichlet(np.ones(n)) * m
min_val = min(result)
max_val = max(result)
while max_val > (max_prob + FLOATING_POINT_COMPARISON):
new_min = min_val + (max_val - max_prob)
# This adjustment prevents the new max from always being max_prob.
adjustment = (max_prob - new_min) * np.random.rand()
result[np.argmin(result)] = new_min + adjustment
result[np.argmax(result)] = max_prob - adjustment
min_val = min(result)
max_val = max(result)
min_val = min(result)
max_val = max(result)
while min_val < (min_prob - FLOATING_POINT_COMPARISON):
min_val = min(result)
max_val = max(result)
new_max = max_val - (min_prob - min_val)
# This adjustment prevents the new min from always being min_prob.
adjustment = (new_max - min_prob) * np.random.rand()
result[np.argmax(result)] = new_max - adjustment
result[np.argmin(result)] = min_prob + adjustment
min_val = min(result)
max_val = max(result)
return result
def randomly_distribute_N_balls_into_K_bins(
N, # int
K, # int
*,
max_balls_per_bin=None,
min_balls_per_bin=None,
) -> np.ndarray:
"""Returns a uniformly random numpy integer array of length `N` that sums
to `K`.
Parameters
----------
N : int
Number of balls.
K : int
Number of bins.
max_balls_per_bin : int
Ensure that each bin contains at most `max_balls_per_bin` balls.
min_balls_per_bin : int
Ensure that each bin contains at least `min_balls_per_bin` balls.
Returns
-------
int_array : np.array
Length `N` array that sums to `K`.
"""
if N == 0:
return np.zeros(K, dtype=int)
if max_balls_per_bin is None:
max_balls_per_bin = N
else:
max_balls_per_bin = min(max_balls_per_bin, N)
if min_balls_per_bin is None:
min_balls_per_bin = 0
else:
min_balls_per_bin = min(min_balls_per_bin, N / K)
if N / float(K) > max_balls_per_bin:
N = max_balls_per_bin * K
arr = np.round(
generate_n_rand_probabilities_that_sum_to_m(
n=K,
m=1,
max_prob=max_balls_per_bin / float(N),
min_prob=min_balls_per_bin / float(N),
)
* N
)
while sum(arr) != N:
while sum(arr) > N: # pragma: no cover
arr[np.argmax(arr)] -= 1
while sum(arr) < N:
arr[np.argmin(arr)] += 1
return arr.astype(int)
cleanlab/classification.py
+1062
View File
File diff suppressed because it is too large Load Diff
cleanlab/count.py
+1489
View File
File diff suppressed because it is too large Load Diff
cleanlab/data_valuation.py
+127
View File
@@ -0,0 +1,127 @@
"""
Methods for quantifying the value of each data point in a Machine Learning dataset.
Data Valuation helps us assess individual training data points' contributions to a ML model's predictive performance.
"""
from typing import Callable, Optional, Union
import numpy as np
from scipy.sparse import csr_matrix
from cleanlab.internal.neighbor.knn_graph import create_knn_graph_and_index
def _knn_shapley_score(neighbor_indices: np.ndarray, y: np.ndarray, k: int) -> np.ndarray:
"""Compute the Data Shapley values of data points using neighbor indices in a K-Nearest Neighbors (KNN) graph.
This function leverages equations (18) and (19) from the paper available at https://arxiv.org/abs/1908.08619
for computational efficiency.
Parameters
----------
neighbor_indices :
A 2D array where each row contains the indices of the k-nearest neighbors for each data point.
y :
A 1D array of target values corresponding to the data points.
k :
The number of nearest neighbors to consider for each data point.
Notes
-----
- The training set is used as its own test set for the KNN-Shapley value computation, meaning y_test is the same as y_train.
- `neighbor_indices` are assumed to be pre-sorted by distance, with the nearest neighbors appearing first, and with at least `k` neighbors.
- Unlike the referenced paper, this implementation does not account for an upper error bound epsilon.
Consequently, K* is treated as equal to K instead of K* = max(K, 1/epsilon).
- This simplification implies that the term min(K, j + 1) will always be j + 1, which is offset by the
corresponding denominator term in the inner loop.
- Dividing by K in the end achieves the same result as dividing by K* in the paper.
- The pre-allocated `scores` array incorporates equation (18) for j = k - 1, ensuring efficient computation.
"""
N = y.shape[0]
scores = np.zeros((N, N))
for y_alpha, s_alpha, idx in zip(y, scores, neighbor_indices):
y_neighbors = y[idx]
ans_matches = (y_neighbors == y_alpha).flatten()
for j in range(k - 2, -1, -1):
s_alpha[idx[j]] = s_alpha[idx[j + 1]] + float(
int(ans_matches[j]) - int(ans_matches[j + 1])
)
return np.mean(scores / k, axis=0)
def data_shapley_knn(
labels: np.ndarray,
*,
features: Optional[np.ndarray] = None,
knn_graph: Optional[csr_matrix] = None,
metric: Optional[Union[str, Callable]] = None,
k: int = 10,
) -> np.ndarray:
"""
Compute the Data Shapley values of data points using a K-Nearest Neighbors (KNN) graph.
This function calculates the contribution (Data Shapley value) of each data point in a dataset
for model training, either directly from data features or using a precomputed KNN graph.
The examples in the dataset with lowest data valuation scores contribute least
to a trained ML model’s performance (those whose value falls below a threshold are flagged with this type of issue).
The data valuation score is an approximate Data Shapley value, calculated based on the labels of the top k nearest neighbors of an example. Details on this KNN-Shapley value can be found in these papers:
https://arxiv.org/abs/1908.08619 and https://arxiv.org/abs/1911.07128.
Parameters
----------
labels :
An array of labels for the data points(only for multi-class classification datasets).
features :
Feature embeddings (vector representations) of every example in the dataset.
Necessary if `knn_graph` is not supplied.
If provided, this must be a 2D array with shape (num_examples, num_features).
knn_graph :
A precomputed sparse KNN graph. If not provided, it will be computed from the `features` using the specified `metric`.
metric : Optional[str or Callable], default=None
The distance metric for KNN graph construction.
Supports metrics available in ``sklearn.neighbors.NearestNeighbors``
Default metric is ``"cosine"`` for ``dim(features) > 3``, otherwise ``"euclidean"`` for lower-dimensional data.
The euclidean is computed with an efficient implementation from scikit-learn when the number of examples is greater than 100.
When the number of examples is 100 or fewer, a more numerically stable version of the euclidean distance from scipy is used.
k :
The number of neighbors to consider for the KNN graph and Data Shapley value computation.
Must be less than the total number of data points.
The value may not exceed the number of neighbors of each data point stored in the KNN graph.
Returns
-------
scores :
An array of transformed Data Shapley values for each data point, calibrated to indicate their relative importance.
These scores have been adjusted to fall within 0 to 1.
Values closer to 1 indicate data points that are highly influential and positively contribute to a trained ML model's performance.
Conversely, scores below 0.5 indicate data points estimated to negatively impact model performance.
Raises
------
ValueError
If neither `knn_graph` nor `features` are provided, or if `k` is larger than the number of examples in `features`.
Examples
--------
>>> import numpy as np
>>> from cleanlab.data_valuation import data_shapley_knn
>>> labels = np.array([0, 1, 0, 1, 0])
>>> features = np.array([[0, 1, 2, 3, 4]]).T
>>> data_shapley_knn(labels=labels, features=features, k=4)
array([0.55 , 0.525, 0.55 , 0.525, 0.55 ])
"""
if knn_graph is None and features is None:
raise ValueError("Either knn_graph or features must be provided.")
# Use provided knn_graph or compute it from features
if knn_graph is None:
knn_graph, _ = create_knn_graph_and_index(features, n_neighbors=k, metric=metric)
num_examples = labels.shape[0]
distances = knn_graph.indices.reshape(num_examples, -1)
scores = _knn_shapley_score(neighbor_indices=distances, y=labels, k=k)
return 0.5 * (scores + 1)
cleanlab/datalab/__init__.py
View File
cleanlab/datalab/datalab.py
+623
View File
@@ -0,0 +1,623 @@
"""
Datalab offers a unified audit to detect all kinds of issues in data and labels.
.. note::
.. include:: optional_dependencies.rst
"""
from __future__ import annotations
import warnings
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
import numpy as np
import pandas as pd
import cleanlab
from cleanlab.datalab.internal.adapter.constants import DEFAULT_CLEANVISION_ISSUES
from cleanlab.datalab.internal.adapter.imagelab import create_imagelab
from cleanlab.datalab.internal.data import Data
from cleanlab.datalab.internal.display import _Displayer
from cleanlab.datalab.internal.helper_factory import (
_DataIssuesBuilder,
issue_finder_factory,
report_factory,
)
from cleanlab.datalab.internal.issue_manager_factory import (
list_default_issue_types as _list_default_issue_types,
list_possible_issue_types as _list_possible_issue_types,
)
from cleanlab.datalab.internal.serialize import _Serializer
from cleanlab.datalab.internal.task import Task
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
from datasets.arrow_dataset import Dataset
from scipy.sparse import csr_matrix
DatasetLike = Union[Dataset, pd.DataFrame, Dict[str, Any], List[Dict[str, Any]], str]
__all__ = ["Datalab"]
class Datalab:
"""
A single object to automatically detect all kinds of issues in datasets.
This is how we recommend you interface with the cleanlab library if you want to audit the quality of your data and detect issues within it.
If you have other specific goals (or are doing a less standard ML task not supported by Datalab), then consider using the other methods across the library.
Datalab tracks intermediate state (e.g. data statistics) from certain cleanlab functions that can be re-used across other cleanlab functions for better efficiency.
Parameters
----------
data : Union[Dataset, pd.DataFrame, dict, list, str]
Dataset-like object that can be converted to a Hugging Face Dataset object.
It should contain the labels for all examples, identified by a
`label_name` column in the Dataset object.
Supported formats:
- datasets.Dataset
- pandas.DataFrame
- dict (keys are strings, values are arrays/lists of length ``N``)
- list (list of dictionaries that each have the same keys)
- str
- path to a local file: Text (.txt), CSV (.csv), JSON (.json)
- or a dataset identifier on the Hugging Face Hub
task : str
The type of machine learning task that the dataset is used for.
Supported tasks:
- "classification" (default): Multiclass classification
- "regression" : Regression
- "multilabel" : Multilabel classification
label_name : str, optional
The name of the label column in the dataset.
image_key : str, optional
Optional key that can be specified for image datasets to point to the field (column) containing the actual images themselves (as PIL objects).
If specified, additional image-specific issue types will be checked for in the dataset.
See the `CleanVision package <https://github.com/cleanlab/cleanvision?tab=readme-ov-file#clean-your-data-for-better-computer-vision>`_ for descriptions of these image-specific issue types.
Currently, this argument is only supported for data formatted as a Hugging Face ``datasets.Dataset`` object.
verbosity : int, optional
The higher the verbosity level, the more information
Datalab prints when auditing a dataset.
Valid values are 0 through 4. Default is 1.
Examples
--------
>>> import datasets
>>> from cleanlab import Datalab
>>> data = datasets.load_dataset("glue", "sst2", split="train")
>>> datalab = Datalab(data, label_name="label")
"""
def __init__(
self,
data: "DatasetLike",
task: str = "classification",
label_name: Optional[str] = None,
image_key: Optional[str] = None,
verbosity: int = 1,
) -> None:
# Assume continuous values of labels for regression task
# Map labels to integers for classification task
self.task = Task.from_str(task)
self._data = Data(data, self.task, label_name)
self.data = self._data._data
self._labels = self._data.labels
self._label_map = self._labels.label_map
self.label_name = self._labels.label_name
self._data_hash = self._data._data_hash
self.cleanlab_version = cleanlab.version.__version__
self.verbosity = verbosity
self._imagelab = create_imagelab(dataset=self.data, image_key=image_key)
# Create the builder for DataIssues
builder = _DataIssuesBuilder(self._data)
builder.set_imagelab(self._imagelab).set_task(self.task)
self.data_issues = builder.build()
# todo: check displayer methods
def __repr__(self) -> str:
return _Displayer(data_issues=self.data_issues, task=self.task).__repr__()
def __str__(self) -> str:
return _Displayer(data_issues=self.data_issues, task=self.task).__str__()
@property
def labels(self) -> Union[np.ndarray, List[List[int]]]:
"""Labels of the dataset, in a [0, 1, ..., K-1] format."""
return self._labels.labels
@property
def has_labels(self) -> bool:
"""Whether the dataset has labels, and that they are in a [0, 1, ..., K-1] format."""
return self._labels.is_available
@property
def class_names(self) -> List[str]:
"""Names of the classes in the dataset.
If the dataset has no labels, returns an empty list.
"""
return self._labels.class_names
def find_issues(
self,
*,
pred_probs: Optional[np.ndarray] = None,
features: Optional[npt.NDArray] = None,
knn_graph: Optional[csr_matrix] = None,
issue_types: Optional[Dict[str, Any]] = None,
) -> None:
"""
Checks the dataset for all sorts of common issues in real-world data (in both labels and feature values).
You can use Datalab to find issues in your data, utilizing *any* model you have already trained.
This method only interacts with your model via its predictions or embeddings (and other functions thereof).
The more of these inputs you provide, the more types of issues Datalab can detect in your dataset/labels.
If you provide a subset of these inputs, Datalab will output what insights it can based on the limited information from your model.
NOTE
----
The issues are saved in the ``self.issues`` attribute of the ``Datalab`` object, but are not returned.
Parameters
----------
pred_probs :
Out-of-sample predicted class probabilities made by the model for every example in the dataset.
To best detect label issues, provide this input obtained from the most accurate model you can produce.
For classification data, this must be a 2D array with shape ``(num_examples, K)`` where ``K`` is the number of classes in the dataset.
Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.
For regression data, this must be a 1D array with shape ``(num_examples,)`` containing the predicted value for each example.
For multilabel classification data, this must be a 2D array with shape ``(num_examples, K)`` where ``K`` is the number of classes in the dataset.
Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name.
features : Optional[np.ndarray]
Feature embeddings (vector representations) of every example in the dataset.
If provided, this must be a 2D array with shape (num_examples, num_features).
knn_graph :
Sparse matrix of precomputed distances between examples in the dataset in a k nearest neighbor graph.
If provided, this must be a square CSR matrix with shape ``(num_examples, num_examples)`` and ``(k*num_examples)`` non-zero entries (``k`` is the number of nearest neighbors considered for each example),
evenly distributed across the rows.
Each non-zero entry in this matrix is a distance between a pair of examples in the dataset. Self-distances must be omitted
(i.e. diagonal must be all zeros, k nearest neighbors for each example do not include the example itself).
This CSR format uses three 1D arrays (`data`, `indices`, `indptr`) to store a 2D matrix ``M``:
- `data`: 1D array containing all the non-zero elements of matrix ``M``, listed in a row-wise fashion (but sorted within each row).
- `indices`: 1D array storing the column indices in matrix ``M`` of these non-zero elements. Each entry in `indices` corresponds to an entry in `data`, indicating the column of ``M`` containing this entry.
- `indptr`: 1D array indicating the start and end indices in `data` for each row of matrix ``M``. The non-zero elements of the i-th row of ``M`` are stored from ``data[indptr[i]]`` to ``data[indptr[i+1]]``.
Within each row of matrix ``M`` (defined by the ranges in `indptr`), the corresponding non-zero entries (distances) of `knn_graph` must be sorted in ascending order (specifically in the segments of the `data` array that correspond to each row of ``M``). The `indices` array must also reflect this ordering, maintaining the correct column positions for these sorted distances.
This type of matrix is returned by the method: `sklearn.neighbors.NearestNeighbors.kneighbors_graph <https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html#sklearn.neighbors.NearestNeighbors.kneighbors_graph>`_.
Below is an example to illustrate:
.. code-block:: python
knn_graph.todense()
# matrix([[0. , 0.3, 0.2],
# [0.3, 0. , 0.4],
# [0.2, 0.4, 0. ]])
knn_graph.data
# array([0.2, 0.3, 0.3, 0.4, 0.2, 0.4])
# Here, 0.2 and 0.3 are the sorted distances in the first row, 0.3 and 0.4 in the second row, and so on.
knn_graph.indices
# array([2, 1, 0, 2, 0, 1])
# Corresponding neighbor indices for the distances from the `data` array.
knn_graph.indptr
# array([0, 2, 4, 6])
# The non-zero entries in the first row are stored from `knn_graph.data[0]` to `knn_graph.data[2]`, the second row from `knn_graph.data[2]` to `knn_graph.data[4]`, and so on.
For any duplicated examples i,j whose distance is 0, there should be an *explicit* zero stored in the matrix, i.e. ``knn_graph[i,j] = 0``.
If both `knn_graph` and `features` are provided, the `knn_graph` will take precendence.
If `knn_graph` is not provided, it is constructed based on the provided `features`.
If neither `knn_graph` nor `features` are provided, certain issue types like (near) duplicates will not be considered.
.. seealso::
See the
`scipy.sparse.csr_matrix documentation <https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html>`_
for more details on the CSR matrix format.
issue_types :
Collection specifying which types of issues to consider in audit and any non-default parameter settings to use.
If unspecified, a default set of issue types and recommended parameter settings is considered.
This is a dictionary of dictionaries, where the keys are the issue types of interest
and the values are dictionaries of parameter values that control how each type of issue is detected (only for advanced users).
More specifically, the values are constructor keyword arguments passed to the corresponding ``IssueManager``,
which is responsible for detecting the particular issue type.
.. seealso::
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`
Examples
--------
Here are some ways to provide inputs to :py:meth:`find_issues`:
- Passing ``pred_probs``:
.. code-block:: python
>>> from sklearn.linear_model import LogisticRegression
>>> import numpy as np
>>> from cleanlab import Datalab
>>> X = np.array([[0, 1], [1, 1], [2, 2], [2, 0]])
>>> y = np.array([0, 1, 1, 0])
>>> clf = LogisticRegression(random_state=0).fit(X, y)
>>> pred_probs = clf.predict_proba(X)
>>> lab = Datalab(data={"X": X, "y": y}, label_name="y")
>>> lab.find_issues(pred_probs=pred_probs)
- Passing ``features``:
.. code-block:: python
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.neighbors import NearestNeighbors
>>> import numpy as np
>>> from cleanlab import Datalab
>>> X = np.array([[0, 1], [1, 1], [2, 2], [2, 0]])
>>> y = np.array([0, 1, 1, 0])
>>> lab = Datalab(data={"X": X, "y": y}, label_name="y")
>>> lab.find_issues(features=X)
.. note::
You can pass both ``pred_probs`` and ``features`` to :py:meth:`find_issues` for a more comprehensive audit.
- Passing a ``knn_graph``:
.. code-block:: python
>>> from sklearn.neighbors import NearestNeighbors
>>> import numpy as np
>>> from cleanlab import Datalab
>>> X = np.array([[0, 1], [1, 1], [2, 2], [2, 0]])
>>> y = np.array([0, 1, 1, 0])
>>> nbrs = NearestNeighbors(n_neighbors=2, metric="euclidean").fit(X)
>>> knn_graph = nbrs.kneighbors_graph(mode="distance")
>>> knn_graph # Pass this to Datalab
<4x4 sparse matrix of type '<class 'numpy.float64'>'
with 8 stored elements in Compressed Sparse Row format>
>>> knn_graph.toarray() # DO NOT PASS knn_graph.toarray() to Datalab, only pass the sparse matrix itself
array([[0. , 1. , 2.23606798, 0. ],
[1. , 0. , 1.41421356, 0. ],
[0. , 1.41421356, 0. , 2. ],
[0. , 1.41421356, 2. , 0. ]])
>>> lab = Datalab(data={"X": X, "y": y}, label_name="y")
>>> lab.find_issues(knn_graph=knn_graph)
- Configuring issue types:
Suppose you want to only consider label issues. Just pass a dictionary with the key "label" and an empty dictionary as the value (to use default label issue parameters).
.. code-block:: python
>>> issue_types = {"label": {}}
>>> # lab.find_issues(pred_probs=pred_probs, issue_types=issue_types)
If you are advanced user who wants greater control, you can pass keyword arguments to the issue manager that handles the label issues.
For example, if you want to pass the keyword argument "clean_learning_kwargs"
to the constructor of the :py:class:`LabelIssueManager <cleanlab.datalab.internal.issue_manager.label.LabelIssueManager>`, you would pass:
.. code-block:: python
>>> issue_types = {
... "label": {
... "clean_learning_kwargs": {
... "prune_method": "prune_by_noise_rate",
... },
... },
... }
>>> # lab.find_issues(pred_probs=pred_probs, issue_types=issue_types)
"""
if issue_types is not None and not issue_types:
warnings.warn(
"No issue types were specified so no issues will be found in the dataset. Set `issue_types` as None to consider a default set of issues."
)
return None
issue_finder = issue_finder_factory(self._imagelab)(
datalab=self, task=self.task, verbosity=self.verbosity
)
issue_finder.find_issues(
pred_probs=pred_probs,
features=features,
knn_graph=knn_graph,
issue_types=issue_types,
)
if self.verbosity:
print(
f"\nAudit complete. {self.data_issues.issue_summary['num_issues'].sum()} issues found in the dataset."
)
def report(
self,
*,
num_examples: int = 5,
verbosity: Optional[int] = None,
include_description: bool = True,
show_summary_score: bool = False,
show_all_issues: bool = False,
) -> None:
"""Prints informative summary of all issues.
Parameters
----------
num_examples :
Number of examples to show for each type of issue.
The report shows the top `num_examples` instances in the dataset that suffer the most from each type of issue.
verbosity :
Higher verbosity levels add more information to the report.
include_description :
Whether or not to include a description of each issue type in the report.
Consider setting this to ``False`` once you're familiar with how each issue type is defined.
show_summary_score :
Whether or not to include the overall severity score of each issue type in the report.
These scores are not comparable across different issue types,
see the ``issue_summary`` documentation to learn more.
show_all_issues :
Whether or not the report should show all issue types that were checked for, or only the types of issues detected in the dataset.
With this set to ``True``, the report may include more types of issues that were not detected in the dataset.
See Also
--------
For advanced usage, see documentation for the
:py:class:`Reporter <cleanlab.datalab.internal.report.Reporter>` class.
"""
if verbosity is None:
verbosity = self.verbosity
if self.data_issues.issue_summary.empty:
print("Please specify some `issue_types` in datalab.find_issues() to see a report.\n")
return
reporter = report_factory(self._imagelab)(
data_issues=self.data_issues,
task=self.task,
verbosity=verbosity,
include_description=include_description,
show_summary_score=show_summary_score,
show_all_issues=show_all_issues,
imagelab=self._imagelab,
)
reporter.report(num_examples=num_examples)
@property
def issues(self) -> pd.DataFrame:
"""Issues found in each example from the dataset."""
return self.data_issues.issues
@issues.setter
def issues(self, issues: pd.DataFrame) -> None:
self.data_issues.issues = issues
@property
def issue_summary(self) -> pd.DataFrame:
"""Summary of issues found in the dataset and the overall severity of each type of issue.
Each type of issue has a summary score, which is usually defined as an average of
per-example issue-severity scores (over all examples in the dataset).
So these summary scores are not directly tied to the number of examples estimated to exhibit
a particular type of issue. Issue-severity (ie. quality of each example) is measured differently for each issue type,
and these per-example scores are only comparable across different examples for the same issue-type, but are not comparable across different issue types.
For instance, label quality might be scored via estimated likelihood of the given label,
whereas outlier quality might be scored via distance to K-nearest-neighbors in feature space (fundamentally incomparable quantities).
For some issue types, the summary score is not an average of per-example scores, but rather a global statistic of the dataset
(eg. for `non_iid` issue type, the p-value for hypothesis test that data are IID).
In summary, you can compare these summary scores across datasets for the same issue type, but never compare them across different issue types.
Examples
-------
If checks for "label" and "outlier" issues were run,
then the issue summary will look something like this:
>>> datalab.issue_summary
issue_type score
outlier 0.123
label 0.456
"""
return self.data_issues.issue_summary
@issue_summary.setter
def issue_summary(self, issue_summary: pd.DataFrame) -> None:
self.data_issues.issue_summary = issue_summary
@property
def info(self) -> Dict[str, Dict[str, Any]]:
"""Information and statistics about the dataset issues found.
Examples
-------
If checks for "label" and "outlier" issues were run,
then the info will look something like this:
>>> datalab.info
{
"label": {
"given_labels": [0, 1, 0, 1, 1, 1, 1, 1, 0, 1, ...],
"predicted_label": [0, 0, 0, 1, 0, 1, 0, 1, 0, 1, ...],
...,
},
"outlier": {
"nearest_neighbor": [3, 7, 1, 2, 8, 4, 5, 9, 6, 0, ...],
"distance_to_nearest_neighbor": [0.123, 0.789, 0.456, ...],
...,
},
}
"""
return self.data_issues.info
@info.setter
def info(self, info: Dict[str, Dict[str, Any]]) -> None:
self.data_issues.info = info
def get_issues(self, issue_name: Optional[str] = None) -> pd.DataFrame:
"""
Use this after finding issues to see which examples suffer from which types of issues.
Parameters
----------
issue_name : str or None
The type of issue to focus on. If `None`, returns full DataFrame summarizing all of the types of issues detected in each example from the dataset.
Raises
------
ValueError
If `issue_name` is not a type of issue previously considered in the audit.
Returns
-------
specific_issues :
A DataFrame where each row corresponds to an example from the dataset and columns specify:
whether this example exhibits a particular type of issue, and how severely (via a numeric quality score where lower values indicate more severe instances of the issue).
The quality scores lie between 0-1 and are directly comparable between examples (for the same issue type), but not across different issue types.
Additional columns may be present in the DataFrame depending on the type of issue specified.
"""
# Validate issue_name
if issue_name is not None and issue_name not in self.list_possible_issue_types():
raise ValueError(
f"""Invalid issue_name: {issue_name}. Please specify a valid issue_name from the list of possible issue types.
Either, specify one of the following: {self.list_possible_issue_types()}
or set issue_name as None to get all issue types.
"""
)
return self.data_issues.get_issues(issue_name=issue_name)
def get_issue_summary(self, issue_name: Optional[str] = None) -> pd.DataFrame:
"""Summarize the issues found in dataset of a particular type,
including how severe this type of issue is overall across the dataset.
See the documentation of the ``issue_summary`` attribute to learn more.
Parameters
----------
issue_name :
Name of the issue type to summarize. If `None`, summarizes each of the different issue types previously considered in the audit.
Returns
-------
issue_summary :
DataFrame where each row corresponds to a type of issue, and columns quantify:
the number of examples in the dataset estimated to exhibit this type of issue,
and the overall severity of the issue across the dataset (via a numeric quality score where lower values indicate that the issue is overall more severe).
The quality scores lie between 0-1 and are directly comparable between multiple datasets (for the same issue type), but not across different issue types.
"""
return self.data_issues.get_issue_summary(issue_name=issue_name)
def get_info(self, issue_name: Optional[str] = None) -> Dict[str, Any]:
"""Get the info for the issue_name key.
This function is used to get the info for a specific issue_name. If the info is not computed yet, it will raise an error.
Parameters
----------
issue_name :
The issue name for which the info is required.
Returns
-------
:py:meth:`info <cleanlab.datalab.internal.data_issues.DataIssues.get_info>` :
The info for the issue_name.
"""
return self.data_issues.get_info(issue_name)
def list_possible_issue_types(self) -> List[str]:
"""Returns a list of all registered issue types.
Any issue type that is not in this list cannot be used in the :py:meth:`find_issues` method.
See Also
--------
:py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
"""
possible_issue_types = _list_possible_issue_types(task=self.task)
if self._imagelab is not None:
possible_issue_types.extend(DEFAULT_CLEANVISION_ISSUES.keys())
return possible_issue_types
def list_default_issue_types(self) -> List[str]:
"""Returns a list of the issue types that are run by default
when :py:meth:`find_issues` is called without specifying `issue_types`.
See Also
--------
:py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
"""
default_issue_types = _list_default_issue_types(task=self.task)
if self._imagelab is not None:
default_issue_types.extend(DEFAULT_CLEANVISION_ISSUES.keys())
return default_issue_types
def save(self, path: str, force: bool = False) -> None:
"""Saves this Datalab object to file (all files are in folder at `path/`).
We do not guarantee saved Datalab can be loaded from future versions of cleanlab.
Parameters
----------
path :
Folder in which all information about this Datalab should be saved.
force :
If ``True``, overwrites any existing files in the folder at `path`. Use this with caution!
NOTE
----
You have to save the Dataset yourself separately if you want it saved to file.
"""
_Serializer.serialize(path=path, datalab=self, force=force)
save_message = f"Saved Datalab to folder: {path}"
print(save_message)
@staticmethod
def load(path: str, data: Optional[Dataset] = None) -> "Datalab":
"""Loads Datalab object from a previously saved folder.
Parameters
----------
`path` :
Path to the folder previously specified in ``Datalab.save()``.
`data` :
The dataset used to originally construct the Datalab.
Remember the dataset is not saved as part of the Datalab,
you must save/load the data separately.
Returns
-------
`datalab` :
A Datalab object that is identical to the one originally saved.
"""
datalab = _Serializer.deserialize(path=path, data=data)
load_message = f"Datalab loaded from folder: {path}"
print(load_message)
return datalab
cleanlab/datalab/internal/__init__.py
View File
cleanlab/datalab/internal/adapter/__init__.py
View File
cleanlab/datalab/internal/adapter/constants.py
+19
View File
@@ -0,0 +1,19 @@
SPURIOUS_CORRELATION_ISSUE = {
"spurious_correlations": {
"threshold": 0.3,
},
} # Default issue type for spurious correlation in Datalab
DEFAULT_CLEANVISION_ISSUES = {
"dark": {},
"light": {},
"low_information": {
"threshold": 0.15,
},
"odd_aspect_ratio": {},
"odd_size": {},
"grayscale": {},
"blurry": {},
} # list of CleanVision issue types considered by default in Datalab may differ from CleanVision package default
IMAGELAB_ISSUES_MAX_PREVALENCE = 0.1
cleanlab/datalab/internal/adapter/imagelab.py
+430
View File
@@ -0,0 +1,430 @@
"""An internal wrapper around the Imagelab class from the CleanVision package to incorporate it into Datalab.
This allows low-quality images to be detected alongside other issues in computer vision datasets.
The methods/classes in this module are just intended for internal use.
"""
import warnings
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Type, cast, Union
import numpy as np
import numpy.typing as npt
import pandas as pd
from scipy.sparse import csr_matrix
from cleanlab.datalab.internal.adapter.constants import (
DEFAULT_CLEANVISION_ISSUES,
IMAGELAB_ISSUES_MAX_PREVALENCE,
SPURIOUS_CORRELATION_ISSUE,
)
from cleanlab.datalab.internal.data import Data
from cleanlab.datalab.internal.data_issues import DataIssues, _InfoStrategy
from cleanlab.datalab.internal.issue_finder import IssueFinder
from cleanlab.datalab.internal.report import Reporter
from cleanlab.datalab.internal.task import Task
from cleanlab.datalab.internal.spurious_correlation import SpuriousCorrelations
if TYPE_CHECKING: # pragma: no cover
from cleanvision import Imagelab
from datasets.arrow_dataset import Dataset
def create_imagelab(dataset: "Dataset", image_key: Optional[str]) -> Optional["Imagelab"]:
"""Creates Imagelab instance for running CleanVision checks. CleanVision checks are only supported for
huggingface datasets as of now.
Parameters
----------
dataset: datasets.Dataset
Huggingface dataset used by Imagelab
image_key: str
key for image feature in the huggingface dataset
Returns
-------
Imagelab
"""
imagelab = None
if not image_key:
return imagelab
try:
from cleanvision import Imagelab
from datasets.arrow_dataset import Dataset
if isinstance(dataset, Dataset):
imagelab = Imagelab(hf_dataset=dataset, image_key=image_key)
else:
raise ValueError(
"For now, only huggingface datasets are supported for running cleanvision checks inside cleanlab. You can easily convert most datasets to the huggingface dataset format."
)
except ImportError:
raise ImportError(
"Cannot import required image packages. Please install them via: `pip install cleanlab[image]` or just install cleanlab with "
"all optional dependencies via: `pip install cleanlab[all]`"
)
return imagelab
class ImagelabDataIssuesAdapter(DataIssues):
"""
Class that collects and stores information and statistics on issues found in a dataset.
Parameters
----------
data :
The data object for which the issues are being collected.
strategy :
Strategy used for processing info dictionaries.
Parameters
----------
issues : pd.DataFrame
Stores information about each individual issue found in the data,
on a per-example basis.
issue_summary : pd.DataFrame
Summarizes the overall statistics for each issue type.
info : dict
A dictionary that contains information and statistics about the data and each issue type.
"""
def __init__(self, data: Data, strategy: Type[_InfoStrategy]) -> None:
super().__init__(data, strategy)
def _update_issues_imagelab(self, imagelab: "Imagelab", overlapping_issues: List[str]) -> None:
overwrite_columns = [f"is_{issue_type}_issue" for issue_type in overlapping_issues]
overwrite_columns.extend([f"{issue_type}_score" for issue_type in overlapping_issues])
if overwrite_columns:
warnings.warn(
f"Overwriting columns {overwrite_columns} in self.issues with "
f"columns from imagelab."
)
self.issues.drop(columns=overwrite_columns, inplace=True)
new_columnns = list(set(imagelab.issues.columns).difference(self.issues.columns))
self.issues = self.issues.join(imagelab.issues[new_columnns], how="outer")
def filter_based_on_max_prevalence(self, issue_summary: pd.DataFrame, max_num: int):
removed_issues = issue_summary[issue_summary["num_images"] > max_num]["issue_type"].tolist()
if len(removed_issues) > 0:
print(
f"Removing {', '.join(removed_issues)} from potential issues in the dataset as it exceeds max_prevalence={IMAGELAB_ISSUES_MAX_PREVALENCE}"
)
return issue_summary[issue_summary["num_images"] <= max_num].copy()
def collect_issues_from_imagelab(self, imagelab: "Imagelab", issue_types: List[str]) -> None:
"""
Collect results from Imagelab and update datalab.issues and datalab.issue_summary
Parameters
----------
imagelab: Imagelab
Imagelab instance that run all the checks for image issue types
"""
overlapping_issues = list(set(self.issue_summary["issue_type"]) & set(issue_types))
self._update_issues_imagelab(imagelab, overlapping_issues)
if overlapping_issues:
warnings.warn(
f"Overwriting {overlapping_issues} rows in self.issue_summary from imagelab."
)
self.issue_summary = self.issue_summary[
~self.issue_summary["issue_type"].isin(overlapping_issues)
]
imagelab_summary_copy = imagelab.issue_summary.copy()
imagelab_summary_copy = self.filter_based_on_max_prevalence(
imagelab_summary_copy, int(IMAGELAB_ISSUES_MAX_PREVALENCE * len(self.issues))
)
imagelab_summary_copy.rename({"num_images": "num_issues"}, axis=1, inplace=True)
self.issue_summary = pd.concat(
[self.issue_summary, imagelab_summary_copy], axis=0, ignore_index=True
)
for issue_type in issue_types:
self._update_issue_info(issue_type, imagelab.info[issue_type])
def get_info(self, issue_name: Optional[str] = None) -> Dict[str, Any]:
# Extend method for fetching info about spurious correlations
if issue_name != "spurious_correlations":
return super().get_info(issue_name)
correlations_info = self.info.get("spurious_correlations", {})
if not correlations_info:
raise ValueError(
"Spurious correlations have not been calculated. Run find_issues() first."
)
return correlations_info
class CorrelationVisualizer:
"""Class to visualize images corresponding to the extreme (minimum and maximum) individual
scores for each of the detected correlated properties.
"""
def __init__(self):
# Wrapper for VizManager that's from the optional cleanvision dependency
try:
from cleanvision.utils.viz_manager import VizManager
self.viz_manager = VizManager
except ImportError:
raise ImportError(
"cleanvision is required for correlation visualization. Please install it to use this feature."
)
def visualize(
self, images: List, title_info: Dict, ncols: int = 2, cell_size: tuple = (2, 2)
) -> None:
self.viz_manager.individual_images(
images=images,
title_info=title_info,
ncols=ncols,
cell_size=cell_size,
)
class CorrelationReporter:
"""Class to report spurious correlations between image features and class labels detected in the data.
If no spurious correlations are found, the class will not report anything.
"""
def __init__(self, data_issues: "DataIssues", imagelab: "Imagelab"):
self.imagelab: "Imagelab" = imagelab
self.data_issues = data_issues
self.threshold = data_issues.get_info("spurious_correlations").get("threshold")
if not self.threshold:
raise ValueError(
"Spurious correlations have not been calculated. Run find_issues() first."
)
self.visualizer = CorrelationVisualizer()
def report(self) -> None:
"""Reports spurious correlations between image features and class labels detected in the data,
if any are found.
"""
correlated_properties = self._get_correlated_properties()
if not correlated_properties:
return
self._print_correlation_summary()
correlations_df = cast(
pd.DataFrame, self.data_issues.get_info("spurious_correlations").get("correlations_df")
)
filtered_correlations_df = self._get_filtered_correlated_properties(
correlations_df, correlated_properties
)
print(filtered_correlations_df.to_string(index=False) + "\n")
self._visualize_extremes(correlated_properties, self.data_issues)
def _print_correlation_summary(self) -> None:
print("\n\n")
report_correlation_header = "Summary of (potentially spurious) correlations between image properties and class labels detected in the data:\n\n"
report_correlation_metric = "Lower scores below correspond to images properties that are more strongly correlated with the class labels.\n\n"
print(report_correlation_header + report_correlation_metric)
def _visualize_extremes(
self, correlated_properties: List[str], data_issues: "DataIssues"
) -> None:
report_extremal_images = "Here are the images corresponding to the extreme (minimum and maximum) individual scores for each of the detected correlated properties:\n\n"
print(report_extremal_images)
issues = data_issues.get_issues()
correlated_indices = {
prop: [issues[prop].idxmin(), issues[prop].idxmax()] for prop in correlated_properties
}
self._visualize(correlated_indices, issues)
def _visualize(self, correlated_indices: Dict[str, List[Any]], issues: pd.DataFrame) -> None:
for prop, image_ids in correlated_indices.items():
print(
f"{'Images with minimum and maximum individual scores for ' + prop.replace('_score', '') + ' issue:'}\n"
)
title_info = {"scores": [f"score: {issues.loc[id, prop]:.4f}" for id in image_ids]}
self.visualizer.visualize(
images=[self.imagelab._dataset[id] for id in image_ids],
title_info=title_info,
)
def _get_correlated_properties(self) -> List[str]:
correlations_df = self.data_issues.get_info("spurious_correlations").get("correlations_df")
if correlations_df is None or correlations_df.empty:
return []
return correlations_df.query("score < @self.threshold")["property"].tolist()
def _get_filtered_correlated_properties(
self, correlations_df: pd.DataFrame, correlated_properties: List[str]
) -> pd.DataFrame:
query_str = "property in @correlated_properties"
filtered_correlations_df = correlations_df.query(query_str)
filtered_correlations_df.loc[:, "property"] = filtered_correlations_df["property"].apply(
lambda x: x.replace("_score", "")
)
return filtered_correlations_df
class ImagelabReporterAdapter(Reporter):
def __init__(
self,
data_issues: "DataIssues",
imagelab: "Imagelab",
task: Task,
verbosity: int = 1,
include_description: bool = True,
show_summary_score: bool = False,
show_all_issues: bool = False,
):
super().__init__(
data_issues=data_issues,
task=task,
verbosity=verbosity,
include_description=include_description,
show_summary_score=show_summary_score,
show_all_issues=show_all_issues,
)
self.imagelab = imagelab
self.correlation_reporter: Optional[CorrelationReporter] = None
try:
self.correlation_reporter = CorrelationReporter(data_issues, imagelab)
except:
# Spurious correlations have not been calculated
self.correlation_reporter = None
def report(self, num_examples: int) -> None:
super().report(num_examples)
self._report_imagelab(num_examples)
# Only report spurious correlations if they've been calculated & detected
if self.correlation_reporter is not None:
self.correlation_reporter.report()
def _report_imagelab(self, num_examples):
print("\n\n")
self.imagelab.report(
num_images=num_examples,
max_prevalence=IMAGELAB_ISSUES_MAX_PREVALENCE,
print_summary=False,
verbosity=0,
show_id=True,
)
class ImagelabIssueFinderAdapter(IssueFinder):
def __init__(self, datalab, task, verbosity):
super().__init__(datalab, task, verbosity)
self.imagelab = self.datalab._imagelab
def _get_imagelab_issue_types(self, issue_types, **kwargs):
if issue_types is None:
return DEFAULT_CLEANVISION_ISSUES
if "image_issue_types" not in issue_types:
return None
issue_types_copy = {}
for issue_type, params in issue_types["image_issue_types"].items():
if not params:
issue_types_copy[issue_type] = DEFAULT_CLEANVISION_ISSUES[issue_type]
else:
issue_types_copy[issue_type] = params
return issue_types_copy
def find_issues(
self,
*,
pred_probs: Optional[np.ndarray] = None,
features: Optional[npt.NDArray] = None,
knn_graph: Optional[csr_matrix] = None,
issue_types: Optional[Dict[str, Any]] = None,
) -> None:
issue_types_to_ignore_in_datalab = ["image_issue_types", "spurious_correlations"]
datalab_issue_types = (
{k: v for k, v in issue_types.items() if k not in issue_types_to_ignore_in_datalab}
if issue_types
else issue_types
)
super().find_issues(
pred_probs=pred_probs,
features=features,
knn_graph=knn_graph,
issue_types=datalab_issue_types,
)
issue_types_copy = self._get_imagelab_issue_types(issue_types)
if issue_types_copy:
try:
if self.verbosity:
print(f'Finding {", ".join(issue_types_copy.keys())} images ...')
self.imagelab.find_issues(issue_types=issue_types_copy, verbose=False)
self.datalab.data_issues.collect_statistics(self.imagelab)
self.datalab.data_issues.collect_issues_from_imagelab(
self.imagelab, issue_types_copy.keys()
)
except Exception as e:
print(f"Error in checking for image issues: {e}")
# if issue_types is neither 'None' nor empty dictionary (non-trivial) but
# there is no mention of 'spurious_correlations', we return.
if issue_types and "spurious_correlations" not in issue_types:
return
# Check if all vision issue scores are computed
imagelab_columns = self.imagelab.issues.columns.tolist()
if all(
default_cleanvision_issue + "_score" not in imagelab_columns
for default_cleanvision_issue in DEFAULT_CLEANVISION_ISSUES.keys()
):
print("Skipping spurious correlations check: Image property scores not available.")
print(
"To include this check, run find_issues() without parameters to compute all scores."
)
return
# Spurious correlation part must be run
print("Finding spurious correlation issues in the dataset ...")
# the else part of the following must contain 'spurious_correlations' key
spurious_correlation_issue_types = (
SPURIOUS_CORRELATION_ISSUE["spurious_correlations"]
if not issue_types
else issue_types["spurious_correlations"]
)
# If threshold is not expicitly given (e.g. lab.find_issues("issue_types={"spurious_correlations": {}"))
# we extract the default value from SPURIOUS_CORRELATION_ISSUE
spurious_correlation_issue_threshold = spurious_correlation_issue_types.get(
"threshold", SPURIOUS_CORRELATION_ISSUE["spurious_correlations"]["threshold"]
)
try:
if self.datalab.has_labels:
self.datalab.data_issues.info["spurious_correlations"] = (
handle_spurious_correlations(
imagelab_issues=self.imagelab.issues,
labels=self.datalab.labels,
threshold=spurious_correlation_issue_threshold,
)
)
except Exception as e:
print(f"Error in checking for spurious correlations: {e}")
def handle_spurious_correlations(
*,
imagelab_issues: pd.DataFrame,
labels: Union[np.ndarray, List[List[int]]],
threshold: float,
**_,
) -> Dict[str, Any]:
imagelab_columns = imagelab_issues.columns.tolist()
score_columns = [col for col in imagelab_columns if col.endswith("_score")]
correlations_df = SpuriousCorrelations(
data=imagelab_issues[score_columns], labels=labels
).calculate_correlations()
return {
"correlations_df": correlations_df,
"threshold": threshold,
}
cleanlab/datalab/internal/data.py
+431
View File
@@ -0,0 +1,431 @@
"""Classes and methods for datasets that are loaded into Datalab."""
import os
from typing import Any, Callable, Dict, List, Mapping, Optional, Union, cast, TYPE_CHECKING, Tuple
from cleanlab.datalab.internal.task import Task
try:
import datasets
except ImportError as error:
raise ImportError(
"Cannot import datasets package. "
"Please install it and try again, or just install cleanlab with "
"all optional dependencies via: `pip install 'cleanlab[all]'`"
) from error
from abc import ABC, abstractmethod
import numpy as np
import pandas as pd
from datasets.arrow_dataset import Dataset
from datasets import ClassLabel
# Import Column types for compatibility with datasets 4.0.0+
try:
from datasets.arrow_dataset import Column
from datasets.iterable_dataset import IterableColumn
except ImportError:
# For backwards compatibility with older datasets versions
Column = None
IterableColumn = None
from cleanlab.internal.validation import labels_to_array, labels_to_list_multilabel
if TYPE_CHECKING: # pragma: no cover
DatasetLike = Union[Dataset, pd.DataFrame, Dict[str, Any], List[Dict[str, Any]], str]
class DataFormatError(ValueError):
"""Exception raised when the data is not in a supported format."""
def __init__(self, data: Any):
self.data = data
message = (
f"Unsupported data type: {type(data)}\n"
"Supported types: "
"datasets.Dataset, pandas.DataFrame, dict, list, str"
)
super().__init__(message)
class DatasetDictError(ValueError):
"""Exception raised when a DatasetDict is passed to Datalab.
Usually, this means that a dataset identifier was passed to Datalab, but
the dataset is a DatasetDict, which contains multiple splits of the dataset.
"""
def __init__(self):
message = (
"Please pass a single dataset, not a DatasetDict. "
"Try specifying a split, e.g. `dataset = load_dataset('dataset', split='train')` "
"then pass `dataset` to Datalab."
)
super().__init__(message)
class DatasetLoadError(ValueError):
"""Exception raised when a dataset cannot be loaded.
Parameters
----------
dataset_type: type
The type of dataset that failed to load.
"""
def __init__(self, dataset_type: type):
message = f"Failed to load dataset from {dataset_type}.\n"
super().__init__(message)
class Data:
"""
Class that holds and validates datasets for Datalab.
Internally, the data is stored as a datasets.Dataset object and the labels
are integers (ranging from 0 to K-1, where K is the number of classes) stored
in a numpy array.
Parameters
----------
data :
Dataset to be audited by Datalab.
Several formats are supported, which will internally be converted to a Dataset object.
Supported formats:
- datasets.Dataset
- pandas.DataFrame
- dict
- keys are strings
- values are arrays or lists of equal length
- list
- list of dictionaries with the same keys
- str
- path to a local file
- Text (.txt)
- CSV (.csv)
- JSON (.json)
- or a dataset identifier on the Hugging Face Hub
It checks if the string is a path to a file that exists locally, and if not,
it assumes it is a dataset identifier on the Hugging Face Hub.
label_name : Union[str, List[str]]
Name of the label column in the dataset.
task :
The task associated with the dataset. This is used to determine how to
to format the labels.
Note:
- If the task is a classification task, the labels
will be mapped to integers, e.g. [0, 1, ..., K-1] where K is the number
of classes. If the task is a regression task, the labels will not be
mapped to integers.
- If the task is a multilabel task, the labels will be formatted as a
list of lists, e.g. [[0, 1], [1, 2], [0, 2]] where each sublist contains
the labels for a single example. If the task is not a multilabel task,
the labels will be formatted as a 1D numpy array.
Warnings
--------
Optional dependencies:
- datasets :
Dataset, DatasetDict and load_dataset are imported from datasets.
This is an optional dependency of cleanlab, but is required for
:py:class:`Datalab <cleanlab.datalab.datalab.Datalab>` to work.
"""
def __init__(
self,
data: "DatasetLike",
task: Task,
label_name: Optional[str] = None,
) -> None:
self._validate_data(data)
self._data = self._load_data(data)
self._data_hash = hash(self._data)
self.labels: Label
label_class = MultiLabel if task.is_multilabel else MultiClass
map_to_int = task.is_classification
self.labels = label_class(data=self._data, label_name=label_name, map_to_int=map_to_int)
def _load_data(self, data: "DatasetLike") -> Dataset:
"""Checks the type of dataset and uses the correct loader method and
assigns the result to the data attribute."""
dataset_factory_map: Dict[type, Callable[..., Dataset]] = {
Dataset: lambda x: x,
pd.DataFrame: Dataset.from_pandas,
dict: self._load_dataset_from_dict,
list: self._load_dataset_from_list,
str: self._load_dataset_from_string,
}
if not isinstance(data, tuple(dataset_factory_map.keys())):
raise DataFormatError(data)
return dataset_factory_map[type(data)](data)
def __len__(self) -> int:
return len(self._data)
def __eq__(self, other) -> bool:
if isinstance(other, Data):
# Equality checks
hashes_are_equal = self._data_hash == other._data_hash
labels_are_equal = self.labels == other.labels
return all([hashes_are_equal, labels_are_equal])
return False
def __hash__(self) -> int:
return self._data_hash
@property
def class_names(self) -> List[str]:
return self.labels.class_names
@property
def has_labels(self) -> bool:
"""Check if labels are available."""
return self.labels.is_available
@staticmethod
def _validate_data(data) -> None:
if isinstance(data, datasets.DatasetDict):
raise DatasetDictError()
if not isinstance(data, (Dataset, pd.DataFrame, dict, list, str)):
raise DataFormatError(data)
@staticmethod
def _load_dataset_from_dict(data_dict: Dict[str, Any]) -> Dataset:
try:
return Dataset.from_dict(data_dict)
except Exception as error:
raise DatasetLoadError(dict) from error
@staticmethod
def _load_dataset_from_list(data_list: List[Dict[str, Any]]) -> Dataset:
try:
return Dataset.from_list(data_list)
except Exception as error:
raise DatasetLoadError(list) from error
@staticmethod
def _load_dataset_from_string(data_string: str) -> Dataset:
if not os.path.exists(data_string):
try:
dataset = datasets.load_dataset(data_string)
return cast(Dataset, dataset)
except Exception as error:
raise DatasetLoadError(str) from error
factory: Dict[str, Callable[[str], Any]] = {
".txt": Dataset.from_text,
".csv": Dataset.from_csv,
".json": Dataset.from_json,
}
extension = os.path.splitext(data_string)[1]
if extension not in factory:
raise DatasetLoadError(type(data_string))
dataset = factory[extension](data_string)
dataset_cast = cast(Dataset, dataset)
return dataset_cast
class Label(ABC):
"""
Class to represent labels in a dataset.
It stores the labels as a numpy array and maps them to integers if necessary.
If a mapping is not necessary, e.g. for regression tasks, the mapping will be an empty dictionary.
Parameters
----------
data :
A Hugging Face Dataset object.
label_name : str
Name of the label column in the dataset.
map_to_int : bool
Whether to map the labels to integers, e.g. [0, 1, ..., K-1] where K is the number of classes.
If False, the labels are not mapped to integers, e.g. for regression tasks.
"""
def __init__(
self, *, data: Dataset, label_name: Optional[str] = None, map_to_int: bool = True
) -> None:
self._data = data
self.label_name = label_name
self.labels = labels_to_array([])
self.label_map: Mapping[Union[str, int], Any] = {}
if label_name is not None:
self.labels, self.label_map = self._extract_labels(data, label_name, map_to_int)
self._validate_labels()
def __len__(self) -> int:
if self.labels is None:
return 0
return len(self.labels)
def __eq__(self, __value: object) -> bool:
if isinstance(__value, Label):
labels_are_equal = np.array_equal(self.labels, __value.labels)
names_are_equal = self.label_name == __value.label_name
maps_are_equal = self.label_map == __value.label_map
return all([labels_are_equal, names_are_equal, maps_are_equal])
return False
def __getitem__(self, __index: Union[int, slice, np.ndarray]) -> np.ndarray:
return self.labels[__index]
def __bool__(self) -> bool:
return self.is_available
@property
def class_names(self) -> List[str]:
"""A list of class names that are present in the dataset.
Without labels, this will return an empty list.
"""
return list(self.label_map.values())
@property
def is_available(self) -> bool:
"""Check if labels are available."""
empty_labels = self.labels is None or len(self.labels) == 0
empty_label_map = self.label_map is None or len(self.label_map) == 0
return not (empty_labels or empty_label_map)
def _validate_labels(self) -> None:
if self.label_name not in self._data.column_names:
raise ValueError(f"Label column '{self.label_name}' not found in dataset.")
labels = self._data[self.label_name]
error_message = (
f"Expected labels to be numpy array, list, or Column type, got {type(labels)}"
)
assert _is_valid_label_column(labels), error_message
assert len(labels) == len(self._data)
@abstractmethod
def _extract_labels(self, *args, **kwargs) -> Any:
"""Extract labels from the dataset and formats them"""
raise NotImplementedError
class MultiLabel(Label):
def __init__(self, data, label_name, map_to_int):
super().__init__(data=data, label_name=label_name, map_to_int=map_to_int)
def _extract_labels(
self, data: Dataset, label_name: str, map_to_int: bool
) -> Tuple[List[List[int]], Dict[int, Any]]:
# Convert Column types to list for compatibility with validation functions
raw_labels = data[label_name]
converted_labels = _convert_column_to_list(raw_labels)
labels: List[List[int]] = labels_to_list_multilabel(converted_labels)
# label_map needs to be lexicographically sorted. np.unique should sort it
unique_labels = np.unique([x for ele in labels for x in ele])
label_map = {label: i for i, label in enumerate(unique_labels)}
formatted_labels = [[label_map[item] for item in label] for label in labels]
inverse_map = {i: label for label, i in label_map.items()}
return formatted_labels, inverse_map
class MultiClass(Label):
def __init__(self, data, label_name, map_to_int):
super().__init__(data=data, label_name=label_name, map_to_int=map_to_int)
def _extract_labels(self, data: Dataset, label_name: str, map_to_int: bool):
"""
Picks out labels from the dataset and formats them to be [0, 1, ..., K-1]
where K is the number of classes. Also returns a mapping from the formatted
labels to the original labels in the dataset.
Note: This function is not meant to be used directly. It is used by
``cleanlab.data.Data`` to extract the formatted labels from the dataset
and stores them as attributes.
Parameters
----------
data : datasets.Dataset
A Hugging Face Dataset object.
label_name : str
Name of the column in the dataset that contains the labels.
map_to_int : bool
Whether to map the labels to integers, e.g. [0, 1, ..., K-1] where K is the number of classes.
If False, the labels are not mapped to integers, e.g. for regression tasks.
Returns
-------
formatted_labels : np.ndarray
Labels in the format [0, 1, ..., K-1] where K is the number of classes.
inverse_map : dict
Mapping from the formatted labels to the original labels in the dataset.
"""
labels = labels_to_array(data[label_name]) # type: ignore[assignment]
if labels.ndim != 1:
raise ValueError("labels must be 1D numpy array.")
if not map_to_int:
# Don't map labels to integers, e.g. for regression tasks
return labels, {}
label_name_feature = data.features[label_name]
if isinstance(label_name_feature, ClassLabel):
label_map = {
label: label_name_feature.str2int(label) for label in label_name_feature.names
}
formatted_labels = labels
else:
label_map = {label: i for i, label in enumerate(np.unique(labels))}
formatted_labels = np.vectorize(label_map.get, otypes=[int])(labels)
inverse_map = {i: label for label, i in label_map.items()}
return formatted_labels, inverse_map
def _is_valid_label_column(labels: Any) -> bool:
"""Helper function to check if labels are a valid type including datasets 4.0.0+ Column types.
Parameters
----------
labels : Any
The labels object to validate.
Returns
-------
bool
True if labels are a valid type (numpy array, list, or Column types).
"""
valid_types = [np.ndarray, list]
if Column is not None:
valid_types.append(Column)
if IterableColumn is not None:
valid_types.append(IterableColumn)
return isinstance(labels, tuple(valid_types))
def _convert_column_to_list(labels):
"""Helper function to convert Column types to list for compatibility with validation functions.
Parameters
----------
labels : Any
The labels object to convert.
Returns
-------
list or original type
Converted to list if it's a Column type, otherwise returns original object.
"""
if Column is not None and isinstance(labels, Column):
return list(labels)
elif IterableColumn is not None and isinstance(labels, IterableColumn):
return list(labels)
return labels
cleanlab/datalab/internal/data_issues.py
+413
View File
@@ -0,0 +1,413 @@
"""
Module for the :py:class:`DataIssues` class, which serves as a central repository for storing
information and statistics about issues found in a dataset.
It collects information from various
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`
instances and keeps track of each issue, a summary for each type of issue,
related information and statistics about the issues.
The collected information can be accessed using the
`~cleanlab.datalab.internal.data_issues.DataIssues.get_info` method.
We recommend using that method instead of this module, which is just intended for internal use.
"""
from __future__ import annotations
import warnings
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Type, Union
import numpy as np
import pandas as pd
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.internal.data import Data
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanvision import Imagelab
class _InfoStrategy(ABC):
"""
Abstract base class for strategies that fetch information about data issues.
Subclasses must implement the `get_info` method, which takes a `Data` object, a dictionary of
information about data issues, and an optional issue name, and returns a dictionary of
information about the specified issue, augmented with dataset about the dataset as a whole.
This class also provides a helper method, `_get_info_helper`, which takes an information
dictionary and an optional issue name, and returns a copy of the information dictionary for
the specified issue. If the issue name is `None`, this method returns `None`.
"""
@staticmethod
@abstractmethod
def get_info(
data: Data,
info: Dict[str, Dict[str, Any]],
issue_name: Optional[str] = None,
) -> Dict[str, Any]:
"""
Get information about a data issue from an information dictionary.
Parameters
----------
info : dict
A dictionary of information about data issues.
issue_name : str or None, optional (default=None)
The name of the issue to get information about. If `None`, this method returns `None`.
Returns
-------
dict or None
A copy of the information dictionary for the specified issue, or `None` if the issue
name is `None`.
Raises
------
ValueError
If the specified issue name is not found in the information dictionary.
"""
pass # pragma: no cover
@staticmethod
def _get_info_helper(
info: Dict[str, Dict[str, Any]],
issue_name: Optional[str] = None,
) -> Optional[Dict[str, Any]]:
if issue_name is None:
return None
if issue_name not in info:
raise ValueError(
f"issue_name {issue_name} not found in self.info. These have not been computed yet."
)
info = info[issue_name].copy()
return info
class _ClassificationInfoStrategy(_InfoStrategy):
"""Strategy for computing information about data issues related to classification tasks."""
@staticmethod
def get_info(
data: Data,
info: Dict[str, Dict[str, Any]],
issue_name: Optional[str] = None,
) -> Dict[str, Any]:
info_extracted = _InfoStrategy._get_info_helper(info=info, issue_name=issue_name)
info = info_extracted if info_extracted is not None else info
if issue_name in ["label", "class_imbalance"]:
if data.labels.is_available is False:
raise ValueError(
"The labels are not available. "
"Most likely, no label column was provided when creating the Data object."
)
# Labels that are stored as integers may need to be converted to strings.
label_map = data.labels.label_map
if not label_map:
raise ValueError("The label map is not available.")
for key in ["given_label", "predicted_label"]:
labels = info.get(key, None)
if labels is not None:
info[key] = np.vectorize(label_map.get)(labels)
info["class_names"] = list(label_map.values()) # type: ignore
return info
class _RegressionInfoStrategy(_InfoStrategy):
"""Strategy for computing information about data issues related to regression tasks."""
@staticmethod
def get_info(
data: Data,
info: Dict[str, Dict[str, Any]],
issue_name: Optional[str] = None,
) -> Dict[str, Any]:
info_extracted = _InfoStrategy._get_info_helper(info=info, issue_name=issue_name)
info = info_extracted if info_extracted is not None else info
if issue_name == "label":
for key in ["given_label", "predicted_label"]:
labels = info.get(key, None)
if labels is not None:
info[key] = labels
return info
class _MultilabelInfoStrategy(_InfoStrategy):
"""Strategy for computing information about data issues related to multilabel tasks."""
@staticmethod
def get_info(
data: Data,
info: Dict[str, Dict[str, Any]],
issue_name: Optional[str] = None,
) -> Dict[str, Any]:
info_extracted = _InfoStrategy._get_info_helper(info=info, issue_name=issue_name)
info = info_extracted if info_extracted is not None else info
if issue_name == "label":
if data.labels.is_available is False:
raise ValueError(
"The labels are not available. "
"Most likely, no label column was provided when creating the Data object."
)
# Labels that are stored as integers may need to be converted to strings.
label_map = data.labels.label_map
if not label_map:
raise ValueError("The label map is not available.")
for key in ["given_label", "predicted_label"]:
labels = info.get(key, None)
if labels is not None:
info[key] = [list(map(label_map.get, label)) for label in labels] # type: ignore
info["class_names"] = list(label_map.values()) # type: ignore
return info
class DataIssues:
"""
Class that collects and stores information and statistics on issues found in a dataset.
Parameters
----------
data :
The data object for which the issues are being collected.
strategy :
Strategy used for processing info dictionaries.
Attributes
----------
issues : pd.DataFrame
Stores information about each individual issue found in the data,
on a per-example basis.
issue_summary : pd.DataFrame
Summarizes the overall statistics for each issue type.
info : dict
A dictionary that contains information and statistics about the data and each issue type.
"""
def __init__(self, data: Data, strategy: Type[_InfoStrategy]) -> None:
self.issues: pd.DataFrame = pd.DataFrame(index=range(len(data)))
self.issue_summary: pd.DataFrame = pd.DataFrame(
columns=["issue_type", "score", "num_issues"]
).astype({"score": np.float64, "num_issues": np.int64})
self.info: Dict[str, Dict[str, Any]] = {
"statistics": get_data_statistics(data),
}
self._data = data
self._strategy = strategy
def get_info(self, issue_name: Optional[str] = None) -> Dict[str, Any]:
return self._strategy.get_info(data=self._data, info=self.info, issue_name=issue_name)
@property
def statistics(self) -> Dict[str, Any]:
"""Returns the statistics dictionary.
Shorthand for self.info["statistics"].
"""
return self.info["statistics"]
def get_issues(self, issue_name: Optional[str] = None) -> pd.DataFrame:
"""
Use this after finding issues to see which examples suffer from which types of issues.
Parameters
----------
issue_name : str or None
The type of issue to focus on. If `None`, returns full DataFrame summarizing all of the types of issues detected in each example from the dataset.
Raises
------
ValueError
If `issue_name` is not a type of issue previously considered in the audit.
Returns
-------
specific_issues :
A DataFrame where each row corresponds to an example from the dataset and columns specify:
whether this example exhibits a particular type of issue and how severely (via a numeric quality score where lower values indicate more severe instances of the issue).
Additional columns may be present in the DataFrame depending on the type of issue specified.
"""
if self.issues.empty:
raise ValueError(
"""No issues available for retrieval. Please check the following before using `get_issues`:
1. Ensure `find_issues` was executed. If not, please run it with the necessary parameters.
2. If `find_issues` was run but you're seeing this message,
it may have encountered limitations preventing full analysis.
However, partial checks can still provide valuable insights.
Review `find_issues` output carefully for any specific actions needed
to facilitate a more comprehensive analysis before calling `get_issues`.
"""
)
if issue_name is None:
return self.issues
columns = [col for col in self.issues.columns if issue_name in col]
if not columns:
raise ValueError(
f"""No columns found for issue type '{issue_name}'. Ensure the following:
1. `find_issues` has been executed. If it hasn't, please run it.
2. Check `find_issues` output to verify that the issue type '{issue_name}' was included in the checks to
ensure it was not excluded accidentally before the audit.
3. Review `find_issues` output for any errors or warnings that might indicate the check for '{issue_name}' issues failed to complete.
This can provide better insights into what adjustments may be necessary.
"""
)
specific_issues = self.issues[columns]
info = self.get_info(issue_name=issue_name)
if issue_name == "label":
specific_issues = specific_issues.assign(
given_label=info["given_label"], predicted_label=info["predicted_label"]
)
if issue_name == "near_duplicate":
column_dict = {
k: info.get(k)
for k in ["near_duplicate_sets", "distance_to_nearest_neighbor"]
if info.get(k) is not None
}
specific_issues = specific_issues.assign(**column_dict)
if issue_name == "class_imbalance":
specific_issues = specific_issues.assign(given_label=info["given_label"])
return specific_issues
def get_issue_summary(self, issue_name: Optional[str] = None) -> pd.DataFrame:
"""Summarize the issues found in dataset of a particular type,
including how severe this type of issue is overall across the dataset.
Parameters
----------
issue_name :
Name of the issue type to summarize. If `None`, summarizes each of the different issue types previously considered in the audit.
Returns
-------
issue_summary :
DataFrame where each row corresponds to a type of issue, and columns quantify:
the number of examples in the dataset estimated to exhibit this type of issue,
and the overall severity of the issue across the dataset (via a numeric quality score where lower values indicate that the issue is overall more severe).
"""
if self.issue_summary.empty:
raise ValueError(
"No issues found in the dataset. "
"Call `find_issues` before calling `get_issue_summary`."
)
if issue_name is None:
return self.issue_summary
row_mask = self.issue_summary["issue_type"] == issue_name
if not any(row_mask):
raise ValueError(f"Issue type {issue_name} not found in the summary.")
return self.issue_summary[row_mask].reset_index(drop=True)
def collect_statistics(self, issue_manager: Union[IssueManager, "Imagelab"]) -> None:
"""Update the statistics in the info dictionary.
Parameters
----------
statistics :
A dictionary of statistics to add/update in the info dictionary.
Examples
--------
A common use case is to reuse the KNN-graph across multiple issue managers.
To avoid recomputing the KNN-graph for each issue manager,
we can pass it as a statistic to the issue managers.
>>> from scipy.sparse import csr_matrix
>>> weighted_knn_graph = csr_matrix(...)
>>> issue_manager_that_computes_knn_graph = ...
"""
key = "statistics"
statistics: Dict[str, Any] = issue_manager.info.get(key, {})
if statistics:
self.info[key].update(statistics)
def _update_issues(self, issue_manager):
overlapping_columns = list(set(self.issues.columns) & set(issue_manager.issues.columns))
if overlapping_columns:
warnings.warn(
f"Overwriting columns {overlapping_columns} in self.issues with "
f"columns from issue manager {issue_manager}."
)
self.issues.drop(columns=overlapping_columns, inplace=True)
self.issues = self.issues.join(issue_manager.issues, how="outer")
def _update_issue_info(self, issue_name, new_info):
if issue_name in self.info:
warnings.warn(f"Overwriting key {issue_name} in self.info")
self.info[issue_name] = new_info
def collect_issues_from_issue_manager(self, issue_manager: IssueManager) -> None:
"""
Collects results from an IssueManager and update the corresponding
attributes of the Datalab object.
This includes:
- self.issues
- self.issue_summary
- self.info
Parameters
----------
issue_manager :
IssueManager object to collect results from.
"""
self._update_issues(issue_manager)
if issue_manager.issue_name in self.issue_summary["issue_type"].values:
warnings.warn(
f"Overwriting row in self.issue_summary with "
f"row from issue manager {issue_manager}."
)
self.issue_summary = self.issue_summary[
self.issue_summary["issue_type"] != issue_manager.issue_name
]
issue_column_name: str = f"is_{issue_manager.issue_name}_issue"
num_issues: int = int(issue_manager.issues[issue_column_name].sum())
self.issue_summary = pd.concat(
[
self.issue_summary,
issue_manager.summary.assign(num_issues=num_issues),
],
axis=0,
ignore_index=True,
)
self._update_issue_info(issue_manager.issue_name, issue_manager.info)
def collect_issues_from_imagelab(self, imagelab: "Imagelab", issue_types: List[str]) -> None:
pass # pragma: no cover
def set_health_score(self) -> None:
"""Set the health score for the dataset based on the issue summary.
Currently, the health score is the mean of the scores for each issue type.
"""
self.info["statistics"]["health_score"] = self.issue_summary["score"].mean()
def get_data_statistics(data: Data) -> Dict[str, Any]:
"""Get statistics about a dataset.
This function is called to initialize the "statistics" info in all `Datalab` objects.
Parameters
----------
data : Data
Data object containing the dataset.
"""
statistics: Dict[str, Any] = {
"num_examples": len(data),
"multi_label": False,
"health_score": None,
}
if data.labels.is_available:
class_names = data.class_names
statistics["class_names"] = class_names
statistics["num_classes"] = len(class_names)
return statistics
cleanlab/datalab/internal/display.py
+135
View File
@@ -0,0 +1,135 @@
"""
Module that handles the string representation of Datalab objects.
"""
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Dict, List, Optional, Type
from cleanlab.datalab.internal.task import Task
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.internal.data_issues import DataIssues
class RepresentationStrategy(ABC):
def __init__(self, data_issues: "DataIssues"):
self.data_issues = data_issues
@property
def checks_run(self) -> bool:
return not self.data_issues.issues.empty
@property
def num_examples(self) -> Optional[int]:
return self.data_issues.get_info("statistics").get("num_examples")
@property
def num_classes(self) -> Optional[int]:
return self.data_issues.get_info("statistics").get("num_classes")
@property
def issues_identified(self) -> str:
return (
self.data_issues.issue_summary["num_issues"].sum() if self.checks_run else "Not checked"
)
def show_task(self, task: "Task") -> str:
return f"task={str(task).capitalize()}"
def show_checks_run(self) -> str:
return f"checks_run={self.checks_run}"
def show_num_examples(self) -> str:
return f"num_examples={self.num_examples}" if self.num_examples is not None else ""
def show_num_classes(self) -> str:
return f"num_classes={self.num_classes}" if self.num_classes is not None else ""
def show_issues_identified(self) -> str:
return f"issues_identified={self.issues_identified}"
@abstractmethod
def represent(self) -> str:
pass
def to_string(self, task: "Task") -> str:
"""What is displayed if user executes: print(datalab)"""
info_list = [
f"Task: {str(task).capitalize()}",
f"Checks run: {'Yes' if self.checks_run else 'No'}",
f"Number of examples: {self.num_examples if self.num_examples is not None else 'Unknown'}",
f"Number of classes: {self.num_classes if self.num_classes is not None else 'Unknown'}",
f"Issues identified: {self.issues_identified}",
]
return "Datalab:\n" + "\n".join(info_list)
class ClassificationRepresentation(RepresentationStrategy):
def represent(self) -> str:
display_strings: List[str] = [
self.show_task(Task.CLASSIFICATION),
self.show_checks_run(),
self.show_num_examples(),
self.show_num_classes(),
self.show_issues_identified(),
]
# Drop empty strings
display_strings = [s for s in display_strings if bool(s)]
display_str = ", ".join(display_strings)
return f"Datalab({display_str})"
class RegressionRepresentation(RepresentationStrategy):
def represent(self) -> str:
display_strings: List[str] = [
self.show_task(Task.REGRESSION),
self.show_checks_run(),
self.show_num_examples(),
self.show_issues_identified(),
]
# Drop empty strings
display_strings = [s for s in display_strings if bool(s)]
display_str = ", ".join(display_strings)
return f"Datalab({display_str})"
class MultilabelRepresentation(RepresentationStrategy):
def represent(self) -> str:
display_strings: List[str] = [
self.show_task(Task.MULTILABEL),
self.show_checks_run(),
self.show_num_examples(),
self.show_num_classes(),
self.show_issues_identified(),
]
# Drop empty strings
display_strings = [s for s in display_strings if bool(s)]
display_str = ", ".join(display_strings)
return f"Datalab({display_str})"
class _Displayer:
def __init__(self, data_issues: "DataIssues", task: "Task") -> None:
self.data_issues = data_issues
self.task = task
self.representation_strategy = self._get_representation_strategy()
def _get_representation_strategy(self) -> RepresentationStrategy:
strategies: Dict[str, Type[RepresentationStrategy]] = {
"classification": ClassificationRepresentation,
"regression": RegressionRepresentation,
"multilabel": MultilabelRepresentation,
}
strategy_class = strategies.get(self.task.value)
if strategy_class is None:
raise ValueError(f"Unsupported task type: {self.task}")
return strategy_class(self.data_issues)
def __repr__(self) -> str:
"""What is displayed in console if user executes: >>> datalab"""
return self.representation_strategy.represent()
def __str__(self) -> str:
"""What is displayed if user executes: print(datalab)"""
return self.representation_strategy.to_string(self.task)
cleanlab/datalab/internal/helper_factory.py
+92
View File
@@ -0,0 +1,92 @@
from typing import Dict, Optional, Type
from cleanlab.datalab.internal.adapter.imagelab import (
ImagelabDataIssuesAdapter,
ImagelabIssueFinderAdapter,
ImagelabReporterAdapter,
)
from cleanlab.datalab.internal.data import Data
from cleanlab.datalab.internal.data_issues import (
_InfoStrategy,
DataIssues,
_ClassificationInfoStrategy,
_RegressionInfoStrategy,
_MultilabelInfoStrategy,
)
from cleanlab.datalab.internal.issue_finder import IssueFinder
from cleanlab.datalab.internal.report import Reporter
from cleanlab.datalab.internal.task import Task
def issue_finder_factory(imagelab):
if imagelab:
return ImagelabIssueFinderAdapter
else:
return IssueFinder
def report_factory(imagelab):
if imagelab:
return ImagelabReporterAdapter
else:
return Reporter
class _DataIssuesBuilder:
"""A helper class for constructing DataIssues instances.
It uses the builder pattern to allow users to specify the desired
configuration of the DataIssues instance.
It uses the `set_X` naming convention for methods that set the
desired configuration, before calling the `build` method to
construct the DataIssues instance.
"""
def __init__(self, data: Data):
self.data = data
self.imagelab = None
self.task: Optional[Task] = None
def set_imagelab(self, imagelab):
self.imagelab = imagelab
return self
def set_task(self, task: Task):
"""Set the task that the data is intended for.
Parameters
----------
task : Task
Specific machine learning task that the datset is intended for.
See details about supported tasks in :py:class:`Task <cleanlab.datalab.internal.task.Task>`.
"""
self.task = task
return self
def build(self) -> DataIssues:
data_issues_class = self._data_issues_factory()
strategy = self._select_info_strategy()
return data_issues_class(self.data, strategy)
def _data_issues_factory(self) -> Type[DataIssues]:
"""Factory method that selects the appropriate class for
constructing the DataIssues instance.
"""
if self.imagelab:
return ImagelabDataIssuesAdapter
else:
return DataIssues
def _select_info_strategy(self) -> Type[_InfoStrategy]:
"""The DataIssues class takes in a strategy class
for processing info dictionaries. This method selects
the appropriate strategy class based on the task during
the `build` method-call.
"""
_default_return = _ClassificationInfoStrategy
strategy_lookup: Dict[Task, Type[_InfoStrategy]] = {
Task.REGRESSION: _RegressionInfoStrategy,
Task.MULTILABEL: _MultilabelInfoStrategy,
}
if self.task is None:
return _default_return
return strategy_lookup.get(self.task, _default_return)
cleanlab/datalab/internal/issue_finder.py
+489
View File
@@ -0,0 +1,489 @@
"""
Module for the :class:`IssueFinder` class, which is responsible for configuring,
creating and running issue managers.
It determines which types of issues to look for, instatiates the IssueManagers
via a factory, run the issue managers
(:py:meth:`IssueManager.find_issues <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager.find_issues>`),
and collects the results to :py:class:`DataIssues <cleanlab.datalab.internal.data_issues.DataIssues>`.
.. note::
This module is not intended to be used directly. Instead, use the public-facing
:py:meth:`Datalab.find_issues <cleanlab.datalab.datalab.Datalab.find_issues>` method.
"""
from __future__ import annotations
import warnings
from typing import TYPE_CHECKING, Any, Dict, Optional
import numpy as np
from scipy.sparse import csr_matrix
from cleanlab.datalab.internal.issue_manager_factory import (
_IssueManagerFactory,
list_default_issue_types,
)
from cleanlab.datalab.internal.model_outputs import (
MultiClassPredProbs,
MultiLabelPredProbs,
RegressionPredictions,
)
from cleanlab.datalab.internal.task import Task
if TYPE_CHECKING: # pragma: no cover
from typing import Callable
import numpy.typing as npt
from cleanlab.datalab.datalab import Datalab
_CLASSIFICATION_ARGS_DICT = {
"label": ["pred_probs", "features"],
"outlier": ["pred_probs", "features", "knn_graph"],
"near_duplicate": ["features", "knn_graph"],
"non_iid": ["pred_probs", "features", "knn_graph"],
# The underperforming_group issue type requires a pair of inputs: (pred_probs, <any_of_the_other_three>)
"underperforming_group": ["pred_probs", "features", "knn_graph", "cluster_ids"],
"data_valuation": ["features", "knn_graph"],
"class_imbalance": [],
"null": ["features"],
}
_REGRESSION_ARGS_DICT = {
"label": ["features", "predictions"],
"outlier": ["features", "knn_graph"],
"near_duplicate": ["features", "knn_graph"],
"non_iid": ["features", "knn_graph"],
"data_valuation": ["features", "knn_graph"],
"null": ["features"],
}
_MULTILABEL_ARGS_DICT = {
"label": ["pred_probs"],
"outlier": ["features", "knn_graph"],
"near_duplicate": ["features", "knn_graph"],
"non_iid": ["features", "knn_graph"],
"data_valuation": ["features", "knn_graph"],
"null": ["features"],
}
def _resolve_required_args_for_classification(**kwargs):
"""Resolves the required arguments for each issue type intended for classification tasks."""
initial_args_dict = _CLASSIFICATION_ARGS_DICT.copy()
args_dict = {
issue_type: {arg: kwargs.get(arg, None) for arg in initial_args_dict[issue_type]}
for issue_type in initial_args_dict
}
# Some issue types (like class-imbalance) have no required args.
# This conditional lambda is used to include them in args dict.
keep_empty_argument = lambda k: not len(_CLASSIFICATION_ARGS_DICT[k])
# Remove None values from argument list, rely on default values in IssueManager
args_dict = {
k: {k2: v2 for k2, v2 in v.items() if v2 is not None}
for k, v in args_dict.items()
if (v or keep_empty_argument(k))
}
# Prefer `knn_graph` over `features` if both are provided.
for v in args_dict.values():
if "cluster_ids" in v and ("knn_graph" in v or "features" in v):
warnings.warn(
"`cluster_ids` have been provided with `knn_graph` or `features`."
"Issue managers that require cluster labels will prefer"
"`cluster_ids` over computation of cluster labels using"
"`knn_graph` or `features`. "
)
if "knn_graph" in v and "features" in v:
warnings.warn(
"Both `features` and `knn_graph` were provided. "
"Most issue managers will likely prefer using `knn_graph` "
"instead of `features` for efficiency."
)
# Only keep issue types that have at least one argument
# or those that require no arguments.
args_dict = {k: v for k, v in args_dict.items() if (v or keep_empty_argument(k))}
return args_dict
def _resolve_required_args_for_regression(**kwargs):
"""Resolves the required arguments for each issue type intended for regression tasks."""
initial_args_dict = _REGRESSION_ARGS_DICT.copy()
args_dict = {
issue_type: {arg: kwargs.get(arg, None) for arg in initial_args_dict[issue_type]}
for issue_type in initial_args_dict
}
# Some issue types have no required args.
# This conditional lambda is used to include them in args dict.
keep_empty_argument = lambda k: not len(_REGRESSION_ARGS_DICT[k])
# Remove None values from argument list, rely on default values in IssueManager
args_dict = {
k: {k2: v2 for k2, v2 in v.items() if v2 is not None}
for k, v in args_dict.items()
if v or keep_empty_argument(k)
}
# Only keep issue types that have at least one argument
# or those that require no arguments.
args_dict = {k: v for k, v in args_dict.items() if (v or keep_empty_argument(k))}
return args_dict
def _resolve_required_args_for_multilabel(**kwargs):
"""Resolves the required arguments for each issue type intended for multilabel tasks."""
initial_args_dict = _MULTILABEL_ARGS_DICT.copy()
args_dict = {
issue_type: {arg: kwargs.get(arg, None) for arg in initial_args_dict[issue_type]}
for issue_type in initial_args_dict
}
# Some issue types have no required args.
# This conditional lambda is used to include them in args dict.
keep_empty_argument = lambda k: not len(_MULTILABEL_ARGS_DICT[k])
# Remove None values from argument list, rely on default values in IssueManager
args_dict = {
k: {k2: v2 for k2, v2 in v.items() if v2 is not None}
for k, v in args_dict.items()
if v or keep_empty_argument(k) # Allow label issues to require no arguments
}
# Only keep issue types that have at least one argument
# or those that require no arguments.
args_dict = {k: v for k, v in args_dict.items() if (v or keep_empty_argument(k))}
return args_dict
def _select_strategy_for_resolving_required_args(task: Task) -> Callable:
"""Helper function that selects the strategy for resolving required arguments for each issue type.
Each strategy resolves the required arguments for each issue type.
This is a helper function that filters out any issue manager
that does not have the required arguments.
This does not consider custom hyperparameters for each issue type.
Parameters
----------
task : str
The type of machine learning task that the dataset is used for.
Returns
-------
args_dict :
Dictionary of required arguments for each issue type, if available.
"""
strategies = {
Task.CLASSIFICATION: _resolve_required_args_for_classification,
Task.REGRESSION: _resolve_required_args_for_regression,
Task.MULTILABEL: _resolve_required_args_for_multilabel,
}
selected_strategy = strategies.get(task, None)
if selected_strategy is None:
raise ValueError(f"No strategy for resolving required arguments for task '{task}'")
return selected_strategy
class IssueFinder:
"""
The IssueFinder class is responsible for managing the process of identifying
issues in the dataset by handling the creation and execution of relevant
IssueManagers. It serves as a coordinator or helper class for the Datalab class
to encapsulate the specific behavior of the issue finding process.
At a high level, the IssueFinder is responsible for:
- Determining which types of issues to look for.
- Instantiating the appropriate IssueManagers using a factory.
- Running the IssueManagers' `find_issues` methods.
- Collecting the results into a DataIssues instance.
Parameters
----------
datalab : Datalab
The Datalab instance associated with this IssueFinder.
task : str
The type of machine learning task that the dataset is used for.
verbosity : int
Controls the verbosity of the output during the issue finding process.
Note
----
This class is not intended to be used directly. Instead, use the
`Datalab.find_issues` method which internally utilizes an IssueFinder instance.
"""
def __init__(self, datalab: "Datalab", task: Task, verbosity=1):
self.datalab = datalab
self.task = task
self.verbosity = verbosity
def find_issues(
self,
*,
pred_probs: Optional[np.ndarray] = None,
features: Optional[npt.NDArray] = None,
knn_graph: Optional[csr_matrix] = None,
issue_types: Optional[Dict[str, Any]] = None,
) -> None:
"""
Checks the dataset for all sorts of common issues in real-world data (in both labels and feature values).
You can use Datalab to find issues in your data, utilizing *any* model you have already trained.
This method only interacts with your model via its predictions or embeddings (and other functions thereof).
The more of these inputs you provide, the more types of issues Datalab can detect in your dataset/labels.
If you provide a subset of these inputs, Datalab will output what insights it can based on the limited information from your model.
Note
----
This method is not intended to be used directly. Instead, use the
:py:meth:`Datalab.find_issues <cleanlab.datalab.datalab.Datalab.find_issues>` method.
Note
----
The issues are saved in the ``self.datalab.data_issues.issues`` attribute, but are not returned.
Parameters
----------
pred_probs :
Out-of-sample predicted class probabilities made by the model for every example in the dataset.
To best detect label issues, provide this input obtained from the most accurate model you can produce.
If provided for classification, this must be a 2D array with shape ``(num_examples, K)`` where K is the number of classes in the dataset.
If provided for regression, this must be a 1D array with shape ``(num_examples,)``.
features : Optional[np.ndarray]
Feature embeddings (vector representations) of every example in the dataset.
If provided, this must be a 2D array with shape (num_examples, num_features).
knn_graph :
Sparse matrix representing distances between examples in the dataset in a k nearest neighbor graph.
For details, refer to the documentation of the same argument in :py:class:`Datalab.find_issues <cleanlab.datalab.datalab.Datalab.find_issues>`
issue_types :
Collection specifying which types of issues to consider in audit and any non-default parameter settings to use.
If unspecified, a default set of issue types and recommended parameter settings is considered.
This is a dictionary of dictionaries, where the keys are the issue types of interest
and the values are dictionaries of parameter values that control how each type of issue is detected (only for advanced users).
More specifically, the values are constructor keyword arguments passed to the corresponding ``IssueManager``,
which is responsible for detecting the particular issue type.
.. seealso::
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`
"""
issue_types_copy = self.get_available_issue_types(
pred_probs=pred_probs,
features=features,
knn_graph=knn_graph,
issue_types=issue_types,
)
if not issue_types_copy:
return None
new_issue_managers = [
factory(datalab=self.datalab, **issue_types_copy.get(factory.issue_name, {}))
for factory in _IssueManagerFactory.from_list(
list(issue_types_copy.keys()), task=self.task
)
]
failed_managers = []
data_issues = self.datalab.data_issues
for issue_manager, arg_dict in zip(new_issue_managers, issue_types_copy.values()):
try:
if self.verbosity:
print(f"Finding {issue_manager.issue_name} issues ...")
issue_manager.find_issues(**arg_dict)
data_issues.collect_statistics(issue_manager)
data_issues.collect_issues_from_issue_manager(issue_manager)
except Exception as e:
print(f"Error in {issue_manager.issue_name}: {e}")
failed_managers.append(issue_manager)
if failed_managers:
print(f"Failed to check for these issue types: {failed_managers}")
data_issues.set_health_score()
def _set_issue_types(
self,
issue_types: Optional[Dict[str, Any]],
required_defaults_dict: Dict[str, Any],
) -> Dict[str, Any]:
"""Set necessary configuration for each IssueManager in a dictionary.
While each IssueManager defines default values for its arguments,
the Datalab class needs to organize the calls to each IssueManager
with different arguments, some of which may be user-provided.
Parameters
----------
issue_types :
Dictionary of issue types and argument configuration for their respective IssueManagers.
If None, then the `required_defaults_dict` is used.
required_defaults_dict :
Dictionary of default parameter configuration for each issue type.
Returns
-------
issue_types_copy :
Dictionary of issue types and their parameter configuration.
The input `issue_types` is copied and updated with the necessary default values.
"""
if issue_types is not None:
issue_types_copy = issue_types.copy()
self._check_missing_args(required_defaults_dict, issue_types_copy)
else:
issue_types_copy = required_defaults_dict.copy()
# keep only default issue types
issue_types_copy = {
issue: issue_types_copy[issue]
for issue in list_default_issue_types(self.task)
if issue in issue_types_copy
}
# Check that all required arguments are provided.
self._validate_issue_types_dict(issue_types_copy, required_defaults_dict)
# Remove None values from argument list, rely on default values in IssueManager
for key, value in issue_types_copy.items():
issue_types_copy[key] = {k: v for k, v in value.items() if v is not None}
return issue_types_copy
@staticmethod
def _check_missing_args(required_defaults_dict, issue_types):
for key, issue_type_value in issue_types.items():
missing_args = set(required_defaults_dict.get(key, {})) - set(issue_type_value.keys())
# Impute missing arguments with default values.
missing_dict = {
missing_arg: required_defaults_dict[key][missing_arg]
for missing_arg in missing_args
}
issue_types[key].update(missing_dict)
@staticmethod
def _validate_issue_types_dict(
issue_types: Dict[str, Any], required_defaults_dict: Dict[str, Any]
) -> None:
missing_required_args_dict = {}
for issue_name, required_args in required_defaults_dict.items():
if issue_name in issue_types:
missing_args = set(required_args.keys()) - set(issue_types[issue_name].keys())
if missing_args:
missing_required_args_dict[issue_name] = missing_args
if any(missing_required_args_dict.values()):
error_message = ""
for issue_name, missing_required_args in missing_required_args_dict.items():
error_message += f"Required argument {missing_required_args} for issue type {issue_name} was not provided.\n"
raise ValueError(error_message)
def get_available_issue_types(self, **kwargs):
"""Returns a dictionary of issue types that can be used in :py:meth:`Datalab.find_issues
<cleanlab.datalab.datalab.Datalab.find_issues>` method."""
pred_probs = kwargs.get("pred_probs", None)
features = kwargs.get("features", None)
knn_graph = kwargs.get("knn_graph", None)
issue_types = kwargs.get("issue_types", None)
model_output = None
if pred_probs is not None:
model_output_dict = {
Task.REGRESSION: RegressionPredictions,
Task.CLASSIFICATION: MultiClassPredProbs,
Task.MULTILABEL: MultiLabelPredProbs,
}
model_output_class = model_output_dict.get(self.task)
if model_output_class is None:
raise ValueError(f"Unknown task type '{self.task}'")
model_output = model_output_class(pred_probs)
if model_output is not None:
# A basic trick to assign the model output to the correct argument
# E.g. Datalab accepts only `pred_probs`, but those are assigned to the `predictions` argument for regression-related issue_managers
kwargs.update({model_output.argument: model_output.collect()})
# Determine which parameters are required for each issue type
strategy_for_resolving_required_args = _select_strategy_for_resolving_required_args(
self.task
)
required_args_per_issue_type = strategy_for_resolving_required_args(**kwargs)
issue_types_copy = self._set_issue_types(issue_types, required_args_per_issue_type)
if issue_types is None:
# Only run default issue types if no issue types are specified
issue_types_copy = {
issue: issue_types_copy[issue]
for issue in list_default_issue_types(self.task)
if issue in issue_types_copy
}
drop_label_check = (
"label" in issue_types_copy
and not self.datalab.has_labels
and self.task != Task.REGRESSION
)
if drop_label_check:
warnings.warn("No labels were provided. " "The 'label' issue type will not be run.")
issue_types_copy.pop("label")
outlier_check_needs_features = (
self.task == "classification"
and "outlier" in issue_types_copy
and not self.datalab.has_labels
)
if outlier_check_needs_features:
no_features = features is None
no_knn_graph = knn_graph is None
pred_probs_given = issue_types_copy["outlier"].get("pred_probs", None) is not None
only_pred_probs_given = pred_probs_given and no_features and no_knn_graph
if only_pred_probs_given:
warnings.warn(
"No labels were provided. " "The 'outlier' issue type will not be run."
)
issue_types_copy.pop("outlier")
drop_class_imbalance_check = (
"class_imbalance" in issue_types_copy
and not self.datalab.has_labels
and self.task == Task.CLASSIFICATION
)
if drop_class_imbalance_check:
issue_types_copy.pop("class_imbalance")
required_pairs_for_underperforming_group = [
("pred_probs", "features"),
("pred_probs", "knn_graph"),
("pred_probs", "cluster_ids"),
]
drop_underperforming_group_check = "underperforming_group" in issue_types_copy and not any(
all(
key in issue_types_copy["underperforming_group"]
and issue_types_copy["underperforming_group"].get(key) is not None
for key in pair
)
for pair in required_pairs_for_underperforming_group
)
if drop_underperforming_group_check:
issue_types_copy.pop("underperforming_group")
return issue_types_copy
cleanlab/datalab/internal/issue_manager/__init__.py
+10
View File
@@ -0,0 +1,10 @@
from .issue_manager import IssueManager # isort:skip
from .duplicate import NearDuplicateIssueManager
from .label import LabelIssueManager
from .outlier import OutlierIssueManager
from .noniid import NonIIDIssueManager
from .imbalance import ClassImbalanceIssueManager
from .underperforming_group import UnderperformingGroupIssueManager
from .data_valuation import DataValuationIssueManager
from .null import NullIssueManager
from .identifier_column import IdentifierColumnIssueManager
cleanlab/datalab/internal/issue_manager/data_valuation.py
+189
View File
@@ -0,0 +1,189 @@
from __future__ import annotations
from typing import (
TYPE_CHECKING,
Any,
Callable,
ClassVar,
Dict,
List,
Optional,
Union,
)
import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
from cleanlab.data_valuation import data_shapley_knn
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.datalab.internal.issue_manager.knn_graph_helpers import (
num_neighbors_in_knn_graph,
set_knn_graph,
)
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
import pandas as pd
from cleanlab.datalab.datalab import Datalab
class DataValuationIssueManager(IssueManager):
"""
Detect which examples in a dataset are least valuable via an approximate Data Shapely value.
Examples
--------
.. code-block:: python
>>> from cleanlab import Datalab
>>> import numpy as np
>>> from sklearn.neighbors import NearestNeighbors
>>>
>>> # Generate two distinct clusters
>>> X = np.vstack([
... np.random.normal(-1, 1, (25, 2)),
... np.random.normal(1, 1, (25, 2)),
... ])
>>> y = np.array([0]*25 + [1]*25)
>>>
>>> # Initialize Datalab with data
>>> lab = Datalab(data={"y": y}, label_name="y")
>>>
>>> # Creating a knn_graph for data valuation
>>> knn = NearestNeighbors(n_neighbors=10).fit(X)
>>> knn_graph = knn.kneighbors_graph(mode='distance')
>>>
>>> # Specifying issue types for data valuation
>>> issue_types = {"data_valuation": {}}
>>> lab.find_issues(knn_graph=knn_graph, issue_types=issue_types)
"""
description: ClassVar[
str
] = """
Examples that contribute minimally to a model's training
receive lower valuation scores.
Since the original knn-shapley value is in [-1, 1], we transform it to [0, 1] by:
.. math::
0.5 \times (\text{shapley} + 1)
here shapley is the original knn-shapley value.
"""
issue_name: ClassVar[str] = "data_valuation"
issue_score_key: ClassVar[str]
verbosity_levels: ClassVar[Dict[int, List[str]]] = {
0: [],
1: [],
2: [],
3: ["average_data_valuation"],
}
DEFAULT_THRESHOLD = 0.5
def __init__(
self,
datalab: Datalab,
metric: Optional[Union[str, Callable]] = None,
threshold: Optional[float] = None,
k: int = 10,
**kwargs,
):
super().__init__(datalab)
self.metric = metric
self.k = k
self.threshold = threshold if threshold is not None else self.DEFAULT_THRESHOLD
def find_issues(
self,
features: Optional[npt.NDArray] = None,
**kwargs,
) -> None:
"""Calculate the data valuation score with a provided or existing knn graph.
Based on KNN-Shapley value described in https://arxiv.org/abs/1911.07128
The larger the score, the more valuable the data point is, the more contribution it will make to the model's training.
Parameters
----------
knn_graph : csr_matrix
A sparse matrix representing the knn graph.
"""
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"Expected labels to be a numpy array of shape (n_samples,) to use with DataValuationIssueManager, "
f"but got {type(labels)} instead."
)
raise TypeError(error_msg)
knn_graph, self.metric, _ = set_knn_graph(
features=features,
find_issues_kwargs=kwargs,
metric=self.metric,
k=self.k,
statistics=self.datalab.get_info("statistics"),
)
# TODO: Check self.k against user-provided knn-graphs across all issue managers
num_neighbors = num_neighbors_in_knn_graph(knn_graph)
if self.k > num_neighbors:
raise ValueError(
f"The provided knn graph has {num_neighbors} neighbors, which is less than the required {self.k} neighbors. "
"Please ensure that the knn graph you provide has at least as many neighbors as the required value of k."
)
scores = data_shapley_knn(labels, knn_graph=knn_graph, k=self.k)
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": scores < self.threshold,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=scores.mean())
self.info = self.collect_info(issues=self.issues, knn_graph=knn_graph)
def collect_info(self, issues: pd.DataFrame, knn_graph: csr_matrix) -> dict:
issues_info = {
"num_low_valuation_issues": sum(issues[f"is_{self.issue_name}_issue"]),
"average_data_valuation": issues[self.issue_score_key].mean(),
}
params_dict = {
"metric": self.metric,
"k": self.k,
"threshold": self.threshold,
}
statistics_dict = self._build_statistics_dictionary(knn_graph=knn_graph)
info_dict = {
**issues_info,
**params_dict,
**statistics_dict,
}
return info_dict
def _build_statistics_dictionary(self, knn_graph: csr_matrix) -> Dict[str, Dict[str, Any]]:
statistics_dict: Dict[str, Dict[str, Any]] = {"statistics": {}}
# Add the knn graph as a statistic if necessary
graph_key = "weighted_knn_graph"
old_knn_graph = self.datalab.get_info("statistics").get(graph_key, None)
old_graph_exists = old_knn_graph is not None
prefer_new_graph = (
not old_graph_exists
or (old_knn_graph is not None and knn_graph.nnz > old_knn_graph.nnz)
or self.metric != self.datalab.get_info("statistics").get("knn_metric", None)
)
if prefer_new_graph:
statistics_dict["statistics"][graph_key] = knn_graph
if self.metric is not None:
statistics_dict["statistics"]["knn_metric"] = self.metric
return statistics_dict
cleanlab/datalab/internal/issue_manager/duplicate.py
+228
View File
@@ -0,0 +1,228 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable, ClassVar, Dict, List, Optional, Union
import warnings
import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.datalab.internal.issue_manager.knn_graph_helpers import set_knn_graph
from cleanlab.internal.constants import EPSILON
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
from cleanlab.datalab.datalab import Datalab
class NearDuplicateIssueManager(IssueManager):
"""Manages issues related to near-duplicate examples."""
description: ClassVar[
str
] = """A (near) duplicate issue refers to two or more examples in
a dataset that are extremely similar to each other, relative
to the rest of the dataset. The examples flagged with this issue
may be exactly duplicated, or lie atypically close together when
represented as vectors (i.e. feature embeddings).
"""
issue_name: ClassVar[str] = "near_duplicate"
verbosity_levels = {
0: [],
1: [],
2: ["threshold"],
}
def __init__(
self,
datalab: Datalab,
metric: Optional[Union[str, Callable]] = None,
threshold: float = 0.13,
k: int = 10,
**_,
):
super().__init__(datalab)
self.metric = metric
self.threshold = self._set_threshold(threshold)
self.k = k
self.near_duplicate_sets: List[List[int]] = []
def find_issues(
self,
features: Optional[npt.NDArray] = None,
**kwargs,
) -> None:
knn_graph, self.metric, _ = set_knn_graph(
features=features,
find_issues_kwargs=kwargs,
metric=self.metric,
k=self.k,
statistics=self.datalab.get_info("statistics"),
)
N = knn_graph.shape[0]
nn_distances = knn_graph.data.reshape(N, -1)[:, 0]
median_nn_distance = max(np.median(nn_distances), EPSILON) # avoid threshold = 0
self.near_duplicate_sets = self._neighbors_within_radius(
knn_graph, self.threshold, median_nn_distance
)
# Flag every example in a near-duplicate set as a near-duplicate issue
all_near_duplicates = np.unique(np.concatenate(self.near_duplicate_sets))
is_issue_column = np.zeros(N, dtype=bool)
is_issue_column[all_near_duplicates] = True
temperature = 1.0 / median_nn_distance
scores = _compute_scores_with_exp_transform(nn_distances, temperature=temperature)
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_issue_column,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=scores.mean())
self.info = self.collect_info(knn_graph=knn_graph, median_nn_distance=median_nn_distance)
@staticmethod
def _neighbors_within_radius(knn_graph: csr_matrix, threshold: float, median: float):
"""Returns a list of lists of indices of near-duplicate examples.
Each list of indices represents a set of near-duplicate examples.
If the list is empty for a given example, then that example is not
a near-duplicate of any other example.
"""
N = knn_graph.shape[0]
distances = knn_graph.data.reshape(N, -1)
# Create a mask for the threshold
mask = distances < threshold * median
# Update the indptr to reflect the new number of neighbors
indptr = np.zeros(knn_graph.indptr.shape, dtype=knn_graph.indptr.dtype)
indptr[1:] = np.cumsum(mask.sum(axis=1))
# Filter the knn_graph based on the threshold
indices = knn_graph.indices[mask.ravel()]
near_duplicate_sets = [indices[indptr[i] : indptr[i + 1]] for i in range(N)]
# Second pass over the data is required to ensure each item is included in the near-duplicate sets of its own near-duplicates.
# This is important because a "near-duplicate" relationship is reciprocal.
# For example, if item A is a near-duplicate of item B, then item B should also be considered a near-duplicate of item A.
# NOTE: This approach does not assure that the sets are ordered by increasing distance.
for i, near_duplicates in enumerate(near_duplicate_sets):
for j in near_duplicates:
if i not in near_duplicate_sets[j]:
near_duplicate_sets[j] = np.append(near_duplicate_sets[j], i)
return near_duplicate_sets
def collect_info(self, knn_graph: csr_matrix, median_nn_distance: float) -> dict:
issues_dict = {
"average_near_duplicate_score": self.issues[self.issue_score_key].mean(),
"near_duplicate_sets": self.near_duplicate_sets,
}
params_dict = {
"metric": self.metric,
"k": self.k,
"threshold": self.threshold,
}
N = knn_graph.shape[0]
dists = knn_graph.data.reshape(N, -1)[:, 0]
nn_ids = knn_graph.indices.reshape(N, -1)[:, 0]
knn_info_dict = {
"nearest_neighbor": nn_ids.tolist(),
"distance_to_nearest_neighbor": dists.tolist(),
"median_distance_to_nearest_neighbor": median_nn_distance,
}
statistics_dict = self._build_statistics_dictionary(knn_graph=knn_graph)
info_dict = {
**issues_dict,
**params_dict,
**knn_info_dict,
**statistics_dict,
}
return info_dict
def _build_statistics_dictionary(self, knn_graph: csr_matrix) -> Dict[str, Dict[str, Any]]:
statistics_dict: Dict[str, Dict[str, Any]] = {"statistics": {}}
# Add the knn graph as a statistic if necessary
graph_key = "weighted_knn_graph"
old_knn_graph = self.datalab.get_info("statistics").get(graph_key, None)
old_graph_exists = old_knn_graph is not None
prefer_new_graph = (
not old_graph_exists
or (old_knn_graph is not None and knn_graph.nnz > old_knn_graph.nnz)
or self.metric != self.datalab.get_info("statistics").get("knn_metric", None)
)
if prefer_new_graph:
statistics_dict["statistics"][graph_key] = knn_graph
if self.metric is not None:
statistics_dict["statistics"]["knn_metric"] = self.metric
return statistics_dict
def _set_threshold(
self,
threshold: float,
) -> float:
"""Computes nearest-neighbors thresholding for near-duplicate detection."""
if threshold < 0:
warnings.warn(
f"Computed threshold {threshold} is less than 0. "
"Setting threshold to 0."
"This may indicate that either the only a few examples are in the dataset, "
"or the data is heavily skewed."
)
threshold = 0
return threshold
def _compute_scores_with_exp_transform(nn_distances: np.ndarray, temperature: float) -> np.ndarray:
r"""Compute near-duplicate scores from nearest neighbor distances.
This is a non-linear transformation of the nearest neighbor distances that
maps distances to scores in the range [0, 1].
Note
----
This transformation is given by the following formula:
.. math::
\text{score}(d, t) = 1 - e^{-dt}
where :math:`d` is the nearest neighbor distance and :math:`t > 0` is a temperature parameter.
Parameters
----------
nn_distances :
The nearest neighbor distances for each example.
Returns
-------
scores :
The near-duplicate scores for each example. The scores are in the range [0, 1].
A lower score indicates that an example is more likely to be a near-duplicate than
an example with a higher score.
A score of 0 indicates that an example has an exact duplicate.
"""
if temperature <= 0:
raise ValueError("Temperature must be greater than 0.")
scores = 1 - np.exp(-temperature * nn_distances)
# Ensure that for nn_distances approximately equal to 0, the score is set to 0
inds = np.isclose(nn_distances, 0)
scores[inds] = 0
return scores
cleanlab/datalab/internal/issue_manager/identifier_column.py
+131
View File
@@ -0,0 +1,131 @@
from typing import ClassVar, List, Optional, Union
import numpy as np
import numpy.typing as npt
import pandas as pd
from cleanlab.datalab.internal.issue_manager import IssueManager
class IdentifierColumnIssueManager(IssueManager):
"""Manages issues related to identifier columns in feature columns"""
description: ClassVar[
str
] = """Checks whether there is an identifier_column in the features of a dataset.
Identifier columns are defined as a column i in features such that
set(features[:,i]) = set(c, c+1, ..., c+n) for some integer c,
where n = num-rows of features. If there is such a column, the dataset has
the identifier_column issue
"""
issue_name: ClassVar[str] = "identifier_column"
verbosity_levels = {
0: [],
1: ["identifier_columns"],
2: [],
}
def _is_sequential(self, arr: npt.NDArray) -> bool:
"""
Check if the elements in the array are sequential.
Parameters:
arr: The input array.
Returns:
A boolean indicating whether the elements in the array are sequential.
"""
if arr.size == 0:
return False
unique_sorted = np.unique(arr) # Returns a sorted unique list
min_val, max_val = unique_sorted[0], unique_sorted[-1]
expected_range = np.arange(min_val, max_val + 1)
if expected_range.size == 1 or unique_sorted.size != expected_range.size:
return False
return bool((expected_range == unique_sorted).all())
def _prepare_features(
self, features: Optional[Union[npt.NDArray, pd.DataFrame, list, dict]]
) -> Union[npt.NDArray, List[npt.NDArray]]:
"""
Prepare the features for issue check.
Args:
features: The input features.
Returns:
features: features as npt.NDArray
"""
if isinstance(features, np.ndarray):
return features.T # Transpose if it's a NumPy array
# to keep the datatype of the string columns for dicts and pandas dataframes consistent
# we convert the string columns to dtype=str, otherwise we ran into error in our tests
elif isinstance(features, pd.DataFrame):
result = []
for col in features.columns:
if pd.api.types.is_string_dtype(
features[col]
): # detect string columns in both pandas 2.x and 3.x.
arr = np.array(features[col].values).astype(str)
else:
arr = np.array(features[col].values)
result.append(arr)
return result
elif isinstance(features, dict):
result = []
for value in features.values():
if isinstance(value[0], str):
arr = np.array(value).astype(str)
else:
arr = np.array(value)
result.append(arr)
return result
elif isinstance(features, list):
for col_list in features:
if not isinstance(col_list, list) and not isinstance(col_list, np.ndarray):
raise ValueError(
"features must be a list of lists or numpy arrays if a list is passed."
)
return [np.array(col_list) for col_list in features]
else:
raise ValueError("features must be a numpy array, pandas DataFrame, list, or dict.")
def find_issues(
self, features: Optional[Union[npt.NDArray, pd.DataFrame, list, dict]], **kwargs
) -> None:
"""
Find identifier columns in the given dataset.
Parameters:
features (Optional[npt.NDArray | pd.DataFrame | list | dict]):
The dataset to check for identifier columns.
Returns:
None
"""
if features is None:
raise ValueError("features must be provided to check for identifier columns.")
processed_features = self._prepare_features(features)
is_identifier_column = np.array(
[
np.issubdtype(feature.dtype, np.integer) and self._is_sequential(feature)
for feature in processed_features
]
)
identifier_column_indices = np.where(is_identifier_column)
# this issue does not reflect rows at all so we set the score to 1.0 for all rows in the issue attribute
# and set the is_identifier_column_issue to False
num_rows = processed_features[0].size
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": False,
self.issue_score_key: np.ones(num_rows),
},
)
# score in summary should be 1.0 if the issue is not present and 0.0 if at least one column is an identifier column
self.summary = self.make_summary(score=1.0 - float(is_identifier_column.any()))
# more elegant way to set the score in summary
self.info = {
"identifier_columns": identifier_column_indices[0].tolist(),
"num_identifier_columns": identifier_column_indices[0].size,
}
cleanlab/datalab/internal/issue_manager/imbalance.py
+78
View File
@@ -0,0 +1,78 @@
from __future__ import annotations
from typing import TYPE_CHECKING, ClassVar
import numpy as np
import pandas as pd
from cleanlab.datalab.internal.issue_manager import IssueManager
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.datalab import Datalab
class ClassImbalanceIssueManager(IssueManager):
"""Manages issues related to imbalance class examples.
Parameters
----------
datalab:
The Datalab instance that this issue manager searches for issues in.
threshold:
Minimum fraction of samples of each class that are present in a dataset without class imbalance.
"""
description: ClassVar[str] = (
"""Examples belonging to the most under-represented class in the dataset."""
)
issue_name: ClassVar[str] = "class_imbalance"
verbosity_levels = {
0: ["Rarest Class"],
1: [],
2: [],
}
def __init__(self, datalab: Datalab, threshold: float = 0.1, **_):
super().__init__(datalab)
self.threshold = threshold
def find_issues(
self,
**kwargs,
) -> None:
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"Expected labels to be a numpy array of shape (n_samples,) to use with ClassImbalanceIssueManager, "
f"but got {type(labels)} instead."
)
raise TypeError(error_msg)
K = len(self.datalab.class_names)
class_probs = np.bincount(labels) / len(labels)
rarest_class_idx = int(np.argmin(class_probs))
# solely one class is identified as rarest, ties go to class w smaller integer index
scores = np.where(labels == rarest_class_idx, class_probs[rarest_class_idx], 1)
imbalance_exists = class_probs[rarest_class_idx] < self.threshold * (1 / K)
rarest_class_issue = rarest_class_idx if imbalance_exists else -1
is_issue_column = labels == rarest_class_issue
rarest_class_name = self.datalab._label_map.get(rarest_class_issue, "NA")
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_issue_column,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=class_probs[rarest_class_idx])
self.info = self.collect_info(class_name=rarest_class_name, labels=labels)
def collect_info(self, class_name: str, labels: np.ndarray) -> dict:
params_dict = {
"threshold": self.threshold,
"Rarest Class": class_name,
"given_label": labels,
}
info_dict = {**params_dict}
return info_dict
cleanlab/datalab/internal/issue_manager/issue_manager.py
+330
View File
@@ -0,0 +1,330 @@
from __future__ import annotations
from abc import ABC, ABCMeta, abstractmethod
from itertools import chain
from typing import TYPE_CHECKING, Any, ClassVar, Dict, List, Optional, Set, Tuple, Type, TypeVar
import json
import numpy as np
import pandas as pd
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.datalab import Datalab
T = TypeVar("T", bound="IssueManager")
TM = TypeVar("TM", bound="IssueManagerMeta")
class IssueManagerMeta(ABCMeta):
"""Metaclass for IssueManager that adds issue_score_key to the class.
:meta private:
"""
issue_name: ClassVar[str]
issue_score_key: ClassVar[str]
verbosity_levels: ClassVar[Dict[int, List[str]]] = {
0: [],
1: [],
2: [],
3: [],
}
def __new__(
meta: Type[TM],
name: str,
bases: Tuple[Type[Any], ...],
class_dict: Dict[str, Any],
) -> TM: # Classes that inherit from ABC don't need to be modified
if ABC in bases:
return super().__new__(meta, name, bases, class_dict)
# Ensure that the verbosity levels don't have keys other than those in ["issue", "info"]
verbosity_levels = class_dict.get("verbosity_levels", meta.verbosity_levels)
for level, level_list in verbosity_levels.items():
if not isinstance(level_list, list):
raise ValueError(
f"Verbosity levels must be lists. "
f"Got {level_list} in {name}.verbosity_levels"
)
prohibited_keys = [key for key in level_list if not isinstance(key, str)]
if prohibited_keys:
raise ValueError(
f"Verbosity levels must be lists of strings. "
f"Got {prohibited_keys} in {name}.verbosity_levels[{level}]"
)
# Concrete classes need to have an issue_name attribute
if "issue_name" not in class_dict:
raise TypeError("IssueManagers need an issue_name class variable")
# Add issue_score_key to class
class_dict["issue_score_key"] = f"{class_dict['issue_name']}_score"
return super().__new__(meta, name, bases, class_dict)
class IssueManager(ABC, metaclass=IssueManagerMeta):
"""Base class for managing data issues of a particular type in a Datalab.
For each example in a dataset, the IssueManager for a particular type of issue should compute:
- A numeric severity score between 0 and 1,
with values near 0 indicating severe instances of the issue.
- A boolean `is_issue` value, which is True
if we believe this example suffers from the issue in question.
`is_issue` may be determined by thresholding the severity score
(with an a priori determined reasonable threshold value),
or via some other means (e.g. Confident Learning for flagging label issues).
The IssueManager should also report:
- A global value between 0 and 1 summarizing how severe this issue is in the dataset overall
(e.g. the average severity across all examples in dataset
or count of examples where `is_issue=True`).
- Other interesting `info` about the issue and examples in the dataset,
and statistics estimated from current dataset that may be reused
to score this issue in future data.
For example, `info` for label issues could contain the:
confident_thresholds, confident_joint, predicted label for each example, etc.
Another example is for (near)-duplicate detection issue, where `info` could contain:
which set of examples in the dataset are all (nearly) identical.
Implementing a new IssueManager:
- Define the `issue_name` class attribute, e.g. "label", "duplicate", "outlier", etc.
- Implement the abstract methods `find_issues` and `collect_info`.
- `find_issues` is responsible for computing computing the `issues` and `summary` dataframes.
- `collect_info` is responsible for computing the `info` dict. It is called by `find_issues`,
once the manager has set the `issues` and `summary` dataframes as instance attributes.
"""
description: ClassVar[str] = ""
"""Short text that summarizes the type of issues handled by this IssueManager.
:meta hide-value:
"""
issue_name: ClassVar[str]
"""Returns a key that is used to store issue summary results about the assigned Lab."""
issue_score_key: ClassVar[str]
"""Returns a key that is used to store issue score results about the assigned Lab."""
verbosity_levels: ClassVar[Dict[int, List[str]]] = {
0: [],
1: [],
2: [],
3: [],
}
"""A dictionary of verbosity levels and their corresponding dictionaries of
report items to print.
:meta hide-value:
Example
-------
>>> verbosity_levels = {
... 0: [],
... 1: ["some_info_key"],
... 2: ["additional_info_key"],
... }
"""
def __init__(self, datalab: Datalab, **_):
self.datalab = datalab
self.info: Dict[str, Any] = {}
self.issues: pd.DataFrame = pd.DataFrame()
self.summary: pd.DataFrame = pd.DataFrame()
def __repr__(self):
class_name = self.__class__.__name__
return class_name
@classmethod
def __init_subclass__(cls):
required_class_variables = [
"issue_name",
]
for var in required_class_variables:
if not hasattr(cls, var):
raise NotImplementedError(f"Class {cls.__name__} must define class variable {var}")
@abstractmethod
def find_issues(self, *args, **kwargs) -> None:
"""Finds occurrences of this particular issue in the dataset.
Computes the `issues` and `summary` dataframes. Calls `collect_info` to compute the `info` dict.
"""
raise NotImplementedError
def collect_info(self, *args, **kwargs) -> dict:
"""Collects data for the info attribute of the Datalab.
NOTE
----
This method is called by :py:meth:`find_issues` after :py:meth:`find_issues` has set the `issues` and `summary` dataframes
as instance attributes.
"""
raise NotImplementedError
@classmethod
def make_summary(cls, score: float) -> pd.DataFrame:
"""Construct a summary dataframe.
Parameters
----------
score :
The overall score for this issue.
Returns
-------
summary :
A summary dataframe.
"""
if not 0 <= score <= 1:
raise ValueError(f"Score must be between 0 and 1. Got {score}.")
return pd.DataFrame(
{
"issue_type": [cls.issue_name],
"score": [score],
},
)
@classmethod
def report(
cls,
issues: pd.DataFrame,
summary: pd.DataFrame,
info: Dict[str, Any],
num_examples: int = 5,
verbosity: int = 0,
include_description: bool = False,
info_to_omit: Optional[List[str]] = None,
) -> str:
"""Compose a report of the issues found by this IssueManager.
Parameters
----------
issues :
An issues dataframe.
Example
-------
>>> import pandas as pd
>>> issues = pd.DataFrame(
... {
... "is_X_issue": [True, False, True],
... "X_score": [0.2, 0.9, 0.4],
... },
... )
summary :
The summary dataframe.
Example
-------
>>> summary = pd.DataFrame(
... {
... "issue_type": ["X"],
... "score": [0.5],
... },
... )
info :
The info dict.
Example
-------
>>> info = {
... "A": "val_A",
... "B": ["val_B1", "val_B2"],
... }
num_examples :
The number of examples to print.
verbosity :
The verbosity level of the report.
include_description :
Whether to include a description of the issue in the report.
Returns
-------
report_str :
A string containing the report.
"""
max_verbosity = max(cls.verbosity_levels.keys())
top_level = max_verbosity + 1
if verbosity not in list(cls.verbosity_levels.keys()) + [top_level]:
raise ValueError(
f"Verbosity level {verbosity} not supported. "
f"Supported levels: {cls.verbosity_levels.keys()}"
f"Use verbosity={top_level} to print all info."
)
if issues.empty:
print(f"No issues found")
topk_ids = issues.sort_values(by=cls.issue_score_key, ascending=True).index[:num_examples]
score = summary["score"].loc[0]
report_str = f"{' ' + cls.issue_name + ' issues ':-^60}\n\n"
if include_description and cls.description:
description = cls.description
if verbosity == 0:
description = description.split("\n\n", maxsplit=1)[0]
report_str += "About this issue:\n\t" + description + "\n\n"
report_str += (
f"Number of examples with this issue: {issues[f'is_{cls.issue_name}_issue'].sum()}\n"
f"Overall dataset quality in terms of this issue: {score:.4f}\n\n"
)
info_to_print: Set[str] = set()
_info_to_omit = set(issues.columns).union(info_to_omit or [])
verbosity_levels_values = chain.from_iterable(
list(cls.verbosity_levels.values())[: verbosity + 1]
)
info_to_print.update(set(verbosity_levels_values) - _info_to_omit)
if verbosity == top_level:
info_to_print.update(set(info.keys()) - _info_to_omit)
report_str += "Examples representing most severe instances of this issue:\n"
report_str += issues.loc[topk_ids].to_string()
def truncate(s, max_len=4) -> str:
if hasattr(s, "shape") or hasattr(s, "ndim"):
s = np.array(s)
if s.ndim > 1:
description = f"array of shape {s.shape}\n"
with np.printoptions(threshold=max_len):
if s.ndim == 2:
description += f"{s}"
if s.ndim > 2:
description += f"{s}"
return description
s = s.tolist()
if isinstance(s, list):
if all([isinstance(s_, list) for s_ in s]):
return truncate(np.array(s, dtype=object), max_len=max_len)
if len(s) > max_len:
s = s[:max_len] + ["..."]
return str(s)
if info_to_print:
info_to_print_dict = {key: info[key] for key in info_to_print}
# Print the info dict, truncating arrays to 4 elements,
report_str += f"\n\nAdditional Information: "
for key, value in info_to_print_dict.items():
if key == "statistics":
continue
if isinstance(value, dict):
report_str += f"\n{key}:\n{json.dumps(value, indent=4)}"
elif isinstance(value, pd.DataFrame):
max_rows = 5
df_str = value.head(max_rows).to_string()
if len(value) > max_rows:
df_str += f"\n... (total {len(value)} rows)"
report_str += f"\n{key}:\n{df_str}"
else:
report_str += f"\n{key}: {truncate(value)}"
return report_str
cleanlab/datalab/internal/issue_manager/knn_graph_helpers.py
+69
View File
@@ -0,0 +1,69 @@
import numpy.typing as npt
from scipy.sparse import csr_matrix
from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, cast
from cleanlab.internal.neighbor.knn_graph import create_knn_graph_and_index
from cleanlab.typing import Metric
if TYPE_CHECKING:
from sklearn.neighbors import NearestNeighbors
def num_neighbors_in_knn_graph(knn_graph: csr_matrix) -> int:
"""Calculate the number of neighbors per row in a knn graph."""
return knn_graph.nnz // knn_graph.shape[0]
def _process_knn_graph_from_inputs(
user_find_issues_kwargs: Dict[str, Any], statistics: Dict[str, Any], k_for_recomputation: int
) -> Optional[csr_matrix]:
"""Determine if a knn_graph is provided in the kwargs or if one is already stored in the associated Datalab instance."""
provided_knn_graph: Optional[csr_matrix] = user_find_issues_kwargs.get("knn_graph", None)
existing_knn_graph = statistics.get("weighted_knn_graph", None)
knn_graph: Optional[csr_matrix] = None
if provided_knn_graph is not None:
knn_graph = provided_knn_graph
elif existing_knn_graph is not None:
knn_graph = existing_knn_graph
num_neighbors = num_neighbors_in_knn_graph(knn_graph) if knn_graph is not None else -1
needs_recompute = k_for_recomputation > num_neighbors
if needs_recompute:
# If the provided knn graph is insufficient, then we need to recompute the knn graph
# with the provided features
knn_graph = None
return knn_graph
def knn_exists(kwargs: Dict[str, Any], statistics: Dict[str, Any], k_needed: int) -> bool:
"""Check if a sufficiently large knn graph exists in the kwargs or statistics."""
return (
_process_knn_graph_from_inputs(kwargs, statistics, k_for_recomputation=k_needed) is not None
)
def set_knn_graph(
features: Optional[npt.NDArray],
find_issues_kwargs: Dict[str, Any],
metric: Optional[Metric],
k: int,
statistics: Dict[str, Any],
) -> Tuple[csr_matrix, Metric, Optional["NearestNeighbors"]]:
# This only fetches graph (optionally)
knn_graph = _process_knn_graph_from_inputs(
find_issues_kwargs, statistics, k_for_recomputation=k
)
old_knn_metric = statistics.get("knn_metric", metric)
missing_knn_graph = knn_graph is None
metric_changes = metric and metric != old_knn_metric
knn: Optional[NearestNeighbors] = None
if missing_knn_graph or metric_changes:
assert features is not None, "Features must be provided to compute the knn graph."
knn_graph, knn = create_knn_graph_and_index(features, n_neighbors=k, metric=metric)
metric = knn.metric
return cast(csr_matrix, knn_graph), cast(Metric, metric), knn
cleanlab/datalab/internal/issue_manager/label.py
+273
View File
@@ -0,0 +1,273 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, ClassVar, Dict, Optional
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import OneHotEncoder
from cleanlab.classification import CleanLearning
from cleanlab.count import get_confident_thresholds
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.internal.validation import assert_valid_inputs
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
import pandas as pd
from cleanlab.datalab.datalab import Datalab
class LabelIssueManager(IssueManager):
"""Manages label issues in a Datalab.
Parameters
----------
datalab :
A Datalab instance.
k :
The number of nearest neighbors to consider when computing pred_probs from features.
Only applicable if features are provided and pred_probs are not.
clean_learning_kwargs :
Keyword arguments to pass to the :py:meth:`CleanLearning <cleanlab.classification.CleanLearning>` constructor.
health_summary_parameters :
Keyword arguments to pass to the :py:meth:`health_summary <cleanlab.dataset.health_summary>` function.
"""
description: ClassVar[
str
] = """Examples whose given label is estimated to be potentially incorrect
(e.g. due to annotation error) are flagged as having label issues.
"""
issue_name: ClassVar[str] = "label"
verbosity_levels = {
0: [],
1: [],
2: [],
3: ["classes_by_label_quality", "overlapping_classes"],
}
def __init__(
self,
datalab: Datalab,
k: int = 10,
clean_learning_kwargs: Optional[Dict[str, Any]] = None,
health_summary_parameters: Optional[Dict[str, Any]] = None,
**_,
):
super().__init__(datalab)
self.cl = CleanLearning(**(clean_learning_kwargs or {}))
self.k = k
self.health_summary_parameters: Dict[str, Any] = (
health_summary_parameters.copy() if health_summary_parameters else {}
)
self._find_issues_inputs: Dict[str, bool] = {"features": False, "pred_probs": False}
self._reset()
@staticmethod
def _process_find_label_issues_kwargs(**kwargs) -> Dict[str, Any]:
"""Searches for keyword arguments that are meant for the
CleanLearning.find_label_issues method call
Examples
--------
>>> from cleanlab.datalab.internal.issue_manager.label import LabelIssueManager
>>> LabelIssueManager._process_find_label_issues_kwargs(thresholds=[0.1, 0.9])
{'thresholds': [0.1, 0.9]}
"""
accepted_kwargs = [
"thresholds",
"noise_matrix",
"inverse_noise_matrix",
"save_space",
"clf_kwargs",
"validation_func",
]
return {k: v for k, v in kwargs.items() if k in accepted_kwargs and v is not None}
def _reset(self) -> None:
"""Reset the attributes of this manager based on the available datalab info
and the keyword arguments stored as instance attributes.
This allows the builder to use pre-computed info from the datalab to speed up
some computations in the :py:meth:`find_issues` method.
"""
if not self.health_summary_parameters:
statistics_dict = self.datalab.get_info("statistics")
self.health_summary_parameters = {
"labels": self.datalab.labels,
"class_names": list(self.datalab._label_map.values()),
"num_examples": statistics_dict.get("num_examples"),
"joint": statistics_dict.get("joint", None),
"confident_joint": statistics_dict.get("confident_joint", None),
"multi_label": statistics_dict.get("multi_label", None),
"asymmetric": statistics_dict.get("asymmetric", None),
"verbose": False,
}
self.health_summary_parameters = {
k: v for k, v in self.health_summary_parameters.items() if v is not None
}
def find_issues(
self,
pred_probs: Optional[npt.NDArray] = None,
features: Optional[npt.NDArray] = None,
**kwargs,
) -> None:
"""Find label issues in the datalab.
Parameters
----------
pred_probs :
The predicted probabilities for each example.
features :
The features for each example.
"""
if pred_probs is not None:
self._find_issues_inputs.update({"pred_probs": True})
if pred_probs is None:
self._find_issues_inputs.update({"features": True})
if features is None:
raise ValueError(
"Either pred_probs or features must be provided to find label issues."
)
# produce out-of-sample pred_probs from features
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"Expected labels to be a numpy array of shape (n_samples,) to use in LabelIssueManager, "
f"but got {type(labels)} instead."
)
raise TypeError(error_msg)
knn = KNeighborsClassifier(n_neighbors=self.k + 1)
knn.fit(features, labels)
pred_probs = knn.predict_proba(features)
encoder = OneHotEncoder()
label_transform = labels.reshape(-1, 1)
one_hot_label = encoder.fit_transform(label_transform)
# adjust pred_probs so it is out-of-sample
pred_probs = np.asarray(
(pred_probs - 1 / (self.k + 1) * one_hot_label) * (self.k + 1) / self.k
)
self.health_summary_parameters.update({"pred_probs": pred_probs})
# Find examples with label issues
labels = self.datalab.labels
self.issues = self.cl.find_label_issues(
labels=labels,
pred_probs=pred_probs,
**self._process_find_label_issues_kwargs(**kwargs),
)
self.issues.rename(columns={"label_quality": self.issue_score_key}, inplace=True)
summary_dict = self.get_health_summary(pred_probs=pred_probs)
# Get a summarized dataframe of the label issues
self.summary = self.make_summary(score=summary_dict["overall_label_health_score"])
confident_thresholds = get_confident_thresholds(labels=labels, pred_probs=pred_probs)
# Collect info about the label issues
self.info = self.collect_info(
issues=self.issues,
summary_dict=summary_dict,
confident_thresholds=confident_thresholds,
)
# Drop columns from issues that are in the info
self.issues = self.issues.drop(columns=["given_label", "predicted_label"])
def get_health_summary(self, pred_probs) -> dict:
"""Returns a short summary of the health of this Lab."""
from cleanlab.dataset import health_summary
# Validate input
self._validate_pred_probs(pred_probs)
summary_kwargs = self._get_summary_parameters(pred_probs)
summary = health_summary(**summary_kwargs)
return summary
def _get_summary_parameters(self, pred_probs) -> Dict["str", Any]:
"""Collects a set of input parameters for the health summary function based on
any info available in the datalab.
Parameters
----------
pred_probs :
The predicted probabilities for each example.
kwargs :
Keyword arguments to pass to the health summary function.
Returns
-------
summary_parameters :
A dictionary of parameters to pass to the health summary function.
"""
if "confident_joint" in self.health_summary_parameters:
summary_parameters = {
"confident_joint": self.health_summary_parameters["confident_joint"]
}
elif all([x in self.health_summary_parameters for x in ["joint", "num_examples"]]):
summary_parameters = {
k: self.health_summary_parameters[k] for k in ["joint", "num_examples"]
}
else:
summary_parameters = {
"pred_probs": pred_probs,
"labels": self.datalab.labels,
}
summary_parameters["class_names"] = self.health_summary_parameters["class_names"]
for k in ["asymmetric", "verbose"]:
# Start with the health_summary_parameters, then override with kwargs
if k in self.health_summary_parameters:
summary_parameters[k] = self.health_summary_parameters[k]
return (
summary_parameters # will be called in `dataset.health_summary(**summary_parameters)`
)
def collect_info(
self, issues: pd.DataFrame, summary_dict: dict, confident_thresholds: np.ndarray
) -> dict:
issues_info = {
"num_label_issues": sum(issues[f"is_{self.issue_name}_issue"]),
"average_label_quality": issues[self.issue_score_key].mean(),
"given_label": issues["given_label"].tolist(),
"predicted_label": issues["predicted_label"].tolist(),
}
health_summary_info = {
"confident_joint": summary_dict["joint"],
"classes_by_label_quality": summary_dict["classes_by_label_quality"],
"overlapping_classes": summary_dict["overlapping_classes"],
}
cl_info = {}
for k in self.cl.__dict__:
if k not in ["py", "noise_matrix", "inverse_noise_matrix", "confident_joint"]:
continue
cl_info[k] = self.cl.__dict__[k]
info_dict = {
**issues_info,
**health_summary_info,
**cl_info,
"confident_thresholds": confident_thresholds.tolist(),
"find_issues_inputs": self._find_issues_inputs,
}
return info_dict
def _validate_pred_probs(self, pred_probs) -> None:
assert_valid_inputs(X=None, y=self.datalab.labels, pred_probs=pred_probs)
cleanlab/datalab/internal/issue_manager/multilabel/__init__.py
+1
View File
@@ -0,0 +1 @@
from .label import MultilabelIssueManager
cleanlab/datalab/internal/issue_manager/multilabel/label.py
+135
View File
@@ -0,0 +1,135 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, ClassVar, Dict, List
import pandas as pd
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.internal.multilabel_utils import onehot2int
from cleanlab.multilabel_classification.filter import find_label_issues
from cleanlab.multilabel_classification.rank import get_label_quality_scores
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
import pandas as pd
from cleanlab.datalab.datalab import Datalab
class MultilabelIssueManager(IssueManager):
"""Manages label issues in Datalab for multilabel tasks.
Parameters
----------
datalab :
A Datalab instance.
"""
description: ClassVar[
str
] = """Examples whose given label(s) are estimated to be potentially incorrect
(e.g. due to annotation error) are flagged as having label issues.
"""
_PREDICTED_LABEL_THRESH = 0.5
"""Internal variable specifying threshold for predicted label."""
issue_name: ClassVar[str] = "label"
verbosity_levels = {
0: [],
1: [],
2: [],
3: [],
}
def __init__(
self,
datalab: Datalab,
**_,
):
super().__init__(datalab)
@staticmethod
def _process_find_label_issues_kwargs(**kwargs: Dict[str, Any]) -> Dict[str, Any]:
"""Searches for keyword arguments that are meant for the
multilabel_classification.filter.find_label_issues method call.
Examples
--------
>>> from cleanlab.datalab.internal.issue_manager.multilabel.label import MultilabelIssueManager
>>> MultilabelIssueManager._process_find_label_issues_kwargs(frac_noise=0.9)
{'frac_noise': 0.9}
"""
accepted_kwargs = [
"filter_by",
"frac_noise",
"num_to_remove_per_class",
"min_examples_per_class",
"confident_joint",
"n_jobs",
"verbose",
"low_memory",
]
return {k: v for k, v in kwargs.items() if k in accepted_kwargs and v is not None}
@staticmethod
def _process_get_label_quality_scores_kwargs(**kwargs: Dict[str, Any]) -> Dict[str, Any]:
"""Searches for keyword arguments that are meant for the
multilabel_classification.rank.get_label_quality_scores method call.
Examples
--------
>>> from cleanlab.datalab.internal.issue_manager.multilabel.label import MultilabelIssueManager
>>> MultilabelIssueManager._process_get_label_quality_scores_kwargs(method="self_confidence")
{'method': 'self_confidence'}
"""
accepted_kwargs = ["method", "adjust_pred_probs", "aggregator_kwargs"]
return {k: v for k, v in kwargs.items() if k in accepted_kwargs and v is not None}
def find_issues(
self,
pred_probs: npt.NDArray,
**kwargs,
) -> None:
"""Find label issues in a multilabel dataset.
Parameters
----------
pred_probs :
The predicted probabilities for each example.
"""
predicted_labels = onehot2int(pred_probs > self._PREDICTED_LABEL_THRESH)
# Find examples with label issues
assert isinstance(self.datalab.labels, List) # Type Narrowing
is_issue_column = find_label_issues(
labels=self.datalab.labels,
pred_probs=pred_probs,
**self._process_find_label_issues_kwargs(**kwargs),
)
scores = get_label_quality_scores(
labels=self.datalab.labels,
pred_probs=pred_probs,
**self._process_get_label_quality_scores_kwargs(**kwargs),
)
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_issue_column,
self.issue_score_key: scores,
},
)
# Get a summarized dataframe of the label issues
self.summary = self.make_summary(score=scores.mean())
# Collect info about the label issues
self.info = self.collect_info(self.datalab.labels, predicted_labels)
def collect_info(
self, given_labels: List[List[int]], predicted_labels: List[List[int]]
) -> Dict[str, Any]:
issues_info = {
"given_label": given_labels,
"predicted_label": predicted_labels,
}
return issues_info
cleanlab/datalab/internal/issue_manager/noniid.py
+446
View File
@@ -0,0 +1,446 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable, ClassVar, Dict, Optional, Union, cast
import itertools
from scipy.stats import gaussian_kde
import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.datalab.internal.issue_manager.knn_graph_helpers import knn_exists, set_knn_graph
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
from cleanlab.datalab.datalab import Datalab
def simplified_kolmogorov_smirnov_test(
neighbor_histogram: npt.NDArray[np.float64],
non_neighbor_histogram: npt.NDArray[np.float64],
) -> float:
"""Computes the Kolmogorov-Smirnov statistic between two groups of data.
The statistic is the largest difference between the empirical cumulative
distribution functions (ECDFs) of the two groups.
Parameters
----------
neighbor_histogram :
Histogram data for the nearest neighbor group.
non_neighbor_histogram :
Histogram data for the non-neighbor group.
Returns
-------
statistic :
The KS statistic between the two ECDFs.
Note
----
- Both input arrays should have the same length.
- The input arrays are histograms, which means they contain the count
or frequency of values in each group. The data in the histograms
should be normalized so that they sum to one.
To calculate the KS statistic, the function first calculates the ECDFs
for both input arrays, which are step functions that show the cumulative
sum of the data up to each point. The function then calculates the
largest absolute difference between the two ECDFs.
"""
neighbor_cdf = np.cumsum(neighbor_histogram)
non_neighbor_cdf = np.cumsum(non_neighbor_histogram)
statistic = np.max(np.abs(neighbor_cdf - non_neighbor_cdf))
return statistic
class NonIIDIssueManager(IssueManager):
"""Manages issues related to non-iid data distributions.
Parameters
----------
datalab :
The Datalab instance that this issue manager searches for issues in.
metric :
The distance metric used to compute the KNN graph of the examples in the dataset.
If set to `None`, the metric will be automatically selected based on the dimensionality
of the features used to represent the examples in the dataset.
k :
The number of nearest neighbors to consider when computing the KNN graph of the examples.
num_permutations :
The number of trials to run when performing permutation testing to determine whether
the distribution of index-distances between neighbors in the dataset is IID or not.
Note
----
This class will only flag a single example as an issue if the dataset is considered non-IID. This type of issue
is more relevant to the entire dataset as a whole, rather than to individual examples.
"""
description: ClassVar[
str
] = """Whether the dataset exhibits statistically significant
violations of the IID assumption like:
changepoints or shift, drift, autocorrelation, etc.
The specific violation considered is whether the
examples are ordered such that almost adjacent examples
tend to have more similar feature values.
"""
issue_name: ClassVar[str] = "non_iid"
verbosity_levels = {
0: ["p-value"],
1: [],
2: [],
}
def __init__(
self,
datalab: Datalab,
metric: Optional[Union[str, Callable]] = None,
k: int = 10,
num_permutations: int = 25,
seed: Optional[int] = 0,
significance_threshold: float = 0.05,
**_,
):
super().__init__(datalab)
self.metric = metric
self.k = k
self.num_permutations = num_permutations
self.tests = {
"ks": simplified_kolmogorov_smirnov_test,
}
self.background_distribution = None
self.seed = seed
self.significance_threshold = significance_threshold
# TODO: Temporary flag introduced to decide on storing knn graphs based on pred_probs.
# Revisit and finalize the implementation.
self._skip_storing_knn_graph_for_pred_probs: bool = False
@staticmethod
def _determine_optional_features(
features: Optional[npt.NDArray],
pred_probs: Optional[np.ndarray],
) -> Optional[npt.NDArray]:
"""
Determines the feature array to be used for constructing a knn-graph. Prioritizing the original features array over pred_probs.
If neither are provided, returns None.
Parameters
----------
features :
Original feature array or None.
pred_probs :
Predicted probabilities array or None.
Returns
-------
features_to_use :
Either the original feature array or the predicted probabilities array,
intended for constructing the knn-graph.
Notes
-----
A knn-graph constructed from predicted probabilities should not be stored in the statistics. But this kind
of knn-graph is allowed for the purpose of running a non-IID check.
"""
if features is not None:
return features
if pred_probs is not None:
return pred_probs
return None
def find_issues(
self,
features: Optional[npt.NDArray] = None,
pred_probs: Optional[np.ndarray] = None,
**kwargs,
) -> None:
statistics = self.datalab.get_info("statistics")
# Crucial when building knn graphs with pred_probs instead of features, where only the
# latter is preferred for storage.
self._determine_if_knn_graph_storage_should_be_skipped(
features, pred_probs, kwargs, statistics, self.k
)
knn_graph, self.metric, _ = set_knn_graph(
features=self._determine_optional_features(features, pred_probs),
find_issues_kwargs=kwargs,
metric=self.metric,
k=self.k,
statistics=statistics,
)
self.neighbor_index_choices = self._get_neighbors(knn_graph=knn_graph)
self.num_neighbors = self.k
indices = np.arange(self.N)
self.neighbor_index_distances = np.abs(indices.reshape(-1, 1) - self.neighbor_index_choices)
self.statistics = self._get_statistics(self.neighbor_index_distances)
self.p_value = self._permutation_test(num_permutations=self.num_permutations)
scores = self._score_dataset()
issue_mask = np.zeros(self.N, dtype=bool)
if self.p_value < self.significance_threshold:
issue_mask[scores.argmin()] = True
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": issue_mask,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=self.p_value)
self.info = self.collect_info(knn_graph=knn_graph)
def _determine_if_knn_graph_storage_should_be_skipped(
self, features, pred_probs, kwargs, statistics, k
) -> None:
"""Decide whether to skip storing the knn graph based on the availability of pred_probs.
Should only happend when a new knn graph needs to be computed, and that it
can only be computed from pred_probs.
"""
sufficient_knn_graph_available = knn_exists(kwargs, statistics, k)
pred_probs_needed = (
not sufficient_knn_graph_available and features is None and pred_probs is not None
)
if pred_probs_needed:
self._skip_storing_knn_graph_for_pred_probs = True
def collect_info(self, knn_graph: csr_matrix) -> dict:
issues_dict = {
"p-value": self.p_value,
}
params_dict = {
"metric": self.metric,
"k": self.k,
}
statistics_dict = self._build_statistics_dictionary(knn_graph=knn_graph)
info_dict = {
**issues_dict,
**params_dict, # type: ignore[arg-type]
**statistics_dict, # type: ignore[arg-type]
}
return info_dict
def _build_statistics_dictionary(self, knn_graph: csr_matrix) -> Dict[str, Dict[str, Any]]:
statistics_dict: Dict[str, Dict[str, Any]] = {"statistics": {}}
if self._skip_storing_knn_graph_for_pred_probs:
return statistics_dict
# Add the knn graph as a statistic if necessary
graph_key = "weighted_knn_graph"
old_knn_graph = self.datalab.get_info("statistics").get(graph_key, None)
old_graph_exists = old_knn_graph is not None
prefer_new_graph = (
(knn_graph is not None and not old_graph_exists)
or (old_knn_graph is not None and knn_graph.nnz > old_knn_graph.nnz)
or self.metric != self.datalab.get_info("statistics").get("knn_metric", None)
)
if prefer_new_graph:
statistics_dict["statistics"][graph_key] = knn_graph
if self.metric is not None:
statistics_dict["statistics"]["knn_metric"] = self.metric
return statistics_dict
def _permutation_test(self, num_permutations) -> float:
N = self.N
if self.seed is not None:
np.random.seed(self.seed)
perms = np.fromiter(
itertools.chain.from_iterable(
np.random.permutation(N) for i in range(num_permutations)
),
dtype=int,
).reshape(num_permutations, N)
neighbor_index_choices = self.neighbor_index_choices
neighbor_index_choices = neighbor_index_choices.reshape(1, *neighbor_index_choices.shape)
perm_neighbor_choices = perms[:, neighbor_index_choices].reshape(
num_permutations, *neighbor_index_choices.shape[1:]
)
neighbor_index_distances = np.abs(perms[..., None] - perm_neighbor_choices).reshape(
num_permutations, -1
)
statistics = []
for neighbor_index_dist in neighbor_index_distances:
stats = self._get_statistics(
neighbor_index_dist,
)
statistics.append(stats)
ks_stats = np.array([stats["ks"] for stats in statistics])
ks_stats_kde = gaussian_kde(ks_stats)
p_value = ks_stats_kde.integrate_box(self.statistics["ks"], 100)
return p_value
def _score_dataset(self) -> npt.NDArray[np.float64]:
"""This function computes a variant of the KS statistic for each
datapoint. Rather than computing the maximum difference
between the CDF of the neighbor distances (foreground
distribution) and the CDF of the all index distances
(background distribution), we compute the absolute difference
in area-under-the-curve of the two CDFs.
The foreground distribution is computed by sampling the
neighbor distances from the KNN graph, but the background
distribution is computed analytically. The background CDF for
a datapoint i can be split up into three parts. Let d = min(i,
N - i - 1).
1. For 0 < j <= d, the slope of the CDF is 2 / (N - 1) since
there are two datapoints in the dataset that are distance j
from datapoint i. We call this threshold the 'double distance
threshold'
2. For d < j <= N - d - 1, the slope of the CDF is
1 / (N - 1) since there is only one datapoint in the dataset
that is distance j from datapoint i.
3. For j > N - d - 1, the slope of the CDF is 0 and is
constant at 1.0 since there are no datapoints in the dataset
that are distance j from datapoint i.
We compute the area differences on each of the k intervals for
which the foreground CDF is constant which allows for the
possibility that the background CDF may intersect the
foreground CDF on this interval. We do not account for these
cases when computing absolute AUC difference.
Our algorithm is simple, sort the k sampled neighbor
distances. Then, for each of the k neighbor distances sampled,
compute the AUC for each CDF up to that point. Then, subtract
from each area the previous area in the sorted order to get
the AUC of the CDF on the interval between those two
points. Subtract the background interval AUCs from the
foreground interval AUCs, take the absolute value, and
sum. The algorithm is vectorized such that this statistic is
computed for each of the N datapoints simultaneously.
The statistics are then normalized by their respective maximum
possible distance (N - d - 1) and then mapped to [0,1] via
tanh.
"""
N = self.N
sorted_neighbors = np.sort(self.neighbor_index_distances, axis=1)
# find the maximum distance that occurs with double probability
middle_idx = np.floor((N - 1) / 2).astype(int)
double_distances = np.arange(N).reshape(N, 1)
double_distances[double_distances > middle_idx] -= N - 1
double_distances = np.abs(double_distances)
sorted_neighbors = np.hstack([sorted_neighbors, np.ones((N, 1)) * (N - 1)]).astype(int)
# the set of distances that are less than the double distance threshold
set_beginning = sorted_neighbors <= double_distances
# the set of distances that are greater than the double distance threshold but have nonzero probability
set_middle = (sorted_neighbors > double_distances) & (
sorted_neighbors <= (N - double_distances - 1)
)
# the set of distances that occur with 0 probability
set_end = sorted_neighbors > (N - double_distances - 1)
shifted_neighbors = np.zeros(sorted_neighbors.shape)
shifted_neighbors[:, 1:] = sorted_neighbors[:, :-1]
diffs = sorted_neighbors - shifted_neighbors # the distances between the sorted indices
area_beginning = (double_distances**2) / (N - 1)
length = N - 2 * double_distances - 1
a = 2 * double_distances / (N - 1)
area_middle = 0.5 * (a + 1) * length
# compute the area under the CDF for each of the indices in sorted_neighbors
background_area = np.zeros(diffs.shape)
background_diffs = np.zeros(diffs.shape)
background_area[set_beginning] = ((sorted_neighbors**2) / (N - 1))[set_beginning]
background_area[set_middle] = (
area_beginning
+ 0.5
* (
(sorted_neighbors + 3 * double_distances)
* (sorted_neighbors - double_distances)
/ (N - 1)
)
)[set_middle]
background_area[set_end] = (
area_beginning + area_middle + (sorted_neighbors - (N - double_distances - 1) * 1.0)
)[set_end]
# compute the area under the CDF between indices in sorted_neighbors
shifted_background = np.zeros(background_area.shape)
shifted_background[:, 1:] = background_area[:, :-1]
background_diffs = background_area - shifted_background
# compute the foreground CDF and AUC between indices in sorted_neighbors
foreground_cdf = np.arange(sorted_neighbors.shape[1]) / (sorted_neighbors.shape[1] - 1)
foreground_diffs = foreground_cdf.reshape(1, -1) * diffs
# compute the differences between foreground and background area intervals
area_diffs = np.abs(foreground_diffs - background_diffs)
stats = np.sum(area_diffs, axis=1)
# normalize scores by the index and transform to [0, 1]
indices = np.arange(N)
reverse = N - indices
normalizer = np.where(indices > reverse, indices, reverse)
scores = stats / normalizer
scores = np.tanh(-1 * scores) + 1
return scores
def _get_neighbors(self, knn_graph: csr_matrix) -> np.ndarray:
"""
Given a knn graph, returns an (N, k) array in
which j is in A[i] if item i and j are nearest neighbors.
"""
self.N = knn_graph.shape[0]
kneighbors = knn_graph.indices.reshape(self.N, -1)
return kneighbors
def _get_statistics(
self,
neighbor_index_distances,
) -> dict[str, float]:
neighbor_index_distances = neighbor_index_distances.flatten()
sorted_neighbors = np.sort(neighbor_index_distances)
sorted_neighbors = np.hstack([sorted_neighbors, np.ones((1)) * (self.N - 1)]).astype(int)
if self.background_distribution is None:
self.background_distribution = (self.N - np.arange(1, self.N)) / (
self.N * (self.N - 1) / 2
)
background_distribution = cast(np.ndarray, self.background_distribution)
background_cdf = np.cumsum(background_distribution)
foreground_cdf = np.arange(sorted_neighbors.shape[0]) / (sorted_neighbors.shape[0] - 1)
statistic = np.max(np.abs(foreground_cdf - background_cdf[sorted_neighbors - 1]))
statistics = {"ks": statistic}
return statistics
cleanlab/datalab/internal/issue_manager/null.py
+210
View File
@@ -0,0 +1,210 @@
from __future__ import annotations
from collections import Counter
from typing import TYPE_CHECKING, Any, ClassVar, Dict, List, Optional
import numpy as np
import pandas as pd
from cleanlab.datalab.internal.issue_manager import IssueManager
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
class NullIssueManager(IssueManager):
"""Manages issues related to null/missing values in the rows of features.
Parameters
----------
datalab :
The Datalab instance that this issue manager searches for issues in.
"""
description: ClassVar[
str
] = """Examples identified with the null issue correspond to rows that have null/missing values across all feature columns (i.e. the entire row is missing values).
"""
issue_name: ClassVar[str] = "null"
verbosity_levels = {
0: [],
1: [],
2: ["most_common_issue"],
}
@staticmethod
def _calculate_null_issues(
features: npt.NDArray[Any],
) -> tuple[npt.NDArray[np.bool_], npt.NDArray[np.float64], npt.NDArray[np.bool_]]:
"""Tracks the number of null values in each row of a feature array,
computes quality scores based on the fraction of null values in each row,
and returns a boolean array indicating whether each row only has null values."""
cols = features.shape[1]
null_tracker = pd.isna(features)
non_null_count = cols - null_tracker.sum(axis=1)
scores = non_null_count / cols
is_null_issue = non_null_count == 0
return is_null_issue, scores, null_tracker
def find_issues(
self,
features: Optional[npt.NDArray | pd.DataFrame] = None,
**kwargs,
) -> None:
if features is None:
raise ValueError("features must be provided to check for null values.")
# Support features as a numpy array. Temporarily allow this issuecheck to convert a DataFrame to a numpy array.
if isinstance(features, pd.DataFrame):
features = features.to_numpy()
is_null_issue, scores, null_tracker = self._calculate_null_issues(features=features)
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_null_issue,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=scores.mean())
self.info = self.collect_info(null_tracker)
@staticmethod
def _most_common_issue(
null_tracker: np.ndarray,
) -> dict[str, dict[str, str | int | list[int] | list[int | None]]]:
"""
Identify and return the most common null value pattern across all rows
and count the number of rows with this pattern.
Parameters
------------
null_tracker : np.ndarray
A boolean array of the same shape as features, where True indicates null/missing entries.
Returns
--------
Dict[str, Any]
A dictionary containing the most common issue pattern and the count of rows with this pattern.
"""
# Convert the boolean null_tracker matrix into a list of strings.
most_frequent_pattern = "no_null"
rows_affected: List[int] = []
occurrence_of_most_frequent_pattern = 0
if np.any(null_tracker, axis=None):
null_row_indices = np.where(np.any(null_tracker, axis=1))[0]
null_patterns_as_strings = [
"".join(map(str, null_tracker[i].astype(int).tolist())) for i in null_row_indices
]
# Use Counter to efficiently count occurrences and find the most common pattern.
pattern_counter = Counter(null_patterns_as_strings)
(
most_frequent_pattern,
occurrence_of_most_frequent_pattern,
) = pattern_counter.most_common(1)[0]
rows_affected = []
for idx, row in enumerate(null_patterns_as_strings):
if row == most_frequent_pattern:
rows_affected.append(int(null_row_indices[idx]))
return {
"most_common_issue": {
"pattern": most_frequent_pattern,
"rows_affected": rows_affected,
"count": occurrence_of_most_frequent_pattern,
}
}
@staticmethod
def _column_impact(null_tracker: np.ndarray) -> Dict[str, List[float]]:
"""
Calculate and return the impact of null values per column, represented as the proportion
of rows having null values in each column.
Parameters
----------
null_tracker : np.ndarray
A boolean array of the same shape as features, where True indicates null/missing entries.
Returns
-------
Dict[str, List[float]]
A dictionary containing the impact per column, with values being a list
where each element is the percentage of rows having null values in the corresponding column.
"""
# Calculate proportion of nulls in each column
proportion_of_nulls_per_column = null_tracker.mean(axis=0)
# Return result as a dictionary containing a list of proportions
return {"column_impact": proportion_of_nulls_per_column.tolist()}
def collect_info(self, null_tracker: np.ndarray) -> dict:
most_common_issue = self._most_common_issue(null_tracker=null_tracker)
column_impact = self._column_impact(null_tracker=null_tracker)
average_null_score = {"average_null_score": self.issues[self.issue_score_key].mean()}
issues_dict = {**average_null_score, **most_common_issue, **column_impact}
info_dict: Dict[str, Any] = {**issues_dict}
return info_dict
@classmethod
def report(cls, *args, **kwargs) -> str:
"""
Return a report of issues found by the NullIssueManager.
This method extends the superclass method by identifying and reporting
specific issues related to null values in the dataset.
Parameters
----------
*args : list
Variable length argument list.
**kwargs : dict
Arbitrary keyword arguments.
Returns
-------
report_str :
A string containing the report.
See Also
--------
:meth:`cleanlab.datalab.Datalab.report`
Notes
-----
This method differs from other IssueManager report methods. It checks for issues
and prompts the user to address them to enable other issue managers to run effectively.
"""
# Generate the base report using the superclass method
original_report = super().report(*args, **kwargs)
# Retrieve the 'issues' dataframe from keyword arguments
issues = kwargs["issues"]
# Identify examples that have null values in all features
issue_filter = f"is_{cls.issue_name}_issue"
examples_with_full_nulls = issues.query(issue_filter).index.tolist()
# Identify examples that have some null values (but not in all features)
partial_null_filter = f"{cls.issue_score_key} < 1.0 and not {issue_filter}"
examples_with_partial_nulls = issues.query(partial_null_filter).index.tolist()
# Append information about examples with null values in all features
if examples_with_full_nulls:
report_addition = (
f"\n\nFound {len(examples_with_full_nulls)} examples with null values in all features. "
f"These examples should be removed from the dataset before running other issue managers."
# TODO: Add a link to the documentation on how to handle null examples
)
original_report += report_addition
# Append information about examples with some null values
if examples_with_partial_nulls:
report_addition = (
f"\n\nFound {len(examples_with_partial_nulls)} examples with null values in some features. "
f"Please address these issues before running other issue managers."
# TODO: Add a link to the documentation on how to handle partially null examples
)
original_report += report_addition
return original_report
cleanlab/datalab/internal/issue_manager/outlier.py
+301
View File
@@ -0,0 +1,301 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, ClassVar, Dict, Optional, Tuple
from scipy.sparse import csr_matrix
from scipy.stats import iqr
import numpy as np
import pandas as pd
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.datalab.internal.issue_manager.knn_graph_helpers import knn_exists, set_knn_graph
from cleanlab.internal.outlier import correct_precision_errors
from cleanlab.outlier import OutOfDistribution, transform_distances_to_scores
if TYPE_CHECKING: # pragma: no cover
from sklearn.neighbors import NearestNeighbors
import numpy.typing as npt
from cleanlab.datalab.datalab import Datalab
from cleanlab.typing import Metric
class OutlierIssueManager(IssueManager):
"""Manages issues related to out-of-distribution examples."""
description: ClassVar[
str
] = """Examples that are very different from the rest of the dataset
(i.e. potentially out-of-distribution or rare/anomalous instances).
"""
issue_name: ClassVar[str] = "outlier"
verbosity_levels = {
0: [],
1: [],
2: ["average_ood_score"],
3: [],
}
DEFAULT_THRESHOLDS = {
"features": 0.37037,
"pred_probs": 0.13,
}
"""Default thresholds for outlier detection.
If outlier detection is performed on the features, an example whose average
distance to their k nearest neighbors is greater than
Q3_avg_dist + (1 / threshold - 1) * IQR_avg_dist is considered an outlier.
If outlier detection is performed on the predicted probabilities, an example
whose average score is lower than threshold * median_outlier_score is
considered an outlier.
"""
def __init__(
self,
datalab: Datalab,
k: int = 10,
t: int = 1,
metric: Optional[Metric] = None,
scaling_factor: Optional[float] = None,
threshold: Optional[float] = None,
**kwargs,
):
super().__init__(datalab)
ood_kwargs = kwargs.get("ood_kwargs", {})
valid_ood_params = OutOfDistribution.DEFAULT_PARAM_DICT.keys()
params = {
key: value
for key, value in ((k, kwargs.get(k, None)) for k in valid_ood_params)
if value is not None
}
# Simplified API: directly specify k and metric instead of NearestNeighbors object
# This reduces dependency on OutOfDistribution and aligns with Datalab's approach
params["k"] = k
self.k = k
self.t = t
self.metric: Optional[Metric] = metric
self.scaling_factor = scaling_factor
if params:
ood_kwargs["params"] = params
# OutOfDistribution still used for pred-prob based outlier detection
self.ood: OutOfDistribution = OutOfDistribution(**ood_kwargs)
self._find_issues_inputs: Dict[str, bool] = {
"features": False,
"pred_probs": False,
"knn_graph": False,
}
# Used for both methods of outlier detection
self.threshold = threshold
def find_issues(
self,
features: Optional[npt.NDArray] = None,
pred_probs: Optional[np.ndarray] = None,
**kwargs,
) -> None:
statistics = self.datalab.get_info("statistics")
# Determine if we can use kNN-based outlier detection
knn_graph_works: bool = self._knn_graph_works(features, kwargs, statistics, self.k)
knn_graph = None
knn = None
if knn_graph_works:
# Set up or retrieve the kNN graph
knn_graph, self.metric, knn = set_knn_graph(
features=features,
find_issues_kwargs=kwargs,
metric=self.metric,
k=self.k,
statistics=statistics,
)
# Compute distances and thresholds for outlier detection
distances = knn_graph.data.reshape(knn_graph.shape[0], -1)
assert isinstance(distances, np.ndarray)
(
self.threshold,
issue_threshold, # Useful info for detecting issues in test data
is_issue_column,
) = self._compute_threshold_and_issue_column_from_distances(distances, self.threshold)
# Calculate outlier scores based on average distances
avg_distances = distances.mean(axis=1)
median_avg_distance = np.median(avg_distances)
self._find_issues_inputs.update({"knn_graph": True})
# Ensure scaling factor is not too small to avoid numerical issues
if self.scaling_factor is None:
self.scaling_factor = float(
max(median_avg_distance, 100 * np.finfo(np.float64).eps)
)
scores = transform_distances_to_scores(
avg_distances, t=self.t, scaling_factor=self.scaling_factor
)
# Apply precision error correction if metric is available
_metric = self.metric
if _metric is not None:
_metric = _metric if isinstance(_metric, str) else _metric.__name__
scores = correct_precision_errors(scores, avg_distances, _metric)
elif pred_probs is not None:
# Fallback to prediction probabilities-based outlier detection
scores = self._score_with_pred_probs(pred_probs, **kwargs)
self._find_issues_inputs.update({"pred_probs": True})
# Set threshold for pred_probs-based detection
if self.threshold is None:
self.threshold = self.DEFAULT_THRESHOLDS["pred_probs"]
if not 0 <= self.threshold:
raise ValueError(f"threshold must be non-negative, but got {self.threshold}.")
issue_threshold = float(
self.threshold * np.median(scores)
) # Useful info for detecting issues in test data
is_issue_column = scores < issue_threshold
else:
# Handle case where neither kNN nor pred_probs-based detection is possible
if (
kwargs.get("knn_graph", None) is not None
or statistics.get("weighted_knn_graph", None) is not None
):
raise ValueError(
"knn_graph is provided, but not sufficiently large to compute the scores based on the provided hyperparameters."
)
raise ValueError(f"Either features pred_probs must be provided.")
# Store results
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_issue_column,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=scores.mean())
self.info = self.collect_info(issue_threshold=issue_threshold, knn_graph=knn_graph, knn=knn)
def _knn_graph_works(self, features, kwargs, statistics, k: int) -> bool:
"""Decide whether to skip the knn-based outlier detection and rely on pred_probs instead."""
sufficient_knn_graph_available = knn_exists(kwargs, statistics, k)
return (features is not None) or sufficient_knn_graph_available
def _compute_threshold_and_issue_column_from_distances(
self, distances: np.ndarray, threshold: Optional[float] = None
) -> Tuple[float, float, np.ndarray]:
avg_distances = distances.mean(axis=1)
if threshold:
if not (isinstance(threshold, (int, float)) and 0 <= threshold <= 1):
raise ValueError(
f"threshold must be a number between 0 and 1, got {threshold} of type {type(threshold)}."
)
if threshold is None:
threshold = OutlierIssueManager.DEFAULT_THRESHOLDS["features"]
def compute_issue_threshold(avg_distances: np.ndarray, threshold: float) -> float:
q3_distance = np.percentile(avg_distances, 75)
iqr_scale = 1 / threshold - 1 if threshold != 0 else np.inf
issue_threshold = q3_distance + iqr_scale * iqr(avg_distances)
return float(issue_threshold)
issue_threshold = compute_issue_threshold(avg_distances, threshold)
return threshold, issue_threshold, avg_distances > issue_threshold
def collect_info(
self,
*,
issue_threshold: float,
knn_graph: Optional[csr_matrix],
knn: Optional["NearestNeighbors"],
) -> dict:
issues_dict = {
"average_ood_score": self.issues[self.issue_score_key].mean(),
"threshold": self.threshold,
"issue_threshold": issue_threshold,
}
pred_probs_issues_dict: Dict[str, Any] = {}
feature_issues_dict = {}
if knn_graph is not None:
N = knn_graph.shape[0]
k = knn_graph.nnz // N
dists = knn_graph.data.reshape(N, -1)[:, 0]
nn_ids = knn_graph.indices.reshape(N, -1)[:, 0]
feature_issues_dict.update(
{
"k": self.k, # type: ignore[union-attr]
"nearest_neighbor": nn_ids.tolist(),
"distance_to_nearest_neighbor": dists.tolist(),
"metric": self.metric, # type: ignore[union-attr]
"scaling_factor": self.scaling_factor,
"t": self.t,
"knn": knn,
}
)
if self.ood.params["confident_thresholds"] is not None:
pass #
statistics_dict = self._build_statistics_dictionary(knn_graph=knn_graph)
ood_params_dict = {
"ood": self.ood,
**self.ood.params,
}
knn_dict = {
**pred_probs_issues_dict,
**feature_issues_dict,
}
info_dict: Dict[str, Any] = {
**issues_dict,
**ood_params_dict, # type: ignore[arg-type]
**knn_dict,
**statistics_dict,
"find_issues_inputs": self._find_issues_inputs,
}
return info_dict
def _build_statistics_dictionary(
self, *, knn_graph: Optional[csr_matrix]
) -> Dict[str, Dict[str, Any]]:
statistics_dict: Dict[str, Dict[str, Any]] = {"statistics": {}}
# Add the knn graph as a statistic if necessary
graph_key = "weighted_knn_graph"
old_knn_graph = self.datalab.get_info("statistics").get(graph_key, None)
old_graph_exists = old_knn_graph is not None
prefer_new_graph = (
not old_graph_exists
or (
isinstance(knn_graph, csr_matrix)
and old_knn_graph is not None
and knn_graph.nnz > old_knn_graph.nnz
)
or self.metric != self.datalab.get_info("statistics").get("knn_metric", None)
)
if prefer_new_graph:
if knn_graph is not None:
statistics_dict["statistics"][graph_key] = knn_graph
if self.metric is not None:
statistics_dict["statistics"]["knn_metric"] = self.metric
return statistics_dict
def _score_with_pred_probs(self, pred_probs: np.ndarray, **kwargs) -> np.ndarray:
# Remove "threshold" from kwargs if it exists
kwargs.pop("threshold", None)
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"labels must be a numpy array of shape (n_samples,) to use the OutlierIssueManager "
f"with pred_probs, but got {type(labels)}."
)
raise TypeError(error_msg)
scores = self.ood.fit_score(pred_probs=pred_probs, labels=labels, **kwargs)
return scores
cleanlab/datalab/internal/issue_manager/regression/__init__.py
+1
View File
@@ -0,0 +1 @@
from .label import RegressionLabelIssueManager
cleanlab/datalab/internal/issue_manager/regression/label.py
+241
View File
@@ -0,0 +1,241 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, ClassVar, Dict, Optional
import numpy as np
import pandas as pd
from cleanlab.regression.learn import CleanLearning
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.regression.rank import get_label_quality_scores
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.datalab import Datalab
class RegressionLabelIssueManager(IssueManager):
"""Manages label issues in a Datalab for regression tasks.
Parameters
----------
datalab :
A Datalab instance.
clean_learning_kwargs :
Keyword arguments to pass to the :py:meth:`regression.learn.CleanLearning <cleanlab.regression.learn.CleanLearning>` constructor.
threshold :
The threshold to use to determine if an example has a label issue. It is a multiplier
of the median label quality score that sets the absolute threshold. Only used if
predictions are provided to `~RegressionLabelIssueManager.find_issues`, not if
features are provided. Default is 0.05.
"""
description: ClassVar[
str
] = """Examples whose given label is estimated to be potentially incorrect
(e.g. due to annotation error) are flagged as having label issues.
"""
issue_name: ClassVar[str] = "label"
verbosity_levels = {
0: [],
1: [],
2: [],
3: [], # TODO
}
def __init__(
self,
datalab: Datalab,
clean_learning_kwargs: Optional[Dict[str, Any]] = None,
threshold: float = 0.05,
health_summary_parameters: Optional[Dict[str, Any]] = None,
**_,
):
super().__init__(datalab)
self.cl = CleanLearning(**(clean_learning_kwargs or {}))
# This is a field for prioritizing features only when using a custom model
self._uses_custom_model = "model" in (clean_learning_kwargs or {})
self.threshold = threshold
def find_issues(
self,
features: Optional[np.ndarray] = None,
predictions: Optional[np.ndarray] = None,
**kwargs,
) -> None:
"""Find label issues in the datalab.
.. admonition:: Priority Order for finding issues:
1. Custom Model: Requires `features` to be passed to this method. Used if a model is set up in the constructor.
2. Predictions: Uses `predictions` if provided and no model is set up in the constructor.
3. Default Model: Defaults to a standard model using `features` if no model or predictions are provided.
"""
if features is None and predictions is None:
raise ValueError(
"Regression requires numerical `features` or `predictions` "
"to be passed in as an argument to `find_issues`."
)
if features is None and self._uses_custom_model:
raise ValueError(
"Regression requires numerical `features` to be passed in as an argument to `find_issues` "
"when using a custom model."
)
# If features are provided and either a custom model is used or no predictions are provided
use_features = features is not None and (self._uses_custom_model or predictions is None)
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"Expected labels to be a numpy array of shape (n_samples,) to use with RegressionLabelIssueManager, "
f"but got {type(labels)} instead."
)
raise TypeError(error_msg)
if use_features:
assert features is not None # mypy won't narrow the type for some reason
self.issues = find_issues_with_features(
features=features,
y=labels,
cl=self.cl,
**kwargs, # function sanitizes kwargs
)
self.issues.rename(columns={"label_quality": self.issue_score_key}, inplace=True)
# Otherwise, if predictions are provided, process them
else:
assert predictions is not None # mypy won't narrow the type for some reason
self.issues = find_issues_with_predictions(
predictions=predictions,
y=labels,
**{**kwargs, **{"threshold": self.threshold}}, # function sanitizes kwargs
)
# Get a summarized dataframe of the label issues
self.summary = self.make_summary(score=self.issues[self.issue_score_key].mean())
# Collect info about the label issues
self.info = self.collect_info(issues=self.issues)
# Drop columns from issues that are in the info
self.issues = self.issues.drop(columns=["given_label", "predicted_label"])
def collect_info(self, issues: pd.DataFrame) -> dict:
issues_info = {
"num_label_issues": sum(issues[f"is_{self.issue_name}_issue"]),
"average_label_quality": issues[self.issue_score_key].mean(),
"given_label": issues["given_label"].tolist(),
"predicted_label": issues["predicted_label"].tolist(),
}
# health_summary_info, cl_info kept just for consistency with classification, but it could be just return issues_info
health_summary_info: dict = {}
cl_info: dict = {}
info_dict = {
**issues_info,
**health_summary_info,
**cl_info,
}
return info_dict
def find_issues_with_predictions(
predictions: np.ndarray,
y: np.ndarray,
threshold: float,
**kwargs,
) -> pd.DataFrame:
"""Find label issues in a regression dataset based on predictions.
This uses a threshold to determine if an example has a label issue
based on the quality score.
Parameters
----------
predictions :
The predictions from a regression model.
y :
The given labels.
threshold :
The threshold to use to determine if an example has a label issue. It is a multiplier
of the median label quality score that sets the absolute threshold.
**kwargs :
Various keyword arguments.
Returns
-------
issues :
A dataframe of the issues. It contains the following columns:
- is_label_issue : bool
True if the example has a label issue.
- label_score : float
The quality score of the label.
- given_label : float
The given label. It is the same as the y parameter.
- predicted_label : float
The predicted label. It is the same as the predictions parameter.
"""
_accepted_kwargs = ["method"]
_kwargs = {k: kwargs.get(k) for k in _accepted_kwargs}
_kwargs = {k: v for k, v in _kwargs.items() if v is not None}
quality_scores = get_label_quality_scores(labels=y, predictions=predictions, **_kwargs)
median_score = np.median(quality_scores)
is_label_issue_mask = quality_scores < median_score * threshold
issues = pd.DataFrame(
{
"is_label_issue": is_label_issue_mask,
"label_score": quality_scores,
"given_label": y,
"predicted_label": predictions,
}
)
return issues
def find_issues_with_features(
features: np.ndarray,
y: np.ndarray,
cl: CleanLearning,
**kwargs,
) -> pd.DataFrame:
"""Find label issues in a regression dataset based on features.
This delegates the work to the CleanLearning.find_label_issues method.
Parameters
----------
features :
The numerical features from a regression dataset.
y :
The given labels.
**kwargs :
Various keyword arguments.
Returns
-------
issues :
A dataframe of the issues. It contains the following columns:
- is_label_issue : bool
True if the example has a label issue.
- label_score : float
The quality score of the label.
- given_label : float
The given label. It is the same as the y parameter.
- predicted_label : float
The predicted label. It is determined by the CleanLearning.find_label_issues method.
"""
_accepted_kwargs = [
"uncertainty",
"coarse_search_range",
"fine_search_size",
"save_space",
"model_kwargs",
]
_kwargs = {k: v for k, v in kwargs.items() if k in _accepted_kwargs and v is not None}
return cl.find_label_issues(X=features, y=y, **_kwargs)
cleanlab/datalab/internal/issue_manager/underperforming_group.py
+345
View File
@@ -0,0 +1,345 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable, ClassVar, Dict, Optional, Union, Tuple
import warnings
import inspect
import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
from sklearn.cluster import DBSCAN
from cleanlab.datalab.internal.issue_manager import IssueManager
from cleanlab.datalab.internal.issue_manager.knn_graph_helpers import set_knn_graph
from cleanlab.rank import get_self_confidence_for_each_label
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
from cleanlab.datalab.datalab import Datalab
CLUSTERING_ALGO = "DBSCAN"
CLUSTERING_PARAMS_DEFAULT = {"metric": "precomputed"}
class UnderperformingGroupIssueManager(IssueManager):
"""
Manages issues related to underperforming group examples.
Note: The `min_cluster_samples` argument should not be confused with the
`min_samples` argument of sklearn.cluster.DBSCAN.
Examples
--------
>>> from cleanlab import Datalab
>>> import numpy as np
>>> X = np.random.normal(size=(50, 2))
>>> y = np.random.randint(2, size=50)
>>> pred_probs = X / X.sum(axis=1, keepdims=True)
>>> data = {"X": X, "y": y}
>>> lab = Datalab(data, label_name="y")
>>> issue_types={"underperforming_group": {"clustering_kwargs": {"eps": 0.5}}}
>>> lab.find_issues(pred_probs=pred_probs, features=X, issue_types=issue_types)
"""
description: ClassVar[
str
] = """An underperforming group refers to a cluster of similar examples
(i.e. a slice) in the dataset for which the ML model predictions
are particularly poor (loss evaluation over this subpopulation is high).
"""
issue_name: ClassVar[str] = "underperforming_group"
verbosity_levels = {
0: [],
1: [],
2: ["threshold"],
}
OUTLIER_CLUSTER_LABELS: ClassVar[Tuple[int]] = (-1,)
"""Specifies labels considered as outliers by the clustering algorithm."""
NO_UNDERPERFORMING_CLUSTER_ID: ClassVar[int] = min(OUTLIER_CLUSTER_LABELS) - 1
"""Constant to signify absence of any underperforming cluster."""
def __init__(
self,
datalab: Datalab,
metric: Optional[Union[str, Callable]] = None,
threshold: float = 0.1,
k: int = 10,
clustering_kwargs: Dict[str, Any] = {},
min_cluster_samples: int = 5,
**_: Any,
):
super().__init__(datalab)
self.metric = metric
self.threshold = self._set_threshold(threshold)
self.k = k
self.clustering_kwargs = clustering_kwargs
self.min_cluster_samples = min_cluster_samples
def find_issues(
self,
pred_probs: npt.NDArray,
features: Optional[npt.NDArray] = None,
cluster_ids: Optional[npt.NDArray[np.int_]] = None,
**kwargs: Any,
) -> None:
labels = self.datalab.labels
if not isinstance(labels, np.ndarray):
error_msg = (
f"Labels must be a numpy array of shape (n_samples,) for UnderperformingGroupIssueManager. "
f"Got {type(labels)} instead."
)
raise TypeError(error_msg)
if cluster_ids is None:
statistics = self.datalab.get_info("statistics")
knn_graph, self.metric, _ = set_knn_graph(
features, kwargs, self.metric, self.k, statistics
)
cluster_ids = self.perform_clustering(knn_graph)
performed_clustering = True
else:
if self.clustering_kwargs:
warnings.warn(
"`clustering_kwargs` will not be used since `cluster_ids` have been passed."
)
performed_clustering = False
knn_graph = None
unique_cluster_ids = self.filter_cluster_ids(cluster_ids)
if not unique_cluster_ids.size:
raise ValueError(
"No meaningful clusters were generated for determining underperforming group."
)
n_clusters = len(unique_cluster_ids)
cluster_id_to_score, worst_cluster_id, worst_cluster_ratio = (
self.get_underperforming_clusters(cluster_ids, unique_cluster_ids, labels, pred_probs)
)
is_issue_column = cluster_ids == worst_cluster_id
scores = np.ones(is_issue_column.shape[0])
for cluster_id, cluster_score in cluster_id_to_score.items():
scores[cluster_ids == cluster_id] = cluster_score
self.issues = pd.DataFrame(
{
f"is_{self.issue_name}_issue": is_issue_column,
self.issue_score_key: scores,
},
)
self.summary = self.make_summary(score=worst_cluster_ratio)
self.info = self.collect_info(
knn_graph=knn_graph,
n_clusters=n_clusters,
cluster_ids=cluster_ids,
performed_clustering=performed_clustering,
worst_cluster_id=worst_cluster_id,
)
def perform_clustering(self, knn_graph: csr_matrix) -> npt.NDArray[np.int_]:
"""Perform clustering of datapoints using a knn graph as distance matrix.
Args:
knn_graph (csr_matrix): Sparse Distance Matrix.
Returns:
cluster_ids (npt.NDArray[np.int_]): Cluster IDs for each datapoint.
"""
DBSCAN_VALID_KEYS = inspect.signature(DBSCAN).parameters.keys()
dbscan_params = {
key: value
for key, value in ((k, self.clustering_kwargs.get(k, None)) for k in DBSCAN_VALID_KEYS)
if value is not None
}
dbscan_params["metric"] = "precomputed"
clusterer = DBSCAN(**dbscan_params)
cluster_ids = clusterer.fit_predict(
knn_graph.copy()
) # Copy to avoid modification by DBSCAN
return cluster_ids
def filter_cluster_ids(self, cluster_ids: npt.NDArray[np.int_]) -> npt.NDArray[np.int_]:
"""Remove outlier clusters and return IDs of clusters with at least `self.min_cluster_samples` number of datapoints.
Args:
cluster_ids (npt.NDArray[np.int_]): Cluster IDs for each datapoint.
Returns:
unique_cluster_ids (npt.NDArray[np.int_]): List of unique cluster IDs after
removing outlier clusters and clusters with less than `self.min_cluster_samples`
number of datapoints.
"""
unique_cluster_ids = np.array(
[label for label in set(cluster_ids) if label not in self.OUTLIER_CLUSTER_LABELS]
)
frequencies = np.bincount(cluster_ids[~np.isin(cluster_ids, self.OUTLIER_CLUSTER_LABELS)])
unique_cluster_ids = np.array(
[
cluster_id
for cluster_id in unique_cluster_ids
if frequencies[cluster_id] >= self.min_cluster_samples
]
)
return unique_cluster_ids
def get_underperforming_clusters(
self,
cluster_ids: npt.NDArray[np.int_],
unique_cluster_ids: npt.NDArray[np.int_],
labels: npt.NDArray,
pred_probs: npt.NDArray,
) -> Tuple[Dict[int, float], int, float]:
"""Get ID and quality score of each underperforming cluster.
Args:
cluster_ids (npt.NDArray[np.int_]): Cluster IDs corresponding to each sample
unique_cluster_ids (npt.NDArray[np.int_]): Unique cluster IDs excluding noisy clusters
labels (npt.NDArray): Label of each sample
pred_probs (npt.NDArray): Prediction probability
Returns:
Tuple[Dict[int, float], int, float]: (Cluster IDs and their scores, Worst Cluster ID, Worst Cluster Quality Score)
"""
worst_cluster_ratio = 1.0 # Largest possible probability value
worst_cluster_id = min(unique_cluster_ids) - 1
# For calculating mean_performance of the dataset, choose labels and pred-probs of samples belonging to non-noisy clusters
filtered_cluster_id_mask = np.isin(cluster_ids, unique_cluster_ids)
filtered_labels = labels[filtered_cluster_id_mask]
filtered_pred_probs = pred_probs[filtered_cluster_id_mask]
mean_performance = get_self_confidence_for_each_label(
filtered_labels, filtered_pred_probs
).mean()
cluster_ids_to_score = {}
for cluster_id in unique_cluster_ids:
cluster_mask = cluster_ids == cluster_id
cur_cluster_ids = labels[cluster_mask]
cur_cluster_pred_probs = pred_probs[cluster_mask]
cluster_performance = get_self_confidence_for_each_label(
cur_cluster_ids, cur_cluster_pred_probs
).mean()
if cluster_performance < mean_performance:
cluster_ids_to_score[cluster_id] = cluster_performance / mean_performance
if cluster_performance < worst_cluster_ratio:
worst_cluster_ratio = cluster_ids_to_score[cluster_id]
worst_cluster_id = cluster_id
worst_cluster_id = (
worst_cluster_id
if worst_cluster_ratio < self.threshold
else self.NO_UNDERPERFORMING_CLUSTER_ID
)
return cluster_ids_to_score, worst_cluster_id, worst_cluster_ratio
def collect_info(
self,
knn_graph: csr_matrix,
n_clusters: int,
cluster_ids: npt.NDArray[np.int_],
performed_clustering: bool,
worst_cluster_id: int,
) -> Dict[str, Any]:
params_dict = {
"k": self.k,
"metric": self.metric,
"threshold": self.threshold,
}
knn_info_dict = {}
if knn_graph is not None:
N = knn_graph.shape[0]
dists = knn_graph.data.reshape(N, -1)[:, 0]
nn_ids = knn_graph.indices.reshape(N, -1)[:, 0]
knn_info_dict = {
"nearest_neighbor": nn_ids.tolist(),
"distance_to_nearest_neighbor": dists.tolist(),
}
statistics_dict = self._build_statistics_dictionary(knn_graph=knn_graph)
cluster_stat_dict = self._get_cluster_statistics(
n_clusters=n_clusters,
cluster_ids=cluster_ids,
performed_clustering=performed_clustering,
worst_cluster_id=worst_cluster_id,
)
info_dict = {
**params_dict,
**knn_info_dict,
**statistics_dict,
**cluster_stat_dict,
}
return info_dict
def _build_statistics_dictionary(self, knn_graph: csr_matrix) -> Dict[str, Dict[str, Any]]:
statistics_dict: Dict[str, Dict[str, Any]] = {"statistics": {}}
# Add the knn graph as a statistic if necessary
graph_key = "weighted_knn_graph"
old_knn_graph = self.datalab.get_info("statistics").get(graph_key, None)
old_graph_exists = old_knn_graph is not None
prefer_new_graph = (
not old_graph_exists
or (
isinstance(knn_graph, csr_matrix)
and old_knn_graph is not None
and knn_graph.nnz > old_knn_graph.nnz
)
or self.metric != self.datalab.get_info("statistics").get("knn_metric", None)
)
if prefer_new_graph:
if knn_graph is not None:
statistics_dict["statistics"][graph_key] = knn_graph
if self.metric is not None:
statistics_dict["statistics"]["knn_metric"] = self.metric
return statistics_dict
def _get_cluster_statistics(
self,
n_clusters: int,
cluster_ids: npt.NDArray[np.int_],
performed_clustering: bool,
worst_cluster_id: int,
) -> Dict[str, Dict[str, Any]]:
"""Get relevant cluster statistics.
Args:
n_clusters (int): Number of clusters
cluster_ids (npt.NDArray[np.int_]): Cluster IDs for each datapoint.
performed_clustering (bool): Set to True to indicate that clustering was performed on
`features` passed to `find_issues`. Set to False to suggest that `cluster_ids` were explicitly
passed to `find_issues`.
worst_cluster_id (int): Uderperforming cluster ID.
Returns:
cluster_stats (Dict[str, Dict[str, Any]]): Cluster Statistics
"""
cluster_stats: Dict[str, Dict[str, Any]] = {
"clustering": {
"algorithm": None,
"params": {},
"stats": {
"n_clusters": n_clusters,
"cluster_ids": cluster_ids,
"underperforming_cluster_id": worst_cluster_id,
},
}
}
if performed_clustering:
cluster_stats["clustering"].update(
{"algorithm": CLUSTERING_ALGO, "params": CLUSTERING_PARAMS_DEFAULT}
)
return cluster_stats
def _set_threshold(
self,
threshold: float,
) -> float:
"""Computes nearest-neighbors thresholding for near-duplicate detection."""
if threshold < 0:
warnings.warn(
f"Computed threshold {threshold} is less than 0. "
"Setting threshold to 0."
"This may indicate that either the only a few examples are in the dataset, "
"or the data is heavily skewed."
)
threshold = 0
return threshold
cleanlab/datalab/internal/issue_manager_factory.py
+267
View File
@@ -0,0 +1,267 @@
"""The factory module provides a factory class for constructing concrete issue managers
and a decorator for registering new issue managers.
This module provides the :py:meth:`register` decorator for users to register new subclasses of
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`
in the registry. Each IssueManager detects some particular type of issue in a dataset.
Note
----
The :class:`REGISTRY` variable is used by the factory class to keep track
of registered issue managers.
The factory class is used as an implementation detail by
:py:class:`Datalab <cleanlab.datalab.datalab.Datalab>`,
which provides a simplified API for constructing concrete issue managers.
:py:class:`Datalab <cleanlab.datalab.datalab.Datalab>` is intended to be used by users
and provides detailed documentation on how to use the API.
Warning
-------
Neither the :class:`REGISTRY` variable nor the factory class should be used directly by users.
"""
from __future__ import annotations
from typing import Dict, List, Type
from cleanlab.datalab.internal.issue_manager import (
ClassImbalanceIssueManager,
DataValuationIssueManager,
IssueManager,
LabelIssueManager,
NearDuplicateIssueManager,
NonIIDIssueManager,
ClassImbalanceIssueManager,
UnderperformingGroupIssueManager,
DataValuationIssueManager,
OutlierIssueManager,
NullIssueManager,
)
from cleanlab.datalab.internal.issue_manager.regression import RegressionLabelIssueManager
from cleanlab.datalab.internal.issue_manager.multilabel.label import MultilabelIssueManager
from cleanlab.datalab.internal.task import Task
REGISTRY: Dict[Task, Dict[str, Type[IssueManager]]] = {
Task.CLASSIFICATION: {
"outlier": OutlierIssueManager,
"label": LabelIssueManager,
"near_duplicate": NearDuplicateIssueManager,
"non_iid": NonIIDIssueManager,
"class_imbalance": ClassImbalanceIssueManager,
"underperforming_group": UnderperformingGroupIssueManager,
"data_valuation": DataValuationIssueManager,
"null": NullIssueManager,
},
Task.REGRESSION: {
"label": RegressionLabelIssueManager,
"outlier": OutlierIssueManager,
"near_duplicate": NearDuplicateIssueManager,
"non_iid": NonIIDIssueManager,
"data_valuation": DataValuationIssueManager,
"null": NullIssueManager,
},
Task.MULTILABEL: {
"label": MultilabelIssueManager,
"outlier": OutlierIssueManager,
"near_duplicate": NearDuplicateIssueManager,
"non_iid": NonIIDIssueManager,
"data_valuation": DataValuationIssueManager,
"null": NullIssueManager,
},
}
"""Registry of issue managers that can be constructed from a task and issue type
and used in the Datalab class.
:meta hide-value:
Currently, the following issue managers are registered by default for a given task:
- Classification:
- ``"outlier"``: :py:class:`OutlierIssueManager <cleanlab.datalab.internal.issue_manager.outlier.OutlierIssueManager>`
- ``"label"``: :py:class:`LabelIssueManager <cleanlab.datalab.internal.issue_manager.label.LabelIssueManager>`
- ``"near_duplicate"``: :py:class:`NearDuplicateIssueManager <cleanlab.datalab.internal.issue_manager.duplicate.NearDuplicateIssueManager>`
- ``"non_iid"``: :py:class:`NonIIDIssueManager <cleanlab.datalab.internal.issue_manager.noniid.NonIIDIssueManager>`
- ``"class_imbalance"``: :py:class:`ClassImbalanceIssueManager <cleanlab.datalab.internal.issue_manager.imbalance.ClassImbalanceIssueManager>`
- ``"underperforming_group"``: :py:class:`UnderperformingGroupIssueManager <cleanlab.datalab.internal.issue_manager.underperforming_group.UnderperformingGroupIssueManager>`
- ``"data_valuation"``: :py:class:`DataValuationIssueManager <cleanlab.datalab.internal.issue_manager.data_valuation.DataValuationIssueManager>`
- ``"null"``: :py:class:`NullIssueManager <cleanlab.datalab.internal.issue_manager.null.NullIssueManager>`
- Regression:
- ``"label"``: :py:class:`RegressionLabelIssueManager <cleanlab.datalab.internal.issue_manager.regression.label.RegressionLabelIssueManager>`
- ``"outlier"``: :py:class:`OutlierIssueManager <cleanlab.datalab.internal.issue_manager.outlier.OutlierIssueManager>`
- ``"near_duplicate"``: :py:class:`NearDuplicateIssueManager <cleanlab.datalab.internal.issue_manager.duplicate.NearDuplicateIssueManager>`
- ``"non_iid"``: :py:class:`NonIIDIssueManager <cleanlab.datalab.internal.issue_manager.noniid.NonIIDIssueManager>`
- ``"null"``: :py:class:`NullIssueManager <cleanlab.datalab.internal.issue_manager.null.NullIssueManager>`
- Multilabel:
- ``"label"``: :py:class:`MultilabelIssueManager <cleanlab.datalab.internal.issue_manager.multilabel.label.MultilabelIssueManager>`
- ``"outlier"``: :py:class:`OutlierIssueManager <cleanlab.datalab.internal.issue_manager.outlier.OutlierIssueManager>`
- ``"near_duplicate"``: :py:class:`NearDuplicateIssueManager <cleanlab.datalab.internal.issue_manager.duplicate.NearDuplicateIssueManager>`
- ``"non_iid"``: :py:class:`NonIIDIssueManager <cleanlab.datalab.internal.issue_manager.noniid.NonIIDIssueManager>`
- ``"null"``: :py:class:`NullIssueManager <cleanlab.datalab.internal.issue_manager.null.NullIssueManager>`
Warning
-------
This variable should not be used directly by users.
"""
# Construct concrete issue manager with a from_str method
class _IssueManagerFactory:
"""Factory class for constructing concrete issue managers."""
@classmethod
def from_str(cls, issue_type: str, task: Task) -> Type[IssueManager]:
"""Constructs a concrete issue manager class from a string."""
if isinstance(issue_type, list):
raise ValueError(
"issue_type must be a string, not a list. Try using from_list instead."
)
if task not in REGISTRY:
raise ValueError(f"Invalid task type: {task}, must be in {list(REGISTRY.keys())}")
if issue_type not in REGISTRY[task]:
raise ValueError(f"Invalid issue type: {issue_type} for task {task}")
return REGISTRY[task][issue_type]
@classmethod
def from_list(cls, issue_types: List[str], task: Task) -> List[Type[IssueManager]]:
"""Constructs a list of concrete issue manager classes from a list of strings."""
return [cls.from_str(issue_type, task) for issue_type in issue_types]
def register(cls: Type[IssueManager], task: str = str(Task.CLASSIFICATION)) -> Type[IssueManager]:
"""Registers the issue manager factory.
Parameters
----------
cls :
A subclass of
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`.
task :
Specific machine learning task like classification or regression.
See :py:meth:`Task.from_str <cleanlab.datalab.internal.task.Task.from_str>`` for more details,
to see which task type corresponds to which string.
Returns
-------
cls :
The same class that was passed in.
Example
-------
When defining a new subclass of
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`,
you can register it like so:
.. code-block:: python
from cleanlab import IssueManager
from cleanlab.datalab.internal.issue_manager_factory import register
@register
class MyIssueManager(IssueManager):
issue_name: str = "my_issue"
def find_issues(self, **kwargs):
# Some logic to find issues
pass
or in a function call:
.. code-block:: python
from cleanlab import IssueManager
from cleanlab.datalab.internal.issue_manager_factory import register
class MyIssueManager(IssueManager):
issue_name: str = "my_issue"
def find_issues(self, **kwargs):
# Some logic to find issues
pass
register(MyIssueManager, task="classification")
"""
if not issubclass(cls, IssueManager):
raise ValueError(f"Class {cls} must be a subclass of IssueManager")
name: str = str(cls.issue_name)
try:
_task = Task.from_str(task)
if _task not in REGISTRY:
raise ValueError(f"Invalid task type: {_task}, must be in {list(REGISTRY.keys())}")
except KeyError:
raise ValueError(f"Invalid task type: {task}, must be in {list(REGISTRY.keys())}")
if name in REGISTRY[_task]:
print(
f"Warning: Overwriting existing issue manager {name} with {cls} for task {_task}."
"This may cause unexpected behavior."
)
REGISTRY[_task][name] = cls
return cls
def list_possible_issue_types(task: Task) -> List[str]:
"""Returns a list of all registered issue types.
Any issue type that is not in this list cannot be used in the :py:meth:`find_issues` method.
See Also
--------
:py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
"""
return list(REGISTRY.get(task, []))
def list_default_issue_types(task: Task) -> List[str]:
"""Returns a list of the issue types that are run by default
when :py:meth:`find_issues` is called without specifying `issue_types`.
task :
Specific machine learning task supported by Datalab.
See Also
--------
:py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
"""
default_issue_types_dict = {
Task.CLASSIFICATION: [
"null",
"label",
"outlier",
"near_duplicate",
"non_iid",
"class_imbalance",
"underperforming_group",
],
Task.REGRESSION: [
"null",
"label",
"outlier",
"near_duplicate",
"non_iid",
],
Task.MULTILABEL: [
"null",
"label",
"outlier",
"near_duplicate",
"non_iid",
],
}
if task not in default_issue_types_dict:
task = Task.CLASSIFICATION
default_issue_types = default_issue_types_dict[task]
return default_issue_types
cleanlab/datalab/internal/model_outputs.py
+116
View File
@@ -0,0 +1,116 @@
"""
This module contains the ModelOutput class, which is used internally within Datalab
to represent model outputs (e.g. predictions, probabilities, etc.) and process them
for issue finding.
This class and associated naming conventions are subject to change and is not meant
to be used by users.
"""
from abc import ABC, abstractmethod
import numpy as np
from dataclasses import dataclass
@dataclass
class ModelOutput(ABC):
"""
An abstract class for representing model outputs (e.g. predictions, probabilities, etc.)
for internal use within Datalab. This class is not meant to be used by users.
It is used internally within the issue-finding process Datalab runs to assign
types to the data and process it accordingly.
Parameters
----------
data : array-like
The model outputs. Not to be confused with the data used to train the model.
This is mainly intended for NumPy arrays.
"""
data: np.ndarray
@abstractmethod
def validate(self):
"""
Validate the data format and content.
E.g. a pred_probs object used for classification
should be a 2D array with values between 0 and 1 and sum to 1 for each row.
"""
pass
@abstractmethod
def collect(self):
"""
Fetch the data for issue finding.
Usually this is just the data itself, but sometimes it may be a transformation
of the data (e.g. a 1D array of predictions from a 2D array of predicted probabilities).
"""
pass
class MultiClassPredProbs(ModelOutput):
"""
A class for representing a model's predicted probabilities for each class
in a multi-class classification problem. This class is not meant to be used by users.
"""
argument = "pred_probs"
def validate(self):
pred_probs = self.data
if pred_probs.ndim != 2:
raise ValueError("pred_probs must be a 2D array for multi-class classification")
if not np.all((pred_probs >= 0) & (pred_probs <= 1)):
incorrect_range = (np.min(pred_probs), np.max(pred_probs))
raise ValueError(
"Expected pred_probs to be between 0 and 1 for multi-label classification,"
f" but got values in range {incorrect_range} instead."
)
if not np.allclose(np.sum(pred_probs, axis=1), 1):
raise ValueError("pred_probs must sum to 1 for each row for multi-class classification")
def collect(self):
return self.data
class RegressionPredictions(ModelOutput):
"""
A class for representing a model's predictions for a regression problem.
This class is not meant to be used by users.
"""
argument = "predictions"
def validate(self):
predictions = self.data
if predictions.ndim != 1:
raise ValueError("pred_probs must be a 1D array for regression")
def collect(self):
return self.data
class MultiLabelPredProbs(ModelOutput):
"""
A class for representing a model's predicted probabilities for each class
in a multilabel classification problem. This class is not meant to be used by users.
"""
argument = "pred_probs"
def validate(self):
pred_probs = self.data
if pred_probs.ndim != 2:
raise ValueError(
f"Expected pred_probs to be a 2D array for multi-label classification,"
" but got {pred_probs.ndim}D array instead."
)
if not np.all((pred_probs >= 0) & (pred_probs <= 1)):
incorrect_range = (np.min(pred_probs), np.max(pred_probs))
raise ValueError(
"Expected pred_probs to be between 0 and 1 for multi-label classification,"
f" but got values in range {incorrect_range} instead."
)
def collect(self):
return self.data
cleanlab/datalab/internal/report.py
+194
View File
@@ -0,0 +1,194 @@
"""
Module that handles reporting of all types of issues identified in the data.
"""
from typing import TYPE_CHECKING, List
import pandas as pd
from cleanlab.datalab.internal.adapter.constants import DEFAULT_CLEANVISION_ISSUES
from cleanlab.datalab.internal.issue_manager_factory import _IssueManagerFactory
from cleanlab.datalab.internal.task import Task
if TYPE_CHECKING: # pragma: no cover
from cleanlab.datalab.internal.data_issues import DataIssues
class Reporter:
"""Class that generates a report about the issues stored in a :py:class:`DataIssues` object.
Parameters
----------
data_issues :
The :py:class:`DataIssues` object containing the issues to report on. This is usually
generated by the :py:class:`Datalab` class, stored in the :py:attr:`data_issues` attribute,
and then passed to the :py:class:`Reporter` class to generate a report.
task :
Specific machine learning task that the datset is intended for.
See details about supported tasks in :py:class:`Task <cleanlab.datalab.internal.task.Task>`.
verbosity :
The default verbosity of the report to generate. Each :py:class`IssueManager`
specifies the available verbosity levels and what additional information
is included at each level.
include_description :
Whether to include the description of each issue type in the report. The description
is included by default, but can be excluded by setting this parameter to ``False``.
Note
----
This class is not intended to be used directly. Instead, use the
`Datalab.find_issues` method which internally utilizes an IssueFinder instance.
"""
def __init__(
self,
data_issues: "DataIssues",
task: Task,
verbosity: int = 1,
include_description: bool = True,
show_summary_score: bool = False,
show_all_issues: bool = False,
**kwargs,
):
self.data_issues = data_issues
self.task = task
self.verbosity = verbosity
self.include_description = include_description
self.show_summary_score = show_summary_score
self.show_all_issues = show_all_issues
def _get_empty_report(self) -> str:
"""This method is used to return a report when there are
no issues found in the data with Datalab.find_issues().
"""
report_str = "No issues found in the data. Good job!"
if not self.show_summary_score:
recommendation_msg = (
"Try re-running Datalab.report() with "
"`show_summary_score = True` and `show_all_issues = True`."
)
report_str += f"\n\n{recommendation_msg}"
return report_str
def report(self, num_examples: int) -> None:
"""Prints a report about identified issues in the data.
Parameters
----------
num_examples :
The number of examples to include in the report for each issue type.
"""
print(self.get_report(num_examples=num_examples))
def get_report(self, num_examples: int) -> str:
"""Constructs a report about identified issues in the data.
Parameters
----------
num_examples :
The number of examples to include in the report for each issue type.
Returns
-------
report_str :
A string containing the report.
Examples
--------
>>> from cleanlab.datalab.internal.report import Reporter
>>> reporter = Reporter(data_issues=data_issues, include_description=False)
>>> report_str = reporter.get_report(num_examples=5)
>>> print(report_str)
"""
report_str = ""
issue_summary = self.data_issues.issue_summary
should_return_empty_report = not (
self.show_all_issues or issue_summary.empty or issue_summary["num_issues"].sum() > 0
)
if should_return_empty_report:
return self._get_empty_report()
issue_summary_sorted = issue_summary.sort_values(by="num_issues", ascending=False)
report_str += self._write_summary(summary=issue_summary_sorted)
issue_types = self._get_issue_types(issue_summary_sorted)
def add_issue_to_report(issue_name: str) -> bool:
"""Returns True if the issue should be added to the report.
It is excluded if show_all_issues is False and there are no issues of that type
found in the data.
"""
if self.show_all_issues:
return True
summary = self.data_issues.get_issue_summary(issue_name=issue_name)
has_issues = summary["num_issues"][0] > 0
return has_issues
issue_reports = [
_IssueManagerFactory.from_str(issue_type=key, task=self.task).report(
issues=self.data_issues.get_issues(issue_name=key),
summary=self.data_issues.get_issue_summary(issue_name=key),
info=self.data_issues.get_info(issue_name=key),
num_examples=num_examples,
verbosity=self.verbosity,
include_description=self.include_description,
)
for key in issue_types
]
report_str += "\n\n\n".join(issue_reports)
return report_str
def _write_summary(self, summary: pd.DataFrame) -> str:
statistics = self.data_issues.get_info("statistics")
num_examples = statistics["num_examples"]
num_classes = statistics.get(
"num_classes"
) # This may not be required for all types of datasets in the future (e.g. unlabeled/regression)
dataset_information = f"Dataset Information: num_examples: {num_examples}"
if num_classes is not None:
dataset_information += f", num_classes: {num_classes}"
if not self.show_all_issues:
# Drop any items in the issue_summary that have no issues (any issue detected in data needs to have num_issues > 0)
summary = summary.query("num_issues > 0")
report_header = (
f"{dataset_information}\n\n"
+ "Here is a summary of various issues found in your data:\n\n"
)
report_footer = (
"\n\n"
+ "Learn about each issue: https://docs.cleanlab.ai/stable/cleanlab/datalab/guide/issue_type_description.html\n"
+ "See which examples in your dataset exhibit each issue via: `datalab.get_issues(<ISSUE_NAME>)`\n\n"
+ "Data indices corresponding to top examples of each issue are shown below.\n\n\n"
)
if self.show_summary_score:
return (
report_header
+ summary.to_string(index=False)
+ "\n\n"
+ "(Note: A lower score indicates a more severe issue across all examples in the dataset.)"
+ report_footer
)
return (
report_header + summary.drop(columns=["score"]).to_string(index=False) + report_footer
)
def _get_issue_types(self, issue_summary: pd.DataFrame) -> List[str]:
issue_types = [
issue_type
for issue_type, num_issues in zip(
issue_summary["issue_type"].tolist(), issue_summary["num_issues"].tolist()
)
if issue_type not in DEFAULT_CLEANVISION_ISSUES
and (self.show_all_issues or num_issues > 0)
]
return issue_types
cleanlab/datalab/internal/serialize.py
+123
View File
@@ -0,0 +1,123 @@
from __future__ import annotations
import os
import pickle
import warnings
from typing import TYPE_CHECKING, Optional
import pandas as pd
import cleanlab
from cleanlab.datalab.internal.data import Data
if TYPE_CHECKING: # pragma: no cover
from datasets.arrow_dataset import Dataset
from cleanlab.datalab.datalab import Datalab
# Constants:
OBJECT_FILENAME = "datalab.pkl"
ISSUES_FILENAME = "issues.csv"
ISSUE_SUMMARY_FILENAME = "summary.csv"
INFO_FILENAME = "info.pkl"
DATA_DIRNAME = "data"
class _Serializer:
@staticmethod
def _save_data_issues(path: str, datalab: Datalab) -> None:
"""Saves the issues to disk."""
issues_path = os.path.join(path, ISSUES_FILENAME)
datalab.data_issues.issues.to_csv(issues_path, index=False)
issue_summary_path = os.path.join(path, ISSUE_SUMMARY_FILENAME)
datalab.data_issues.issue_summary.to_csv(issue_summary_path, index=False)
@staticmethod
def _save_data(path: str, datalab: Datalab) -> None:
"""Saves the dataset to disk."""
data_path = os.path.join(path, DATA_DIRNAME)
datalab.data.save_to_disk(data_path)
@staticmethod
def _validate_version(datalab: Datalab) -> None:
current_version = cleanlab.__version__ # type: ignore[attr-defined]
datalab_version = datalab.cleanlab_version
if current_version != datalab_version:
warnings.warn(
f"Saved Datalab was created using different version of cleanlab "
f"({datalab_version}) than current version ({current_version}). "
f"Things may be broken!"
)
@classmethod
def serialize(cls, path: str, datalab: Datalab, force: bool) -> None:
"""Serializes the datalab object to disk.
Parameters
----------
path : str
Path to save the datalab object to.
datalab : Datalab
The datalab object to save.
force : bool
If True, will overwrite existing files at the specified path.
"""
path_exists = os.path.exists(path)
if not path_exists:
os.mkdir(path)
else:
if not force:
raise FileExistsError("Please specify a new path or set force=True")
print(f"WARNING: Existing files will be overwritten by newly saved files at: {path}")
# Save the datalab object to disk.
with open(os.path.join(path, OBJECT_FILENAME), "wb") as f:
pickle.dump(datalab, f)
# Save the issues to disk. Use placeholder method for now.
cls._save_data_issues(path=path, datalab=datalab)
# Save the dataset to disk
cls._save_data(path=path, datalab=datalab)
@classmethod
def deserialize(cls, path: str, data: Optional[Dataset] = None) -> Datalab:
"""Deserializes the datalab object from disk."""
if not os.path.exists(path):
raise ValueError(f"No folder found at specified path: {path}")
with open(os.path.join(path, OBJECT_FILENAME), "rb") as f:
datalab: Datalab = pickle.load(f)
cls._validate_version(datalab)
# Load the issues from disk.
issues_path = os.path.join(path, ISSUES_FILENAME)
if not hasattr(datalab.data_issues, "issues") and os.path.exists(issues_path):
datalab.data_issues.issues = pd.read_csv(issues_path)
issue_summary_path = os.path.join(path, ISSUE_SUMMARY_FILENAME)
if not hasattr(datalab.data_issues, "issue_summary") and os.path.exists(issue_summary_path):
datalab.data_issues.issue_summary = pd.read_csv(issue_summary_path)
if data is not None:
if hash(data) != hash(datalab._data):
raise ValueError(
"Data has been modified since Lab was saved. "
"Cannot load Lab with modified data."
)
if len(data) != len(datalab.labels):
raise ValueError(
f"Length of data ({len(data)}) does not match length of labels ({len(datalab.labels)})"
)
datalab._data = Data(data, datalab.task, datalab.label_name)
datalab.data = datalab._data._data
return datalab
cleanlab/datalab/internal/spurious_correlation.py
+113
View File
@@ -0,0 +1,113 @@
from dataclasses import dataclass
from typing import List, Optional, Union
import warnings
import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
warnings.filterwarnings("ignore")
@dataclass
class SpuriousCorrelations:
data: pd.DataFrame
labels: Union[np.ndarray, list]
properties_of_interest: Optional[List[str]] = None
def __post_init__(self):
# Must have same number of rows
if not len(self.data) == len(self.labels):
raise ValueError(
"The number of rows in the data dataframe must be the same as the number of labels."
)
# Set default properties_of_interest if not provided
if self.properties_of_interest is None:
self.properties_of_interest = self.data.columns.tolist()
if not all(isinstance(p, str) for p in self.properties_of_interest):
raise TypeError("properties_of_interest must be a list of strings.")
def calculate_correlations(self) -> pd.DataFrame:
"""Calculates the spurious correlation scores for each property of interest found in the dataset."""
baseline_accuracy = self._get_baseline()
assert (
self.properties_of_interest is not None
), "properties_of_interest must be set, but is None."
property_scores = {
str(property_of_interest): self.calculate_spurious_correlation(
property_of_interest, baseline_accuracy
)
for property_of_interest in self.properties_of_interest
}
data_score = pd.DataFrame(list(property_scores.items()), columns=["property", "score"])
return data_score
def _get_baseline(self) -> float:
"""Calculates the baseline accuracy of the dataset. The baseline model is predicting the most common label."""
baseline_accuracy = np.bincount(self.labels).argmax() / len(self.labels)
return float(baseline_accuracy)
def calculate_spurious_correlation(
self, property_of_interest, baseline_accuracy: float
) -> float:
"""Scores the dataset based on a given property of interest.
Parameters
----------
property_of_interest :
The property of interest to score the dataset on.
baseline_accuracy :
The accuracy of the baseline model.
Returns
-------
score :
A correlation score of the dataset's labels to the property of interest.
"""
X = self.data[property_of_interest].values.reshape(-1, 1)
y = self.labels
mean_accuracy = _train_and_eval(X, y)
return relative_room_for_improvement(baseline_accuracy, float(mean_accuracy))
def _train_and_eval(X, y, cv=5) -> float:
classifier = GaussianNB() # TODO: Make this a parameter
cv_accuracies = cross_val_score(classifier, X, y, cv=cv, scoring="accuracy")
mean_accuracy = float(np.mean(cv_accuracies))
return mean_accuracy
def relative_room_for_improvement(
baseline_accuracy: float, mean_accuracy: float, eps: float = 1e-8
) -> float:
"""
Calculate the relative room for improvement given a baseline and trial accuracy.
This function computes the ratio of the difference between perfect accuracy (1.0)
and the trial accuracy to the difference between perfect accuracy and the baseline accuracy.
If the baseline accuracy is perfect (i.e., 1.0), an epsilon value is added to the denominator
to avoid division by zero.
Parameters
----------
baseline_accuracy :
The accuracy of the baseline model. Must be between 0 and 1.
mean_accuracy :
The accuracy of the trial model being compared. Must be between 0 and 1.
eps :
A small constant to avoid division by zero when baseline accuracy is 1. Defaults to 1e-8.
Returns
-------
score :
The relative room for improvement, bounded between 0 and 1.
"""
numerator = 1 - mean_accuracy
denominator = 1 - baseline_accuracy
if baseline_accuracy == 1:
denominator += eps
return min(1, numerator / denominator)
cleanlab/datalab/internal/task.py
+130
View File
@@ -0,0 +1,130 @@
"""
This module contains the Task enum, which internally represents the tasks
supported by Datalab, so that the appropriate task-specific logic can be applied.
This class and associated naming conventions are subject to change and is not meant
to be used by users.
"""
from enum import Enum
class Task(Enum):
"""
Represents a task supported by Datalab.
Datalab supports the following tasks:
* **Classification**: for predicting discrete class labels.
* **Regression**: for predicting continuous numerical values.
* **Multilabel**: for predicting multiple binary labels simultaneously.
Example
-------
>>> task = Task.CLASSIFICATION
>>> task
<Task.CLASSIFICATION: 'classification'>
"""
CLASSIFICATION = "classification"
"""Classification task."""
REGRESSION = "regression"
"""Regression task."""
MULTILABEL = "multilabel"
"""Multilabel task."""
def __str__(self):
"""
Returns the string representation of the task.
Returns:
str: The string representation of the task.
"""
return self.value
@classmethod
def from_str(cls, task_str: str) -> "Task":
"""
Converts a string representation of a task to a Task enum value.
Parameters
----------
task_str :
The string representation of the task.
Returns
-------
Task :
The corresponding Task enum value.
Raises
------
ValueError :
If the provided task_str is not a valid task supported by Datalab.
Examples
--------
>>> Task.from_str("classification")
<Task.CLASSIFICATION: 'classification'>
>>> print(Task.from_str("regression"))
regression
"""
_value_to_enum = {task.value: task for task in Task}
try:
return _value_to_enum[task_str]
except KeyError:
valid_tasks = list(_value_to_enum.keys())
raise ValueError(f"Invalid task: {task_str}. Datalab only supports {valid_tasks}.")
@property
def is_classification(self):
"""
Checks if the task is classification.
Returns
-------
bool :
True if the task is classification, False otherwise.
Examples
--------
>>> task = Task.CLASSIFICATION
>>> print(task.is_classification)
True
"""
return self == Task.CLASSIFICATION
@property
def is_regression(self):
"""
Checks if the task is regression.
Returns
-------
bool :
True if the task is regression, False otherwise.
Examples
--------
>>> task = Task.CLASSIFICATION
>>> print(task.is_regression)
False
"""
return self == Task.REGRESSION
@property
def is_multilabel(self):
"""
Checks if the task is multilabel.
Returns
-------
bool :
True if the task is multilabel, False otherwise.
Examples
--------
>>> task = Task.CLASSIFICATION
>>> print(task.is_multilabel)
False
"""
return self == Task.MULTILABEL
cleanlab/dataset.py
+514
View File
@@ -0,0 +1,514 @@
"""
Provides dataset-level and class-level overviews of issues in your classification dataset.
If your task allows you to modify the classes in your dataset, this module can help you determine
which classes to remove (see `~cleanlab.dataset.rank_classes_by_label_quality`)
and which classes to merge (see `~cleanlab.dataset.find_overlapping_classes`).
"""
from typing import Optional, cast
import numpy as np
import pandas as pd
from cleanlab.count import estimate_joint, num_label_issues
from cleanlab.internal.constants import EPSILON
def rank_classes_by_label_quality(
labels=None,
pred_probs=None,
*,
class_names=None,
num_examples=None,
joint=None,
confident_joint=None,
multi_label=False,
) -> pd.DataFrame:
"""
Returns a Pandas DataFrame with all classes and three overall class label quality scores
(details about each score are listed in the Returns parameter). By default, classes are ordered
by "Label Quality Score", ascending, so the most problematic classes are reported first.
Score values are unnormalized and may tend to be very small. What matters is their relative
ranking across the classes.
This method works by providing any one (and only one) of the following inputs:
1. ``labels`` and ``pred_probs``, or
2. ``joint`` and ``num_examples``, or
3. ``confident_joint``
Only provide **exactly one of the above input options**, do not provide a combination.
Examples
--------
>>> from cleanlab.dataset import rank_classes_by_label_quality
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.model_selection import cross_val_predict
>>> data, labels = get_data_labels_from_dataset()
>>> yourFavoriteModel = LogisticRegression()
>>> pred_probs = cross_val_predict(yourFavoriteModel, data, labels, cv=3, method="predict_proba")
>>> df = rank_classes_by_label_quality(labels=labels, pred_probs=pred_probs)
**Parameters**: For parameter info, see the docstring of `~cleanlab.dataset.find_overlapping_classes`.
Returns
-------
overall_label_quality : pd.DataFrame
Pandas DataFrame with cols "Class Index", "Label Issues", "Inverse Label Issues",
"Label Issues", "Inverse Label Noise", "Label Quality Score",
with a description of each of these columns below.
The length of the DataFrame is ``num_classes`` (one row per class).
Noise scores are between 0 and 1, where 0 implies no label issues
in the class. The "Label Quality Score" is also between 0 and 1 where 1 implies
perfect quality. Columns:
* *Class Index*: The index of the class in 0, 1, ..., K-1.
* *Label Issues*: ``count(given_label = k, true_label != k)``, estimated number of examples in the dataset that are labeled as class k but should have a different label.
* *Inverse Label Issues*: ``count(given_label != k, true_label = k)``, estimated number of examples in the dataset that should actually be labeled as class k but have been given another label.
* *Label Noise*: ``prob(true_label != k | given_label = k)``, estimated proportion of examples in the dataset that are labeled as class k but should have a different label. For each class k: this is computed by dividing the number of examples with "Label Issues" that were labeled as class k by the total number of examples labeled as class k.
* *Inverse Label Noise*: ``prob(given_label != k | true_label = k)``, estimated proportion of examples in the dataset that should actually be labeled as class k but have been given another label.
* *Label Quality Score*: ``p(true_label = k | given_label = k)``. This is the proportion of examples with given label k that have been labeled correctly, i.e. ``1 - label_noise``.
By default, the DataFrame is ordered by "Label Quality Score", ascending.
"""
if multi_label:
raise ValueError(
"For multilabel data, please instead call: multilabel_classification.dataset.overall_multilabel_health_score()"
)
if joint is None:
joint = estimate_joint(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
)
if num_examples is None:
num_examples = _get_num_examples(labels=labels)
given_label_noise = joint.sum(axis=1) - joint.diagonal() # p(s=k) - p(s=k,y=k) = p(y!=k, s=k)
true_label_noise = joint.sum(axis=0) - joint.diagonal() # p(y=k) - p(s=k,y=k) = p(s!=k,y=k)
given_conditional_noise = given_label_noise / np.clip(
joint.sum(axis=1), a_min=EPSILON, a_max=None
) # p(y!=k, s=k) / p(s=k) , avoiding division by 0
true_conditional_noise = true_label_noise / np.clip(
joint.sum(axis=0), a_min=EPSILON, a_max=None
) # p(s!=k, y=k) / p(y=k) , avoiding division by 0
df = pd.DataFrame(
{
"Class Index": np.arange(len(joint)),
"Label Issues": (given_label_noise * num_examples).round().astype(int),
"Inverse Label Issues": (true_label_noise * num_examples).round().astype(int),
"Label Noise": given_conditional_noise, # p(y!=k | s=k)
"Inverse Label Noise": true_conditional_noise, # p(s!=k | y=k)
# Below could equivalently be computed as: joint.diagonal() / joint.sum(axis=1)
"Label Quality Score": 1 - given_conditional_noise, # p(y=k | s=k)
}
)
if class_names is not None:
df.insert(loc=0, column="Class Name", value=class_names)
return df.sort_values(by="Label Quality Score", ascending=True).reset_index(drop=True)
def find_overlapping_classes(
labels=None,
pred_probs=None,
*,
asymmetric=False,
class_names=None,
num_examples=None,
joint=None,
confident_joint=None,
multi_label=False,
) -> pd.DataFrame:
"""Returns the pairs of classes that are often mislabeled as one another.
Consider merging the top pairs of classes returned by this method each into a single class.
If the dataset is labeled by human annotators, consider clearly defining the
difference between the classes prior to having annotators label the data.
This method provides two scores in the Pandas DataFrame that is returned:
* **Num Overlapping Examples**: The number of examples where the two classes overlap
* **Joint Probability**: `(num overlapping examples / total number of examples in the dataset`).
This method works by providing any one (and only one) of the following inputs:
1. ``labels`` and ``pred_probs``, or
2. ``joint`` and ``num_examples``, or
3. ``confident_joint``
Only provide **exactly one of the above input options**, do not provide a combination.
This method uses the joint distribution of noisy and true labels to compute ontological
issues via the approach published in `Northcutt et al.,
2021 <https://jair.org/index.php/jair/article/view/12125>`_.
Examples
--------
>>> from cleanlab.dataset import find_overlapping_classes
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.model_selection import cross_val_predict
>>> data, labels = get_data_labels_from_dataset()
>>> yourFavoriteModel = LogisticRegression()
>>> pred_probs = cross_val_predict(yourFavoriteModel, data, labels, cv=3, method="predict_proba")
>>> df = find_overlapping_classes(labels=labels, pred_probs=pred_probs)
Note
----
The joint distribution of noisy and true labels is asymmetric, and therefore the joint
probability ``p(given="vehicle", true="truck") != p(true="truck", given="vehicle")``.
This is intuitive. Images of trucks (true label) are much more likely to be labeled as a car
(given label) than images of cars (true label) being frequently mislabeled as truck (given
label). cleanlab takes these differences into account for you automatically via the joint
distribution. If you do not want this behavior, simply set ``asymmetric=False``.
This method estimates how often the annotators confuse two classes.
This differs from just using a similarity matrix or confusion matrix,
as these summarize characteristics of the predictive model rather than the data labelers (i.e. annotators).
Instead, this method works even if the model that generated `pred_probs` tends to be more confident in some classes than others.
Parameters
----------
labels : np.ndarray or list, optional
An array_like (of length N) of noisy labels for the classification dataset, i.e. some labels may be erroneous.
Elements must be integers in the set 0, 1, ..., K-1, where K is the number of classes.
All the classes (0, 1, ..., and K-1) should be present in ``labels``, such that
``len(set(labels)) == pred_probs.shape[1]`` for standard multi-class classification with single-labeled data (e.g. ``labels = [1,0,2,1,1,0...]``).
For multi-label classification where each example can belong to multiple classes (e.g. ``labels = [[1,2],[1],[0],[],...]``),
your labels should instead satisfy: ``len(set(k for l in labels for k in l)) == pred_probs.shape[1])``.
pred_probs : np.ndarray, optional
An array of shape ``(N, K)`` of model-predicted probabilities,
``P(label=k|x)``. Each row of this matrix corresponds
to an example `x` and contains the model-predicted probabilities that
`x` belongs to each possible class, for each of the K classes. The
columns must be ordered such that these probabilities correspond to
class 0, 1, ..., K-1. `pred_probs` should have been computed using 3 (or
higher) fold cross-validation.
asymmetric : bool, optional
If ``asymmetric=True``, returns separate estimates for both pairs (class1, class2) and (class2, class1). Use this
for finding "is a" relationships where for example "class1 is a class2".
In this case, num overlapping examples counts the number of examples that have been labeled as class1 which should actually have been labeled as class2.
If ``asymmetric=False``, the pair (class1, class2) will only be returned once with an arbitrary order.
In this case, their estimated score is the sum: ``score(class1, class2) + score(class2, class1))``.
class_names : Iterable[str]
A list or other iterable of the string class names. The list should be in the order that
matches the class indices. So if class 0 is 'dog' and class 1 is 'cat', then
``class_names = ['dog', 'cat']``.
num_examples : int or None, optional
The number of examples in the dataset, i.e. ``len(labels)``. You only need to provide this if
you use this function with the joint, e.g. ``find_overlapping_classes(joint=joint)``, otherwise
this is automatically computed via ``sum(confident_joint)`` or ``len(labels)``.
joint : np.ndarray, optional
An array of shape ``(K, K)``, where K is the number of classes,
representing the estimated joint distribution of the noisy labels and
true labels. The sum of all entries in this matrix must be 1 (valid
probability distribution). Each entry in the matrix captures the co-occurence joint
probability of a true label and a noisy label, i.e. ``p(noisy_label=i, true_label=j)``.
**Important**. If you input the joint, you must also input `num_examples`.
confident_joint : np.ndarray, optional
An array of shape ``(K, K)`` representing the confident joint, the matrix used for identifying label issues, which
estimates a confident subset of the joint distribution of the noisy and true labels, ``P_{noisy label, true label}``.
Entry ``(j, k)`` in the matrix is the number of examples confidently counted into the pair of ``(noisy label=j, true label=k)`` classes.
The `confident_joint` can be computed using :py:func:`count.compute_confident_joint <cleanlab.count.compute_confident_joint>`.
If not provided, it is computed from the given (noisy) `labels` and `pred_probs`.
Returns
-------
overlapping_classes : pd.DataFrame
Pandas DataFrame with columns "Class Index A", "Class Index B",
"Num Overlapping Examples", "Joint Probability" and a description of each below.
Each row corresponds to a pair of classes.
* *Class Index A*: the index of a class in 0, 1, ..., K-1.
* *Class Index B*: the index of a different class (from Class A) in 0, 1, ..., K-1.
* *Num Overlapping Examples*: estimated number of labels overlapping between the two classes.
* *Joint Probability*: the *Num Overlapping Examples* divided by the number of examples in the dataset.
By default, the DataFrame is ordered by "Joint Probability" descending.
"""
def _2d_matrix_to_row_column_value_list(matrix):
"""Create a list<tuple> [(row_index, col_index, value)] representation of matrix.
Parameters
----------
matrix : np.ndarray<float>
Any valid np.ndarray 2-d dimensional matrix.
Returns
-------
list<tuple>
A [(row_index, col_index, value)] representation of matrix.
"""
return [(*i, v) for i, v in np.ndenumerate(matrix)]
if multi_label:
raise ValueError(
"For multilabel data, please instead call: multilabel_classification.dataset.common_multilabel_issues()"
)
if joint is None:
joint = estimate_joint(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
)
if num_examples is None:
num_examples = _get_num_examples(labels=labels, confident_joint=confident_joint)
if asymmetric:
rcv_list = _2d_matrix_to_row_column_value_list(joint)
# Remove diagonal elements
rcv_list = [tup for tup in rcv_list if tup[0] != tup[1]]
else: # symmetric
# Sum the upper and lower triangles and remove the lower triangle and the diagonal
sym_joint = np.triu(joint) + np.tril(joint).T
rcv_list = _2d_matrix_to_row_column_value_list(sym_joint)
# Provide values only in (the upper triangle) of the matrix.
rcv_list = [tup for tup in rcv_list if tup[0] < tup[1]]
df = pd.DataFrame(rcv_list, columns=["Class Index A", "Class Index B", "Joint Probability"])
num_overlapping = (df["Joint Probability"] * num_examples).round().astype(int)
df.insert(loc=2, column="Num Overlapping Examples", value=num_overlapping)
if class_names is not None:
df.insert(
loc=0, column="Class Name A", value=df["Class Index A"].apply(lambda x: class_names[x])
)
df.insert(
loc=1, column="Class Name B", value=df["Class Index B"].apply(lambda x: class_names[x])
)
return df.sort_values(by="Joint Probability", ascending=False).reset_index(drop=True)
def overall_label_health_score(
labels=None,
pred_probs=None,
*,
num_examples=None,
confident_joint=None,
joint=None,
multi_label=False,
verbose=True,
) -> float:
"""Returns a single score between 0 and 1 measuring the overall quality of all labels in a dataset.
Intuitively, the score is the average correctness of the given labels across all examples in the
dataset. So a score of 1 suggests your data is perfectly labeled and a score of 0.5 suggests
half of the examples in the dataset may be incorrectly labeled. Thus, a higher
score implies a higher quality dataset.
This method works by providing any one (and only one) of the following inputs:
1. ``labels`` and ``pred_probs``, or
2. ``joint`` and ``num_examples``, or
3. ``confident_joint``
Only provide **exactly one of the above input options**, do not provide a combination.
Examples
--------
>>> from cleanlab.dataset import overall_label_health_score
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.model_selection import cross_val_predict
>>> data, labels = get_data_labels_from_dataset()
>>> yourFavoriteModel = LogisticRegression()
>>> pred_probs = cross_val_predict(yourFavoriteModel, data, labels, cv=3, method="predict_proba")
>>> score = overall_label_health_score(labels=labels, pred_probs=pred_probs) # doctest: +SKIP
**Parameters**: For parameter info, see the docstring of `~cleanlab.dataset.find_overlapping_classes`.
Returns
-------
health_score : float
A score between 0 and 1, where 1 implies all labels in the dataset are estimated to be correct.
A score of 0.5 implies that half of the dataset's labels are estimated to have issues.
"""
if multi_label:
raise ValueError(
"For multilabel data, please instead call: multilabel_classification.dataset.overall_multilabel_health_score()"
)
if num_examples is None:
num_examples = _get_num_examples(labels=labels, confident_joint=confident_joint)
if pred_probs is None or labels is None:
if joint is None:
joint = estimate_joint(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
)
joint_trace = joint.trace()
num_issues = (num_examples * (1 - joint_trace)).round().astype(int)
health_score = joint_trace
else:
num_issues = num_label_issues(
labels=labels, pred_probs=pred_probs, confident_joint=confident_joint
)
health_score = 1 - num_issues / num_examples
if verbose:
print(
f" * Overall, about {(1 - health_score):.0%} ({num_issues:,} of the {num_examples:,}) "
f"labels in your dataset have potential issues.\n"
f" ** The overall label health score for this dataset is: {health_score:.2f}."
)
return health_score
def health_summary(
labels=None,
pred_probs=None,
*,
asymmetric=False,
class_names=None,
num_examples=None,
joint=None,
confident_joint=None,
multi_label=False,
verbose=True,
) -> dict:
"""Prints a health summary of your dataset.
This summary includes useful statistics like:
* The classes with the most and least label issues.
* Classes that overlap and could potentially be merged.
* Overall label quality scores, summarizing how accurate the labels appear overall.
This method works by providing any one (and only one) of the following inputs:
1. ``labels`` and ``pred_probs``, or
2. ``joint`` and ``num_examples``, or
3. ``confident_joint``
Only provide **exactly one of the above input options**, do not provide a combination.
Examples
--------
>>> from cleanlab.dataset import health_summary
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.model_selection import cross_val_predict
>>> data, labels = get_data_labels_from_dataset()
>>> yourFavoriteModel = LogisticRegression()
>>> pred_probs = cross_val_predict(yourFavoriteModel, data, labels, cv=3, method="predict_proba")
>>> summary = health_summary(labels=labels, pred_probs=pred_probs) # doctest: +SKIP
**Parameters**: For parameter info, see the docstring of `~cleanlab.dataset.find_overlapping_classes`.
Returns
-------
summary : dict
A dictionary containing keys (see the corresponding functions' documentation to understand the values):
- ``"overall_label_health_score"``, corresponding to `~cleanlab.dataset.overall_label_health_score`
- ``"joint"``, corresponding to :py:func:`count.estimate_joint <cleanlab.count.estimate_joint>`
- ``"classes_by_label_quality"``, corresponding to `~cleanlab.dataset.rank_classes_by_label_quality`
- ``"overlapping_classes"``, corresponding to `~cleanlab.dataset.find_overlapping_classes`
"""
from cleanlab.internal.util import smart_display_dataframe
if multi_label:
raise ValueError(
"For multilabel data, please call multilabel_classification.dataset.health_summary"
)
if joint is None:
joint = estimate_joint(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
)
if num_examples is None:
num_examples = _get_num_examples(labels=labels)
if verbose:
longest_line = (
f"| for your dataset with {num_examples:,} examples "
f"and {len(joint):,} classes. |\n"
)
print(
"-" * (len(longest_line) - 1)
+ "\n"
+ f"| Generating a Cleanlab Dataset Health Summary{' ' * (len(longest_line) - 49)}|\n"
+ longest_line
+ f"| Note, Cleanlab is not a medical doctor... yet.{' ' * (len(longest_line) - 51)}|\n"
+ "-" * (len(longest_line) - 1)
+ "\n",
)
df_class_label_quality = rank_classes_by_label_quality(
labels=labels,
pred_probs=pred_probs,
class_names=class_names,
num_examples=num_examples,
joint=joint,
confident_joint=confident_joint,
)
if verbose:
print("Overall Class Quality and Noise across your dataset (below)")
print("-" * 60, "\n", flush=True)
smart_display_dataframe(df_class_label_quality)
df_overlapping_classes = find_overlapping_classes(
labels=labels,
pred_probs=pred_probs,
asymmetric=asymmetric,
class_names=class_names,
num_examples=num_examples,
joint=joint,
confident_joint=confident_joint,
)
if verbose:
print(
"\nClass Overlap. In some cases, you may want to merge classes in the top rows (below)"
+ "\n"
+ "-" * 83
+ "\n",
flush=True,
)
smart_display_dataframe(df_overlapping_classes)
print()
health_score = overall_label_health_score(
labels=labels,
pred_probs=pred_probs,
num_examples=num_examples,
confident_joint=confident_joint,
verbose=verbose,
)
if verbose:
print("\nGenerated with <3 from Cleanlab.\n")
return {
"overall_label_health_score": health_score,
"joint": joint,
"classes_by_label_quality": df_class_label_quality,
"overlapping_classes": df_overlapping_classes,
}
def _get_num_examples(labels=None, confident_joint: Optional[np.ndarray] = None) -> int:
"""Helper method that finds the number of examples from the parameters or throws an error
if neither parameter is provided.
**Parameters:** For information about the arguments to this method, see the documentation of `dataset.find_overlapping_classes`
Returns
-------
num_examples : int
The number of examples in the dataset.
Raises
------
ValueError
If `labels` is None."""
if labels is None and confident_joint is None:
raise ValueError(
"Error: num_examples is None. You must either provide confident_joint, "
"or provide both num_example and joint as input parameters."
)
_confident_joint = cast(np.ndarray, confident_joint)
num_examples = len(labels) if labels is not None else cast(int, np.sum(_confident_joint))
return num_examples
cleanlab/experimental/README.md
+16
View File
@@ -0,0 +1,16 @@
# Useful methods/models adapted for use with cleanlab
Methods in this `experimental` module are bleeding edge and may have sharp edges. They are not guaranteed to be stable between different cleanlab versions.
Some of these files include various models that can be used with cleanlab to find issues in specific types of data. These require dependencies on deep learning and other machine learning packages that are not official cleanlab dependencies. You must install these dependencies on your own if you wish to use them.
The modules and required dependencies are as follows:
* mnist_pytorch.py - a cleanlab-compatible simplified AlexNet for MNIST using PyTorch
- torch
- torchvision
* cifar_cnn.py - a cleanlab-compatible Convolutional Neural Network for CIFAR using PyTorch, trainable via CoTeaching
- torch
- torchvision
* coteaching.py - an algorithm to train neural networks with noisy labels
- torch
cleanlab/experimental/__init__.py
View File
cleanlab/experimental/coteaching.py
+229
View File
@@ -0,0 +1,229 @@
"""
Implements the co-teaching algorithm for training neural networks on noisily-labeled data (Han et al., 2018).
This module requires PyTorch (https://pytorch.org/get-started/locally/).
Example using this algorithm with cleanlab to achieve state of the art on CIFAR-10
for learning with noisy labels is provided within: https://github.com/cleanlab/examples/
``cifar_cnn.py`` provides an example model that can be trained via this algorithm.
"""
# Significant code was adapted from the following GitHub:
# https://github.com/bhanML/Co-teaching/blob/master/loss.py
# See (Han et al., 2018).
import torch
import torch.nn.functional as F
from torch.autograd import Variable
import numpy as np
MINIMUM_BATCH_SIZE = 16
# Loss function for Co-Teaching
def loss_coteaching(
y_1,
y_2,
t,
forget_rate,
class_weights=None,
):
"""Co-Teaching Loss function.
Parameters
----------
y_1 : Tensor array
Output logits from model 1
y_2 : Tensor array
Output logits from model 2
t : np.ndarray
List of Noisy Labels (t means targets)
forget_rate : float
Decimal between 0 and 1 for how quickly the models forget what they learn.
Just use rate_schedule[epoch] for this value
class_weights : Tensor array, shape (Number of classes x 1), Default: None
A np.torch.tensor list of length number of classes with weights
"""
loss_1 = F.cross_entropy(y_1, t, reduce=False, weight=class_weights)
ind_1_sorted = np.argsort(loss_1.data.cpu())
loss_1_sorted = loss_1[ind_1_sorted]
loss_2 = F.cross_entropy(y_2, t, reduce=False, weight=class_weights)
ind_2_sorted = np.argsort(loss_2.data.cpu())
remember_rate = 1 - forget_rate
num_remember = int(remember_rate * len(loss_1_sorted))
ind_1_update = ind_1_sorted[:num_remember]
ind_2_update = ind_2_sorted[:num_remember]
# Share updates between the two models.
# TODO: these class weights should take into account the ind_mask filters.
loss_1_update = F.cross_entropy(y_1[ind_2_update], t[ind_2_update], weight=class_weights)
loss_2_update = F.cross_entropy(y_2[ind_1_update], t[ind_1_update], weight=class_weights)
return (
torch.sum(loss_1_update) / num_remember,
torch.sum(loss_2_update) / num_remember,
)
def initialize_lr_scheduler(lr=0.001, epochs=250, epoch_decay_start=80):
"""Scheduler to adjust learning rate and betas for Adam Optimizer"""
mom1 = 0.9
mom2 = 0.9 # Original author had this set to 0.1
alpha_plan = [lr] * epochs
beta1_plan = [mom1] * epochs
for i in range(epoch_decay_start, epochs):
alpha_plan[i] = float(epochs - i) / (epochs - epoch_decay_start) * lr
beta1_plan[i] = mom2
return alpha_plan, beta1_plan
def adjust_learning_rate(optimizer, epoch, alpha_plan, beta1_plan):
"""Scheduler to adjust learning rate and betas for Adam Optimizer"""
for param_group in optimizer.param_groups:
param_group["lr"] = alpha_plan[epoch]
param_group["betas"] = (beta1_plan[epoch], 0.999) # Only change beta1
def forget_rate_scheduler(epochs, forget_rate, num_gradual, exponent):
"""Tells Co-Teaching what fraction of examples to forget at each epoch."""
# define how many things to forget at each rate schedule
forget_rate_schedule = np.ones(epochs) * forget_rate
forget_rate_schedule[:num_gradual] = np.linspace(0, forget_rate**exponent, num_gradual)
return forget_rate_schedule
# Train the Model
def train(
train_loader,
epoch,
model1,
optimizer1,
model2,
optimizer2,
args,
forget_rate_schedule,
class_weights,
accuracy,
):
"""PyTorch training function.
Parameters
----------
train_loader : torch.utils.data.DataLoader
epoch : int
model1 : PyTorch class inheriting nn.Module
Must define __init__ and forward(self, x,)
optimizer1 : PyTorch torch.optim.Adam
model2 : PyTorch class inheriting nn.Module
Must define __init__ and forward(self, x,)
optimizer2 : PyTorch torch.optim.Adam
args : parser.parse_args() object
Must contain num_iter_per_epoch, print_freq, and epochs
forget_rate_schedule : np.ndarray of length number of epochs
Tells Co-Teaching loss what fraction of examples to forget about.
class_weights : Tensor array, shape (Number of classes x 1), Default: None
A np.torch.tensor list of length number of classes with weights
accuracy : function
A function of the form accuracy(output, target, topk=(1,)) for
computing top1 and top5 accuracy given output and true targets."""
train_total = 0
train_correct = 0
train_total2 = 0
train_correct2 = 0
# Prepare models for training
model1.train()
model2.train()
for i, (images, labels) in enumerate(train_loader):
if i == len(train_loader) - 1 and len(labels) < MINIMUM_BATCH_SIZE:
# Edge case -- the last leftover batch is small (potentially size 1)
# This will happen if, for example, you train on 35101 examples with
# batch size of 450. The last batch will be size 1.
# If you update the weights based on the gradient from one example
# if that example is noisy, you will add tons of noise to your net
# and accuracy will actually go down with each epoch.
# To avoid this, do not train on the last batch if it's small.
continue
images = Variable(images).cuda()
labels = Variable(labels).cuda()
# Forward + Backward + Optimize
logits1 = model1(images)
prec1, _ = accuracy(logits1, labels, topk=(1, 5))
train_total += 1
train_correct += prec1
logits2 = model2(images)
prec2, _ = accuracy(logits2, labels, topk=(1, 5))
train_total2 += 1
train_correct2 += prec2
loss_1, loss_2 = loss_coteaching(
logits1,
logits2,
labels,
forget_rate=forget_rate_schedule[epoch],
class_weights=class_weights,
)
optimizer1.zero_grad()
loss_1.backward()
optimizer1.step()
optimizer2.zero_grad()
loss_2.backward()
optimizer2.step()
if (i + 1) % args.print_freq == 0:
print(
"Epoch [%d/%d], Iter [%d/%d] Training Accuracy1: %.4F, "
"Training Accuracy2: %.4f, Loss1: %.4f, Loss2: %.4f "
% (
epoch + 1,
args.epochs,
i + 1,
len(train_loader.dataset) // args.batch_size,
prec1,
prec2,
loss_1.data.item(),
loss_2.data.item(),
)
)
train_acc1 = float(train_correct) / float(train_total)
train_acc2 = float(train_correct2) / float(train_total2)
return train_acc1, train_acc2
# Evaluate the Model
def evaluate(test_loader, model1, model2):
print("Evaluating Co-Teaching Model")
model1.eval() # Change model to 'eval' mode.
correct1 = 0
total1 = 0
for images, labels in test_loader:
images = Variable(images).cuda()
logits1 = model1(images)
outputs1 = F.softmax(logits1, dim=1)
_, pred1 = torch.max(outputs1.data, 1)
total1 += labels.size(0)
correct1 += (pred1.cpu() == labels).sum()
model2.eval() # Change model to 'eval' mode
correct2 = 0
total2 = 0
for images, labels in test_loader:
images = Variable(images).cuda()
logits2 = model2(images)
outputs2 = F.softmax(logits2, dim=1)
_, pred2 = torch.max(outputs2.data, 1)
total2 += labels.size(0)
correct2 += (pred2.cpu() == labels).sum()
acc1 = 100 * float(correct1) / float(total1)
acc2 = 100 * float(correct2) / float(total2)
return acc1, acc2
cleanlab/experimental/label_issues_batched.py
+760
View File
@@ -0,0 +1,760 @@
"""
Implementation of :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>`
that does not need much memory by operating in mini-batches.
You can also use this approach to estimate label quality scores or the number of label issues
for big datasets with limited memory.
With default settings, the results returned from this approach closely approximate those returned from:
``cleanlab.filter.find_label_issues(..., filter_by="low_self_confidence", return_indices_ranked_by="self_confidence")``
To run this approach, either use the ``find_label_issues_batched()`` convenience function defined in this module,
or follow the examples script for the ``LabelInspector`` class if you require greater customization.
"""
import numpy as np
from typing import Optional, List, Tuple, Any
from cleanlab.count import get_confident_thresholds, _reduce_issues
from cleanlab.rank import find_top_issues, _compute_label_quality_scores
from cleanlab.typing import LabelLike
from cleanlab.internal.util import value_counts_fill_missing_classes
from cleanlab.internal.constants import (
CONFIDENT_THRESHOLDS_LOWER_BOUND,
FLOATING_POINT_COMPARISON,
CLIPPING_LOWER_BOUND,
)
import platform
import multiprocessing as mp
try:
import psutil
PSUTIL_EXISTS = True
except ImportError: # pragma: no cover
PSUTIL_EXISTS = False
# global variable for multiproc on linux
adj_confident_thresholds_shared: np.ndarray
labels_shared: LabelLike
pred_probs_shared: np.ndarray
def find_label_issues_batched(
labels: Optional[LabelLike] = None,
pred_probs: Optional[np.ndarray] = None,
*,
labels_file: Optional[str] = None,
pred_probs_file: Optional[str] = None,
batch_size: int = 10000,
n_jobs: Optional[int] = 1,
verbose: bool = True,
quality_score_kwargs: Optional[dict] = None,
num_issue_kwargs: Optional[dict] = None,
return_mask: bool = False,
) -> np.ndarray:
"""
Variant of :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>`
that requires less memory by reading from `pred_probs`, `labels` in mini-batches.
To avoid loading big `pred_probs`, `labels` arrays into memory,
provide these as memory-mapped objects like Zarr arrays or memmap arrays instead of regular numpy arrays.
See: https://pythonspeed.com/articles/mmap-vs-zarr-hdf5/
With default settings, the results returned from this method closely approximate those returned from:
``cleanlab.filter.find_label_issues(..., filter_by="low_self_confidence", return_indices_ranked_by="self_confidence")``
This function internally implements the example usage script of the ``LabelInspector`` class,
but you can further customize that script by running it yourself instead of this function.
See the documentation of ``LabelInspector`` to learn more about how this method works internally.
Parameters
----------
labels: np.ndarray-like object, optional
1D array of given class labels for each example in the dataset, (int) values in ``0,1,2,...,K-1``.
To avoid loading big objects into memory, you should pass this as a memory-mapped object like:
Zarr array loaded with ``zarr.convenience.open(YOURFILE.zarr, mode="r")``,
or memmap array loaded with ``np.load(YOURFILE.npy, mmap_mode="r")``.
Tip: You can save an existing numpy array to Zarr via: ``zarr.convenience.save_array(YOURFILE.zarr, your_array)``,
or to .npy file that can be loaded with mmap via: ``np.save(YOURFILE.npy, your_array)``.
pred_probs: np.ndarray-like object, optional
2D array of model-predicted class probabilities (floats) for each example in the dataset.
To avoid loading big objects into memory, you should pass this as a memory-mapped object like:
Zarr array loaded with ``zarr.convenience.open(YOURFILE.zarr, mode="r")``
or memmap array loaded with ``np.load(YOURFILE.npy, mmap_mode="r")``.
labels_file: str, optional
Specify this instead of `labels` if you want this method to load from file for you into a memmap array.
Path to .npy file where the entire 1D `labels` numpy array is stored on disk (list format is not supported).
This is loaded using: ``np.load(labels_file, mmap_mode="r")``
so make sure this file was created via: ``np.save()`` or other compatible methods (.npz not supported).
pred_probs_file: str, optional
Specify this instead of `pred_probs` if you want this method to load from file for you into a memmap array.
Path to .npy file where the entire `pred_probs` numpy array is stored on disk.
This is loaded using: ``np.load(pred_probs_file, mmap_mode="r")``
so make sure this file was created via: ``np.save()`` or other compatible methods (.npz not supported).
batch_size : int, optional
Size of mini-batches to use for estimating the label issues.
To maximize efficiency, try to use the largest `batch_size` your memory allows.
n_jobs: int, optional
Number of processes for multiprocessing (default value = 1). Only used on Linux.
If `n_jobs=None`, will use either the number of: physical cores if psutil is installed, or logical cores otherwise.
verbose : bool, optional
Whether to suppress print statements or not.
quality_score_kwargs : dict, optional
Keyword arguments to pass into :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>`.
num_issue_kwargs : dict, optional
Keyword arguments to :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`
to control estimation of the number of label issues.
The only supported kwarg here for now is: `estimation_method`.
return_mask : bool, optional
Determines what is returned by this method: If `return_mask=True`, return a boolean mask.
If `False`, return a list of indices specifying examples with label issues, sorted by label quality score.
Returns
-------
label_issues : np.ndarray
If `return_mask` is `True`, returns a boolean **mask** for the entire dataset
where ``True`` represents a label issue and ``False`` represents an example that is
accurately labeled with high confidence.
If `return_mask` is `False`, returns an array containing **indices** of examples identified to have
label issues (i.e. those indices where the mask would be ``True``), sorted by likelihood that the corresponding label is correct.
--------
>>> batch_size = 10000 # for efficiency, set this to as large of a value as your memory can handle
>>> # Just demonstrating how to save your existing numpy labels, pred_probs arrays to compatible .npy files:
>>> np.save("LABELS.npy", labels_array)
>>> np.save("PREDPROBS.npy", pred_probs_array)
>>> # You can load these back into memmap arrays via: labels = np.load("LABELS.npy", mmap_mode="r")
>>> # and then run this method on the memmap arrays, or just run it directly on the .npy files like this:
>>> issues = find_label_issues_batched(labels_file="LABELS.npy", pred_probs_file="PREDPROBS.npy", batch_size=batch_size)
>>> # This method also works with Zarr arrays:
>>> import zarr
>>> # Just demonstrating how to save your existing numpy labels, pred_probs arrays to compatible .zarr files:
>>> zarr.convenience.save_array("LABELS.zarr", labels_array)
>>> zarr.convenience.save_array("PREDPROBS.zarr", pred_probs_array)
>>> # You can load from such files into Zarr arrays:
>>> labels = zarr.convenience.open("LABELS.zarr", mode="r")
>>> pred_probs = zarr.convenience.open("PREDPROBS.zarr", mode="r")
>>> # This method can be directly run on Zarr arrays, memmap arrays, or regular numpy arrays:
>>> issues = find_label_issues_batched(labels=labels, pred_probs=pred_probs, batch_size=batch_size)
"""
if labels_file is not None:
if labels is not None:
raise ValueError("only specify one of: `labels` or `labels_file`")
if not isinstance(labels_file, str):
raise ValueError(
"labels_file must be str specifying path to .npy file containing the array of labels"
)
labels = np.load(labels_file, mmap_mode="r")
assert isinstance(labels, np.ndarray)
if pred_probs_file is not None:
if pred_probs is not None:
raise ValueError("only specify one of: `pred_probs` or `pred_probs_file`")
if not isinstance(pred_probs_file, str):
raise ValueError(
"pred_probs_file must be str specifying path to .npy file containing 2D array of pred_probs"
)
pred_probs = np.load(pred_probs_file, mmap_mode="r")
assert isinstance(pred_probs, np.ndarray)
if verbose:
print(
f"mmap-loaded numpy arrays have: {len(pred_probs)} examples, {pred_probs.shape[1]} classes"
)
if labels is None:
raise ValueError("must provide one of: `labels` or `labels_file`")
if pred_probs is None:
raise ValueError("must provide one of: `pred_probs` or `pred_probs_file`")
assert pred_probs is not None
if len(labels) != len(pred_probs):
raise ValueError(
f"len(labels)={len(labels)} does not match len(pred_probs)={len(pred_probs)}. Perhaps an issue loading mmap numpy arrays from file."
)
lab = LabelInspector(
num_class=pred_probs.shape[1],
verbose=verbose,
n_jobs=n_jobs,
quality_score_kwargs=quality_score_kwargs,
num_issue_kwargs=num_issue_kwargs,
)
n = len(labels)
if verbose:
from tqdm.auto import tqdm
pbar = tqdm(desc="number of examples processed for estimating thresholds", total=n)
i = 0
while i < n:
end_index = i + batch_size
labels_batch = labels[i:end_index]
pred_probs_batch = pred_probs[i:end_index, :]
i = end_index
lab.update_confident_thresholds(labels_batch, pred_probs_batch)
if verbose:
pbar.update(batch_size)
# Next evaluate the quality of the labels (run this on full dataset you want to evaluate):
if verbose:
pbar.close()
pbar = tqdm(desc="number of examples processed for checking labels", total=n)
i = 0
while i < n:
end_index = i + batch_size
labels_batch = labels[i:end_index]
pred_probs_batch = pred_probs[i:end_index, :]
i = end_index
_ = lab.score_label_quality(labels_batch, pred_probs_batch)
if verbose:
pbar.update(batch_size)
if verbose:
pbar.close()
label_issues_indices = lab.get_label_issues()
label_issues_mask = np.zeros(len(labels), dtype=bool)
label_issues_mask[label_issues_indices] = True
mask = _reduce_issues(pred_probs=pred_probs, labels=labels)
label_issues_mask[mask] = False
if return_mask:
return label_issues_mask
return np.where(label_issues_mask)[0]
class LabelInspector:
"""
Class for finding label issues in big datasets where memory becomes a problem for other cleanlab methods.
Only create one such object per dataset and do not try to use the same ``LabelInspector`` across 2 datasets.
For efficiency, this class does little input checking.
You can first run :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>`
on a small subset of your data to verify your inputs are properly formatted.
Do NOT modify any of the attributes of this class yourself!
Multi-label classification is not supported by this class, it is only for multi-class classification.
The recommended usage demonstrated in the examples script below involves two passes over your data:
one pass to compute `confident_thresholds`, another to evaluate each label.
To maximize efficiency, try to use the largest batch_size your memory allows.
To reduce runtime further, you can run the first pass on a subset of your dataset
as long as it contains enough data from each class to estimate `confident_thresholds` accurately.
In the examples script below:
- `labels` is a (big) 1D ``np.ndarray`` of class labels represented as integers in ``0,1,...,K-1``.
- ``pred_probs`` = is a (big) 2D ``np.ndarray`` of predicted class probabilities,
where each row is an example, each column represents a class.
`labels` and `pred_probs` can be stored in a file instead where you load chunks of them at a time.
Methods to load arrays in chunks include: ``np.load(...,mmap_mode='r')``, ``numpy.memmap()``,
HDF5 or Zarr files, see: https://pythonspeed.com/articles/mmap-vs-zarr-hdf5/
Examples
--------
>>> n = len(labels)
>>> batch_size = 10000 # you can change this in between batches, set as big as your RAM allows
>>> lab = LabelInspector(num_class = pred_probs.shape[1])
>>> # First compute confident thresholds (for faster results, can also do this on a random subset of your data):
>>> i = 0
>>> while i < n:
>>> end_index = i + batch_size
>>> labels_batch = labels[i:end_index]
>>> pred_probs_batch = pred_probs[i:end_index,:]
>>> i = end_index
>>> lab.update_confident_thresholds(labels_batch, pred_probs_batch)
>>> # See what we calculated:
>>> confident_thresholds = lab.get_confident_thresholds()
>>> # Evaluate the quality of the labels (run this on full dataset you want to evaluate):
>>> i = 0
>>> while i < n:
>>> end_index = i + batch_size
>>> labels_batch = labels[i:end_index]
>>> pred_probs_batch = pred_probs[i:end_index,:]
>>> i = end_index
>>> batch_results = lab.score_label_quality(labels_batch, pred_probs_batch)
>>> # Indices of examples with label issues, sorted by label quality score (most severe to least severe):
>>> indices_of_examples_with_issues = lab.get_label_issues()
>>> # If your `pred_probs` and `labels` are arrays already in memory,
>>> # then you can use this shortcut for all of the above:
>>> indices_of_examples_with_issues = find_label_issues_batched(labels, pred_probs, batch_size=10000)
Parameters
----------
num_class : int
The number of classes in your multi-class classification task.
store_results : bool, optional
Whether this object will store all label quality scores, a 1D array of shape ``(N,)``
where ``N`` is the total number of examples in your dataset.
Set this to False if you encounter memory problems even for small batch sizes (~1000).
If ``False``, you can still identify the label issues yourself by aggregating
the label quality scores for each batch, sorting them across all batches, and returning the top ``T`` indices
with ``T = self.get_num_issues()``.
verbose : bool, optional
Whether to suppress print statements or not.
n_jobs: int, optional
Number of processes for multiprocessing (default value = 1). Only used on Linux.
If `n_jobs=None`, will use either the number of: physical cores if psutil is installed, or logical cores otherwise.
quality_score_kwargs : dict, optional
Keyword arguments to pass into :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>`.
num_issue_kwargs : dict, optional
Keyword arguments to :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`
to control estimation of the number of label issues.
The only supported kwarg here for now is: `estimation_method`.
"""
def __init__(
self,
*,
num_class: int,
store_results: bool = True,
verbose: bool = True,
quality_score_kwargs: Optional[dict] = None,
num_issue_kwargs: Optional[dict] = None,
n_jobs: Optional[int] = 1,
):
if quality_score_kwargs is None:
quality_score_kwargs = {}
if num_issue_kwargs is None:
num_issue_kwargs = {}
self.num_class = num_class
self.store_results = store_results
self.verbose = verbose
self.quality_score_kwargs = quality_score_kwargs # extra arguments for ``rank.get_label_quality_scores()`` to control label quality scoring
self.num_issue_kwargs = num_issue_kwargs # extra arguments for ``count.num_label_issues()`` to control estimation of the number of label issues (only supported argument for now is: `estimation_method`).
self.off_diagonal_calibrated = False
if num_issue_kwargs.get("estimation_method") == "off_diagonal_calibrated":
# store extra attributes later needed for calibration:
self.off_diagonal_calibrated = True
self.prune_counts = np.zeros(self.num_class)
self.class_counts = np.zeros(self.num_class)
self.normalization = np.zeros(self.num_class)
else:
self.prune_count = 0 # number of label issues estimated based on data seen so far (only used when estimation_method is not calibrated)
if self.store_results:
self.label_quality_scores: List[float] = []
self.confident_thresholds = np.zeros(
(num_class,)
) # current estimate of thresholds based on data seen so far
self.examples_per_class = np.zeros(
(num_class,)
) # current counts of examples with each given label seen so far
self.examples_processed_thresh = (
0 # number of examples seen so far for estimating thresholds
)
self.examples_processed_quality = 0 # number of examples seen so far for estimating label quality and number of label issues
# Determine number of cores for multiprocessing:
self.n_jobs: Optional[int] = None
os_name = platform.system()
if os_name != "Linux":
self.n_jobs = 1
if n_jobs is not None and n_jobs != 1 and self.verbose:
print(
"n_jobs is overridden to 1 because multiprocessing is only supported for Linux."
)
elif n_jobs is not None:
self.n_jobs = n_jobs
else:
if PSUTIL_EXISTS:
self.n_jobs = psutil.cpu_count(logical=False) # physical cores
if not self.n_jobs:
# switch to logical cores
self.n_jobs = mp.cpu_count()
if self.verbose:
print(
f"Multiprocessing will default to using the number of logical cores ({self.n_jobs}). To default to number of physical cores: pip install psutil"
)
def get_confident_thresholds(self, silent: bool = False) -> np.ndarray:
"""
Fetches already-computed confident thresholds from the data seen so far
in same format as: :py:func:`count.get_confident_thresholds <cleanlab.count.get_confident_thresholds>`.
Returns
-------
confident_thresholds : np.ndarray
An array of shape ``(K, )`` where ``K`` is the number of classes.
"""
if self.examples_processed_thresh < 1:
raise ValueError(
"Have not computed any confident_thresholds yet. Call `update_confident_thresholds()` first."
)
else:
if self.verbose and not silent:
print(
f"Total number of examples used to estimate confident thresholds: {self.examples_processed_thresh}"
)
return self.confident_thresholds
def get_num_issues(self, silent: bool = False) -> int:
"""
Fetches already-computed estimate of the number of label issues in the data seen so far
in the same format as: :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`.
Note: The estimated number of issues may differ from :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`
by 1 due to rounding differences.
Returns
-------
num_issues : int
The estimated number of examples with label issues in the data seen so far.
"""
if self.examples_processed_quality < 1:
raise ValueError(
"Have not evaluated any labels yet. Call `score_label_quality()` first."
)
else:
if self.verbose and not silent:
print(
f"Total number of examples whose labels have been evaluated: {self.examples_processed_quality}"
)
if self.off_diagonal_calibrated:
calibrated_prune_counts = (
self.prune_counts
* self.class_counts
/ np.clip(self.normalization, a_min=CLIPPING_LOWER_BOUND, a_max=None)
) # avoid division by 0
return np.rint(np.sum(calibrated_prune_counts)).astype("int")
else: # not calibrated
return self.prune_count
def get_quality_scores(self) -> np.ndarray:
"""
Fetches already-computed estimate of the label quality of each example seen so far
in the same format as: :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>`.
Returns
-------
label_quality_scores : np.ndarray
Contains one score (between 0 and 1) per example seen so far.
Lower scores indicate more likely mislabeled examples.
"""
if not self.store_results:
raise ValueError(
"Must initialize the LabelInspector with `store_results` == True. "
"Otherwise you can assemble the label quality scores yourself based on "
"the scores returned for each batch of data from `score_label_quality()`"
)
else:
return np.asarray(self.label_quality_scores)
def get_label_issues(self) -> np.ndarray:
"""
Fetches already-computed estimate of indices of examples with label issues in the data seen so far,
in the same format as: :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>`
with its `return_indices_ranked_by` argument specified.
Note: this method corresponds to ``filter.find_label_issues(..., filter_by=METHOD1, return_indices_ranked_by=METHOD2)``
where by default: ``METHOD1="low_self_confidence"``, ``METHOD2="self_confidence"``
or if this object was instantiated with ``quality_score_kwargs = {"method": "normalized_margin"}`` then we instead have:
``METHOD1="low_normalized_margin"``, ``METHOD2="normalized_margin"``.
Note: The estimated number of issues may differ from :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>`
by 1 due to rounding differences.
Returns
-------
issue_indices : np.ndarray
Indices of examples with label issues, sorted by label quality score.
"""
if not self.store_results:
raise ValueError(
"Must initialize the LabelInspector with `store_results` == True. "
"Otherwise you can identify label issues yourself based on the scores from all "
"the batches of data and the total number of issues returned by `get_num_issues()`"
)
if self.examples_processed_quality < 1:
raise ValueError(
"Have not evaluated any labels yet. Call `score_label_quality()` first."
)
if self.verbose:
print(
f"Total number of examples whose labels have been evaluated: {self.examples_processed_quality}"
)
return find_top_issues(self.get_quality_scores(), top=self.get_num_issues(silent=True))
def update_confident_thresholds(self, labels: LabelLike, pred_probs: np.ndarray):
"""
Updates the estimate of confident_thresholds stored in this class using a new batch of data.
Inputs should be in same format as for: :py:func:`count.get_confident_thresholds <cleanlab.count.get_confident_thresholds>`.
Parameters
----------
labels: np.ndarray or list
Given class labels for each example in the batch, values in ``0,1,2,...,K-1``.
pred_probs: np.ndarray
2D array of model-predicted class probabilities for each example in the batch.
"""
labels = _batch_check(labels, pred_probs, self.num_class)
batch_size = len(labels)
batch_thresholds = get_confident_thresholds(
labels, pred_probs
) # values for missing classes may exceed 1 but should not matter since we multiply by this class counts in the batch
batch_class_counts = value_counts_fill_missing_classes(labels, num_classes=self.num_class)
self.confident_thresholds = (
self.examples_per_class * self.confident_thresholds
+ batch_class_counts * batch_thresholds
) / np.clip(
self.examples_per_class + batch_class_counts, a_min=1, a_max=None
) # avoid division by 0
self.confident_thresholds = np.clip(
self.confident_thresholds, a_min=CONFIDENT_THRESHOLDS_LOWER_BOUND, a_max=None
)
self.examples_per_class += batch_class_counts
self.examples_processed_thresh += batch_size
def score_label_quality(
self,
labels: LabelLike,
pred_probs: np.ndarray,
*,
update_num_issues: bool = True,
) -> np.ndarray:
"""
Scores the label quality of each example in the provided batch of data,
and also updates the number of label issues stored in this class.
Inputs should be in same format as for: :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>`.
Parameters
----------
labels: np.ndarray
Given class labels for each example in the batch, values in ``0,1,2,...,K-1``.
pred_probs: np.ndarray
2D array of model-predicted class probabilities for each example in the batch of data.
update_num_issues: bool, optional
Whether or not to update the number of label issues or only compute label quality scores.
For lower runtimes, set this to ``False`` if you only want to score label quality and not find label issues.
Returns
-------
label_quality_scores : np.ndarray
Contains one score (between 0 and 1) for each example in the batch of data.
"""
labels = _batch_check(labels, pred_probs, self.num_class)
batch_size = len(labels)
scores = _compute_label_quality_scores(
labels,
pred_probs,
confident_thresholds=self.get_confident_thresholds(silent=True),
**self.quality_score_kwargs,
)
class_counts = value_counts_fill_missing_classes(labels, num_classes=self.num_class)
if update_num_issues:
self._update_num_label_issues(labels, pred_probs, **self.num_issue_kwargs)
self.examples_processed_quality += batch_size
if self.store_results:
self.label_quality_scores += list(scores)
return scores
def _update_num_label_issues(
self,
labels: LabelLike,
pred_probs: np.ndarray,
**kwargs,
):
"""
Update the estimate of num_label_issues stored in this class using a new batch of data.
Kwargs are ignored here for now (included for forwards compatibility).
Instead of being specified here, `estimation_method` should be declared when this class is initialized.
"""
# whether to match the output of count.num_label_issues exactly
# default is False, which gives significant speedup on large batches
# and empirically matches num_label_issues even on input sizes of
# 1M x 10k
thorough = False
if self.examples_processed_thresh < 1:
raise ValueError(
"Have not computed any confident_thresholds yet. Call `update_confident_thresholds()` first."
)
if self.n_jobs == 1:
adj_confident_thresholds = self.confident_thresholds - FLOATING_POINT_COMPARISON
pred_class = np.argmax(pred_probs, axis=1)
batch_size = len(labels)
if thorough:
# add margin for floating point comparison operations:
pred_gt_thresholds = pred_probs >= adj_confident_thresholds
max_ind = np.argmax(pred_probs * pred_gt_thresholds, axis=1)
if not self.off_diagonal_calibrated:
mask = (max_ind != labels) & (pred_class != labels)
else:
# calibrated
# should we change to above?
mask = pred_class != labels
else:
max_ind = pred_class
mask = pred_class != labels
if not self.off_diagonal_calibrated:
prune_count_batch = np.sum(
(
pred_probs[np.arange(batch_size), max_ind]
>= adj_confident_thresholds[max_ind]
)
& mask
)
self.prune_count += prune_count_batch
else: # calibrated
self.class_counts += value_counts_fill_missing_classes(
labels, num_classes=self.num_class
)
to_increment = (
pred_probs[np.arange(batch_size), max_ind] >= adj_confident_thresholds[max_ind]
)
for class_label in range(self.num_class):
labels_equal_to_class = labels == class_label
self.normalization[class_label] += np.sum(labels_equal_to_class & to_increment)
self.prune_counts[class_label] += np.sum(
labels_equal_to_class
& to_increment
& (max_ind != labels)
# & (pred_class != labels)
# This is not applied in num_label_issues(..., estimation_method="off_diagonal_custom"). Do we want to add it?
)
else: # multiprocessing implementation
global adj_confident_thresholds_shared
adj_confident_thresholds_shared = self.confident_thresholds - FLOATING_POINT_COMPARISON
global labels_shared, pred_probs_shared
labels_shared = labels
pred_probs_shared = pred_probs
# good values for this are ~1000-10000 in benchmarks where pred_probs has 1B entries:
processes = 5000
if len(labels) <= processes:
chunksize = 1
else:
chunksize = len(labels) // processes
inds = split_arr(np.arange(len(labels)), chunksize)
if thorough:
use_thorough = np.ones(len(inds), dtype=bool)
else:
use_thorough = np.zeros(len(inds), dtype=bool)
args = zip(inds, use_thorough)
# Use fork method explicitly for Python 3.14+ compatibility
# Falls back to default method if fork is not available
try:
ctx = mp.get_context("fork")
pool_class = ctx.Pool
except (RuntimeError, ValueError):
# fork not available (Windows) or already set, use default
pool_class = mp.Pool
with pool_class(self.n_jobs) as pool:
if not self.off_diagonal_calibrated:
prune_count_batch = np.sum(
np.asarray(list(pool.imap_unordered(_compute_num_issues, args)))
)
self.prune_count += prune_count_batch
else:
results = list(pool.imap_unordered(_compute_num_issues_calibrated, args))
for result in results:
class_label = result[0]
self.class_counts[class_label] += 1
self.normalization[class_label] += result[1]
self.prune_counts[class_label] += result[2]
def split_arr(arr: np.ndarray, chunksize: int) -> List[np.ndarray]:
"""
Helper function to split array into chunks for multiprocessing.
"""
return np.split(arr, np.arange(chunksize, arr.shape[0], chunksize), axis=0)
def _compute_num_issues(arg: Tuple[np.ndarray, bool]) -> int:
"""
Helper function for `_update_num_label_issues` multiprocessing without calibration.
"""
ind = arg[0]
thorough = arg[1]
label = labels_shared[ind]
pred_prob = pred_probs_shared[ind, :]
pred_class = np.argmax(pred_prob, axis=-1)
batch_size = len(label)
if thorough:
pred_gt_thresholds = pred_prob >= adj_confident_thresholds_shared
max_ind = np.argmax(pred_prob * pred_gt_thresholds, axis=-1)
prune_count_batch = np.sum(
(pred_prob[np.arange(batch_size), max_ind] >= adj_confident_thresholds_shared[max_ind])
& (max_ind != label)
& (pred_class != label)
)
else:
prune_count_batch = np.sum(
(
pred_prob[np.arange(batch_size), pred_class]
>= adj_confident_thresholds_shared[pred_class]
)
& (pred_class != label)
)
return prune_count_batch
def _compute_num_issues_calibrated(arg: Tuple[np.ndarray, bool]) -> Tuple[Any, int, int]:
"""
Helper function for `_update_num_label_issues` multiprocessing with calibration.
"""
ind = arg[0]
thorough = arg[1]
label = labels_shared[ind]
pred_prob = pred_probs_shared[ind, :]
batch_size = len(label)
pred_class = np.argmax(pred_prob, axis=-1)
if thorough:
pred_gt_thresholds = pred_prob >= adj_confident_thresholds_shared
max_ind = np.argmax(pred_prob * pred_gt_thresholds, axis=-1)
to_inc = (
pred_prob[np.arange(batch_size), max_ind] >= adj_confident_thresholds_shared[max_ind]
)
prune_count_batch = to_inc & (max_ind != label)
normalization_batch = to_inc
else:
to_inc = (
pred_prob[np.arange(batch_size), pred_class]
>= adj_confident_thresholds_shared[pred_class]
)
normalization_batch = to_inc
prune_count_batch = to_inc & (pred_class != label)
return (label, normalization_batch, prune_count_batch)
def _batch_check(labels: LabelLike, pred_probs: np.ndarray, num_class: int) -> np.ndarray:
"""
Basic checks to ensure batch of data looks ok. For efficiency, this check is quite minimal.
Returns
-------
labels : np.ndarray
`labels` formatted as a 1D array.
"""
batch_size = pred_probs.shape[0]
labels = np.asarray(labels)
if len(labels) != batch_size:
raise ValueError("labels and pred_probs must have same length")
if pred_probs.shape[1] != num_class:
raise ValueError("num_class must equal pred_probs.shape[1]")
return labels
cleanlab/experimental/mnist_pytorch.py
+369
View File
@@ -0,0 +1,369 @@
"""
A cleanlab-compatible PyTorch ConvNet classifier that can be used to find
label issues in image data.
This is a good example to reference for making your own bespoke model compatible with cleanlab.
You must have PyTorch installed: https://pytorch.org/get-started/locally/
"""
from sklearn.base import BaseEstimator
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable
from torch.utils.data.sampler import SubsetRandomSampler
import numpy as np
MNIST_TRAIN_SIZE = 60000
MNIST_TEST_SIZE = 10000
SKLEARN_DIGITS_TRAIN_SIZE = 1247
SKLEARN_DIGITS_TEST_SIZE = 550
def get_mnist_dataset(loader): # pragma: no cover
"""Downloads MNIST as PyTorch dataset.
Parameters
----------
loader : str (values: 'train' or 'test')."""
dataset = datasets.MNIST(
root="../data",
train=(loader == "train"),
download=True,
transform=transforms.Compose(
[transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
),
)
return dataset
def get_sklearn_digits_dataset(loader):
"""Downloads Sklearn handwritten digits dataset.
Uses the last SKLEARN_DIGITS_TEST_SIZE examples as the test
This is (hard-coded) -- do not change.
Parameters
----------
loader : str (values: 'train' or 'test')."""
from torch.utils.data import Dataset
from sklearn.datasets import load_digits
class TorchDataset(Dataset):
"""Abstracts a numpy array as a PyTorch dataset."""
def __init__(self, data, targets, transform=None):
self.data = torch.from_numpy(data).float()
self.targets = torch.from_numpy(targets).long()
self.transform = transform
def __getitem__(self, index):
x = self.data[index]
y = self.targets[index]
if self.transform:
x = self.transform(x)
return x, y
def __len__(self):
return len(self.data)
transform = transforms.Compose(
[
transforms.ToPILImage(),
transforms.Resize(28),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,)),
]
)
# Get sklearn digits dataset
X_all, y_all = load_digits(return_X_y=True)
X_all = X_all.reshape((len(X_all), 8, 8))
y_train = y_all[:-SKLEARN_DIGITS_TEST_SIZE]
y_test = y_all[-SKLEARN_DIGITS_TEST_SIZE:]
X_train = X_all[:-SKLEARN_DIGITS_TEST_SIZE]
X_test = X_all[-SKLEARN_DIGITS_TEST_SIZE:]
if loader == "train":
return TorchDataset(X_train, y_train, transform=transform)
elif loader == "test":
return TorchDataset(X_test, y_test, transform=transform)
else: # prama: no cover
raise ValueError("loader must be either str 'train' or str 'test'.")
class SimpleNet(nn.Module):
"""Basic Pytorch CNN for MNIST-like data."""
def __init__(self):
super(SimpleNet, self).__init__()
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
self.conv2_drop = nn.Dropout2d()
self.fc1 = nn.Linear(320, 50)
self.fc2 = nn.Linear(50, 10)
def forward(self, x, T=1.0):
x = F.relu(F.max_pool2d(self.conv1(x), 2))
x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
x = x.view(-1, 320)
x = F.relu(self.fc1(x))
x = F.dropout(x, training=self.training)
x = self.fc2(x)
x = F.log_softmax(x, dim=1)
return x
class CNN(BaseEstimator): # Inherits sklearn classifier
"""Wraps a PyTorch CNN for the MNIST dataset within an sklearn template
Defines ``.fit()``, ``.predict()``, and ``.predict_proba()`` functions. This
template enables the PyTorch CNN to flexibly be used within the sklearn
architecture -- meaning it can be passed into functions like
cross_val_predict as if it were an sklearn model. The cleanlab library
requires that all models adhere to this basic sklearn template and thus,
this class allows a PyTorch CNN to be used in for learning with noisy
labels among other things.
Parameters
----------
batch_size: int
epochs: int
log_interval: int
lr: float
momentum: float
no_cuda: bool
seed: int
test_batch_size: int, default=None
dataset: {'mnist', 'sklearn-digits'}
loader: {'train', 'test'}
Set to 'test' to force fit() and predict_proba() on test_set
Note
----
Be careful setting the ``loader`` param, it will override every other loader
If you set this to 'test', but call .predict(loader = 'train')
then .predict() will still predict on test!
Attributes
----------
batch_size: int
epochs: int
log_interval: int
lr: float
momentum: float
no_cuda: bool
seed: int
test_batch_size: int, default=None
dataset: {'mnist', 'sklearn-digits'}
loader: {'train', 'test'}
Set to 'test' to force fit() and predict_proba() on test_set
Methods
-------
fit
fits the model to data.
predict
get the fitted model's prediction on test data
predict_proba
get the fitted model's probability distribution over classes for test data
"""
def __init__(
self,
batch_size=64,
epochs=6,
log_interval=50, # Set to None to not print
lr=0.01,
momentum=0.5,
no_cuda=False,
seed=1,
test_batch_size=None,
dataset="mnist",
loader=None,
):
self.batch_size = batch_size
self.epochs = epochs
self.log_interval = log_interval
self.lr = lr
self.momentum = momentum
self.no_cuda = no_cuda
self.seed = seed
self.cuda = not self.no_cuda and torch.cuda.is_available()
torch.manual_seed(self.seed)
if self.cuda: # pragma: no cover
torch.cuda.manual_seed(self.seed)
# Instantiate PyTorch model
self.model = SimpleNet()
if self.cuda: # pragma: no cover
self.model.cuda()
self.loader_kwargs = {"num_workers": 1, "pin_memory": True} if self.cuda else {}
self.loader = loader
self._set_dataset(dataset)
if test_batch_size is not None:
self.test_batch_size = test_batch_size
else:
self.test_batch_size = self.test_size
def _set_dataset(self, dataset):
self.dataset = dataset
if dataset == "mnist":
# pragma: no cover
self.get_dataset = get_mnist_dataset
self.train_size = MNIST_TRAIN_SIZE
self.test_size = MNIST_TEST_SIZE
elif dataset == "sklearn-digits":
self.get_dataset = get_sklearn_digits_dataset
self.train_size = SKLEARN_DIGITS_TRAIN_SIZE
self.test_size = SKLEARN_DIGITS_TEST_SIZE
else: # pragma: no cover
raise ValueError("dataset must be 'mnist' or 'sklearn-digits'.")
# XXX this is a pretty weird sklearn estimator that does data loading
# internally in `fit`, and it supports multiple datasets and is aware of
# which dataset it's using; if we weren't doing this, we wouldn't need to
# override `get_params` / `set_params`
def get_params(self, deep=True):
return {
"batch_size": self.batch_size,
"epochs": self.epochs,
"log_interval": self.log_interval,
"lr": self.lr,
"momentum": self.momentum,
"no_cuda": self.no_cuda,
"test_batch_size": self.test_batch_size,
"dataset": self.dataset,
}
def set_params(self, **parameters): # pragma: no cover
for parameter, value in parameters.items():
if parameter != "dataset":
setattr(self, parameter, value)
if "dataset" in parameters:
self._set_dataset(parameters["dataset"])
return self
def fit(self, train_idx, train_labels=None, sample_weight=None, loader="train"):
"""This function adheres to sklearn's "fit(X, y)" format for
compatibility with scikit-learn. ** All inputs should be numpy
arrays, not pyTorch Tensors train_idx is not X, but instead a list of
indices for X (and y if train_labels is None). This function is a
member of the cnn class which will handle creation of X, y from the
train_idx via the train_loader."""
if self.loader is not None:
loader = self.loader
if train_labels is not None and len(train_idx) != len(train_labels):
raise ValueError("Check that train_idx and train_labels are the same length.")
if sample_weight is not None: # pragma: no cover
if len(sample_weight) != len(train_labels):
raise ValueError(
"Check that train_labels and sample_weight " "are the same length."
)
class_weight = sample_weight[np.unique(train_labels, return_index=True)[1]]
class_weight = torch.from_numpy(class_weight).float()
if self.cuda:
class_weight = class_weight.cuda()
else:
class_weight = None
train_dataset = self.get_dataset(loader)
# Use provided labels if not None o.w. use MNIST dataset training labels
if train_labels is not None:
# Create sparse tensor of train_labels with (-1)s for labels not
# in train_idx. We avoid train_data[idx] because train_data may
# very large, i.e. ImageNet
sparse_labels = (
np.zeros(self.train_size if loader == "train" else self.test_size, dtype=int) - 1
)
sparse_labels[train_idx] = train_labels
train_dataset.targets = sparse_labels
train_loader = torch.utils.data.DataLoader(
dataset=train_dataset,
# sampler=SubsetRandomSampler(train_idx if train_idx is not None
# else range(self.train_size)),
sampler=SubsetRandomSampler(train_idx),
batch_size=self.batch_size,
**self.loader_kwargs,
)
optimizer = optim.SGD(self.model.parameters(), lr=self.lr, momentum=self.momentum)
# Train for self.epochs epochs
for epoch in range(1, self.epochs + 1):
# Enable dropout and batch norm layers
self.model.train()
for batch_idx, (data, target) in enumerate(train_loader):
if self.cuda: # pragma: no cover
data, target = data.cuda(), target.cuda()
data, target = Variable(data), Variable(target).long()
optimizer.zero_grad()
output = self.model(data)
loss = F.nll_loss(output, target, class_weight)
loss.backward()
optimizer.step()
if self.log_interval is not None and batch_idx % self.log_interval == 0:
print(
"TrainEpoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}".format(
epoch,
batch_idx * len(data),
len(train_idx),
100.0 * batch_idx / len(train_loader),
loss.item(),
),
)
def predict(self, idx=None, loader=None):
"""Get predicted labels from trained model."""
# get the index of the max probability
probs = self.predict_proba(idx, loader)
return probs.argmax(axis=1)
def predict_proba(self, idx=None, loader=None):
if self.loader is not None:
loader = self.loader
if loader is None:
is_test_idx = (
idx is not None
and len(idx) == self.test_size
and np.all(np.array(idx) == np.arange(self.test_size))
)
loader = "test" if is_test_idx else "train"
dataset = self.get_dataset(loader)
# Filter by idx
if idx is not None:
if (loader == "train" and len(idx) != self.train_size) or (
loader == "test" and len(idx) != self.test_size
):
dataset.data = dataset.data[idx]
dataset.targets = dataset.targets[idx]
loader = torch.utils.data.DataLoader(
dataset=dataset,
batch_size=self.batch_size if loader == "train" else self.test_batch_size,
**self.loader_kwargs,
)
# sets model.train(False) inactivating dropout and batch-norm layers
self.model.eval()
# Run forward pass on model to compute outputs
outputs = []
for data, _ in loader:
if self.cuda: # pragma: no cover
data = data.cuda()
with torch.no_grad():
data = Variable(data)
output = self.model(data)
outputs.append(output)
# Outputs are log_softmax (log probabilities)
outputs = torch.cat(outputs, dim=0)
# Convert to probabilities and return the numpy array of shape N x K
out = outputs.cpu().numpy() if self.cuda else outputs.numpy()
pred = np.exp(out)
return pred
cleanlab/experimental/span_classification.py
+106
View File
@@ -0,0 +1,106 @@
"""
Methods to find label issues in span classification datasets (text data), each token in a sentence receives one or more class labels.
The underlying label error detection algorithms are in `cleanlab.token_classification`.
"""
import numpy as np
from typing import List, Tuple, Optional
from cleanlab.token_classification.filter import find_label_issues as find_label_issues_token
from cleanlab.token_classification.summary import display_issues as display_issues_token
from cleanlab.token_classification.rank import (
get_label_quality_scores as get_label_quality_scores_token,
)
def find_label_issues(
labels: list,
pred_probs: list,
):
"""Identifies tokens with label issues in a span classification dataset.
Tokens identified with issues will be ranked by their individual label quality score.
To rank the sentences based on their overall label quality, use :py:func:`experimental.span_classification.get_label_quality_scores <cleanlab.experimental.span_classification.get_label_quality_scores>`
Parameters
----------
labels:
Nested list of given labels for all tokens.
Refer to documentation for this argument in :py:func:`token_classification.filter.find_label_issues <cleanlab.token_classification.filter.find_label_issues>` for further details.
Note: Currently, only a single span class is supported.
pred_probs:
An array of shape ``(T, K)`` of model-predicted class probabilities.
Refer to documentation for this argument in :py:func:`token_classification.filter.find_label_issues <cleanlab.token_classification.filter.find_label_issues>` for further details.
Returns
-------
issues:
List of label issues identified by cleanlab, such that each element is a tuple ``(i, j)``, which
indicates that the `j`-th token of the `i`-th sentence has a label issue.
These tuples are ordered in `issues` list based on the likelihood that the corresponding token is mislabeled.
Use :py:func:`experimental.span_classification.get_label_quality_scores <cleanlab.experimental.span_classification.get_label_quality_scores>`
to view these issues within the original sentences.
Examples
--------
>>> import numpy as np
>>> from cleanlab.experimental.span_classification import find_label_issues
>>> labels = [[0, 0, 1, 1], [1, 1, 0]]
>>> pred_probs = [
... np.array([0.9, 0.9, 0.9, 0.1]),
... np.array([0.1, 0.1, 0.9]),
... ]
>>> find_label_issues(labels, pred_probs)
"""
pred_probs_token = _get_pred_prob_token(pred_probs)
return find_label_issues_token(labels, pred_probs_token)
def display_issues(
issues: list,
tokens: List[List[str]],
*,
labels: Optional[list] = None,
pred_probs: Optional[list] = None,
exclude: List[Tuple[int, int]] = [],
class_names: Optional[List[str]] = None,
top: int = 20,
) -> None:
"""
See documentation of :py:meth:`token_classification.summary.display_issues<cleanlab.token_classification.summary.display_issues>` for description.
"""
display_issues_token(
issues,
tokens,
labels=labels,
pred_probs=pred_probs,
exclude=exclude,
class_names=class_names,
top=top,
)
def get_label_quality_scores(
labels: list,
pred_probs: list,
**kwargs,
) -> Tuple[np.ndarray, list]:
"""
See documentation of :py:meth:`token_classification.rank.get_label_quality_scores<cleanlab.token_classification.rank.get_label_quality_scores>` for description.
"""
pred_probs_token = _get_pred_prob_token(pred_probs)
return get_label_quality_scores_token(labels, pred_probs_token, **kwargs)
def _get_pred_prob_token(pred_probs: list) -> list:
"""Converts pred_probs for span classification to pred_probs for token classification."""
pred_probs_token = []
for probs in pred_probs:
pred_probs_token.append(np.stack([1 - probs, probs], axis=1))
return pred_probs_token
cleanlab/filter.py
+952
View File
@@ -0,0 +1,952 @@
"""
Methods to identify which examples have label issues in a classification dataset.
The documentation below assumes a dataset with ``N`` examples and ``K`` classes.
This module is for standard (multi-class) classification where each example is labeled as belonging to exactly one of K classes (e.g. ``labels = np.array([0,0,1,0,2,1])``).
Some methods here also work for multi-label classification data where each example can be labeled as belonging to multiple classes (e.g. ``labels = [[1,2],[1],[0],[],...]``),
but we encourage using the methods in the ``cleanlab.multilabel_classification`` module instead for such data.
"""
import numpy as np
from sklearn.metrics import confusion_matrix
import multiprocessing
import sys
import warnings
from typing import Any, Dict, Optional, Tuple, List
from functools import reduce
import platform
from cleanlab.count import calibrate_confident_joint, num_label_issues, _reduce_issues
from cleanlab.rank import order_label_issues, get_label_quality_scores
import cleanlab.internal.multilabel_scorer as ml_scorer
from cleanlab.internal.validation import assert_valid_inputs
from cleanlab.internal.util import (
value_counts_fill_missing_classes,
round_preserving_row_totals,
get_num_classes,
)
from cleanlab.internal.multilabel_utils import stack_complement, get_onehot_num_classes, int2onehot
from cleanlab.typing import LabelLike
from cleanlab.multilabel_classification.filter import find_multilabel_issues_per_class
# tqdm is a package to print time-to-complete when multiprocessing is used.
# This package is not necessary, but when installed improves user experience for large datasets.
try:
import tqdm.auto as tqdm
tqdm_exists = True
except ImportError as e: # pragma: no cover
tqdm_exists = False
w = """To see estimated completion times for methods in cleanlab.filter, "pip install tqdm"."""
warnings.warn(w)
# psutil is a package used to count physical cores for multiprocessing
# This package is not necessary, because we can always fall back to logical cores as the default
try:
import psutil
psutil_exists = True
except ImportError as e: # pragma: no cover
psutil_exists = False
# global variable for find_label_issues multiprocessing
pred_probs_by_class: Dict[int, np.ndarray]
prune_count_matrix_cols: Dict[int, np.ndarray]
def find_label_issues(
labels: LabelLike,
pred_probs: np.ndarray,
*,
return_indices_ranked_by: Optional[str] = None,
rank_by_kwargs: Optional[Dict[str, Any]] = None,
filter_by: str = "prune_by_noise_rate",
frac_noise: float = 1.0,
num_to_remove_per_class: Optional[List[int]] = None,
min_examples_per_class=1,
confident_joint: Optional[np.ndarray] = None,
n_jobs: Optional[int] = None,
verbose: bool = False,
multi_label: bool = False,
) -> np.ndarray:
"""
Identifies potentially bad labels in a classification dataset using confident learning.
Returns a boolean mask for the entire dataset where ``True`` represents
an example identified with a label issue and ``False`` represents an example that seems correctly labeled.
Instead of a mask, you can obtain indices of the examples with label issues in your dataset
(sorted by issue severity) by specifying the `return_indices_ranked_by` argument.
This determines which label quality score is used to quantify severity,
and is useful to view only the top-`J` most severe issues in your dataset.
The number of indices returned as issues is controlled by `frac_noise`: reduce its
value to identify fewer label issues. If you aren't sure, leave this set to 1.0.
Tip: if you encounter the error "pred_probs is not defined", try setting
``n_jobs=1``.
Parameters
----------
labels : np.ndarray or list
A discrete vector of noisy labels for a classification dataset, i.e. some labels may be erroneous.
*Format requirements*: for dataset with K classes, each label must be integer in 0, 1, ..., K-1.
For a standard (multi-class) classification dataset where each example is labeled with one class,
`labels` should be 1D array of shape ``(N,)``, for example: ``labels = [1,0,2,1,1,0...]``.
pred_probs : np.ndarray, optional
An array of shape ``(N, K)`` of model-predicted class probabilities,
``P(label=k|x)``. Each row of this matrix corresponds
to an example `x` and contains the model-predicted probabilities that
`x` belongs to each possible class, for each of the K classes. The
columns must be ordered such that these probabilities correspond to
class 0, 1, ..., K-1.
**Note**: Returned label issues are most accurate when they are computed based on out-of-sample `pred_probs` from your model.
To obtain out-of-sample predicted probabilities for every datapoint in your dataset, you can use :ref:`cross-validation <pred_probs_cross_val>`.
This is encouraged to get better results.
return_indices_ranked_by : {None, 'self_confidence', 'normalized_margin', 'confidence_weighted_entropy'}, default=None
Determines what is returned by this method: either a boolean mask or list of indices np.ndarray.
If ``None``, this function returns a boolean mask (``True`` if example at index is label error).
If not ``None``, this function returns a sorted array of indices of examples with label issues
(instead of a boolean mask). Indices are sorted by label quality score which can be one of:
- ``'normalized_margin'``: ``normalized margin (p(label = k) - max(p(label != k)))``
- ``'self_confidence'``: ``[pred_probs[i][labels[i]] for i in label_issues_idx]``
- ``'confidence_weighted_entropy'``: ``entropy(pred_probs) / self_confidence``
rank_by_kwargs : dict, optional
Optional keyword arguments to pass into scoring functions for ranking by
label quality score (see :py:func:`rank.get_label_quality_scores
<cleanlab.rank.get_label_quality_scores>`).
filter_by : {'prune_by_class', 'prune_by_noise_rate', 'both', 'confident_learning', 'predicted_neq_given', 'low_normalized_margin', 'low_self_confidence'}, default='prune_by_noise_rate'
Method to determine which examples are flagged as having label issue, so you can filter/prune them from the dataset. Options:
- ``'prune_by_noise_rate'``: filters examples with *high probability* of being mislabeled for every non-diagonal in the confident joint (see `prune_counts_matrix` in `filter.py`). These are the examples where (with high confidence) the given label is unlikely to match the predicted label for the example.
- ``'prune_by_class'``: filters the examples with *smallest probability* of belonging to their given class label for every class.
- ``'both'``: filters only those examples that would be filtered by both ``'prune_by_noise_rate'`` and ``'prune_by_class'``.
- ``'confident_learning'``: filters the examples counted as part of the off-diagonals of the confident joint. These are the examples that are confidently predicted to be a different label than their given label.
- ``'predicted_neq_given'``: filters examples for which the predicted class (i.e. argmax of the predicted probabilities) does not match the given label.
- ``'low_normalized_margin'``: filters the examples with *smallest* normalized margin label quality score. The number of issues returned matches :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`.
- ``'low_self_confidence'``: filters the examples with *smallest* self confidence label quality score. The number of issues returned matches :py:func:`count.num_label_issues <cleanlab.count.num_label_issues>`.
frac_noise : float, default=1.0
Used to only return the "top" ``frac_noise * num_label_issues``. The choice of which "top"
label issues to return is dependent on the `filter_by` method used. It works by reducing the
size of the off-diagonals of the `joint` distribution of given labels and true labels
proportionally by `frac_noise` prior to estimating label issues with each method.
This parameter only applies for `filter_by=both`, `filter_by=prune_by_class`, and
`filter_by=prune_by_noise_rate` methods and currently is unused by other methods.
When ``frac_noise=1.0``, return all "confident" estimated noise indices (recommended).
frac_noise * number_of_mislabeled_examples_in_class_k.
num_to_remove_per_class : array_like
An iterable of length K, the number of classes.
E.g. if K = 3, ``num_to_remove_per_class=[5, 0, 1]`` would return
the indices of the 5 most likely mislabeled examples in class 0,
and the most likely mislabeled example in class 2.
Note
----
Only set this parameter if ``filter_by='prune_by_class'``.
You may use with ``filter_by='prune_by_noise_rate'``, but
if ``num_to_remove_per_class=k``, then either k-1, k, or k+1
examples may be removed for any class due to rounding error. If you need
exactly 'k' examples removed from every class, you should use
``filter_by='prune_by_class'``.
min_examples_per_class : int, default=1
Minimum number of examples per class to avoid flagging as label issues.
This is useful to avoid deleting too much data from one class
when pruning noisy examples in datasets with rare classes.
confident_joint : np.ndarray, optional
An array of shape ``(K, K)`` representing the confident joint, the matrix used for identifying label issues, which
estimates a confident subset of the joint distribution of the noisy and true labels, ``P_{noisy label, true label}``.
Entry ``(j, k)`` in the matrix is the number of examples confidently counted into the pair of ``(noisy label=j, true label=k)`` classes.
The `confident_joint` can be computed using :py:func:`count.compute_confident_joint <cleanlab.count.compute_confident_joint>`.
If not provided, it is computed from the given (noisy) `labels` and `pred_probs`.
n_jobs : optional
Number of processing threads used by multiprocessing. Default ``None``
sets to the number of cores on your CPU (physical cores if you have ``psutil`` package installed, otherwise logical cores).
Set this to 1 to *disable* parallel processing (if its causing issues).
Windows users may see a speed-up with ``n_jobs=1``.
verbose : optional
If ``True``, prints when multiprocessing happens.
Returns
-------
label_issues : np.ndarray
If `return_indices_ranked_by` left unspecified, returns a boolean **mask** for the entire dataset
where ``True`` represents a label issue and ``False`` represents an example that is
accurately labeled with high confidence.
If `return_indices_ranked_by` is specified, returns a shorter array of **indices** of examples identified to have
label issues (i.e. those indices where the mask would be ``True``), sorted by likelihood that the corresponding label is correct.
Note
----
Obtain the *indices* of examples with label issues in your dataset by setting `return_indices_ranked_by`.
"""
if not rank_by_kwargs:
rank_by_kwargs = {}
assert filter_by in [
"low_normalized_margin",
"low_self_confidence",
"prune_by_noise_rate",
"prune_by_class",
"both",
"confident_learning",
"predicted_neq_given",
] # TODO: change default to confident_learning ?
allow_one_class = False
if isinstance(labels, np.ndarray) or all(isinstance(lab, int) for lab in labels):
if set(labels) == {0}: # occurs with missing classes in multi-label settings
allow_one_class = True
assert_valid_inputs(
X=None,
y=labels,
pred_probs=pred_probs,
multi_label=multi_label,
allow_one_class=allow_one_class,
)
if filter_by in [
"confident_learning",
"predicted_neq_given",
"low_normalized_margin",
"low_self_confidence",
] and (frac_noise != 1.0 or num_to_remove_per_class is not None):
warn_str = (
"frac_noise and num_to_remove_per_class parameters are only supported"
" for filter_by 'prune_by_noise_rate', 'prune_by_class', and 'both'. They "
"are not supported for methods 'confident_learning', 'predicted_neq_given', "
"'low_normalized_margin' or 'low_self_confidence'."
)
warnings.warn(warn_str)
if (num_to_remove_per_class is not None) and (
filter_by
in [
"confident_learning",
"predicted_neq_given",
"low_normalized_margin",
"low_self_confidence",
]
):
# TODO - add support for these filters
raise ValueError(
"filter_by 'confident_learning', 'predicted_neq_given', 'low_normalized_margin' "
"or 'low_self_confidence' is not supported (yet) when setting 'num_to_remove_per_class'"
)
if filter_by == "confident_learning" and isinstance(confident_joint, np.ndarray):
warn_str = (
"The supplied `confident_joint` is ignored when `filter_by = 'confident_learning'`; confident joint will be "
"re-estimated from the given labels. To use your supplied `confident_joint`, please specify a different "
"`filter_by` value."
)
warnings.warn(warn_str)
K = get_num_classes(
labels=labels, pred_probs=pred_probs, label_matrix=confident_joint, multi_label=multi_label
)
# Boolean set to true if dataset is large
big_dataset = K * len(labels) > 1e8
# Set-up number of multiprocessing threads
# On Windows/macOS, when multi_label is True, multiprocessing is much slower
# even for faily large input arrays, so we default to n_jobs=1 in this case
os_name = platform.system()
if n_jobs is None:
if multi_label and os_name != "Linux":
n_jobs = 1
else:
if psutil_exists:
n_jobs = psutil.cpu_count(logical=False) # physical cores
elif big_dataset:
print(
"To default `n_jobs` to the number of physical cores for multiprocessing in find_label_issues(), please: `pip install psutil`.\n"
"Note: You can safely ignore this message. `n_jobs` only affects runtimes, results will be the same no matter its value.\n"
"Since psutil is not installed, `n_jobs` was set to the number of logical cores by default.\n"
"Disable this message by either installing psutil or specifying the `n_jobs` argument."
) # pragma: no cover
if not n_jobs:
# either psutil does not exist
# or psutil can return None when physical cores cannot be determined
# switch to logical cores
n_jobs = multiprocessing.cpu_count()
else:
assert n_jobs >= 1
if multi_label:
if not isinstance(labels, list):
raise TypeError("`labels` must be list when `multi_label=True`.")
warnings.warn(
"The multi_label argument to filter.find_label_issues() is deprecated and will be removed in future versions. Please use `multilabel_classification.filter.find_label_issues()` instead.",
DeprecationWarning,
)
return _find_label_issues_multilabel(
labels,
pred_probs,
return_indices_ranked_by,
rank_by_kwargs,
filter_by,
frac_noise,
num_to_remove_per_class,
min_examples_per_class,
confident_joint,
n_jobs,
verbose,
)
# Else this is standard multi-class classification
# Number of examples in each class of labels
label_counts = value_counts_fill_missing_classes(labels, K, multi_label=multi_label)
# Ensure labels are of type np.ndarray()
labels = np.asarray(labels)
if confident_joint is None or filter_by == "confident_learning":
from cleanlab.count import compute_confident_joint
confident_joint, cl_error_indices = compute_confident_joint(
labels=labels,
pred_probs=pred_probs,
multi_label=multi_label,
return_indices_of_off_diagonals=True,
)
if filter_by in ["low_normalized_margin", "low_self_confidence"]:
# TODO: consider setting adjust_pred_probs to true based on benchmarks (or adding it kwargs, or ignoring and leaving as false by default)
scores = get_label_quality_scores(
labels,
pred_probs,
method=filter_by[4:],
adjust_pred_probs=False,
)
num_errors = num_label_issues(
labels, pred_probs, multi_label=multi_label # TODO: Check usage of multilabel
)
# Find label issues O(nlogn) solution (mapped to boolean mask later in the method)
cl_error_indices = np.argsort(scores)[:num_errors]
# The following is the O(n) fastest solution (check for one-off errors), but the problem is if lots of the scores are identical you will overcount,
# you can end up returning more or less and they aren't ranked in the boolean form so there's no way to drop the highest scores randomly
# boundary = np.partition(scores, num_errors)[num_errors] # O(n) solution
# label_issues_mask = scores <= boundary
if filter_by in ["prune_by_noise_rate", "prune_by_class", "both"]:
# Create `prune_count_matrix` with the number of examples to remove in each class and
# leave at least min_examples_per_class examples per class.
# `prune_count_matrix` is transposed relative to the confident_joint.
prune_count_matrix = _keep_at_least_n_per_class(
prune_count_matrix=confident_joint.T,
n=min_examples_per_class,
frac_noise=frac_noise,
)
if num_to_remove_per_class is not None:
# Estimate joint probability distribution over label issues
psy = prune_count_matrix / np.sum(prune_count_matrix, axis=1)
noise_per_s = psy.sum(axis=1) - psy.diagonal()
# Calibrate labels.t. noise rates sum to num_to_remove_per_class
tmp = (psy.T * num_to_remove_per_class / noise_per_s).T
np.fill_diagonal(tmp, label_counts - num_to_remove_per_class)
prune_count_matrix = round_preserving_row_totals(tmp)
# Prepare multiprocessing shared data
# On Linux with Python <3.14, multiprocessing is started with fork,
# so data can be shared with global variables + COW
# On Window/macOS, processes are started with spawn,
# so data will need to be pickled to the subprocesses through input args
# In Python 3.14+, global variable sharing is no longer reliable even on Linux
chunksize = max(1, K // n_jobs)
use_global_vars = n_jobs == 1 or (os_name == "Linux" and sys.version_info < (3, 14))
if use_global_vars:
global pred_probs_by_class, prune_count_matrix_cols
pred_probs_by_class = {k: pred_probs[labels == k] for k in range(K)}
prune_count_matrix_cols = {k: prune_count_matrix[:, k] for k in range(K)}
args = [[k, min_examples_per_class, None] for k in range(K)]
else:
args = [
[k, min_examples_per_class, [pred_probs[labels == k], prune_count_matrix[:, k]]]
for k in range(K)
]
# Perform Pruning with threshold probabilities from BFPRT algorithm in O(n)
# Operations are parallelized across all CPU processes
if filter_by == "prune_by_class" or filter_by == "both":
if n_jobs > 1:
with multiprocessing.Pool(n_jobs) as p:
if verbose: # pragma: no cover
print("Parallel processing label issues by class.")
sys.stdout.flush()
if big_dataset and tqdm_exists:
label_issues_masks_per_class = list(
tqdm.tqdm(p.imap(_prune_by_class, args, chunksize=chunksize), total=K)
)
else:
label_issues_masks_per_class = p.map(_prune_by_class, args, chunksize=chunksize)
else:
label_issues_masks_per_class = [_prune_by_class(arg) for arg in args]
label_issues_mask = np.zeros(len(labels), dtype=bool)
for k, mask in enumerate(label_issues_masks_per_class):
if len(mask) > 1:
label_issues_mask[labels == k] = mask
if filter_by == "both":
label_issues_mask_by_class = label_issues_mask
if filter_by == "prune_by_noise_rate" or filter_by == "both":
if n_jobs > 1:
with multiprocessing.Pool(n_jobs) as p:
if verbose: # pragma: no cover
print("Parallel processing label issues by noise rate.")
sys.stdout.flush()
if big_dataset and tqdm_exists:
label_issues_masks_per_class = list(
tqdm.tqdm(p.imap(_prune_by_count, args, chunksize=chunksize), total=K)
)
else:
label_issues_masks_per_class = p.map(_prune_by_count, args, chunksize=chunksize)
else:
label_issues_masks_per_class = [_prune_by_count(arg) for arg in args]
label_issues_mask = np.zeros(len(labels), dtype=bool)
for k, mask in enumerate(label_issues_masks_per_class):
if len(mask) > 1:
label_issues_mask[labels == k] = mask
if filter_by == "both":
label_issues_mask = label_issues_mask & label_issues_mask_by_class
if filter_by in ["confident_learning", "low_normalized_margin", "low_self_confidence"]:
label_issues_mask = np.zeros(len(labels), dtype=bool)
label_issues_mask[cl_error_indices] = True
if filter_by == "predicted_neq_given":
label_issues_mask = find_predicted_neq_given(labels, pred_probs, multi_label=multi_label)
if filter_by not in ["low_self_confidence", "low_normalized_margin"]:
# Remove label issues if model prediction is close to given label
mask = _reduce_issues(pred_probs=pred_probs, labels=labels)
label_issues_mask[mask] = False
if verbose:
print("Number of label issues found: {}".format(sum(label_issues_mask)))
# TODO: run count.num_label_issues() and adjust the total issues found here to match
if return_indices_ranked_by is not None:
er = order_label_issues(
label_issues_mask=label_issues_mask,
labels=labels,
pred_probs=pred_probs,
rank_by=return_indices_ranked_by,
rank_by_kwargs=rank_by_kwargs,
)
return er
return label_issues_mask
def _find_label_issues_multilabel(
labels: list,
pred_probs: np.ndarray,
return_indices_ranked_by: Optional[str] = None,
rank_by_kwargs={},
filter_by: str = "prune_by_noise_rate",
frac_noise: float = 1.0,
num_to_remove_per_class: Optional[List[int]] = None,
min_examples_per_class=1,
confident_joint: Optional[np.ndarray] = None,
n_jobs: Optional[int] = None,
verbose: bool = False,
low_memory: bool = False,
) -> np.ndarray:
"""
Finds label issues in multi-label classification data where each example can belong to more than one class.
This is done via a one-vs-rest reduction for each class and the results are subsequently aggregated across all classes.
Here `labels` must be formatted as an iterable of iterables, e.g. ``List[List[int]]``.
"""
if filter_by in ["low_normalized_margin", "low_self_confidence"] and not low_memory:
num_errors = sum(
find_label_issues(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
multi_label=True,
filter_by="confident_learning",
)
)
y_one, num_classes = get_onehot_num_classes(labels, pred_probs)
label_quality_scores = ml_scorer.get_label_quality_scores(
labels=y_one,
pred_probs=pred_probs,
)
cl_error_indices = np.argsort(label_quality_scores)[:num_errors]
label_issues_mask = np.zeros(len(labels), dtype=bool)
label_issues_mask[cl_error_indices] = True
if return_indices_ranked_by is not None:
label_quality_scores_issues = ml_scorer.get_label_quality_scores(
labels=y_one[label_issues_mask],
pred_probs=pred_probs[label_issues_mask],
method=ml_scorer.MultilabelScorer(
base_scorer=ml_scorer.ClassLabelScorer.from_str(return_indices_ranked_by),
),
base_scorer_kwargs=rank_by_kwargs,
)
return cl_error_indices[np.argsort(label_quality_scores_issues)]
return label_issues_mask
per_class_issues = find_multilabel_issues_per_class(
labels,
pred_probs,
return_indices_ranked_by,
rank_by_kwargs,
filter_by,
frac_noise,
num_to_remove_per_class,
min_examples_per_class,
confident_joint,
n_jobs,
verbose,
low_memory,
)
if return_indices_ranked_by is None:
assert isinstance(per_class_issues, np.ndarray)
return per_class_issues.sum(axis=1) >= 1
else:
label_issues_list, labels_list, pred_probs_list = per_class_issues
label_issues_idx = reduce(np.union1d, label_issues_list)
y_one, num_classes = get_onehot_num_classes(labels, pred_probs)
label_quality_scores = ml_scorer.get_label_quality_scores(
labels=y_one,
pred_probs=pred_probs,
method=ml_scorer.MultilabelScorer(
base_scorer=ml_scorer.ClassLabelScorer.from_str(return_indices_ranked_by),
),
base_scorer_kwargs=rank_by_kwargs,
)
label_quality_scores_issues = label_quality_scores[label_issues_idx]
return label_issues_idx[np.argsort(label_quality_scores_issues)]
def _keep_at_least_n_per_class(
prune_count_matrix: np.ndarray, n: int, *, frac_noise: float = 1.0
) -> np.ndarray:
"""Make sure every class has at least n examples after removing noise.
Functionally, increase each column, increases the diagonal term #(true_label=k,label=k)
of prune_count_matrix until it is at least n, distributing the amount
increased by subtracting uniformly from the rest of the terms in the
column. When frac_noise = 1.0, return all "confidently" estimated
noise indices, otherwise this returns frac_noise fraction of all
the noise counts, with diagonal terms adjusted to ensure column
totals are preserved.
Parameters
----------
prune_count_matrix : np.ndarray of shape (K, K), K = number of classes
A counts of mislabeled examples in every class. For this function.
NOTE prune_count_matrix is transposed relative to confident_joint.
n : int
Number of examples to make sure are left in each class.
frac_noise : float, default=1.0
Used to only return the "top" ``frac_noise * num_label_issues``. The choice of which "top"
label issues to return is dependent on the `filter_by` method used. It works by reducing the
size of the off-diagonals of the `prune_count_matrix` of given labels and true labels
proportionally by `frac_noise` prior to estimating label issues with each method.
When frac_noise=1.0, return all "confident" estimated noise indices (recommended).
Returns
-------
prune_count_matrix : np.ndarray of shape (K, K), K = number of classes
This the same as the confident_joint, but has been transposed and the counts are adjusted.
"""
prune_count_matrix_diagonal = np.diagonal(prune_count_matrix)
# Set diagonal terms less than n, to n.
new_diagonal = np.maximum(prune_count_matrix_diagonal, n)
# Find how much diagonal terms were increased.
diff_per_col = new_diagonal - prune_count_matrix_diagonal
# Count non-zero, non-diagonal items per column
# np.maximum(*, 1) makes this never 0 (we divide by this next)
num_noise_rates_per_col = np.maximum(
np.count_nonzero(prune_count_matrix, axis=0) - 1.0,
1.0,
)
# Uniformly decrease non-zero noise rates by the same amount
# that the diagonal items were increased
new_mat = prune_count_matrix - diff_per_col / num_noise_rates_per_col
# Originally zero noise rates will now be negative, fix them back to zero
new_mat[new_mat < 0] = 0
# Round diagonal terms (correctly labeled examples)
np.fill_diagonal(new_mat, new_diagonal)
# Reduce (multiply) all noise rates (non-diagonal) by frac_noise and
# increase diagonal by the total amount reduced in each column
# to preserve column counts.
new_mat = _reduce_prune_counts(new_mat, frac_noise)
# These are counts, so return a matrix of ints.
return round_preserving_row_totals(new_mat).astype(int)
def _reduce_prune_counts(prune_count_matrix: np.ndarray, frac_noise: float = 1.0) -> np.ndarray:
"""Reduce (multiply) all prune counts (non-diagonal) by frac_noise and
increase diagonal by the total amount reduced in each column to
preserve column counts.
Parameters
----------
prune_count_matrix : np.ndarray of shape (K, K), K = number of classes
A counts of mislabeled examples in every class. For this function, it
does not matter what the rows or columns are, but the diagonal terms
reflect the number of correctly labeled examples.
frac_noise : float
Used to only return the "top" ``frac_noise * num_label_issues``. The choice of which "top"
label issues to return is dependent on the `filter_by` method used. It works by reducing the
size of the off-diagonals of the `prune_count_matrix` of given labels and true labels
proportionally by `frac_noise` prior to estimating label issues with each method.
When frac_noise=1.0, return all "confident" estimated noise indices (recommended).
"""
new_mat = prune_count_matrix * frac_noise
np.fill_diagonal(new_mat, prune_count_matrix.diagonal())
np.fill_diagonal(
new_mat,
prune_count_matrix.diagonal() + np.sum(prune_count_matrix - new_mat, axis=0),
)
# These are counts, so return a matrix of ints.
return new_mat.astype(int)
def find_predicted_neq_given(
labels: LabelLike, pred_probs: np.ndarray, *, multi_label: bool = False
) -> np.ndarray:
"""A simple baseline approach that considers ``argmax(pred_probs) != labels`` as the examples with label issues.
Parameters
----------
labels : np.ndarray or list
Labels in the same format expected by the `~cleanlab.filter.find_label_issues` function.
pred_probs : np.ndarray
Predicted-probabilities in the same format expected by the `~cleanlab.filter.find_label_issues` function.
multi_label : bool, optional
Whether each example may have multiple labels or not (see documentation for the `~cleanlab.filter.find_label_issues` function).
Returns
-------
label_issues_mask : np.ndarray
A boolean mask for the entire dataset where ``True`` represents a
label issue and ``False`` represents an example that is accurately
labeled with high confidence.
"""
assert_valid_inputs(X=None, y=labels, pred_probs=pred_probs, multi_label=multi_label)
if multi_label:
if not isinstance(labels, list):
raise TypeError("`labels` must be list when `multi_label=True`.")
else:
return _find_predicted_neq_given_multilabel(labels=labels, pred_probs=pred_probs)
else:
return np.argmax(pred_probs, axis=1) != np.asarray(labels)
def _find_predicted_neq_given_multilabel(labels: list, pred_probs: np.ndarray) -> np.ndarray:
"""
Parameters
----------
labels : list
List of noisy labels for multi-label classification where each example can belong to multiple classes
(e.g. ``labels = [[1,2],[1],[0],[],...]`` indicates the first example in dataset belongs to both class 1 and class 2).
pred_probs : np.ndarray
Predicted-probabilities in the same format expected by the `~cleanlab.filter.find_label_issues` function.
Returns
-------
label_issues_mask : np.ndarray
A boolean mask for the entire dataset where ``True`` represents a
label issue and ``False`` represents an example that is accurately
labeled with high confidence.
"""
y_one, num_classes = get_onehot_num_classes(labels, pred_probs)
pred_neq: np.ndarray = np.zeros(y_one.shape).astype(bool)
for class_num, (label, pred_prob_for_class) in enumerate(zip(y_one.T, pred_probs.T)):
pred_probs_binary = stack_complement(pred_prob_for_class)
pred_neq[:, class_num] = find_predicted_neq_given(
labels=label, pred_probs=pred_probs_binary
)
return pred_neq.sum(axis=1) >= 1
def find_label_issues_using_argmax_confusion_matrix(
labels: np.ndarray,
pred_probs: np.ndarray,
*,
calibrate: bool = True,
filter_by: str = "prune_by_noise_rate",
) -> np.ndarray:
"""A baseline approach that uses the confusion matrix
of ``argmax(pred_probs)`` and labels as the confident joint and then uses cleanlab
(confident learning) to find the label issues using this matrix.
The only difference between this and `~cleanlab.filter.find_label_issues` is that it uses the confusion matrix
based on the argmax and given label instead of using the confident joint
from :py:func:`count.compute_confident_joint
<cleanlab.count.compute_confident_joint>`.
Parameters
----------
labels : np.ndarray
An array of shape ``(N,)`` of noisy labels, i.e. some labels may be erroneous.
Elements must be in the set 0, 1, ..., K-1, where K is the number of classes.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted probabilities,
``P(label=k|x)``. Each row of this matrix corresponds
to an example `x` and contains the model-predicted probabilities that
`x` belongs to each possible class, for each of the K classes. The
columns must be ordered such that these probabilities correspond to
class 0, 1, ..., K-1. `pred_probs` should have been computed using 3 (or
higher) fold cross-validation.
calibrate : bool, default=True
Set to ``True`` to calibrate the confusion matrix created by ``pred != given labels``.
This calibration adjusts the confusion matrix / confident joint so that the
prior (given noisy labels) is correct based on the original labels.
filter_by : str, default='prune_by_noise_rate'
See `filter_by` argument of `~cleanlab.filter.find_label_issues`.
Returns
-------
label_issues_mask : np.ndarray
A boolean mask for the entire dataset where ``True`` represents a
label issue and ``False`` represents an example that is accurately
labeled with high confidence.
"""
assert_valid_inputs(X=None, y=labels, pred_probs=pred_probs, multi_label=False)
confident_joint = confusion_matrix(np.argmax(pred_probs, axis=1), labels).T
if calibrate:
confident_joint = calibrate_confident_joint(confident_joint, labels)
return find_label_issues(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
filter_by=filter_by,
)
# Multiprocessing helper functions:
mp_params: Dict[str, Any] = {} # Globals to be shared across threads in multiprocessing
def _to_np_array(
mp_arr: bytearray, dtype="int32", shape: Optional[Tuple[int, int]] = None
) -> np.ndarray: # pragma: no cover
"""multipropecessing Helper function to convert a multiprocessing
RawArray to a numpy array."""
arr = np.frombuffer(mp_arr, dtype=dtype)
if shape is None:
return arr
return arr.reshape(shape)
def _init(
__labels,
__label_counts,
__prune_count_matrix,
__pcm_shape,
__pred_probs,
__pred_probs_shape,
__multi_label,
__min_examples_per_class,
): # pragma: no cover
"""Shares memory objects across child processes.
ASSUMES none of these will be changed by child processes!"""
mp_params["labels"] = __labels
mp_params["label_counts"] = __label_counts
mp_params["prune_count_matrix"] = __prune_count_matrix
mp_params["pcm_shape"] = __pcm_shape
mp_params["pred_probs"] = __pred_probs
mp_params["pred_probs_shape"] = __pred_probs_shape
mp_params["multi_label"] = __multi_label
mp_params["min_examples_per_class"] = __min_examples_per_class
def _get_shared_data() -> Any: # pragma: no cover
"""multiprocessing helper function to extract numpy arrays from
shared RawArray types used to shared data across process."""
label_counts = _to_np_array(mp_params["label_counts"])
prune_count_matrix = _to_np_array(
mp_arr=mp_params["prune_count_matrix"],
shape=mp_params["pcm_shape"],
)
pred_probs = _to_np_array(
mp_arr=mp_params["pred_probs"],
dtype="float32",
shape=mp_params["pred_probs_shape"],
)
min_examples_per_class = mp_params["min_examples_per_class"]
multi_label = mp_params["multi_label"]
labels = _to_np_array(mp_params["labels"]) # type: ignore
return (
labels,
label_counts,
prune_count_matrix,
pred_probs,
multi_label,
min_examples_per_class,
)
# TODO figure out what the types inside args are.
def _prune_by_class(args: list) -> np.ndarray:
"""multiprocessing Helper function for find_label_issues()
that assumes globals and produces a mask for class k for each example by
removing the examples with *smallest probability* of
belonging to their given class label.
Parameters
----------
k : int (between 0 and num classes - 1)
The class of interest."""
k, min_examples_per_class, arrays = args
if arrays is None:
pred_probs = pred_probs_by_class[k]
prune_count_matrix = prune_count_matrix_cols[k]
else:
pred_probs = arrays[0]
prune_count_matrix = arrays[1]
label_counts = pred_probs.shape[0]
label_issues = np.zeros(label_counts, dtype=bool)
if label_counts > min_examples_per_class: # No prune if not at least min_examples_per_class
num_issues = label_counts - prune_count_matrix[k]
# Get return_indices_ranked_by of the smallest prob of class k for examples with noisy label k
# rank = np.partition(class_probs, num_issues)[num_issues]
if num_issues >= 1:
class_probs = pred_probs[:, k]
order = np.argsort(class_probs)
label_issues[order[:num_issues]] = True
return label_issues
warnings.warn(
f"May not flag all label issues in class: {k}, it has too few examples (see argument: `min_examples_per_class`)"
)
return label_issues
# TODO figure out what the types inside args are.
def _prune_by_count(args: list) -> np.ndarray:
"""multiprocessing Helper function for find_label_issues() that assumes
globals and produces a mask for class k for each example by
removing the example with noisy label k having *largest margin*,
where
margin of example := prob of given label - max prob of non-given labels
Parameters
----------
k : int (between 0 and num classes - 1)
The true_label class of interest."""
k, min_examples_per_class, arrays = args
if arrays is None:
pred_probs = pred_probs_by_class[k]
prune_count_matrix = prune_count_matrix_cols[k]
else:
pred_probs = arrays[0]
prune_count_matrix = arrays[1]
label_counts = pred_probs.shape[0]
label_issues_mask = np.zeros(label_counts, dtype=bool)
if label_counts <= min_examples_per_class:
warnings.warn(
f"May not flag all label issues in class: {k}, it has too few examples (see `min_examples_per_class` argument)"
)
return label_issues_mask
K = pred_probs.shape[1]
if K < 1:
raise ValueError("Must have at least 1 class.")
for j in range(K):
num2prune = prune_count_matrix[j]
# Only prune for noise rates, not diagonal entries
if k != j and num2prune > 0:
# num2prune's largest p(true class k) - p(noisy class k)
# for x with true label j
margin = pred_probs[:, j] - pred_probs[:, k]
order = np.argsort(-margin)
label_issues_mask[order[:num2prune]] = True
return label_issues_mask
# TODO: decide if we want to keep this based on TODO above. If so move to utils. Add unit test for this.
def _multiclass_crossval_predict(
labels: list, pred_probs: np.ndarray
) -> np.ndarray: # pragma: no cover
"""Returns a numpy 2D array of one-hot encoded
multiclass predictions. Each row in the array
provides the predictions for a particular example.
The boundary condition used to threshold predictions
is computed by maximizing the F1 ROC curve.
Parameters
----------
labels : list of lists (length N)
These are multiclass labels. Each list in the list contains all the
labels for that example.
pred_probs : np.ndarray (shape (N, K))
P(label=k|x) is a matrix with K model-predicted probabilities.
Each row of this matrix corresponds to an example `x` and contains the model-predicted
probabilities that `x` belongs to each possible class.
The columns must be ordered such that these probabilities correspond to class 0,1,2,...
`pred_probs` should have been computed using 3 (or higher) fold cross-validation."""
from sklearn.metrics import f1_score
boundaries = np.arange(0.05, 0.9, 0.05)
K = get_num_classes(
labels=labels,
pred_probs=pred_probs,
multi_label=True,
)
labels_one_hot = int2onehot(labels, K)
f1s = [
f1_score(
labels_one_hot,
(pred_probs > boundary).astype(np.uint8),
average="micro",
)
for boundary in boundaries
]
boundary = boundaries[np.argmax(f1s)]
pred = (pred_probs > boundary).astype(np.uint8)
return pred
cleanlab/internal/__init__.py
View File
cleanlab/internal/constants.py
+38
View File
@@ -0,0 +1,38 @@
FLOATING_POINT_COMPARISON = 1e-6 # floating point comparison for fuzzy equals
CLIPPING_LOWER_BOUND = 1e-6 # lower-bound clipping threshold for expected behavior
CONFIDENT_THRESHOLDS_LOWER_BOUND = (
2 * FLOATING_POINT_COMPARISON
) # lower bound imposed to clip confident thresholds from below, has to be larger than floating point comparison
TINY_VALUE = 1e-100 # very tiny value for clipping
# Object Detection Constants
EUC_FACTOR = 0.1 # Factor to control magnitude of euclidian distance. Increasing the factor makes the distances between two objects go to zero more rapidly.
MAX_ALLOWED_BOX_PRUNE = 0.97 # This is max allowed percent of boxes that are pruned before a warning is thrown given a specific threshold. Pruning too many boxes negatively affects performance.
IOU_THRESHOLD = (
0.5 # Threshold for considering the predicted box and annotated box to be overlapping
)
EPSILON = 1e-6 # Small value to prevent division by zero
ALPHA = 0.9 # Param for objectlab, weight between IoU and distance when considering similarity matrix. High alpha means considering IoU more strongly over distance
LOW_PROBABILITY_THRESHOLD = 0.5 # Param for get_label_quality_score, lowest predicted class probability threshold allowed when considering predicted boxes to identify badly located label boxes.
HIGH_PROBABILITY_THRESHOLD = 0.95 # Param for objectlab, high probability threshold for considering predicted boxes to identify overlooked and swapped label boxes
TEMPERATURE = 0.1 # Param for objectlab, temperature of the softmin function used to pool the per-box quality scores for an error subtype across all boxes into a single subtype score for the image. With a lower temperature, softmin pooling acts more like minimum pooling, alternatively it acts more like mean pooling with high temperature.
LABEL_OVERLAP_THRESHOLD = 0.95 # Param for objectlab, minimum IoU threshold for deciding when two boxes overlap used for deciding which objects have multiple conflicting annotations.
OVERLOOKED_THRESHOLD_FACTOR = 0.8 # Param for find_label_issues. Per-box label quality score threshold scale factor to determine max score for a box to be considered an overlooked issue
BADLOC_THRESHOLD_FACTOR = 0.8 # Param for find_label_issues. Per-box label quality score threshold scale factor to determine max score for a box to be considered a bad location issue
SWAP_THRESHOLD_FACTOR = 0.8 # Param for find_label_issues. Per-box label quality score threshold scale factor to determine max score for a box to be considered a swap issue
AP_SCALE_FACTOR = 0.25 # Param for find_label_issues. Scale factor for per-class precision to determine is_issue.
CUSTOM_SCORE_WEIGHT_OVERLOOKED = (
1 / 3
) # Param for get_label_quality_score, weight to determine how much to value overlooked scores over other subtypes when deciding the overall label quality score for an image.
CUSTOM_SCORE_WEIGHT_BADLOC = (
1 / 3
) # Param for get_label_quality_score, weight to determine how much to value badloc scores over other subtypes when deciding issues
CUSTOM_SCORE_WEIGHT_SWAP = (
1 / 3
) # Param for get_label_quality_score, weight to determine how much to value swap scores over other subtypes when deciding issues
MAX_CLASS_TO_SHOW = 10 # Number of classes to show in legend during the visualize method. Classes over max_class_to_show are cut off.
cleanlab/internal/label_quality_utils.py
+118
View File
@@ -0,0 +1,118 @@
"""Helper methods used internally for computing label quality scores."""
import warnings
import numpy as np
from typing import Optional
from scipy.special import xlogy
from cleanlab.count import get_confident_thresholds
def _subtract_confident_thresholds(
labels: Optional[np.ndarray],
pred_probs: np.ndarray,
multi_label: bool = False,
confident_thresholds: Optional[np.ndarray] = None,
) -> np.ndarray:
"""
Return adjusted predicted probabilities by subtracting the class confident thresholds and renormalizing.
The confident class threshold for a class j is the expected (average) "self-confidence" for class j.
The purpose of this adjustment is to handle class imbalance.
Parameters
----------
labels : np.ndarray
Labels in the same format expected by the `cleanlab.count.get_confident_thresholds()` method.
If labels is None, confident_thresholds needs to be passed in as it will not be calculated.
pred_probs : np.ndarray (shape (N, K))
Predicted-probabilities in the same format expected by the `cleanlab.count.get_confident_thresholds()` method.
confident_thresholds : np.ndarray (shape (K,))
Pre-calculated confident thresholds. If passed in, function will subtract these thresholds instead of calculating
confident_thresholds from the given labels and pred_probs.
multi_label : bool, optional
If ``True``, labels should be an iterable (e.g. list) of iterables, containing a
list of labels for each example, instead of just a single label.
The multi-label setting supports classification tasks where an example has 1 or more labels.
Example of a multi-labeled `labels` input: ``[[0,1], [1], [0,2], [0,1,2], [0], [1], ...]``.
The major difference in how this is calibrated versus single-label is that
the total number of errors considered is based on the number of labels,
not the number of examples. So, the calibrated `confident_joint` will sum
to the number of total labels.
Returns
-------
pred_probs_adj : np.ndarray (float)
Adjusted pred_probs.
"""
# Get expected (average) self-confidence for each class
# TODO: Test this for multi-label
if confident_thresholds is None:
if labels is None:
raise ValueError(
"Cannot calculate confident_thresholds without labels. Pass in either labels or already calculated "
"confident_thresholds parameter. "
)
confident_thresholds = get_confident_thresholds(labels, pred_probs, multi_label=multi_label)
# Subtract the class confident thresholds
pred_probs_adj = pred_probs - confident_thresholds
# Re-normalize by shifting data to take care of negative values from the subtraction
pred_probs_adj += confident_thresholds.max()
pred_probs_adj /= pred_probs_adj.sum(axis=1, keepdims=True)
return pred_probs_adj
def get_normalized_entropy(
pred_probs: np.ndarray, min_allowed_prob: Optional[float] = None
) -> np.ndarray:
"""Return the normalized entropy of pred_probs.
Normalized entropy is between 0 and 1. Higher values of entropy indicate higher uncertainty in the model's prediction of the correct label.
Read more about normalized entropy `on Wikipedia <https://en.wikipedia.org/wiki/Entropy_(information_theory)>`_.
Normalized entropy is used in active learning for uncertainty sampling: https://medium.com/data-science/uncertainty-sampling-cheatsheet-ec57bc067c0b
Unlike label-quality scores, entropy only depends on the model's predictions, not the given label.
Parameters
----------
pred_probs : np.ndarray (shape (N, K))
Each row of this matrix corresponds to an example x and contains the model-predicted
probabilities that x belongs to each possible class: P(label=k|x)
min_allowed_prob : float, default: None, deprecated
Minimum allowed probability value. If not `None` (default),
entries of `pred_probs` below this value will be clipped to this value.
.. deprecated:: 2.5.0
This keyword is deprecated and should be left to the default.
The entropy is well-behaved even if `pred_probs` contains zeros,
clipping is unnecessary and (slightly) changes the results.
Returns
-------
entropy : np.ndarray (shape (N, ))
Each element is the normalized entropy of the corresponding row of ``pred_probs``.
Raises
------
ValueError
An error is raised if any of the probabilities is not in the interval [0, 1].
"""
if np.any(pred_probs < 0) or np.any(pred_probs > 1):
raise ValueError("All probabilities are required to be in the interval [0, 1].")
num_classes = pred_probs.shape[1]
if min_allowed_prob is not None:
warnings.warn(
"Using `min_allowed_prob` is not necessary anymore and will be removed.",
DeprecationWarning,
)
pred_probs = np.clip(pred_probs, a_min=min_allowed_prob, a_max=None)
# Note that dividing by log(num_classes) changes the base of the log which rescales entropy to 0-1 range
return -np.sum(xlogy(pred_probs, pred_probs), axis=1) / np.log(num_classes)
cleanlab/internal/latent_algebra.py
+312
View File
@@ -0,0 +1,312 @@
"""
Contains mathematical functions relating the latent terms,
``P(given_label)``, ``P(given_label | true_label)``, ``P(true_label | given_label)``, ``P(true_label)``, etc. together.
For every function here, if the inputs are exact, the output is guaranteed to be exact.
Every function herein is the computational equivalent of a mathematical equation having a closed, exact form.
If the inputs are inexact, the error will of course propagate.
Throughout `K` denotes the number of classes in the classification task.
"""
import warnings
import numpy as np
from typing import Tuple
from cleanlab.internal.util import value_counts, clip_values, clip_noise_rates
from cleanlab.internal.constants import TINY_VALUE, CLIPPING_LOWER_BOUND
def compute_ps_py_inv_noise_matrix(
labels, noise_matrix
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""Compute ``ps := P(labels=k), py := P(true_labels=k)``, and the inverse noise matrix.
Parameters
----------
labels : np.ndarray
A discrete vector of noisy labels, i.e. some labels may be erroneous.
*Format requirements*: for dataset with `K` classes, labels must be in ``{0,1,...,K-1}``.
noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of noise_matrix sum to 1."""
ps = value_counts(labels) / float(len(labels)) # p(labels=k)
py, inverse_noise_matrix = compute_py_inv_noise_matrix(ps, noise_matrix)
return ps, py, inverse_noise_matrix
def compute_py_inv_noise_matrix(ps, noise_matrix) -> Tuple[np.ndarray, np.ndarray]:
"""Compute py := P(true_label=k), and the inverse noise matrix.
Parameters
----------
ps : np.ndarray
Array of shape ``(K, )`` or ``(1, K)``.
The fraction (prior probability) of each observed, NOISY class ``P(labels = k)``.
noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of noise_matrix sum to 1."""
# 'py' is p(true_labels=k) = noise_matrix^(-1) * p(labels=k)
# because in *vector computation*: P(label=k|true_label=k) * p(true_label=k) = P(label=k)
# The pseudo-inverse is used when noise_matrix is not invertible.
py = np.linalg.inv(noise_matrix).dot(ps)
# No class should have probability 0, so we use .000001
# Make sure valid probabilities that sum to 1.0
py = clip_values(py, low=CLIPPING_LOWER_BOUND, high=1.0, new_sum=1.0)
# All the work is done in this function (below)
return py, compute_inv_noise_matrix(py=py, noise_matrix=noise_matrix, ps=ps)
def compute_inv_noise_matrix(py, noise_matrix, *, ps=None) -> np.ndarray:
"""Compute the inverse noise matrix if py := P(true_label=k) is given.
Parameters
----------
py : np.ndarray (shape (K, 1))
The fraction (prior probability) of each TRUE class label, P(true_label = k)
noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of noise_matrix sum to 1.
ps : np.ndarray
Array of shape ``(K, 1)`` containing the fraction (prior probability) of each NOISY given label, ``P(labels = k)``.
`ps` is easily computable from py and should only be provided if it has already been precomputed, to increase code efficiency.
Examples
--------
For loop based implementation:
.. code:: python
# Number of classes
K = len(py)
# 'ps' is p(labels=k) = noise_matrix * p(true_labels=k)
# because in *vector computation*: P(label=k|true_label=k) * p(true_label=k) = P(label=k)
if ps is None:
ps = noise_matrix.dot(py)
# Estimate the (K, K) inverse noise matrix P(true_label = k_y | label = k_s)
inverse_noise_matrix = np.empty(shape=(K,K))
# k_s is the class value k of noisy label `label == k`
for k_s in range(K):
# k_y is the (guessed) class value k of true label y
for k_y in range(K):
# P(true_label|label) = P(label|y) * P(true_label) / P(labels)
inverse_noise_matrix[k_y][k_s] = noise_matrix[k_s][k_y] * \
py[k_y] / ps[k_s]
"""
joint = noise_matrix * py
ps = joint.sum(axis=1) if ps is None else ps
inverse_noise_matrix = joint.T / np.clip(ps, a_min=TINY_VALUE, a_max=None)
# Clip inverse noise rates P(true_label=k_s|true_label=k_y) into proper range [0,1)
return clip_noise_rates(inverse_noise_matrix)
def compute_noise_matrix_from_inverse(ps, inverse_noise_matrix, *, py=None) -> np.ndarray:
"""Compute the noise matrix ``P(label=k_s|true_label=k_y)``.
Parameters
----------
py : np.ndarray
Array of shape ``(K, 1)`` containing the fraction (prior probability) of each TRUE class label, ``P(true_label = k)``.
inverse_noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form P(true_label=k_y|label=k_s) representing
the estimated fraction observed examples in each class k_s, that are
mislabeled examples from every other class k_y. If None, the
inverse_noise_matrix will be computed from pred_probs and labels.
Assumes columns of inverse_noise_matrix sum to 1.
ps : np.ndarray
Array of shape ``(K, 1)`` containing the fraction (prior probability) of each observed NOISY label, P(labels = k).
`ps` is easily computable from `py` and should only be provided if it has already been precomputed, to increase code efficiency.
Returns
-------
noise_matrix : np.ndarray
Array of shape ``(K, K)``, where `K` = number of classes, whose columns sum to 1.
A conditional probability matrix of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Examples
--------
For loop based implementation:
.. code:: python
# Number of classes labels
K = len(ps)
# 'py' is p(true_label=k) = inverse_noise_matrix * p(true_label=k)
# because in *vector computation*: P(true_label=k|label=k) * p(label=k) = P(true_label=k)
if py is None:
py = inverse_noise_matrix.dot(ps)
# Estimate the (K, K) noise matrix P(labels = k_s | true_labels = k_y)
noise_matrix = np.empty(shape=(K,K))
# k_s is the class value k of noisy label `labels == k`
for k_s in range(K):
# k_y is the (guessed) class value k of true label y
for k_y in range(K):
# P(labels|y) = P(true_label|labels) * P(labels) / P(true_label)
noise_matrix[k_s][k_y] = inverse_noise_matrix[k_y][k_s] * \
ps[k_s] / py[k_y]
"""
joint = (inverse_noise_matrix * ps).T
py = joint.sum(axis=0) if py is None else py
noise_matrix = joint / np.clip(py, a_min=TINY_VALUE, a_max=None)
# Clip inverse noise rates P(true_label=k_y|true_label=k_s) into proper range [0,1)
return clip_noise_rates(noise_matrix)
def compute_py(
ps, noise_matrix, inverse_noise_matrix, *, py_method="cnt", true_labels_class_counts=None
) -> np.ndarray:
"""Compute ``py := P(true_labels=k)`` from ``ps := P(labels=k)``, `noise_matrix`, and
`inverse_noise_matrix`.
This method is ** ROBUST ** when ``py_method = 'cnt'``
It may work well even when the noise matrices are estimated
poorly by using the diagonals of the matrices
instead of all the probabilities in the entire matrix.
Parameters
----------
ps : np.ndarray
Array of shape ``(K, )`` or ``(1, K)`` containing the fraction (prior probability) of each observed, noisy label, P(labels = k)
noise_matrix : np.ndarray
A conditional probability matrix ( of shape ``(K, K)``) of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of noise_matrix sum to 1.
inverse_noise_matrix : np.ndarray of shape (K, K), K = number of classes
A conditional probability matrix ( of shape ``(K, K)``) of the form ``P(true_label=k_y|label=k_s)`` representing
the estimated fraction observed examples in each class `k_s`, that are
mislabeled examples from every other class `k_y`. If ``None``, the
inverse_noise_matrix will be computed from `pred_probs` and `labels`.
Assumes columns of `inverse_noise_matrix` sum to 1.
py_method : str (Options: ["cnt", "eqn", "marginal", "marginal_ps"])
How to compute the latent prior ``p(true_label=k)``. Default is "cnt" as it often
works well even when the noise matrices are estimated poorly by using
the matrix diagonals instead of all the probabilities.
true_labels_class_counts : np.ndarray
Array of shape ``(K, )`` or ``(1, K)`` containing the marginal counts of the confident joint
(like ``cj.sum(axis = 0)``).
Returns
-------
py : np.ndarray
Array of shape ``(K, )`` or ``(1, K)``.
The fraction (prior probability) of each TRUE class label, ``P(true_label = k)``."""
if len(np.shape(ps)) > 2 or (len(np.shape(ps)) == 2 and np.shape(ps)[0] != 1):
w = "Input parameter np.ndarray ps has shape " + str(np.shape(ps))
w += ", but shape should be (K, ) or (1, K)"
warnings.warn(w)
if py_method == "marginal" and true_labels_class_counts is None:
msg = (
'py_method == "marginal" requires true_labels_class_counts, '
"but true_labels_class_counts is None. "
)
msg += " Provide parameter true_labels_class_counts."
raise ValueError(msg)
if py_method == "cnt":
# Computing py this way avoids dividing by zero noise rates.
# More robust bc error est_p(true_label|labels) / est_p(labels|y) ~ p(true_label|labels) / p(labels|y)
py = (
inverse_noise_matrix.diagonal()
/ np.clip(noise_matrix.diagonal(), a_min=TINY_VALUE, a_max=None)
* ps
)
# Equivalently: py = (true_labels_class_counts / labels_class_counts) * ps
elif py_method == "eqn":
py = np.linalg.inv(noise_matrix).dot(ps)
elif py_method == "marginal":
py = true_labels_class_counts / np.clip(
float(sum(true_labels_class_counts)), a_min=TINY_VALUE, a_max=None
)
elif py_method == "marginal_ps":
py = np.dot(inverse_noise_matrix, ps)
else:
err = "py_method {}".format(py_method)
err += " should be in [cnt, eqn, marginal, marginal_ps]"
raise ValueError(err)
# Clip py (0,1), s.t. no class should have prob 0, hence 1e-6
py = clip_values(py, low=CLIPPING_LOWER_BOUND, high=1.0, new_sum=1.0)
return py
def compute_pyx(pred_probs, noise_matrix, inverse_noise_matrix):
"""Compute ``pyx := P(true_label=k|x)`` from ``pred_probs := P(label=k|x)``, `noise_matrix` and
`inverse_noise_matrix`.
This method is ROBUST - meaning it works well even when the
noise matrices are estimated poorly by only using the diagonals of the
matrices which tend to be easy to estimate correctly.
Parameters
----------
pred_probs : np.ndarray
``P(label=k|x)`` is a ``(N x K)`` matrix with K model-predicted probabilities.
Each row of this matrix corresponds to an example `x` and contains the model-predicted
probabilities that `x` belongs to each possible class.
The columns must be ordered such that these probabilities correspond to class 0,1,2,...
`pred_probs` should have been computed using 3 (or higher) fold cross-validation.
noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form ``P(label=k_s|true_label=k_y)`` containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of `noise_matrix` sum to 1.
inverse_noise_matrix : np.ndarray
A conditional probability matrix (of shape ``(K, K)``) of the form ``P(true_label=k_y|label=k_s)`` representing
the estimated fraction observed examples in each class `k_s`, that are
mislabeled examples from every other class `k_y`. If None, the
inverse_noise_matrix will be computed from `pred_probs` and `labels`.
Assumes columns of `inverse_noise_matrix` sum to 1.
Returns
-------
pyx : np.ndarray
``P(true_label=k|x)`` is a ``(N, K)`` matrix of model-predicted probabilities.
Each row of this matrix corresponds to an example `x` and contains the model-predicted
probabilities that `x` belongs to each possible class.
The columns must be ordered such that these probabilities correspond to class 0,1,2,...
`pred_probs` should have been computed using 3 (or higher) fold cross-validation."""
if len(np.shape(pred_probs)) != 2:
raise ValueError(
"Input parameter np.ndarray 'pred_probs' has shape "
+ str(np.shape(pred_probs))
+ ", but shape should be (N, K)"
)
pyx = (
pred_probs
* inverse_noise_matrix.diagonal()
/ np.clip(noise_matrix.diagonal(), a_min=TINY_VALUE, a_max=None)
)
# Make sure valid probabilities that sum to 1.0
return np.apply_along_axis(
func1d=clip_values, axis=1, arr=pyx, **{"low": 0.0, "high": 1.0, "new_sum": 1.0}
)
cleanlab/internal/multiannotator_utils.py
+352
View File
@@ -0,0 +1,352 @@
"""
Helper methods used internally in cleanlab.multiannotator
"""
import warnings
from typing import Optional, Tuple
import numpy as np
import pandas as pd
from cleanlab.internal.numerics import softmax
from cleanlab.internal.util import get_num_classes, value_counts
from cleanlab.internal.validation import assert_valid_class_labels
from cleanlab.typing import LabelLike
SMALL_CONST = 1e-30
def assert_valid_inputs_multiannotator(
labels_multiannotator: np.ndarray,
pred_probs: Optional[np.ndarray] = None,
ensemble: bool = False,
allow_single_label: bool = False,
annotator_ids: Optional[pd.Index] = None,
) -> None:
"""Validate format of multi-annotator labels"""
# Check that labels_multiannotator is a 2D array
if labels_multiannotator.ndim != 2:
raise ValueError(
"labels_multiannotator must be a 2D array or dataframe, "
"each row represents an example and each column represents an annotator."
)
# Raise error if labels are not formatted properly
if any([isinstance(label, str) for label in labels_multiannotator.ravel()]):
raise ValueError(
"Labels cannot be strings, they must be zero-indexed integers corresponding to class indices."
)
# Raise error if labels_multiannotator has NaN rows
nan_row_mask = np.isnan(labels_multiannotator).all(axis=1)
if nan_row_mask.any():
nan_rows = list(np.where(nan_row_mask)[0])
raise ValueError(
"labels_multiannotator cannot have rows with all NaN, each example must have at least one label.\n"
f"Examples {nan_rows} do not have any labels."
)
# Raise error if labels_multiannotator has NaN columns
nan_col_mask = np.isnan(labels_multiannotator).all(axis=0)
if nan_col_mask.any():
if annotator_ids is not None:
nan_columns = list(annotator_ids[np.where(nan_col_mask)[0]])
else:
nan_columns = list(np.where(nan_col_mask)[0])
raise ValueError(
"labels_multiannotator cannot have columns with all NaN, each annotator must annotator at least one example.\n"
f"Annotators {nan_columns} did not label any examples."
)
if not allow_single_label:
# Raise error if labels_multiannotator has <= 1 column
if labels_multiannotator.shape[1] <= 1:
raise ValueError(
"labels_multiannotator must have more than one column.\n"
"If there is only one annotator, use cleanlab.rank.get_label_quality_scores instead"
)
# Raise error if labels_multiannotator only has 1 label per example
if (np.sum(~np.isnan(labels_multiannotator), axis=1) == 1).all():
raise ValueError(
"Each example only has one label, collapse the labels into a 1-D array and use "
"cleanlab.rank.get_label_quality_scores instead"
)
# Raise warning if no examples with 2 or more annotators agree
# TODO: might shift this later in the code to avoid extra compute
has_agreement = np.zeros(labels_multiannotator.shape[0], dtype=bool)
for i in np.unique(labels_multiannotator):
has_agreement |= (labels_multiannotator == i).sum(axis=1) > 1
if not has_agreement.any():
warnings.warn("Annotators do not agree on any example. Check input data.")
# Check labels
all_labels_flatten = labels_multiannotator.ravel()
all_labels_flatten = all_labels_flatten[~np.isnan(all_labels_flatten)]
assert_valid_class_labels(all_labels_flatten, allow_one_class=True)
# Raise error if number of classes in labels_multiannoator does not match number of classes in pred_probs
if pred_probs is not None:
if not isinstance(pred_probs, np.ndarray):
raise TypeError("pred_probs must be a numpy array.")
if ensemble:
if pred_probs.ndim != 3:
error_message = "pred_probs must be a 3d array."
if pred_probs.ndim == 2:
error_message += " If you have a 2d pred_probs array, use the non-ensemble version of this function."
raise ValueError(error_message)
if pred_probs.shape[1] != len(labels_multiannotator):
raise ValueError("each pred_probs and labels_multiannotator must have same length.")
num_classes = pred_probs.shape[2]
else:
if pred_probs.ndim != 2:
error_message = "pred_probs must be a 2d array."
if pred_probs.ndim == 3:
error_message += " If you have a 3d pred_probs array, use the ensemble version of this function."
raise ValueError(error_message)
if len(pred_probs) != len(labels_multiannotator):
raise ValueError("pred_probs and labels_multiannotator must have same length.")
num_classes = pred_probs.shape[1]
highest_class = np.nanmax(labels_multiannotator) + 1
# this allows for missing labels, but not missing columns in pred_probs
if num_classes < highest_class:
raise ValueError(
f"pred_probs must have at least {int(highest_class)} columns based on the largest class label "
"which appears in labels_multiannotator. Perhaps some rarely-annotated classes were lost while "
"establishing consensus labels used to train your classifier."
)
def assert_valid_pred_probs(
pred_probs: Optional[np.ndarray] = None,
pred_probs_unlabeled: Optional[np.ndarray] = None,
ensemble: bool = False,
):
"""Validate format of pred_probs for multiannotator active learning functions"""
if pred_probs is None and pred_probs_unlabeled is None:
raise ValueError(
"pred_probs and pred_probs_unlabeled cannot both be None, specify at least one of the two."
)
if ensemble:
if pred_probs is not None:
if not isinstance(pred_probs, np.ndarray):
raise TypeError("pred_probs must be a numpy array.")
if pred_probs.ndim != 3:
error_message = "pred_probs must be a 3d array."
if pred_probs.ndim == 2: # pragma: no cover
error_message += " If you have a 2d pred_probs array (ie. only one predictor), use the non-ensemble version of this function."
raise ValueError(error_message)
if pred_probs_unlabeled is not None:
if not isinstance(pred_probs_unlabeled, np.ndarray):
raise TypeError("pred_probs_unlabeled must be a numpy array.")
if pred_probs_unlabeled.ndim != 3:
error_message = "pred_probs_unlabeled must be a 3d array."
if pred_probs_unlabeled.ndim == 2: # pragma: no cover
error_message += " If you have a 2d pred_probs_unlabeled array, use the non-ensemble version of this function."
raise ValueError(error_message)
if pred_probs is not None and pred_probs_unlabeled is not None:
if pred_probs.shape[2] != pred_probs_unlabeled.shape[2]:
raise ValueError(
"pred_probs and pred_probs_unlabeled must have the same number of classes"
)
else:
if pred_probs is not None:
if not isinstance(pred_probs, np.ndarray):
raise TypeError("pred_probs must be a numpy array.")
if pred_probs.ndim != 2:
error_message = "pred_probs must be a 2d array."
if pred_probs.ndim == 3: # pragma: no cover
error_message += " If you have a 3d pred_probs array, use the ensemble version of this function."
raise ValueError(error_message)
if pred_probs_unlabeled is not None:
if not isinstance(pred_probs_unlabeled, np.ndarray):
raise TypeError("pred_probs_unlabeled must be a numpy array.")
if pred_probs_unlabeled.ndim != 2:
error_message = "pred_probs_unlabeled must be a 2d array."
if pred_probs_unlabeled.ndim == 3: # pragma: no cover
error_message += " If you have a 3d pred_probs_unlabeled array, use the non-ensemble version of this function."
raise ValueError(error_message)
if pred_probs is not None and pred_probs_unlabeled is not None:
if pred_probs.shape[1] != pred_probs_unlabeled.shape[1]:
raise ValueError(
"pred_probs and pred_probs_unlabeled must have the same number of classes"
)
def format_multiannotator_labels(labels: LabelLike) -> Tuple[pd.DataFrame, dict]:
"""Takes an array of labels and formats it such that labels are in the set ``0, 1, ..., K-1``,
where ``K`` is the number of classes. The labels are assigned based on lexicographic order.
Returns
-------
formatted_labels
Returns pd.DataFrame of shape ``(N,M)``. The return labels will be properly formatted and can be passed to
cleanlab.multiannotator functions.
mapping
A dictionary showing the mapping of new to old labels, such that ``mapping[k]`` returns the name of the k-th class.
"""
if isinstance(labels, pd.DataFrame):
np_labels = labels.values
elif isinstance(labels, np.ndarray):
np_labels = labels
else:
raise TypeError("labels must be 2D numpy array or pandas DataFrame")
unique_labels = pd.unique(np_labels.ravel())
try:
unique_labels = unique_labels[~np.isnan(unique_labels)]
unique_labels.sort()
except TypeError: # np.unique / np.sort cannot handle string values or pd.NA types
nan_mask = np.array([(l is np.nan) or (l is pd.NA) or (l == "nan") for l in unique_labels])
unique_labels = unique_labels[~nan_mask]
unique_labels.sort()
# convert float labels (that arose because np.nan is float type) to int
if unique_labels.dtype == "float":
unique_labels = unique_labels.astype("int")
label_map = {label: i for i, label in enumerate(unique_labels)}
inverse_map = {i: label for label, i in label_map.items()}
if isinstance(labels, np.ndarray):
labels = pd.DataFrame(labels)
formatted_labels = labels.replace(label_map)
return formatted_labels, inverse_map
def check_consensus_label_classes(
labels_multiannotator: np.ndarray,
consensus_label: np.ndarray,
consensus_method: str,
) -> None:
"""Check if any classes no longer appear in the set of consensus labels (established using the consensus_method stated)"""
unique_ma_labels = np.unique(labels_multiannotator)
unique_ma_labels = unique_ma_labels[~np.isnan(unique_ma_labels)]
labels_set_difference = set(unique_ma_labels) - set(consensus_label)
if len(labels_set_difference) > 0:
print(
"CAUTION: Number of unique classes has been reduced from the original data when establishing consensus labels "
f"using consensus method '{consensus_method}', likely due to some classes being rarely annotated. "
"If training a classifier on these consensus labels, it will never see any of the omitted classes unless you "
"manually replace some of the consensus labels.\n"
f"Classes in the original data but not in consensus labels: {list(map(int, labels_set_difference))}"
)
def compute_soft_cross_entropy(
labels_multiannotator: np.ndarray,
pred_probs: np.ndarray,
) -> float:
"""Compute soft cross entropy between the annotators' empirical label distribution and model pred_probs"""
num_classes = get_num_classes(pred_probs=pred_probs)
empirical_label_distribution = np.full((len(labels_multiannotator), num_classes), np.nan)
for i, labels in enumerate(labels_multiannotator):
labels_subset = labels[~np.isnan(labels)]
empirical_label_distribution[i, :] = value_counts(
labels_subset, num_classes=num_classes
) / len(labels_subset)
clipped_pred_probs = np.clip(pred_probs, a_min=SMALL_CONST, a_max=None)
soft_cross_entropy = -np.sum(
empirical_label_distribution * np.log(clipped_pred_probs), axis=1
) / np.log(num_classes)
return soft_cross_entropy
def find_best_temp_scaler(
labels_multiannotator: np.ndarray,
pred_probs: np.ndarray,
coarse_search_range: list = [0.1, 0.2, 0.5, 0.8, 1, 2, 3, 5, 8],
fine_search_size: int = 4,
) -> float:
"""Find the best temperature scaling factor that minimizes the soft cross entropy between the annotators' empirical label distribution
and model pred_probs"""
soft_cross_entropy_coarse = np.full(len(coarse_search_range), np.nan)
log_pred_probs = np.log(
pred_probs, where=pred_probs > 0, out=np.full(pred_probs.shape, -np.inf)
)
for i, curr_temp in enumerate(coarse_search_range):
scaled_pred_probs = softmax(log_pred_probs, temperature=curr_temp, axis=1, shift=False)
soft_cross_entropy_coarse[i] = np.mean(
compute_soft_cross_entropy(labels_multiannotator, scaled_pred_probs)
)
min_entropy_ind = np.argmin(soft_cross_entropy_coarse)
fine_search_range = _set_fine_search_range(
coarse_search_range, fine_search_size, min_entropy_ind
)
soft_cross_entropy_fine = np.full(len(fine_search_range), np.nan)
for i, curr_temp in enumerate(fine_search_range):
scaled_pred_probs = softmax(log_pred_probs, temperature=curr_temp, axis=1, shift=False)
soft_cross_entropy_fine[i] = np.mean(
compute_soft_cross_entropy(labels_multiannotator, scaled_pred_probs)
)
best_temp = fine_search_range[np.argmin(soft_cross_entropy_fine)]
return best_temp
def _set_fine_search_range(
coarse_search_range: list, fine_search_size: int, min_entropy_ind: np.intp
) -> np.ndarray:
fine_search_range = np.array([])
if min_entropy_ind != 0:
fine_search_range = np.append(
np.linspace(
coarse_search_range[min_entropy_ind - 1],
coarse_search_range[min_entropy_ind],
fine_search_size,
endpoint=False,
),
fine_search_range,
)
if min_entropy_ind != len(coarse_search_range) - 1:
fine_search_range = np.append(
fine_search_range,
np.linspace(
coarse_search_range[min_entropy_ind],
coarse_search_range[min_entropy_ind + 1],
fine_search_size + 1,
endpoint=True,
),
)
return fine_search_range
def temp_scale_pred_probs(
pred_probs: np.ndarray,
temp: float,
) -> np.ndarray:
"""Scales pred_probs by the given temperature factor. Temperature of <1 will sharpen the pred_probs while temperatures of >1 will smoothen it."""
# clip pred_probs to prevent taking log of 0
pred_probs = np.clip(pred_probs, a_min=SMALL_CONST, a_max=None)
pred_probs = pred_probs / np.sum(pred_probs, axis=1)[:, np.newaxis]
# apply temperate scale
scaled_pred_probs = softmax(np.log(pred_probs), temperature=temp, axis=1, shift=False)
scaled_pred_probs = (
scaled_pred_probs / np.sum(scaled_pred_probs, axis=1)[:, np.newaxis]
) # normalize
return scaled_pred_probs
cleanlab/internal/multilabel_scorer.py
+652
View File
@@ -0,0 +1,652 @@
"""
Helper classes and functions used internally to compute label quality scores in multi-label classification.
"""
from enum import Enum
from typing import Callable, Dict, Optional, Union
import numpy as np
from sklearn.model_selection import cross_val_predict
from cleanlab.internal.label_quality_utils import _subtract_confident_thresholds
from cleanlab.internal.multilabel_utils import _is_multilabel, stack_complement
from cleanlab.internal.numerics import softmax
from cleanlab.rank import (
get_confidence_weighted_entropy_for_each_label,
get_normalized_margin_for_each_label,
get_self_confidence_for_each_label,
)
class _Wrapper:
"""Helper class for wrapping callable functions as attributes of an Enum instead of
setting them as methods of the Enum class.
This class is only intended to be used internally for the ClassLabelScorer or
other cases where functions are used for enumeration values.
"""
def __init__(self, f: Callable) -> None:
self.f = f
def __call__(self, *args, **kwargs):
return self.f(*args, **kwargs)
def __repr__(self):
return self.f.__name__
class ClassLabelScorer(Enum):
"""Enum for the different methods to compute label quality scores."""
SELF_CONFIDENCE = _Wrapper(get_self_confidence_for_each_label)
"""Returns the self-confidence label-quality score for each datapoint.
See also
--------
cleanlab.rank.get_self_confidence_for_each_label
"""
NORMALIZED_MARGIN = _Wrapper(get_normalized_margin_for_each_label)
"""Returns the "normalized margin" label-quality score for each datapoint.
See also
--------
cleanlab.rank.get_normalized_margin_for_each_label
"""
CONFIDENCE_WEIGHTED_ENTROPY = _Wrapper(get_confidence_weighted_entropy_for_each_label)
"""Returns the "confidence weighted entropy" label-quality score for each datapoint.
See also
--------
cleanlab.rank.get_confidence_weighted_entropy_for_each_label
"""
def __call__(self, labels: np.ndarray, pred_probs: np.ndarray, **kwargs) -> np.ndarray:
"""Returns the label-quality scores for each datapoint based on the given labels and predicted probabilities.
See the documentation for each method for more details.
Example
-------
>>> import numpy as np
>>> from cleanlab.internal.multilabel_scorer import ClassLabelScorer
>>> labels = np.array([0, 0, 0, 1, 1, 1])
>>> pred_probs = np.array([
... [0.9, 0.1],
... [0.8, 0.2],
... [0.7, 0.3],
... [0.2, 0.8],
... [0.75, 0.25],
... [0.1, 0.9],
... ])
>>> ClassLabelScorer.SELF_CONFIDENCE(labels, pred_probs)
array([0.9 , 0.8 , 0.7 , 0.8 , 0.25, 0.9 ])
"""
pred_probs = self._adjust_pred_probs(labels, pred_probs, **kwargs)
return self.value(labels, pred_probs)
def _adjust_pred_probs(
self, labels: np.ndarray, pred_probs: np.ndarray, **kwargs
) -> np.ndarray:
"""Returns adjusted predicted probabilities by subtracting the class confident thresholds and renormalizing.
This is used to adjust the predicted probabilities for the SELF_CONFIDENCE and NORMALIZED_MARGIN methods.
"""
if kwargs.get("adjust_pred_probs", False) is True:
if self == ClassLabelScorer.CONFIDENCE_WEIGHTED_ENTROPY:
raise ValueError(f"adjust_pred_probs is not currently supported for {self}.")
pred_probs = _subtract_confident_thresholds(labels, pred_probs)
return pred_probs
@classmethod
def from_str(cls, method: str) -> "ClassLabelScorer":
"""Constructs an instance of the ClassLabelScorer enum based on the given method name.
Parameters
----------
method:
The name of the scoring method to use.
Returns
-------
scorer:
An instance of the ClassLabelScorer enum.
Raises
------
ValueError:
If the given method name is not a valid method name.
It must be one of the following: "self_confidence", "normalized_margin", or "confidence_weighted_entropy".
Example
-------
>>> from cleanlab.internal.multilabel_scorer import ClassLabelScorer
>>> ClassLabelScorer.from_str("self_confidence")
<ClassLabelScorer.SELF_CONFIDENCE: get_self_confidence_for_each_label>
"""
try:
return cls[method.upper()]
except KeyError:
raise ValueError(f"Invalid method name: {method}")
def exponential_moving_average(
s: np.ndarray,
*,
alpha: Optional[float] = None,
axis: int = 1,
**_,
) -> np.ndarray:
r"""Exponential moving average (EMA) score aggregation function.
For a score vector s = (s_1, ..., s_K) with K scores, the values
are sorted in *descending* order and the exponential moving average
of the last score is calculated, denoted as EMA_K according to the
note below.
Note
----
The recursive formula for the EMA at step :math:`t = 2, ..., K` is:
.. math::
\text{EMA}_t = \alpha \cdot s_t + (1 - \alpha) \cdot \text{EMA}_{t-1}, \qquad 0 \leq \alpha \leq 1
We set :math:`\text{EMA}_1 = s_1` as the largest score in the sorted vector s.
:math:`\alpha` is the "forgetting factor" that gives more weight to the
most recent scores, and successively less weight to the previous scores.
Parameters
----------
s :
Scores to be transformed.
alpha :
Discount factor that determines the weight of the previous EMA score.
Higher alpha means that the previous EMA score has a lower weight while
the current score has a higher weight.
Its value must be in the interval [0, 1].
If alpha is None, it is set to 2 / (K + 1) where K is the number of scores.
axis :
Axis along which the scores are sorted.
Returns
-------
s_ema :
Exponential moving average score.
Examples
--------
>>> from cleanlab.internal.multilabel_scorer import exponential_moving_average
>>> import numpy as np
>>> s = np.array([[0.1, 0.2, 0.3]])
>>> exponential_moving_average(s, alpha=0.5)
np.array([0.175])
"""
K = s.shape[1]
s_sorted = np.fliplr(np.sort(s, axis=axis))
if alpha is None:
# One conventional choice for alpha is 2/(K + 1), where K is the number of periods in the moving average.
alpha = float(2 / (K + 1))
if not (0 <= alpha <= 1):
raise ValueError(f"alpha must be in the interval [0, 1], got {alpha}")
s_T = s_sorted.T
s_ema, s_next = s_T[0], s_T[1:]
for s_i in s_next:
s_ema = alpha * s_i + (1 - alpha) * s_ema
return s_ema
def softmin(
s: np.ndarray,
*,
temperature: float = 0.1,
axis: int = 1,
**_,
) -> np.ndarray:
"""Softmin score aggregation function.
Parameters
----------
s :
Input array.
temperature :
Temperature parameter. Too small values may cause numerical underflow and NaN scores.
axis :
Axis along which to apply the function.
Returns
-------
Softmin score.
"""
return np.einsum(
"ij,ij->i", s, softmax(x=1 - s, temperature=temperature, axis=axis, shift=True)
)
class Aggregator:
"""Helper class for aggregating the label quality scores for each class into a single score for each datapoint.
Parameters
----------
method:
The method to compute the label quality scores for each class.
If passed as a callable, your function should take in a 1D array of K scores and return a single aggregated score.
See `~cleanlab.internal.multilabel_scorer.exponential_moving_average` for an example of such a function.
Alternatively, this can be a str value to specify a built-in function, possible values are the keys of the ``Aggregator``'s `possible_methods` attribute.
kwargs:
Additional keyword arguments to pass to the aggregation function when it is called.
"""
possible_methods: Dict[str, Callable[..., np.ndarray]] = {
"exponential_moving_average": exponential_moving_average,
"softmin": softmin,
}
def __init__(self, method: Union[str, Callable], **kwargs):
if isinstance(method, str): # convert to callable
if method in self.possible_methods:
method = self.possible_methods[method]
else:
raise ValueError(
f"Invalid aggregation method specified: '{method}', must be one of the following: {list(self.possible_methods.keys())}"
)
self._validate_method(method)
self.method = method
self.kwargs = kwargs
@staticmethod
def _validate_method(method) -> None:
if not callable(method):
raise TypeError(f"Expected callable method, got {type(method)}")
@staticmethod
def _validate_scores(scores: np.ndarray) -> None:
if not (isinstance(scores, np.ndarray) and scores.ndim == 2):
raise ValueError(
f"Expected 2D array for scores, got {type(scores)} with shape {scores.shape}"
)
def __call__(self, scores: np.ndarray, **kwargs) -> np.ndarray:
"""Returns the label quality scores for each datapoint based on the given label quality scores for each class.
Parameters
----------
scores:
The label quality scores for each class.
Returns
-------
aggregated_scores:
A single label quality score for each datapoint.
"""
self._validate_scores(scores)
kwargs["axis"] = 1
updated_kwargs = {**self.kwargs, **kwargs}
return self.method(scores, **updated_kwargs)
def __repr__(self):
return f"Aggregator(method={self.method.__name__}, kwargs={self.kwargs})"
class MultilabelScorer:
"""Aggregates label quality scores across different classes to produce one score per example in multi-label classification tasks.
Parameters
----------
base_scorer:
The method to compute the label quality scores for each class.
See the documentation for the ClassLabelScorer enum for more details.
aggregator:
The method to aggregate the label quality scores for each class into a single score for each datapoint.
Defaults to the EMA (exponential moving average) aggregator with forgetting factor ``alpha=0.8``.
See the documentation for the Aggregator class for more details.
See also
--------
exponential_moving_average
strict:
Flag for performing strict validation of the input data.
"""
def __init__(
self,
base_scorer: ClassLabelScorer = ClassLabelScorer.SELF_CONFIDENCE,
aggregator: Union[Aggregator, Callable] = Aggregator(exponential_moving_average, alpha=0.8),
*,
strict: bool = True,
):
self.base_scorer = base_scorer
if not isinstance(aggregator, Aggregator):
self.aggregator = Aggregator(aggregator)
else:
self.aggregator = aggregator
self.strict = strict
def __call__(
self,
labels: np.ndarray,
pred_probs: np.ndarray,
base_scorer_kwargs: Optional[dict] = None,
**aggregator_kwargs,
) -> np.ndarray:
"""
Computes a quality score for each label in a multi-label classification problem
based on out-of-sample predicted probabilities.
For each example, the label quality scores for each class are aggregated into a single overall label quality score.
Parameters
----------
labels:
A 2D array of shape (n_samples, n_labels) with binary labels.
pred_probs:
A 2D array of shape (n_samples, n_labels) with predicted probabilities.
kwargs:
Additional keyword arguments to pass to the base_scorer and the aggregator.
base_scorer_kwargs:
Keyword arguments to pass to the base_scorer
aggregator_kwargs:
Additional keyword arguments to pass to the aggregator.
Returns
-------
scores:
A 1D array of shape (n_samples,) with the quality scores for each datapoint.
Examples
--------
>>> from cleanlab.internal.multilabel_scorer import MultilabelScorer, ClassLabelScorer
>>> import numpy as np
>>> labels = np.array([[0, 1, 0], [1, 0, 1]])
>>> pred_probs = np.array([[0.1, 0.9, 0.1], [0.4, 0.1, 0.9]])
>>> scorer = MultilabelScorer()
>>> scores = scorer(labels, pred_probs)
>>> scores
array([0.9, 0.5])
>>> scorer = MultilabelScorer(
... base_scorer = ClassLabelScorer.NORMALIZED_MARGIN,
... aggregator = np.min, # Use the "worst" label quality score for each example.
... )
>>> scores = scorer(labels, pred_probs)
>>> scores
array([0.9, 0.4])
"""
if self.strict:
self._validate_labels_and_pred_probs(labels, pred_probs)
scores = self.get_class_label_quality_scores(labels, pred_probs, base_scorer_kwargs)
return self.aggregate(scores, **aggregator_kwargs)
def aggregate(
self,
class_label_quality_scores: np.ndarray,
**kwargs,
) -> np.ndarray:
"""Aggregates the label quality scores for each class into a single overall label quality score for each example.
Parameters
----------
class_label_quality_scores:
A 2D array of shape (n_samples, n_labels) with the label quality scores for each class.
See also
--------
get_class_label_quality_scores
kwargs:
Additional keyword arguments to pass to the aggregator.
Returns
-------
scores:
A 1D array of shape (n_samples,) with the quality scores for each datapoint.
Examples
--------
>>> from cleanlab.internal.multilabel_scorer import MultilabelScorer
>>> import numpy as np
>>> class_label_quality_scores = np.array([[0.9, 0.9, 0.3],[0.4, 0.9, 0.6]])
>>> scorer = MultilabelScorer() # Use the default aggregator (exponential moving average) with default parameters.
>>> scores = scorer.aggregate(class_label_quality_scores)
>>> scores
array([0.42, 0.452])
>>> new_scores = scorer.aggregate(class_label_quality_scores, alpha=0.5) # Use the default aggregator with custom parameters.
>>> new_scores
array([0.6, 0.575])
Warning
-------
Make sure that keyword arguments correspond to the aggregation function used.
I.e. the ``exponential_moving_average`` function supports an ``alpha`` keyword argument, but ``np.min`` does not.
"""
return self.aggregator(class_label_quality_scores, **kwargs)
def get_class_label_quality_scores(
self,
labels: np.ndarray,
pred_probs: np.ndarray,
base_scorer_kwargs: Optional[dict] = None,
) -> np.ndarray:
"""Computes separate label quality scores for each class.
Parameters
----------
labels:
A 2D array of shape (n_samples, n_labels) with binary labels.
pred_probs:
A 2D array of shape (n_samples, n_labels) with predicted probabilities.
base_scorer_kwargs:
Keyword arguments to pass to the base scoring-function.
Returns
-------
class_label_quality_scores:
A 2D array of shape (n_samples, n_labels) with the quality scores for each label.
Examples
--------
>>> from cleanlab.internal.multilabel_scorer import MultilabelScorer
>>> import numpy as np
>>> labels = np.array([[0, 1, 0], [1, 0, 1]])
>>> pred_probs = np.array([[0.1, 0.9, 0.7], [0.4, 0.1, 0.6]])
>>> scorer = MultilabelScorer() # Use the default base scorer (SELF_CONFIDENCE)
>>> class_label_quality_scores = scorer.get_label_quality_scores_per_class(labels, pred_probs)
>>> class_label_quality_scores
array([[0.9, 0.9, 0.3],
[0.4, 0.9, 0.6]])
"""
class_label_quality_scores = np.zeros(shape=labels.shape)
if base_scorer_kwargs is None:
base_scorer_kwargs = {}
for i, (label_i, pred_prob_i) in enumerate(zip(labels.T, pred_probs.T)):
pred_prob_i_two_columns = stack_complement(pred_prob_i)
class_label_quality_scores[:, i] = self.base_scorer(
label_i, pred_prob_i_two_columns, **base_scorer_kwargs
)
return class_label_quality_scores
@staticmethod
def _validate_labels_and_pred_probs(labels: np.ndarray, pred_probs: np.ndarray) -> None:
"""
Checks that (multi-)labels are in the proper binary indicator format and that
they are compatible with the predicted probabilities.
"""
# Only allow dense matrices for labels for now
if not isinstance(labels, np.ndarray):
raise TypeError("Labels must be a numpy array.")
if not _is_multilabel(labels):
raise ValueError("Labels must be in multi-label format.")
if labels.shape != pred_probs.shape:
raise ValueError("Labels and predicted probabilities must have the same shape.")
def get_label_quality_scores(
labels,
pred_probs,
*,
method: MultilabelScorer = MultilabelScorer(),
base_scorer_kwargs: Optional[dict] = None,
**aggregator_kwargs,
) -> np.ndarray:
"""Computes a quality score for each label in a multi-label classification problem
based on out-of-sample predicted probabilities.
Parameters
----------
labels:
A 2D array of shape (N, K) with binary labels.
pred_probs:
A 2D array of shape (N, K) with predicted probabilities.
method:
A scoring+aggregation method for computing the label quality scores of examples in a multi-label classification setting.
base_scorer_kwargs:
Keyword arguments to pass to the class-label scorer.
aggregator_kwargs:
Additional keyword arguments to pass to the aggregator.
Returns
-------
scores:
A 1D array of shape (N,) with the quality scores for each datapoint.
Examples
--------
>>> import cleanlab.internal.multilabel_scorer as ml_scorer
>>> import numpy as np
>>> labels = np.array([[0, 1, 0], [1, 0, 1]])
>>> pred_probs = np.array([[0.1, 0.9, 0.1], [0.4, 0.1, 0.9]])
>>> scores = ml_scorer.get_label_quality_scores(labels, pred_probs, method=ml_scorer.MultilabelScorer())
>>> scores
array([0.9, 0.5])
See also
--------
MultilabelScorer:
See the documentation for the MultilabelScorer class for more examples of scoring methods and aggregation methods.
"""
return method(labels, pred_probs, base_scorer_kwargs=base_scorer_kwargs, **aggregator_kwargs)
# Probabilities
def multilabel_py(y: np.ndarray) -> np.ndarray:
"""Compute the prior probability of each label in a multi-label classification problem.
Parameters
----------
y :
A 2d array of binarized multi-labels of shape (N, K) where N is the number of samples and K is the number of classes.
Returns
-------
py :
A 2d array of prior probabilities of shape (K,2) where the first column is the probability of the label being 0
and the second column is the probability of the label being 1 for each class.
Examples
--------
>>> from cleanlab.internal.multilabel_scorer import multilabel_py
>>> import numpy as np
>>> y = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
>>> multilabel_py(y)
array([[0.5, 0.5],
[0.5, 0.5]])
>>> y = np.array([[0, 0], [0, 1], [1, 0], [1, 0], [1, 0]])
>>> multilabel_py(y)
array([[0.4, 0.6],
[0.8, 0.2]])
"""
N, _ = y.shape
fraction_0 = np.sum(y == 0, axis=0) / N
fraction_1 = 1 - fraction_0
py = np.column_stack((fraction_0, fraction_1))
return py
# Cross-validation helpers
def _get_split_generator(labels, cv):
_, multilabel_ids = np.unique(labels, axis=0, return_inverse=True)
split_generator = cv.split(X=multilabel_ids, y=multilabel_ids)
return split_generator
def get_cross_validated_multilabel_pred_probs(X, labels: np.ndarray, *, clf, cv) -> np.ndarray:
"""Get predicted probabilities for a multi-label classifier via cross-validation.
Note
----
The labels are reformatted to a "multi-class" format internally to support a wider range of cross-validation strategies.
If you have a multi-label dataset with `K` classes, the labels are reformatted to a "multi-class" format with up to `2**K` classes
(i.e. the number of possible class-assignment configurations).
It is unlikely that you'll all `2**K` configurations in your dataset.
Parameters
----------
X :
A 2d array of features of shape (N, M) where N is the number of samples and M is the number of features.
labels :
A 2d array of binarized multi-labels of shape (N, K) where N is the number of samples and K is the number of classes.
clf :
A multi-label classifier with a ``predict_proba`` method.
cv :
A cross-validation splitter with a ``split`` method that returns a generator of train/test indices.
Returns
-------
pred_probs :
A 2d array of predicted probabilities of shape (N, K) where N is the number of samples and K is the number of classes.
Note
----
The predicted probabilities are not expected to sum to 1 for each sample in the case of multi-label classification.
Examples
--------
>>> import numpy as np
>>> from sklearn.model_selection import KFold
>>> from sklearn.multiclass import OneVsRestClassifier
>>> from sklearn.ensemble import RandomForestClassifier
>>> from cleanlab.internal.multilabel_scorer import get_cross_validated_multilabel_pred_probs
>>> np.random.seed(0)
>>> X = np.random.rand(16, 2)
>>> labels = np.random.randint(0, 2, size=(16, 2))
>>> clf = OneVsRestClassifier(RandomForestClassifier())
>>> cv = KFold(n_splits=2)
>>> get_cross_validated_multilabel_pred_probs(X, labels, clf=clf, cv=cv)
"""
split_generator = _get_split_generator(labels, cv)
pred_probs = cross_val_predict(clf, X, labels, cv=split_generator, method="predict_proba")
return pred_probs
cleanlab/internal/multilabel_utils.py
+90
View File
@@ -0,0 +1,90 @@
"""
Helper functions used internally for multi-label classification tasks.
"""
from typing import List, Optional, Tuple
import numpy as np
from cleanlab.internal.util import get_num_classes
def _is_multilabel(y: np.ndarray) -> bool:
"""Checks whether `y` is in a multi-label indicator matrix format.
Sparse matrices are not supported.
"""
if not (isinstance(y, np.ndarray) and y.ndim == 2 and y.shape[1] > 1):
return False
return np.array_equal(np.unique(y), [0, 1])
def stack_complement(pred_prob_slice: np.ndarray) -> np.ndarray:
"""
Extends predicted probabilities of a single class to two columns.
Parameters
----------
pred_prob_slice:
A 1D array with predicted probabilities for a single class.
Example
-------
>>> pred_prob_slice = np.array([0.1, 0.9, 0.3, 0.8])
>>> stack_complement(pred_prob_slice)
array([[0.9, 0.1],
[0.1, 0.9],
[0.7, 0.3],
[0.2, 0.8]])
"""
return np.vstack((1 - pred_prob_slice, pred_prob_slice)).T
def get_onehot_num_classes(
labels: list, pred_probs: Optional[np.ndarray] = None
) -> Tuple[np.ndarray, int]:
"""Returns OneHot encoding of MultiLabel Data, and number of classes"""
num_classes = get_num_classes(labels=labels, pred_probs=pred_probs)
try:
y_one = int2onehot(labels, K=num_classes)
except TypeError:
raise ValueError(
"wrong format for labels, should be a list of list[indices], please check the documentation in find_label_issues for further information"
)
return y_one, num_classes
def int2onehot(labels: list, K: int) -> np.ndarray:
"""Convert multi-label classification `labels` from a ``List[List[int]]`` format to a onehot matrix.
This returns a binarized format of the labels as a multi-hot vector for each example, where the entries in this vector are 1 for each class that applies to this example and 0 otherwise.
Parameters
----------
labels: list of lists of integers
e.g. [[0,1], [3], [1,2,3], [1], [2]]
All integers from 0,1,...,K-1 must be represented.
K: int
The number of classes."""
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer(classes=range(K))
return mlb.fit_transform(labels)
def onehot2int(onehot_matrix: np.ndarray) -> List[List[int]]:
"""Convert multi-label classification `labels` from a onehot matrix format to a ``List[List[int]]`` format that can be used with other cleanlab functions.
Parameters
----------
onehot_matrix: 2D np.ndarray of 0s and 1s
A matrix representation of multi-label classification labels in a binarized format as a multi-hot vector for each example.
The entries in this vector are 1 for each class that applies to this example and 0 otherwise.
Returns
-------
labels: list of lists of integers
e.g. [[0,1], [3], [1,2,3], [1], [2]]
All integers from 0,1,...,K-1 must be represented."""
return [np.where(row)[0].tolist() for row in onehot_matrix]
cleanlab/internal/neighbor/__init__.py
+1
View File
@@ -0,0 +1 @@
from .knn_graph import features_to_knn
cleanlab/internal/neighbor/knn_graph.py
+578
View File
@@ -0,0 +1,578 @@
from __future__ import annotations
from typing import List, Optional, TYPE_CHECKING, Tuple
import numpy as np
from scipy.sparse import csr_matrix
from scipy.linalg import circulant
from sklearn.neighbors import NearestNeighbors
if TYPE_CHECKING:
from cleanlab.typing import FeatureArray, Metric
from cleanlab.internal.neighbor.metric import decide_default_metric
from cleanlab.internal.neighbor.search import construct_knn
DEFAULT_K = 10
"""Default number of neighbors to consider in the k-nearest neighbors search,
unless the size of the feature array is too small or the user specifies a different value.
This should be the largest desired value of k for all desired issue types that require a KNN graph.
E.g. if near duplicates wants k=1 but outliers wants 10, then DEFAULT_K should be 10. This way, all issue types can rely on the same KNN graph.
"""
def features_to_knn(
features: Optional[FeatureArray],
*,
n_neighbors: Optional[int] = None,
metric: Optional[Metric] = None,
**sklearn_knn_kwargs,
) -> NearestNeighbors:
"""Build and fit a k-nearest neighbors search object from an array of numerical features.
Parameters
----------
features :
The input feature array, with shape (N, M), where N is the number of samples and M is the number of features.
n_neighbors :
The number of nearest neighbors to consider. If None, a default value is determined based on the feature array size.
metric :
The distance metric to use for computing distances between points. If None, the metric is determined based on the feature array shape.
**sklearn_knn_kwargs :
Additional keyword arguments to be passed to the search index constructor.
Returns
-------
knn :
A k-nearest neighbors search object fitted to the input feature array.
Examples
--------
>>> import numpy as np
>>> from cleanlab.internal.neighbor import features_to_knn
>>> features = np.random.rand(100, 10)
>>> knn = features_to_knn(features)
>>> knn
NearestNeighbors(metric='cosine', n_neighbors=10)
"""
if features is None:
raise ValueError("Both knn and features arguments cannot be None at the same time.")
# Use provided metric if available, otherwise decide based on the features.
metric = metric or decide_default_metric(features)
# Decide the number of neighbors to use in the KNN search.
n_neighbors = _configure_num_neighbors(features, n_neighbors)
knn = construct_knn(n_neighbors, metric, **sklearn_knn_kwargs)
return knn.fit(features)
def construct_knn_graph_from_index(
knn: NearestNeighbors,
correction_features: Optional[FeatureArray] = None,
) -> csr_matrix:
"""Construct a sparse distance matrix representation of KNN graph out of a fitted NearestNeighbors search object.
Parameters
----------
knn :
A NearestNeighbors object that has been fitted to a feature array.
The KNN graph is constructed based on the distances and indices of each feature row's nearest neighbors.
correction_features :
The input feature array used to fit the NearestNeighbors object.
If provided, the function the distances and indices of the neighbors will be corrected based on exact
duplicates in the feature array.
If not provided, no correction will be applied.
Warning
-------
This function is designed to handle a specific case where a KNN index is used to construct a KNN graph by itself,
and there is a need to detect and correct for exact duplicates in the feature array. However, relying on this
function for such corrections is generally discouraged. There are other functions in the module that handle
KNN graph construction with feature corrections in a more flexible and robust manner. Use this function only
when there is a special need to correct distances and indices based on the feature array provided.
Returns
-------
knn_graph :
A sparse, weighted adjacency matrix representing the KNN graph of the feature array.
Note
----
This is *not* intended to construct a KNN graph of test data. It is only used to construct a KNN graph of the data used to fit the NearestNeighbors object.
Examples
--------
>>> import numpy as np
>>> from cleanlab.internal.neighbor.knn_graph import features_to_knn, construct_knn_graph_from_index
>>> features = np.array([
... [0.701, 0.701],
... [0.900, 0.436],
... [0.000, 1.000],
... ])
>>> knn = features_to_knn(features, n_neighbors=1)
>>> knn_graph = construct_knn_graph_from_index(knn)
>>> knn_graph.toarray() # For demonstration purposes only. It is generally a bad idea to transform to dense matrix for large graphs.
array([[0. , 0.33140006, 0. ],
[0.33140006, 0. , 0. ],
[0.76210367, 0. , 0. ]])
"""
# Perform self-querying to get the distances and indices of the nearest neighbors
distances, indices = knn.kneighbors(X=None, return_distance=True)
# Correct the distances and indices if the correction_features array is provided
if correction_features is not None:
distances, indices = correct_knn_distances_and_indices(
features=correction_features, distances=distances, indices=indices
)
N, K = distances.shape
# Pointers to the row elements distances[indptr[i]:indptr[i+1]],
# and their corresponding column indices indices[indptr[i]:indptr[i+1]].
indptr = np.arange(0, N * K + 1, K)
return csr_matrix((distances.reshape(-1), indices.reshape(-1), indptr), shape=(N, N))
def create_knn_graph_and_index(
features: Optional[FeatureArray],
*,
n_neighbors: Optional[int] = None,
metric: Optional[Metric] = None,
correct_exact_duplicates: bool = True,
**sklearn_knn_kwargs,
) -> Tuple[csr_matrix, NearestNeighbors]:
"""Calculate the KNN graph from the features if it is not provided in the kwargs.
Parameters
----------
features :
The input feature array, with shape (N, M), where N is the number of samples and M is the number of features.
n_neighbors :
The number of nearest neighbors to consider. If None, a default value is determined based on the feature array size.
metric :
The distance metric to use for computing distances between points. If None, the metric is determined based on the feature array shape.
correct_exact_duplicates :
Whether to correct the KNN graph to ensure that exact duplicates have zero mutual distance, and they are correctly included in the KNN graph.
**sklearn_knn_kwargs :
Additional keyword arguments to be passed to the search index constructor.
Raises
------
ValueError :
If `features` is None, as it's required to construct a KNN graph from scratch.
Returns
-------
knn_graph :
A sparse, weighted adjacency matrix representing the KNN graph of the feature array.
knn :
A k-nearest neighbors search object fitted to the input feature array. This object can be used to query the nearest neighbors of new data points.
Examples
--------
>>> import numpy as np
>>> from cleanlab.internal.neighbor.knn_graph import create_knn_graph_and_index
>>> features = np.array([
... [0.701, 0.701],
... [0.900, 0.436],
... [0.000, 1.000],
... ])
>>> knn_graph, knn = create_knn_graph_and_index(features, n_neighbors=1)
>>> knn_graph.toarray() # For demonstration purposes only. It is generally a bad idea to transform to dense matrix for large graphs.
array([[0. , 0.33140006, 0. ],
[0.33140006, 0. , 0. ],
[0.76210367, 0. , 0. ]])
>>> knn
NearestNeighbors(metric=<function euclidean at ...>, n_neighbors=1) # For demonstration purposes only. The actual metric may vary.
"""
# Construct NearestNeighbors object
knn = features_to_knn(features, n_neighbors=n_neighbors, metric=metric, **sklearn_knn_kwargs)
# Build graph from NearestNeighbors object
knn_graph = construct_knn_graph_from_index(knn)
# Ensure that exact duplicates found with np.unique aren't accidentally missed in the KNN graph
if correct_exact_duplicates:
assert features is not None
knn_graph = correct_knn_graph(features, knn_graph)
return knn_graph, knn
def correct_knn_graph(features: FeatureArray, knn_graph: csr_matrix) -> csr_matrix:
"""
Corrects a k-nearest neighbors (KNN) graph by handling exact duplicates in the feature array.
This utility function takes a precomputed KNN graph and the corresponding feature array,
identifies sets of exact duplicate feature vectors, and corrects the KNN graph to properly
reflect these duplicates. The corrected KNN graph is returned as a sparse CSR matrix.
Parameters
----------
features : np.ndarray
The input feature array, with shape (N, M), where N is the number of samples and M is the number of features.
knn_graph : csr_matrix
A sparse matrix of shape (N, N) representing the k-nearest neighbors graph.
The graph is expected to be in CSR (Compressed Sparse Row) format.
Returns
-------
csr_matrix
A corrected KNN graph in CSR format with adjusted distances and indices to properly handle
exact duplicates in the feature array.
Notes
-----
- This function assumes that the input `knn_graph` is already computed and provided in CSR format.
- The function modifies the KNN graph to ensure that exact duplicates are represented with zero distance
and correctly updated neighbor indices.
- This function is useful for post-processing a KNN graph when exact duplicates were not handled during
the initial KNN computation.
"""
N = features.shape[0]
distances, indices = knn_graph.data.reshape(N, -1), knn_graph.indices.reshape(N, -1)
corrected_distances, corrected_indices = correct_knn_distances_and_indices(
features, distances, indices
)
N = features.shape[0]
return csr_matrix(
(corrected_distances.reshape(-1), corrected_indices.reshape(-1), knn_graph.indptr),
shape=(N, N),
)
def _compute_exact_duplicate_sets(features: FeatureArray) -> List[np.ndarray]:
"""
Computes the sets of exact duplicate points in the feature array.
This function groups indices of points that have identical feature vectors.
It returns a list of arrays, where each array contains the indices of points that are exact duplicates
of each other.
Parameters
----------
features : np.ndarray
The input feature array, with shape (N, M), where N is the number of samples and M is the number of features.
Returns
-------
exact_duplicate_sets
A list of 1D arrays, where each array contains the indices of exact duplicate points in the dataset.
Only sets with two or more duplicates are included in the list. If no exact duplicates are found, an empty list is returned.
Examples
--------
>>> features = np.array([[1, 2], [3, 4], [1, 2], [5, 6], [3, 4]])
>>> _compute_exact_duplicate_sets(features)
[array([0, 2]), array([1, 4])] # The row value [1, 2] appears in rows 0 and 2, and [3, 4] appears in rows 1 and 4.
Notes
-----
- This function uses `np.unique` to find unique feature vectors and their inverse indices.
- This function is intended to be used internally within this module.
"""
# Use np.unique to catch inverse indices of all unique feature sets
_, unique_inverse, unique_counts = np.unique(
features, return_inverse=True, return_counts=True, axis=0
)
# Collect different sets of exact duplicates in the dataset
exact_duplicate_sets = [
np.where(unique_inverse == u)[0] for u in set(unique_inverse) if unique_counts[u] > 1
]
return exact_duplicate_sets
def correct_knn_distances_and_indices_with_exact_duplicate_sets_inplace(
distances: np.ndarray,
indices: np.ndarray,
exact_duplicate_sets: List[np.ndarray],
) -> None:
"""
Corrects the distances and indices arrays of k-nearest neighbors (KNN) graphs by handling sets
of exact duplicates explicitly. This function modifies the input arrays in-place.
This function ensures that exact duplicates are correctly represented in the KNN graph.
It modifies the `distances` and `indices` arrays so that each set of exact duplicates
points to itself with zero distance, and adjusts the nearest neighbors accordingly.
Parameters
----------
distances :
A 2D array of shape (N, k) representing the distances between each point of the N points and their k-nearest neighbors.
This array will be modified in-place to reflect the corrections for exact duplicates (whose mutual distances are explicitly set to zero).
indices :
A 2D array of shape (N, k) representing the indices of the nearest neighbors for each of the N points.
This array will be modified in-place to reflect the corrections for exact duplicates.
exact_duplicate_sets :
A list of 1D arrays, each containing the indices of points that are exact duplicates of each other.
These sets will be used to correct the KNN graph by ensuring that duplicates are reflected as nearest neighbors
with zero distance.
High-Level Overview
-------------------
The function operates in two main scenarios based on the size of the duplicate sets relative to k:
1. **Duplicate Set Size >= k + 1**:
- All nearest neighbors are exact duplicates.
- The `indices` array is updated such that the first k+1 entries for each duplicate set point are used to represent the nearest neighbors
of all points in the duplicate set.
- The rows of the `distances` array belonging to the duplicate set are set to zero.
2. **Duplicate Set Size < k + 1**:
- Some of the nearest neighbors are not exact duplicates.
- Non-duplicate neighbors are shifted to the back of the list.
- The `indices` and `distances` arrays are updated accordingly to reflect the duplicates at the front with zero distance.
User Considerations
-------------------
- **Input Validity**: Ensure that the `distances` and `indices` arrays have the correct shape and correspond to the same KNN graph.
- **In-Place Modifications**: The function modifies the input arrays directly. If the original data is needed, make a copy before calling the function.
- **Duplicate Set Size**: The function is optimized for cases where the number of exact duplicates can be larger than k. Ensure the duplicate sets are accurately identified.
- **Performance**: The function uses efficient NumPy operations, but performance can be affected by the size of the input arrays and the number of duplicate sets.
Capabilities
------------
- Handles exact duplicate sets efficiently, ensuring correct KNN graph representation.
- Maintains zero distances for exact duplicates.
- Adjusts neighbor indices to reflect the presence of duplicates.
Limitations
-----------
- Assumes that the input arrays (`distances` and `indices`) come from a precomputed KNN graph.
- Does not handle near-duplicates or merge non-duplicate neighbors.
- Requires careful construction of `exact_duplicate_sets` to avoid misidentification.
"""
# Number of neighbors
k = distances.shape[1]
for duplicate_inds in exact_duplicate_sets:
# Determine the number of same points to include, respecting the limit of k
num_same = len(duplicate_inds)
num_same_included = min(num_same - 1, k) # ensure we do not exceed k neighbors
sorted_first_k_duplicate_inds = _prepare_neighborhood_of_first_k_duplicates(
duplicate_inds, num_same_included
)
if num_same >= k + 1:
# All nearest neighbors are exact duplicates
# We only pass in the ciruclant matrix of nearest neighbors
indices[duplicate_inds[: k + 1]] = sorted_first_k_duplicate_inds
# But the rest will just take the k first duplicate ids
indices[duplicate_inds[k + 1 :]] = duplicate_inds[:k]
# Finally, set the distances between exact duplicates to zero
distances[duplicate_inds] = 0
else:
# Some of the nearest neighbors aren't exact duplicates, move those to the back
# Get indices and distances from knn that are not the same as i
different_point_mask = np.isin(indices[duplicate_inds], duplicate_inds, invert=True)
# Get the indices of the first m True values in each row of the mask
true_indices = np.argsort(~different_point_mask, axis=1)[:, :-num_same_included]
# Copy the values to the last m columns in dists
distances[duplicate_inds, -(k - num_same_included) :] = distances[
duplicate_inds, true_indices.T
].T
indices[duplicate_inds, -(k - num_same_included) :] = indices[
duplicate_inds, true_indices.T
].T
# We can pass the circulant matrix to a slice
indices[duplicate_inds, :num_same_included] = sorted_first_k_duplicate_inds
# Finally, set the distances between exact duplicates to zero
distances[duplicate_inds, :num_same_included] = 0
return None
def _prepare_neighborhood_of_first_k_duplicates(duplicate_inds, num_same_included):
"""
Prepare a matrix representing the neighborhoods of duplicate items.
This function constructs a matrix where each row corresponds to an item
and contains the indices of its nearest neighbors (excluding itself), up
to a specified number `k`.
Parameters:
-----------
duplicate_inds : list
A list of indices that represent duplicate items.
num_same_included : int
An integer `k` representing the number of neighbors to include for
each item.
Returns:
--------
np.ndarray
A matrix where each row contains the sorted indices of the nearest
neighbors for the corresponding item.
Explanation:
------------
1. Extract the Base for the Circulant Matrix:
- The function extracts the first `k+1` elements from `duplicate_inds`
to form the base of the circulant matrix. This approach ensures that
even if the set of duplicate items is larger, we only need to consider
the first `k` duplicates as the nearest neighbors, avoiding conflicts
with the items themselves.
2. Create the Circulant Matrix:
- A circulant matrix is generated from the base, where each row is a
cyclic permutation of the previous row.
3. Slice the Matrix to Exclude the First Column:
- The first column is removed to ensure each row represents the neighbors
without including the item itself.
4. Sort the Neighborhood Indices:
- The rows of the sliced matrix are sorted to ensure a consistent order
of neighbors.
Example:
--------
Given a set of 5 duplicate items `[A, B, C, D, E]` and `k=2`, the function
processes this as follows:
1. `circulant_base` for `k=2` would be `[A, B, C]`.
2. The `circulant_matrix` might look like:
```
[A B C]
[B C A]
[C A B]
```
3. Removing the first column results in:
```
[B C]
[C A]
[A B]
```
4. Sorting each row gives the final matrix:
```
[B C]
[A C]
[A B]
```
This matrix indicates that:
- The nearest neighbors of `A` are `[B, C]`.
- The nearest neighbors of `B` are `[A, C]`.
- The nearest neighbors of `C` are `[A, B]`.
For `k=2`, the neighbors of `D`, `E`, onwards could be any of the above.
The function constructs a sorted matrix of nearest neighbors for a list of
duplicate items, ensuring an equal distribution of neighbors up to a specified
number `k`. This process is necessary for tasks requiring an understanding of
the local neighborhood structure among duplicate examples. By using only the first
`k+1` elements, the function avoids the need to construct a larger circulant
matrix, simplifying the computation and ensuring no conflicts among the rest of the items.
"""
circulant_base = duplicate_inds[: num_same_included + 1]
circulant_matrix = circulant(circulant_base)
sliced_circulant_matrix = circulant_matrix[:, 1:]
sorted_first_k_duplicate_inds = np.sort(sliced_circulant_matrix, axis=1)
return sorted_first_k_duplicate_inds
def correct_knn_distances_and_indices(
features: FeatureArray,
distances: np.ndarray,
indices: np.ndarray,
exact_duplicate_sets: Optional[List[np.ndarray]] = None,
) -> tuple[np.ndarray, np.ndarray]:
"""
Corrects the distances and indices of a k-nearest neighbors (KNN) graph
based on all exact duplicates detected in the feature array.
Parameters
----------
features :
The feature array used to construct the KNN graph.
distances :
The distances between each point and its k nearest neighbors.
indices :
The indices of the k nearest neighbors for each point.
exact_duplicate_sets:
A list of numpy arrays, where each array contains the indices of exact duplicates in the feature array. If not provided, it will be computed from the feature array.
Returns
-------
corrected_distances :
The corrected distances between each point and its k nearest neighbors. Exact duplicates (based on the feature array) are ensured to have zero mutual distance.
corrected_indices :
The corrected indices of the k nearest neighbors for each point. Exact duplicates are ensured to be included in the k nearest neighbors, unless the number of exact duplicates exceeds k.
Example
-------
>>> import numpy as np
>>> X = np.array(
... [
... [0, 0],
... [0, 0], # Exact duplicate of the previous point
... [1, 1], # The distances between this point and the others is sqrt(2) (equally distant from both)
... ]
... )
>>> distances = np.array( # Distance to the 1-NN of each point
... [
... [np.sqrt(2)], # Should be [0]
... [1e-16], # Should be [0]
... [np.sqrt(2)],
... ]
... )
>>> indices = np.array( # Index of the 1-NN of each point
... [
... [2], # Should be [1]
... [0],
... [1], # Might be [0] or [1]
... ]
... )
>>> corrected_distances, corrected_indices = correct_knn_distances_and_indices(X, distances, indices)
>>> corrected_distances
array([[0.], [0.], [1.41421356]])
>>> corrected_indices
array([[1], [0], [0]])
"""
if exact_duplicate_sets is None:
exact_duplicate_sets = _compute_exact_duplicate_sets(features)
# Prepare the output arrays
corrected_distances = np.copy(distances)
corrected_indices = np.copy(indices)
correct_knn_distances_and_indices_with_exact_duplicate_sets_inplace(
distances=corrected_distances,
indices=corrected_indices,
exact_duplicate_sets=exact_duplicate_sets,
)
return corrected_distances, corrected_indices
def _configure_num_neighbors(features: FeatureArray, k: Optional[int]):
# Error if the provided value is greater or equal to the number of examples.
N = features.shape[0]
k_larger_than_dataset = k is not None and k >= N
if k_larger_than_dataset:
raise ValueError(
f"Number of nearest neighbors k={k} cannot exceed the number of examples N={len(features)} passed into the estimator (knn)."
)
# Either use the provided value or select a default value based on the feature array size.
k = k or min(DEFAULT_K, N - 1)
return k
cleanlab/internal/neighbor/metric.py
+107
View File
@@ -0,0 +1,107 @@
from scipy.spatial.distance import euclidean
from cleanlab.typing import FeatureArray, Metric
HIGH_DIMENSION_CUTOFF: int = 3
"""
If the number of columns (M) in the `features` array is greater than this cutoff value,
then by default, K-nearest-neighbors will use the "cosine" metric.
The cosine metric is more suitable for high-dimensional data.
Otherwise the "euclidean" distance will be used.
"""
ROW_COUNT_CUTOFF: int = 100
"""
Only affects settings where Euclidean metrics would be used by default.
If the number of rows (N) in the `features` array is greater than this cutoff value,
then by default, Euclidean distances are computed via the "euclidean" metric
(implemented in sklearn for efficiency reasons).
Otherwise, Euclidean distances are by default computed via
the ``euclidean`` metric from scipy (slower but numerically more precise/accurate).
"""
# Metric decision functions
def _euclidean_large_dataset() -> str:
return "euclidean"
def _euclidean_small_dataset() -> Metric:
return euclidean
def _cosine_metric() -> str:
return "cosine"
def decide_euclidean_metric(features: FeatureArray) -> Metric:
"""
Decide the appropriate Euclidean metric implementation based on the size of the dataset.
Parameters
----------
features :
The input features array.
Returns
-------
metric :
A string or a callable representing a specific implementation of computing the euclidean distance.
Note
----
A choice is made between two implementations
of the euclidean metric based on the number of rows in the feature array.
If the number of rows (N) in the feature array is greater than another predefined
cutoff value (ROW_COUNT_CUTOFF), the ``"euclidean"`` metric is used. This
is because the euclidean metric performs better on larger datasets.
If neither condition is met, the ``euclidean`` metric function from scipy is returned.
See also
--------
ROW_COUNT_CUTOFF: The cutoff value for the number of rows in the feature array.
sklearn.metrics.pairwise.euclidean_distances: The euclidean metric function from scikit-learn.
scipy.spatial.distance.euclidean: The euclidean metric function from scipy.
"""
num_rows = features.shape[0]
if num_rows > ROW_COUNT_CUTOFF:
return _euclidean_large_dataset()
else:
return _euclidean_small_dataset()
# Main function to decide the metric
def decide_default_metric(features: FeatureArray) -> Metric:
"""
Decide the KNN metric to be used based on the shape of the feature array.
Parameters
----------
features :
The input feature array, with shape (N, M), where N is the number of samples and M is the number of features.
Returns
-------
metric :
The distance metric to be used for neighbor search. It can be either a string
representing the metric name ("cosine" or "euclidean") or a callable
representing the metric function from scipy (euclidean).
Note
----
The decision of which metric to use is based on the shape of the feature array.
If the number of columns (M) in the feature array is greater than a predefined
cutoff value (HIGH_DIMENSION_CUTOFF), the "cosine" metric is used. This is because the cosine
metric is more suitable for high-dimensional data.
Otherwise, a euclidean metric is used.
That is handled by the :py:meth:`~cleanlab.internal.neighbor.metric.decide_euclidean_metric` function.
See Also
--------
HIGH_DIMENSION_CUTOFF: The cutoff value for the number of columns in the feature array.
sklearn.metrics.pairwise.cosine_distances: The cosine metric function from scikit-learn
"""
if features.shape[1] > HIGH_DIMENSION_CUTOFF:
return _cosine_metric()
return decide_euclidean_metric(features)
cleanlab/internal/neighbor/search.py
+75
View File
@@ -0,0 +1,75 @@
from __future__ import annotations
from typing import TYPE_CHECKING
from sklearn.neighbors import NearestNeighbors
if TYPE_CHECKING:
from cleanlab.typing import Metric
def construct_knn(n_neighbors: int, metric: Metric, **knn_kwargs) -> NearestNeighbors:
"""
Constructs a k-nearest neighbors search object. You can implement a similar method to run cleanlab with your own approximate-KNN library.
Parameters
----------
n_neighbors :
The number of nearest neighbors to consider.
metric :
The distance metric to use for computing distances between points.
See :py:mod:`~cleanlab.internal.neighbor.metric` for more information.
**knn_kwargs:
Additional keyword arguments to be passed to the search index constructor.
See https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html for more details on the available options.
Returns
-------
knn :
A k-nearest neighbors search object compatible with the scikit-learn NearestNeighbors class interface.
Implements:
- `fit` method: Accepts a feature array `X` to fit the model.
This enables subsequent neighbor searches on the data.
- `kneighbors` method: Finds the K-neighbors of a point, returning distances and indices of the k-nearest neighbors. Handles two scenarios:
1. When a query array `features: np.ndarray` is provided, it returns the distances and indices for each point in the query array.
2. When no query array is provided (`features = None`), it returns neighbors for each indexed point without considering the query point as its own neighbor.
Optionally, allows re-specification of the number of neighbors for each query point, defaulting to the constructor's value if not specified.
Attributes:
- `n_neighbors`: Number of neighbors to consider.
- `metric`: Distance metric used to compute distances between points.
- `metric_params`: Additional parameters for the distance metric function.
Optional:
- `kneighbors_graph` method: Not required but can be implemented for convenience.
Responsibility shifted to :py:ref:`construct_knn_graph_from_index <cleanlab.internal.neighbor.neighbor.construct_knn_graph_from_index>`.
Fitted Attributes:
- `n_features_in_`: Number of features observed during fit.
- `effective_metric_params_`: Metric parameters used in distance computation.
- `effective_metric_`: Metric used for computing distances to neighbors.
- `n_samples_fit_`: Number of samples in the fitted data.
Additional:
- `__sklearn_is_fitted__`: Method returning a boolean indicating if the object is fitted,
useful for conducting an is_fitted validation, which verifies the presence of fitted attributes (typically ending with a trailing underscore).
The above specifications ensure compatibility and provide a clear directive for developers needing to integrate alternative k-nearest neighbors implementations or modify existing functionalities.
Note
----
The `metric` argument should be a callable that takes two arguments (the two points) and returns the distance between them.
The additional keyword arguments (`**knn_kwargs`) are passed directly to the underlying k-nearest neighbors search algorithm.
"""
sklearn_knn = NearestNeighbors(n_neighbors=n_neighbors, metric=metric, **knn_kwargs)
return sklearn_knn
cleanlab/internal/numerics.py
+39
View File
@@ -0,0 +1,39 @@
from typing import Optional
import numpy as np
from cleanlab.internal.constants import EPSILON
def softmax(
x: np.ndarray, temperature: float = 1.0, axis: Optional[int] = None, shift: bool = False
) -> np.ndarray:
"""Softmax function.
Parameters
----------
x : np.ndarray
Input array.
temperature : float
Temperature of the softmax function.
axis : Optional[int]
Axis to apply the softmax function. If None, the softmax function is
applied to all elements of the input array.
shift : bool
Whether to shift the input array before applying the softmax function.
This is useful to avoid numerical issues when the input array contains
large values, that could result in overflows when applying the exponential
function.
Returns
-------
np.ndarray
Softmax function applied to the input array.
"""
x = x / max(temperature, EPSILON)
if shift:
x = x - np.max(x, axis=axis, keepdims=True)
exp_x = np.exp(x)
return exp_x / np.sum(exp_x, axis=axis, keepdims=True)
cleanlab/internal/object_detection_utils.py
+89
View File
@@ -0,0 +1,89 @@
"""
Helper functions used internally for object detection tasks.
"""
from typing import Any, Dict, List, Optional
import numpy as np
from cleanlab.internal.numerics import softmax
def bbox_xyxy_to_xywh(bbox: List[float]) -> Optional[List[float]]:
"""Converts bounding box coodrinate types from x1y1,x2y2 to x,y,w,h"""
if len(bbox) == 4:
x1, y1, x2, y2 = bbox
w = x2 - x1
h = y2 - y1
return [x1, y1, w, h]
else:
print("Wrong bbox shape", len(bbox))
return None
def softmin1d(scores: np.ndarray, temperature: float = 0.99, axis: int = 0) -> float:
"""Returns softmin of passed in scores."""
scores = np.array(scores)
softmax_scores = softmax(x=-1 * scores, temperature=temperature, axis=axis, shift=True)
return np.dot(softmax_scores, scores)
def calculate_bounding_box_areas(rectangle_coordinates):
"""Calculate areas of bounding boxes represented by numpy array with x1, y1, x2, y2 format"""
widths = rectangle_coordinates[:, 2] - rectangle_coordinates[:, 0]
heights = rectangle_coordinates[:, 3] - rectangle_coordinates[:, 1]
return widths * heights
def assert_valid_aggregation_weights(aggregation_weights: Dict[str, Any]) -> None:
"""assert aggregation weights are in the proper format"""
weights = np.array(list(aggregation_weights.values()))
if (not np.isclose(np.sum(weights), 1.0)) or (np.min(weights) < 0.0):
raise ValueError(
f"""Aggregation weights should be non-negative and must sum to 1.0
"""
)
def assert_valid_inputs(
labels: List[Dict[str, Any]],
predictions,
method: Optional[str] = None,
threshold: Optional[float] = None,
):
"""Asserts proper input format."""
if len(labels) != len(predictions):
raise ValueError(
f"labels and predictions length needs to match. len(labels) == {len(labels)} while len(predictions) == {len(predictions)}."
)
# Typecheck labels and predictions
if not isinstance(labels[0], dict):
raise ValueError(
f"Labels has to be a list of dicts. Instead it is list of {type(labels[0])}."
)
# check last column of predictions is probabilities ( < 1.)?
if not isinstance(predictions[0], (list, np.ndarray)):
raise ValueError(
f"Prediction has to be a list or np.ndarray. Instead it is type {type(predictions[0])}."
)
if not predictions[0][0].shape[1] == 5:
raise ValueError(
f"Prediction values have to be of format [x1,y1,x2,y2,pred_prob]. Please refer to the documentation for predicted probabilities under object_detection.rank.get_label_quality_scores for details"
)
valid_methods = ["objectlab"]
if method is not None and method not in valid_methods:
raise ValueError(
f"""
{method} is not a valid object detection scoring method!
Please choose a valid scoring_method: {valid_methods}
"""
)
if threshold is not None and threshold > 1.0:
raise ValueError(
f"""
Threshold is a cutoff of predicted probabilities and therefore should be <= 1.
"""
)
cleanlab/internal/outlier.py
+112
View File
@@ -0,0 +1,112 @@
"""
Helper functions used internally for outlier detection tasks.
"""
from typing import Optional
import numpy as np
from cleanlab.internal.constants import EPSILON
def transform_distances_to_scores(
avg_distances: np.ndarray, t: int, scaling_factor: float
) -> np.ndarray:
"""Returns an outlier score for each example based on its average distance to its k nearest neighbors.
The transformation of a distance, :math:`d` , to a score, :math:`o` , is based on the following formula:
.. math::
o = \\exp\\left(-dt\\right)
where :math:`t` scales the distance to a score in the range [0,1].
Parameters
----------
avg_distances : np.ndarray
An array of distances of shape ``(N)``, where N is the number of examples.
Each entry represents an example's average distance to its k nearest neighbors.
t : int
A sensitivity parameter that modulates the strength of the transformation from distances to scores.
Higher values of `t` result in more pronounced differentiation between the scores of examples
lying in the range [0,1].
scaling_factor : float
A scaling factor used to normalize the distances before they are converted into scores. A valid
scaling factor is any positive number. The choice of scaling factor should be based on the
distribution of distances between neighboring examples. A good rule of thumb is to set the
scaling factor to the median distance between neighboring examples. A lower scaling factor
results in more pronounced differentiation between the scores of examples lying in the range [0,1].
Returns
-------
ood_features_scores : np.ndarray
An array of outlier scores of shape ``(N,)`` for N examples.
Examples
--------
>>> import numpy as np
>>> from cleanlab.outlier import transform_distances_to_scores
>>> distances = np.array([[0.0, 0.1, 0.25],
... [0.15, 0.2, 0.3]])
>>> avg_distances = np.mean(distances, axis=1)
>>> transform_distances_to_scores(avg_distances, t=1, scaling_factor=1)
array([0.88988177, 0.80519832])
"""
# Map ood_features_scores to range 0-1 with 0 = most concerning
return np.exp(-t * avg_distances / max(scaling_factor, EPSILON))
def correct_precision_errors(
scores: np.ndarray,
avg_distances: np.ndarray,
metric: str,
C: int = 100,
p: Optional[int] = None,
):
"""
Ensure that scores where avg_distances are below the tolerance threshold get a score of one.
Parameters
----------
scores :
An array of scores of shape ``(N)``, where N is the number of examples.
Each entry represents a score between 0 and 1.
avg_distances :
An array of distances of shape ``(N)``, where N is the number of examples.
Each entry represents an example's average distance to its k nearest neighbors.
metric :
The metric used by the knn algorithm to calculate the distances.
It must be 'cosine', 'euclidean' or 'minkowski', otherwise this function does nothing.
C :
Multiplier used to increase the tolerance of the acceptable precision differences.
It is a multiplicative factor of the machine epsilon that is used to calculate the tolerance.
For the type of values that are used in the distances, a value of 100 should be a sensible
default value for small values of the distances, below the order of 1.
p :
This value is only used when metric is 'minkowski'.
A ValueError will be raised if metric is 'minkowski' and 'p' was not provided.
Returns
-------
fixed_scores :
An array of scores of shape ``(N,)`` for N examples with scores between 0 and 1.
"""
if metric == "cosine":
tolerance = C * np.finfo(np.float64).epsneg
elif metric == "euclidean":
tolerance = np.sqrt(C * np.finfo(np.float64).eps)
elif metric == "minkowski":
if p is None:
raise ValueError("When metric is 'minkowski' you must specify the 'p' parameter")
tolerance = (C * np.finfo(np.float64).eps) ** (1 / p)
else:
return scores
candidates_mask = avg_distances < tolerance
scores[candidates_mask] = 1
return scores
cleanlab/internal/regression_utils.py
+113
View File
@@ -0,0 +1,113 @@
"""
Helper functions internally used in cleanlab.regression.
"""
import numpy as np
import pandas as pd
from numpy.typing import ArrayLike
from typing import Tuple, Union
def assert_valid_prediction_inputs(
labels: ArrayLike,
predictions: ArrayLike,
method: str,
) -> Tuple[np.ndarray, np.ndarray]:
"""Checks that ``labels``, ``predictions``, ``method`` are correctly formatted."""
# Load array_like input as numpy array. If not raise error.
try:
labels = np.asarray(labels)
except:
raise ValueError(f"labels must be array_like.")
try:
predictions = np.asarray(predictions)
except:
raise ValueError(f"predictions must be array_like.")
# Check if labels and predictions are 1-D and numeric
valid_labels = check_dimension_and_datatype(check_input=labels, text="labels")
valid_predictions = check_dimension_and_datatype(check_input=predictions, text="predictions")
# Check if number of examples are same.
assert (
valid_labels.shape == valid_predictions.shape
), f"Number of examples in labels {labels.shape} and predictions {predictions.shape} are not same."
# Check if inputs have missing values
check_missing_values(valid_labels, text="labels")
check_missing_values(valid_predictions, text="predictions")
# Check if method is among allowed scoring method
scoring_methods = ["residual", "outre"]
if method not in scoring_methods:
raise ValueError(f"Specified method '{method}' must be one of: {scoring_methods}.")
# return 1-D numpy array
return valid_labels, valid_predictions
def assert_valid_regression_inputs(
X: Union[np.ndarray, pd.DataFrame],
y: ArrayLike,
) -> Tuple[np.ndarray, np.ndarray]:
"""
Checks that regression inputs are properly formatted and returns the inputs in numpy array format.
"""
try:
X = np.asarray(X)
except:
raise ValueError(f"X must be array_like.")
y = check_dimension_and_datatype(y, "y")
check_missing_values(y, text="y")
if len(X) != len(y):
raise ValueError("X and y must have same length.")
return X, y
def check_dimension_and_datatype(check_input: ArrayLike, text: str) -> np.ndarray:
"""
Raises errors related to:
1. If input is empty
2. If input is not 1-D
3. If input is not numeric
If all the checks are passed, it returns the squeezed 1-D array required by the main algorithm.
"""
try:
check_input = np.asarray(check_input)
except:
raise ValueError(f"{text} could not be converted to numpy array, check input.")
# Check if input is empty
if not check_input.size:
raise ValueError(f"{text} cannot be empty array.")
# Remove axis with length one
check_input = np.squeeze(check_input)
# Check if input is 1-D
if check_input.ndim != 1:
raise ValueError(
f"Expected 1-Dimensional inputs for {text}, got {check_input.ndim} dimensions."
)
# Check if datatype is numeric
if not np.issubdtype(check_input.dtype, np.number):
raise ValueError(
f"Expected {text} to contain numeric values, got values of type {check_input.dtype}."
)
return check_input
def check_missing_values(check_input: np.ndarray, text: str):
"""Raise error if there are any missing values in Numpy array."""
if np.isnan(check_input).any():
raise ValueError(f"{text} cannot contain missing values.")
cleanlab/internal/segmentation_utils.py
+58
View File
@@ -0,0 +1,58 @@
"""
Helper functions used internally for segmentation tasks.
"""
from typing import Optional, List
import numpy as np
def _get_valid_optional_params(
batch_size: Optional[int] = None,
n_jobs: Optional[int] = None,
):
"""Takes in optional args and returns good values for them if they are None."""
if batch_size is None:
batch_size = 10000
if batch_size <= 0:
raise ValueError(f"Batch size must be greater than 0, got {batch_size}")
return batch_size, n_jobs
def _get_summary_optional_params(
class_names: Optional[List[str]] = None,
exclude: Optional[List[int]] = None,
top: Optional[int] = None,
):
"""Takes in optional args and returns good values for them if they are None for summary functions."""
if exclude is None:
exclude = []
if top is None:
top = 20
return class_names, exclude, top
def _check_input(labels: np.ndarray, pred_probs: np.ndarray) -> None:
"""
Checks that the input labels and predicted probabilities are valid.
Parameters
----------
labels:
Array of shape ``(N, H, W)`` of integer labels, where `N` is the number of images in the dataset and `H` and `W` are the height and width of the images.
pred_probs:
Array of shape ``(N, K, H, W)`` of predicted probabilities, where `N` is the number of images in the dataset, `K` is the number of classes, and `H` and `W` are the height and width of the images.
"""
if len(labels.shape) != 3:
raise ValueError("labels must have a shape of (N, H, W)")
if len(pred_probs.shape) != 4:
raise ValueError("pred_probs must have a shape of (N, K, H, W)")
num_images, height, width = labels.shape
num_images_pred, num_classes, height_pred, width_pred = pred_probs.shape
if num_images != num_images_pred or height != height_pred or width != width_pred:
raise ValueError("labels and pred_probs must have matching dimensions for N, H, and W")
cleanlab/internal/token_classification_utils.py
+276
View File
@@ -0,0 +1,276 @@
"""
Helper methods used internally in cleanlab.token_classification
"""
from __future__ import annotations
import re
import string
import numpy as np
from termcolor import colored
from typing import List, Optional, Callable, Tuple, TypeVar, TYPE_CHECKING
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
T = TypeVar("T", bound=npt.NBitBase)
def get_sentence(words: List[str]) -> str:
"""
Get sentence formed by a list of words with minor processing for readability
Parameters
----------
words:
list of word-level tokens
Returns
----------
sentence:
sentence formed by list of word-level tokens
Examples
--------
>>> from cleanlab.internal.token_classification_utils import get_sentence
>>> words = ["This", "is", "a", "sentence", "."]
>>> get_sentence(words)
'This is a sentence.'
"""
sentence = ""
for word in words:
if word not in string.punctuation or word in ["-", "("]:
word = " " + word
sentence += word
sentence = sentence.replace(" '", "'").replace("( ", "(").strip()
return sentence
def filter_sentence(
sentences: List[str],
condition: Optional[Callable[[str], bool]] = None,
) -> Tuple[List[str], List[bool]]:
"""
Filter sentence based on some condition, and returns filter mask
Parameters
----------
sentences:
list of sentences
condition:
sentence filtering condition
Returns
---------
sentences:
list of sentences filtered
mask:
boolean mask such that `mask[i] == True` if the i'th sentence is included in the
filtered sentence, otherwise `mask[i] == False`
Examples
--------
>>> from cleanlab.internal.token_classification_utils import filter_sentence
>>> sentences = ["Short sentence.", "This is a longer sentence."]
>>> condition = lambda x: len(x.split()) > 2
>>> long_sentences, _ = filter_sentence(sentences, condition)
>>> long_sentences
['This is a longer sentence.']
>>> document = ["# Headline", "Sentence 1.", "&", "Sentence 2."]
>>> sentences, mask = filter_sentence(document)
>>> sentences, mask
(['Sentence 1.', 'Sentence 2.'], [False, True, False, True])
"""
if not condition:
condition = lambda sentence: len(sentence) > 1 and "#" not in sentence
mask = list(map(condition, sentences))
sentences = [sentence for m, sentence in zip(mask, sentences) if m]
return sentences, mask
def process_token(token: str, replace: List[Tuple[str, str]] = [("#", "")]) -> str:
"""
Replaces special characters in the tokens
Parameters
----------
token:
token which potentially contains special characters
replace:
list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2
Returns
---------
processed_token:
processed token whose special character has been replaced
Note
----
Only applies to characters in the original input token.
Examples
--------
>>> from cleanlab.internal.token_classification_utils import process_token
>>> token = "#Comment"
>>> process_token("#Comment")
'Comment'
Specify custom replacement rules
>>> replace = [("C", "a"), ("a", "C")]
>>> process_token("Cleanlab", replace)
'aleCnlCb'
"""
replace_dict = {re.escape(k): v for (k, v) in replace}
pattern = "|".join(replace_dict.keys())
compiled_pattern = re.compile(pattern)
replacement = lambda match: replace_dict[re.escape(match.group(0))]
processed_token = compiled_pattern.sub(replacement, token)
return processed_token
def mapping(entities: List[int], maps: List[int]) -> List[int]:
"""
Map a list of entities to its corresponding entities
Parameters
----------
entities:
a list of given entities
maps:
a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`
Returns
---------
mapped_entities:
a list of mapped entities
Examples
--------
>>> unique_identities = [0, 1, 2, 3, 4] # ["O", "B-PER", "I-PER", "B-LOC", "I-LOC"]
>>> maps = [0, 1, 1, 2, 2] # ["O", "PER", "PER", "LOC", "LOC"]
>>> mapping(unique_identities, maps)
[0, 1, 1, 2, 2] # ["O", "PER", "PER", "LOC", "LOC"]
>>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)
[0, 0, 2, 2, 2, 2, 0, 1] # ["O", "O", "LOC", "LOC", "LOC", "LOC", "O", "PER"]
"""
f = lambda x: maps[x]
return list(map(f, entities))
def merge_probs(
probs: npt.NDArray["np.floating[T]"], maps: List[int]
) -> npt.NDArray["np.floating[T]"]:
"""
Merges model-predictive probabilities with desired mapping
Parameters
----------
probs:
A 2D np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model
maps:
a list of mapped index, such that the probability of the token being in the i'th class is mapped to the
`maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the
returned probability is re-normalized.
Returns
---------
probs_merged:
A 2D np.array of shape ``(N, K')``, where `K'` is the number of new classes. Probabilities are merged and
re-normalized if necessary.
Examples
--------
>>> import numpy as np
>>> from cleanlab.internal.token_classification_utils import merge_probs
>>> probs = np.array([
... [0.55, 0.0125, 0.0375, 0.1, 0.3],
... [0.1, 0.8, 0, 0.075, 0.025],
... ])
>>> maps = [0, 1, 1, 2, 2]
>>> merge_probs(probs, maps)
array([[0.55, 0.05, 0.4 ],
[0.1 , 0.8 , 0.1 ]])
"""
old_classes = probs.shape[1]
map_size = np.max(maps) + 1
probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)
for i in range(old_classes):
if maps[i] >= 0:
probs_merged[:, maps[i]] += probs[:, i]
if -1 in maps:
row_sums = probs_merged.sum(axis=1)
probs_merged /= row_sums[:, np.newaxis]
return probs_merged
def color_sentence(sentence: str, word: str) -> str:
"""
Searches for a given token in the sentence and returns the sentence where the given token is colored red
Parameters
----------
sentence:
a sentence where the word is searched
word:
keyword to find in `sentence`. Assumes the word exists in the sentence.
Returns
---------
colored_sentence:
`sentence` where the every occurrence of the word is colored red, using ``termcolor.colored``
Examples
--------
>>> from cleanlab.internal.token_classification_utils import color_sentence
>>> sentence = "This is a sentence."
>>> word = "sentence"
>>> color_sentence(sentence, word)
'This is a \x1b[31msentence\x1b[0m.'
Also works for multiple occurrences of the word
>>> document = "This is a sentence. This is another sentence."
>>> word = "sentence"
>>> color_sentence(document, word)
'This is a \x1b[31msentence\x1b[0m. This is another \x1b[31msentence\x1b[0m.'
"""
colored_word = colored(word, "red", force_color=True)
return _replace_sentence(sentence=sentence, word=word, new_word=colored_word)
def _replace_sentence(sentence: str, word: str, new_word: str) -> str:
"""
Searches for a given token in the sentence and returns the sentence where the given token has been replaced by
`new_word`.
Parameters
----------
sentence:
a sentence where the word is searched
word:
keyword to find in `sentence`. Assumes the word exists in the sentence.
new_word:
the word to replace the keyword with
Returns
---------
new_sentence:
`sentence` where the every occurrence of the word is replaced by `colored_word`
"""
new_sentence, number_of_substitions = re.subn(
r"\b{}\b".format(re.escape(word)), new_word, sentence
)
if number_of_substitions == 0:
# Use basic string manipulation if regex fails
new_sentence = sentence.replace(word, new_word)
return new_sentence
cleanlab/internal/util.py
+616
View File
@@ -0,0 +1,616 @@
"""
Ancillary helper methods used internally throughout this package; mostly related to Confident Learning algorithms.
"""
from typing import Optional, Tuple, Union
import numpy as np
import pandas as pd
from cleanlab.internal.constants import FLOATING_POINT_COMPARISON, TINY_VALUE
from cleanlab.internal.validation import labels_to_array
from cleanlab.typing import DatasetLike, LabelLike
def remove_noise_from_class(noise_matrix: np.ndarray, class_without_noise: int) -> np.ndarray:
"""A helper function in the setting of PU learning.
Sets all P(label=class_without_noise|true_label=any_other_class) = 0
in noise_matrix for pulearning setting, where we have
generalized the positive class in PU learning to be any
class of choosing, denoted by class_without_noise.
Parameters
----------
noise_matrix : np.ndarray of shape (K, K), K = number of classes
A conditional probability matrix of the form P(label=k_s|true_label=k_y) containing
the fraction of examples in every class, labeled as every other class.
Assumes columns of noise_matrix sum to 1.
class_without_noise : int
Integer value of the class that has no noise. Traditionally,
this is 1 (positive) for PU learning."""
# Number of classes
K = len(noise_matrix)
cwn = class_without_noise
x = np.copy(noise_matrix)
# Set P( labels = cwn | y != cwn) = 0 (no noise)
class_arange = np.arange(K)
x[cwn, class_arange[class_arange != cwn]] = 0.0
# Normalize columns by increasing diagonal terms
# Ensures noise_matrix is a valid probability matrix
np.fill_diagonal(x, 1 - (np.sum(x, axis=0) - np.diag(x)))
return x
def clip_noise_rates(noise_matrix: np.ndarray) -> np.ndarray:
"""Clip all noise rates to proper range [0,1), but
do not modify the diagonal terms because they are not
noise rates.
ASSUMES noise_matrix columns sum to 1.
Parameters
----------
noise_matrix : np.ndarray of shape (K, K), K = number of classes
A conditional probability matrix containing the fraction of
examples in every class, labeled as every other class.
Diagonal terms are not noise rates, but are consistency P(label=k|true_label=k)
Assumes columns of noise_matrix sum to 1"""
# Preserve because diagonal entries are not noise rates.
diagonal = np.diagonal(noise_matrix)
# Clip all noise rates (efficiently).
noise_matrix = np.clip(noise_matrix, 0, 0.9999)
# Put unmodified diagonal back.
np.fill_diagonal(noise_matrix, diagonal)
# Re-normalized noise_matrix so that columns sum to one.
noise_matrix = noise_matrix / np.clip(noise_matrix.sum(axis=0), a_min=TINY_VALUE, a_max=None)
return noise_matrix
def clip_values(x, low=0.0, high=1.0, new_sum: Optional[float] = None) -> np.ndarray:
"""Clip all values in p to range [low,high].
Preserves sum of x.
Parameters
----------
x : np.ndarray
An array / list of values to be clipped.
low : float
values in x greater than 'low' are clipped to this value
high : float
values in x greater than 'high' are clipped to this value
new_sum : float
normalizes x after clipping to sum to new_sum
Returns
-------
x : np.ndarray
A list of clipped values, summing to the same sum as x."""
if len(x.shape) > 1:
raise TypeError(
f"only size-1 arrays can be converted to Python scalars but 'x' had shape {x.shape}"
)
prev_sum = np.sum(x) if new_sum is None else new_sum # Store previous sum
x = np.clip(x, low, high) # Clip all values (efficiently)
x = (
x * prev_sum / np.clip(np.sum(x), a_min=TINY_VALUE, a_max=None)
) # Re-normalized values to sum to previous sum
return x
def value_counts(x, *, num_classes: Optional[int] = None, multi_label=False) -> np.ndarray:
"""Returns an np.ndarray of shape (K, 1), with the
value counts for every unique item in the labels list/array,
where K is the number of unique entries in labels.
Works for both single-labeled and multi-labeled data.
Parameters
----------
x : list or np.ndarray (one dimensional)
A list of discrete objects, like lists or strings, for
example, class labels 'y' when training a classifier.
e.g. ["dog","dog","cat"] or [1,2,0,1,1,0,2]
num_classes : int (default: None)
Setting this fills the value counts for missing classes with zeros.
For example, if x = [0, 0, 1, 1, 3] then setting ``num_classes=5`` returns
[2, 2, 0, 1, 0] whereas setting ``num_classes=None`` would return [2, 2, 1]. This assumes
your labels come from the set [0, 1,... num_classes=1] even if some classes are missing.
multi_label : bool, optional
If ``True``, labels should be an iterable (e.g. list) of iterables, containing a
list of labels for each example, instead of just a single label.
Assumes all classes in pred_probs.shape[1] are represented in labels.
The multi-label setting supports classification tasks where an example has 1 or more labels.
Example of a multi-labeled `labels` input: ``[[0,1], [1], [0,2], [0,1,2], [0], [1], ...]``.
The major difference in how this is calibrated versus single-label is that
the total number of errors considered is based on the number of labels,
not the number of examples. So, the calibrated `confident_joint` will sum
to the number of total labels."""
# Efficient method if x is pd.Series, np.ndarray, or list
if multi_label:
x = [z for lst in x for z in lst] # Flatten
unique_classes, counts = np.unique(x, return_counts=True)
# Early exit if num_classes is not provided or redundant
if num_classes is None or num_classes == len(unique_classes):
return counts
# Else, there are missing classes
labels_are_integers = np.issubdtype(np.array(x).dtype, np.integer)
if labels_are_integers and num_classes <= np.max(unique_classes):
raise ValueError(f"Required: num_classes > max(x), but {num_classes} <= {np.max(x)}.")
# Add zero counts for all missing classes in [0, 1,..., num_classes-1]
total_counts = np.zeros(num_classes, dtype=int)
# Fill in counts for classes that are present.
# If labels are integers, unique_classes can be used directly as indices to place counts
# into the correct positions in total_counts array.
# If labels are strings, use a slice to fill counts sequentially since strings do not map to indices.
count_ids = unique_classes if labels_are_integers else slice(len(unique_classes))
total_counts[count_ids] = counts
# Return counts with zeros for all missing classes.
return total_counts
def value_counts_fill_missing_classes(x, num_classes, *, multi_label=False) -> np.ndarray:
"""Same as ``internal.util.value_counts`` but requires that num_classes is provided and
always fills missing classes with zero counts.
See ``internal.util.value_counts`` for parameter docstrings."""
return value_counts(x, num_classes=num_classes, multi_label=multi_label)
def get_missing_classes(labels, *, pred_probs=None, num_classes=None, multi_label=False):
"""Find which classes are present in ``pred_probs`` but not present in ``labels``.
See ``count.compute_confident_joint`` for parameter docstrings."""
if pred_probs is None and num_classes is None:
raise ValueError("Both pred_probs and num_classes are None. You must provide exactly one.")
if pred_probs is not None and num_classes is not None:
raise ValueError("Both pred_probs and num_classes are not None. Only one may be provided.")
if num_classes is None:
num_classes = pred_probs.shape[1]
unique_classes = get_unique_classes(labels, multi_label=multi_label)
return sorted(set(range(num_classes)).difference(unique_classes))
def round_preserving_sum(iterable) -> np.ndarray:
"""Rounds an iterable of floats while retaining the original summed value.
The name of each parameter is required. The type and description of each
parameter is optional, but should be included if not obvious.
The while loop in this code was adapted from:
https://github.com/cgdeboer/iteround
Parameters
-----------
iterable : list<float> or np.ndarray<float>
An iterable of floats
Returns
-------
list<int> or np.ndarray<int>
The iterable rounded to int, preserving sum."""
floats = np.asarray(iterable, dtype=float)
ints = floats.round()
orig_sum = np.sum(floats).round()
int_sum = np.sum(ints).round()
# Adjust the integers so that they sum to orig_sum
while abs(int_sum - orig_sum) > FLOATING_POINT_COMPARISON:
diff = np.round(orig_sum - int_sum)
increment = -1 if int(diff < 0.0) else 1
changes = min(int(abs(diff)), len(iterable))
# Orders indices by difference. Increments # of changes.
indices = np.argsort(floats - ints)[::-increment][:changes]
for i in indices:
ints[i] = ints[i] + increment
int_sum = np.sum(ints).round()
return ints.astype(int)
def round_preserving_row_totals(confident_joint) -> np.ndarray:
"""Rounds confident_joint cj to type int
while preserving the totals of reach row.
Assumes that cj is a 2D np.ndarray of type float.
Parameters
----------
confident_joint : 2D np.ndarray<float> of shape (K, K)
See compute_confident_joint docstring for details.
Returns
-------
confident_joint : 2D np.ndarray<int> of shape (K,K)
Rounded to int while preserving row totals."""
return np.apply_along_axis(
func1d=round_preserving_sum,
axis=1,
arr=confident_joint,
).astype(int)
def estimate_pu_f1(s, prob_s_eq_1) -> float:
"""Computes Claesen's estimate of f1 in the pulearning setting.
Parameters
----------
s : iterable (list or np.ndarray)
Binary label (whether each element is labeled or not) in pu learning.
prob_s_eq_1 : iterable (list or np.ndarray)
The probability, for each example, whether it has label=1 P(label=1|x)
Output (float)
------
Claesen's estimate for f1 in the pulearning setting."""
pred = np.asarray(prob_s_eq_1) >= 0.5
true_positives = sum((np.asarray(s) == 1) & (np.asarray(pred) == 1))
all_positives = sum(s)
recall = true_positives / float(all_positives)
frac_positive = sum(pred) / float(len(s))
return recall**2 / (2.0 * frac_positive) if frac_positive != 0 else np.nan
def confusion_matrix(true, pred) -> np.ndarray:
"""Implements a confusion matrix for true labels
and predicted labels. true and pred MUST BE the same length
and have the same distinct set of class labels represented.
Results are identical (and similar computation time) to:
"sklearn.metrics.confusion_matrix"
However, this function avoids the dependency on sklearn.
Parameters
----------
true : np.ndarray 1d
Contains labels.
Assumes true and pred contains the same set of distinct labels.
pred : np.ndarray 1d
A discrete vector of noisy labels, i.e. some labels may be erroneous.
*Format requirements*: for dataset with K classes, labels must be in {0,1,...,K-1}.
Returns
-------
confusion_matrix : np.ndarray (2D)
matrix of confusion counts with true on rows and pred on columns."""
assert len(true) == len(pred)
true_classes = np.unique(true)
pred_classes = np.unique(pred)
K_true = len(true_classes) # Number of classes in true
K_pred = len(pred_classes) # Number of classes in pred
map_true = dict(zip(true_classes, range(K_true)))
map_pred = dict(zip(pred_classes, range(K_pred)))
result = np.zeros((K_true, K_pred))
for i in range(len(true)):
result[map_true[true[i]]][map_pred[pred[i]]] += 1
return result
def print_square_matrix(
matrix,
left_name="s",
top_name="y",
title=" A square matrix",
short_title="s,y",
round_places=2,
):
"""Pretty prints a matrix.
Parameters
----------
matrix : np.ndarray
the matrix to be printed
left_name : str
the name of the variable on the left of the matrix
top_name : str
the name of the variable on the top of the matrix
title : str
Prints this string above the printed square matrix.
short_title : str
A short title (6 characters or fewer) like P(labels|y) or P(labels,y).
round_places : int
Number of decimals to show for each matrix value."""
short_title = short_title[:6]
K = len(matrix) # Number of classes
# Make sure matrix is 2d array
if len(np.shape(matrix)) == 1:
matrix = np.array([matrix])
print()
print(title, "of shape", matrix.shape)
print(" " + short_title + "".join(["\t" + top_name + "=" + str(i) for i in range(K)]))
print("\t---" * K)
for i in range(K):
entry = "\t".join([str(z) for z in list(matrix.round(round_places)[i, :])])
print(left_name + "=" + str(i) + " |\t" + entry)
print("\tTrace(matrix) =", np.round(np.trace(matrix), round_places))
print()
def print_noise_matrix(noise_matrix, round_places=2):
"""Pretty prints the noise matrix."""
print_square_matrix(
noise_matrix,
title=" Noise Matrix (aka Noisy Channel) P(given_label|true_label)",
short_title="p(s|y)",
round_places=round_places,
)
def print_inverse_noise_matrix(inverse_noise_matrix, round_places=2):
"""Pretty prints the inverse noise matrix."""
print_square_matrix(
inverse_noise_matrix,
left_name="y",
top_name="s",
title=" Inverse Noise Matrix P(true_label|given_label)",
short_title="p(y|s)",
round_places=round_places,
)
def print_joint_matrix(joint_matrix, round_places=2):
"""Pretty prints the joint label noise matrix."""
print_square_matrix(
joint_matrix,
title=" Joint Label Noise Distribution Matrix P(given_label, true_label)",
short_title="p(s,y)",
round_places=round_places,
)
def compress_int_array(int_array, num_possible_values) -> np.ndarray:
"""Compresses dtype of np.ndarray<int> if num_possible_values is small enough."""
try:
compressed_type = None
if num_possible_values < np.iinfo(np.dtype("int16")).max:
compressed_type = "int16"
elif num_possible_values < np.iinfo(np.dtype("int32")).max: # pragma: no cover
compressed_type = "int32" # pragma: no cover
if compressed_type is not None:
int_array = int_array.astype(compressed_type)
return int_array
except Exception: # int_array may not even be numpy array, keep as is then
return int_array
def train_val_split(
X, labels, train_idx, holdout_idx
) -> Tuple[DatasetLike, DatasetLike, LabelLike, LabelLike]:
"""Splits data into training/validation sets based on given indices"""
labels_train, labels_holdout = (
labels[train_idx],
labels[holdout_idx],
) # labels are always np.ndarray
split_completed = False
if isinstance(X, (pd.DataFrame, pd.Series)):
X_train, X_holdout = X.iloc[train_idx], X.iloc[holdout_idx]
split_completed = True
if not split_completed:
try: # check if X is pytorch Dataset object using lazy import
import torch
if isinstance(X, torch.utils.data.Dataset): # special splitting for pytorch Dataset
X_train = torch.utils.data.Subset(X, train_idx)
X_holdout = torch.utils.data.Subset(X, holdout_idx)
split_completed = True
except Exception:
pass
if not split_completed:
try:
X_train, X_holdout = X[train_idx], X[holdout_idx]
except Exception:
raise ValueError(
"Cleanlab cannot split this form of dataset (required for cross-validation). "
"Try a different data format, "
"or implement the cross-validation yourself and instead provide out-of-sample `pred_probs`"
)
return X_train, X_holdout, labels_train, labels_holdout
def subset_X_y(X, labels, mask) -> Tuple[DatasetLike, LabelLike]:
"""Extracts subset of features/labels where mask is True"""
labels = subset_labels(labels, mask)
X = subset_data(X, mask)
return X, labels
def subset_labels(labels, mask) -> Union[list, np.ndarray, pd.Series]:
"""Extracts subset of labels where mask is True"""
try: # filtering labels as if it is array or DataFrame
return labels[mask]
except Exception:
try: # filtering labels as if it is list
return [l for idx, l in enumerate(labels) if mask[idx]]
except Exception:
raise TypeError("labels must be 1D np.ndarray, list, or pd.Series.")
def subset_data(X, mask) -> DatasetLike:
"""Extracts subset of data examples where mask (np.ndarray) is True"""
try:
import torch
if isinstance(X, torch.utils.data.Dataset):
mask_idx_list = list(np.nonzero(mask)[0])
return torch.utils.data.Subset(X, mask_idx_list)
except Exception:
pass
try:
return X[mask]
except Exception:
raise TypeError("Data features X must be subsettable with boolean mask array: X[mask]")
def is_torch_dataset(X) -> bool:
try:
import torch
if isinstance(X, torch.utils.data.Dataset):
return True
except Exception:
pass
return False # assumes this cannot be torch dataset if torch cannot be imported
def csr_vstack(a, b) -> DatasetLike:
"""Takes in 2 csr_matrices and appends the second one to the bottom of the first one.
Alternative to scipy.sparse.vstack. Returns a sparse matrix.
"""
a.data = np.hstack((a.data, b.data))
a.indices = np.hstack((a.indices, b.indices))
a.indptr = np.hstack((a.indptr, (b.indptr + a.nnz)[1:]))
a._shape = (a.shape[0] + b.shape[0], b.shape[1])
return a
def append_extra_datapoint(to_data, from_data, index) -> DatasetLike:
"""Appends an extra datapoint to the data object ``to_data``.
This datapoint is taken from the data object ``from_data`` at the corresponding index.
One place this could be useful is ensuring no missing classes after train/validation split.
"""
if not (type(from_data) is type(to_data)):
raise ValueError("Cannot append datapoint from different type of data object.")
if isinstance(to_data, np.ndarray):
return np.vstack([to_data, from_data[index]])
elif isinstance(from_data, (pd.DataFrame, pd.Series)):
X_extra = from_data.iloc[[index]] # type: ignore
to_data = pd.concat([to_data, X_extra])
return to_data.reset_index(drop=True)
else:
try:
X_extra = from_data[index]
try:
return to_data.append(X_extra)
except Exception: # special append for sparse matrix
return csr_vstack(to_data, X_extra)
except Exception:
raise TypeError("Data features X must support: X.append(X[i])")
def get_num_classes(labels=None, pred_probs=None, label_matrix=None, multi_label=None) -> int:
"""Determines the number of classes based on information considered in a
canonical ordering. label_matrix can be: noise_matrix, inverse_noise_matrix, confident_joint,
or any other K x K matrix where K = number of classes.
"""
if pred_probs is not None: # pred_probs is number 1 source of truth
return pred_probs.shape[1]
if label_matrix is not None: # matrix dimension is number 2 source of truth
if label_matrix.shape[0] != label_matrix.shape[1]:
raise ValueError(f"label matrix must be K x K, not {label_matrix.shape}")
else:
return label_matrix.shape[0]
if labels is None:
raise ValueError("Cannot determine number of classes from None input")
return num_unique_classes(labels, multi_label=multi_label)
def num_unique_classes(labels, multi_label=None) -> int:
"""Finds the number of unique classes for both single-labeled
and multi-labeled labels. If multi_label is set to None (default)
this method will infer if multi_label is True or False based on
the format of labels.
This allows for a more general form of multiclass labels that looks
like this: [1, [1,2], [0], [0, 1], 2, 1]"""
return len(get_unique_classes(labels, multi_label))
def get_unique_classes(labels, multi_label=None) -> set:
"""Returns the set of unique classes for both single-labeled
and multi-labeled labels. If multi_label is set to None (default)
this method will infer if multi_label is True or False based on
the format of labels.
This allows for a more general form of multiclass labels that looks
like this: [1, [1,2], [0], [0, 1], 2, 1]"""
if multi_label is None:
multi_label = any(isinstance(l, list) for l in labels)
if multi_label:
return set(l for grp in labels for l in list(grp))
else:
return set(labels)
def format_labels(labels: LabelLike) -> Tuple[np.ndarray, dict]:
"""Takes an array of labels and formats it such that labels are in the set ``0, 1, ..., K-1``,
where ``K`` is the number of classes. The labels are assigned based on lexicographic order.
This is useful for mapping string class labels to the integer format required by many cleanlab (and sklearn) functions.
Returns
-------
formatted_labels
Returns np.ndarray of shape ``(N,)``. The return labels will be properly formatted and can be passed to other cleanlab functions.
mapping
A dictionary showing the mapping of new to old labels, such that ``mapping[k]`` returns the name of the k-th class.
"""
labels = labels_to_array(labels)
if labels.ndim != 1:
raise ValueError("labels must be 1D numpy array.")
unique_labels = np.unique(labels)
label_map = {label: i for i, label in enumerate(unique_labels)}
formatted_labels = np.array([label_map[l] for l in labels])
inverse_map = {i: label for label, i in label_map.items()}
return formatted_labels, inverse_map
def smart_display_dataframe(df): # pragma: no cover
"""Display a pandas dataframe if in a jupyter notebook, otherwise print it to console."""
try:
from IPython.display import display
display(df)
except Exception:
print(df)
def force_two_dimensions(X) -> DatasetLike:
"""
Enforce the dimensionality of a dataset to two dimensions for the use of CleanLearning default classifier,
which is `sklearn.linear_model.LogisticRegression
<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html>`_.
Parameters
----------
X : np.ndarray or DatasetLike
Returns
-------
X : np.ndarray or DatasetLike
The original dataset reduced to two dimensions, so that the dataset will have the shape ``(N, sum(...))``,
where N is still the number of examples.
"""
if X is not None and len(X.shape) > 2:
X = X.reshape((len(X), -1))
return X
cleanlab/internal/validation.py
+212
View File
@@ -0,0 +1,212 @@
"""
Checks to ensure valid inputs for various methods.
"""
from cleanlab.typing import LabelLike, DatasetLike
from cleanlab.internal.constants import FLOATING_POINT_COMPARISON
from typing import Any, List, Optional, Union
import warnings
import numpy as np
import pandas as pd
def assert_valid_inputs(
X: DatasetLike,
y: LabelLike,
pred_probs: Optional[np.ndarray] = None,
multi_label: bool = False,
allow_missing_classes: bool = True,
allow_one_class: bool = False,
) -> None:
"""Checks that ``X``, ``labels``, ``pred_probs`` are correctly formatted."""
if not isinstance(y, (list, np.ndarray, np.generic, pd.Series, pd.DataFrame)):
raise TypeError("labels should be a numpy array or pandas Series.")
if not multi_label:
y = labels_to_array(y)
assert_valid_class_labels(
y=y, allow_missing_classes=allow_missing_classes, allow_one_class=allow_one_class
)
allow_empty_X = True
if pred_probs is None:
allow_empty_X = False
if not allow_empty_X:
assert_nonempty_input(X)
try:
num_examples = len(X)
len_supported = True
except:
len_supported = False
if not len_supported:
try:
num_examples = X.shape[0]
shape_supported = True
except:
shape_supported = False
if (not len_supported) and (not shape_supported):
raise TypeError("Data features X must support either: len(X) or X.shape[0]")
if num_examples != len(y):
raise ValueError(
f"X and labels must be same length, but X is length {num_examples} and labels is length {len(y)}."
)
assert_indexing_works(X, length_X=num_examples)
if pred_probs is not None:
if not isinstance(pred_probs, (np.ndarray, np.generic)):
raise TypeError("pred_probs must be a numpy array.")
if len(pred_probs) != len(y):
raise ValueError("pred_probs and labels must have same length.")
if len(pred_probs.shape) != 2:
raise ValueError("pred_probs array must have shape: num_examples x num_classes.")
if not multi_label:
assert isinstance(y, np.ndarray)
highest_class = max(y) + 1
else:
assert isinstance(y, list)
assert all(isinstance(y_i, list) for y_i in y)
highest_class = max([max(y_i) for y_i in y if len(y_i) != 0]) + 1
if pred_probs.shape[1] < highest_class:
raise ValueError(
f"pred_probs must have at least {highest_class} columns, based on the largest class index which appears in labels."
)
# Check for valid probabilities.
if (np.min(pred_probs) < 0 - FLOATING_POINT_COMPARISON) or (
np.max(pred_probs) > 1 + FLOATING_POINT_COMPARISON
):
raise ValueError("Values in pred_probs must be between 0 and 1.")
if X is not None:
warnings.warn("When X and pred_probs are both provided, the former may be ignored.")
def assert_valid_class_labels(
y: np.ndarray,
allow_missing_classes: bool = True,
allow_one_class: bool = False,
) -> None:
"""Checks that ``labels`` is properly formatted, i.e. a 1D numpy array where labels are zero-based
integers (not multi-label).
"""
if y.ndim != 1:
raise ValueError("Labels must be 1D numpy array.")
if any([isinstance(label, str) for label in y]):
raise ValueError(
"Labels cannot be strings, they must be zero-indexed integers corresponding to class indices."
)
if not np.equal(np.mod(y, 1), 0).all(): # check that labels are integers
raise ValueError("Labels must be zero-indexed integers corresponding to class indices.")
if min(y) < 0:
raise ValueError("Labels must be positive integers corresponding to class indices.")
unique_classes = np.unique(y)
if (not allow_one_class) and (len(unique_classes) < 2):
raise ValueError("Labels must contain at least 2 classes.")
if not allow_missing_classes:
if (unique_classes != np.arange(len(unique_classes))).any():
msg = "cleanlab requires zero-indexed integer labels (0,1,2,..,K-1), but in "
msg += "your case: np.unique(labels) = {}. ".format(str(unique_classes))
msg += "Every class in (0,1,2,..,K-1) must be present in labels as well."
raise TypeError(msg)
def assert_nonempty_input(X: Any) -> None:
"""Ensures input is not None."""
if X is None:
raise ValueError("Data features X cannot be None. Currently X is None.")
def assert_indexing_works(
X: DatasetLike, idx: Optional[List[int]] = None, length_X: Optional[int] = None
) -> None:
"""Ensures we can do list-based indexing into ``X`` and ``y``.
``length_X`` is an optional argument since sparse matrix ``X``
does not support: ``len(X)`` and we want this method to work for sparse ``X``
(in addition to many other types of ``X``).
"""
if idx is None:
if length_X is None:
length_X = 2 # pragma: no cover
idx = [0, length_X - 1]
is_indexed = False
try:
if isinstance(X, (pd.DataFrame, pd.Series)):
_ = X.iloc[idx] # type: ignore[call-overload]
is_indexed = True
except Exception:
pass
if not is_indexed:
try: # check if X is pytorch Dataset object using lazy import
import torch
if isinstance(X, torch.utils.data.Dataset): # indexing for pytorch Dataset
_ = torch.utils.data.Subset(X, idx) # type: ignore[call-overload]
is_indexed = True
except Exception:
pass
if not is_indexed:
try:
_ = X[idx] # type: ignore[call-overload]
except Exception:
msg = (
"Data features X must support list-based indexing; i.e. one of these must work: \n"
)
msg += "1) X[index_list] where say index_list = [0,1,3,10], or \n"
msg += "2) X.iloc[index_list] if X is pandas DataFrame."
raise TypeError(msg)
def labels_to_array(y: Union[LabelLike, np.generic]) -> np.ndarray:
"""Converts different types of label objects to 1D numpy array and checks their validity.
Parameters
----------
y : Union[LabelLike, np.generic]
Labels to convert to 1D numpy array. Can be a list, numpy array, pandas Series, or pandas DataFrame.
Returns
-------
labels_array : np.ndarray
1D numpy array of labels.
"""
if isinstance(y, pd.Series):
y_series: np.ndarray = y.to_numpy()
return y_series
elif isinstance(y, pd.DataFrame):
y_arr = y.values
assert isinstance(y_arr, np.ndarray)
if y_arr.shape[1] != 1:
raise ValueError("labels must be one dimensional.")
return y_arr.flatten()
else: # y is list, np.ndarray, or some other tuple-like object
try:
return np.asarray(y)
except:
raise ValueError(
"List of labels must be convertable to 1D numpy array via: np.ndarray(labels)."
)
def labels_to_list_multilabel(y: List) -> List[List[int]]:
"""Converts different types of label objects to nested list and checks their validity.
Parameters
----------
y : List
Labels to convert to nested list. Supports only list type.
Returns
-------
labels_list : List[List[int]]
Nested list of labels.
"""
if not isinstance(y, list):
raise ValueError("Unsupported Label format")
if not all(isinstance(x, list) for x in y):
raise ValueError("Each element in list of labels must be a list.")
return y
cleanlab/models/README.md
+9
View File
@@ -0,0 +1,9 @@
# Useful models adapted for use with cleanlab
Methods in this ``models`` module are not guaranteed to be stable between different ``cleanlab`` versions.
Some of these files include various models that can be used with cleanlab to find issues in specific types of data. These require dependencies on deep learning and other machine learning packages that are not official cleanlab dependencies. You must install these dependencies on your own if you wish to use them.
The dependencies are as follows:
* keras.py - a wrapper to make any Keras model compatible with cleanlab and sklearn
- tensorflow
cleanlab/models/__init__.py
View File
cleanlab/multiannotator.py
+1923
View File
File diff suppressed because it is too large Load Diff
cleanlab/multilabel_classification/__init__.py
+4
View File
@@ -0,0 +1,4 @@
from .rank import get_label_quality_scores
from . import rank
from . import dataset
from . import filter
cleanlab/multilabel_classification/dataset.py
+326
View File
@@ -0,0 +1,326 @@
"""
Methods to summarize overall labeling issues across a multi-label classification dataset.
Here each example can belong to one or more classes, or none of the classes at all.
Unlike in standard multi-class classification, model-predicted class probabilities need not sum to 1 for each row in multi-label classification.
"""
import pandas as pd
import numpy as np
from typing import Optional, cast, Dict, Any # noqa: F401
from cleanlab.multilabel_classification.filter import (
find_multilabel_issues_per_class,
find_label_issues,
)
from cleanlab.internal.multilabel_utils import get_onehot_num_classes
from collections import defaultdict
def common_multilabel_issues(
labels=list,
pred_probs=None,
*,
class_names=None,
confident_joint=None,
) -> pd.DataFrame:
"""Summarizes which classes in a multi-label dataset appear most often mislabeled overall.
Since classes are not mutually exclusive in multi-label classification, this method summarizes the label issues for each class independently of the others.
Parameters
----------
labels : List[List[int]]
List of noisy labels for multi-label classification where each example can belong to multiple classes.
Refer to documentation for this argument in :py:func:`multilabel_classification.filter.find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted class probabilities.
Refer to documentation for this argument in :py:func:`multilabel_classification.filter.find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
class_names : Iterable[str], optional
A list or other iterable of the string class names. Its order must match the label indices.
If class 0 is 'dog' and class 1 is 'cat', then ``class_names = ['dog', 'cat']``.
If provided, the returned DataFrame will have an extra *Class Name* column with this info.
confident_joint : np.ndarray, optional
An array of shape ``(K, 2, 2)`` representing a one-vs-rest formatted confident joint.
Refer to documentation for this argument in :py:func:`multilabel_classification.filter.find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for details.
Returns
-------
common_multilabel_issues : pd.DataFrame
DataFrame where each row corresponds to a class summarized by the following columns:
- *Class Name*: The name of the class if class_names is provided.
- *Class Index*: The index of the class.
- *In Given Label*: Whether the Class is originally annotated True or False in the given label.
- *In Suggested Label*: Whether the Class should be True or False in the suggested label (based on model's prediction).
- *Num Examples*: Number of examples flagged as a label issue where this Class is True/False "In Given Label" but cleanlab estimates the annotation should actually be as specified "In Suggested Label". I.e. the number of examples in your dataset where this Class was labeled as True but likely should have been False (or vice versa).
- *Issue Probability*: The *Num Examples* column divided by the total number of examples in the dataset; i.e. the relative overall frequency of each type of label issue in your dataset.
By default, the rows in this DataFrame are ordered by "Issue Probability" (descending).
"""
num_examples = _get_num_examples_multilabel(labels=labels, confident_joint=confident_joint)
summary_issue_counts = defaultdict(list)
y_one, num_classes = get_onehot_num_classes(labels, pred_probs)
label_issues_list, labels_list, pred_probs_list = find_multilabel_issues_per_class(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
return_indices_ranked_by="self_confidence",
)
for class_num, (label, issues_for_class) in enumerate(zip(y_one.T, label_issues_list)):
binary_label_issues = np.zeros(len(label)).astype(bool)
binary_label_issues[issues_for_class] = True
true_but_false_count = sum(np.logical_and(label == 1, binary_label_issues))
false_but_true_count = sum(np.logical_and(label == 0, binary_label_issues))
if class_names is not None:
summary_issue_counts["Class Name"].append(class_names[class_num])
summary_issue_counts["Class Index"].append(class_num)
summary_issue_counts["In Given Label"].append(True)
summary_issue_counts["In Suggested Label"].append(False)
summary_issue_counts["Num Examples"].append(true_but_false_count)
summary_issue_counts["Issue Probability"].append(true_but_false_count / num_examples)
if class_names is not None:
summary_issue_counts["Class Name"].append(class_names[class_num])
summary_issue_counts["Class Index"].append(class_num)
summary_issue_counts["In Given Label"].append(False)
summary_issue_counts["In Suggested Label"].append(True)
summary_issue_counts["Num Examples"].append(false_but_true_count)
summary_issue_counts["Issue Probability"].append(false_but_true_count / num_examples)
return (
pd.DataFrame.from_dict(summary_issue_counts)
.sort_values(by=["Issue Probability"], ascending=False)
.reset_index(drop=True)
)
def rank_classes_by_multilabel_quality(
labels=None,
pred_probs=None,
*,
class_names=None,
joint=None,
confident_joint=None,
) -> pd.DataFrame:
"""
Returns a DataFrame with three overall label quality scores per class for a multi-label dataset.
These numbers summarize all examples annotated with the class (details listed below under the Returns parameter).
By default, classes are ordered by "Label Quality Score", so the most problematic classes are reported first in the DataFrame.
Score values are unnormalized and may be very small. What matters is their relative ranking across the classes.
**Parameters**:
For information about the arguments to this method, see the documentation of
`~cleanlab.multilabel_classification.dataset.common_multilabel_issues`.
Returns
-------
overall_label_quality : pd.DataFrame
Pandas DataFrame with one row per class and columns: "Class Index", "Label Issues",
"Inverse Label Issues", "Label Issues", "Inverse Label Noise", "Label Quality Score".
Some entries are overall quality scores between 0 and 1, summarizing how good overall the labels
appear to be for that class (lower values indicate more erroneous labels).
Other entries are estimated counts of annotation errors related to this class.
Here is what each column represents:
- *Class Name*: The name of the class if class_names is provided.
- *Class Index*: The index of the class in 0, 1, ..., K-1.
- *Label Issues*: Estimated number of examples in the dataset that are labeled as belonging to class k but actually should not belong to this class.
- *Inverse Label Issues*: Estimated number of examples in the dataset that should actually be labeled as class k but did not receive this label.
- *Label Noise*: Estimated proportion of examples in the dataset that are labeled as class k but should not be. For each class k: this is computed by dividing the number of examples with "Label Issues" that were labeled as class k by the total number of examples labeled as class k.
- *Inverse Label Noise*: Estimated proportion of examples in the dataset that should actually be labeled as class k but did not receive this label.
- *Label Quality Score*: Estimated proportion of examples labeled as class k that have been labeled correctly, i.e. ``1 - label_noise``.
By default, the DataFrame is ordered by "Label Quality Score" (in ascending order), so the classes with the most label issues appear first.
"""
issues_df = common_multilabel_issues(
labels=labels, pred_probs=pred_probs, class_names=class_names, confident_joint=joint
)
issues_dict = defaultdict(defaultdict) # type: Dict[str, Any]
num_examples = _get_num_examples_multilabel(labels=labels, confident_joint=confident_joint)
return_columns = [
"Class Name",
"Class Index",
"Label Issues",
"Inverse Label Issues",
"Label Noise",
"Inverse Label Noise",
"Label Quality Score",
]
if class_names is None:
return_columns = return_columns[1:]
for class_num, row in issues_df.iterrows():
if row["In Given Label"]:
if class_names is not None:
issues_dict[row["Class Index"]]["Class Name"] = row["Class Name"]
issues_dict[row["Class Index"]]["Label Issues"] = int(
row["Issue Probability"] * num_examples
)
issues_dict[row["Class Index"]]["Label Noise"] = row["Issue Probability"]
issues_dict[row["Class Index"]]["Label Quality Score"] = (
1 - issues_dict[row["Class Index"]]["Label Noise"]
)
else:
if class_names is not None:
issues_dict[row["Class Index"]]["Class Name"] = row["Class Name"]
issues_dict[row["Class Index"]]["Inverse Label Issues"] = int(
row["Issue Probability"] * num_examples
)
issues_dict[row["Class Index"]]["Inverse Label Noise"] = row["Issue Probability"]
issues_df_dict = defaultdict(list)
for i in issues_dict:
issues_df_dict["Class Index"].append(i)
for j in issues_dict[i]:
issues_df_dict[j].append(issues_dict[i][j])
return (
pd.DataFrame.from_dict(issues_df_dict)
.sort_values(by="Label Quality Score", ascending=True)
.reset_index(drop=True)
)[return_columns]
def _get_num_examples_multilabel(labels=None, confident_joint: Optional[np.ndarray] = None) -> int:
"""Helper method that finds the number of examples from the parameters or throws an error
if neither parameter is provided.
Parameters
----------
For parameter info, see the docstring of `~cleanlab.multilabel_classification.dataset.common_multilabel_issues`.
Returns
-------
num_examples : int
The number of examples in the dataset.
Raises
------
ValueError
If `labels` is None.
"""
if labels is None and confident_joint is None:
raise ValueError(
"Error: num_examples is None. You must either provide confident_joint, "
"or provide both num_example and joint as input parameters."
)
_confident_joint = cast(np.ndarray, confident_joint)
num_examples = len(labels) if labels is not None else cast(int, np.sum(_confident_joint[0]))
return num_examples
def overall_multilabel_health_score(
labels=None,
pred_probs=None,
*,
confident_joint=None,
) -> float:
"""Returns a single score between 0 and 1 measuring the overall quality of all labels in a multi-label classification dataset.
Intuitively, the score is the average correctness of the given labels across all examples in the
dataset. So a score of 1 suggests your data is perfectly labeled and a score of 0.5 suggests
half of the examples in the dataset may be incorrectly labeled. Thus, a higher
score implies a higher quality dataset.
**Parameters**: For information about the arguments to this method, see the documentation of
`~cleanlab.multilabel_classification.dataset.common_multilabel_issues`.
Returns
-------
health_score : float
A overall score between 0 and 1, where 1 implies all labels in the dataset are estimated to be correct.
A score of 0.5 implies that half of the dataset's labels are estimated to have issues.
"""
num_examples = _get_num_examples_multilabel(labels=labels)
issues = find_label_issues(
labels=labels, pred_probs=pred_probs, confident_joint=confident_joint
)
return 1.0 - sum(issues) / num_examples
def multilabel_health_summary(
labels=None,
pred_probs=None,
*,
class_names=None,
num_examples=None,
confident_joint=None,
verbose=True,
) -> Dict:
"""Prints a health summary of your multi-label dataset.
This summary includes useful statistics like:
* The classes with the most and least label issues.
* Overall label quality scores, summarizing how accurate the labels appear across the entire dataset.
**Parameters**: For information about the arguments to this method, see the documentation of
`~cleanlab.multilabel_classification.dataset.common_multilabel_issues`.
Returns
-------
summary : dict
A dictionary containing keys (see the corresponding functions' documentation to understand the values):
- ``"overall_label_health_score"``, corresponding to output of `~cleanlab.multilabel_classification.dataset.overall_multilabel_health_score`
- ``"classes_by_multilabel_quality"``, corresponding to output of `~cleanlab.multilabel_classification.dataset.rank_classes_by_multilabel_quality`
- ``"common_multilabel_issues"``, corresponding to output of `~cleanlab.multilabel_classification.dataset.common_multilabel_issues`
"""
from cleanlab.internal.util import smart_display_dataframe
if num_examples is None:
num_examples = _get_num_examples_multilabel(labels=labels)
if verbose:
longest_line = f"| for your dataset with {num_examples:,} examples "
print(
"-" * (len(longest_line) - 1)
+ "\n"
+ f"| Generating a Cleanlab Dataset Health Summary{' ' * (len(longest_line) - 49)}|\n"
+ longest_line
+ f"| Note, Cleanlab is not a medical doctor... yet.{' ' * (len(longest_line) - 51)}|\n"
+ "-" * (len(longest_line) - 1)
+ "\n",
)
df_class_label_quality = rank_classes_by_multilabel_quality(
labels=labels,
pred_probs=pred_probs,
class_names=class_names,
confident_joint=confident_joint,
)
if verbose:
print("Overall Class Quality and Noise across your dataset (below)")
print("-" * 60, "\n", flush=True)
smart_display_dataframe(df_class_label_quality)
df_common_issues = common_multilabel_issues(
labels=labels,
pred_probs=pred_probs,
class_names=class_names,
confident_joint=confident_joint,
)
if verbose:
print(
"\nCommon multilabel issues are" + "\n" + "-" * 83 + "\n",
flush=True,
)
smart_display_dataframe(df_common_issues)
print()
health_score = overall_multilabel_health_score(
labels=labels,
pred_probs=pred_probs,
confident_joint=confident_joint,
)
if verbose:
print("\nGenerated with <3 from Cleanlab.\n")
return {
"overall_multilabel_health_score": health_score,
"classes_by_multilabel_quality": df_class_label_quality,
"common_multilabel_issues": df_common_issues,
}
cleanlab/multilabel_classification/filter.py
+303
View File
@@ -0,0 +1,303 @@
"""
Methods to flag which examples have label issues in multi-label classification datasets.
Here each example can belong to one or more classes, or none of the classes at all.
Unlike in standard multi-class classification, model-predicted class probabilities need not sum to 1 for each row in multi-label classification.
"""
import warnings
import inspect
from typing import Optional, Union, Tuple, List, Any
import numpy as np
def find_label_issues(
labels: list,
pred_probs: np.ndarray,
return_indices_ranked_by: Optional[str] = None,
rank_by_kwargs={},
filter_by: str = "prune_by_noise_rate",
frac_noise: float = 1.0,
num_to_remove_per_class: Optional[List[int]] = None,
min_examples_per_class=1,
confident_joint: Optional[np.ndarray] = None,
n_jobs: Optional[int] = None,
verbose: bool = False,
low_memory: bool = False,
) -> np.ndarray:
"""
Identifies potentially mislabeled examples in a multi-label classification dataset.
An example is flagged as with a label issue if *any* of the classes appear to be incorrectly annotated for this example.
Parameters
----------
labels : List[List[int]]
List of noisy labels for multi-label classification where each example can belong to multiple classes.
This is an iterable of iterables where the i-th element of `labels` corresponds to a list of classes that the i-th example belongs to,
according to the original data annotation (e.g. ``labels = [[1,2],[1],[0],..]``).
This method will return the indices i where the inner list ``labels[i]`` is estimated to have some error.
For a dataset with K classes, each class must be represented as an integer in 0, 1, ..., K-1 within the labels.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted class probabilities.
Each row of this matrix corresponds to an example `x`
and contains the predicted probability that `x` belongs to each possible class,
for each of the K classes (along its columns).
The columns need not sum to 1 but must be ordered such that
these probabilities correspond to class 0, 1, ..., K-1.
Note
----
Estimated label quality scores are most accurate when they are computed based on out-of-sample ``pred_probs`` from your model.
To obtain out-of-sample predicted probabilities for every example in your dataset, you can use :ref:`cross-validation <pred_probs_cross_val>`.
This is encouraged to get better results.
return_indices_ranked_by : {None, 'self_confidence', 'normalized_margin', 'confidence_weighted_entropy'}, default = None
This function can return a boolean mask (if None) or an array of the example-indices with issues sorted based on the specified ranking method.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
rank_by_kwargs : dict, optional
Optional keyword arguments to pass into scoring functions for ranking by
label quality score (see :py:func:`rank.get_label_quality_scores
<cleanlab.rank.get_label_quality_scores>`).
filter_by : {'prune_by_class', 'prune_by_noise_rate', 'both', 'confident_learning', 'predicted_neq_given', 'low_normalized_margin', 'low_self_confidence'}, default='prune_by_noise_rate'
The specific Confident Learning method to determine precisely which examples have label issues in a dataset.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
frac_noise : float, default = 1.0
This will return the "top" frac_noise * num_label_issues estimated label errors, dependent on the filtering method used,
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
num_to_remove_per_class : array_like
An iterable that specifies the number of mislabeled examples to return from each class.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
min_examples_per_class : int, default = 1
The minimum number of examples required per class below which examples from this class will not be flagged as label issues.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
confident_joint : np.ndarray, optional
An array of shape ``(K, 2, 2)`` representing a one-vs-rest formatted confident joint, as is appropriate for multi-label classification tasks.
Entry ``(c, i, j)`` in this array is the number of examples confidently counted into a ``(class c, noisy label=i, true label=j)`` bin,
where `i, j` are either 0 or 1 to denote whether this example belongs to class `c` or not
(recall examples can belong to multiple classes in multi-label classification).
The `confident_joint` can be computed using :py:func:`count.compute_confident_joint <cleanlab.count.compute_confident_joint>` with ``multi_label=True``.
If not provided, it is computed from the given (noisy) `labels` and `pred_probs`.
n_jobs : optional
Number of processing threads used by multiprocessing.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
verbose : optional
If ``True``, prints when multiprocessing happens.
low_memory: bool, default=False
Set as ``True`` if you have a big dataset with limited memory.
Uses :py:func:`experimental.label_issues_batched.find_label_issues_batched <cleanlab.experimental.label_issues_batched>`
Returns
-------
label_issues : np.ndarray
If `return_indices_ranked_by` left unspecified, returns a boolean **mask** for the entire dataset
where ``True`` represents an example suffering from some label issue and
``False`` represents an example that appears accurately labeled.
If `return_indices_ranked_by` is specified, this method instead returns a list of **indices** of examples identified with
label issues (i.e. those indices where the mask would be ``True``).
Indices are sorted by the likelihood that *all* classes are correctly annotated for the corresponding example.
Note
----
Obtain the *indices* of examples with label issues in your dataset by setting
`return_indices_ranked_by`.
"""
from cleanlab.filter import _find_label_issues_multilabel
if low_memory:
if rank_by_kwargs:
warnings.warn(f"`rank_by_kwargs` is not used when `low_memory=True`.")
func_signature = inspect.signature(find_label_issues)
default_args = {
k: v.default
for k, v in func_signature.parameters.items()
if v.default is not inspect.Parameter.empty
}
arg_values = {
"filter_by": filter_by,
"num_to_remove_per_class": num_to_remove_per_class,
"confident_joint": confident_joint,
"n_jobs": n_jobs,
"num_to_remove_per_class": num_to_remove_per_class,
"frac_noise": frac_noise,
"min_examples_per_class": min_examples_per_class,
}
for arg_name, arg_val in arg_values.items():
if arg_val != default_args[arg_name]:
warnings.warn(f"`{arg_name}` is not used when `low_memory=True`.")
return _find_label_issues_multilabel(
labels=labels,
pred_probs=pred_probs,
return_indices_ranked_by=return_indices_ranked_by,
rank_by_kwargs=rank_by_kwargs,
filter_by=filter_by,
frac_noise=frac_noise,
num_to_remove_per_class=num_to_remove_per_class,
min_examples_per_class=min_examples_per_class,
confident_joint=confident_joint,
n_jobs=n_jobs,
verbose=verbose,
low_memory=low_memory,
)
def find_multilabel_issues_per_class(
labels: list,
pred_probs: np.ndarray,
return_indices_ranked_by: Optional[str] = None,
rank_by_kwargs={},
filter_by: str = "prune_by_noise_rate",
frac_noise: float = 1.0,
num_to_remove_per_class: Optional[List[int]] = None,
min_examples_per_class=1,
confident_joint: Optional[np.ndarray] = None,
n_jobs: Optional[int] = None,
verbose: bool = False,
low_memory: bool = False,
) -> Union[np.ndarray, Tuple[List[np.ndarray], List[Any], List[np.ndarray]]]:
"""
Identifies potentially bad labels for each example and each class in a multi-label classification dataset.
Whereas `~cleanlab.multilabel_classification.filter.find_label_issues`
estimates which examples have an erroneous annotation for *any* class, this method estimates which specific classes are incorrectly annotated as well.
This method returns a list of size K, the number of classes in the dataset.
Parameters
----------
labels : List[List[int]]
List of noisy labels for multi-label classification where each example can belong to multiple classes.
Refer to documentation for this argument in `~cleanlab.multilabel_classification.filter.find_label_issues` for further details.
This method will identify whether ``labels[i][k]`` appears correct, for every example ``i`` and class ``k``.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted class probabilities.
Refer to documentation for this argument in `~cleanlab.multilabel_classification.filter.find_label_issues` for further details.
return_indices_ranked_by : {None, 'self_confidence', 'normalized_margin', 'confidence_weighted_entropy'}, default = None
This function can return a boolean mask (if this argument is ``None``) or a sorted array of indices based on the specified ranking method (if not ``None``).
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
rank_by_kwargs : dict, optional
Optional keyword arguments to pass into scoring functions for ranking by.
label quality score (see :py:func:`rank.get_label_quality_scores
<cleanlab.rank.get_label_quality_scores>`).
filter_by : {'prune_by_class', 'prune_by_noise_rate', 'both', 'confident_learning', 'predicted_neq_given', 'low_normalized_margin', 'low_self_confidence'}, default = 'prune_by_noise_rate'
The specific method that can be used to filter or prune examples with label issues from a dataset.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
frac_noise : float, default = 1.0
This will return the "top" frac_noise * num_label_issues estimated label errors, dependent on the filtering method used,
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
num_to_remove_per_class : array_like
This parameter is an iterable that specifies the number of mislabeled examples to return from each class.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
min_examples_per_class : int, default = 1
The minimum number of examples required per class to avoid flagging as label issues.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
confident_joint : np.ndarray, optional
An array of shape ``(K, 2, 2)`` representing a one-vs-rest formatted confident joint.
Refer to documentation for this argument in `~cleanlab.multilabel_classification.filter.find_label_issues` for details.
n_jobs : optional
Number of processing threads used by multiprocessing.
Refer to documentation for this argument in :py:func:`filter.find_label_issues <cleanlab.filter.find_label_issues>` for details.
verbose : optional
If ``True``, prints when multiprocessing happens.
Returns
-------
per_class_label_issues : list(np.ndarray)
By default, this is a list of length K containing the examples where each class appears incorrectly annotated.
``per_class_label_issues[k]`` is a Boolean mask of the same length as the dataset,
where ``True`` values indicate examples where class ``k`` appears incorrectly annotated.
For more details, refer to `~cleanlab.multilabel_classification.filter.find_label_issues`.
Otherwise if `return_indices_ranked_by` is not ``None``, then this method returns 3 objects (each of length K, the number of classes): `label_issues_list`, `labels_list`, `pred_probs_list`.
- *label_issues_list*: an ordered list of indices of examples where class k appears incorrectly annotated, sorted by the likelihood that class k is correctly annotated.
- *labels_list*: a binary one-hot representation of the original labels, useful if you want to compute label quality scores.
- *pred_probs_list*: a one-vs-rest representation of the original predicted probabilities of shape ``(N, 2)``, useful if you want to compute label quality scores.
``pred_probs_list[k][i][0]`` is the estimated probability that example ``i`` belongs to class ``k``, and is equal to: ``1 - pred_probs_list[k][i][1]``.
"""
import cleanlab.filter
from cleanlab.internal.multilabel_utils import get_onehot_num_classes, stack_complement
from cleanlab.experimental.label_issues_batched import find_label_issues_batched
y_one, num_classes = get_onehot_num_classes(labels, pred_probs)
if return_indices_ranked_by is None:
bissues = np.zeros(y_one.shape).astype(bool)
else:
label_issues_list = []
labels_list = []
pred_probs_list = []
if confident_joint is not None and not low_memory:
confident_joint_shape = confident_joint.shape
if confident_joint_shape == (num_classes, num_classes):
warnings.warn(
f"The new recommended format for `confident_joint` in multi_label settings is (num_classes,2,2) as output by compute_confident_joint(...,multi_label=True). Your K x K confident_joint in the old format is being ignored."
)
confident_joint = None
elif confident_joint_shape != (num_classes, 2, 2):
raise ValueError("confident_joint should be of shape (num_classes, 2, 2)")
for class_num, (label, pred_prob_for_class) in enumerate(zip(y_one.T, pred_probs.T)):
pred_probs_binary = stack_complement(pred_prob_for_class)
if low_memory:
quality_score_kwargs = (
{"method": return_indices_ranked_by} if return_indices_ranked_by else None
)
binary_label_issues = find_label_issues_batched(
labels=label,
pred_probs=pred_probs_binary,
verbose=verbose,
quality_score_kwargs=quality_score_kwargs,
return_mask=return_indices_ranked_by is None,
)
else:
if confident_joint is None:
conf = None
else:
conf = confident_joint[class_num]
if num_to_remove_per_class is not None:
ml_num_to_remove_per_class = [num_to_remove_per_class[class_num], 0]
else:
ml_num_to_remove_per_class = None
binary_label_issues = cleanlab.filter.find_label_issues(
labels=label,
pred_probs=pred_probs_binary,
return_indices_ranked_by=return_indices_ranked_by,
frac_noise=frac_noise,
rank_by_kwargs=rank_by_kwargs,
filter_by=filter_by,
num_to_remove_per_class=ml_num_to_remove_per_class,
min_examples_per_class=min_examples_per_class,
confident_joint=conf,
n_jobs=n_jobs,
verbose=verbose,
)
if return_indices_ranked_by is None:
bissues[:, class_num] = binary_label_issues
else:
label_issues_list.append(binary_label_issues)
labels_list.append(label)
pred_probs_list.append(pred_probs_binary)
if return_indices_ranked_by is None:
return bissues
else:
return label_issues_list, labels_list, pred_probs_list
cleanlab/multilabel_classification/rank.py
+179
View File
@@ -0,0 +1,179 @@
"""
Methods to rank the severity of label issues in multi-label classification datasets.
Here each example can belong to one or more classes, or none of the classes at all.
Unlike in standard multi-class classification, model-predicted class probabilities need not sum to 1 for each row in multi-label classification.
"""
from __future__ import annotations
import numpy as np # noqa: F401: Imported for type annotations
from typing import List, TypeVar, Dict, Any, Optional, Tuple, TYPE_CHECKING
from cleanlab.internal.validation import assert_valid_inputs
from cleanlab.internal.util import get_num_classes
from cleanlab.internal.multilabel_utils import int2onehot
from cleanlab.internal.multilabel_scorer import MultilabelScorer, ClassLabelScorer, Aggregator
if TYPE_CHECKING: # pragma: no cover
import numpy.typing as npt
T = TypeVar("T", bound=npt.NBitBase)
def _labels_to_binary(
labels: List[List[int]],
pred_probs: npt.NDArray["np.floating[T]"],
) -> np.ndarray:
"""Validate the inputs to the multilabel scorer. Also transform the labels to a binary representation."""
assert_valid_inputs(
X=None, y=labels, pred_probs=pred_probs, multi_label=True, allow_one_class=True
)
num_classes = get_num_classes(labels=labels, pred_probs=pred_probs, multi_label=True)
binary_labels = int2onehot(labels, K=num_classes)
return binary_labels
def _create_multilabel_scorer(
method: str,
adjust_pred_probs: bool,
aggregator_kwargs: Optional[Dict[str, Any]] = None,
) -> Tuple[MultilabelScorer, Dict]:
"""This function acts as a factory that creates a MultilabelScorer."""
base_scorer = ClassLabelScorer.from_str(method)
base_scorer_kwargs = {"adjust_pred_probs": adjust_pred_probs}
if aggregator_kwargs:
aggregator = Aggregator(**aggregator_kwargs)
scorer = MultilabelScorer(base_scorer, aggregator)
else:
scorer = MultilabelScorer(base_scorer)
return scorer, base_scorer_kwargs
def get_label_quality_scores(
labels: List[List[int]],
pred_probs: npt.NDArray["np.floating[T]"],
*,
method: str = "self_confidence",
adjust_pred_probs: bool = False,
aggregator_kwargs: Dict[str, Any] = {"method": "exponential_moving_average", "alpha": 0.8},
) -> npt.NDArray["np.floating[T]"]:
"""Computes a label quality score for each example in a multi-label classification dataset.
Scores are between 0 and 1 with lower scores indicating examples whose label more likely contains an error.
For each example, this method internally computes a separate score for each individual class
and then aggregates these per-class scores into an overall label quality score for the example.
Parameters
----------
labels : List[List[int]]
List of noisy labels for multi-label classification where each example can belong to multiple classes.
Refer to documentation for this argument in :py:func:`multilabel_classification.filter.find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted class probabilities.
Refer to documentation for this argument in :py:func:`multilabel_classification.filter.find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
method : {"self_confidence", "normalized_margin", "confidence_weighted_entropy"}, default = "self_confidence"
Method to calculate separate per-class annotation scores for an example that are then aggregated into an overall label quality score for the example.
These scores are separately calculated for each class based on the corresponding column of `pred_probs` in a one-vs-rest manner,
and are standard label quality scores for binary classification (based on whether the class should or should not apply to this example).
See also
--------
:py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>` function for details about each option.
adjust_pred_probs : bool, default = False
Account for class imbalance in the label-quality scoring by adjusting predicted probabilities.
Refer to documentation for this argument in :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>` for details.
aggregator_kwargs : dict, default = {"method": "exponential_moving_average", "alpha": 0.8}
A dictionary of hyperparameter values to use when aggregating per-class scores into an overall label quality score for each example.
Options for ``"method"`` include: ``"exponential_moving_average"`` or ``"softmin"`` or your own callable function.
See :py:class:`internal.multilabel_scorer.Aggregator <cleanlab.internal.multilabel_scorer.Aggregator>` for details about each option and other possible hyperparameters.
To get a score for each class annotation for each example, use the `~cleanlab.multilabel_classification.rank.get_label_quality_scores_per_class` method instead.
Returns
-------
label_quality_scores : np.ndarray
A 1D array of shape ``(N,)`` with a label quality score (between 0 and 1) for each example in the dataset.
Lower scores indicate examples whose label is more likely to contain some annotation error (for any of the classes).
Examples
--------
>>> from cleanlab.multilabel_classification import get_label_quality_scores
>>> import numpy as np
>>> labels = [[1], [0,2]]
>>> pred_probs = np.array([[0.1, 0.9, 0.1], [0.4, 0.1, 0.9]])
>>> scores = get_label_quality_scores(labels, pred_probs)
>>> scores
array([0.9, 0.5])
"""
binary_labels = _labels_to_binary(labels, pred_probs)
scorer, base_scorer_kwargs = _create_multilabel_scorer(
method=method,
adjust_pred_probs=adjust_pred_probs,
aggregator_kwargs=aggregator_kwargs,
)
return scorer(binary_labels, pred_probs, base_scorer_kwargs=base_scorer_kwargs)
def get_label_quality_scores_per_class(
labels: List[List[int]],
pred_probs: npt.NDArray["np.floating[T]"],
*,
method: str = "self_confidence",
adjust_pred_probs: bool = False,
) -> np.ndarray:
"""
Computes a quality score quantifying how likely each individual class annotation is correct in a multi-label classification dataset.
This is similar to `~cleanlab.multilabel_classification.rank.get_label_quality_scores`
but instead returns the per-class results without aggregation.
For a dataset with K classes, each example receives K scores from this method.
Refer to documentation in `~cleanlab.multilabel_classification.rank.get_label_quality_scores` for details.
Parameters
----------
labels : List[List[int]]
List of noisy labels for multi-label classification where each example can belong to multiple classes.
Refer to documentation for this argument in :py:func:`find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
pred_probs : np.ndarray
An array of shape ``(N, K)`` of model-predicted class probabilities.
Refer to documentation for this argument in :py:func:`find_label_issues <cleanlab.multilabel_classification.filter.find_label_issues>` for further details.
method : {"self_confidence", "normalized_margin", "confidence_weighted_entropy"}, default = "self_confidence"
Method to calculate separate per-class annotation scores (that quantify how likely a particular class annotation is correct for a particular example).
Refer to documentation for this argument in `~cleanlab.multilabel_classification.rank.get_label_quality_scores` for further details.
adjust_pred_probs : bool, default = False
Account for class imbalance in the label-quality scoring by adjusting predicted probabilities.
Refer to documentation for this argument in :py:func:`rank.get_label_quality_scores <cleanlab.rank.get_label_quality_scores>` for details.
Returns
-------
label_quality_scores : list(np.ndarray)
A list containing K arrays, each of shape (N,). Here K is the number of classes in the dataset and N is the number of examples.
``label_quality_scores[k][i]`` is a score between 0 and 1 quantifying how likely the annotation for class ``k`` is correct for example ``i``.
Examples
--------
>>> from cleanlab.multilabel_classification import get_label_quality_scores
>>> import numpy as np
>>> labels = [[1], [0,2]]
>>> pred_probs = np.array([[0.1, 0.9, 0.1], [0.4, 0.1, 0.9]])
>>> scores = get_label_quality_scores(labels, pred_probs)
>>> scores
array([0.9, 0.5])
"""
binary_labels = _labels_to_binary(labels, pred_probs)
scorer, base_scorer_kwargs = _create_multilabel_scorer(
method=method,
adjust_pred_probs=adjust_pred_probs,
)
return scorer.get_class_label_quality_scores(
labels=binary_labels, pred_probs=pred_probs, base_scorer_kwargs=base_scorer_kwargs
)
cleanlab/object_detection/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from . import rank
from . import filter
from . import summary
cleanlab/object_detection/filter.py
+405
View File
@@ -0,0 +1,405 @@
"""Methods to find label issues in an object detection dataset, where each annotated bounding box in an image receives its own class label."""
from collections import defaultdict
from multiprocessing import Pool
from typing import Any, Dict, List, Optional, Tuple, Union
import numpy as np
from cleanlab.internal.constants import (
ALPHA,
HIGH_PROBABILITY_THRESHOLD,
LOW_PROBABILITY_THRESHOLD,
OVERLOOKED_THRESHOLD_FACTOR,
BADLOC_THRESHOLD_FACTOR,
SWAP_THRESHOLD_FACTOR,
AP_SCALE_FACTOR,
)
from cleanlab.internal.object_detection_utils import assert_valid_inputs
from cleanlab.object_detection.rank import (
_get_valid_inputs_for_compute_scores,
_separate_label,
_separate_prediction,
compute_badloc_box_scores,
compute_overlooked_box_scores,
compute_swap_box_scores,
get_label_quality_scores,
issues_from_scores,
_get_overlap_matrix,
)
def find_label_issues(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
*,
return_indices_ranked_by_score: Optional[bool] = False,
overlapping_label_check: Optional[bool] = True,
) -> np.ndarray:
"""
Identifies potentially mislabeled images in an object detection dataset.
An image is flagged with a label issue if *any* of its bounding boxes appear incorrectly annotated.
This includes images for which a bounding box: should have been annotated but is missing,
has been annotated with the wrong class, or has been annotated in a suboptimal location.
Suppose the dataset has ``N`` images, ``K`` possible class labels.
If ``return_indices_ranked_by_score`` is ``False``, a boolean mask of length ``N`` is returned,
indicating whether each image has a label issue (``True``) or not (``False``).
If ``return_indices_ranked_by_score`` is ``True``, the indices of images flagged with label issues are returned,
sorted with the most likely-mislabeled images ordered first.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
This is a list of ``N`` dictionaries such that ``labels[i]`` contains the given labels for the `i`-th image in the following format:
``{'bboxes': np.ndarray((L,4)), 'labels': np.ndarray((L,)), 'image_name': str}`` where ``L`` is the number of annotated bounding boxes
for the `i`-th image and ``bboxes[l]`` is a bounding box of coordinates in ``[x1,y1,x2,y2]`` format and with given class label ``labels[j]``.
``image_name`` is an optional part of the labels that can be used to later refer to specific images.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`].
For more information on proper labels formatting, check out the `MMDetection library <https://mmdetection.readthedocs.io/en/dev-3.x/advanced_guides/customize_dataset.html>`_.
predictions:
Predictions output by a trained object detection model.
For the most accurate results, predictions should be out-of-sample to avoid overfitting, eg. obtained via :ref:`cross-validation <pred_probs_cross_val>`.
This is a list of ``N`` ``np.ndarray`` such that ``predictions[i]`` corresponds to the model prediction for the `i`-th image.
For each possible class ``k`` in 0, 1, ..., K-1: ``predictions[i][k]`` is a ``np.ndarray`` of shape ``(M,5)``,
where ``M`` is the number of predicted bounding boxes for class ``k``. Here the five columns correspond to ``[x1,y1,x2,y2,pred_prob]``,
where ``[x1,y1,x2,y2]`` are coordinates of the bounding box predicted by the model and ``pred_prob`` is the model's confidence in the predicted class label for this bounding box.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`]. The last column, pred_prob, represents the predicted probability that the bounding box contains an object of the class k.
For more information see the `MMDetection package <https://github.com/open-mmlab/mmdetection>`_ for an example object detection library that outputs predictions in the correct format.
return_indices_ranked_by_score:
Determines what is returned by this method (see description of return value for details).
overlapping_label_check : bool, default = True
If True, boxes annotated with more than one class label have their swap score penalized. Set this to False if you are not concerned when two very similar boxes exist with different class labels in the given annotations.
Returns
-------
label_issues : np.ndarray
Specifies which images are identified to have a label issue.
If ``return_indices_ranked_by_score = False``, this function returns a boolean mask of length ``N`` (``True`` entries indicate which images have label issue).
If ``return_indices_ranked_by_score = True``, this function returns a (shorter) array of indices of images with label issues, sorted by how likely the image is mislabeled.
More precisely, indices are sorted by image label quality score calculated via :py:func:`object_detection.rank.get_label_quality_scores <cleanlab.object_detection.rank.get_label_quality_scores>`.
"""
scoring_method = "objectlab"
assert_valid_inputs(
labels=labels,
predictions=predictions,
method=scoring_method,
)
is_issue = _find_label_issues(
labels,
predictions,
scoring_method=scoring_method,
return_indices_ranked_by_score=return_indices_ranked_by_score,
overlapping_label_check=overlapping_label_check,
)
return is_issue
def _find_label_issues(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
*,
scoring_method: Optional[str] = "objectlab",
return_indices_ranked_by_score: Optional[bool] = True,
overlapping_label_check: Optional[bool] = True,
):
"""Internal function to find label issues based on passed in method."""
if scoring_method == "objectlab":
auxiliary_inputs = _get_valid_inputs_for_compute_scores(ALPHA, labels, predictions)
per_class_scores = _get_per_class_ap(labels, predictions)
lab_list = [_separate_label(label)[1] for label in labels]
pred_list = [_separate_prediction(pred)[1] for pred in predictions]
pred_thresholds_list = _process_class_list(pred_list, per_class_scores)
lab_thresholds_list = _process_class_list(lab_list, per_class_scores)
overlooked_scores_per_box = compute_overlooked_box_scores(
alpha=ALPHA,
high_probability_threshold=HIGH_PROBABILITY_THRESHOLD,
auxiliary_inputs=auxiliary_inputs,
)
overlooked_issues_per_box = _find_label_issues_per_box(
overlooked_scores_per_box, pred_thresholds_list, OVERLOOKED_THRESHOLD_FACTOR
)
overlooked_issues_per_image = _pool_box_scores_per_image(overlooked_issues_per_box)
badloc_scores_per_box = compute_badloc_box_scores(
alpha=ALPHA,
low_probability_threshold=LOW_PROBABILITY_THRESHOLD,
auxiliary_inputs=auxiliary_inputs,
)
badloc_issues_per_box = _find_label_issues_per_box(
badloc_scores_per_box, lab_thresholds_list, BADLOC_THRESHOLD_FACTOR
)
badloc_issues_per_image = _pool_box_scores_per_image(badloc_issues_per_box)
swap_scores_per_box = compute_swap_box_scores(
alpha=ALPHA,
high_probability_threshold=HIGH_PROBABILITY_THRESHOLD,
overlapping_label_check=overlapping_label_check,
auxiliary_inputs=auxiliary_inputs,
)
swap_issues_per_box = _find_label_issues_per_box(
swap_scores_per_box, lab_thresholds_list, SWAP_THRESHOLD_FACTOR
)
swap_issues_per_image = _pool_box_scores_per_image(swap_issues_per_box)
issues_per_image = (
overlooked_issues_per_image + badloc_issues_per_image + swap_issues_per_image
)
is_issue = issues_per_image > 0
else:
is_issue = np.full(
shape=[
len(labels),
],
fill_value=-1,
)
if return_indices_ranked_by_score:
scores = get_label_quality_scores(labels, predictions)
sorted_scores_idx = issues_from_scores(scores, threshold=1.0)
is_issue_idx = np.where(is_issue == True)[0]
sorted_issue_mask = np.in1d(sorted_scores_idx, is_issue_idx, assume_unique=True)
issue_idx = sorted_scores_idx[sorted_issue_mask]
return issue_idx
else:
return is_issue
def _find_label_issues_per_box(
scores_per_box: List[np.ndarray], threshold_classes, threshold_factor=1.0
) -> List[np.ndarray]:
"""Takes in a list of size ``N`` where each index is an array of scores for each bounding box in the `n-th` example
and a threshold. Each box below or equal to the corresponding threshold in threshold_classes will be marked as an issue.
Returns a list of size ``N`` where each index is a boolean array of length number of boxes per example `n`
marking if a specific box is an issue - 1 or not - 0."""
is_issue_per_box = []
for idx, score_per_box in enumerate(scores_per_box):
if len(score_per_box) == 0: # if no for specific image, then image not an issue
is_issue_per_box.append(np.array([False]))
else:
score_per_box[np.isnan(score_per_box)] = 1.0
score_per_box = score_per_box
issue_per_box = []
for i in range(len(score_per_box)):
issue_per_box.append(
score_per_box[i] <= threshold_classes[idx][i] * threshold_factor
)
is_issue_per_box.append(np.array(issue_per_box, bool))
return is_issue_per_box
def _pool_box_scores_per_image(is_issue_per_box: List[np.ndarray]) -> np.ndarray:
"""Takes in a list of size ``N`` where each index is a boolean array of length number of boxes per image `n `
marking if a specific box is an issue - 1 or not - 0.
Returns a list of size ``N`` where each index marks if the image contains an issue - 1 or not - 0.
Images are marked as issues if 1 or more bounding boxes in the image is an issue."""
is_issue = np.zeros(
shape=[
len(
is_issue_per_box,
)
]
)
for idx, issue_per_box in enumerate(is_issue_per_box):
if np.sum(issue_per_box) > 0:
is_issue[idx] = 1
return is_issue
def _process_class_list(class_list: List[np.ndarray], class_dict: Dict[int, float]) -> List:
"""
Converts a list of classes represented as numpy arrays using a class-to-float dictionary,
and returns a list where each class is replaced by its corresponding float value from the dictionary.
Args:
class_list (List[np.ndarray]): A list of classes represented as numpy arrays.
class_dict (Dict[int, float]): A dictionary mapping class indices to their corresponding float values.
Returns:
List[float]: A list of float values corresponding to the classes in the input list.
"""
class_l2 = []
for i in class_list:
l3 = [class_dict[j] for j in i]
class_l2.append(l3)
return class_l2
def _calculate_ap_per_class(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
*,
iou_threshold: Optional[float] = 0.5,
num_procs: int = 1,
) -> List:
"""
Computes the average precision for each class based on provided labels and predictions.
It uses an Intersection over Union (IoU) threshold and supports parallel processing with a specified number of processes.
"""
num_images = len(predictions)
num_scale = 1
num_classes = len(predictions[0])
if num_images > 1:
num_procs = min(num_procs, num_images)
pool = Pool(num_procs)
ap_per_class_list = []
for class_num in range(num_classes):
pred_bboxes, lab_bboxes = _filter_by_class(labels, predictions, class_num)
if num_images > 1:
tpfp = pool.starmap(
_calculate_true_positives_false_positives,
zip(pred_bboxes, lab_bboxes, [iou_threshold for _ in range(num_images)]),
)
else:
tpfp = [
_calculate_true_positives_false_positives(
pred_bboxes[0],
lab_bboxes[0],
iou_threshold,
)
]
true_positives, false_positives = tuple(zip(*tpfp))
num_gts = np.zeros(num_scale, dtype=int)
for j, bbox in enumerate(lab_bboxes):
num_gts[0] += bbox.shape[0]
pred_bboxes = np.vstack(pred_bboxes)
sort_inds = np.argsort(-pred_bboxes[:, -1])
true_positives = np.hstack(true_positives)[:, sort_inds]
false_positives = np.hstack(false_positives)[:, sort_inds]
true_positives = np.cumsum(true_positives, axis=1)
false_positives = np.cumsum(false_positives, axis=1)
eps = np.finfo(np.float32).eps
recalls = true_positives / np.maximum(num_gts[:, np.newaxis], eps)
precisions = true_positives / np.maximum((true_positives + false_positives), eps)
recalls = recalls[0, :]
precisions = precisions[0, :]
ap = _calculate_average_precision(recalls, precisions)
ap_per_class_list.append(ap)
if num_images > 1:
pool.close()
return ap_per_class_list
def _filter_by_class(
labels: List[Dict[str, Any]], predictions: List[np.ndarray], class_num: int
) -> Tuple[List, List]:
"""
Filters predictions and labels based on a specific class number.
"""
pred_bboxes = [prediction[class_num] for prediction in predictions]
lab_bboxes = []
for label in labels:
gt_inds = label["labels"] == class_num
lab_bboxes.append(label["bboxes"][gt_inds, :])
return pred_bboxes, lab_bboxes
def _calculate_true_positives_false_positives(
pred_bboxes: np.ndarray,
lab_bboxes: np.ndarray,
iou_threshold: Optional[float] = 0.5,
return_false_negative: bool = False,
) -> Union[Tuple[np.ndarray, np.ndarray], Tuple[np.ndarray, np.ndarray, np.ndarray]]:
"""Calculates true positives (TP) and false positives (FP) for object detection tasks.
It takes predicted bounding boxes, ground truth bounding boxes, and an optional Intersection over Union (IoU) threshold as inputs.
If return_false_negative is True, it returns an array of False negatives as well.
"""
num_preds = pred_bboxes.shape[0]
num_labels = lab_bboxes.shape[0]
num_scales = 1
true_positives = np.zeros((num_scales, num_preds), dtype=np.float32)
false_positives = np.zeros((num_scales, num_preds), dtype=np.float32)
if lab_bboxes.shape[0] == 0:
false_positives[...] = 1
if return_false_negative:
return true_positives, false_positives, np.array([], dtype=np.float32)
else:
return true_positives, false_positives
ious = _get_overlap_matrix(pred_bboxes, lab_bboxes)
ious_max = ious.max(axis=1)
ious_argmax = ious.argmax(axis=1)
sorted_indices = np.argsort(-pred_bboxes[:, -1])
is_covered = np.zeros(num_labels, dtype=bool)
for index in sorted_indices:
if ious_max[index] >= iou_threshold:
matching_label = ious_argmax[index]
if not is_covered[matching_label]:
is_covered[matching_label] = True
true_positives[0, index] = 1
else:
false_positives[0, index] = 1
else:
false_positives[0, index] = 1
if return_false_negative:
false_negatives = np.zeros((num_scales, num_labels), dtype=np.float32)
for label_index in range(num_labels):
if not is_covered[label_index]:
false_negatives[0, label_index] = 1
return true_positives, false_positives, false_negatives
return true_positives, false_positives
def _calculate_average_precision(
recall_values: np.ndarray, precision_values: np.ndarray
) -> np.ndarray:
"""Computes the average precision (AP) for a set of recall and precision values. It takes arrays of recall and precision values as inputs."""
recall_values = recall_values[np.newaxis, :]
precision_values = precision_values[np.newaxis, :]
num_scales = recall_values.shape[0]
average_precision = np.zeros(num_scales, dtype=np.float32)
zeros_matrix = np.zeros((num_scales, 1), dtype=recall_values.dtype)
ones_matrix = np.ones((num_scales, 1), dtype=recall_values.dtype)
modified_recall = np.hstack((zeros_matrix, recall_values, ones_matrix))
modified_precision = np.hstack((zeros_matrix, precision_values, zeros_matrix))
for i in range(modified_precision.shape[1] - 1, 0, -1):
modified_precision[:, i - 1] = np.maximum(
modified_precision[:, i - 1], modified_precision[:, i]
)
for i in range(num_scales):
index = np.where(modified_recall[i, 1:] != modified_recall[i, :-1])[0]
average_precision[i] = np.sum(
(modified_recall[i, index + 1] - modified_recall[i, index])
* modified_precision[i, index + 1]
)
return average_precision
def _get_per_class_ap(
labels: List[Dict[str, Any]], predictions: List[np.ndarray]
) -> Dict[int, float]:
"""Computes the Average Precision (AP) for each class in an object detection task.
It takes a list of label dictionaries and a list of prediction arrays as inputs.
It calculates AP values for different Intersection over Union (IoU) thresholds, averages them per class, and then scales the AP values.
"""
iou_thrs = np.linspace(0.5, 0.95, int(np.round((0.95 - 0.5) / 0.05)) + 1, endpoint=True)
class_num_to_iou_list = defaultdict(list)
for threshold in iou_thrs:
ap_per_class = _calculate_ap_per_class(labels, predictions, iou_threshold=threshold)
for class_num in range(0, len(ap_per_class)):
class_num_to_iou_list[class_num].append(ap_per_class[class_num])
class_num_to_AP = {}
for class_num in class_num_to_iou_list:
class_num_to_AP[class_num] = np.mean(class_num_to_iou_list[class_num]) * AP_SCALE_FACTOR
return class_num_to_AP
cleanlab/object_detection/rank.py
+1110
View File
File diff suppressed because it is too large Load Diff
cleanlab/object_detection/summary.py
+757
View File
@@ -0,0 +1,757 @@
"""
Methods to display examples and their label issues in an object detection dataset.
Here each image can have multiple objects, each with its own bounding box and class label.
"""
from multiprocessing import Pool
from typing import (
Optional,
Any,
Dict,
Tuple,
Union,
List,
TYPE_CHECKING,
TypeVar,
DefaultDict,
cast,
)
import numpy as np
import collections
from cleanlab.internal.constants import (
MAX_CLASS_TO_SHOW,
ALPHA,
EPSILON,
TINY_VALUE,
)
from cleanlab.object_detection.filter import (
_filter_by_class,
_calculate_true_positives_false_positives,
)
from cleanlab.object_detection.rank import (
_get_valid_inputs_for_compute_scores,
_separate_prediction,
_separate_label,
_get_prediction_type,
)
from cleanlab.internal.object_detection_utils import bbox_xyxy_to_xywh
if TYPE_CHECKING:
from PIL.Image import Image as Image # pragma: no cover
else:
Image = TypeVar("Image")
def object_counts_per_image(
labels=None,
predictions=None,
*,
auxiliary_inputs=None,
) -> Tuple[List, List]:
"""Return the number of annotated and predicted objects for each image in the dataset.
This method can help you discover images with abnormally many/few object annotations.
Parameters
----------
labels :
Annotated boxes and class labels in the original dataset, which may contain some errors.
This is a list of ``N`` dictionaries such that ``labels[i]`` contains the given labels for the `i`-th image in the following format:
``{'bboxes': np.ndarray((L,4)), 'labels': np.ndarray((L,)), 'image_name': str}`` where ``L`` is the number of annotated bounding boxes
for the `i`-th image and ``bboxes[l]`` is a bounding box of coordinates in ``[x1,y1,x2,y2]`` format with given class label ``labels[j]``.
``image_name`` is an optional part of the labels that can be used to later refer to specific images.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`].
For more information on proper labels formatting, check out the `MMDetection library <https://mmdetection.readthedocs.io/en/dev-3.x/advanced_guides/customize_dataset.html>`_.
predictions :
Predictions output by a trained object detection model.
For the most accurate results, predictions should be out-of-sample to avoid overfitting, eg. obtained via :ref:`cross-validation <pred_probs_cross_val>`.
This is a list of ``N`` ``np.ndarray`` such that ``predictions[i]`` corresponds to the model prediction for the `i`-th image.
For each possible class ``k`` in 0, 1, ..., K-1: ``predictions[i][k]`` is a ``np.ndarray`` of shape ``(M,5)``,
where ``M`` is the number of predicted bounding boxes for class ``k``. Here the five columns correspond to ``[x1,y1,x2,y2,pred_prob]``,
where ``[x1,y1,x2,y2]`` are coordinates of the bounding box predicted by the model
and ``pred_prob`` is the model's confidence in the predicted class label for this bounding box.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`]. The last column, pred_prob, represents the predicted probability that the bounding box contains an object of the class k.
For more information see the `MMDetection package <https://github.com/open-mmlab/mmdetection>`_ for an example object detection library that outputs predictions in the correct format.
auxiliary_inputs: optional
Auxiliary inputs to be used in the computation of counts.
The `auxiliary_inputs` can be computed using :py:func:`rank._get_valid_inputs_for_compute_scores <cleanlab.object_detection.rank._get_valid_inputs_for_compute_scores>`.
It is internally computed from the given `labels` and `predictions`.
Returns
-------
object_counts: Tuple[List, List]
A tuple containing two lists. The first is an array of shape ``(N,)`` containing the number of annotated objects for each image in the dataset.
The second is an array of shape ``(N,)`` containing the number of predicted objects for each image in the dataset.
"""
if auxiliary_inputs is None:
auxiliary_inputs = _get_valid_inputs_for_compute_scores(ALPHA, labels, predictions)
return (
[len(sample["lab_bboxes"]) for sample in auxiliary_inputs],
[len(sample["pred_bboxes"]) for sample in auxiliary_inputs],
)
def bounding_box_size_distribution(
labels=None,
predictions=None,
*,
auxiliary_inputs=None,
class_names: Optional[Dict[Any, Any]] = None,
sort: bool = False,
) -> Tuple[Dict[Any, List], Dict[Any, List]]:
"""Return the distribution over sizes of annotated and predicted bounding boxes across the dataset, broken down by each class.
This method can help you find annotated/predicted boxes for a particular class that are abnormally big/small.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
predictions:
Predictions output by a trained object detection model.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
auxiliary_inputs: optional
Auxiliary inputs to be used in the computation of counts.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
class_names: optional
A dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``.
You can use this argument to control the classes for which the size distribution is computed.
sort: bool
If True, the returned dictionaries are sorted by the number of instances of each class in the dataset in descending order.
Returns
-------
bbox_sizes: Tuple[Dict[Any, List], Dict[Any, List]]
A tuple containing two dictionaries. Each maps each class label to a list of the sizes of annotated bounding boxes for that class in the label and prediction datasets, respectively.
"""
if auxiliary_inputs is None:
auxiliary_inputs = _get_valid_inputs_for_compute_scores(ALPHA, labels, predictions)
lab_area: Dict[Any, list] = collections.defaultdict(list)
pred_area: Dict[Any, list] = collections.defaultdict(list)
for sample in auxiliary_inputs:
_get_bbox_areas(sample["lab_labels"], sample["lab_bboxes"], lab_area, class_names)
_get_bbox_areas(sample["pred_labels"], sample["pred_bboxes"], pred_area, class_names)
if sort:
lab_area = dict(sorted(lab_area.items(), key=lambda x: -len(x[1])))
pred_area = dict(sorted(pred_area.items(), key=lambda x: -len(x[1])))
return lab_area, pred_area
def class_label_distribution(
labels=None,
predictions=None,
*,
auxiliary_inputs=None,
class_names: Optional[Dict[Any, Any]] = None,
) -> Tuple[Dict[Any, float], Dict[Any, float]]:
"""Returns the distribution of class labels associated with all annotated bounding boxes (or predicted bounding boxes) in the dataset.
This method can help you understand which classes are: rare or over/under-predicted by the model overall.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
predictions:
Predictions output by a trained object detection model.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
auxiliary_inputs: optional
Auxiliary inputs to be used in the computation of counts.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
class_names: optional
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``.
Returns
-------
class_distribution: Tuple[Dict[Any, float], Dict[Any, float]]
A tuple containing two dictionaries. The first is a dictionary mapping each class label to its frequency in the dataset annotations.
The second is a dictionary mapping each class label to its frequency in the model predictions across all images in the dataset.
"""
if auxiliary_inputs is None:
auxiliary_inputs = _get_valid_inputs_for_compute_scores(ALPHA, labels, predictions)
lab_freq: DefaultDict[Any, int] = collections.defaultdict(int)
pred_freq: DefaultDict[Any, int] = collections.defaultdict(int)
for sample in auxiliary_inputs:
_get_class_instances(sample["lab_labels"], lab_freq, class_names)
_get_class_instances(sample["pred_labels"], pred_freq, class_names)
label_norm = _normalize_by_total(lab_freq)
pred_norm = _normalize_by_total(pred_freq)
return label_norm, pred_norm
def get_sorted_bbox_count_idxs(labels, predictions):
"""
Returns a tuple of idxs and bounding box counts of images sorted from highest to lowest number of bounding boxes.
This plot can help you discover images with abnormally many/few object annotations.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
predictions:
Predictions output by a trained object detection model.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
Returns
-------
sorted_idxs: List[Tuple[int, int]], List[Tuple[int, int]]
A tuple containing two lists. The first is an array of shape ``(N,)`` containing the number of annotated objects for each image in the dataset.
The second is an array of shape ``(N,)`` containing the number of predicted objects for each image in the dataset.
"""
lab_count, pred_count = object_counts_per_image(labels, predictions)
lab_grouped = list(enumerate(lab_count))
pred_grouped = list(enumerate(pred_count))
sorted_lab = sorted(lab_grouped, key=lambda x: x[1], reverse=True)
sorted_pred = sorted(pred_grouped, key=lambda x: x[1], reverse=True)
return sorted_lab, sorted_pred
def plot_class_size_distributions(
labels, predictions, class_names=None, class_to_show=MAX_CLASS_TO_SHOW, **kwargs
):
"""
Plots the size distributions for bounding boxes for each class.
This plot can help you find annotated/predicted boxes for a particular class that are abnormally big/small.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
predictions:
Predictions output by a trained object detection model.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
class_names: optional
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``.
You can use this argument to control the classes for which the size distribution is plotted.
class_to_show: optional
The number of classes to show in the plots. Classes over `class_to_show` are hidden. If this argument is provided, then the classes are sorted by the number of instances in the dataset.
Defaults to `MAX_CLASS_TO_SHOW` which is set to 10.
kwargs:
Additional keyword arguments to pass to ``plt.show()`` (matplotlib.pyplot.show).
"""
try:
import matplotlib.pyplot as plt
except ImportError as e:
raise ImportError(
"This functionality requires matplotlib. Install it via: `pip install matplotlib`"
)
lab_boxes, pred_boxes = bounding_box_size_distribution(
labels,
predictions,
class_names=class_names,
sort=True if class_to_show is not None else False,
)
for i, c in enumerate(lab_boxes.keys()):
if i >= class_to_show:
break
fig, axs = plt.subplots(1, 2, figsize=(10, 5))
fig.suptitle(f"Size distributions for bounding box for class {c}")
for i, l in enumerate([lab_boxes, pred_boxes]):
axs[i].hist(l[c], bins="auto")
axs[i].set_xlabel("box area (pixels)")
axs[i].set_ylabel("count")
axs[i].set_title("annotated" if i == 0 else "predicted")
plt.show(**kwargs)
def plot_class_distribution(labels, predictions, class_names=None, **kwargs):
"""
Plots the distribution of class labels associated with all annotated bounding boxes and predicted bounding boxes in the dataset.
This plot can help you understand which classes are rare or over/under-predicted by the model overall.
Parameters
----------
labels:
Annotated boxes and class labels in the original dataset, which may contain some errors.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
predictions:
Predictions output by a trained object detection model.
Refer to documentation for this argument in :py:func:`object_counts_per_image <cleanlab.object_detection.summary.object_counts_per_image>` for further details.
class_names: optional
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``.
kwargs:
Additional keyword arguments to pass to ``plt.show()`` (matplotlib.pyplot.show).
"""
try:
import matplotlib.pyplot as plt
except ImportError as e:
raise ImportError(
"This functionality requires matplotlib. Install it via: `pip install matplotlib`"
)
lab_dist, pred_dist = class_label_distribution(labels, predictions, class_names=class_names)
fig, axs = plt.subplots(1, 2, figsize=(10, 5))
fig.suptitle(f"Distribution of classes in the dataset")
for i, d in enumerate([lab_dist, pred_dist]):
axs[i].pie(d.values(), labels=d.keys(), autopct="%1.1f%%")
axs[i].set_title("Annotated" if i == 0 else "Predicted")
plt.show(**kwargs)
def visualize(
image: Union[str, np.ndarray, Image],
*,
label: Optional[Dict[str, Any]] = None,
prediction: Optional[np.ndarray] = None,
prediction_threshold: Optional[float] = None,
overlay: bool = True,
class_names: Optional[Dict[Any, Any]] = None,
figsize: Optional[Tuple[int, int]] = None,
save_path: Optional[str] = None,
**kwargs,
) -> None:
"""Display the annotated bounding boxes (given labels) and predicted bounding boxes (model predictions) for a particular image.
Given labels are shown in red, model predictions in blue.
Parameters
----------
image:
Image object loaded into memory or full path to the image file. If path is provided, image is loaded into memory.
label:
The given label for a single image in the format ``{'bboxes': np.ndarray((L,4)), 'labels': np.ndarray((L,))}`` where
``L`` is the number of bounding boxes for the `i`-th image and ``bboxes[j]`` is in the format ``[x1,y1,x2,y2]`` with given label ``labels[j]``.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`].
prediction:
A prediction for a single image in the format ``np.ndarray((K,))`` and ``prediction[k]`` is of shape ``np.ndarray(N,5)``
where ``M`` is the number of predicted bounding boxes for class ``k`` and the five columns correspond to ``[x,y,x,y,pred_prob]`` where
``[x1,y1,x2,y2]`` are the bounding box coordinates predicted by the model and ``pred_prob`` is the model's confidence in ``predictions[i]``.
Note: Here, ``(x1,y1)`` corresponds to the top-left and ``(x2,y2)`` corresponds to the bottom-right corner of the bounding box with respect to the image matrix [e.g. `XYXY in Keras <https://keras.io/api/keras_cv/bounding_box/formats/>`, `Detectron 2 <https://detectron2.readthedocs.io/en/latest/modules/utils.html#detectron2.utils.visualizer.Visualizer.draw_box>`]. The last column, pred_prob, represents the predicted probability that the bounding box contains an object of the class k.
prediction_threshold:
All model-predicted bounding boxes with confidence (`pred_prob`)
below this threshold are omitted from the visualization.
overlay: bool
If True, display a single image with given labels and predictions overlaid.
If False, display two images (side by side) with the left image showing the model predictions and the right image showing the given label.
class_names:
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``.
save_path:
Path to save figure at. If a path is provided, the figure is saved. To save in a specific image format, add desired file extension to the end of `save_path`. Allowed file extensions are: 'png', 'pdf', 'ps', 'eps', and 'svg'.
figsize:
Optional figure size for plotting the image.
Corresponds to ``matplotlib.figure.figsize``.
kwargs:
Additional keyword arguments to pass to ``plt.show()`` (matplotlib.pyplot.show).
"""
try:
import matplotlib.pyplot as plt
from matplotlib.axes import Axes
except ImportError as e:
raise ImportError(
"This functionality requires matplotlib. Install it via: `pip install matplotlib`"
)
# Create figure and axes
if isinstance(image, str):
image = plt.imread(image)
if prediction is not None:
prediction_type = _get_prediction_type(prediction)
pbbox, plabels, pred_probs = _separate_prediction(
prediction, prediction_type=prediction_type
)
if prediction_threshold is not None:
keep_idx = np.where(pred_probs > prediction_threshold)
pbbox = pbbox[keep_idx]
plabels = plabels[keep_idx]
if label is not None:
abbox, alabels = _separate_label(label)
if overlay:
figsize = (8, 5) if figsize is None else figsize
fig, ax = plt.subplots(frameon=False, figsize=figsize)
plt.axis("off")
ax.imshow(image)
if label is not None:
fig, ax = _draw_boxes(
fig, ax, abbox, alabels, edgecolor="r", linestyle="-", linewidth=1
)
if prediction is not None:
_, _ = _draw_boxes(fig, ax, pbbox, plabels, edgecolor="b", linestyle="-.", linewidth=1)
else:
figsize = (14, 10) if figsize is None else figsize
fig, axes = plt.subplots(nrows=1, ncols=2, frameon=False, figsize=figsize)
axes = cast(Tuple[Axes, Axes], axes)
axes[0].axis("off")
axes[0].imshow(image)
axes[1].axis("off")
axes[1].imshow(image)
if label is not None:
fig, ax = _draw_boxes(
fig, axes[0], abbox, alabels, edgecolor="r", linestyle="-", linewidth=1
)
if prediction is not None:
_, _ = _draw_boxes(
fig, axes[1], pbbox, plabels, edgecolor="b", linestyle="-.", linewidth=1
)
bbox_extra_artists = None
if label or prediction is not None:
legend, plt = _plot_legend(class_names, label, prediction)
bbox_extra_artists = (legend,)
if save_path:
allowed_image_formats = set(["png", "pdf", "ps", "eps", "svg"])
image_format: Optional[str] = None
if save_path.split(".")[-1] in allowed_image_formats and "." in save_path:
image_format = save_path.split(".")[-1]
plt.savefig(
save_path,
format=image_format,
bbox_extra_artists=bbox_extra_artists,
bbox_inches="tight",
transparent=True,
pad_inches=0.5,
)
plt.show(**kwargs)
def _get_per_class_confusion_matrix_dict_(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
iou_threshold: Optional[float] = 0.5,
num_procs: int = 1,
) -> DefaultDict[int, Dict[str, int]]:
"""
Returns a confusion matrix dictionary for each class containing the number of True Positive, False Positive, and False Negative detections from the object detection model.
"""
num_classes = len(predictions[0])
num_images = len(predictions)
pool = Pool(num_procs)
counter_dict: DefaultDict[int, dict[str, int]] = collections.defaultdict(
lambda: {"TP": 0, "FP": 0, "FN": 0}
)
for class_num in range(num_classes):
pred_bboxes, lab_bboxes = _filter_by_class(labels, predictions, class_num)
tpfpfn = pool.starmap(
_calculate_true_positives_false_positives,
zip(
pred_bboxes,
lab_bboxes,
[iou_threshold for _ in range(num_images)],
[True for _ in range(num_images)],
),
)
for image_idx, (tp, fp, fn) in enumerate(tpfpfn): # type: ignore
counter_dict[class_num]["TP"] += np.sum(tp)
counter_dict[class_num]["FP"] += np.sum(fp)
counter_dict[class_num]["FN"] += np.sum(fn)
return counter_dict
def _sort_dict_to_list(index_value_dict):
"""
Convert a dictionary to a list sorted by index and return the values in that order.
Parameters:
- index_value_dict (dict): The input dictionary where keys represent indices and values are the corresponding elements.
Returns:
list: A list containing the values from the input dictionary, sorted by index.
Example:
>>> my_dict = {'0': '0', '1': '1', '2': '2', '3': '3', '4': '4'}
>>> sort_dict_to_list(my_dict)
['0', '1', '2', '3', '4']
"""
sorted_list = [
value for key, value in sorted(index_value_dict.items(), key=lambda x: int(x[0]))
]
return sorted_list
def get_average_per_class_confusion_matrix(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
num_procs: int = 1,
class_names: Optional[Dict[Any, Any]] = None,
) -> Dict[Union[int, str], Dict[str, float]]:
"""
Compute a confusion matrix dictionary for each class containing the average number of True Positive, False Positive, and False Negative detections from the object detection model across a range of Intersection over Union thresholds.
At each IoU threshold, the metrics are calculated as follows:
- True Positive (TP): Instances where the model correctly identifies the class with IoU above the threshold.
- False Positive (FP): Instances where the model predicts the class, but IoU is below the threshold.
- False Negative (FN): Instances where the ground truth class is not predicted by the model.
The average confusion matrix provides insights into the model strengths and potential biases.
Note: lower TP at certain IoU thresholds does not necessarily imply that everything else is FP, instead it indicates that, at those specific IoU thresholds, the model is not performing as well in terms of correctly identifying class instances. The other metrics (FP and FN) provide additional information about the model's behavior.
Note: Since we average over many IoU thresholds, 'TP', 'FP', and 'FN' may contain float values representing the average across these thresholds.
Parameters
----------
labels:
A list of ``N`` dictionaries such that ``labels[i]`` contains the given labels for the `i`-th image.
Refer to documentation for this argument in :py:func:`object_detection.filter.find_label_issues <cleanlab.object_detection.filter.find_label_issues>` for further details.
predictions:
A list of ``N`` ``np.ndarray`` such that ``predictions[i]`` corresponds to the model predictions for the `i`-th image.
Refer to documentation for this argument in :py:func:`object_detection.filter.find_label_issues <cleanlab.object_detection.filter.find_label_issues>` for further details.
num_procs:
Number of processes for parallelization. Default is 1.
class_names:
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``
Returns
-------
avg_metrics: dict
A distionary containing the average confusion matrix.
The default range of Intersection over Union thresholds is from 0.5 to 0.95 with a step size of 0.05.
"""
iou_thrs = np.linspace(0.5, 0.95, int(np.round((0.95 - 0.5) / 0.05)) + 1, endpoint=True)
num_classes = len(predictions[0])
if class_names is None:
class_names = {str(i): int(i) for i in list(range(num_classes))}
class_names = _sort_dict_to_list(class_names)
avg_metrics = {class_num: {"TP": 0.0, "FP": 0.0, "FN": 0.0} for class_num in class_names}
for iou_threshold in iou_thrs:
results_dict = _get_per_class_confusion_matrix_dict_(
labels, predictions, iou_threshold, num_procs
)
for class_num in results_dict:
tp = results_dict[class_num]["TP"]
fp = results_dict[class_num]["FP"]
fn = results_dict[class_num]["FN"]
avg_metrics[class_names[class_num]]["TP"] += tp
avg_metrics[class_names[class_num]]["FP"] += fp
avg_metrics[class_names[class_num]]["FN"] += fn
num_thresholds = len(iou_thrs) * len(results_dict)
for class_name in avg_metrics:
avg_metrics[class_name]["TP"] /= num_thresholds
avg_metrics[class_name]["FP"] /= num_thresholds
avg_metrics[class_name]["FN"] /= num_thresholds
return avg_metrics
def calculate_per_class_metrics(
labels: List[Dict[str, Any]],
predictions: List[np.ndarray],
num_procs: int = 1,
class_names=None,
) -> Dict[Union[int, str], Dict[str, float]]:
"""
Calculate the object detection model's precision, recall, and F1 score for each class in the dataset.
These metrics can help you identify model strengths and weaknesses, and provide reference statistics for model evaluation and comparisons.
Parameters
----------
labels:
A list of ``N`` dictionaries such that ``labels[i]`` contains the given labels for the `i`-th image.
Refer to documentation for this argument in :py:func:`object_detection.filter.find_label_issues <cleanlab.object_detection.filter.find_label_issues>` for further details.
predictions:
A list of ``N`` ``np.ndarray`` such that ``predictions[i]`` corresponds to the model predictions for the `i`-th image.
Refer to documentation for this argument in :py:func:`object_detection.filter.find_label_issues <cleanlab.object_detection.filter.find_label_issues>` for further details.
num_procs:
Number of processes for parallelization. Default is 1.
class_names:
Optional dictionary mapping one-hot-encoded class labels back to their original class names in the format ``{"integer-label": "original-class-name"}``
Returns
-------
per_class_metrics: dict
A dictionary containing per-class metrics computed from the object detection model's average confusion matrix values across a range of Intersection over Union thresholds.
The default range of Intersection over Union thresholds is from 0.5 to 0.95 with a step size of 0.05.
"""
avg_metrics = get_average_per_class_confusion_matrix(
labels, predictions, num_procs, class_names=class_names
)
avg_metrics_dict = {}
for class_name in avg_metrics:
tp = avg_metrics[class_name]["TP"]
fp = avg_metrics[class_name]["FP"]
fn = avg_metrics[class_name]["FN"]
precision = tp / (tp + fp + TINY_VALUE) # Avoid division by zero
recall = tp / (tp + fn + TINY_VALUE) # Avoid division by zero
f1 = 2 * (precision * recall) / (precision + recall + TINY_VALUE) # Avoid division by zero
avg_metrics_dict[class_name] = {
"average precision": precision,
"average recall": recall,
"average f1": f1,
}
return avg_metrics_dict
def _normalize_by_total(freq):
"""Helper function to normalize a frequency distribution."""
total = sum(freq.values())
return {k: round(v / (total + EPSILON), 2) for k, v in freq.items()}
def _get_bbox_areas(labels, boxes, class_area_dict, class_names=None) -> None:
"""Helper function to compute the area of bounding boxes for each class."""
for cl, bbox in zip(labels, boxes):
if class_names is not None:
if str(cl) not in class_names:
continue
cl = class_names[str(cl)]
class_area_dict[cl].append((bbox[2] - bbox[0]) * (bbox[3] - bbox[1]))
def _get_class_instances(labels, class_instances_dict, class_names=None) -> None:
"""Helper function to count the number of class instances in each image."""
for cl in labels:
if class_names is not None:
cl = class_names[str(cl)]
class_instances_dict[cl] += 1
def _plot_legend(class_names, label, prediction):
colors = ["black"]
colors.extend(["red"] if label is not None else [])
colors.extend(["blue"] if prediction is not None else [])
markers = [None]
markers.extend(["s"] if label is not None else [])
markers.extend(["s"] if prediction is not None else [])
labels = [r"$\bf{Legend}$"]
labels.extend(["given label"] if label is not None else [])
labels.extend(["predicted label"] if prediction is not None else [])
if class_names:
colors += ["black"] + ["black"] * min(len(class_names), MAX_CLASS_TO_SHOW)
markers += [None] + [f"${class_key}$" for class_key in class_names.keys()]
labels += [r"$\bf{classes}$"] + list(class_names.values())
try:
import matplotlib.pyplot as plt
except ImportError as e:
raise ImportError(
"This functionality requires matplotlib. Install it via: `pip install matplotlib`"
)
f = lambda m, c: plt.plot([], [], marker=m, color=c, ls="none")[0]
handles = [f(marker, color) for marker, color in zip(markers, colors)]
legend = plt.legend(
handles, labels, bbox_to_anchor=(1.04, 0.05), loc="lower left", borderaxespad=0
)
return legend, plt
def _draw_labels(ax, rect, label, edgecolor):
"""Helper function to draw labels on an axis."""
rx, ry = rect.get_xy()
c_xleft = rx + 10
c_xright = rx + rect.get_width() - 10
c_ytop = ry + 12
if edgecolor == "r":
cx, cy = c_xleft, c_ytop
else: # edgecolor == b
cx, cy = c_xright, c_ytop
l = ax.annotate(
label, (cx, cy), fontsize=8, fontweight="bold", color="white", ha="center", va="center"
)
l.set_bbox(dict(facecolor=edgecolor, alpha=0.35, edgecolor=edgecolor, pad=2))
return ax
def _draw_boxes(fig, ax, bboxes, labels, edgecolor="g", linestyle="-", linewidth=3):
"""Helper function to draw bboxes and labels on an axis."""
bboxes = [bbox_xyxy_to_xywh(box) for box in bboxes]
try:
from matplotlib.patches import Rectangle
except Exception as e:
raise ImportError(
"This functionality requires matplotlib. Install it via: `pip install matplotlib`"
)
for (x, y, w, h), label in zip(bboxes, labels):
rect = Rectangle(
(x, y),
w,
h,
linewidth=linewidth,
linestyle=linestyle,
edgecolor=edgecolor,
facecolor="none",
)
ax.add_patch(rect)
if labels is not None:
ax = _draw_labels(ax, rect, label, edgecolor)
return fig, ax

Some files were not shown because too many files have changed in this diff Show More