wehub-resource-sync
209 lines
9.3 KiB
ReStructuredText
209 lines
9.3 KiB
ReStructuredText
:og:title: Open-Source Documentation | Cleanlab
|
|
:og:description: Get started with Cleanlab Open-Source, learn about capabilities, and follow tutorials to improve your own Data and Models.
|
|
|
|
cleanlab open-source documentation
|
|
==================================
|
|
|
|
`cleanlab <https://github.com/cleanlab/cleanlab>`_ **automatically detects data and label issues in your ML datasets.**
|
|
|
|
| This helps you improve your data and train robust ML models on noisy real-world datasets. cleanlab has already found thousands of `label errors <https://labelerrors.com>`_ in ImageNet, MNIST, and other popular ML benchmarking datasets. Beyond handling label errors, this is a comprehensive open-source library implementing many data-centric AI capabilities. Start using automation to improve your data!
|
|
|
|
Quickstart
|
|
==========
|
|
|
|
1. Install ``cleanlab``
|
|
-----------------------
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: uv
|
|
|
|
.. code-block:: bash
|
|
|
|
uv pip install cleanlab
|
|
|
|
To install the package with all optional dependencies:
|
|
|
|
.. code-block:: bash
|
|
|
|
uv pip install "cleanlab[all]"
|
|
|
|
.. tab:: pip
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install cleanlab
|
|
|
|
To install the package with all optional dependencies:
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install "cleanlab[all]"
|
|
|
|
|
|
.. tab:: conda
|
|
|
|
.. code-block:: bash
|
|
|
|
conda install -c cleanlab cleanlab
|
|
|
|
.. tab:: source
|
|
|
|
.. code-block:: bash
|
|
|
|
uv pip install git+https://github.com/cleanlab/cleanlab.git
|
|
|
|
Or with pip:
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install git+https://github.com/cleanlab/cleanlab.git
|
|
|
|
To install the package with all optional dependencies:
|
|
|
|
.. code-block:: bash
|
|
|
|
uv pip install "git+https://github.com/cleanlab/cleanlab.git#egg=cleanlab[all]"
|
|
|
|
|
|
2. Check your data for all sorts of issues
|
|
------------------------------------------
|
|
|
|
cleanlab automatically detects various issues in *any dataset that a classifier can be trained on*. The cleanlab package *works with any ML model* by operating on model outputs (predicted class probabilities or feature embeddings) -- it doesn't require that a particular model created those outputs. For any classification dataset, use your trained model to produce `pred_probs` (predicted class probabilities) and/or `feature_embeddings` (numeric vector representations of each datapoint). To automatically check your dataset for common real-world issues (like label errors, outliers, near duplicates, IID violations, underperforming groups, ...), simply run these few lines of code:
|
|
|
|
.. code-block:: python
|
|
|
|
from cleanlab import Datalab
|
|
|
|
lab = Datalab(data=your_dataset, label_name="column_name_of_labels")
|
|
lab.find_issues(features=feature_embeddings, pred_probs=pred_probs)
|
|
lab.report() # summarize issues in dataset, how severe they are in each data point, ...
|
|
|
|
While other data quality tools only catch limited types of data issues based on manually pre-defined validation rules, cleanlab applies automated data-centric AI techniques using your trained ML model to detect many more types of data issues that would otherwise be hard to catch. Don't dive into ML model improvement without first using AI to help check your data!
|
|
|
|
|
|
3. Handle label errors and train robust models with noisy labels
|
|
----------------------------------------------------------------
|
|
|
|
Mislabeled data is a particularly concerning issue plaguing real-world datasets. To use a scikit-learn-compatible model for classification with noisy labels, you don't need to train a model to find label issues -- you can pass the untrained model object, data, and labels into :py:meth:`CleanLearning.find_label_issues <cleanlab.classification.CleanLearning.find_label_issues>` and cleanlab will handle model training for you.
|
|
|
|
.. code-block:: python
|
|
|
|
from cleanlab.classification import CleanLearning
|
|
|
|
# This works with any sklearn-compatible model - just input data + labels and cleanlab will detect label issues ツ
|
|
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)
|
|
|
|
:py:class:`CleanLearning <cleanlab.classification.CleanLearning>` also works with models from most standard ML frameworks by wrapping the model for scikit-learn compliance, e.g. pytorch (can use `skorch <https://github.com/skorch-dev/skorch>`_ package), etc.
|
|
|
|
:py:meth:`find_label_issues <cleanlab.classification.CleanLearning.find_label_issues>` returns a boolean mask flagging which examples have label issues and a numeric label quality score for each example quantifying our confidence that its label is correct.
|
|
|
|
Beyond standard classification tasks, cleanlab can also detect mislabeled examples in: `multi-label data <tutorials/multilabel_classification.ipynb>`_ (e.g. image/document tagging), `sequence prediction <tutorials/token_classification.ipynb>`_ (e.g. entity recognition), and `data labeled by multiple annotators <tutorials/multiannotator.ipynb>`_ (e.g. crowdsourcing).
|
|
|
|
.. important::
|
|
Cleanlab performs better if the ``pred_probs`` from your model are **out-of-sample**. Details on how to compute out-of-sample predicted probabilities for your entire dataset are :ref:`here <pred_probs_cross_val>`.
|
|
|
|
cleanlab's :py:class:`CleanLearning <cleanlab.classification.CleanLearning>` class trains a more robust version of any existing (`scikit-learn <https://scikit-learn.org/>`_ `compatible <https://scikit-learn.org/stable/developers/develop.html>`_) classification model, `clf`, by fitting it to an automatically filtered version of your dataset with low-quality data removed. It returns a model trained only on the clean data, from which you can get predictions in the same way as your existing classifier.
|
|
|
|
.. code-block:: python
|
|
|
|
from sklearn.linear_model import LogisticRegression
|
|
from cleanlab.classification import CleanLearning
|
|
|
|
cl = CleanLearning(clf=LogisticRegression()) # any sklearn-compatible classifier
|
|
cl.fit(train_data, labels)
|
|
|
|
# Estimate the predictions you would have gotten if you trained without mislabeled data
|
|
predictions = cl.predict(test_data)
|
|
|
|
|
|
4. Dataset curation: fix dataset-level issues
|
|
---------------------------------------------
|
|
|
|
cleanlab's `dataset <tutorials/dataset_health.ipynb>`_ module helps you deal with dataset-level issues -- :py:meth:`find overlapping classes <cleanlab.dataset.find_overlapping_classes>` (classes to merge), :py:meth:`rank class-level label quality <cleanlab.dataset.rank_classes_by_label_quality>` (classes to keep/delete), and :py:meth:`measure overall dataset health <cleanlab.dataset.overall_label_health_score>` (to track dataset quality as you make adjustments).
|
|
|
|
View all dataset-level issues in one line of code with :py:meth:`dataset.health_summary() <cleanlab.dataset.health_summary>`.
|
|
|
|
.. code-block:: python
|
|
|
|
from cleanlab.dataset import health_summary
|
|
|
|
health_summary(labels, pred_probs, class_names=class_names)
|
|
|
|
|
|
5. Improve your data via many other techniques
|
|
----------------------------------------------
|
|
|
|
Beyond handling label errors, cleanlab supports other data-centric AI capabilities including:
|
|
|
|
- Detecting outliers and out-of-distribution examples in both training and future test data `(tutorial) <tutorials/outliers.ipynb>`_
|
|
- Analyzing data labeled by multiple annotators to estimate consensus labels and their quality `(tutorial) <tutorials/multiannotator.ipynb>`_
|
|
- Active learning with multiple annotators to identify which data is most informative to label or re-label next `(tutorial) <https://github.com/cleanlab/examples/blob/master/active_learning_multiannotator/active_learning.ipynb>`_
|
|
|
|
|
|
If you have questions, check out our `FAQ <tutorials/faq.ipynb>`_ and feel free to open a Github Issue!
|
|
|
|
|
|
.. toctree::
|
|
:hidden:
|
|
|
|
Quickstart <self>
|
|
|
|
.. toctree::
|
|
:hidden:
|
|
:caption: Tutorials
|
|
|
|
Datalab Tutorials <tutorials/datalab/index>
|
|
Improving ML Performance <tutorials/improving_ml_performance>
|
|
CleanLearning Tutorials <tutorials/clean_learning/index>
|
|
Workflows of Data-Centric AI <tutorials/indepth_overview>
|
|
Analyze Dataset-level Issues <tutorials/dataset_health>
|
|
Outlier Detection <tutorials/outliers>
|
|
Improving Consensus Labels for Multiannotator Data <tutorials/multiannotator>
|
|
Multi-Label Classification <tutorials/multilabel_classification>
|
|
Noisy Labels in Regression <tutorials/regression>
|
|
Token Classification (text) <tutorials/token_classification>
|
|
Image Segmentation <tutorials/segmentation>
|
|
Object Detection <tutorials/object_detection>
|
|
Predicted Probabilities via Cross Validation <tutorials/pred_probs_cross_val>
|
|
FAQ <tutorials/faq>
|
|
|
|
.. toctree::
|
|
:caption: API Reference
|
|
:hidden:
|
|
:maxdepth: 3
|
|
|
|
cleanlab/datalab/index
|
|
cleanlab/classification
|
|
cleanlab/filter
|
|
cleanlab/rank
|
|
cleanlab/count
|
|
cleanlab/dataset
|
|
cleanlab/outlier
|
|
cleanlab/multiannotator
|
|
cleanlab/data_valuation
|
|
cleanlab/multilabel_classification/index
|
|
cleanlab/regression/index
|
|
cleanlab/token_classification/index
|
|
cleanlab/segmentation/index
|
|
cleanlab/object_detection/index
|
|
cleanlab/benchmarking/index
|
|
cleanlab/models/index
|
|
cleanlab/experimental/index
|
|
cleanlab/internal/index
|
|
|
|
.. toctree::
|
|
:caption: Guides
|
|
:hidden:
|
|
|
|
Datalab issue types <cleanlab/datalab/guide/index>
|
|
|
|
.. toctree::
|
|
:caption: Links
|
|
:hidden:
|
|
|
|
GitHub <https://github.com/cleanlab/cleanlab>
|
|
PyPI <https://pypi.org/project/cleanlab/>
|
|
Conda <https://anaconda.org/conda-forge/cleanlab>
|
|
Blog <https://cleanlab.ai/blog/>
|