wehub-resource-sync
226 lines
8.0 KiB
ReStructuredText
226 lines
8.0 KiB
ReStructuredText
.. _issue_manager_creating_your_own:
|
|
|
|
Creating Your Own Issues Manager
|
|
================================
|
|
|
|
|
|
|
|
This guide walks through the process of creating your own
|
|
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>`
|
|
to detect a custom-defined type of issue alongside the pre-defined issue types in
|
|
:py:class:`Datalab <cleanlab.datalab.datalab.Datalab>`.
|
|
|
|
.. seealso::
|
|
|
|
- :py:meth:`register <cleanlab.datalab.internal.issue_manager_factory.register>`:
|
|
You can either use this function at runtime to register a new issue manager:
|
|
|
|
.. code-block:: python
|
|
|
|
from cleanlab.datalab.internal.issue_manager_factory import register
|
|
register(MyIssueManager) # Defaults to task="classification"
|
|
# register(MyIssueManagerForRegression, task="regression") # Alternative for regression tasks
|
|
|
|
or add as a decorator to the class definition (currently only works for classification tasks):
|
|
|
|
.. code-block:: python
|
|
|
|
@register
|
|
class MyIssueManager(IssueManager):
|
|
...
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
As a starting point for this guide, we'll import the necessary things for the next section and create a dummy dataset.
|
|
|
|
.. note::
|
|
|
|
.. include:: ../optional_dependencies.rst
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
import numpy as np
|
|
import pandas as pd
|
|
from cleanlab import IssueManager
|
|
|
|
# Create a dummy dataset
|
|
N = 20
|
|
data = pd.DataFrame(
|
|
{
|
|
"text": [f"example {i}" for i in range(N)],
|
|
"label": np.random.randint(0, 2, N),
|
|
},
|
|
)
|
|
|
|
|
|
Implementing IssueManagers
|
|
--------------------------
|
|
|
|
.. _basic_issue_manager:
|
|
|
|
Basic Issue Check
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
To create a basic issue manager, inherit from the
|
|
:py:class:`IssueManager <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager>` class,
|
|
assign a name to the class as the class-variable, `issue_name`, and implement the
|
|
:py:meth:`find_issues <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager.find_issues>` method.
|
|
|
|
The :py:meth:`find_issues <cleanlab.datalab.internal.issue_manager.issue_manager.IssueManager.find_issues>`
|
|
method should mark each example in the dataset as an issue or not with a boolean array.
|
|
It should also provide a score for each example in the dataset that quantifies the quality of the example
|
|
with regards to the issue.
|
|
|
|
.. code-block:: python
|
|
|
|
class Basic(IssueManager):
|
|
# Assign a name to the issue
|
|
issue_name = "basic"
|
|
def find_issues(self, **kwargs) -> None:
|
|
# Compute scores for each example
|
|
scores = np.random.rand(len(self.datalab.data))
|
|
|
|
# Construct a dataframe where examples are marked for issues
|
|
# and the score for each example is included.
|
|
self.issues = pd.DataFrame(
|
|
{
|
|
f"is_{self.issue_name}_issue" : scores < 0.1,
|
|
self.issue_score_key : scores,
|
|
},
|
|
)
|
|
|
|
# Score the dataset as a whole based on this issue type
|
|
self.summary = self.make_summary(score = scores.mean())
|
|
|
|
|
|
.. _intermediate_issue_manager:
|
|
|
|
Intermediate Issue Check
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
To create an intermediate issue:
|
|
|
|
- Perform the same steps as in the :ref:`basic issue check <basic_issue_manager>` section.
|
|
- Populate the `info` attribute with a dictionary of information about the identified issues.
|
|
|
|
The information can be included in a report generated by :py:class:`Datalab <cleanlab.datalab.datalab.Datalab>`,
|
|
if you add any of the keys to the `verbosity_levels` class-attribute.
|
|
Optionally, you can also add a description of the type of issue this issue manager handles to the `description` class-attribute.
|
|
|
|
.. code-block:: python
|
|
|
|
class Intermediate(IssueManager):
|
|
issue_name = "intermediate"
|
|
# Add a dictionary of information to include in the report
|
|
verbosity_levels = {
|
|
0: [],
|
|
1: ["std"],
|
|
2: ["raw_scores"],
|
|
}
|
|
# Add a description of the issue
|
|
description = "Intermediate issues are a bit more involved than basic issues."
|
|
def find_issues(self, *, intermediate_arg: int, **kwargs) -> None:
|
|
N = len(self.datalab.data)
|
|
raw_scores = np.random.rand(N)
|
|
std = raw_scores.std()
|
|
threshold = min(0, raw_scores.mean() - std)
|
|
sin_filter = np.sin(intermediate_arg * np.arange(N) / N)
|
|
kernel = sin_filter ** 2
|
|
scores = kernel * raw_scores
|
|
self.issues = pd.DataFrame(
|
|
{
|
|
f"is_{self.issue_name}_issue" : scores < threshold,
|
|
self.issue_score_key : scores,
|
|
},
|
|
)
|
|
self.summary = self.make_summary(score = scores.mean())
|
|
|
|
# Useful information that will be available in the Datalab instance
|
|
self.info = {
|
|
"std": std,
|
|
"raw_scores": raw_scores,
|
|
"kernel": kernel,
|
|
}
|
|
|
|
Advanced Issue Check
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
There could be different types of issues detected in a dataset. A local issue which affects individual data points in a dataset and can be tracked via `Datalab.issues` dataframe (to see which data points are exhibiting this type of issue). Alternatively, a global issue which affects the overall dataset but is not easily attributable to individual data points (hard to say one data point exhibits the issue but another does not). Even for global issues, we recommend trying to assign a per data point score (and boolean) if possible, see the Non-IID IssueManager as an example of this. Note that a global issue must have num_issues greater than 0 in its `issue_summary`, otherwise it won't show up in `Datalab.report()` by default.
|
|
|
|
|
|
Use with Datalab
|
|
----------------
|
|
|
|
We can create a
|
|
:py:class:`Datalab <cleanlab.datalab.datalab.Datalab>`
|
|
instance and run issue checks with the custom issue managers we created like so:
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
from cleanlab.datalab.internal.issue_manager_factory import register
|
|
from cleanlab import Datalab
|
|
|
|
|
|
# Register the issue manager
|
|
for issue_manager in [Basic, Intermediate]:
|
|
register(issue_manager)
|
|
|
|
# Instantiate a datalab instance
|
|
datalab = Datalab(data, label_name="label")
|
|
|
|
# Run the issue check
|
|
issue_types = {"basic": {}, "intermediate": {"intermediate_arg": 2}}
|
|
datalab.find_issues(issue_types=issue_types)
|
|
|
|
# Print report
|
|
datalab.report(verbosity=0)
|
|
|
|
|
|
The report will look something like this:
|
|
|
|
.. code-block:: text
|
|
|
|
Here is a summary of the different kinds of issues found in the data:
|
|
|
|
issue_type score num_issues
|
|
basic 0.477762 2
|
|
intermediate 0.286455 0
|
|
|
|
(Note: A lower score indicates a more severe issue across all examples in the dataset.)
|
|
|
|
|
|
------------------------------------------- basic issues -------------------------------------------
|
|
|
|
Number of examples with this issue: 2
|
|
Overall dataset quality in terms of this issue: 0.4778
|
|
|
|
Examples representing most severe instances of this issue:
|
|
is_basic_issue basic_score
|
|
13 True 0.003042
|
|
8 True 0.058117
|
|
11 False 0.121908
|
|
15 False 0.169312
|
|
17 False 0.229044
|
|
|
|
|
|
--------------------------------------- intermediate issues ----------------------------------------
|
|
|
|
About this issue:
|
|
Intermediate issues are a bit more involved than basic issues.
|
|
|
|
Number of examples with this issue: 0
|
|
Overall dataset quality in terms of this issue: 0.2865
|
|
|
|
Examples representing most severe instances of this issue:
|
|
is_intermediate_issue intermediate_score kernel
|
|
0 False 0.000000 0.0
|
|
1 False 0.007059 0.009967
|
|
3 False 0.010995 0.087332
|
|
2 False 0.016296 0.03947
|
|
11 False 0.019459 0.794251
|