wehub-resource-sync
41 lines
2.0 KiB
Markdown
41 lines
2.0 KiB
Markdown
### MLflow evaluation Examples
|
|
|
|
The examples in this directory demonstrate how to use the `mlflow.evaluate()` API. Specifically,
|
|
they show how to evaluate a PyFunc model on a specified dataset using the builtin default evaluator
|
|
and specified extra metrics, where the resulting metrics & artifacts are logged to MLflow Tracking.
|
|
They also show how to specify validation thresholds for the resulting metrics to validate the quality
|
|
of your model. See full list of examples below:
|
|
|
|
- Example `evaluate_on_binary_classifier.py` evaluates an xgboost `XGBClassifier` model on dataset loaded by
|
|
`shap.datasets.adult`.
|
|
- Example `evaluate_on_multiclass_classifier.py` evaluates a scikit-learn `LogisticRegression` model on dataset
|
|
generated by `sklearn.datasets.make_classification`.
|
|
- Example `evaluate_on_regressor.py` evaluate as scikit-learn `LinearRegression` model on dataset loaded by
|
|
`sklearn.datasets.load_diabetes`
|
|
- Example `evaluate_with_custom_metrics.py` evaluates a scikit-learn `LinearRegression`
|
|
model with a custom metric function on dataset loaded by `sklearn.datasets.load_diabetes`
|
|
- Example `evaluate_with_custom_metrics_comprehensive.py` evaluates a scikit-learn `LinearRegression` model
|
|
with a comprehensive list of custom metric functions on dataset loaded by `sklearn.datasets.load_diabetes`
|
|
- Example `evaluate_with_model_validation.py` trains both a candidate xgboost `XGBClassifier` model
|
|
and a baseline `DummyClassifier` model on dataset loaded by `shap.datasets.adult`. Then, it validates
|
|
the candidate model against specified thresholds on both builtin and extra metrics and the dummy model.
|
|
|
|
#### Prerequisites
|
|
|
|
```
|
|
pip install scikit-learn xgboost shap>=0.40 matplotlib
|
|
```
|
|
|
|
#### How to run the examples
|
|
|
|
Run in this directory with Python.
|
|
|
|
```sh
|
|
python evaluate_on_binary_classifier.py
|
|
python evaluate_on_multiclass_classifier.py
|
|
python evaluate_on_regressor.py
|
|
python evaluate_with_custom_metrics.py
|
|
python evaluate_with_custom_metrics_comprehensive.py
|
|
python evaluate_with_model_validation.py
|
|
```
|