wehub-resource-sync
225 lines
9.3 KiB
ReStructuredText
225 lines
9.3 KiB
ReStructuredText
mlflow.metrics
|
|
==============
|
|
|
|
The ``mlflow.metrics`` module helps you quantitatively and qualitatively measure your models.
|
|
|
|
.. autoclass:: mlflow.metrics.EvaluationMetric
|
|
|
|
These :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>` are used by the :py:func:`mlflow.evaluate()` API, either computed automatically depending on the ``model_type`` or specified via the ``extra_metrics`` parameter.
|
|
|
|
The following code demonstrates how to use :py:func:`mlflow.evaluate()` with an :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>`.
|
|
|
|
.. code-block:: python
|
|
|
|
import mlflow
|
|
from mlflow.metrics.genai import EvaluationExample, answer_similarity
|
|
|
|
eval_df = pd.DataFrame(
|
|
{
|
|
"inputs": [
|
|
"What is MLflow?",
|
|
],
|
|
"ground_truth": [
|
|
"MLflow is the largest open source AI engineering platform for agents, LLM applications, and ML models. It was developed by Databricks, a company that specializes in data and AI solutions. MLflow is designed to address the challenges that data scientists and AI engineers face when developing, evaluating, and deploying AI applications.",
|
|
],
|
|
}
|
|
)
|
|
|
|
example = EvaluationExample(
|
|
input="What is MLflow?",
|
|
output="MLflow is the largest open source AI engineering platform "
|
|
"for agents, LLM applications, and ML models, including tracing, "
|
|
"evaluation, prompt management, experiment tracking, and deployment.",
|
|
score=4,
|
|
justification="The definition effectively explains what MLflow is "
|
|
"its purpose, and its developer. It could be more concise for a 5-score.",
|
|
grading_context={
|
|
"ground_truth": "MLflow is the largest open source AI engineering "
|
|
"platform for agents, LLM applications, and ML models. It was "
|
|
"developed by Databricks, a company that specializes in data and "
|
|
"AI solutions. MLflow is designed to address the challenges that "
|
|
"data scientists and AI engineers face when developing, evaluating, "
|
|
"and deploying AI applications."
|
|
},
|
|
)
|
|
answer_similarity_metric = answer_similarity(examples=[example])
|
|
results = mlflow.evaluate(
|
|
logged_model.model_uri,
|
|
eval_df,
|
|
targets="ground_truth",
|
|
model_type="question-answering",
|
|
extra_metrics=[answer_similarity_metric],
|
|
)
|
|
|
|
Information about how an :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>` is calculated, such as the grading prompt used is available via the ``metric_details`` property.
|
|
|
|
.. code-block:: python
|
|
|
|
import mlflow
|
|
from mlflow.metrics.genai import relevance
|
|
|
|
my_relevance_metric = relevance()
|
|
print(my_relevance_metric.metric_details)
|
|
|
|
Evaluation results are stored as :py:class:`MetricValue <mlflow.metrics.MetricValue>`. Aggregate results are logged to the MLflow run as metrics, while per-example results are logged to the MLflow run as artifacts in the form of an evaluation table.
|
|
|
|
.. autoclass:: mlflow.metrics.MetricValue
|
|
|
|
We provide the following builtin factory functions to create :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>` for evaluating models. These metrics are computed automatically depending on the ``model_type``. For more information on the ``model_type`` parameter, see :py:func:`mlflow.evaluate()` API.
|
|
|
|
Regressor Metrics
|
|
-----------------
|
|
|
|
.. autofunction:: mlflow.metrics.mae
|
|
|
|
.. autofunction:: mlflow.metrics.mape
|
|
|
|
.. autofunction:: mlflow.metrics.max_error
|
|
|
|
.. autofunction:: mlflow.metrics.mse
|
|
|
|
.. autofunction:: mlflow.metrics.rmse
|
|
|
|
.. autofunction:: mlflow.metrics.r2_score
|
|
|
|
Classifier Metrics
|
|
------------------
|
|
|
|
.. autofunction:: mlflow.metrics.precision_score
|
|
|
|
.. autofunction:: mlflow.metrics.recall_score
|
|
|
|
.. autofunction:: mlflow.metrics.f1_score
|
|
|
|
Text Metrics
|
|
------------
|
|
|
|
.. autofunction:: mlflow.metrics.ari_grade_level
|
|
|
|
.. autofunction:: mlflow.metrics.flesch_kincaid_grade_level
|
|
|
|
Question Answering Metrics
|
|
---------------------------
|
|
|
|
Includes all of the above **Text Metrics** as well as the following:
|
|
|
|
.. autofunction:: mlflow.metrics.exact_match
|
|
|
|
.. autofunction:: mlflow.metrics.rouge1
|
|
|
|
.. autofunction:: mlflow.metrics.rouge2
|
|
|
|
.. autofunction:: mlflow.metrics.rougeL
|
|
|
|
.. autofunction:: mlflow.metrics.rougeLsum
|
|
|
|
.. autofunction:: mlflow.metrics.toxicity
|
|
|
|
.. autofunction:: mlflow.metrics.token_count
|
|
|
|
.. autofunction:: mlflow.metrics.latency
|
|
|
|
.. autofunction:: mlflow.metrics.bleu
|
|
|
|
Retriever Metrics
|
|
-----------------
|
|
|
|
The following metrics are built-in metrics for the ``'retriever'`` model type, meaning they will be
|
|
automatically calculated with a default ``retriever_k`` value of 3.
|
|
|
|
To evaluate document retrieval models, it is recommended to use a dataset with the following
|
|
columns:
|
|
|
|
- Input queries
|
|
- Retrieved relevant doc IDs
|
|
- Ground-truth doc IDs
|
|
|
|
Alternatively, you can also provide a function through the ``model`` parameter to represent
|
|
your retrieval model. The function should take a Pandas DataFrame containing input queries and
|
|
ground-truth relevant doc IDs, and return a DataFrame with a column of retrieved relevant doc IDs.
|
|
|
|
A "doc ID" is a string or integer that uniquely identifies a document. Each row of the retrieved and
|
|
ground-truth doc ID columns should consist of a list or numpy array of doc IDs.
|
|
|
|
Parameters:
|
|
|
|
- ``targets``: A string specifying the column name of the ground-truth relevant doc IDs
|
|
- ``predictions``: A string specifying the column name of the retrieved relevant doc IDs in either
|
|
the static dataset or the Dataframe returned by the ``model`` function
|
|
- ``retriever_k``: A positive integer specifying the number of retrieved docs IDs to consider for
|
|
each input query. ``retriever_k`` defaults to 3. You can change ``retriever_k`` by using the
|
|
:py:func:`mlflow.evaluate` API:
|
|
|
|
1. .. code-block:: python
|
|
|
|
# with a model and using `evaluator_config`
|
|
mlflow.evaluate(
|
|
model=retriever_function,
|
|
data=data,
|
|
targets="ground_truth",
|
|
model_type="retriever",
|
|
evaluators="default",
|
|
evaluator_config={"retriever_k": 5}
|
|
)
|
|
2. .. code-block:: python
|
|
|
|
# with a static dataset and using `extra_metrics`
|
|
mlflow.evaluate(
|
|
data=data,
|
|
predictions="predictions_param",
|
|
targets="targets_param",
|
|
model_type="retriever",
|
|
extra_metrics = [
|
|
mlflow.metrics.precision_at_k(5),
|
|
mlflow.metrics.precision_at_k(6),
|
|
mlflow.metrics.recall_at_k(5),
|
|
mlflow.metrics.ndcg_at_k(5)
|
|
]
|
|
)
|
|
|
|
NOTE: In the 2nd method, it is recommended to omit the ``model_type`` as well, or else
|
|
``precision@3`` and ``recall@3`` will be calculated in addition to ``precision@5``,
|
|
``precision@6``, ``recall@5``, and ``ndcg_at_k@5``.
|
|
|
|
.. autofunction:: mlflow.metrics.precision_at_k
|
|
|
|
.. autofunction:: mlflow.metrics.recall_at_k
|
|
|
|
.. autofunction:: mlflow.metrics.ndcg_at_k
|
|
|
|
Users create their own :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>` using the :py:func:`make_metric <mlflow.metrics.make_metric>` factory function
|
|
|
|
.. autofunction:: mlflow.metrics.make_metric
|
|
|
|
.. automodule:: mlflow.metrics
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|
|
:exclude-members: MetricValue, EvaluationMetric, make_metric, EvaluationExample, ari_grade_level, flesch_kincaid_grade_level, exact_match, rouge1, rouge2, rougeL, rougeLsum, toxicity, answer_similarity, answer_correctness, faithfulness, answer_relevance, mae, mape, max_error, mse, rmse, r2_score, precision_score, recall_score, f1_score, token_count, latency, precision_at_k, recall_at_k, ndcg_at_k, bleu
|
|
|
|
Generative AI Metrics
|
|
---------------------
|
|
|
|
We also provide generative AI ("genai") :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>`\s for evaluating text models. These metrics use an LLM to evaluate the quality of a model's output text. Note that your use of a third party LLM service (e.g., OpenAI) for evaluation may be subject to and governed by the LLM service's terms of use. The following factory functions help you customize the intelligent metric to your use case.
|
|
|
|
.. automodule:: mlflow.metrics.genai
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|
|
:exclude-members: EvaluationExample, make_genai_metric
|
|
|
|
You can also create your own generative AI :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>`\s using the :py:func:`make_genai_metric <mlflow.metrics.genai.make_genai_metric>` factory function.
|
|
|
|
.. autofunction:: mlflow.metrics.genai.make_genai_metric
|
|
|
|
When using generative AI :py:class:`EvaluationMetric <mlflow.metrics.EvaluationMetric>`\s, it is important to pass in an :py:class:`EvaluationExample <mlflow.metrics.genai.EvaluationExample>`
|
|
|
|
.. autoclass:: mlflow.metrics.genai.EvaluationExample
|
|
|
|
Users must set the appropriate environment variables for the LLM service they are using for
|
|
evaluation. For example, if you are using OpenAI's API, you must set the ``OPENAI_API_KEY``
|
|
environment variable. If using Azure OpenAI, you must also set the ``OPENAI_API_TYPE``,
|
|
``OPENAI_API_VERSION``, ``OPENAI_API_BASE``, and ``OPENAI_DEPLOYMENT_NAME`` environment variables.
|
|
See `Azure OpenAI documentation <https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints>`_
|
|
Users do not need to set these environment variables if they are using a gateway route.
|