wehub-resource-sync
591 lines
18 KiB
Plaintext
591 lines
18 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "42084110-295b-493a-9b3e-5d8d29ff78b3",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"# LLM RAG Evaluation with MLflow Example Notebook\n",
|
|
"\n",
|
|
"In this notebook, we will demonstrate how to evaluate various a RAG system with MLflow."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "bdff35e3-0e09-48b8-87ce-78759de88998",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"We need to set our OpenAI API key, since we will be using GPT-4 for our LLM-judged metrics.\n",
|
|
"\n",
|
|
"In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
|
|
"\n",
|
|
"`OPENAI_API_KEY=<your openai API key>`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "fb946228-62fb-4d68-9732-75935c9cb401",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"import mlflow"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "273d1345-95d7-435a-a7b6-a5f3dbb3f073",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"## Create a RAG system\n",
|
|
"\n",
|
|
"Use Langchain and Chroma to create a RAG system that answers questions based on the MLflow documentation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "2c28d0ad-f469-46ab-a2b4-c5e8db50a729",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.chains import RetrievalQA\n",
|
|
"from langchain.document_loaders import WebBaseLoader\n",
|
|
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
|
|
"from langchain.llms import OpenAI\n",
|
|
"from langchain.text_splitter import CharacterTextSplitter\n",
|
|
"from langchain.vectorstores import Chroma"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "83a7e77e-6717-472a-86dc-02e2c356ddef",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"loader = WebBaseLoader(\"https://mlflow.org/docs/latest/index.html\")\n",
|
|
"\n",
|
|
"documents = loader.load()\n",
|
|
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
|
|
"texts = text_splitter.split_documents(documents)\n",
|
|
"\n",
|
|
"embeddings = OpenAIEmbeddings()\n",
|
|
"docsearch = Chroma.from_documents(texts, embeddings)\n",
|
|
"\n",
|
|
"qa = RetrievalQA.from_chain_type(\n",
|
|
" llm=OpenAI(temperature=0),\n",
|
|
" chain_type=\"stuff\",\n",
|
|
" retriever=docsearch.as_retriever(),\n",
|
|
" return_source_documents=True,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "fd70bcf6-7c44-44d3-9435-567b82611e1c",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"## Evaluate the RAG system using `mlflow.evaluate()`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "de1bc359-2e40-459c-bea4-bed35a117988",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"Create a simple function that runs each input through the RAG chain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "667ec809-2bb5-4170-9937-6804386b41ec",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def model(input_df):\n",
|
|
" answer = []\n",
|
|
" for index, row in input_df.iterrows():\n",
|
|
" answer.append(qa(row[\"questions\"]))\n",
|
|
"\n",
|
|
" return answer"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "d1064306-b7f3-4b3e-825c-4353d808f21d",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"Create an eval dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "a5481491-e4a9-42ea-8a3f-f527faffd04d",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"eval_df = pd.DataFrame({\n",
|
|
" \"questions\": [\n",
|
|
" \"What is MLflow?\",\n",
|
|
" \"How to run mlflow.evaluate()?\",\n",
|
|
" \"How to log_table()?\",\n",
|
|
" \"How to load_table()?\",\n",
|
|
" ],\n",
|
|
"})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "9c3c8023-8feb-427a-b36d-34cd1853a5dc",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"source": [
|
|
"Create a faithfulness metric"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "3882b940-9c25-41ce-a301-72d8c0c90aaa",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from mlflow.metrics.genai.metric_definitions import faithfulness\n",
|
|
"\n",
|
|
"faithfulness_metric = faithfulness(model=\"openai:/gpt-4\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {
|
|
"byteLimit": 2048000,
|
|
"rowLimit": 10000
|
|
},
|
|
"inputWidgets": {},
|
|
"nuid": "ea40ce52-6ac7-4c20-9669-d24f80a6cebe",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023/10/23 13:13:16 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
|
|
"Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
|
|
"Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
|
|
"Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
|
|
"Using pad_token, but it is not set yet.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "23e9a5f58f1b4930ac47c88259156e1d",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/1 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: token_count\n",
|
|
"2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: toxicity\n",
|
|
"2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: perplexity\n",
|
|
"Using pad_token, but it is not set yet.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "2c6fd2067bad4404ad5550d56e23407e",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/1 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: flesch_kincaid_grade_level\n",
|
|
"2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ari_grade_level\n",
|
|
"2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: exact_match\n",
|
|
"2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: faithfulness\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'toxicity/v1/mean': 0.0002736186215770431, 'toxicity/v1/variance': 2.856656765360073e-08, 'toxicity/v1/p90': 0.0004570253004203551, 'toxicity/v1/ratio': 0.0, 'perplexity/v1/mean': 70.08646988868713, 'perplexity/v1/variance': 5233.465638493719, 'perplexity/v1/p90': 149.10144042968753, 'flesch_kincaid_grade_level/v1/mean': 7.625, 'flesch_kincaid_grade_level/v1/variance': 23.836875, 'flesch_kincaid_grade_level/v1/p90': 13.150000000000002, 'ari_grade_level/v1/mean': 9.450000000000001, 'ari_grade_level/v1/variance': 32.262499999999996, 'ari_grade_level/v1/p90': 15.870000000000001, 'faithfulness/v1/mean': 4.0, 'faithfulness/v1/variance': 3.0, 'faithfulness/v1/p90': 5.0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results = mlflow.evaluate(\n",
|
|
" model,\n",
|
|
" eval_df,\n",
|
|
" model_type=\"question-answering\",\n",
|
|
" evaluators=\"default\",\n",
|
|
" predictions=\"result\",\n",
|
|
" extra_metrics=[faithfulness_metric, mlflow.metrics.latency()],\n",
|
|
" evaluator_config={\n",
|
|
" \"col_mapping\": {\n",
|
|
" \"inputs\": \"questions\",\n",
|
|
" \"context\": \"source_documents\",\n",
|
|
" }\n",
|
|
" },\n",
|
|
")\n",
|
|
"print(results.metrics)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+cell": {
|
|
"cellMetadata": {},
|
|
"inputWidgets": {},
|
|
"nuid": "989a0861-5153-44e6-a19d-efcae7fe6cb5",
|
|
"showTitle": false,
|
|
"title": ""
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "4a4883be06c94983a171da51d14b40a3",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
"Downloading artifacts: 0%| | 0/1 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>questions</th>\n",
|
|
" <th>outputs</th>\n",
|
|
" <th>query</th>\n",
|
|
" <th>source_documents</th>\n",
|
|
" <th>latency</th>\n",
|
|
" <th>token_count</th>\n",
|
|
" <th>toxicity/v1/score</th>\n",
|
|
" <th>perplexity/v1/score</th>\n",
|
|
" <th>flesch_kincaid_grade_level/v1/score</th>\n",
|
|
" <th>ari_grade_level/v1/score</th>\n",
|
|
" <th>faithfulness/v1/score</th>\n",
|
|
" <th>faithfulness/v1/justification</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>What is MLflow?</td>\n",
|
|
" <td>MLflow is an open source platform for managin...</td>\n",
|
|
" <td>What is MLflow?</td>\n",
|
|
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
|
|
" <td>3.970739</td>\n",
|
|
" <td>176</td>\n",
|
|
" <td>0.000208</td>\n",
|
|
" <td>28.626591</td>\n",
|
|
" <td>15.4</td>\n",
|
|
" <td>18.9</td>\n",
|
|
" <td>5</td>\n",
|
|
" <td>The output provided by the model is a detailed...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>How to run MLflow.evaluate()?</td>\n",
|
|
" <td>\\n\\nYou can run MLflow.evaluate() by using the...</td>\n",
|
|
" <td>How to run MLflow.evaluate()?</td>\n",
|
|
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
|
|
" <td>1.083653</td>\n",
|
|
" <td>39</td>\n",
|
|
" <td>0.000179</td>\n",
|
|
" <td>44.533493</td>\n",
|
|
" <td>4.7</td>\n",
|
|
" <td>4.5</td>\n",
|
|
" <td>5</td>\n",
|
|
" <td>The output states that \"You can run MLflow.eva...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>How to log_table()?</td>\n",
|
|
" <td>\\n\\nYou can use the log_table() function in ML...</td>\n",
|
|
" <td>How to log_table()?</td>\n",
|
|
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
|
|
" <td>2.833117</td>\n",
|
|
" <td>114</td>\n",
|
|
" <td>0.000564</td>\n",
|
|
" <td>13.269521</td>\n",
|
|
" <td>7.9</td>\n",
|
|
" <td>8.8</td>\n",
|
|
" <td>1</td>\n",
|
|
" <td>The output provides a detailed explanation of ...</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>How to load_table()?</td>\n",
|
|
" <td>load_table() is not a function in MLflow.</td>\n",
|
|
" <td>How to load_table()?</td>\n",
|
|
" <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
|
|
" <td>3.736170</td>\n",
|
|
" <td>11</td>\n",
|
|
" <td>0.000144</td>\n",
|
|
" <td>193.916275</td>\n",
|
|
" <td>2.5</td>\n",
|
|
" <td>5.6</td>\n",
|
|
" <td>5</td>\n",
|
|
" <td>The output states that \"load_table() is not a ...</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>"
|
|
],
|
|
"text/plain": [
|
|
" questions \\\n",
|
|
"0 What is MLflow? \n",
|
|
"1 How to run MLflow.evaluate()? \n",
|
|
"2 How to log_table()? \n",
|
|
"3 How to load_table()? \n",
|
|
"\n",
|
|
" outputs \\\n",
|
|
"0 MLflow is an open source platform for managin... \n",
|
|
"1 \\n\\nYou can run MLflow.evaluate() by using the... \n",
|
|
"2 \\n\\nYou can use the log_table() function in ML... \n",
|
|
"3 load_table() is not a function in MLflow. \n",
|
|
"\n",
|
|
" query \\\n",
|
|
"0 What is MLflow? \n",
|
|
"1 How to run MLflow.evaluate()? \n",
|
|
"2 How to log_table()? \n",
|
|
"3 How to load_table()? \n",
|
|
"\n",
|
|
" source_documents latency token_count \\\n",
|
|
"0 [{'lc_attributes': {}, 'lc_namespace': ['langc... 3.970739 176 \n",
|
|
"1 [{'lc_attributes': {}, 'lc_namespace': ['langc... 1.083653 39 \n",
|
|
"2 [{'lc_attributes': {}, 'lc_namespace': ['langc... 2.833117 114 \n",
|
|
"3 [{'lc_attributes': {}, 'lc_namespace': ['langc... 3.736170 11 \n",
|
|
"\n",
|
|
" toxicity/v1/score perplexity/v1/score \\\n",
|
|
"0 0.000208 28.626591 \n",
|
|
"1 0.000179 44.533493 \n",
|
|
"2 0.000564 13.269521 \n",
|
|
"3 0.000144 193.916275 \n",
|
|
"\n",
|
|
" flesch_kincaid_grade_level/v1/score ari_grade_level/v1/score \\\n",
|
|
"0 15.4 18.9 \n",
|
|
"1 4.7 4.5 \n",
|
|
"2 7.9 8.8 \n",
|
|
"3 2.5 5.6 \n",
|
|
"\n",
|
|
" faithfulness/v1/score faithfulness/v1/justification \n",
|
|
"0 5 The output provided by the model is a detailed... \n",
|
|
"1 5 The output states that \"You can run MLflow.eva... \n",
|
|
"2 1 The output provides a detailed explanation of ... \n",
|
|
"3 5 The output states that \"load_table() is not a ... "
|
|
]
|
|
},
|
|
"execution_count": 18,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"results.tables[\"eval_results_table\"]"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"application/vnd.databricks.v1+notebook": {
|
|
"dashboards": [],
|
|
"language": "python",
|
|
"notebookMetadata": {
|
|
"pythonIndentUnit": 2
|
|
},
|
|
"notebookName": "LLM Evaluation Examples -- RAG",
|
|
"widgets": {}
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "mlflow-dev-env",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.8.17"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|