{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "42084110-295b-493a-9b3e-5d8d29ff78b3",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# LLM RAG Evaluation with MLflow Example Notebook\n",
    "\n",
    "In this notebook, we will demonstrate how to evaluate various a RAG system with MLflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "bdff35e3-0e09-48b8-87ce-78759de88998",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "We need to set our OpenAI API key, since we will be using GPT-4 for our LLM-judged metrics.\n",
    "\n",
    "In order to set your private key safely, please be sure to either export your key through a command-line terminal for your current instance, or, for a permanent addition to all user-based sessions, configure your favored environment management configuration file (i.e., .bashrc, .zshrc) to have the following entry:\n",
    "\n",
    "`OPENAI_API_KEY=<your openai API key>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fb946228-62fb-4d68-9732-75935c9cb401",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "import mlflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "273d1345-95d7-435a-a7b6-a5f3dbb3f073",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Create a RAG system\n",
    "\n",
    "Use Langchain and Chroma to create a RAG system that answers questions based on the MLflow documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2c28d0ad-f469-46ab-a2b4-c5e8db50a729",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain.document_loaders import WebBaseLoader\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.llms import OpenAI\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.vectorstores import Chroma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "83a7e77e-6717-472a-86dc-02e2c356ddef",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "loader = WebBaseLoader(\"https://mlflow.org/docs/latest/index.html\")\n",
    "\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "texts = text_splitter.split_documents(documents)\n",
    "\n",
    "embeddings = OpenAIEmbeddings()\n",
    "docsearch = Chroma.from_documents(texts, embeddings)\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=OpenAI(temperature=0),\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=docsearch.as_retriever(),\n",
    "    return_source_documents=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "fd70bcf6-7c44-44d3-9435-567b82611e1c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Evaluate the RAG system using `mlflow.evaluate()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "de1bc359-2e40-459c-bea4-bed35a117988",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Create a simple function that runs each input through the RAG chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "667ec809-2bb5-4170-9937-6804386b41ec",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "def model(input_df):\n",
    "    answer = []\n",
    "    for index, row in input_df.iterrows():\n",
    "        answer.append(qa(row[\"questions\"]))\n",
    "\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d1064306-b7f3-4b3e-825c-4353d808f21d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Create an eval dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a5481491-e4a9-42ea-8a3f-f527faffd04d",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "eval_df = pd.DataFrame({\n",
    "    \"questions\": [\n",
    "        \"What is MLflow?\",\n",
    "        \"How to run mlflow.evaluate()?\",\n",
    "        \"How to log_table()?\",\n",
    "        \"How to load_table()?\",\n",
    "    ],\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "9c3c8023-8feb-427a-b36d-34cd1853a5dc",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Create a faithfulness metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3882b940-9c25-41ce-a301-72d8c0c90aaa",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from mlflow.metrics.genai.metric_definitions import faithfulness\n",
    "\n",
    "faithfulness_metric = faithfulness(model=\"openai:/gpt-4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ea40ce52-6ac7-4c20-9669-d24f80a6cebe",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/10/23 13:13:16 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
      "Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
      "Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
      "Number of requested results 4 is greater than number of elements in index 3, updating n_results = 3\n",
      "Using pad_token, but it is not set yet.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "23e9a5f58f1b4930ac47c88259156e1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: token_count\n",
      "2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: toxicity\n",
      "2023/10/23 13:13:41 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: perplexity\n",
      "Using pad_token, but it is not set yet.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c6fd2067bad4404ad5550d56e23407e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: flesch_kincaid_grade_level\n",
      "2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ari_grade_level\n",
      "2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: exact_match\n",
      "2023/10/23 13:13:44 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: faithfulness\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'toxicity/v1/mean': 0.0002736186215770431, 'toxicity/v1/variance': 2.856656765360073e-08, 'toxicity/v1/p90': 0.0004570253004203551, 'toxicity/v1/ratio': 0.0, 'perplexity/v1/mean': 70.08646988868713, 'perplexity/v1/variance': 5233.465638493719, 'perplexity/v1/p90': 149.10144042968753, 'flesch_kincaid_grade_level/v1/mean': 7.625, 'flesch_kincaid_grade_level/v1/variance': 23.836875, 'flesch_kincaid_grade_level/v1/p90': 13.150000000000002, 'ari_grade_level/v1/mean': 9.450000000000001, 'ari_grade_level/v1/variance': 32.262499999999996, 'ari_grade_level/v1/p90': 15.870000000000001, 'faithfulness/v1/mean': 4.0, 'faithfulness/v1/variance': 3.0, 'faithfulness/v1/p90': 5.0}\n"
     ]
    }
   ],
   "source": [
    "results = mlflow.evaluate(\n",
    "    model,\n",
    "    eval_df,\n",
    "    model_type=\"question-answering\",\n",
    "    evaluators=\"default\",\n",
    "    predictions=\"result\",\n",
    "    extra_metrics=[faithfulness_metric, mlflow.metrics.latency()],\n",
    "    evaluator_config={\n",
    "        \"col_mapping\": {\n",
    "            \"inputs\": \"questions\",\n",
    "            \"context\": \"source_documents\",\n",
    "        }\n",
    "    },\n",
    ")\n",
    "print(results.metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "989a0861-5153-44e6-a19d-efcae7fe6cb5",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a4883be06c94983a171da51d14b40a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading artifacts:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>questions</th>\n",
       "      <th>outputs</th>\n",
       "      <th>query</th>\n",
       "      <th>source_documents</th>\n",
       "      <th>latency</th>\n",
       "      <th>token_count</th>\n",
       "      <th>toxicity/v1/score</th>\n",
       "      <th>perplexity/v1/score</th>\n",
       "      <th>flesch_kincaid_grade_level/v1/score</th>\n",
       "      <th>ari_grade_level/v1/score</th>\n",
       "      <th>faithfulness/v1/score</th>\n",
       "      <th>faithfulness/v1/justification</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is MLflow?</td>\n",
       "      <td>MLflow is an open source platform for managin...</td>\n",
       "      <td>What is MLflow?</td>\n",
       "      <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
       "      <td>3.970739</td>\n",
       "      <td>176</td>\n",
       "      <td>0.000208</td>\n",
       "      <td>28.626591</td>\n",
       "      <td>15.4</td>\n",
       "      <td>18.9</td>\n",
       "      <td>5</td>\n",
       "      <td>The output provided by the model is a detailed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How to run MLflow.evaluate()?</td>\n",
       "      <td>\\n\\nYou can run MLflow.evaluate() by using the...</td>\n",
       "      <td>How to run MLflow.evaluate()?</td>\n",
       "      <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
       "      <td>1.083653</td>\n",
       "      <td>39</td>\n",
       "      <td>0.000179</td>\n",
       "      <td>44.533493</td>\n",
       "      <td>4.7</td>\n",
       "      <td>4.5</td>\n",
       "      <td>5</td>\n",
       "      <td>The output states that \"You can run MLflow.eva...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>How to log_table()?</td>\n",
       "      <td>\\n\\nYou can use the log_table() function in ML...</td>\n",
       "      <td>How to log_table()?</td>\n",
       "      <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
       "      <td>2.833117</td>\n",
       "      <td>114</td>\n",
       "      <td>0.000564</td>\n",
       "      <td>13.269521</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.8</td>\n",
       "      <td>1</td>\n",
       "      <td>The output provides a detailed explanation of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How to load_table()?</td>\n",
       "      <td>load_table() is not a function in MLflow.</td>\n",
       "      <td>How to load_table()?</td>\n",
       "      <td>[{'lc_attributes': {}, 'lc_namespace': ['langc...</td>\n",
       "      <td>3.736170</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000144</td>\n",
       "      <td>193.916275</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.6</td>\n",
       "      <td>5</td>\n",
       "      <td>The output states that \"load_table() is not a ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       questions  \\\n",
       "0                What is MLflow?   \n",
       "1  How to run MLflow.evaluate()?   \n",
       "2            How to log_table()?   \n",
       "3           How to load_table()?   \n",
       "\n",
       "                                             outputs  \\\n",
       "0   MLflow is an open source platform for managin...   \n",
       "1  \\n\\nYou can run MLflow.evaluate() by using the...   \n",
       "2  \\n\\nYou can use the log_table() function in ML...   \n",
       "3          load_table() is not a function in MLflow.   \n",
       "\n",
       "                           query  \\\n",
       "0                What is MLflow?   \n",
       "1  How to run MLflow.evaluate()?   \n",
       "2            How to log_table()?   \n",
       "3           How to load_table()?   \n",
       "\n",
       "                                    source_documents   latency  token_count  \\\n",
       "0  [{'lc_attributes': {}, 'lc_namespace': ['langc...  3.970739          176   \n",
       "1  [{'lc_attributes': {}, 'lc_namespace': ['langc...  1.083653           39   \n",
       "2  [{'lc_attributes': {}, 'lc_namespace': ['langc...  2.833117          114   \n",
       "3  [{'lc_attributes': {}, 'lc_namespace': ['langc...  3.736170           11   \n",
       "\n",
       "   toxicity/v1/score  perplexity/v1/score  \\\n",
       "0           0.000208            28.626591   \n",
       "1           0.000179            44.533493   \n",
       "2           0.000564            13.269521   \n",
       "3           0.000144           193.916275   \n",
       "\n",
       "   flesch_kincaid_grade_level/v1/score  ari_grade_level/v1/score  \\\n",
       "0                                 15.4                      18.9   \n",
       "1                                  4.7                       4.5   \n",
       "2                                  7.9                       8.8   \n",
       "3                                  2.5                       5.6   \n",
       "\n",
       "   faithfulness/v1/score                      faithfulness/v1/justification  \n",
       "0                      5  The output provided by the model is a detailed...  \n",
       "1                      5  The output states that \"You can run MLflow.eva...  \n",
       "2                      1  The output provides a detailed explanation of ...  \n",
       "3                      5  The output states that \"load_table() is not a ...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.tables[\"eval_results_table\"]"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2
   },
   "notebookName": "LLM Evaluation Examples -- RAG",
   "widgets": {}
  },
  "kernelspec": {
   "display_name": "mlflow-dev-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}