wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
373 lines
11 KiB
Plaintext
373 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/evaluation/Deepeval.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 🚀 DeepEval - Open Source Evals with Tracing\n",
|
|
"\n",
|
|
"This code tutorial shows how you can easily trace and evaluate your LlamaIndex Agents. You can read more about the DeepEval framework here: https://docs.confident-ai.com/docs/getting-started\n",
|
|
"\n",
|
|
"LlamaIndex integration with DeepEval allows you to trace your LlamaIndex Agents and evaluate them using DeepEval's default metrics. Read more about the integration here: https://deepeval.com/integrations/frameworks/langchain\n",
|
|
"\n",
|
|
"Feel free to check out our repository here on GitHub: https://github.com/confident-ai/deepeval"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Quickstart\n",
|
|
"\n",
|
|
"Install the following packages:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -q -q llama-index\n",
|
|
"!pip install -U -q deepeval"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This step is optional and only if you want a server-hosted dashboard! (Psst I think you should!)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!deepeval login"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### End-to-End Evals\n",
|
|
"\n",
|
|
"`deepeval` allows you to evaluate LlamaIndex applications end-to-end in under a minute.\n",
|
|
"\n",
|
|
"Create a `FunctionAgent` with a list of metrics you wish to use, and pass it to your LlamaIndex application's `run` method."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import asyncio\n",
|
|
"\n",
|
|
"from llama_index.llms.openai import OpenAI\n",
|
|
"import llama_index.core.instrumentation as instrument\n",
|
|
"\n",
|
|
"from deepeval.integrations.llama_index import (\n",
|
|
" instrument_llama_index,\n",
|
|
" FunctionAgent,\n",
|
|
")\n",
|
|
"from deepeval.metrics import AnswerRelevancyMetric\n",
|
|
"\n",
|
|
"instrument_llama_index(instrument.get_dispatcher())\n",
|
|
"\n",
|
|
"\n",
|
|
"def multiply(a: float, b: float) -> float:\n",
|
|
" \"\"\"Useful for multiplying two numbers.\"\"\"\n",
|
|
" return a * b\n",
|
|
"\n",
|
|
"\n",
|
|
"answer_relevancy_metric = AnswerRelevancyMetric()\n",
|
|
"\n",
|
|
"agent = FunctionAgent(\n",
|
|
" tools=[multiply],\n",
|
|
" llm=OpenAI(model=\"gpt-4o-mini\"),\n",
|
|
" system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
|
|
" metrics=[answer_relevancy_metric],\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def llm_app(input: str):\n",
|
|
" return await agent.run(input)\n",
|
|
"\n",
|
|
"\n",
|
|
"asyncio.run(llm_app(\"What is 2 * 3?\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Evaluations are supported for LlamaIndex `FunctionAgent`, `ReActAgent` and `CodeActAgent`. Only metrics with LLM parameters input and output are eligible for evaluation.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Synchronous\n",
|
|
"\n",
|
|
"Create a `FunctionAgent` with a list of metrics you wish to use, and pass it to your LlamaIndex application's run method."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.dataset import EvaluationDataset, Golden\n",
|
|
"\n",
|
|
"dataset = EvaluationDataset(\n",
|
|
" goldens=[Golden(input=\"What is 3 * 12?\"), Golden(input=\"What is 4 * 13?\")]\n",
|
|
")\n",
|
|
"\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" task = asyncio.create_task(llm_app(golden.input))\n",
|
|
" dataset.evaluate(task)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Asynchronous"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.dataset import EvaluationDataset, Golden\n",
|
|
"import asyncio\n",
|
|
"\n",
|
|
"dataset = EvaluationDataset(\n",
|
|
" goldens=[Golden(input=\"What's 7 * 8?\"), Golden(input=\"What's 7 * 6?\")]\n",
|
|
")\n",
|
|
"\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" task = asyncio.create_task(llm_app(golden.input))\n",
|
|
" dataset.evaluate(task)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### ⚠️ Warning: DeepEval runs using event loops for managing asynchronous operations.\n",
|
|
"\n",
|
|
"Jupyter notebooks already maintain their own event loop, which may lead to unexpected behavior, hangs, or runtime errors when running DeepEval examples directly in a notebook cell.\n",
|
|
"\n",
|
|
"Recommendation: To avoid such issues, run your DeepEval examples in a standalone Python script (.py file) instead of within Jupyter Notebook."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Examples\n",
|
|
"\n",
|
|
"Here are some examples scripts."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Synchronous (End-to-End Evals)\n",
|
|
"import os\n",
|
|
"import deepeval\n",
|
|
"import asyncio\n",
|
|
"\n",
|
|
"from llama_index.llms.openai import OpenAI\n",
|
|
"import llama_index.core.instrumentation as instrument\n",
|
|
"\n",
|
|
"from deepeval.integrations.llama_index import instrument_llama_index\n",
|
|
"from deepeval.integrations.llama_index import FunctionAgent\n",
|
|
"from deepeval.metrics import AnswerRelevancyMetric\n",
|
|
"from deepeval.dataset import EvaluationDataset, Golden\n",
|
|
"\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"\n",
|
|
"load_dotenv()\n",
|
|
"\n",
|
|
"deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
|
|
"instrument_llama_index(instrument.get_dispatcher())\n",
|
|
"\n",
|
|
"\n",
|
|
"def multiply(a: float, b: float) -> float:\n",
|
|
" \"\"\"Useful for multiplying two numbers.\"\"\"\n",
|
|
" return a * b\n",
|
|
"\n",
|
|
"\n",
|
|
"answer_relevancy_metric = AnswerRelevancyMetric()\n",
|
|
"agent = FunctionAgent(\n",
|
|
" tools=[multiply],\n",
|
|
" llm=OpenAI(model=\"gpt-4o-mini\"),\n",
|
|
" system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
|
|
" metrics=[answer_relevancy_metric],\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def llm_app(input: str):\n",
|
|
" return await agent.run(input)\n",
|
|
"\n",
|
|
"\n",
|
|
"dataset = EvaluationDataset(\n",
|
|
" goldens=[Golden(input=\"What is 3 * 12?\"), Golden(input=\"What is 4 * 13?\")]\n",
|
|
")\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" task = asyncio.create_task(llm_app(golden.input))\n",
|
|
" dataset.evaluate(task)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Asynchronous (End-to-End Evals)\n",
|
|
"import os\n",
|
|
"from deepeval.integrations.llama_index import instrument_llama_index\n",
|
|
"import llama_index.core.instrumentation as instrument\n",
|
|
"from deepeval.integrations.llama_index import FunctionAgent\n",
|
|
"from llama_index.llms.openai import OpenAI\n",
|
|
"import asyncio\n",
|
|
"import time\n",
|
|
"\n",
|
|
"import deepeval\n",
|
|
"from deepeval.metrics import AnswerRelevancyMetric\n",
|
|
"from deepeval.dataset import EvaluationDataset, Golden\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"\n",
|
|
"load_dotenv()\n",
|
|
"\n",
|
|
"\n",
|
|
"# Don't forget to setup tracing\n",
|
|
"deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
|
|
"instrument_llama_index(instrument.get_dispatcher())\n",
|
|
"\n",
|
|
"\n",
|
|
"def multiply(a: float, b: float) -> float:\n",
|
|
" \"\"\"Useful for multiplying two numbers.\"\"\"\n",
|
|
" return a * b\n",
|
|
"\n",
|
|
"\n",
|
|
"answer_relevancy_metric = AnswerRelevancyMetric()\n",
|
|
"agent = FunctionAgent(\n",
|
|
" tools=[multiply],\n",
|
|
" llm=OpenAI(model=\"gpt-4o-mini\"),\n",
|
|
" system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
|
|
" metrics=[answer_relevancy_metric],\n",
|
|
")\n",
|
|
"\n",
|
|
"goldens = [Golden(input=\"What's 7 * 8?\"), Golden(input=\"What's 7 * 6?\")]\n",
|
|
"\n",
|
|
"\n",
|
|
"async def llm_app(golden: Golden):\n",
|
|
" await agent.run(golden.input)\n",
|
|
"\n",
|
|
"\n",
|
|
"def main():\n",
|
|
" dataset = EvaluationDataset(goldens=goldens)\n",
|
|
" for golden in dataset.evals_iterator():\n",
|
|
" task = asyncio.create_task(llm_app(golden))\n",
|
|
" dataset.evaluate(task)\n",
|
|
"\n",
|
|
"\n",
|
|
"if __name__ == \"__main__\":\n",
|
|
" main()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"from deepeval.integrations.llama_index import instrument_llama_index\n",
|
|
"import llama_index.core.instrumentation as instrument\n",
|
|
"from deepeval.integrations.llama_index import FunctionAgent\n",
|
|
"from llama_index.llms.openai import OpenAI\n",
|
|
"import asyncio\n",
|
|
"\n",
|
|
"import deepeval\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"\n",
|
|
"load_dotenv()\n",
|
|
"\n",
|
|
"# Don't forget to setup tracing\n",
|
|
"deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
|
|
"instrument_llama_index(instrument.get_dispatcher())\n",
|
|
"\n",
|
|
"\n",
|
|
"def multiply(a: float, b: float) -> float:\n",
|
|
" \"\"\"Useful for multiplying two numbers.\"\"\"\n",
|
|
" return a * b\n",
|
|
"\n",
|
|
"\n",
|
|
"agent = FunctionAgent(\n",
|
|
" tools=[multiply],\n",
|
|
" llm=OpenAI(model=\"gpt-4o-mini\"),\n",
|
|
" system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
|
|
" metric_collection=\"test_collection_1\",\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"async def llm_app(golden: Golden):\n",
|
|
" await agent.run(golden.input)\n",
|
|
"\n",
|
|
"\n",
|
|
"asyncio.run(llm_app(Golden(input=\"What is 3 * 12?\")))"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "base",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|