wehub-resource-sync
262 lines
13 KiB
Plaintext
262 lines
13 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Evaluating CrewAI's `crew` (end-to-end)\n",
|
|
"\n",
|
|
"In this notebook we will demonstrate how you can run evaluations on crews using datasets from Confident AI and DeepEval's dataset iterator."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Install dependencies:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install -U deepeval -U crewai ipywidgets --quiet"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Set your OpenAI API key:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = \"<your-openai-api-key>\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create a crew:\n",
|
|
"\n",
|
|
"This is a simple crew with a single agent and a single task."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
|
|
"\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080\">╭─────────────────────────────────────────────── </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">Execution Traces</span><span style=\"color: #008080; text-decoration-color: #008080\"> ────────────────────────────────────────────────╮</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">🔍 Detailed execution traces are available!</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">View insights including:</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\"> • Agent decision-making process</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\"> • Task execution flow and timing</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\"> • Tool usage details</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
|
|
"<span style=\"color: #008080; text-decoration-color: #008080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[36m╭─\u001b[0m\u001b[36m──────────────────────────────────────────────\u001b[0m\u001b[36m \u001b[0m\u001b[1;36mExecution Traces\u001b[0m\u001b[36m \u001b[0m\u001b[36m───────────────────────────────────────────────\u001b[0m\u001b[36m─╮\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[1;36m🔍 \u001b[0m\u001b[1;36mDetailed execution traces are available!\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[37mView insights including:\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[94m • Agent decision-making process\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[94m • Task execution flow and timing\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[94m • Tool usage details\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m│\u001b[0m \u001b[36m│\u001b[0m\n",
|
|
"\u001b[36m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Would you like to view your execution traces? [y/N] (20s timeout): The biggest open source database, in terms of popularity and widespread use, is MySQL. MySQL is a relational database management system (RDBMS) that is based on Structured Query Language (SQL). It was originally developed in the mid-1990s and is now owned by Oracle Corporation. MySQL is known for its speed, reliability, and ease of use, making it a preferred choice for web applications and various enterprise applications. \n",
|
|
"\n",
|
|
"MySQL supports large databases, and when combined with various storage engines, it can efficiently manage large volumes of data. It is frequently used in scenarios where data integrity and performance are crucial, such as in content management systems like WordPress, e-commerce applications, and many other types of software applications.\n",
|
|
"\n",
|
|
"The community-driven aspect of MySQL, along with its extensive documentation and support, contributes to its status as the largest and most popular open source database. It also has a thriving ecosystem with numerous third-party tools and libraries that further enhance its capabilities. In summary, MySQL stands out as the leading open source database, making it a top choice for developers and organizations looking for robust database solutions at no cost.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from crewai import Task, Crew, Agent\n",
|
|
"\n",
|
|
"agent = Agent(\n",
|
|
" role=\"Consultant\",\n",
|
|
" goal=\"Write clear, concise explanation.\",\n",
|
|
" backstory=\"An expert consultant with a keen eye for software trends.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"task = Task(\n",
|
|
" description=\"Explain the given topic: {topic}\",\n",
|
|
" expected_output=\"A clear and concise explanation.\",\n",
|
|
" agent=agent,\n",
|
|
")\n",
|
|
"\n",
|
|
"crew = Crew(agents=[agent], tasks=[task])\n",
|
|
"\n",
|
|
"result = crew.kickoff(\n",
|
|
" inputs={\"topic\": \"What is the biggest open source database?\"}\n",
|
|
")\n",
|
|
"print(result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Evaluate the agent\n",
|
|
"\n",
|
|
"To evaluate CrewAI's `crew`:\n",
|
|
"\n",
|
|
"1. Instrument the application (using `from deepeval.integrations.crewai import instrument_crewai`)\n",
|
|
"2. Supply metrics to `kickoff`.\n",
|
|
"\n",
|
|
"\n",
|
|
"> (Pro Tip) View your Agent's trace and publish test runs on [Confident AI](https://www.confident-ai.com/). Apart from this you get an in-house dataset editor and more advaced tools to monitor and enventually improve your Agent's performance. Get your API key from [here](https://app.confident-ai.com/)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"os.environ[\"CONFIDENT_API_KEY\"] = \"<your-confident-api-key>\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Overriding of current TracerProvider is not allowed\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from deepeval.integrations.crewai import instrument_crewai\n",
|
|
"\n",
|
|
"instrument_crewai()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Using a dataset from Confident AI:\n",
|
|
"\n",
|
|
"For demo purposes, we will use a public dataset from Confident AI. You can use your own dataset as well. Refer to the [docs](https://deepeval.com/docs/evaluation-end-to-end-llm-evals#setup-your-test-environment) to learn more about how to create your own dataset."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.dataset import EvaluationDataset\n",
|
|
"\n",
|
|
"dataset = EvaluationDataset()\n",
|
|
"dataset.pull(alias=\"topic_agent_queries\", public=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Run evaluations:\n",
|
|
"\n",
|
|
"We will use the `AnswerRelevancyMetric` to evaluate the crew. Dataset iterator will yield golden examples from the dataset."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.metrics import AnswerRelevancyMetric\n",
|
|
"from deepeval.tracing import trace\n",
|
|
"\n",
|
|
"answer_relavancy_metric = AnswerRelevancyMetric()\n",
|
|
"\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" with trace(trace_metrics=[answer_relavancy_metric]):\n",
|
|
" result = crew.kickoff(\n",
|
|
" inputs={\"topic\": golden.input}, metrics=[AnswerRelevancyMetric()]\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Congratulation! You have just evaluated your first CrewAI's `crew` using Deepeval. Try changing Hyperparameters, Agents, Tasks, Metrics and see how your agent performs."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": ".venv",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|