wehub-resource-sync
236 lines
6.4 KiB
Plaintext
236 lines
6.4 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4f286a60",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Using DeepEval with OpenAI\n",
|
|
"\n",
|
|
"This guide will help you to evalaute LLM calls using OpenAI SDK, both as a standalone LLM call and as a part of LLM application. DeepEval's OpenAI integrations takes care of generating LLM spans for OpenAI SDK calls and it is fully compatible with the native `observe` decorator. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "124c28e0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install openai -U deepeval ipywidgets"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4aa94394",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = \"<your-openai-api-key>\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fbcb3b26",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.openai import OpenAI\n",
|
|
"\n",
|
|
"client = OpenAI()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6f236ac7",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Evaluating OpenAI SDK as a standalone LLM call\n",
|
|
"\n",
|
|
"There are 3 simple steps to evaluate OpenAI SDK as a standalone LLM call:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8c42d53e",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Create an evalaution dataset with goldens."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "87638cfa",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.dataset import Golden, EvaluationDataset\n",
|
|
"\n",
|
|
"goldens = [\n",
|
|
" Golden(\n",
|
|
" input=\"What are the top 5 most popular palces to eat in New York City?\"\n",
|
|
" ),\n",
|
|
" Golden(input=\"What is the weather in Paris, France?\"),\n",
|
|
"]\n",
|
|
"\n",
|
|
"dataset = EvaluationDataset(goldens=goldens)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fd14318e",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Select the metrics to evaluate.\n",
|
|
"\n",
|
|
"Note: The current integrations only supports metrics with input, output and tools called. This means that the only eligible metrics are those which have required arguments as `input`, `output` and `tools_called`. However you can still set the other test cases parameters like (`expected_output` or `context`) in the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "31540d2f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.metrics import AnswerRelevancyMetric, BiasMetric\n",
|
|
"\n",
|
|
"metrics = [AnswerRelevancyMetric(), BiasMetric()]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "99e14530",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Run the evals \n",
|
|
"\n",
|
|
"The `evals_iterator` from `EvaluationDataset` object returns a generator of goldens. You can iterate through the goldens and run the evals. If you want to set more parameters for the test cases, you can set them in the `LlmSpanContext` object."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "85d27a82",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.tracing import trace, LlmSpanContext\n",
|
|
"\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" # run OpenAI client\n",
|
|
" with trace(\n",
|
|
" llm_span_context=LlmSpanContext(\n",
|
|
" metrics=metrics,\n",
|
|
" expected_output=golden.expected_output,\n",
|
|
" )\n",
|
|
" ):\n",
|
|
" client.chat.completions.create(\n",
|
|
" model=\"gpt-4o\",\n",
|
|
" messages=[\n",
|
|
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
|
|
" {\"role\": \"user\", \"content\": golden.input},\n",
|
|
" ],\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "69584c14",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Evaluating OpenAI as SDK as a part of LLM application"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5b2194c1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from deepeval.tracing import observe\n",
|
|
"\n",
|
|
"\n",
|
|
"@observe()\n",
|
|
"def retrieve_docs(query):\n",
|
|
" return [\n",
|
|
" \"Paris is the capital and most populous city of France.\",\n",
|
|
" \"It has been a major European center of finance, diplomacy, commerce, and science.\",\n",
|
|
" ]\n",
|
|
"\n",
|
|
"\n",
|
|
"@observe()\n",
|
|
"def llm_app(input):\n",
|
|
" with trace(\n",
|
|
" llm_span_context=LlmSpanContext(\n",
|
|
" metrics=[AnswerRelevancyMetric(), BiasMetric()],\n",
|
|
" ),\n",
|
|
" ):\n",
|
|
" response = client.chat.completions.create(\n",
|
|
" model=\"gpt-4o\",\n",
|
|
" messages=[\n",
|
|
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"\\n\".join(retrieve_docs(input))\n",
|
|
" + \"\\n\\nQuestion: \"\n",
|
|
" + input,\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" )\n",
|
|
" return response.choices[0].message.content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5ec0043f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create dataset\n",
|
|
"dataset = EvaluationDataset(\n",
|
|
" goldens=[\n",
|
|
" Golden(\n",
|
|
" input=\"What are the top 5 most popular palces to eat in New York City?\"\n",
|
|
" ),\n",
|
|
" Golden(input=\"What is the weather in Paris, France?\"),\n",
|
|
" ]\n",
|
|
")\n",
|
|
"\n",
|
|
"# Iterate through goldens\n",
|
|
"for golden in dataset.evals_iterator():\n",
|
|
" # run your LLM application\n",
|
|
" llm_app(input=golden.input)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": ".venv",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|