wehub-resource-sync
422 lines
15 KiB
Plaintext
422 lines
15 KiB
Plaintext
---
|
|
id: openai
|
|
title: OpenAI
|
|
sidebar_label: OpenAI
|
|
---
|
|
|
|
<IntegrationTagsDisplayer native={true} cicdEvals={true} traceability={true} />
|
|
|
|
[OpenAI](https://platform.openai.com/docs/) provides chat completions and responses APIs for building LLM applications.
|
|
|
|
The `deepeval` integration is a drop-in replacement for OpenAI's client. Every `client.chat.completions.create(...)` and `client.responses.create(...)` call becomes an LLM span you can evaluate, without rewriting how you call the API.
|
|
|
|
<AgentTraceTerminal
|
|
title="openai_app · deepeval"
|
|
ariaLabel="Example OpenAI client trace with per-step metric scores"
|
|
lines={[
|
|
{ kind: "cmd", name: "deepeval test run test_openai_app.py" },
|
|
{ kind: "blank" },
|
|
{ kind: "root", prefix: "●", name: "test_openai_app" },
|
|
{ kind: "blank", prefix: "│" },
|
|
{
|
|
kind: "llm",
|
|
prefix: "└─",
|
|
name: "gpt-4o · respond",
|
|
metric: "Answer Relevancy",
|
|
score: "0.93",
|
|
duration: "260ms",
|
|
pass: true,
|
|
},
|
|
{
|
|
kind: "llm",
|
|
prefix: " ",
|
|
name: "",
|
|
metric: "Faithfulness",
|
|
score: "0.41",
|
|
duration: "",
|
|
pass: false,
|
|
},
|
|
{ kind: "blank" },
|
|
{
|
|
kind: "summary",
|
|
name: "Trace score 0.67 · 1/2 metrics passed",
|
|
pass: false,
|
|
},
|
|
]}
|
|
/>
|
|
|
|
`deepeval`'s OpenAI integration enables you to:
|
|
|
|
- **Drop in `deepeval.openai.OpenAI`** — every chat completion or response produces an LLM span with input, output, and `tools_called` captured automatically.
|
|
- **Evaluate LLM calls** with any `deepeval` metric through `LlmSpanContext`.
|
|
- **Run evals from scripts or CI/CD** — same client, different surfaces.
|
|
- **Compose with `@observe` and `with trace(...)`** to evaluate larger flows that wrap one or more OpenAI calls.
|
|
|
|
## Getting Started
|
|
|
|
<Steps>
|
|
|
|
<Step>
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
pip install -U deepeval openai
|
|
```
|
|
|
|
`deepeval.openai.OpenAI` and `deepeval.openai.AsyncOpenAI` import OpenAI's classes and patch them in place. Existing kwargs, async paths, streaming, and tool-calling behavior all work unchanged.
|
|
|
|
</Step>
|
|
|
|
<Step>
|
|
|
|
### Instrument and evaluate
|
|
|
|
Replace `from openai import OpenAI` with `from deepeval.openai import OpenAI`. Wrap each call you want to evaluate in `with trace(llm_span_context=LlmSpanContext(metrics=[...]))`.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.openai import OpenAI
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
client = OpenAI()
|
|
|
|
# Goldens are the inputs you want to evaluate.
|
|
dataset = EvaluationDataset(goldens=[Golden(input="What's the capital of France?")])
|
|
|
|
for golden in dataset.evals_iterator():
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
|
|
client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[
|
|
{"role": "system", "content": "Be concise."},
|
|
{"role": "user", "content": golden.input},
|
|
],
|
|
)
|
|
```
|
|
|
|
Done ✅. You've run your first eval against an OpenAI call with full traceability via `deepeval`.
|
|
|
|
</Step>
|
|
|
|
</Steps>
|
|
|
|
## What gets traced
|
|
|
|
Each patched OpenAI call produces one **LLM span** under the active trace. When the call uses tool-calling, the span's `tools_called` field captures every tool invocation the model returned — no extra wiring needed.
|
|
|
|
- **LLM spans** — one per `chat.completions.create(...)`, `chat.completions.parse(...)`, or `responses.create(...)` call. Captures input messages, output text, token counts, and `tools_called`.
|
|
- **Trace** — auto-created when the call has no parent. If the call runs inside `with trace(...)` or `@observe`, the LLM span nests under that trace instead.
|
|
|
|
```text
|
|
Trace ← auto-created or user-owned
|
|
└── LLM: gpt-4o ← one client.chat.completions.create(...) call
|
|
```
|
|
|
|
The trace and its LLM span are independently evaluable.
|
|
|
|
## Running evals
|
|
|
|
There are two surfaces for running evals against OpenAI calls. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.
|
|
|
|
### In CI/CD (pytest)
|
|
|
|
Use the `deepeval` pytest integration. Each parametrized test invocation becomes one OpenAI call; failing metrics fail the test, which fails the build.
|
|
|
|
```python title="test_openai_app.py" showLineNumbers
|
|
import pytest
|
|
from deepeval import assert_test
|
|
from deepeval.openai import OpenAI
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
client = OpenAI()
|
|
dataset = EvaluationDataset(goldens=[
|
|
Golden(input="What's the capital of France?"),
|
|
Golden(input="Who wrote Hamlet?"),
|
|
])
|
|
|
|
@pytest.mark.parametrize("golden", dataset.goldens)
|
|
def test_openai_app(golden: Golden):
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
|
|
client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[
|
|
{"role": "system", "content": "Be concise."},
|
|
{"role": "user", "content": golden.input},
|
|
],
|
|
)
|
|
assert_test(golden=golden)
|
|
```
|
|
|
|
Run it with:
|
|
|
|
```bash
|
|
deepeval test run test_openai_app.py
|
|
```
|
|
|
|
### In a script
|
|
|
|
Use `EvaluationDataset` + `evals_iterator(...)`. Each `Golden` becomes one OpenAI call; metrics score the resulting LLM span.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
import asyncio
|
|
|
|
from deepeval.openai import AsyncOpenAI
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.evaluate.configs import AsyncConfig
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
client = AsyncOpenAI()
|
|
dataset = EvaluationDataset(goldens=[
|
|
Golden(input="What's the capital of France?"),
|
|
Golden(input="Who wrote Hamlet?"),
|
|
])
|
|
|
|
async def call_openai(prompt: str):
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
|
|
return await client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": prompt}],
|
|
)
|
|
|
|
for golden in dataset.evals_iterator(async_config=AsyncConfig(run_async=True)):
|
|
task = asyncio.create_task(call_openai(golden.input))
|
|
dataset.evaluate(task)
|
|
```
|
|
|
|
Sync (`OpenAI`) and async (`AsyncOpenAI`) clients both work; pick whichever matches your code.
|
|
|
|
## Applying metrics to LLM spans
|
|
|
|
Passing `metrics=[...]` to `LlmSpanContext` evaluates the next OpenAI call's LLM span specifically. The same context manager lets you attach extra evaluation parameters that some metrics need.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.openai import OpenAI
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
|
|
|
|
client = OpenAI()
|
|
|
|
with trace(
|
|
llm_span_context=LlmSpanContext(
|
|
metrics=[AnswerRelevancyMetric(), FaithfulnessMetric()],
|
|
retrieval_context=["Paris is the capital of France."],
|
|
),
|
|
):
|
|
client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": "What's the capital of France?"}],
|
|
)
|
|
```
|
|
|
|
`LlmSpanContext` accepts `metrics`, `expected_output`, `expected_tools`, `context`, `retrieval_context`, and `prompt`. Each one is read by the OpenAI patch when the next LLM span is created.
|
|
|
|
## Customizing trace and span data
|
|
|
|
The patch captures input messages, output text, and `tools_called` automatically. For anything else, the right API depends on where your code runs.
|
|
|
|
- Use `with trace(...)` for trace-level fields (`name`, `tags`, `metadata`, `thread_id`, `user_id`).
|
|
- Use `LlmSpanContext` for LLM-span-level fields the metric needs (`expected_output`, `retrieval_context`, etc.).
|
|
- Use `@observe` to wrap retrieval, post-processing, or any other step you want to see as its own span in the trace.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.openai import OpenAI
|
|
from deepeval.tracing import trace, LlmSpanContext, observe
|
|
|
|
client = OpenAI()
|
|
|
|
@observe(type="retriever")
|
|
def retrieve_docs(query: str) -> list[str]:
|
|
return ["Paris is the capital of France."]
|
|
|
|
@observe()
|
|
def respond_to_user(prompt: str) -> str:
|
|
docs = retrieve_docs(prompt)
|
|
with trace(
|
|
llm_span_context=LlmSpanContext(retrieval_context=docs),
|
|
user_id="user-123",
|
|
tags=["openai", "rag"],
|
|
):
|
|
response = client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[
|
|
{"role": "system", "content": "\n".join(docs)},
|
|
{"role": "user", "content": prompt},
|
|
],
|
|
)
|
|
return response.choices[0].message.content
|
|
```
|
|
|
|
## Advanced patterns
|
|
|
|
The primitives above — `deepeval.openai.OpenAI`, `LlmSpanContext`, `@observe`, `with trace(...)` — compose around one boundary: the patch owns each LLM call's span, and your code chooses what trace to put it inside.
|
|
|
|
### Wrap an OpenAI call in `@observe`
|
|
|
|
When the OpenAI call is part of a larger operation, decorate the outer function with `@observe`. The LLM span nests under your observed span automatically.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.tracing import observe, trace, LlmSpanContext
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
...
|
|
|
|
@observe(name="respond_to_user")
|
|
def respond_to_user(prompt: str) -> str:
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
|
|
response = client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": prompt}],
|
|
)
|
|
return response.choices[0].message.content
|
|
```
|
|
|
|
#### No trace-level metrics required
|
|
|
|
Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because `AnswerRelevancyMetric` is attached to the LLM span, so CI/CD and scripts only need to call the function.
|
|
|
|
This is how you'd run it:
|
|
|
|
<Tabs items={["CI/CD", "Scripts"]}>
|
|
<Tab value="CI/CD">
|
|
|
|
```python title="test_openai_app.py" showLineNumbers
|
|
import pytest
|
|
from deepeval import assert_test
|
|
...
|
|
|
|
@pytest.mark.parametrize("golden", dataset.goldens)
|
|
def test_respond_to_user(golden: Golden):
|
|
respond_to_user(golden.input)
|
|
assert_test(golden=golden)
|
|
```
|
|
|
|
```bash
|
|
deepeval test run test_openai_app.py
|
|
```
|
|
|
|
</Tab>
|
|
<Tab value="Scripts">
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
...
|
|
|
|
for golden in dataset.evals_iterator():
|
|
respond_to_user(golden.input)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### Multiple OpenAI calls under one trace
|
|
|
|
When a single logical unit of work makes several OpenAI calls (e.g. a planner call followed by a respond call), bracket them with `with trace(...)` so the LLM spans share a `trace_id` and show up as siblings under one root.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.tracing import trace
|
|
...
|
|
|
|
def plan_then_respond(prompt: str):
|
|
with trace(name="plan_then_respond"):
|
|
plan = client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": f"Plan: {prompt}"}],
|
|
)
|
|
return client.chat.completions.create(
|
|
model="gpt-4o",
|
|
messages=[{"role": "user", "content": plan.choices[0].message.content}],
|
|
)
|
|
```
|
|
|
|
### Tool-calling models
|
|
|
|
When the model returns tool calls, the LLM span's `tools_called` field captures them automatically. Use `expected_tools` on `LlmSpanContext` if you want to evaluate tool selection with a tool-aware metric.
|
|
|
|
```python title="openai_app.py" showLineNumbers
|
|
from deepeval.test_case import ToolCall
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
...
|
|
|
|
with trace(
|
|
llm_span_context=LlmSpanContext(
|
|
expected_tools=[ToolCall(name="get_weather", input_parameters={"city": "Paris"})],
|
|
),
|
|
):
|
|
client.chat.completions.create(model="gpt-4o", messages=[...], tools=[...])
|
|
```
|
|
|
|
## API reference
|
|
|
|
`LlmSpanContext(...)` accepts the following kwargs. Each is read once when the next OpenAI call's LLM span is created.
|
|
|
|
| Kwarg | Type | Description |
|
|
| ------------------- | ----------- | -------------------------------------------------------------------------------------------------------- |
|
|
| `metrics` | `list` | Metrics applied to the next LLM span. |
|
|
| `prompt` | `Prompt` | Confident AI prompt object; captured on the LLM span for prompt-version analytics. |
|
|
| `expected_output` | `str` | Reference output for metrics that compare against ground truth. |
|
|
| `expected_tools` | `list` | Reference tool calls for tool-aware metrics. |
|
|
| `context` | `list[str]` | Ideal context the model should use when answering. |
|
|
| `retrieval_context` | `list[str]` | Retrieved context the model actually used (Faithfulness, Contextual Relevancy, etc.). |
|
|
|
|
`with trace(...)` accepts trace-level kwargs (`name`, `tags`, `metadata`, `thread_id`, `user_id`, `metrics`, `input`, `output`) — see the [tracing reference](/docs/evaluation-llm-tracing).
|
|
|
|
## FAQs
|
|
|
|
<FAQs
|
|
qas={[
|
|
{
|
|
question: "Can I gate my CI/CD pipeline on these OpenAI evals?",
|
|
answer: (
|
|
<>
|
|
Yes. Keep the same <code>deepeval.openai.OpenAI</code> client, then
|
|
wrap each parametrized <code>pytest</code> case with{" "}
|
|
<code>assert_test(golden=golden)</code> and run{" "}
|
|
<code>deepeval test run</code>. A failing metric fails the test, which
|
|
fails the build.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question:
|
|
"One request makes several OpenAI calls — can I evaluate each call separately?",
|
|
answer: (
|
|
<>
|
|
Yes. Bracket the calls in a single <code>with trace(...)</code> so the
|
|
LLM spans share one root, and attach metrics to a specific call by
|
|
opening that call inside its own{" "}
|
|
<code>LlmSpanContext(metrics=[...])</code>. Each{" "}
|
|
<code>chat.completions.create(...)</code> is its own independently
|
|
scorable LLM span.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question: "Can I see these traces and scores in a UI instead of the terminal?",
|
|
answer: (
|
|
<>
|
|
Yes, and it's optional. Run <code>deepeval login</code> to connect{" "}
|
|
<a href="https://www.confident-ai.com">Confident AI</a> and the same
|
|
instrumented client renders every trace, LLM span, and metric score in
|
|
a shared cloud dashboard — no code changes.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question: "Can I keep evaluating my OpenAI app once it's in production?",
|
|
answer: (
|
|
<>
|
|
Yes. When logged into Confident AI, the drop-in client streams live
|
|
traces in real time, so you can run{" "}
|
|
<a href="https://www.confident-ai.com/docs/llm-tracing/online-evals">
|
|
online evals
|
|
</a>{" "}
|
|
on real production traffic instead of only offline datasets.
|
|
</>
|
|
),
|
|
},
|
|
]}
|
|
/>
|