---
id: openai
title: OpenAI
sidebar_label: OpenAI
---

<IntegrationTagsDisplayer native={true} cicdEvals={true} traceability={true} />

[OpenAI](https://platform.openai.com/docs/) provides chat completions and responses APIs for building LLM applications.

The `deepeval` integration is a drop-in replacement for OpenAI's client. Every `client.chat.completions.create(...)` and `client.responses.create(...)` call becomes an LLM span you can evaluate, without rewriting how you call the API.

<AgentTraceTerminal
  title="openai_app · deepeval"
  ariaLabel="Example OpenAI client trace with per-step metric scores"
  lines={[
    { kind: "cmd", name: "deepeval test run test_openai_app.py" },
    { kind: "blank" },
    { kind: "root", prefix: "●", name: "test_openai_app" },
    { kind: "blank", prefix: "│" },
    {
      kind: "llm",
      prefix: "└─",
      name: "gpt-4o · respond",
      metric: "Answer Relevancy",
      score: "0.93",
      duration: "260ms",
      pass: true,
    },
    {
      kind: "llm",
      prefix: "  ",
      name: "",
      metric: "Faithfulness",
      score: "0.41",
      duration: "",
      pass: false,
    },
    { kind: "blank" },
    {
      kind: "summary",
      name: "Trace score  0.67   ·   1/2 metrics passed",
      pass: false,
    },
  ]}
/>

`deepeval`'s OpenAI integration enables you to:

- **Drop in `deepeval.openai.OpenAI`** — every chat completion or response produces an LLM span with input, output, and `tools_called` captured automatically.
- **Evaluate LLM calls** with any `deepeval` metric through `LlmSpanContext`.
- **Run evals from scripts or CI/CD** — same client, different surfaces.
- **Compose with `@observe` and `with trace(...)`** to evaluate larger flows that wrap one or more OpenAI calls.

## Getting Started

<Steps>

<Step>

### Installation

```bash
pip install -U deepeval openai
```

`deepeval.openai.OpenAI` and `deepeval.openai.AsyncOpenAI` import OpenAI's classes and patch them in place. Existing kwargs, async paths, streaming, and tool-calling behavior all work unchanged.

</Step>

<Step>

### Instrument and evaluate

Replace `from openai import OpenAI` with `from deepeval.openai import OpenAI`. Wrap each call you want to evaluate in `with trace(llm_span_context=LlmSpanContext(metrics=[...]))`.

```python title="openai_app.py" showLineNumbers
from deepeval.openai import OpenAI
from deepeval.tracing import trace, LlmSpanContext
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric

client = OpenAI()

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What's the capital of France?")])

for golden in dataset.evals_iterator():
    with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Be concise."},
                {"role": "user", "content": golden.input},
            ],
        )
```

Done ✅. You've run your first eval against an OpenAI call with full traceability via `deepeval`.

</Step>

</Steps>

## What gets traced

Each patched OpenAI call produces one **LLM span** under the active trace. When the call uses tool-calling, the span's `tools_called` field captures every tool invocation the model returned — no extra wiring needed.

- **LLM spans** — one per `chat.completions.create(...)`, `chat.completions.parse(...)`, or `responses.create(...)` call. Captures input messages, output text, token counts, and `tools_called`.
- **Trace** — auto-created when the call has no parent. If the call runs inside `with trace(...)` or `@observe`, the LLM span nests under that trace instead.

```text
Trace                          ← auto-created or user-owned
└── LLM: gpt-4o                ← one client.chat.completions.create(...) call
```

The trace and its LLM span are independently evaluable.

## Running evals

There are two surfaces for running evals against OpenAI calls. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

### In CI/CD (pytest)

Use the `deepeval` pytest integration. Each parametrized test invocation becomes one OpenAI call; failing metrics fail the test, which fails the build.

```python title="test_openai_app.py" showLineNumbers
import pytest
from deepeval import assert_test
from deepeval.openai import OpenAI
from deepeval.tracing import trace, LlmSpanContext
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import AnswerRelevancyMetric

client = OpenAI()
dataset = EvaluationDataset(goldens=[
    Golden(input="What's the capital of France?"),
    Golden(input="Who wrote Hamlet?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_openai_app(golden: Golden):
    with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Be concise."},
                {"role": "user", "content": golden.input},
            ],
        )
    assert_test(golden=golden)
```

Run it with:

```bash
deepeval test run test_openai_app.py
```

### In a script

Use `EvaluationDataset` + `evals_iterator(...)`. Each `Golden` becomes one OpenAI call; metrics score the resulting LLM span.

```python title="openai_app.py" showLineNumbers
import asyncio

from deepeval.openai import AsyncOpenAI
from deepeval.tracing import trace, LlmSpanContext
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate.configs import AsyncConfig
from deepeval.metrics import AnswerRelevancyMetric

client = AsyncOpenAI()
dataset = EvaluationDataset(goldens=[
    Golden(input="What's the capital of France?"),
    Golden(input="Who wrote Hamlet?"),
])

async def call_openai(prompt: str):
    with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
        return await client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
        )

for golden in dataset.evals_iterator(async_config=AsyncConfig(run_async=True)):
    task = asyncio.create_task(call_openai(golden.input))
    dataset.evaluate(task)
```

Sync (`OpenAI`) and async (`AsyncOpenAI`) clients both work; pick whichever matches your code.

## Applying metrics to LLM spans

Passing `metrics=[...]` to `LlmSpanContext` evaluates the next OpenAI call's LLM span specifically. The same context manager lets you attach extra evaluation parameters that some metrics need.

```python title="openai_app.py" showLineNumbers
from deepeval.openai import OpenAI
from deepeval.tracing import trace, LlmSpanContext
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

client = OpenAI()

with trace(
    llm_span_context=LlmSpanContext(
        metrics=[AnswerRelevancyMetric(), FaithfulnessMetric()],
        retrieval_context=["Paris is the capital of France."],
    ),
):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "What's the capital of France?"}],
    )
```

`LlmSpanContext` accepts `metrics`, `expected_output`, `expected_tools`, `context`, `retrieval_context`, and `prompt`. Each one is read by the OpenAI patch when the next LLM span is created.

## Customizing trace and span data

The patch captures input messages, output text, and `tools_called` automatically. For anything else, the right API depends on where your code runs.

- Use `with trace(...)` for trace-level fields (`name`, `tags`, `metadata`, `thread_id`, `user_id`).
- Use `LlmSpanContext` for LLM-span-level fields the metric needs (`expected_output`, `retrieval_context`, etc.).
- Use `@observe` to wrap retrieval, post-processing, or any other step you want to see as its own span in the trace.

```python title="openai_app.py" showLineNumbers
from deepeval.openai import OpenAI
from deepeval.tracing import trace, LlmSpanContext, observe

client = OpenAI()

@observe(type="retriever")
def retrieve_docs(query: str) -> list[str]:
    return ["Paris is the capital of France."]

@observe()
def respond_to_user(prompt: str) -> str:
    docs = retrieve_docs(prompt)
    with trace(
        llm_span_context=LlmSpanContext(retrieval_context=docs),
        user_id="user-123",
        tags=["openai", "rag"],
    ):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "\n".join(docs)},
                {"role": "user", "content": prompt},
            ],
        )
    return response.choices[0].message.content
```

## Advanced patterns

The primitives above — `deepeval.openai.OpenAI`, `LlmSpanContext`, `@observe`, `with trace(...)` — compose around one boundary: the patch owns each LLM call's span, and your code chooses what trace to put it inside.

### Wrap an OpenAI call in `@observe`

When the OpenAI call is part of a larger operation, decorate the outer function with `@observe`. The LLM span nests under your observed span automatically.

```python title="openai_app.py" showLineNumbers
from deepeval.tracing import observe, trace, LlmSpanContext
from deepeval.metrics import AnswerRelevancyMetric
...

@observe(name="respond_to_user")
def respond_to_user(prompt: str) -> str:
    with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
        )
    return response.choices[0].message.content
```

#### No trace-level metrics required

Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because `AnswerRelevancyMetric` is attached to the LLM span, so CI/CD and scripts only need to call the function.

This is how you'd run it:

<Tabs items={["CI/CD", "Scripts"]}>
<Tab value="CI/CD">

```python title="test_openai_app.py" showLineNumbers
import pytest
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_respond_to_user(golden: Golden):
    respond_to_user(golden.input)
    assert_test(golden=golden)
```

```bash
deepeval test run test_openai_app.py
```

</Tab>
<Tab value="Scripts">

```python title="openai_app.py" showLineNumbers
...

for golden in dataset.evals_iterator():
    respond_to_user(golden.input)
```

</Tab>
</Tabs>

### Multiple OpenAI calls under one trace

When a single logical unit of work makes several OpenAI calls (e.g. a planner call followed by a respond call), bracket them with `with trace(...)` so the LLM spans share a `trace_id` and show up as siblings under one root.

```python title="openai_app.py" showLineNumbers
from deepeval.tracing import trace
...

def plan_then_respond(prompt: str):
    with trace(name="plan_then_respond"):
        plan = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": f"Plan: {prompt}"}],
        )
        return client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": plan.choices[0].message.content}],
        )
```

### Tool-calling models

When the model returns tool calls, the LLM span's `tools_called` field captures them automatically. Use `expected_tools` on `LlmSpanContext` if you want to evaluate tool selection with a tool-aware metric.

```python title="openai_app.py" showLineNumbers
from deepeval.test_case import ToolCall
from deepeval.tracing import trace, LlmSpanContext
...

with trace(
    llm_span_context=LlmSpanContext(
        expected_tools=[ToolCall(name="get_weather", input_parameters={"city": "Paris"})],
    ),
):
    client.chat.completions.create(model="gpt-4o", messages=[...], tools=[...])
```

## API reference

`LlmSpanContext(...)` accepts the following kwargs. Each is read once when the next OpenAI call's LLM span is created.

| Kwarg               | Type        | Description                                                                                              |
| ------------------- | ----------- | -------------------------------------------------------------------------------------------------------- |
| `metrics`           | `list`      | Metrics applied to the next LLM span.                                                                    |
| `prompt`            | `Prompt`    | Confident AI prompt object; captured on the LLM span for prompt-version analytics.                       |
| `expected_output`   | `str`       | Reference output for metrics that compare against ground truth.                                          |
| `expected_tools`    | `list`      | Reference tool calls for tool-aware metrics.                                                             |
| `context`           | `list[str]` | Ideal context the model should use when answering.                                                       |
| `retrieval_context` | `list[str]` | Retrieved context the model actually used (Faithfulness, Contextual Relevancy, etc.).                    |

`with trace(...)` accepts trace-level kwargs (`name`, `tags`, `metadata`, `thread_id`, `user_id`, `metrics`, `input`, `output`) — see the [tracing reference](/docs/evaluation-llm-tracing).

## FAQs

<FAQs
  qas={[
    {
      question: "Can I gate my CI/CD pipeline on these OpenAI evals?",
      answer: (
        <>
          Yes. Keep the same <code>deepeval.openai.OpenAI</code> client, then
          wrap each parametrized <code>pytest</code> case with{" "}
          <code>assert_test(golden=golden)</code> and run{" "}
          <code>deepeval test run</code>. A failing metric fails the test, which
          fails the build.
        </>
      ),
    },
    {
      question:
        "One request makes several OpenAI calls — can I evaluate each call separately?",
      answer: (
        <>
          Yes. Bracket the calls in a single <code>with trace(...)</code> so the
          LLM spans share one root, and attach metrics to a specific call by
          opening that call inside its own{" "}
          <code>LlmSpanContext(metrics=[...])</code>. Each{" "}
          <code>chat.completions.create(...)</code> is its own independently
          scorable LLM span.
        </>
      ),
    },
    {
      question: "Can I see these traces and scores in a UI instead of the terminal?",
      answer: (
        <>
          Yes, and it's optional. Run <code>deepeval login</code> to connect{" "}
          <a href="https://www.confident-ai.com">Confident AI</a> and the same
          instrumented client renders every trace, LLM span, and metric score in
          a shared cloud dashboard — no code changes.
        </>
      ),
    },
    {
      question: "Can I keep evaluating my OpenAI app once it's in production?",
      answer: (
        <>
          Yes. When logged into Confident AI, the drop-in client streams live
          traces in real time, so you can run{" "}
          <a href="https://www.confident-ai.com/docs/llm-tracing/online-evals">
            online evals
          </a>{" "}
          on real production traffic instead of only offline datasets.
        </>
      ),
    },
  ]}
/>
