confident-ai--deepeval/docs/content/integrations/frameworks/langgraph.mdx

---
id: langgraph
title: LangGraph
sidebar_label: LangGraph
---

<IntegrationTagsDisplayer native={true} cicdEvals={true} traceability={true} />

[LangGraph](https://www.langchain.com/langgraph) is a low-level orchestration framework for building stateful, graph-based agent workflows. You compose agents from `StateGraph` nodes and edges, with full control over routing, state, and tool execution.

The `deepeval` integration traces LangGraph runs through LangChain's `CallbackHandler`, which you pass into your graph's runtime config. Every graph run, node, model call, tool call, and nested step becomes a span you can inspect, without rewriting your LangGraph app.

<AgentTraceTerminal
  title="langgraph_agent · deepeval"
  ariaLabel="Example LangGraph agent trace with per-step metric scores"
  lines={[
    { kind: "cmd", name: "deepeval test run test_langgraph_agent.py" },
    { kind: "blank" },
    { kind: "root", prefix: "●", name: "test_langgraph_agent" },
    { kind: "blank", prefix: "│" },
    {
      kind: "agent",
      prefix: "└─",
      name: "weather_graph",
      metric: "Task Completion",
      score: "0.94",
      duration: "190ms",
      pass: true,
    },
    {
      kind: "llm",
      prefix: "   ├─",
      name: "chatbot · gpt-4o-mini",
      metric: "G-Eval",
      score: "0.42",
      duration: "72ms",
      pass: false,
    },
    {
      kind: "tool",
      prefix: "   ├─",
      name: 'get_weather(city="Paris")',
      duration: "32ms",
    },
    {
      kind: "llm",
      prefix: "   └─",
      name: "chatbot · gpt-4o-mini",
      metric: "Faithfulness",
      score: "0.95",
      duration: "78ms",
      pass: true,
    },
    { kind: "blank" },
    {
      kind: "summary",
      name: "Trace score  0.77   ·   2/3 metrics passed",
      pass: false,
    },
  ]}
/>

`deepeval`'s LangGraph integration enables you to:

- **Trace any LangGraph run** — pass `CallbackHandler(...)` through `config={"callbacks": [...]}` per call.
- **Evaluate traces or model / agent components** with `deepeval` metrics.
- **Run evals from scripts or CI/CD** — same callback, different surfaces.
- **Customize trace and span data** through callback kwargs and LangChain metadata.

## Getting Started

<Steps>

<Step>

### Installation

```bash
pip install -U deepeval langgraph langchain-openai
```

LangGraph uses LangChain's callback system, so the `deepeval` integration is per-call. You decide which graph runs are traced by passing `CallbackHandler(...)` into the graph config.

</Step>

<Step>

### Instrument and evaluate

Wire your `StateGraph` (LangGraph's core abstraction), then pass `CallbackHandler(...)` to the invocation you want to evaluate.

```python title="langgraph_agent.py" showLineNumbers
from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What is the weather in Paris?")])

# The `TaskCompletionMetric` is passed into the LangGraph callback.
for golden in dataset.evals_iterator():
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )
```

Done ✅. You've run your first eval with full traceability into LangGraph via `deepeval`.

</Step>

</Steps>

## What gets traced

Each LangGraph run that receives a `CallbackHandler` produces a **trace** — the end-to-end unit your user observes. Inside that trace are **component spans** for each callback LangGraph emits through LangChain:

- **Graph / node spans** — the compiled `StateGraph` invocation and each node it dispatches to.
- **LLM spans** — chat model and completion calls inside a node.
- **Tool spans** — tool calls executed by `ToolNode` (or your own).
- **Retriever spans** — retriever calls, when your graph uses retrieval.

```text
Trace                           ← what the user observes
└── Graph: weather_graph         ← one graph invoke(...) call
    ├── Node: chatbot           ← model picks a tool
    │   └── LLM: gpt-4o-mini
    ├── Node: tools             ← ToolNode runs the tool
    │   └── Tool: get_weather
    └── Node: chatbot           ← model writes the final answer
        └── LLM: gpt-4o-mini
```

The trace and its component spans are independently evaluable.

## Running evals

There are two surfaces for running evals against a LangGraph app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

### In CI/CD (pytest)

Use the `deepeval` pytest integration. Each parametrized test invocation becomes one LangGraph run; failing metrics fail the test, which fails the build.

```python title="test_langgraph_agent.py" showLineNumbers
import pytest
from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval import assert_test
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

llm = init_chat_model("openai:gpt-4o-mini").bind_tools([get_weather])

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
)

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_langgraph_agent(golden: Golden):
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler()]},
    )
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])
```

Run it with:

```bash
deepeval test run test_langgraph_agent.py
```

### In a script

Use `EvaluationDataset` + `evals_iterator(...)`. Each `Golden` becomes one LangGraph run; metrics score the resulting trace through the callback.

```python title="langgraph_agent.py" showLineNumbers
dataset = EvaluationDataset(goldens=[
    Golden(input="What is the weather in Paris?"),
    Golden(input="What is the weather in London?"),
])

for golden in dataset.evals_iterator():
    graph.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )
```

## Applying metrics to components

Passing `metrics=[...]` to `CallbackHandler` evaluates the overall LangGraph run. To evaluate a component instead, attach metrics where the graph creates that component.

### Agent spans (sub-agents)

Wrap the `graph.invoke(...)` in `with next_agent_span(metrics=[...]):`. The `CallbackHandler` drains the staged metric onto the **first agent span** the graph emits — useful for scoring a sub-agent node or subgraph in isolation.

```python title="langgraph_agent.py" showLineNumbers
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import next_agent_span
...

for golden in dataset.evals_iterator():
    with next_agent_span(metrics=[TaskCompletionMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )
```

The same one-shot semantic as `next_llm_span` applies: only the first agent span in the graph run picks up the staged metric.

### LLM calls

Wrap the `graph.invoke(...)` in `with next_llm_span(metrics=[...]):`. The `CallbackHandler` drains the staged metric onto the **first LLM span** the graph emits; later LLM calls on subsequent loop turns get nothing. This is the same one-shot semantic used by `next_*_span` in the Pydantic AI / Strands / AgentCore / Google ADK integrations.

```python title="langgraph_agent.py" showLineNumbers
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import next_llm_span
...

for golden in dataset.evals_iterator():
    with next_llm_span(metrics=[AnswerRelevancyMetric()]):
        graph.invoke(
            {"messages": [{"role": "user", "content": golden.input}]},
            config={"callbacks": [CallbackHandler()]},
        )
```

:::caution[One-shot per run]
`next_llm_span` stages a metric for the **first** LLM span the graph emits inside the `with` block. Later loop iterations through the `chatbot` node won't pick it up. To score every LLM call, drive the loop yourself (`next_llm_span` per `graph.invoke(...)`) or score the run end-to-end with trace-level metrics on `CallbackHandler(metrics=[...])`.
:::

For deterministic tool calls, use tool spans for traceability, inputs, outputs, and metadata. Avoid attaching metrics directly to tool spans.

## Customizing trace and span data

LangGraph is instrumented per-call through LangChain callbacks, so customization happens at the callback or span-staging boundary.

- Use `CallbackHandler(...)` kwargs for trace-level defaults like `name`, `tags`, `metadata`, `thread_id`, and `user_id`.
- Use `next_agent_span(...)` / `next_llm_span(...)` / `next_retriever_span(...)` / `next_tool_span(...)` to stage component-level fields (metrics, metric collections, test cases, custom span metadata) onto the next span the callback opens.
- Use tool spans for deterministic traceability, inputs, outputs, and metadata.

```python title="langgraph_agent.py" showLineNumbers
callback = CallbackHandler(
    name="weather-graph",
    tags=["langgraph", "weather"],
    metadata={"team": "support"},
    user_id="user-123",
)

graph.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Paris?"}]},
    config={"callbacks": [callback]},
)
```

## Advanced patterns

The primitives above — `CallbackHandler(...)` and `next_*_span(...)` — compose around one boundary: LangGraph owns the graph execution lifecycle, and your code chooses where to stage component config for the next span the callback opens.

### Evaluate subagents with `next_*_span`

`next_*_span(metrics=[...])` stages a metric for the next matching span the `CallbackHandler` opens during the graph run. Use this when you want to evaluate a subagent node or model step instead of the full graph. Pick the helper that matches the span you want to score: `next_agent_span(...)` or `next_llm_span(...)`.

```python title="langgraph_agent.py" showLineNumbers
from langchain.chat_models import init_chat_model
from langgraph.graph import StateGraph, MessagesState, START, END
from deepeval.integrations.langchain import CallbackHandler
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import next_agent_span
...

llm = init_chat_model("openai:gpt-4o-mini")

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_edge(START, "chatbot")
    .add_edge("chatbot", END)
    .compile()
)

def run_graph(prompt: str):
    with next_agent_span(metrics=[TaskCompletionMetric()]):
        return graph.invoke(
            {"messages": [{"role": "user", "content": prompt}]},
            config={"callbacks": [CallbackHandler()]},
        )
```

#### No trace-level metrics required

Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because the `TaskCompletionMetric` is staged for the agent span, so CI/CD and scripts only need to run the graph inside the staging block.

This is how you'd run it:

<Tabs items={["CI/CD", "Scripts"]}>
<Tab value="CI/CD">

```python title="test_langgraph_agent.py" showLineNumbers
import pytest
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_component_metrics(golden: Golden):
    run_graph(golden.input)
    assert_test(golden=golden)
```

```bash
deepeval test run test_langgraph_agent.py
```

</Tab>
<Tab value="Scripts">

```python title="langgraph_agent.py" showLineNumbers
...

for golden in dataset.evals_iterator():
    run_graph(golden.input)
```

</Tab>
</Tabs>

### Wrap a LangGraph run in `@observe`

When the LangGraph call is part of a larger operation, decorate the outer function with `@observe`. LangGraph spans nest under your observed span when the callback runs inside it.

```python title="langgraph_agent.py" showLineNumbers
from deepeval.tracing import observe
...

@observe(name="respond_to_user")
def respond_to_user(prompt: str):
    return graph.invoke(
        {"messages": [{"role": "user", "content": prompt}]},
        config={"callbacks": [CallbackHandler()]},
    )
```

## Tracing a LangGraph server deployment

When you deploy with the [LangGraph server](https://langchain-ai.github.io/langgraph/cloud/) (`langgraph dev` or LangGraph Platform), the **server** invokes your graph. So instead you can attach the handler to the compiled graph with `.with_config(...)` and export that. The callback then applies to every run the server executes.

```python title="langgraph_agent.py" showLineNumbers
from langgraph.graph import StateGraph, MessagesState, START, END
from langgraph.prebuilt import ToolNode, tools_condition
from deepeval.integrations.langchain import CallbackHandler
...

graph = (
    StateGraph(MessagesState)
    .add_node(chatbot)
    .add_node("tools", ToolNode([get_weather]))
    .add_edge(START, "chatbot")
    .add_conditional_edges("chatbot", tools_condition)
    .add_edge("tools", "chatbot")
    .compile()
    .with_config(callbacks=[CallbackHandler()])  # baked into every server run
)
```

Point `langgraph.json` at the exported `graph`:

```json title="langgraph.json"
{
  "dependencies": ["."],
  "graphs": { "agent": "./langgraph_agent.py:graph" },
  "env": ".env"
}
```

Then start the server — every request it runs against the graph is traced:

```bash
pip install -U "langgraph-cli[inmem]"
langgraph dev
```

:::note[Modern agent builder]
If you build agents with `create_agent` (`from langchain.agents import create_agent`, which replaces the deprecated `langgraph.prebuilt.create_react_agent`), the same applies — call `.with_config(callbacks=[CallbackHandler()])` on the returned graph before exporting it.
:::

:::caution[Baked-in trace attributes are per-handler, not per-request]
Attributes set on a baked-in `CallbackHandler(...)` — `thread_id`, `user_id`, etc. — are the same for every request the server serves, because the handler is constructed once. For per-request values, attach the callback per call instead of baking it into the graph.
:::

## API reference

`CallbackHandler(...)` accepts the following trace-level kwargs. Each one is a default for runs that use that callback.

| Kwarg               | Type        | Description                                                       |
| ------------------- | ----------- | ----------------------------------------------------------------- |
| `name`              | `str`       | Default trace name.                                               |
| `tags`              | `list[str]` | Tags applied to traces produced by this callback.                 |
| `metadata`          | `dict`      | Trace metadata applied when the callback starts a trace.          |
| `thread_id`         | `str`       | Groups related runs into a single trace thread.                   |
| `user_id`           | `str`       | Actor identifier for the trace.                                   |
| `metrics`           | `list`      | Metrics applied to the LangGraph run.                             |
| `metric_collection` | `str`       | Metric collection applied to the LangGraph run.                   |
| `test_case_id`      | `str`       | Optional test case identifier.                                    |
| `turn_id`           | `str`       | Optional turn identifier for conversational traces.               |

For native tracing helpers (`@observe`, `with trace(...)`, `update_current_trace`, `update_current_span`) see the [tracing reference](/docs/evaluation-llm-tracing).

## FAQs

<FAQs
  qas={[
    {
      question: "Can I evaluate a sub-agent node inside my graph?",
      answer: (
        <>
          Yes. Wrap <code>graph.invoke(...)</code> in{" "}
          <code>with next_agent_span(metrics=[...])</code> and the{" "}
          <code>CallbackHandler</code> drains the metric onto the agent span
          that sub-agent node emits — scoring it on its own. It's one-shot per
          run, so for every loop turn either drive the loop yourself or score
          end-to-end with trace-level metrics.
        </>
      ),
    },
    {
      question: "Can I fail CI when a LangGraph metric regresses?",
      answer: (
        <>
          Yes. Pass <code>CallbackHandler()</code> into the graph config inside a
          parametrized <code>pytest</code> test and assert with{" "}
          <code>assert_test(...)</code> under <code>deepeval test run</code>.
        </>
      ),
    },
    {
      question: "Where do my LangGraph traces show up beyond the console?",
      answer: (
        <>
          Run <code>deepeval login</code> and{" "}
          <a href="https://www.confident-ai.com">Confident AI</a> visualizes the
          full graph trace — every node, model call, and tool call as nested
          spans — with their scores in a shared cloud UI. It's optional.
        </>
      ),
    },
    {
      question: "How do I trace a LangGraph server deployment (langgraph.json)?",
      answer: (
        <>
          The server invokes your graph, so there's no <code>invoke</code> call
          of your own to pass the callback to. Attach it to the compiled graph
          with <code>.with_config(callbacks=[CallbackHandler()])</code> and point{" "}
          <code>langgraph.json</code> at that exported graph — every server run
          is then traced.
        </>
      ),
    },
    {
      question: "Can I keep evaluating a deployed LangGraph app in production?",
      answer: (
        <>
          Yes. Keep passing the <code>CallbackHandler</code> in production and
          group runs with <code>thread_id</code>; logged into Confident AI those
          live traces power{" "}
          <a href="https://www.confident-ai.com/docs/llm-tracing/online-evals">
            online evals
          </a>{" "}
          on real traffic.
        </>
      ),
    },
  ]}
/>