wehub-resource-sync
435 lines
15 KiB
Plaintext
435 lines
15 KiB
Plaintext
---
|
|
id: llamaindex
|
|
title: LlamaIndex
|
|
sidebar_label: LlamaIndex
|
|
---
|
|
|
|
<IntegrationTagsDisplayer native={true} cicdEvals={true} traceability={true} />
|
|
|
|
[LlamaIndex](https://www.llamaindex.ai/) is an orchestration framework for data ingestion, indexing, and retrieval-augmented generation, with first-class agent and workflow primitives.
|
|
|
|
The `deepeval` integration registers a LlamaIndex event handler that turns every dispatch — workflow runs, agent steps, LLM chats, retrieval, and tool calls — into a span you can inspect, without rewriting your LlamaIndex app.
|
|
|
|
<AgentTraceTerminal
|
|
title="llamaindex_agent · deepeval"
|
|
ariaLabel="Example LlamaIndex agent trace with per-step metric scores"
|
|
lines={[
|
|
{ kind: "cmd", name: "deepeval test run test_llamaindex_agent.py" },
|
|
{ kind: "blank" },
|
|
{ kind: "root", prefix: "●", name: "test_llamaindex_agent" },
|
|
{ kind: "blank", prefix: "│" },
|
|
{
|
|
kind: "agent",
|
|
prefix: "└─",
|
|
name: "math_agent",
|
|
metric: "Task Completion",
|
|
score: "0.95",
|
|
duration: "210ms",
|
|
pass: true,
|
|
},
|
|
{
|
|
kind: "llm",
|
|
prefix: " ├─",
|
|
name: "gpt-4o-mini · plan",
|
|
metric: "G-Eval",
|
|
score: "0.43",
|
|
duration: "70ms",
|
|
pass: false,
|
|
},
|
|
{
|
|
kind: "tool",
|
|
prefix: " ├─",
|
|
name: "multiply(a=8, b=6)",
|
|
duration: "32ms",
|
|
},
|
|
{
|
|
kind: "llm",
|
|
prefix: " └─",
|
|
name: "gpt-4o-mini · respond",
|
|
metric: "Faithfulness",
|
|
score: "0.94",
|
|
duration: "76ms",
|
|
pass: true,
|
|
},
|
|
{ kind: "blank" },
|
|
{
|
|
kind: "summary",
|
|
name: "Trace score 0.77 · 2/3 metrics passed",
|
|
pass: false,
|
|
},
|
|
]}
|
|
/>
|
|
|
|
`deepeval`'s LlamaIndex integration enables you to:
|
|
|
|
- **Trace every workflow / agent run** — each `agent.run(...)` produces a trace, and each LLM, tool, and retriever call becomes a component span.
|
|
- **Evaluate traces or model / agent components** with any `deepeval` metric through `LlmSpanContext` and `AgentSpanContext`.
|
|
- **Run evals from scripts or CI/CD** — same dispatcher, different surfaces.
|
|
- **Compose with `@observe` and `with trace(...)`** to evaluate larger flows that wrap one or more LlamaIndex runs.
|
|
|
|
## Getting Started
|
|
|
|
<Steps>
|
|
|
|
<Step>
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
pip install -U deepeval llama-index llama-index-llms-openai
|
|
```
|
|
|
|
The integration registers a `BaseEventHandler` and `BaseSpanHandler` against LlamaIndex's instrumentation dispatcher. After that, every workflow / agent run dispatches events that `deepeval` turns into spans.
|
|
|
|
</Step>
|
|
|
|
<Step>
|
|
|
|
### Instrument and evaluate
|
|
|
|
Call `instrument_llama_index(get_dispatcher())` once at startup. Wrap each agent run in `with trace(agent_span_context=AgentSpanContext(metrics=[...]))` to evaluate the agent span.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
import asyncio
|
|
|
|
from llama_index.llms.openai import OpenAI
|
|
from llama_index.core.agent import FunctionAgent
|
|
import llama_index.core.instrumentation as instrument
|
|
|
|
from deepeval.integrations.llama_index import instrument_llama_index
|
|
from deepeval.tracing import trace, AgentSpanContext
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.evaluate.configs import AsyncConfig
|
|
from deepeval.metrics import TaskCompletionMetric
|
|
|
|
instrument_llama_index(instrument.get_dispatcher())
|
|
|
|
def multiply(a: float, b: float) -> float:
|
|
"""Multiply two numbers."""
|
|
return a * b
|
|
|
|
agent = FunctionAgent(
|
|
tools=[multiply],
|
|
llm=OpenAI(model="gpt-4o-mini"),
|
|
system_prompt="You are a helpful calculator.",
|
|
)
|
|
|
|
async def run_agent(prompt: str):
|
|
with trace(agent_span_context=AgentSpanContext(metrics=[TaskCompletionMetric()])):
|
|
return await agent.run(prompt)
|
|
|
|
# Goldens are the inputs you want to evaluate.
|
|
dataset = EvaluationDataset(goldens=[Golden(input="What is 8 multiplied by 6?")])
|
|
|
|
for golden in dataset.evals_iterator(async_config=AsyncConfig(run_async=True)):
|
|
task = asyncio.create_task(run_agent(golden.input))
|
|
dataset.evaluate(task)
|
|
```
|
|
|
|
Done ✅. You've run your first eval with full traceability into LlamaIndex via `deepeval`.
|
|
|
|
</Step>
|
|
|
|
</Steps>
|
|
|
|
## What gets traced
|
|
|
|
Each LlamaIndex `Workflow` or `agent.run(...)` call produces a **trace** — the end-to-end unit your user observes. Inside that trace are **component spans** for every dispatch LlamaIndex emits:
|
|
|
|
- **Agent spans** — `FunctionAgent.run`, `Workflow.run`, and nested agent steps.
|
|
- **LLM spans** — chat model calls (`LLMChatStartEvent` / `LLMChatEndEvent`).
|
|
- **Tool spans** — `call_tool` / `acall_tool` invocations.
|
|
- **Retriever spans** — retriever calls (`RetrievalEndEvent`) when your app uses retrieval.
|
|
|
|
```text
|
|
Trace ← what the user observes
|
|
└── Agent: math_agent ← one agent.run(...) call
|
|
├── LLM: gpt-4o-mini ← component span: model decides
|
|
├── Tool: multiply ← component span: tool input + output
|
|
└── LLM: gpt-4o-mini ← component span: final answer
|
|
```
|
|
|
|
The trace and its component spans are independently evaluable.
|
|
|
|
## Running evals
|
|
|
|
There are two surfaces for running evals against a LlamaIndex app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.
|
|
|
|
### In CI/CD (pytest)
|
|
|
|
Use the `deepeval` pytest integration. Each parametrized test invocation becomes one `agent.run(...)`; failing metrics fail the test, which fails the build.
|
|
|
|
```python title="test_llamaindex_agent.py" showLineNumbers
|
|
import asyncio
|
|
import pytest
|
|
|
|
from llama_index.llms.openai import OpenAI
|
|
from llama_index.core.agent import FunctionAgent
|
|
import llama_index.core.instrumentation as instrument
|
|
|
|
from deepeval import assert_test
|
|
from deepeval.integrations.llama_index import instrument_llama_index
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.metrics import TaskCompletionMetric
|
|
|
|
instrument_llama_index(instrument.get_dispatcher())
|
|
|
|
def multiply(a: float, b: float) -> float:
|
|
"""Multiply two numbers."""
|
|
return a * b
|
|
|
|
agent = FunctionAgent(
|
|
tools=[multiply],
|
|
llm=OpenAI(model="gpt-4o-mini"),
|
|
system_prompt="You are a helpful calculator.",
|
|
)
|
|
dataset = EvaluationDataset(goldens=[
|
|
Golden(input="What is 8 multiplied by 6?"),
|
|
Golden(input="What is 7 multiplied by 9?"),
|
|
])
|
|
|
|
@pytest.mark.parametrize("golden", dataset.goldens)
|
|
def test_llamaindex_agent(golden: Golden):
|
|
asyncio.run(agent.run(golden.input))
|
|
assert_test(golden=golden, metrics=[TaskCompletionMetric()])
|
|
```
|
|
|
|
Run it with:
|
|
|
|
```bash
|
|
deepeval test run test_llamaindex_agent.py
|
|
```
|
|
|
|
### In a script
|
|
|
|
Use `EvaluationDataset` + `evals_iterator(...)`. Each `Golden` becomes one agent run; metrics score the resulting trace.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
import asyncio
|
|
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.evaluate.configs import AsyncConfig
|
|
from deepeval.metrics import TaskCompletionMetric
|
|
...
|
|
|
|
dataset = EvaluationDataset(goldens=[
|
|
Golden(input="What is 8 multiplied by 6?"),
|
|
Golden(input="What is 7 multiplied by 9?"),
|
|
])
|
|
|
|
for golden in dataset.evals_iterator(
|
|
async_config=AsyncConfig(run_async=True),
|
|
metrics=[TaskCompletionMetric()],
|
|
):
|
|
task = asyncio.create_task(agent.run(golden.input))
|
|
dataset.evaluate(task)
|
|
```
|
|
|
|
LlamaIndex's `agent.run(...)` is async-only, so `evals_iterator` here uses `AsyncConfig(run_async=True)` and `dataset.evaluate(task)` to run goldens concurrently.
|
|
|
|
## Applying metrics to components
|
|
|
|
The `metrics=[...]` you pass to `evals_iterator` evaluates the **trace**. To evaluate a **component** — a specific agent span or LLM call — stage the metric with `AgentSpanContext` or `LlmSpanContext` before the run.
|
|
|
|
### Agent spans
|
|
|
|
Use `AgentSpanContext(metrics=[...])` to score the agent span specifically. Useful when you want a metric on the agent step itself, distinct from the trace.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import trace, AgentSpanContext
|
|
from deepeval.metrics import TaskCompletionMetric
|
|
...
|
|
|
|
async def run_agent(prompt: str):
|
|
with trace(agent_span_context=AgentSpanContext(metrics=[TaskCompletionMetric()])):
|
|
return await agent.run(prompt)
|
|
```
|
|
|
|
### LLM calls
|
|
|
|
Use `LlmSpanContext(metrics=[...])` to score the next LLM span LlamaIndex opens. Useful when you want to evaluate the model's reasoning step in isolation.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
...
|
|
|
|
async def run_agent(prompt: str):
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[AnswerRelevancyMetric()])):
|
|
return await agent.run(prompt)
|
|
```
|
|
|
|
For deterministic tool calls, prefer `update_current_span(...)` to add metadata, inputs, and outputs instead of attaching metrics to the tool span.
|
|
|
|
## Customizing trace and span data
|
|
|
|
The integration captures inputs, outputs, model names, and tool calls automatically. For anything dynamic, the right API depends on where your code runs.
|
|
|
|
- Use `with trace(...)` for trace-level fields (`name`, `tags`, `metadata`, `thread_id`, `user_id`, `metrics`).
|
|
- Use `LlmSpanContext` and `AgentSpanContext` for component-level metric defaults and evaluation parameters.
|
|
- Use `update_current_trace(...)` and `update_current_span(...)` from inside a tool body to mutate fields the framework can't see.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import update_current_span
|
|
|
|
def multiply(a: float, b: float) -> float:
|
|
"""Multiply two numbers."""
|
|
update_current_span(metadata={"deterministic": True})
|
|
return a * b
|
|
```
|
|
|
|
## Advanced patterns
|
|
|
|
The primitives above — `instrument_llama_index`, `LlmSpanContext`, `AgentSpanContext`, `@observe`, `with trace(...)` — compose around one boundary: LlamaIndex owns the dispatcher lifecycle, and your code stages metrics for the spans it produces.
|
|
|
|
### Stage component metrics with span contexts
|
|
|
|
`AgentSpanContext` and `LlmSpanContext` stage metrics for the next matching component span. Use them when you want to evaluate a sub-step instead of the full trace.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import trace, AgentSpanContext
|
|
from deepeval.metrics import TaskCompletionMetric
|
|
...
|
|
|
|
async def run_agent(prompt: str):
|
|
with trace(agent_span_context=AgentSpanContext(metrics=[TaskCompletionMetric()])):
|
|
return await agent.run(prompt)
|
|
```
|
|
|
|
#### No trace-level metrics required
|
|
|
|
Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because `TaskCompletionMetric` is attached to the agent span via `AgentSpanContext`, so CI/CD and scripts only need to run the agent.
|
|
|
|
This is how you'd run it:
|
|
|
|
<Tabs items={["CI/CD", "Scripts"]}>
|
|
<Tab value="CI/CD">
|
|
|
|
```python title="test_llamaindex_agent.py" showLineNumbers
|
|
import asyncio
|
|
import pytest
|
|
from deepeval import assert_test
|
|
...
|
|
|
|
@pytest.mark.parametrize("golden", dataset.goldens)
|
|
def test_agent_span(golden: Golden):
|
|
asyncio.run(run_agent(golden.input))
|
|
assert_test(golden=golden)
|
|
```
|
|
|
|
```bash
|
|
deepeval test run test_llamaindex_agent.py
|
|
```
|
|
|
|
</Tab>
|
|
<Tab value="Scripts">
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
import asyncio
|
|
...
|
|
|
|
for golden in dataset.evals_iterator(async_config=AsyncConfig(run_async=True)):
|
|
task = asyncio.create_task(run_agent(golden.input))
|
|
dataset.evaluate(task)
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
### Wrap an agent run in `@observe`
|
|
|
|
When the agent run is part of a larger operation, decorate the outer function with `@observe`. The LlamaIndex spans nest under your observed span automatically.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import observe
|
|
...
|
|
|
|
@observe(name="respond_to_user")
|
|
async def respond_to_user(prompt: str) -> str:
|
|
result = await agent.run(prompt)
|
|
return str(result)
|
|
```
|
|
|
|
### Evaluate retrieval
|
|
|
|
When your LlamaIndex app uses a retriever, retrieval results are captured automatically on the retriever span. Stage `LlmSpanContext` with `retrieval_context` for any LLM that needs faithfulness-style metrics, or apply a metric directly to the retriever span via the dispatcher event.
|
|
|
|
```python title="llamaindex_agent.py" showLineNumbers
|
|
from deepeval.tracing import trace, LlmSpanContext
|
|
from deepeval.metrics import FaithfulnessMetric
|
|
...
|
|
|
|
async def run_rag(prompt: str):
|
|
with trace(llm_span_context=LlmSpanContext(metrics=[FaithfulnessMetric()])):
|
|
return await query_engine.aquery(prompt)
|
|
```
|
|
|
|
## API reference
|
|
|
|
`AgentSpanContext(...)` and `LlmSpanContext(...)` accept the following kwargs. Each is read once when the next matching span is created.
|
|
|
|
| Kwarg | Type | Description |
|
|
| ------------------- | ----------- | ---------------------------------------------------------------------------------------- |
|
|
| `metrics` | `list` | Metrics applied to the next matching span (agent or LLM). |
|
|
| `expected_output` | `str` | Reference output for metrics that compare against ground truth. |
|
|
| `expected_tools` | `list` | Reference tool calls for tool-aware metrics. |
|
|
| `context` | `list[str]` | Ideal context the model should use when answering. |
|
|
| `retrieval_context` | `list[str]` | Retrieved context the model actually used (LLM-only; Faithfulness, Contextual Relevancy).|
|
|
| `prompt` | `Prompt` | Confident AI prompt object; LLM-only. |
|
|
|
|
`with trace(...)` accepts trace-level kwargs (`name`, `tags`, `metadata`, `thread_id`, `user_id`, `metrics`) — see the [tracing reference](/docs/evaluation-llm-tracing).
|
|
|
|
## FAQs
|
|
|
|
<FAQs
|
|
qas={[
|
|
{
|
|
question: "Can I score the agent step separately from the LLM call?",
|
|
answer: (
|
|
<>
|
|
Yes. Stage a metric on the agent span with{" "}
|
|
<code>AgentSpanContext(metrics=[...])</code>, or target a model step
|
|
with <code>LlmSpanContext(metrics=[...])</code>, inside{" "}
|
|
<code>with trace(...)</code> — so a sub-agent and its underlying LLM
|
|
call are evaluable independently.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question: "Can I run these LlamaIndex evals under Pytest?",
|
|
answer: (
|
|
<>
|
|
Yes. Call <code>instrument_llama_index(get_dispatcher())</code> once,
|
|
run the agent inside a parametrized <code>pytest</code> test, and
|
|
assert with <code>assert_test(...)</code> under{" "}
|
|
<code>deepeval test run</code>.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question: "Is there a cloud view for these traces?",
|
|
answer: (
|
|
<>
|
|
Yes, optionally. <code>deepeval login</code> connects{" "}
|
|
<a href="https://www.confident-ai.com">Confident AI</a>, which renders
|
|
the same dispatcher-traced spans and scores in a shared UI for your
|
|
team.
|
|
</>
|
|
),
|
|
},
|
|
{
|
|
question: "Can I monitor a LlamaIndex app in production?",
|
|
answer: (
|
|
<>
|
|
Yes. The dispatcher keeps emitting spans in production, and once
|
|
logged into Confident AI those live traces support{" "}
|
|
<a href="https://www.confident-ai.com/docs/llm-tracing/online-evals">
|
|
online evals
|
|
</a>{" "}
|
|
on real traffic, not just offline datasets.
|
|
</>
|
|
),
|
|
},
|
|
]}
|
|
/>
|