confident-ai--deepeval/deepeval/integrations

deepeval.integrations

Contributor reference for the framework integrations. Each integration plugs deepeval's tracing / evaluation into a third-party framework using one of four mechanisms.

Note: deepeval.openai, deepeval.anthropic, and deepeval.openai_agents live at the top level of the deepeval package, not under this folder. They're listed here so the matrix is complete.

Integration matrix

Capability columns:

• Bare — calling the framework directly without an enclosing @observe / with trace(...) produces a trace in Confident AI. Each integration auto-creates a trace on first activity (callback fire, OTel root span, internal @observe wrap on the native client, etc.).
• @observe / with trace(...) — when wrapped, the integration's spans flow into deepeval's native trace context: update_current_trace(...) / update_current_span(...) work anywhere in the call stack, single REST POST per trace, no UUID-reconciliation needed.
• evals_iterator — works inside dataset.evals_iterator(...), both end-to-end (metrics=[...] on the iterator) and component-level (@observe(metrics=[...]) on a span). For OTel-mode integrations, ContextAwareSpanProcessor flips to REST routing automatically when trace_manager.is_evaluating is True so spans flow through trace_manager instead of OTLP.
• deepeval test run — works under the pytest tracing-eval entry point (@assert_test, @generate_trace_json, @assert_trace_json).

Integration	Mode	Entry point	Bare	@observe / with trace()	evals_iterator	deepeval test run	Source
OpenAI	Native client wrapper	from deepeval.openai import OpenAI	Yes	Yes	Yes	Yes	deepeval/openai/
Anthropic	Native client wrapper	from deepeval.anthropic import Anthropic	Yes	Yes	Yes	Yes	deepeval/anthropic/
LangChain	Callback handler	CallbackHandler()	Yes	Yes	Yes	Yes	deepeval/integrations/langchain/
LangGraph	Callback handler (LangChain's)	CallbackHandler()	Yes	Yes	Yes	Yes	deepeval/integrations/langchain/
LlamaIndex	Event handler	instrument_llama_index()	Yes	Yes	Yes	Yes	deepeval/integrations/llama_index/
CrewAI	Event listener + wrapper classes	instrument_crewai()	Yes	Yes	Yes	Yes	deepeval/integrations/crewai/
OpenAI Agents	Trace processor + agent wrapper	add_trace_processor(DeepEvalTracingProcessor())	Yes	Yes	Yes	Yes	deepeval/openai_agents/
AgentCore	OpenTelemetry	instrument_agentcore()	Yes	Yes	Yes	Yes	deepeval/integrations/agentcore/
Strands	OpenTelemetry	instrument_strands()	Yes	Yes	Yes	Yes	deepeval/integrations/strands/
Google ADK	OpenTelemetry (via OpenInference)	instrument_google_adk()	Yes	Yes	Yes	Yes	deepeval/integrations/google_adk/
Pydantic AI	OpenTelemetry	DeepEvalInstrumentationSettings(...)	Yes	Yes	Yes	Yes	deepeval/integrations/pydantic_ai/

Every cell is Yes because of the recent OTel POC migrations: native-client / callback-handler / event-listener / trace-processor integrations were already feature-complete via direct trace_manager access, and the four OTel-mode integrations (Pydantic AI, AgentCore, Google ADK, Strands) now follow the same SpanInterceptor + ContextAwareSpanProcessor pattern¹ so their spans behave identically across all four surfaces. New integrations should target the same parity.

Mode reference

• Native client wrapper — drop-in replacement for the vendor SDK's client class (e.g. deepeval.openai.OpenAI instead of openai.OpenAI). Spans are built directly via trace_manager. Lowest friction, but only covers calls that go through that client.
• Callback handler / event listener — registers with the framework's own callback or event API (LangChain BaseCallbackHandler, LlamaIndex BaseEventHandler, CrewAI BaseEventListener, etc.). Spans are built directly via trace_manager. Covers all calls the framework dispatches through that surface — no need to swap clients.
• Trace processor — for frameworks that already have their own tracing pipeline (OpenAI Agents SDK), we plug into it as a processor and translate events into deepeval spans.
• OpenTelemetry — registers an OTel SpanProcessor against the global TracerProvider. The framework (or a community-maintained instrumentor like openinference-instrumentation-google-adk) emits OTel spans; deepeval translates them into Confident span attributes and ships them via OTLP.

Transport reference

• REST — trace_manager posts the full trace to api.confident-ai.com/v1/traces once per trace.
• OTLP — BatchSpanProcessor flushes OTel spans to otel.confident-ai.com/v1/traces on a timer / queue threshold.

OpenInference (generic OTel backend for community instrumentors)

deepeval/integrations/openinference/ is the SpanInterceptor + processor wiring shared by Google ADK and any other community-maintained OpenInference instrumentor. It sets up the TracerProvider, registers OpenInferenceSpanInterceptor (translates OpenInference semantic-convention attributes — openinference.span.kind, llm.input_messages.{idx}, llm.output_messages.{idx}, tool.name, llm.token_count.* — into confident.span.*), and routes spans through ContextAwareSpanProcessor (REST or OTLP).

It is exposed at the top level as deepeval.instrument(...) so users can pair it with any OpenInference instrumentor directly:

import deepeval
from openinference.instrumentation.google_adk import GoogleADKInstrumentor

deepeval.instrument(name="my-app", environment="development")
GoogleADKInstrumentor().instrument()

instrument_google_adk(...) is just a convenience wrapper that calls GoogleADKInstrumentor().instrument() then deepeval.instrument(...) for you.

AgentCore, Strands, and Pydantic AI do NOT delegate here — they have their own SpanInterceptors (AgentCoreSpanInterceptor, StrandsSpanInterceptor, PydanticAISpanInterceptor). AgentCore and Strands both read OTel GenAI semconv (gen_ai.*) attributes — Strands emits these natively, and AgentCore picks them up from the Strands runtime AWS Bedrock typically runs under, plus Traceloop / AWS Bedrock fallbacks; Pydantic AI uses its own logfire-shaped attrs. All four interceptors share the same processor wiring and the same ContextAwareSpanProcessor for routing.

Mixing OTel-mode with @observe

When an OTel-mode integration runs inside an active @observe / with trace(...) context, the OTel span interceptor synchronizes the trace UUID (current_trace_context.uuid = OTel trace_id) so both transports land on the same trace server-side.

For all OTel-mode integrations above, ContextAwareSpanProcessor automatically routes the OTel spans through ConfidentSpanExporter (REST) when a deepeval trace context is active or an evaluation is running — so a mixed trace produces a single REST POST and update_current_trace(...) / update_current_span(...) from anywhere in the call stack land on the right span. Pydantic AI is the reference implementation; AgentCore, Strands, and Google ADK (the latter via the shared openinference/ backend) follow the same pattern.

---

1. Each OTel-mode SpanInterceptor reads trace-level metadata from current_trace_context per span (instead of baking it at instrument_*() time) and pushes a BaseSpan placeholder onto current_span_context for each OTel span's lifetime so update_current_span(...) from anywhere lands in confident.span.* attributes at on_end. The ContextAwareSpanProcessor (deepeval/tracing/otel/context_aware_processor.py) routes spans to REST when a deepeval trace context is active or an evaluation is running, OTLP otherwise. ↩︎