chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Promptfoo result files
|
||||
*-results.json
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
@@ -0,0 +1,118 @@
|
||||
# integration-google-adk (Google ADK Integration)
|
||||
|
||||
This example shows how to evaluate the Python [Google Agent Development Kit (ADK)](https://adk.dev/) in promptfoo with native ADK tracing.
|
||||
|
||||
It demonstrates:
|
||||
|
||||
- an in-process Python provider instead of an `adk api_server` wrapper
|
||||
- native ADK OpenTelemetry spans exported into Promptfoo
|
||||
- multi-turn session state, callbacks, plugins, and artifacts
|
||||
- workflow agents via `SequentialAgent`
|
||||
- trajectory assertions over real ADK tool calls
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example integration-google-adk
|
||||
cd integration-google-adk
|
||||
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
export GOOGLE_API_KEY=your_google_api_key_here
|
||||
npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache
|
||||
npx promptfoo@latest eval -c promptfooconfig.workflow.yaml --no-cache
|
||||
npx promptfoo@latest view
|
||||
```
|
||||
|
||||
The default model is `gemini-2.5-flash`. To use another ADK-supported model, set `ADK_MODEL` before running the eval. Provider-style model strings such as `openai/gpt-5.4-mini` require the optional ADK extensions:
|
||||
|
||||
```bash
|
||||
pip install 'google-adk[extensions]>=1.32.0,<2'
|
||||
export ADK_MODEL=openai/gpt-5.4-mini
|
||||
```
|
||||
|
||||
If Promptfoo is launched outside the activated virtual environment, point the Python provider at it explicitly:
|
||||
|
||||
```bash
|
||||
PROMPTFOO_PYTHON=.venv/bin/python npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
- `agent.py`: ADK app builders, tools, callback, plugin, and workflow agent graph
|
||||
- `provider.py`: Promptfoo Python provider plus ADK-to-Promptfoo trace propagation
|
||||
- `provider_test.py`: focused tests for provider helpers
|
||||
- `promptfooconfig.yaml`: conversational multi-turn eval with state, artifacts, and trajectory assertions
|
||||
- `promptfooconfig.workflow.yaml`: workflow-agent eval with `SequentialAgent`
|
||||
- `requirements.txt`: Python dependencies
|
||||
|
||||
## What The Conversational Eval Covers
|
||||
|
||||
The main config turns one Promptfoo row into a small multi-turn task:
|
||||
|
||||
1. Ask for London weather.
|
||||
2. Ask the agent to save a trip note.
|
||||
3. Ask which city was discussed earlier.
|
||||
|
||||
The provider returns the user-visible answer plus an inspection payload:
|
||||
|
||||
- `session_state` from ADK state
|
||||
- `artifact_names` and `artifacts` from `InMemoryArtifactService`
|
||||
- `plugin_events` recorded by an ADK `BasePlugin`
|
||||
- `event_count` from the ADK session
|
||||
|
||||
The eval asserts that:
|
||||
|
||||
- ADK used both `get_weather` and `save_trip_note`
|
||||
- the tool arguments were correct
|
||||
- the tool sequence was correct
|
||||
- ADK emitted `invoke_agent`, `call_llm`, and `execute_tool` spans
|
||||
- no traced error spans were emitted
|
||||
|
||||
## How Tracing Works
|
||||
|
||||
ADK 1.x already emits OpenTelemetry spans for the important framework steps:
|
||||
|
||||
- `invocation`
|
||||
- `invoke_agent <name>`
|
||||
- `call_llm`
|
||||
- `execute_tool <name>`
|
||||
|
||||
`provider.py` keeps those spans inside Promptfoo's trace by:
|
||||
|
||||
1. reading the W3C `traceparent` from the Promptfoo Python provider context
|
||||
2. creating an OpenTelemetry provider with an OTLP HTTP exporter pointed at Promptfoo's receiver
|
||||
3. starting a small provider span under the Promptfoo parent trace
|
||||
4. letting ADK emit its native child spans beneath it
|
||||
|
||||
Because ADK records `gen_ai.tool.name` and tool-call arguments, Promptfoo can normalize those spans into `trajectory:*` assertions without a custom SDK span converter.
|
||||
|
||||
After an eval, open the Trace Timeline for the row and inspect:
|
||||
|
||||
- `invoke_agent weather_agent`
|
||||
- `call_llm`
|
||||
- `execute_tool get_weather`
|
||||
- `execute_tool save_trip_note`
|
||||
- tool attributes such as `gen_ai.tool.name`
|
||||
- ADK tool arguments captured in `gcp.vertex.agent.tool_call_args`
|
||||
|
||||
## Why This Uses A Python Provider
|
||||
|
||||
The older HTTP shape around `adk api_server` is fine when you need to test a deployed service boundary, but it hides useful framework details from Promptfoo. The in-process provider is the better default when you want:
|
||||
|
||||
- direct control over sessions and state
|
||||
- access to artifacts and plugins
|
||||
- trace assertions on ADK's internal workflow
|
||||
- one eval row to represent a long-horizon task
|
||||
|
||||
Use an HTTP provider when the deployed API itself is what you want to validate.
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Evaluate Google ADK agents](https://promptfoo.dev/docs/guides/evaluate-google-adk)
|
||||
- [ADK technical overview](https://adk.dev/get-started/about/)
|
||||
- [ADK sessions](https://adk.dev/sessions/)
|
||||
- [ADK callbacks](https://adk.dev/callbacks/)
|
||||
- [ADK artifacts](https://adk.dev/artifacts/)
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Google ADK agents used by the Promptfoo integration example."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from google.adk.agents import Agent, SequentialAgent
|
||||
from google.adk.apps import App
|
||||
from google.adk.plugins import BasePlugin
|
||||
from google.adk.tools import ToolContext
|
||||
from google.genai import types
|
||||
|
||||
APP_NAME = "promptfoo_adk_demo"
|
||||
|
||||
WEATHER_REPORTS = {
|
||||
"london": "London is cloudy with light drizzle and 14 C temperatures.",
|
||||
"new york": "New York is sunny with a light breeze and 22 C temperatures.",
|
||||
"tokyo": "Tokyo is clear with mild humidity and 24 C temperatures.",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_city(city: str) -> str:
|
||||
return city.strip().casefold()
|
||||
|
||||
|
||||
def before_agent_callback(callback_context) -> None:
|
||||
"""Track how often ADK enters the main conversational agent."""
|
||||
|
||||
callback_context.state["callback_invocations"] = (
|
||||
int(callback_context.state.get("callback_invocations", 0)) + 1
|
||||
)
|
||||
|
||||
|
||||
def get_weather(city: str, tool_context: ToolContext) -> dict[str, Any]:
|
||||
"""Return sample weather data and remember the most recent city."""
|
||||
|
||||
normalized_city = _normalize_city(city)
|
||||
report = WEATHER_REPORTS.get(
|
||||
normalized_city,
|
||||
f"No sample weather is stored for {city}.",
|
||||
)
|
||||
tool_context.state["last_city"] = city
|
||||
return {
|
||||
"city": city,
|
||||
"status": "success" if normalized_city in WEATHER_REPORTS else "not_found",
|
||||
"report": report,
|
||||
}
|
||||
|
||||
|
||||
async def save_trip_note(
|
||||
city: str,
|
||||
summary: str,
|
||||
tool_context: ToolContext,
|
||||
) -> dict[str, Any]:
|
||||
"""Save a short trip note as an ADK artifact."""
|
||||
|
||||
filename = f"{_normalize_city(city).replace(' ', '-')}-trip-note.md"
|
||||
artifact = types.Part.from_bytes(
|
||||
data=f"# Trip note for {city}\n\n{summary}\n".encode("utf-8"),
|
||||
mime_type="text/markdown",
|
||||
)
|
||||
version = await tool_context.save_artifact(filename=filename, artifact=artifact)
|
||||
tool_context.state["last_saved_artifact"] = filename
|
||||
return {
|
||||
"city": city,
|
||||
"filename": filename,
|
||||
"version": version,
|
||||
}
|
||||
|
||||
|
||||
class AuditPlugin(BasePlugin):
|
||||
"""Record runner lifecycle callbacks so the provider can expose them."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(name="audit_plugin")
|
||||
self.events: list[str] = []
|
||||
|
||||
async def before_run_callback(self, *, invocation_context) -> None:
|
||||
self.events.append(f"before_run:{invocation_context.session.id}")
|
||||
|
||||
async def after_run_callback(self, *, invocation_context) -> None:
|
||||
self.events.append(f"after_run:{invocation_context.session.id}")
|
||||
|
||||
|
||||
def build_conversational_app(model: str) -> tuple[App, AuditPlugin]:
|
||||
"""Build the conversational ADK app used by the main eval."""
|
||||
|
||||
audit_plugin = AuditPlugin()
|
||||
root_agent = Agent(
|
||||
name="weather_agent",
|
||||
model=model,
|
||||
description="Answers weather questions and saves trip notes.",
|
||||
instruction=(
|
||||
"You are a concise travel weather assistant. "
|
||||
"When the user asks about weather, call get_weather. "
|
||||
"When the user asks to save a note, call save_trip_note using the "
|
||||
"city already discussed and a brief summary of the weather. "
|
||||
"When the user asks what city was discussed earlier, answer from the "
|
||||
"conversation and current session state."
|
||||
),
|
||||
tools=[get_weather, save_trip_note],
|
||||
before_agent_callback=before_agent_callback,
|
||||
)
|
||||
return (
|
||||
App(name=APP_NAME, root_agent=root_agent, plugins=[audit_plugin]),
|
||||
audit_plugin,
|
||||
)
|
||||
|
||||
|
||||
def build_workflow_app(model: str) -> App:
|
||||
"""Build a small SequentialAgent workflow for the workflow eval."""
|
||||
|
||||
weather_lookup_agent = Agent(
|
||||
name="weather_lookup_agent",
|
||||
model=model,
|
||||
description="Looks up weather for the requested city.",
|
||||
instruction=(
|
||||
"Call get_weather for the city in the user's request, then return a "
|
||||
"single-sentence weather summary."
|
||||
),
|
||||
tools=[get_weather],
|
||||
output_key="weather_snapshot",
|
||||
)
|
||||
briefing_agent = Agent(
|
||||
name="briefing_agent",
|
||||
model=model,
|
||||
description="Turns the weather snapshot into a compact travel brief.",
|
||||
instruction=(
|
||||
"Use the weather snapshot from state to write a one-sentence trip "
|
||||
"brief that includes the city name and a packing suggestion."
|
||||
),
|
||||
)
|
||||
workflow = SequentialAgent(
|
||||
name="trip_planning_workflow",
|
||||
description="Looks up weather, then writes a travel brief.",
|
||||
sub_agents=[weather_lookup_agent, briefing_agent],
|
||||
)
|
||||
return App(name=APP_NAME, root_agent=workflow)
|
||||
@@ -0,0 +1,55 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Google ADK SequentialAgent workflow with native tracing
|
||||
|
||||
prompts:
|
||||
- '{{task}}'
|
||||
|
||||
providers:
|
||||
- id: file://provider.py:call_workflow_api
|
||||
label: google-adk-workflow
|
||||
config:
|
||||
otlp_endpoint: http://localhost:4318
|
||||
|
||||
tests:
|
||||
- description: Sequential workflow looks up weather before drafting a brief
|
||||
vars:
|
||||
task: Write a one-sentence Tokyo trip brief with a packing suggestion.
|
||||
metadata:
|
||||
tracingEnabled: true
|
||||
testCaseId: google-adk-sequential-workflow
|
||||
assert:
|
||||
- type: contains
|
||||
value: Tokyo
|
||||
- type: contains
|
||||
value: '"weather_snapshot"'
|
||||
- type: trajectory:tool-used
|
||||
value: get_weather
|
||||
- type: trajectory:tool-args-match
|
||||
value:
|
||||
name: get_weather
|
||||
args:
|
||||
city: Tokyo
|
||||
mode: partial
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'invoke_agent trip_planning_workflow'
|
||||
min: 1
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'invoke_agent weather_lookup_agent'
|
||||
min: 1
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'invoke_agent briefing_agent'
|
||||
min: 1
|
||||
- type: trace-error-spans
|
||||
value:
|
||||
max_count: 0
|
||||
|
||||
tracing:
|
||||
enabled: true
|
||||
otlp:
|
||||
http:
|
||||
enabled: true
|
||||
port: 4318
|
||||
acceptFormats: ['json', 'protobuf']
|
||||
@@ -0,0 +1,82 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Google ADK conversational agent with native tracing
|
||||
|
||||
prompts:
|
||||
- '{{task}}'
|
||||
|
||||
providers:
|
||||
- id: file://provider.py:call_api
|
||||
label: google-adk-conversation
|
||||
config:
|
||||
otlp_endpoint: http://localhost:4318
|
||||
|
||||
tests:
|
||||
- description: Multi-turn trip note uses state, callback, plugin, artifact, and tools
|
||||
vars:
|
||||
task: Help me with a short London trip note.
|
||||
steps_json: |
|
||||
[
|
||||
"What's the weather in London?",
|
||||
"Save a short trip note for that city.",
|
||||
"Which city did we discuss earlier?"
|
||||
]
|
||||
metadata:
|
||||
tracingEnabled: true
|
||||
testCaseId: google-adk-multi-turn-note
|
||||
assert:
|
||||
- type: contains
|
||||
value: '"last_city": "London"'
|
||||
- type: contains
|
||||
value: '"last_saved_artifact": "london-trip-note.md"'
|
||||
- type: contains
|
||||
value: '"artifact_names": ["london-trip-note.md"]'
|
||||
- type: contains
|
||||
value: '"callback_invocations": 3'
|
||||
- type: contains
|
||||
value: before_run
|
||||
- type: contains
|
||||
value: after_run
|
||||
- type: trajectory:tool-used
|
||||
value:
|
||||
- get_weather
|
||||
- save_trip_note
|
||||
- type: trajectory:tool-args-match
|
||||
value:
|
||||
name: get_weather
|
||||
args:
|
||||
city: London
|
||||
mode: partial
|
||||
- type: trajectory:tool-args-match
|
||||
value:
|
||||
name: save_trip_note
|
||||
args:
|
||||
city: London
|
||||
mode: partial
|
||||
- type: trajectory:tool-sequence
|
||||
value:
|
||||
steps:
|
||||
- get_weather
|
||||
- save_trip_note
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'invoke_agent weather_agent'
|
||||
min: 3
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'call_llm'
|
||||
min: 3
|
||||
- type: trace-span-count
|
||||
value:
|
||||
pattern: 'execute_tool *'
|
||||
min: 2
|
||||
- type: trace-error-spans
|
||||
value:
|
||||
max_count: 0
|
||||
|
||||
tracing:
|
||||
enabled: true
|
||||
otlp:
|
||||
http:
|
||||
enabled: true
|
||||
port: 4318
|
||||
acceptFormats: ['json', 'protobuf']
|
||||
@@ -0,0 +1,293 @@
|
||||
"""Promptfoo Python providers for the Google ADK integration example."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterator
|
||||
|
||||
from google.adk.artifacts import InMemoryArtifactService
|
||||
from google.adk.runners import Runner
|
||||
from google.adk.sessions import InMemorySessionService
|
||||
from google.genai import types
|
||||
from opentelemetry import context as otel_context
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.propagate import extract
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
|
||||
|
||||
EXAMPLE_DIR = Path(__file__).resolve().parent
|
||||
if str(EXAMPLE_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(EXAMPLE_DIR))
|
||||
|
||||
from agent import APP_NAME, build_conversational_app, build_workflow_app
|
||||
|
||||
DEFAULT_MODEL = "gemini-2.5-flash"
|
||||
DEFAULT_USER_ID = "promptfoo-user"
|
||||
DEFAULT_OTLP_ENDPOINT = "http://localhost:4318"
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _TracerProviderState:
|
||||
provider: TracerProvider | None = None
|
||||
otlp_endpoint: str | None = None
|
||||
|
||||
|
||||
_tracer_provider_state = _TracerProviderState()
|
||||
|
||||
|
||||
def _build_steps(prompt: str, vars_dict: dict[str, Any]) -> list[str]:
|
||||
raw_steps = vars_dict.get("steps_json")
|
||||
if not raw_steps:
|
||||
return [prompt]
|
||||
|
||||
try:
|
||||
steps = json.loads(raw_steps)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError("steps_json must be a JSON array of strings") from exc
|
||||
if not isinstance(steps, list) or not all(isinstance(step, str) for step in steps):
|
||||
raise ValueError("steps_json must be a JSON array of strings")
|
||||
return steps
|
||||
|
||||
|
||||
def _session_id(context: dict[str, Any], vars_dict: dict[str, Any]) -> str:
|
||||
if explicit := vars_dict.get("session_id"):
|
||||
return str(explicit)
|
||||
|
||||
evaluation_id = context.get("evaluationId", "local-eval")
|
||||
test_case_id = context.get("testCaseId", "default-test")
|
||||
repeat_index = context.get("repeatIndex")
|
||||
if repeat_index is None:
|
||||
return f"promptfoo-adk-{evaluation_id}-{test_case_id}"
|
||||
return f"promptfoo-adk-{evaluation_id}-{test_case_id}-repeat-{repeat_index}"
|
||||
|
||||
|
||||
def _model(options: dict[str, Any]) -> str:
|
||||
config = options.get("config", {})
|
||||
return str(config.get("model") or os.getenv("ADK_MODEL") or DEFAULT_MODEL)
|
||||
|
||||
|
||||
def _otlp_endpoint(options: dict[str, Any]) -> str:
|
||||
config = options.get("config", {})
|
||||
return str(config.get("otlp_endpoint") or DEFAULT_OTLP_ENDPOINT)
|
||||
|
||||
|
||||
def _ensure_tracer_provider(otlp_endpoint: str) -> TracerProvider:
|
||||
"""Install a TracerProvider that exports to ``otlp_endpoint`` exactly once.
|
||||
|
||||
OpenTelemetry's global ``set_tracer_provider`` is set-once: a subsequent
|
||||
call is silently rejected with an "Overriding of current TracerProvider is
|
||||
not allowed" warning. We mirror that by installing on the first call and
|
||||
refusing to swap endpoints later, which avoids stacking a second
|
||||
SimpleSpanProcessor onto the same provider (which would cause every span
|
||||
to be exported twice).
|
||||
"""
|
||||
|
||||
if _tracer_provider_state.provider is not None:
|
||||
if _tracer_provider_state.otlp_endpoint != otlp_endpoint:
|
||||
_logger.warning(
|
||||
"TracerProvider already configured for %s; ignoring request to switch to %s",
|
||||
_tracer_provider_state.otlp_endpoint,
|
||||
otlp_endpoint,
|
||||
)
|
||||
return _tracer_provider_state.provider
|
||||
|
||||
exporter = OTLPSpanExporter(endpoint=f"{otlp_endpoint.rstrip('/')}/v1/traces")
|
||||
provider = TracerProvider(
|
||||
resource=Resource.create({"service.name": "promptfoo-google-adk-example"})
|
||||
)
|
||||
provider.add_span_processor(SimpleSpanProcessor(exporter))
|
||||
trace.set_tracer_provider(provider)
|
||||
|
||||
_tracer_provider_state.provider = provider
|
||||
_tracer_provider_state.otlp_endpoint = otlp_endpoint
|
||||
return provider
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _provider_span(context: dict[str, Any], otlp_endpoint: str) -> Iterator[None]:
|
||||
provider = _ensure_tracer_provider(otlp_endpoint)
|
||||
tracer = trace.get_tracer("promptfoo.google-adk.example", "1.0.0")
|
||||
parent_context = extract({"traceparent": context.get("traceparent", "")})
|
||||
token = otel_context.attach(parent_context)
|
||||
try:
|
||||
with tracer.start_as_current_span(
|
||||
"promptfoo_google_adk_provider",
|
||||
attributes={
|
||||
"promptfoo.eval.id": str(context.get("evaluationId") or ""),
|
||||
"promptfoo.test.id": str(context.get("testCaseId") or ""),
|
||||
},
|
||||
):
|
||||
yield
|
||||
finally:
|
||||
otel_context.detach(token)
|
||||
provider.force_flush()
|
||||
|
||||
|
||||
def _final_text_from_events(events: list[Any]) -> str:
|
||||
for event in reversed(events):
|
||||
if (
|
||||
not event.is_final_response()
|
||||
or not event.content
|
||||
or not event.content.parts
|
||||
):
|
||||
continue
|
||||
text = "".join(part.text or "" for part in event.content.parts)
|
||||
if text.strip():
|
||||
return text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
async def _artifact_payloads(
|
||||
artifact_service: InMemoryArtifactService,
|
||||
session_id: str,
|
||||
) -> dict[str, str]:
|
||||
payloads: dict[str, str] = {}
|
||||
artifact_names = await artifact_service.list_artifact_keys(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
)
|
||||
for artifact_name in artifact_names:
|
||||
artifact = await artifact_service.load_artifact(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
filename=artifact_name,
|
||||
)
|
||||
if artifact is None:
|
||||
continue
|
||||
if artifact.text:
|
||||
payloads[artifact_name] = artifact.text
|
||||
elif artifact.inline_data and artifact.inline_data.data:
|
||||
payloads[artifact_name] = artifact.inline_data.data.decode("utf-8")
|
||||
return payloads
|
||||
|
||||
|
||||
async def _run_conversational_provider(
|
||||
prompt: str,
|
||||
options: dict[str, Any],
|
||||
context: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
vars_dict = context.get("vars", {})
|
||||
steps = _build_steps(prompt, vars_dict)
|
||||
session_id = _session_id(context, vars_dict)
|
||||
session_service = InMemorySessionService()
|
||||
artifact_service = InMemoryArtifactService()
|
||||
await session_service.create_session(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
state={},
|
||||
)
|
||||
|
||||
app, audit_plugin = build_conversational_app(_model(options))
|
||||
async with Runner(
|
||||
app=app,
|
||||
session_service=session_service,
|
||||
artifact_service=artifact_service,
|
||||
) as runner:
|
||||
for step in steps:
|
||||
async for _ in runner.run_async(
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
new_message=types.Content(role="user", parts=[types.Part(text=step)]),
|
||||
):
|
||||
pass
|
||||
|
||||
session = await session_service.get_session(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
)
|
||||
if session is None:
|
||||
raise RuntimeError(f"Session {session_id} was not found after the run")
|
||||
|
||||
artifacts = await _artifact_payloads(artifact_service, session_id)
|
||||
summary = {
|
||||
"final_answer": _final_text_from_events(session.events),
|
||||
"session_state": session.state,
|
||||
"artifact_names": sorted(artifacts),
|
||||
"artifacts": artifacts,
|
||||
"plugin_events": audit_plugin.events,
|
||||
"event_count": len(session.events),
|
||||
}
|
||||
return {"output": json.dumps(summary, ensure_ascii=False, sort_keys=True)}
|
||||
|
||||
|
||||
async def _run_workflow_provider(
|
||||
prompt: str,
|
||||
options: dict[str, Any],
|
||||
context: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
vars_dict = context.get("vars", {})
|
||||
session_id = _session_id(context, vars_dict)
|
||||
session_service = InMemorySessionService()
|
||||
await session_service.create_session(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
state={},
|
||||
)
|
||||
|
||||
async with Runner(
|
||||
app=build_workflow_app(_model(options)),
|
||||
session_service=session_service,
|
||||
) as runner:
|
||||
async for _ in runner.run_async(
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
new_message=types.Content(role="user", parts=[types.Part(text=prompt)]),
|
||||
):
|
||||
pass
|
||||
|
||||
session = await session_service.get_session(
|
||||
app_name=APP_NAME,
|
||||
user_id=DEFAULT_USER_ID,
|
||||
session_id=session_id,
|
||||
)
|
||||
if session is None:
|
||||
raise RuntimeError(f"Session {session_id} was not found after the run")
|
||||
|
||||
summary = {
|
||||
"final_answer": _final_text_from_events(session.events),
|
||||
"session_state": session.state,
|
||||
"event_count": len(session.events),
|
||||
}
|
||||
return {"output": json.dumps(summary, ensure_ascii=False, sort_keys=True)}
|
||||
|
||||
|
||||
def call_api(
|
||||
prompt: str, options: dict[str, Any], context: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
"""Run the conversational Google ADK example."""
|
||||
|
||||
try:
|
||||
with _provider_span(context, _otlp_endpoint(options)):
|
||||
return asyncio.run(_run_conversational_provider(prompt, options, context))
|
||||
except Exception as exc:
|
||||
return {"error": str(exc), "output": f"Error: {exc}"}
|
||||
|
||||
|
||||
def call_workflow_api(
|
||||
prompt: str,
|
||||
options: dict[str, Any],
|
||||
context: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Run the SequentialAgent workflow example."""
|
||||
|
||||
try:
|
||||
with _provider_span(context, _otlp_endpoint(options)):
|
||||
return asyncio.run(_run_workflow_provider(prompt, options, context))
|
||||
except Exception as exc:
|
||||
return {"error": str(exc), "output": f"Error: {exc}"}
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Focused tests for provider helper behavior."""
|
||||
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
EXAMPLE_DIR = Path(__file__).resolve().parent
|
||||
if str(EXAMPLE_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(EXAMPLE_DIR))
|
||||
|
||||
import provider
|
||||
|
||||
|
||||
class BuildStepsTests(unittest.TestCase):
|
||||
def test_uses_prompt_when_no_steps_json(self):
|
||||
self.assertEqual(provider._build_steps("hello", {}), ["hello"])
|
||||
|
||||
def test_parses_a_json_array_of_strings(self):
|
||||
self.assertEqual(
|
||||
provider._build_steps("ignored", {"steps_json": '["a", "b"]'}),
|
||||
["a", "b"],
|
||||
)
|
||||
|
||||
def test_rejects_non_array_json(self):
|
||||
with self.assertRaisesRegex(ValueError, "JSON array of strings"):
|
||||
provider._build_steps("unused", {"steps_json": '{"bad": true}'})
|
||||
|
||||
def test_rejects_malformed_json_with_clear_error(self):
|
||||
with self.assertRaisesRegex(ValueError, "JSON array of strings"):
|
||||
provider._build_steps("unused", {"steps_json": "not json"})
|
||||
|
||||
|
||||
class SessionIdTests(unittest.TestCase):
|
||||
def test_explicit_session_id_wins(self):
|
||||
session_id = provider._session_id({}, {"session_id": "fixed"})
|
||||
self.assertEqual(session_id, "fixed")
|
||||
|
||||
def test_uses_repeat_index_when_present(self):
|
||||
session_id = provider._session_id(
|
||||
{"evaluationId": "eval-1", "testCaseId": "case-1", "repeatIndex": 2},
|
||||
{},
|
||||
)
|
||||
self.assertEqual(session_id, "promptfoo-adk-eval-1-case-1-repeat-2")
|
||||
|
||||
def test_falls_back_to_defaults(self):
|
||||
session_id = provider._session_id({}, {})
|
||||
self.assertEqual(session_id, "promptfoo-adk-local-eval-default-test")
|
||||
|
||||
|
||||
class TracerProviderTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
# Module-level state is process-wide; reset it so tests can re-install.
|
||||
provider._configured_tracer_provider = None
|
||||
provider._configured_otlp_endpoint = None
|
||||
|
||||
def test_same_endpoint_returns_cached_provider(self):
|
||||
first = provider._ensure_tracer_provider("http://localhost:4318")
|
||||
second = provider._ensure_tracer_provider("http://localhost:4318")
|
||||
self.assertIs(first, second)
|
||||
self.assertEqual(len(first._active_span_processor._span_processors), 1)
|
||||
|
||||
def test_different_endpoint_keeps_existing_provider(self):
|
||||
first = provider._ensure_tracer_provider("http://localhost:4318")
|
||||
with self.assertLogs(provider._logger, level="WARNING") as captured:
|
||||
second = provider._ensure_tracer_provider("http://localhost:4319")
|
||||
self.assertIs(first, second)
|
||||
# No additional processor stacked on the original provider.
|
||||
self.assertEqual(len(first._active_span_processor._span_processors), 1)
|
||||
self.assertTrue(any("ignoring request to switch" in m for m in captured.output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,8 @@
|
||||
google-adk>=1.32.0,<2
|
||||
|
||||
# provider.py imports these directly for OTLP export. They are also pulled in
|
||||
# transitively by google-adk today, but we pin them so the example keeps
|
||||
# working if google-adk drops or changes its OTel surface.
|
||||
opentelemetry-api>=1.30.0
|
||||
opentelemetry-sdk>=1.30.0
|
||||
opentelemetry-exporter-otlp-proto-http>=1.30.0
|
||||
Reference in New Issue
Block a user