chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
"""Promptfoo provider for Strands Agents SDK.
|
||||
|
||||
This module exposes a call_api function that promptfoo uses to interact
|
||||
with the Strands agent for evaluation.
|
||||
|
||||
The provider follows promptfoo's Python provider interface:
|
||||
https://promptfoo.dev/docs/providers/python/
|
||||
|
||||
Usage in promptfooconfig.yaml:
|
||||
providers:
|
||||
- id: 'file://agent_provider.py:call_api'
|
||||
config:
|
||||
model_id: 'gpt-4o-mini'
|
||||
|
||||
For more information:
|
||||
- Strands Agents SDK: https://github.com/strands-agents/sdk-python
|
||||
- Strands Documentation: https://strandsagents.com/
|
||||
- promptfoo Python Providers: https://promptfoo.dev/docs/providers/python/
|
||||
"""
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from agent import run_agent
|
||||
|
||||
|
||||
def call_api(
|
||||
prompt: str, options: Dict[str, Any], context: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""Main entry point for promptfoo to call the Strands agent.
|
||||
|
||||
This function is called by promptfoo for each test case. It receives the
|
||||
rendered prompt and returns the agent's response.
|
||||
|
||||
Args:
|
||||
prompt: The user's input message (rendered from the prompt template)
|
||||
options: Configuration options from the provider config in YAML
|
||||
context: Context information from promptfoo (vars, test metadata, etc.)
|
||||
|
||||
Returns:
|
||||
Dictionary with 'output' key containing the agent's response.
|
||||
On error, includes both 'error' and 'output' keys.
|
||||
"""
|
||||
try:
|
||||
# Extract model configuration from provider options
|
||||
config = options.get("config", {})
|
||||
model_id = config.get("model_id", "gpt-4o-mini")
|
||||
|
||||
# Run the Strands agent and get the response
|
||||
result = run_agent(prompt, model_id)
|
||||
|
||||
# Return in promptfoo's expected format
|
||||
return {"output": result}
|
||||
except Exception as e:
|
||||
# Return error in a format promptfoo can display
|
||||
return {"error": str(e), "output": f"Error: {str(e)}"}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import os
|
||||
|
||||
print("Testing Strands agent provider...")
|
||||
if os.getenv("OPENAI_API_KEY"):
|
||||
result = call_api("What's the weather in New York?", {}, {})
|
||||
print(f"Result: {result}")
|
||||
else:
|
||||
print("Set OPENAI_API_KEY to test.")
|
||||
Reference in New Issue
Block a user