wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
integration-opentelemetry/built-in (OpenTelemetry Built-in Tracing)
You can run this example with:
npx promptfoo@latest init --example integration-opentelemetry/built-in
cd integration-opentelemetry/built-in
This example demonstrates promptfoo's built-in OpenTelemetry tracing for LLM provider calls.
Quick Start
- Set up environment variables:
# Required for the providers you want to test
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
# Add other provider keys as needed
- Run the evaluation:
npx promptfoo eval -c promptfooconfig.yaml
- View traces in the UI:
npx promptfoo view
Navigate to the Traces tab to see detailed span information.
Configuration
Tracing is enabled by default. Configure via environment variables:
| Variable | Default | Description |
|---|---|---|
PROMPTFOO_DISABLE_TRACING |
false |
Set to true to disable tracing |
OTEL_EXPORTER_OTLP_ENDPOINT |
- | Export traces to external OTLP backend |
OTEL_SERVICE_NAME |
promptfoo |
Service name in traces |
Viewing Traces Externally
With Jaeger
- Start Jaeger:
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4318:4318 \
jaegertracing/all-in-one:latest
- Run eval with OTLP export:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx promptfoo eval
- View at http://localhost:16686
With Honeycomb
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io \
OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY" \
npx promptfoo eval
Trace Attributes
Each LLM call span includes:
GenAI Semantic Conventions
gen_ai.system- Provider system (openai, anthropic, etc.)gen_ai.operation.name- Operation type (chat, completion, embedding)gen_ai.request.model- Requested model namegen_ai.request.max_tokens- Max tokens settinggen_ai.request.temperature- Temperature settinggen_ai.usage.input_tokens- Prompt tokens usedgen_ai.usage.output_tokens- Completion tokens usedgen_ai.usage.total_tokens- Total tokensgen_ai.usage.cached_tokens- Cached tokens (Anthropic)gen_ai.usage.reasoning_tokens- Reasoning tokens (o1 models)gen_ai.response.model- Actual model usedgen_ai.response.id- Provider response IDgen_ai.response.finish_reasons- Finish reasons
Promptfoo Attributes
promptfoo.provider.id- Provider identifierpromptfoo.eval.id- Evaluation run IDpromptfoo.test.index- Test case indexpromptfoo.prompt.label- Prompt label
Supported Providers
All major providers are instrumented:
| Provider | Tracing Support |
|---|---|
| OpenAI | ✓ |
| Anthropic | ✓ |
| Azure OpenAI | ✓ |
| AWS Bedrock | ✓ |
| Google Vertex AI | ✓ |
| Ollama | ✓ |
| Mistral | ✓ |
| Cohere | ✓ |
| Huggingface | ✓ |
| IBM Watsonx | ✓ |
| HTTP | ✓ |
| OpenRouter | ✓ |
| Replicate | ✓ |
| OpenAI-compatible | ✓ (inherited) |
| Cloudflare AI | ✓ (inherited) |