chore: import upstream snapshot with attribution

examples/integration-opentelemetry/python/README.md

@@ -0,0 +1,211 @@
+# integration-opentelemetry/python (Python OpenTelemetry Tracing Example)
+
+This example demonstrates how to use OpenTelemetry with Python to trace the internal operations of your LLM providers during Promptfoo evaluations. It uses the **protobuf format** for trace export, which is the default and most efficient format for the Python OpenTelemetry SDK.
+
+## Quick Start
+
+```bash
+npx promptfoo@latest init --example integration-opentelemetry/python
+cd integration-opentelemetry/python
+
+# Create and activate a virtual environment
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Run the evaluation
+npx promptfoo@latest eval
+npx promptfoo@latest view
+```
+
+## Environment Variables
+
+This example requires no API keys - it uses a simulated provider that demonstrates tracing patterns.
+
+## Overview
+
+This example showcases:
+
+- **Python OpenTelemetry SDK** - Using the official Python SDK for tracing
+- **Protobuf format** - The `opentelemetry-exporter-otlp-proto-http` package sends traces in protobuf format (`application/x-protobuf`), which is more efficient than JSON
+- **Distributed tracing** - Parsing W3C Trace Context from Promptfoo and creating child spans
+- **Trace assertions** - Validating trace structure and performance
+
+## How It Works
+
+1. **Promptfoo starts the OTLP receiver** on port 4318
+2. **Promptfoo generates a trace context** for each test case (W3C Trace Context format)
+3. **The Python provider receives the trace context** via `promptfoo_context['traceparent']`
+4. **The provider creates child spans** using the OpenTelemetry Python SDK
+5. **Traces are exported in protobuf format** to Promptfoo's OTLP endpoint
+6. **Promptfoo correlates traces** with test cases for analysis
+
+## Files in This Example
+
+| File                   | Description                                        |
+| ---------------------- | -------------------------------------------------- |
+| `promptfooconfig.yaml` | Evaluation config with tracing enabled             |
+| `provider.py`          | Python provider with OpenTelemetry instrumentation |
+| `requirements.txt`     | Python dependencies (OpenTelemetry SDK)            |
+
+## Protobuf vs JSON
+
+Python's OpenTelemetry SDK uses **protobuf by default** when using `opentelemetry-exporter-otlp-proto-http`:
+
+| Format   | Content-Type             | Package                                  |
+| -------- | ------------------------ | ---------------------------------------- |
+| Protobuf | `application/x-protobuf` | `opentelemetry-exporter-otlp-proto-http` |
+| JSON     | `application/json`       | `opentelemetry-exporter-otlp-http`       |
+
+Protobuf is more efficient for serialization/deserialization and produces smaller payloads, making it the recommended format for production use.
+
+## Provider Implementation
+
+The key parts of the Python provider:
+
+### 1. Initialize OpenTelemetry
+
+```python
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+
+resource = Resource.create({
+    "service.name": "my-python-provider",
+    "service.version": "1.0.0",
+})
+
+exporter = OTLPSpanExporter(
+    endpoint="http://localhost:4318/v1/traces",
+)
+
+# Use SimpleSpanProcessor for synchronous export
+# This ensures spans are exported before the provider returns
+provider = TracerProvider(resource=resource)
+provider.add_span_processor(SimpleSpanProcessor(exporter))
+trace.set_tracer_provider(provider)
+
+tracer = trace.get_tracer("my-python-provider")
+```
+
+> **Note:** This example uses `SimpleSpanProcessor` for synchronous, immediate export. This ensures spans are sent before the provider returns. For production use with higher throughput, consider `BatchSpanProcessor`, but be sure to call `processor.force_flush()` before returning from your provider.
+
+### 2. Parse Trace Context
+
+```python
+import re
+from opentelemetry.trace import SpanContext, TraceFlags
+
+def parse_traceparent(traceparent: str) -> SpanContext | None:
+    match = re.match(r"^(\d{2})-([a-f0-9]{32})-([a-f0-9]{16})-(\d{2})$", traceparent)
+    if not match:
+        return None
+
+    version, trace_id, parent_id, trace_flags = match.groups()
+
+    return SpanContext(
+        trace_id=int(trace_id, 16),
+        span_id=int(parent_id, 16),
+        is_remote=True,
+        trace_flags=TraceFlags(int(trace_flags, 16)),
+    )
+```
+
+### 3. Create Child Spans
+
+```python
+from opentelemetry.trace import SpanKind, Status, StatusCode
+
+def call_api(prompt: str, options: dict, promptfoo_context: dict) -> dict:
+    traceparent = promptfoo_context.get("traceparent")
+
+    if traceparent:
+        span_context = parse_traceparent(traceparent)
+        ctx = trace.set_span_in_context(trace.NonRecordingSpan(span_context))
+
+        with tracer.start_as_current_span(
+            "my_operation",
+            context=ctx,
+            kind=SpanKind.SERVER,
+        ) as span:
+            # Your provider logic here
+            result = do_work()
+            span.set_status(Status(StatusCode.OK))
+            return {"output": result}
+
+    return {"output": do_work()}
+```
+
+## Trace-Based Assertions
+
+This example uses several trace assertion types:
+
+```yaml
+assert:
+  # Count spans matching a pattern
+  - type: trace-span-count
+    value:
+      pattern: 'retrieve_document_*'
+      min: 3
+      max: 3
+
+  # Check span duration
+  - type: trace-span-duration
+    value:
+      pattern: 'rag_agent_workflow'
+      max: 5000 # milliseconds
+
+  # Check for error spans
+  - type: trace-error-spans
+    value:
+      max_count: 0
+```
+
+## Viewing Traces
+
+After running an evaluation, view traces in the web UI:
+
+```bash
+npx promptfoo@latest view
+```
+
+Click on any test result to see the "Trace Timeline" section.
+
+## Dependencies
+
+| Package                                  | Version  | Purpose                       |
+| ---------------------------------------- | -------- | ----------------------------- |
+| `opentelemetry-api`                      | >=1.28.0 | Core tracing API              |
+| `opentelemetry-sdk`                      | >=1.28.0 | SDK implementation            |
+| `opentelemetry-exporter-otlp-proto-http` | >=1.28.0 | OTLP HTTP exporter (protobuf) |
+| `opentelemetry-semantic-conventions`     | >=0.49b0 | Standard attribute names      |
+
+## Troubleshooting
+
+### Traces Not Appearing
+
+1. Verify `tracing.enabled: true` in config
+2. Check OTLP receiver is running (look for port 4318 in logs)
+3. Ensure `processor.force_flush()` is called before returning
+4. Check the trace context is properly parsed from `promptfoo_context['traceparent']`
+
+### Import Errors
+
+Make sure all dependencies are installed:
+
+```bash
+pip install -r requirements.txt
+```
+
+### Connection Refused
+
+Ensure Promptfoo's OTLP receiver is running on port 4318. The receiver starts automatically when `tracing.enabled: true` is set in your config.
+
+## See Also
+
+- [OpenTelemetry Tracing (JavaScript)](../javascript/) - JavaScript version using JSON format
+- [Promptfoo Tracing Documentation](https://promptfoo.dev/docs/tracing/)

examples/integration-opentelemetry/python/promptfooconfig.yaml

@@ -0,0 +1,112 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: OpenTelemetry tracing with Python (protobuf format)
+
+providers:
+  - python:provider.py
+
+prompts:
+  - 'Explain how {{topic}} works in simple terms'
+
+tests:
+  - vars:
+      topic: 'quantum computing'
+    metadata:
+      tracingEnabled: true
+      testCaseId: 'python-test-1'
+    assert:
+      # Ensure the main workflow span exists
+      - type: trace-span-count
+        value:
+          pattern: 'rag_agent_workflow'
+          min: 1
+          max: 1
+
+      # Ensure we retrieve exactly 3 documents
+      - type: trace-span-count
+        value:
+          pattern: 'retrieve_document_*'
+          min: 3
+          max: 3
+
+      # Ensure all reasoning steps occur
+      - type: trace-span-count
+        value:
+          pattern: 'reasoning_*'
+          min: 3
+
+      # Ensure the LLM generation span exists
+      - type: trace-span-count
+        value:
+          pattern: 'llm_generation'
+          min: 1
+
+      # Ensure the overall workflow completes quickly
+      - type: trace-span-duration
+        value:
+          pattern: 'rag_agent_workflow'
+          max: 5000 # 5 seconds max
+
+      # Ensure no errors occur
+      - type: trace-error-spans
+        value:
+          max_count: 0
+
+  - vars:
+      topic: 'machine learning'
+    metadata:
+      tracingEnabled: true
+      testCaseId: 'python-test-2'
+    assert:
+      - type: trace-span-count
+        value:
+          pattern: 'rag_agent_workflow'
+          min: 1
+          max: 1
+
+      - type: trace-span-count
+        value:
+          pattern: 'retrieve_document_*'
+          min: 3
+          max: 3
+
+      - type: trace-span-count
+        value:
+          pattern: 'reasoning_*'
+          min: 3
+
+      - type: trace-span-duration
+        value:
+          pattern: 'rag_agent_workflow'
+          max: 5000
+
+      - type: trace-error-spans
+        value:
+          max_count: 0
+
+# Default assertions for all test cases
+defaultTest:
+  assert:
+    # Monitor overall latency
+    - type: trace-span-duration
+      value:
+        pattern: '*'
+        max: 2000
+        percentile: 95
+      weight: 0
+      metric: p95_latency
+
+    # Ensure retrieval operations are fast
+    - type: trace-span-duration
+      value:
+        pattern: 'retrieve_document_*'
+        max: 500
+
+# Tracing configuration - note we accept both JSON and protobuf
+tracing:
+  enabled: true
+  otlp:
+    http:
+      enabled: true
+      port: 4318
+      # Python's OTLP exporter uses protobuf by default
+      acceptFormats: ['json', 'protobuf']

examples/integration-opentelemetry/python/provider.py

@@ -0,0 +1,186 @@
+"""
+OpenTelemetry-traced Python provider for Promptfoo.
+
+This provider demonstrates how to instrument a Python application with
+OpenTelemetry and send traces to Promptfoo's OTLP receiver using the
+protobuf format (application/x-protobuf).
+
+The Python OpenTelemetry SDK uses protobuf by default when using the
+`opentelemetry-exporter-otlp-proto-http` package, making it ideal for
+testing protobuf support in Promptfoo.
+"""
+
+import re
+import time
+
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.trace import SpanContext, SpanKind, Status, StatusCode, TraceFlags
+
+# Initialize OpenTelemetry with OTLP HTTP exporter (uses protobuf by default)
+resource = Resource.create(
+    {
+        "service.name": "python-rag-provider",
+        "service.version": "1.0.0",
+        "deployment.environment": "development",
+    }
+)
+
+# Create OTLP exporter pointing to Promptfoo's receiver
+# This uses application/x-protobuf content type by default
+exporter = OTLPSpanExporter(
+    endpoint="http://localhost:4318/v1/traces",
+)
+
+# Use SimpleSpanProcessor for immediate export (synchronous)
+# This ensures spans are exported before the provider returns
+# For production use, consider BatchSpanProcessor for better performance
+provider = TracerProvider(resource=resource)
+processor = SimpleSpanProcessor(exporter)
+provider.add_span_processor(processor)
+trace.set_tracer_provider(provider)
+
+tracer = trace.get_tracer("python-rag-provider", "1.0.0")
+
+
+def parse_traceparent(traceparent: str) -> SpanContext | None:
+    """Parse W3C Trace Context traceparent header."""
+    match = re.match(r"^(\d{2})-([a-f0-9]{32})-([a-f0-9]{16})-(\d{2})$", traceparent)
+    if not match:
+        return None
+
+    version, trace_id, parent_id, trace_flags = match.groups()
+
+    return SpanContext(
+        trace_id=int(trace_id, 16),
+        span_id=int(parent_id, 16),
+        is_remote=True,
+        trace_flags=TraceFlags(int(trace_flags, 16)),
+    )
+
+
+def simulate_document_retrieval(doc_name: str, delay: float = 0.05) -> dict:
+    """Simulate retrieving a document from a knowledge base."""
+    time.sleep(delay)
+    return {
+        "name": doc_name,
+        "content": f"This is the content of {doc_name}",
+        "relevance": 0.95,
+    }
+
+
+def simulate_reasoning_step(step_name: str, delay: float = 0.03) -> str:
+    """Simulate a reasoning step in the RAG pipeline."""
+    time.sleep(delay)
+    return f"Completed reasoning: {step_name}"
+
+
+def simulate_llm_call(prompt: str, delay: float = 0.1) -> str:
+    """Simulate calling an LLM for generation."""
+    time.sleep(delay)
+    return f"Generated response for: {prompt[:50]}..."
+
+
+def call_api(prompt: str, options: dict, promptfoo_context: dict) -> dict:
+    """
+    Main provider entry point called by Promptfoo.
+
+    Args:
+        prompt: The rendered prompt to process
+        options: Provider options from config
+        promptfoo_context: Context including traceparent for distributed tracing
+
+    Returns:
+        dict with 'output' key containing the response
+    """
+    traceparent = promptfoo_context.get("traceparent")
+
+    # If no trace context, run without tracing
+    if not traceparent:
+        return {"output": simulate_llm_call(prompt)}
+
+    # Parse the trace context from Promptfoo
+    span_context = parse_traceparent(traceparent)
+    if not span_context:
+        return {"output": simulate_llm_call(prompt)}
+
+    # Create a context with the parent span
+    ctx = trace.set_span_in_context(trace.NonRecordingSpan(span_context))
+
+    # Run the RAG pipeline within the trace context
+    with tracer.start_as_current_span(
+        "rag_agent_workflow",
+        context=ctx,
+        kind=SpanKind.SERVER,
+        attributes={
+            "rag.prompt_length": len(prompt),
+            "rag.model": "simulated-model",
+        },
+    ) as workflow_span:
+        try:
+            # Phase 1: Document Retrieval
+            documents = []
+            for i in range(3):
+                doc_name = f"document_{i + 1}"
+                with tracer.start_as_current_span(
+                    f"retrieve_document_{i + 1}",
+                    kind=SpanKind.CLIENT,
+                    attributes={
+                        "retrieval.document_name": doc_name,
+                        "retrieval.source": "knowledge_base",
+                    },
+                ) as retrieval_span:
+                    doc = simulate_document_retrieval(doc_name)
+                    documents.append(doc)
+                    retrieval_span.set_attribute(
+                        "retrieval.relevance", doc["relevance"]
+                    )
+
+            workflow_span.set_attribute("rag.documents_retrieved", len(documents))
+
+            # Phase 2: Reasoning Steps
+            reasoning_results = []
+            for i, step in enumerate(
+                ["analyze_query", "rank_documents", "synthesize_context"]
+            ):
+                with tracer.start_as_current_span(
+                    f"reasoning_{step}",
+                    kind=SpanKind.INTERNAL,
+                    attributes={
+                        "reasoning.step_number": i + 1,
+                        "reasoning.step_name": step,
+                    },
+                ) as reasoning_span:
+                    result = simulate_reasoning_step(step)
+                    reasoning_results.append(result)
+                    reasoning_span.set_attribute("reasoning.completed", True)
+
+            # Phase 3: LLM Generation
+            with tracer.start_as_current_span(
+                "llm_generation",
+                kind=SpanKind.CLIENT,
+                attributes={
+                    "llm.model": "simulated-model",
+                    "llm.prompt_tokens": len(prompt.split()),
+                },
+            ) as generation_span:
+                output = simulate_llm_call(prompt)
+                generation_span.set_attribute(
+                    "llm.completion_tokens", len(output.split())
+                )
+
+            workflow_span.set_status(Status(StatusCode.OK))
+
+            return {"output": output}
+
+        except Exception as e:
+            workflow_span.set_status(Status(StatusCode.ERROR, str(e)))
+            workflow_span.record_exception(e)
+            raise
+
+
+# Export the function for Promptfoo
+__all__ = ["call_api"]

examples/integration-opentelemetry/python/requirements.txt

@@ -0,0 +1,9 @@
+# OpenTelemetry SDK for Python
+opentelemetry-api>=1.28.0
+opentelemetry-sdk>=1.28.0
+
+# OTLP HTTP exporter (uses protobuf format by default)
+opentelemetry-exporter-otlp-proto-http>=1.28.0
+
+# Semantic conventions for standard attribute names
+opentelemetry-semantic-conventions>=0.49b0