promptfoo--promptfoo/examples/anthropic/sonnet-5/promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Claude Sonnet 5 agentic reasoning with effort and adaptive thinking

prompts:
  - |
    {{task}}

providers:
  - id: anthropic:messages:claude-sonnet-5
    config:
      # Sonnet 5 is the Claude 5-generation Sonnet: near-Opus agentic capability at
      # Sonnet pricing, with a 1M-token context window.
      #
      # Sonnet 5 deprecates manual sampling controls (temperature/top_p/top_k) at the
      # model level — promptfoo omits them automatically, so don't set them here.
      #
      # Adaptive thinking is opt-in: without an explicit `thinking` block the model
      # runs WITHOUT extended thinking even at high effort. Set it to let the model
      # decide when and how much to reason per request.
      thinking:
        type: adaptive
      effort: high # Sonnet 5's cost-efficient sweet spot; xhigh/max are available for harder work
      max_tokens: 8000

tests:
  # Multi-system bug diagnosis (intermittent failure)
  - vars:
      task: |
        You're debugging a production issue where users can't log in. Here's what you know:

        1. The frontend shows "Authentication failed" after username/password submission
        2. Backend logs show successful JWT generation
        3. Redis cache is returning stale session data
        4. Database shows correct user credentials
        5. The issue only affects 10% of login attempts
        6. It started after deploying a load balancer configuration change

        Diagnose the root cause and propose a fix. Explain your reasoning about what's
        causing the intermittent nature of the bug.
    assert:
      - type: contains-any
        value: ['load balancer', 'session', 'sticky', 'affinity', 'routing']
        reason: Should identify load balancer session routing as the issue
      - type: llm-rubric
        value: |
          The response should:
          1. Identify the root cause (likely session affinity/sticky sessions issue with the load balancer)
          2. Explain why it's intermittent (requests hit different backends with inconsistent session state)
          3. Propose concrete fixes (sticky sessions, a shared session store, or stateless tokens)
          4. Reason about the tradeoffs of the proposed solutions

  # Production-quality code generation with error handling and caching
  - vars:
      task: |
        Write a Python function that:
        1. Fetches user data from a REST API (may timeout or return errors)
        2. Caches results in Redis with a 5-minute TTL
        3. Falls back to the database on a cache miss
        4. Returns a user object or raises an appropriate exception

        Include proper error handling, type hints, and comments explaining design decisions.
    assert:
      - type: contains
        value: 'def'
        reason: Should include a Python function definition
      - type: contains-any
        value: ['try', 'except', 'raise', 'error']
        reason: Should include error handling
      - type: contains-any
        value: ['cache', 'redis', 'ttl']
        reason: Should implement caching logic
      - type: llm-rubric
        value: |
          The code should:
          1. Include type hints (e.g. from typing import ...)
          2. Handle network timeouts and API errors gracefully
          3. Implement the cache-aside pattern correctly
          4. Include docstrings/comments explaining design decisions
          5. Use appropriate exception types
          6. Be production-ready (not a toy example)