chore: import upstream snapshot with attribution

examples/provider-litellm/README.md

@@ -0,0 +1,129 @@
+# provider-litellm (LiteLLM Provider)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-litellm
+cd provider-litellm
+```
+
+This example demonstrates how to use the LiteLLM provider with promptfoo to evaluate multiple models through a unified interface.
+
+## What is LiteLLM?
+
+LiteLLM provides a unified interface to 400+ LLMs. Instead of managing different APIs and authentication methods for each provider, you can use a single interface to access models from OpenAI, Anthropic, Google, and many more.
+
+## Quick Start
+
+1. **Set your API keys**:
+
+   ```bash
+   export OPENAI_API_KEY=your-openai-key
+   # Optional: Add other providers
+   export ANTHROPIC_API_KEY=your-anthropic-key
+   export GOOGLE_AI_API_KEY=your-google-key
+   ```
+
+2. **Start the LiteLLM proxy**:
+
+   ```bash
+   # Use the provided script
+   ./start-proxy.sh
+
+   # Or manually:
+   pip install litellm[proxy]
+   litellm --model gpt-4.1 --model claude-sonnet-4-6 --model gemini-2.5-pro --model text-embedding-3-large
+   ```
+
+3. **Run the evaluation**:
+   ```bash
+   npx promptfoo@latest eval
+   ```
+
+## Features
+
+- **Unified Interface**: Access OpenAI, Anthropic, Google, and 400+ other models through one API
+- **Chat Models**: GPT-4.1, Claude Sonnet 4.6, Gemini 2.5
+- **Embedding Models**: Support for similarity assertions via embedding models
+- **Simple Configuration**: One provider syntax for all models
+- **Cost Tracking**: LiteLLM proxy can track usage across providers
+- **Load Balancing**: Distribute requests across multiple instances
+
+## How It Works
+
+The LiteLLM provider in promptfoo connects to a LiteLLM proxy server (default port 4000). The proxy handles:
+
+- Authentication and routing to various providers
+- Standardizing request/response formats
+- Error handling and retries
+- Optional features like caching and rate limiting
+
+## Configuration Files
+
+- `promptfooconfig.yaml` - Main evaluation configuration
+- `litellm_config.yaml` - LiteLLM proxy server configuration
+- `start-proxy.sh` - Helper script to start the proxy
+
+## Example Configuration
+
+The example evaluates translation and creative writing tasks across three different providers:
+
+```yaml
+providers:
+  - litellm:gpt-4.1
+  - litellm:claude-sonnet-4-6
+  - litellm:gemini-2.5-pro
+
+defaultTest:
+  options:
+    provider:
+      embedding: litellm:embedding:text-embedding-3-large
+```
+
+## Troubleshooting
+
+### Common Errors
+
+- **"Connection refused 0.0.0.0:4000"**: The LiteLLM proxy server is not running. Start it first with `./start-proxy.sh`
+- **"API key not found"**: Set the appropriate environment variables before starting the proxy
+- **"Model not found"**: Ensure the model is included when starting the proxy server
+
+### Verify Setup
+
+1. **Check proxy is running**:
+
+   ```bash
+   curl http://localhost:4000/health
+   ```
+
+2. **Verify API keys**:
+   ```bash
+   echo $OPENAI_API_KEY
+   ```
+
+## Advanced Usage
+
+### Custom Server URL
+
+If your LiteLLM proxy runs on a different host or port:
+
+```yaml
+providers:
+  - id: litellm:gpt-4.1
+    config:
+      apiBaseUrl: https://your-litellm-server.com
+```
+
+### Using Config File
+
+For more complex setups, use the config file:
+
+```bash
+litellm --config litellm_config.yaml
+```
+
+## Learn More
+
+- [LiteLLM Documentation](https://docs.litellm.ai/docs/)
+- [Promptfoo LiteLLM Provider Docs](/docs/providers/litellm)
+- [LiteLLM Proxy Setup](https://docs.litellm.ai/docs/proxy/quick_start)

examples/provider-litellm/litellm_config.yaml

@@ -0,0 +1,26 @@
+# LiteLLM proxy server configuration
+# Usage: litellm --config litellm_config.yaml
+
+model_list:
+  - model_name: gpt-4.1
+    litellm_params:
+      model: openai/gpt-4.1
+      api_key: ${OPENAI_API_KEY}
+
+  - model_name: claude-sonnet-4-6
+    litellm_params:
+      model: anthropic/claude-sonnet-4-6
+      api_key: ${ANTHROPIC_API_KEY}
+
+  - model_name: gemini-2.5-pro
+    litellm_params:
+      model: google/gemini-2.5-pro
+      api_key: ${GOOGLE_AI_API_KEY}
+
+  - model_name: text-embedding-3-large
+    litellm_params:
+      model: openai/text-embedding-3-large
+      api_key: ${OPENAI_API_KEY}
+
+litellm_settings:
+  drop_params: true

examples/provider-litellm/promptfooconfig.yaml

@@ -0,0 +1,46 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: LiteLLM provider example - unified interface for multiple LLMs
+
+# This example demonstrates the LiteLLM provider, which provides a unified
+# interface to 400+ LLMs through a proxy server.
+#
+# REQUIREMENTS:
+# 1. LiteLLM proxy server must be running (use ./start-proxy.sh)
+# 2. Set API keys: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_AI_API_KEY
+
+prompts:
+  - 'Translate this to {{language}}: {{text}}'
+  - 'Write a haiku about {{topic}}'
+
+providers:
+  # All providers use the litellm: prefix and connect through the proxy
+  - litellm:gpt-4.1
+  - litellm:claude-sonnet-4-6
+  - litellm:gemini-2.5-pro
+
+# Configure embedding provider for similarity assertions
+# This also goes through the LiteLLM proxy
+defaultTest:
+  options:
+    provider:
+      embedding: litellm:embedding:text-embedding-3-large
+
+tests:
+  - vars:
+      language: French
+      text: 'Hello, world!'
+    assert:
+      - type: contains
+        value: 'Bonjour'
+      - type: similar
+        value: 'Bonjour, le monde!'
+        threshold: 0.8
+
+  - vars:
+      topic: 'artificial intelligence'
+    assert:
+      - type: javascript
+        value: |
+          // Check if output is a haiku (3 lines)
+          const lines = output.trim().split('\n').filter(line => line.trim());
+          return lines.length === 3;

examples/provider-litellm/start-proxy.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+set -euo pipefail
+
+# Start LiteLLM proxy server for promptfoo example
+
+echo "Starting LiteLLM proxy server..."
+echo ""
+echo "Required environment variables:"
+echo "  - OPENAI_API_KEY (for GPT models and embeddings)"
+echo "  - ANTHROPIC_API_KEY (for Claude models)"
+echo "  - GOOGLE_AI_API_KEY (for Gemini models)"
+echo ""
+
+# Check if litellm is installed
+if ! command -v litellm &>/dev/null; then
+  echo "ERROR: LiteLLM is not installed."
+  echo "Install it with: pip install 'litellm[proxy]'"
+  exit 1
+fi
+
+# Check for at least one API key
+if [ -z "$OPENAI_API_KEY" ] && [ -z "$ANTHROPIC_API_KEY" ] && [ -z "$GOOGLE_AI_API_KEY" ]; then
+  echo "ERROR: No API keys found. Set at least one of the environment variables above."
+  exit 1
+fi
+
+echo "Starting proxy on http://localhost:4000..."
+echo ""
+
+# Start the proxy with models used in the example
+litellm \
+  --model gpt-4.1 \
+  --model claude-sonnet-4-6 \
+  --model gemini-2.5-pro \
+  --model text-embedding-3-large \
+  --port 4000