chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,184 @@
+# integration-helicone (Helicone AI Gateway)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example integration-helicone
+cd integration-helicone
+```
+
+This example demonstrates how to use the Helicone AI Gateway provider in promptfoo to route requests through a self-hosted Helicone AI Gateway instance for unified provider access.
+
+## What This Example Shows
+
+- **Unified Interface**: Use the same OpenAI-compatible syntax to access multiple providers
+- **Load Balancing**: Smart routing based on provider availability and performance
+- **Self-Hosted Gateway**: Full control over your LLM routing infrastructure
+- **Provider Comparison**: Compare responses from different providers through a single interface
+- **Flexible Configuration**: Easy switching between providers and models
+
+## Prerequisites
+
+1. **Helicone AI Gateway**: A running instance (we'll start one locally)
+2. **API Keys**: You'll need at least one provider API key:
+   - OpenAI API key (recommended)
+   - Anthropic API key (optional)
+   - Groq API key (optional)
+
+## Setup
+
+1. **Set Environment Variables**:
+
+   ```bash
+   # Set your provider API keys
+   export OPENAI_API_KEY=your_openai_api_key_here
+   export ANTHROPIC_API_KEY=your_anthropic_api_key_here  # Optional
+   export GROQ_API_KEY=your_groq_api_key_here           # Optional
+   ```
+
+2. **Start Helicone AI Gateway**:
+
+   ```bash
+   # In a separate terminal, start the gateway
+   npx @helicone/ai-gateway@latest
+   ```
+
+   The gateway will start on `http://localhost:8080` by default.
+
+3. **Install promptfoo** (if you haven't already):
+   ```bash
+   npm install -g promptfoo
+   ```
+
+## Running the Example
+
+From this directory, run:
+
+```bash
+promptfoo eval
+```
+
+This will:
+
+- Send the same prompts to all three providers through the Helicone AI Gateway
+- Compare responses and performance across providers
+- Generate a detailed comparison report
+- Show differences in model capabilities and response patterns
+
+## What Happens
+
+1. **Request Routing**: Each request is sent to the local Helicone AI Gateway at `http://localhost:8080`
+2. **Provider Selection**: The gateway routes each request to the appropriate provider (OpenAI, Anthropic, or Groq)
+3. **Unified Interface**: All providers use the same OpenAI-compatible request/response format
+4. **Response Comparison**: promptfoo compares the responses from each provider
+
+## Gateway Features
+
+The Helicone AI Gateway provides several powerful features:
+
+- **Load Balancing**: Automatic routing to the fastest/most reliable provider
+- **Caching**: Built-in response caching to reduce costs and improve latency
+- **Rate Limiting**: Configurable rate limits to prevent abuse
+- **Observability**: Optional integration with Helicone's observability platform
+- **Self-Hosted**: Full control over your infrastructure and data
+
+## Configuration Details
+
+The example configuration includes:
+
+### Provider Setup
+
+```yaml
+providers:
+  - id: helicone:openai/gpt-4o-mini
+    label: 'OpenAI via Helicone Gateway'
+    config:
+      temperature: 0.7
+      max_tokens: 500
+```
+
+### Key Features Demonstrated
+
+1. **Unified Interface**: All providers use the same `helicone:provider/model` format
+2. **OpenAI Compatibility**: Standard OpenAI parameters work across all providers
+3. **Easy Switching**: Change providers by simply updating the model name
+4. **Local Gateway**: All requests go through your self-hosted gateway instance
+
+## Customization
+
+You can modify the configuration to:
+
+1. **Add More Providers**: Include any providers supported by your Helicone AI Gateway
+2. **Change Models**: Specify different models using the `provider/model` format
+3. **Custom Gateway**: Point to a different Helicone AI Gateway instance
+4. **Router Configuration**: Use custom routers for different environments
+
+### Example with Custom Gateway and Router
+
+```yaml
+providers:
+  - id: helicone:openai/gpt-4o
+    config:
+      baseUrl: http://my-gateway.company.com:8080
+      router: production
+      temperature: 0.5
+```
+
+## Advanced Features
+
+### Using Different Gateway Endpoints
+
+Route to different environments using routers:
+
+```yaml
+providers:
+  - id: helicone:openai/gpt-4o
+    config:
+      router: production
+
+  - id: helicone:openai/gpt-4o-mini
+    config:
+      router: development
+```
+
+### Custom Gateway Configuration
+
+If you're running your own Helicone AI Gateway with custom configuration:
+
+```yaml
+providers:
+  - id: helicone:custom-provider/custom-model
+    config:
+      baseUrl: http://localhost:9000
+      headers:
+        Custom-Header: value
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Authentication Error**: Verify your `HELICONE_API_KEY` is correct
+2. **Provider API Key Missing**: Ensure you have valid API keys for the providers you're testing
+3. **No Data in Dashboard**: Check that requests are successfully completing
+
+### Debug Mode
+
+For detailed request logging:
+
+```bash
+LOG_LEVEL=debug promptfoo eval
+```
+
+## Learn More
+
+- [Helicone Documentation](https://docs.helicone.ai/)
+- [promptfoo Helicone Provider Guide](/docs/providers/helicone/)
+- [promptfoo Documentation](https://promptfoo.dev/docs/)
+
+## Next Steps
+
+1. **Explore the Dashboard**: Review the analytics in your Helicone dashboard
+2. **Set Up Alerts**: Configure cost and usage alerts in Helicone
+3. **Optimize Costs**: Use caching and rate limiting to reduce expenses
+4. **Scale Testing**: Add more providers and test cases for comprehensive evaluation
@@ -0,0 +1,91 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: 'Helicone AI Gateway provider comparison'
+
+providers:
+  # Basic usage with different providers through Helicone AI Gateway
+  - id: helicone:openai/gpt-4o-mini
+    label: 'OpenAI via Helicone Gateway'
+    config:
+      temperature: 0.7
+      max_tokens: 500
+
+  - id: helicone:anthropic/claude-sonnet-4-6
+    label: 'Anthropic via Helicone Gateway'
+    config:
+      temperature: 0.7
+      max_tokens: 500
+
+  - id: helicone:groq/llama-3.1-8b-instant
+    label: 'Groq via Helicone Gateway'
+    config:
+      temperature: 0.7
+      max_tokens: 500
+
+prompts:
+  - |
+    You are a helpful AI assistant. Please answer the following question concisely and accurately.
+
+    Question: {{question}}
+
+    Provide a clear, informative response.
+
+tests:
+  - description: 'Basic question answering'
+    vars:
+      question: 'What is machine learning?'
+    assert:
+      - type: contains
+        value: 'algorithm'
+      - type: contains
+        value: 'data'
+      - type: llm-rubric
+        value: 'Response accurately explains machine learning concepts'
+
+  - description: 'Creative writing task'
+    vars:
+      question: 'Write a short story about a robot learning to paint in exactly 3 sentences.'
+    assert:
+      - type: llm-rubric
+        value: 'Story is exactly 3 sentences long'
+      - type: llm-rubric
+        value: 'Story is creative and engaging'
+      - type: contains
+        value: 'robot'
+
+  - description: 'Technical explanation'
+    vars:
+      question: 'Explain the difference between supervised and unsupervised learning.'
+    assert:
+      - type: contains
+        value: 'supervised'
+      - type: contains
+        value: 'unsupervised'
+      - type: llm-rubric
+        value: 'Explanation clearly distinguishes between the two types of learning'
+
+  - description: 'Math problem solving'
+    vars:
+      question: 'If a train travels 60 miles per hour for 2.5 hours, how far does it travel?'
+    assert:
+      - type: contains
+        value: '150'
+      - type: llm-rubric
+        value: 'Calculation is correct and clearly explained'
+
+  - description: 'Code explanation'
+    vars:
+      question: 'What does this Python code do: `[x**2 for x in range(10)]`?'
+    assert:
+      - type: contains
+        value: 'list comprehension'
+      - type: contains
+        value: 'square'
+      - type: llm-rubric
+        value: 'Explanation is accurate and includes the output'
+
+defaultTest:
+  options:
+    # Helicone provides built-in cost tracking
+    # Enable cost tracking for comparison
+    includeMetrics: true