promptfoo--promptfoo/site/docs/providers/litellm.md

sidebar_label, title, description, keywords

sidebar_label	title	description	keywords
LiteLLM	LiteLLM Provider - Access 400+ LLMs with Unified API	Use LiteLLM with promptfoo to evaluate 400+ language models through a unified OpenAI-compatible interface. Supports chat, completion, and embedding models.	litellm llm provider openai compatible language models ai evaluation gpt-4 claude gemini llama mistral embeddings promptfoo

LiteLLM

LiteLLM provides access to 400+ LLMs through a unified OpenAI-compatible interface.

Usage

You can use LiteLLM with promptfoo in three ways:

1. Dedicated LiteLLM provider

The LiteLLM provider supports chat, completion, and embedding models.

Chat models (default)

providers:
  - id: litellm:<model name>
  # or explicitly:
  - id: litellm:chat:<model name>

Example:

providers:
  - id: litellm:gpt-5-mini
  # or
  - id: litellm:chat:gpt-5-mini

Completion models

providers:
  - id: litellm:completion:<model name>

Embedding models

providers:
  - id: litellm:embedding:<model name>

Example:

providers:
  - id: litellm:embedding:text-embedding-3-large

2. Using with LiteLLM proxy server

If you're running a LiteLLM proxy server:

providers:
  - id: litellm:gpt-5-mini # Uses LITELLM_API_KEY env var
    config:
      apiBaseUrl: http://localhost:4000
      # apiKey: "{{ env.LITELLM_API_KEY }}"  # optional, auto-detected

3. Using OpenAI provider with LiteLLM

Since LiteLLM uses the OpenAI format, you can use the OpenAI provider:

providers:
  - id: openai:chat:gpt-5-mini # Uses LITELLM_API_KEY env var
    config:
      apiBaseUrl: http://localhost:4000
      # apiKey: "{{ env.LITELLM_API_KEY }}"  # optional, auto-detected

Configuration

Basic configuration

providers:
  - id: litellm:gpt-5.1-mini # Uses OPENAI_API_KEY env var
    config:
      # apiKey: "{{ env.OPENAI_API_KEY }}"  # optional, auto-detected
      temperature: 0.7
      max_tokens: 1000

Advanced configuration

All LiteLLM parameters are supported:

providers:
  - id: litellm:claude-4-sonnet # Uses ANTHROPIC_API_KEY env var
    config:
      # apiKey: "{{ env.ANTHROPIC_API_KEY }}"  # optional, auto-detected
      temperature: 0.7
      max_tokens: 4096
      top_p: 0.9
      # Any other LiteLLM-supported parameters

Environment Variables

The LiteLLM provider respects standard environment variables:

• LITELLM_API_KEY - API key for the LiteLLM proxy server
• LITELLM_API_BASE - Base URL for the LiteLLM proxy server (default: http://0.0.0.0:4000)
• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• AZURE_API_KEY
• Other provider-specific environment variables

Embedding Configuration

LiteLLM supports embedding models that can be used for similarity metrics and other tasks. You can specify an embedding provider globally or for individual assertions.

1. Set a default embedding provider for all tests

defaultTest:
  options:
    provider:
      embedding:
        id: litellm:embedding:text-embedding-3-large

2. Override the embedding provider for a specific assertion

assert:
  - type: similar
    value: Reference text
    provider:
      id: litellm:embedding:text-embedding-3-large

Additional configuration options can be passed through the config block if needed:

defaultTest:
  options:
    provider:
      embedding:
        id: litellm:embedding:text-embedding-3-large # Uses OPENAI_API_KEY env var
        config:
          # apiKey: "{{ env.OPENAI_API_KEY }}"  # optional, auto-detected

Complete Example

Here's a complete example using multiple LiteLLM models:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: LiteLLM evaluation example

providers:
  # Chat models
  - id: litellm:gpt-5-mini
  - id: litellm:claude-sonnet-4-5 # Uses ANTHROPIC_API_KEY env var
    # config:
    # apiKey: "{{ env.ANTHROPIC_API_KEY }}"  # optional, auto-detected

  # Embedding model for similarity checks
  - id: litellm:embedding:text-embedding-3-large

prompts:
  - 'Translate this to {{language}}: {{text}}'

tests:
  - vars:
      language: French
      text: 'Hello, world!'
    assert:
      - type: contains
        value: 'Bonjour'
      - type: similar
        value: 'Bonjour, le monde!'
        threshold: 0.8
        provider: litellm:embedding:text-embedding-3-large

Supported Models

LiteLLM supports models from all major providers:

• OpenAI: GPT-4.1, GPT-4, GPT-3.5, embeddings, and more
• Anthropic: Claude 4, Claude 3.7, Claude 3.5, Claude 3, and earlier models
• Google: Gemini and PaLM models
• Meta: Llama models
• Mistral: All Mistral models
• And 400+ more models

For a complete list of supported models, see the LiteLLM model documentation.

Supported Parameters

All standard LiteLLM parameters are passed through:

• temperature
• max_tokens
• top_p
• frequency_penalty
• presence_penalty
• stop
• response_format
• tools / functions
• seed
• Provider-specific parameters

Tips

1. Model naming: Use exact model names as specified in LiteLLM's documentation
2. API keys: Set appropriate API keys for each provider
3. Proxy server: Consider running a LiteLLM proxy server for better control
4. Rate limiting: LiteLLM handles rate limiting automatically
5. Cost tracking: LiteLLM provides built-in cost tracking

Troubleshooting

If you encounter issues:

1. Verify API keys are correctly set
2. Check model name matches LiteLLM's documentation
3. Ensure LiteLLM proxy server (if using) is accessible
4. Review provider-specific requirements in LiteLLM docs

See Also

• LiteLLM Documentation
• Provider Configuration
• OpenAI Provider