promptfoo--promptfoo/site/docs/providers/truefoundry.md

sidebar_label, description

sidebar_label	description
TrueFoundry	Configure TrueFoundry's enterprise-grade AI Gateway (LLM, MCP, and Agent Gateway) to connect, observe, and govern agentic AI applications from a single control plane

TrueFoundry

TrueFoundry provides an enterprise-grade AI Gateway that encompasses an LLM Gateway, MCP Gateway, and Agent Gateway. This enables enterprises to connect, observe, and govern agentic AI applications across providers from a single control plane. TrueFoundry's gateway is OpenAI-compatible and integrates seamlessly with promptfoo for testing and evaluation.

The TrueFoundry provider supports:

• Chat completions from multiple LLM providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, and more)
• Embeddings
• Tool use and function calling
• MCP (Model Context Protocol) servers for enhanced capabilities
• Custom metadata and logging configuration
• Real-time observability and monitoring

Setup

To use TrueFoundry, you need to set up your API key:

1. Create a TrueFoundry account and obtain an API key from the TrueFoundry Console.
2. Set the TRUEFOUNDRY_API_KEY environment variable:

export TRUEFOUNDRY_API_KEY=your_api_key_here

Alternatively, you can specify the apiKey in the provider configuration (see below).

Configuration

Configure the TrueFoundry provider in your promptfoo configuration file. The model name should follow the format provider-account/model-name (e.g., openai-main/gpt-5):

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      max_completion_tokens: 100
prompts:
  - Write a funny tweet about {{topic}}
tests:
  - vars:
      topic: cats
  - vars:
      topic: dogs

Note: The model identifier format is provider-account/model-name. The provider-account is the name of your provider integration in TrueFoundry (e.g., openai-main, anthropic-main). You can find available models in the TrueFoundry LLM Playground UI.

Basic Configuration Options

The TrueFoundry provider supports all standard OpenAI configuration options:

• temperature: Controls randomness in output between 0 and 2
• max_tokens: Maximum number of tokens to generate
• max_completion_tokens: Maximum number of tokens that can be generated in the chat completion
• top_p: Alternative to temperature sampling using nucleus sampling
• presence_penalty: Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far
• frequency_penalty: Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far
• stop: Up to 4 sequences where the API will stop generating further tokens
• response_format: Object specifying the format that the model must output (e.g., JSON mode)
• seed: For deterministic sampling (best effort)

Custom API Base URL

For self-hosted or enterprise deployments, you can specify a custom API base URL:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      apiBaseUrl: 'https://your-custom-gateway.example.com'
      temperature: 0.7

If not specified, the default URL https://llm-gateway.truefoundry.com is used.

TrueFoundry-Specific Configuration

TrueFoundry provides additional configuration options for metadata tracking and logging:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      metadata:
        user_id: 'test-user'
        environment: 'production'
        custom_key: 'custom_value'
      loggingConfig:
        enabled: true

Configuration options:

• metadata: Custom metadata to track with each request (object with key-value pairs)
• loggingConfig: Logging configuration for observability (must include enabled: true)

Model Support

TrueFoundry supports models from multiple providers. Use the format provider-account/model-name:

OpenAI Models

providers:
  - truefoundry:openai-main/gpt-5
  - truefoundry:openai-main/gpt-4o
  - truefoundry:openai-main/gpt-4o-mini
  - truefoundry:openai-main/o1
  - truefoundry:openai-main/o1-mini

Anthropic Models

providers:
  - truefoundry:anthropic-main/claude-sonnet-4.5
  - truefoundry:anthropic-main/claude-3-5-sonnet-20241022
  - truefoundry:anthropic-main/claude-3-opus-20240229

Google Gemini Models

providers:
  - truefoundry:vertex-ai-main/gemini-2.5-pro
  - truefoundry:vertex-ai-main/gemini-2.5-flash
  - truefoundry:vertex-ai-main/gemini-1.5-pro

Other Providers

providers:
  - truefoundry:groq-main/llama-3.3-70b-versatile
  - truefoundry:mistral-main/mistral-large-latest
  - truefoundry:cohere-main/embed-english-v3.0 # Embeddings

Embeddings

TrueFoundry supports embedding models through the same unified API:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/text-embedding-3-large
    config:
      metadata:
        user_id: 'embedding-test'
      loggingConfig:
        enabled: true
tests:
  - vars:
      query: 'What is machine learning?'
    assert:
      - type: is-valid-openai-embedding

Cohere Embeddings

When using Cohere models, you must specify the input_type parameter:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:cohere-main/embed-english-v3.0
    config:
      input_type: 'search_query' # Options: search_query, search_document, classification, clustering
      metadata:
        user_id: 'embedding-test'

Multimodal Embeddings (Vertex AI)

TrueFoundry supports multimodal embeddings for images and videos through Vertex AI:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:vertex-ai-main/multimodalembedding@001
    config:
      metadata:
        use_case: 'image-search'

Tool Use and Function Calling

TrueFoundry supports tool use and function calling, compatible with the OpenAI tools format:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      tools:
        - type: function
          function:
            name: get_weather
            description: 'Get the current weather in a given location'
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: 'The city and state, e.g. San Francisco, CA'
                unit:
                  type: string
                  enum:
                    - celsius
                    - fahrenheit
              required:
                - location
      tool_choice: auto
prompts:
  - 'What is the weather in {{location}}?'
tests:
  - vars:
      location: 'San Francisco, CA'

MCP Servers (Model Context Protocol)

TrueFoundry supports MCP servers for enhanced tool capabilities. MCP servers provide access to integrated tools like web search, code execution, and more:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
            - name: 'code_executor'
      iteration_limit: 20
      metadata:
        user_id: 'mcp-test'
      loggingConfig:
        enabled: true
prompts:
  - 'Search the web for {{query}} and summarize the findings'
tests:
  - vars:
      query: 'latest AI developments 2025'

MCP Configuration Options

• mcp_servers: Array of MCP server configurations
  • integration_fqn: Fully qualified name of the integration (e.g., 'common-tools')
  • enable_all_tools: Whether to enable all tools in the integration (boolean)
  • tools: Array of specific tools to enable (each with a name field)
• iteration_limit: Maximum number of iterations for tool calling (default: 20)

Available MCP Integrations

Common integrations include:

• common-tools: Provides web_search, code_executor, and other utilities
• Custom integrations can be configured through your TrueFoundry account

Complete Example

Here's a comprehensive example demonstrating TrueFoundry's capabilities:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: TrueFoundry AI Gateway evaluation

providers:
  - id: truefoundry:openai-main/gpt-5
    label: 'GPT-5 via TrueFoundry'
    config:
      temperature: 0.7
      max_completion_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
      iteration_limit: 10

  - id: truefoundry:anthropic-main/claude-sonnet-4.5
    label: 'Claude Sonnet 4.5 via TrueFoundry'
    config:
      temperature: 0.7
      max_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true

prompts:
  - |
    You are a helpful assistant. Answer the following question:
    {{question}}

tests:
  - vars:
      question: 'What is the capital of France?'
    assert:
      - type: contains
        value: 'Paris'

  - vars:
      question: 'Explain quantum computing in simple terms'
    assert:
      - type: llm-rubric
        value: 'Provides a clear, simple explanation of quantum computing'

  - vars:
      question: 'Search for the latest news about AI and summarize'
    assert:
      - type: llm-rubric
        value: 'Successfully searches and summarizes recent AI news'

Observability and Monitoring

TrueFoundry provides built-in observability features. When loggingConfig.enabled is set to true, all requests are logged and can be monitored through the TrueFoundry dashboard.

Key observability features:

• Request and response logging
• Performance metrics (latency, tokens used)
• Cost tracking
• Error monitoring
• Custom metadata for filtering and analysis

Best Practices

1. Use Metadata: Add meaningful metadata to track requests by user, environment, or feature
2. Enable Logging: Set loggingConfig.enabled: true for production monitoring
3. Model Selection: Choose models based on your use case (speed vs. quality tradeoff)
4. MCP Servers: Use MCP servers for enhanced capabilities like web search and code execution
5. Cost Management: Monitor token usage through TrueFoundry's dashboard

Additional Resources

• TrueFoundry Documentation
• TrueFoundry Blog
• OpenAI Provider Documentation (for additional configuration options)

For more information about TrueFoundry's AI Gateway, visit truefoundry.com/ai-gateway.