promptfoo--promptfoo/site/docs/providers/truefoundry.md

---
sidebar_label: TrueFoundry
description: Configure TrueFoundry's enterprise-grade AI Gateway (LLM, MCP, and Agent Gateway) to connect, observe, and govern agentic AI applications from a single control plane
---

# TrueFoundry

[TrueFoundry](https://www.truefoundry.com/ai-gateway) provides an enterprise-grade AI Gateway that encompasses an LLM Gateway, MCP Gateway, and Agent Gateway. This enables enterprises to connect, observe, and govern agentic AI applications across providers from a single control plane. TrueFoundry's gateway is OpenAI-compatible and integrates seamlessly with promptfoo for testing and evaluation.

The TrueFoundry provider supports:

- Chat completions from multiple LLM providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, and more)
- Embeddings
- Tool use and function calling
- MCP (Model Context Protocol) servers for enhanced capabilities
- Custom metadata and logging configuration
- Real-time observability and monitoring

## Setup

To use TrueFoundry, you need to set up your API key:

1. Create a TrueFoundry account and obtain an API key from the [TrueFoundry Console](https://www.truefoundry.com/).
2. Set the `TRUEFOUNDRY_API_KEY` environment variable:

```sh
export TRUEFOUNDRY_API_KEY=your_api_key_here
```

Alternatively, you can specify the `apiKey` in the provider configuration (see below).

## Configuration

Configure the TrueFoundry provider in your promptfoo configuration file. The model name should follow the format `provider-account/model-name` (e.g., `openai-main/gpt-5`):

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      max_completion_tokens: 100
prompts:
  - Write a funny tweet about {{topic}}
tests:
  - vars:
      topic: cats
  - vars:
      topic: dogs
```

**Note**: The model identifier format is `provider-account/model-name`. The `provider-account` is the name of your provider integration in TrueFoundry (e.g., `openai-main`, `anthropic-main`). You can find available models in the TrueFoundry LLM Playground UI.

### Basic Configuration Options

The TrueFoundry provider supports all standard OpenAI configuration options:

- `temperature`: Controls randomness in output between 0 and 2
- `max_tokens`: Maximum number of tokens to generate
- `max_completion_tokens`: Maximum number of tokens that can be generated in the chat completion
- `top_p`: Alternative to temperature sampling using nucleus sampling
- `presence_penalty`: Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far
- `frequency_penalty`: Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far
- `stop`: Up to 4 sequences where the API will stop generating further tokens
- `response_format`: Object specifying the format that the model must output (e.g., JSON mode)
- `seed`: For deterministic sampling (best effort)

### Custom API Base URL

For self-hosted or enterprise deployments, you can specify a custom API base URL:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      apiBaseUrl: 'https://your-custom-gateway.example.com'
      temperature: 0.7
```

If not specified, the default URL `https://llm-gateway.truefoundry.com` is used.

### TrueFoundry-Specific Configuration

TrueFoundry provides additional configuration options for metadata tracking and logging:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      metadata:
        user_id: 'test-user'
        environment: 'production'
        custom_key: 'custom_value'
      loggingConfig:
        enabled: true
```

Configuration options:

- `metadata`: Custom metadata to track with each request (object with key-value pairs)
- `loggingConfig`: Logging configuration for observability (must include `enabled: true`)

## Model Support

TrueFoundry supports models from multiple providers. Use the format `provider-account/model-name`:

### OpenAI Models

```yaml
providers:
  - truefoundry:openai-main/gpt-5
  - truefoundry:openai-main/gpt-4o
  - truefoundry:openai-main/gpt-4o-mini
  - truefoundry:openai-main/o1
  - truefoundry:openai-main/o1-mini
```

### Anthropic Models

```yaml
providers:
  - truefoundry:anthropic-main/claude-sonnet-4.5
  - truefoundry:anthropic-main/claude-3-5-sonnet-20241022
  - truefoundry:anthropic-main/claude-3-opus-20240229
```

### Google Gemini Models

```yaml
providers:
  - truefoundry:vertex-ai-main/gemini-2.5-pro
  - truefoundry:vertex-ai-main/gemini-2.5-flash
  - truefoundry:vertex-ai-main/gemini-1.5-pro
```

### Other Providers

```yaml
providers:
  - truefoundry:groq-main/llama-3.3-70b-versatile
  - truefoundry:mistral-main/mistral-large-latest
  - truefoundry:cohere-main/embed-english-v3.0 # Embeddings
```

## Embeddings

TrueFoundry supports embedding models through the same unified API:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/text-embedding-3-large
    config:
      metadata:
        user_id: 'embedding-test'
      loggingConfig:
        enabled: true
tests:
  - vars:
      query: 'What is machine learning?'
    assert:
      - type: is-valid-openai-embedding
```

### Cohere Embeddings

When using Cohere models, you must specify the `input_type` parameter:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:cohere-main/embed-english-v3.0
    config:
      input_type: 'search_query' # Options: search_query, search_document, classification, clustering
      metadata:
        user_id: 'embedding-test'
```

### Multimodal Embeddings (Vertex AI)

TrueFoundry supports multimodal embeddings for images and videos through Vertex AI:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:vertex-ai-main/multimodalembedding@001
    config:
      metadata:
        use_case: 'image-search'
```

## Tool Use and Function Calling

TrueFoundry supports tool use and function calling, compatible with the OpenAI tools format:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      tools:
        - type: function
          function:
            name: get_weather
            description: 'Get the current weather in a given location'
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: 'The city and state, e.g. San Francisco, CA'
                unit:
                  type: string
                  enum:
                    - celsius
                    - fahrenheit
              required:
                - location
      tool_choice: auto
prompts:
  - 'What is the weather in {{location}}?'
tests:
  - vars:
      location: 'San Francisco, CA'
```

## MCP Servers (Model Context Protocol)

TrueFoundry supports MCP servers for enhanced tool capabilities. MCP servers provide access to integrated tools like web search, code execution, and more:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
            - name: 'code_executor'
      iteration_limit: 20
      metadata:
        user_id: 'mcp-test'
      loggingConfig:
        enabled: true
prompts:
  - 'Search the web for {{query}} and summarize the findings'
tests:
  - vars:
      query: 'latest AI developments 2025'
```

### MCP Configuration Options

- `mcp_servers`: Array of MCP server configurations
  - `integration_fqn`: Fully qualified name of the integration (e.g., 'common-tools')
  - `enable_all_tools`: Whether to enable all tools in the integration (boolean)
  - `tools`: Array of specific tools to enable (each with a `name` field)
- `iteration_limit`: Maximum number of iterations for tool calling (default: 20)

### Available MCP Integrations

Common integrations include:

- `common-tools`: Provides web_search, code_executor, and other utilities
- Custom integrations can be configured through your TrueFoundry account

## Complete Example

Here's a comprehensive example demonstrating TrueFoundry's capabilities:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: TrueFoundry AI Gateway evaluation

providers:
  - id: truefoundry:openai-main/gpt-5
    label: 'GPT-5 via TrueFoundry'
    config:
      temperature: 0.7
      max_completion_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
      iteration_limit: 10

  - id: truefoundry:anthropic-main/claude-sonnet-4.5
    label: 'Claude Sonnet 4.5 via TrueFoundry'
    config:
      temperature: 0.7
      max_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true

prompts:
  - |
    You are a helpful assistant. Answer the following question:
    {{question}}

tests:
  - vars:
      question: 'What is the capital of France?'
    assert:
      - type: contains
        value: 'Paris'

  - vars:
      question: 'Explain quantum computing in simple terms'
    assert:
      - type: llm-rubric
        value: 'Provides a clear, simple explanation of quantum computing'

  - vars:
      question: 'Search for the latest news about AI and summarize'
    assert:
      - type: llm-rubric
        value: 'Successfully searches and summarizes recent AI news'
```

## Observability and Monitoring

TrueFoundry provides built-in observability features. When `loggingConfig.enabled` is set to `true`, all requests are logged and can be monitored through the TrueFoundry dashboard.

Key observability features:

- Request and response logging
- Performance metrics (latency, tokens used)
- Cost tracking
- Error monitoring
- Custom metadata for filtering and analysis

## Best Practices

1. **Use Metadata**: Add meaningful metadata to track requests by user, environment, or feature
2. **Enable Logging**: Set `loggingConfig.enabled: true` for production monitoring
3. **Model Selection**: Choose models based on your use case (speed vs. quality tradeoff)
4. **MCP Servers**: Use MCP servers for enhanced capabilities like web search and code execution
5. **Cost Management**: Monitor token usage through TrueFoundry's dashboard

## Additional Resources

- [TrueFoundry Documentation](https://docs.truefoundry.com/docs/ai-gateway)
- [TrueFoundry Blog](https://www.truefoundry.com/blog)
- [OpenAI Provider Documentation](/docs/providers/openai/) (for additional configuration options)

For more information about TrueFoundry's AI Gateway, visit [truefoundry.com/ai-gateway](https://www.truefoundry.com/ai-gateway).