promptfoo--promptfoo/examples/redteam-auth

redteam-auth (Red Team Authentication)

You can run this example with:

npx promptfoo@latest init --example redteam-auth
cd redteam-auth

This example demonstrates how to configure authentication for red team evaluations against HTTP endpoints. It includes three configuration files showing different authentication methods:

1. OAuth 2.0 (promptfooconfig.oauth.yaml) - Client credentials flow for server-to-server authentication
2. API Key (promptfooconfig.api_key.yaml) - API key authentication via headers
3. File Auth (promptfooconfig.file.yaml) - Custom token loading via JavaScript, TypeScript, or Python
4. Digital Signature (promptfooconfig.digital_signature.yaml) - Digital signature authentication using private keys

Overview

When running red team evaluations against protected HTTP endpoints, you need to configure authentication. This example shows how to set up OAuth, API key, file-based, and digital signature authentication in your red team target configuration.

Configuration Files

OAuth Authentication

The promptfooconfig.oauth.yaml file demonstrates OAuth 2.0 client credentials flow:

targets:
  - id: http
    config:
      url: https://example-app.promptfoo.app/minnow/chat?auth_type=bearer
      method: POST
      auth:
        type: oauth
        grantType: client_credentials
        clientId: '{{env.PROMPTFOO_TARGET_CLIENT_ID}}'
        clientSecret: '{{env.PROMPTFOO_TARGET_CLIENT_SECRET}}'
        tokenUrl: https://example-app.promptfoo.app/oauth/token
        scopes: []

API Key Authentication

The promptfooconfig.api_key.yaml file demonstrates API key authentication:

targets:
  - id: http
    config:
      url: https://example-app.promptfoo.app/minnow/chat?auth_type=api_key
      method: POST
      auth:
        type: api_key
        value: '{{env.PROMPTFOO_TARGET_API_KEY}}'
        placement: header
        keyName: X-API-Key

File Authentication

The promptfooconfig.file.yaml file demonstrates file-based authentication:

targets:
  - id: http
    config:
      url: https://example-app.promptfoo.app/minnow/chat?auth_type=bearer
      method: POST
      headers:
        Content-Type: application/json
        Authorization: Bearer {{token}}
      body:
        messages: '{{prompt}}'
      auth:
        type: file
        path: ./auth/get-token.js

The bundled auth scripts simulate the client credentials grant used by the OAuth example. They call https://example-app.promptfoo.app/oauth/token, return the access_token as token, and convert expires_in into the optional expiration field.

The auth file returns an object shaped like:

{
  token: string;
  expiration?: number | null;
}

You can also use:

• ./auth/get-token.ts for TypeScript default exports
• ./auth/get-token.py for Python get_auth
• file://./auth/get-token.ts:buildAuth for named exports

Digital Signature Authentication

The promptfooconfig.digital_signature.yaml file demonstrates digital signature authentication:

targets:
  - id: http
    config:
      url: https://example-app.promptfoo.app/minnow/chat?auth_type=digital_signature
      method: POST
      headers:
        'timestamp': '{{signatureTimestamp}}'
        'signature': '{{signature}}'
      signatureAuth:
        enabled: true
        certificateType: pem
        keyInputType: base64
        type: pem
        privateKey: '{{env.PROMPTFOO_AUTH_PRIVATE_KEY}}'
        signatureValidityMs: 80000
        signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'

Environment Variables

This example requires environment variables depending on which authentication method you use:

For OAuth Authentication

• PROMPTFOO_TARGET_CLIENT_ID - Your OAuth client ID
• PROMPTFOO_TARGET_CLIENT_SECRET - Your OAuth client secret

For API Key Authentication

• PROMPTFOO_TARGET_API_KEY - Your API key

For File Authentication

• PROMPTFOO_TARGET_CLIENT_ID - OAuth client ID used by the bundled auth scripts
• PROMPTFOO_TARGET_CLIENT_SECRET - OAuth client secret used by the bundled auth scripts
• PROMPTFOO_TARGET_SCOPES - Optional space-delimited OAuth scopes

For Digital Signature Authentication

• PROMPTFOO_AUTH_PRIVATE_KEY - Your base64-encoded private key (PEM format)

NOTE: The values for these environment variables are available upon request.

Running the Example

1. Set up environment variables:

# For OAuth
export PROMPTFOO_TARGET_CLIENT_ID=your-client-id
export PROMPTFOO_TARGET_CLIENT_SECRET=your-client-secret

# For API Key
export PROMPTFOO_TARGET_API_KEY=your-api-key

# For File Auth
export PROMPTFOO_TARGET_CLIENT_ID=your-client-id
export PROMPTFOO_TARGET_CLIENT_SECRET=your-client-secret
export PROMPTFOO_TARGET_SCOPES=

# For Digital Signature
export PROMPTFOO_AUTH_PRIVATE_KEY=your-base64-encoded-private-key

2. Run the red team evaluation:

# Using OAuth configuration
promptfoo redteam run -c promptfooconfig.oauth.yaml

# Using API Key configuration
promptfoo redteam run -c promptfooconfig.api_key.yaml

# Using file-based authentication
promptfoo redteam run -c promptfooconfig.file.yaml

# Using Digital Signature configuration
promptfoo redteam run -c promptfooconfig.digital_signature.yaml

3. View the results:

promptfoo view

How It Works

OAuth Flow

When using OAuth authentication:

1. The HTTP provider automatically requests an access token from the tokenUrl using client credentials
2. The token is cached and refreshed when it expires (with a 60-second buffer)
3. The token is added to requests as an Authorization: Bearer <token> header

API Key Flow

When using API key authentication:

1. The API key is read from the environment variable
2. The key is added to requests in the specified header (e.g., X-API-Key: <key>)
3. The placement can be header or query (query parameters)

Digital Signature Flow

When using digital signature authentication:

1. A timestamp is generated for each request
2. The signature data is constructed using the signatureDataTemplate (e.g., promptfoo-app{{signatureTimestamp}})
3. The data is signed using the private key from the environment variable
4. The timestamp and signature are added to request headers
5. The signature is valid for the duration specified by signatureValidityMs (80 seconds in the example)

File Auth Flow

When using file-based authentication:

1. Promptfoo loads the configured JavaScript, TypeScript, or Python file
2. The auth function performs its own client credentials token request
3. It returns a token and optional expiration
4. The token is cached on the provider instance
5. If expiration is omitted, the token is reused indefinitely
6. If expiration is provided, the auth function is called again when the token is close to expiring
7. {{token}} is available for templating into headers, query params, bodies, and raw requests

Security Best Practices

• Never commit credentials to version control
• Use environment variables for all sensitive values
• Use the most restrictive scopes necessary for OAuth
• Rotate credentials regularly in production environments
• Keep private keys secure and use base64 encoding when storing in environment variables
• Set appropriate signature validity windows to balance security and usability

Customizing for Your Endpoint

To use this example with your own endpoint:

1. Update the url in the target configuration
2. Update the tokenUrl for OAuth (if applicable)
3. Adjust the body structure to match your API's expected format
4. Update the transformResponse function to extract the response from your API's format
5. For digital signatures, configure the signatureDataTemplate to match your API's expected signature format
6. For file auth, update the auth script in auth/ to fetch or mint tokens for your API
7. Set the appropriate environment variables

For more information, see the HTTP Provider documentation and Red Team documentation.