chore: import upstream snapshot with attribution

examples/provider-http/README.md

@@ -0,0 +1,12 @@
+# provider-http (HTTP Provider)
+
+Examples for using promptfoo's generic HTTP provider to connect to any API endpoint.
+
+## Examples
+
+- [basic](./basic/) - Simple HTTP POST provider with response transformation
+- [auth-signature](./auth-signature/) - PEM-based signature authentication
+- [auth-signature-jks](./auth-signature-jks/) - JKS keystore-based signature authentication
+- [auth-signature-pfx](./auth-signature-pfx/) - PFX certificate-based signature authentication
+- [streaming](./streaming/) - Streaming responses via SSE
+- [tls](./tls/) - TLS configurations: CA certs, mutual TLS, PFX bundles

examples/provider-http/auth-signature-jks/README.md

@@ -0,0 +1,123 @@
+# provider-http/auth-signature-jks (HTTP provider with JKS certificate signature authentication)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/auth-signature-jks
+cd provider-http/auth-signature-jks
+```
+
+## Introduction
+
+This example demonstrates how to setup authentication with an HTTP provider using JKS (Java KeyStore) certificates for cryptographic signature validation.
+
+## Prerequisites
+
+- Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends July 30, 2026; Node.js 24 LTS recommended)
+- A JKS keystore file with a keypair for signing/verification
+
+## Setup
+
+### Installation
+
+1. Install dependencies:
+
+```bash
+npm install
+```
+
+2. **Create a JKS keystore** (if you don't have one):
+
+```bash
+# Generate a self-signed certificate and store it in a JKS keystore
+keytool -genkeypair -alias client -keyalg RSA -keysize 2048 \
+  -keystore clientkeystore.jks -storepass password -keypass password \
+  -dname "CN=PromptFoo Test, OU=Test, O=Test, L=Test, ST=Test, C=US" \
+  -validity 365
+```
+
+3. Start the server:
+
+```bash
+npm start
+```
+
+## Configuration
+
+The example uses the following JKS configuration:
+
+- **Keystore Path**: `./clientkeystore.jks`
+- **Keystore Password**: `password`
+- **Key Alias**: `client`
+- **Key Password**: `password`
+- **Signature Algorithm**: SHA256
+
+**Important**: In production, use environment variables for passwords and secure key management practices.
+
+### Checking Your Keystore
+
+If you're unsure about the alias or contents of your JKS keystore, you can inspect it using:
+
+```bash
+keytool -list -keystore clientkeystore.jks -storepass password
+```
+
+This will show all aliases in the keystore. Update the `keyAlias` in both `app.js` and `promptfooconfig.yaml` to match your actual alias.
+
+## Running Tests
+
+```bash
+# Set the keystore password via environment variable
+export PROMPTFOO_JKS_PASSWORD=password
+
+# Run test cases
+promptfoo eval --no-cache
+
+# View results
+promptfoo view
+```
+
+Alternatively, you can uncomment the `keystorePassword` line in `promptfooconfig.yaml` and run directly:
+
+```bash
+# Run test cases (with password in config)
+promptfoo eval --no-cache
+```
+
+**IMPORTANT**: Be sure to run with `--no-cache` when testing! Otherwise it may cache responses from good signatures.
+
+## How it Works
+
+1. The server loads the JKS keystore and extracts the public key certificate
+2. Incoming requests must include signature headers (`signature`, `timestamp`, `client-id`)
+3. The server validates the timestamp and verifies the signature using the public key
+4. Only requests with valid signatures are processed
+
+## Environment Variables
+
+This example demonstrates using environment variables for sensitive data:
+
+- `PROMPTFOO_JKS_PASSWORD` - Password for the JKS keystore (alternative to config keystorePassword)
+- `KEYSTORE_PASSWORD` - Password for the JKS keystore (used by server)
+- `KEY_PASSWORD` - Password for the private key (used by server)
+
+### Using Environment Variables
+
+You can provide the keystore password in two ways:
+
+1. **Via environment variable (recommended for production):**
+
+   ```bash
+   export PROMPTFOO_JKS_PASSWORD=password
+   promptfoo eval
+   ```
+
+2. **Via configuration file:**
+   ```yaml
+   signatureAuth:
+     type: jks
+     keystorePath: ./clientkeystore.jks
+     keystorePassword: password # Direct config
+   ```
+
+If both are provided, the configuration file value takes precedence, with the environment variable serving as a fallback.

examples/provider-http/auth-signature-jks/app.js

@@ -0,0 +1,114 @@
+const express = require('express');
+const crypto = require('crypto');
+const fs = require('fs');
+const jks = require('jks-js');
+const rateLimit = require('express-rate-limit');
+
+const app = express();
+app.use(express.json());
+
+const chatRateLimiter = rateLimit({
+  windowMs: 60 * 1000,
+  max: 30,
+  standardHeaders: true,
+  legacyHeaders: false,
+});
+
+// Add signature validation configuration for JKS
+const SIGNATURE_CONFIG = {
+  keystorePath: './clientkeystore.jks',
+  keystorePassword: 'password', // In real apps, use environment variables
+  keyAlias: 'client', // Common alias for client certificates
+  keyPassword: 'password', // In real apps, use environment variables
+  signatureHeader: 'signature',
+  timestampHeader: 'timestamp',
+  clientIdHeader: 'client-id',
+  signatureValidityMs: 300000, // 5 minutes
+  signatureDataTemplate: 'promptfoo-app{{timestamp}}',
+  signatureAlgorithm: 'SHA256',
+};
+
+// Load JKS keystore and extract public key
+let publicKey;
+try {
+  const keystoreData = fs.readFileSync(SIGNATURE_CONFIG.keystorePath);
+  const keystore = jks.toPem(keystoreData, SIGNATURE_CONFIG.keystorePassword);
+
+  // Find the certificate by alias
+  const cert = keystore[SIGNATURE_CONFIG.keyAlias];
+  if (!cert) {
+    throw new Error(`Certificate with alias '${SIGNATURE_CONFIG.keyAlias}' not found in keystore`);
+  }
+
+  publicKey = cert.cert;
+  console.log('Successfully loaded JKS keystore and extracted public key');
+} catch (error) {
+  console.error('Error loading JKS keystore:', error.message);
+  console.error('Make sure keystore.jks exists and credentials are correct');
+  process.exit(1);
+}
+
+// Signature validation middleware
+function validateSignature(req, res, next) {
+  try {
+    const signature = req.headers[SIGNATURE_CONFIG.signatureHeader];
+    const timestamp = req.headers[SIGNATURE_CONFIG.timestampHeader];
+    const clientId = req.headers[SIGNATURE_CONFIG.clientIdHeader];
+
+    // Check if all required headers are present
+    if (!signature || !timestamp || !clientId) {
+      console.warn('Request rejected: Missing signature headers');
+      return res.status(401).json({ error: 'Missing signature headers' });
+    }
+
+    // Check timestamp validity
+    const now = Date.now();
+    const requestTime = Number.parseInt(timestamp, 10);
+
+    if (Number.isNaN(requestTime) || now - requestTime > SIGNATURE_CONFIG.signatureValidityMs) {
+      console.warn('Request rejected: Signature expired or invalid timestamp');
+      return res.status(401).json({ error: 'Signature expired or invalid timestamp' });
+    }
+
+    // Generate signature data using the template
+    const signatureData = SIGNATURE_CONFIG.signatureDataTemplate.replace(
+      '{{timestamp}}',
+      timestamp,
+    );
+
+    // Verify signature using the public key from JKS
+    const verify = crypto.createVerify(SIGNATURE_CONFIG.signatureAlgorithm);
+    verify.update(signatureData);
+    const isValid = verify.verify(publicKey, signature, 'base64');
+
+    if (!isValid) {
+      console.warn('Request rejected: Invalid signature');
+      return res.status(401).json({ error: 'Invalid signature' });
+    }
+
+    console.log('JKS signature checks out... continuing');
+    next();
+  } catch (error) {
+    console.error('Error validating signature:', error);
+    return res.status(500).json({ error: 'Error validating signature' });
+  }
+}
+
+app.post('/chat', chatRateLimiter, validateSignature, async (req, res) => {
+  try {
+    return res.json({ message: 'hello from JKS authenticated endpoint' });
+  } catch (error) {
+    console.error('Error processing chat request:', error);
+    return res.status(500).json({ error: error.message });
+  }
+});
+
+const PORT = process.env.PORT || 2346;
+app.listen(PORT, (error) => {
+  if (error) {
+    console.error(`Failed to start server: ${error.message}`);
+    process.exit(1);
+    return;
+  }
+  console.info(`JKS server is running on port ${PORT}`);
+});

examples/provider-http/auth-signature-jks/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "http-provider-auth-signature-jks",
+  "version": "1.0.0",
+  "description": "Example HTTP provider with JKS certificate signature authentication",
+  "main": "app.js",
+  "scripts": {
+    "start": "node app.js",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "dependencies": {
+    "express": "^5.2.1",
+    "express-rate-limit": "^8.2.1",
+    "promptfoo": "^0.121.0",
+    "jks-js": "^1.1.5"
+  },
+  "keywords": [
+    "promptfoo",
+    "authentication",
+    "jks",
+    "certificates",
+    "signature"
+  ],
+  "author": "",
+  "license": "MIT"
+}

examples/provider-http/auth-signature-jks/promptfooconfig.yaml

@@ -0,0 +1,31 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: JKS Certificate Signature Authentication
+targets:
+  - id: http
+    config:
+      url: http://localhost:2346/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+        'client-id': 'promptfoo-app'
+        'timestamp': '{{signatureTimestamp}}'
+        'signature': '{{signature}}'
+      signatureAuth:
+        type: jks
+        keystorePath: ./clientkeystore.jks
+        # keystorePassword: password # Can be provided here or via PROMPTFOO_JKS_PASSWORD env var
+        keyAlias: 'client'
+        keyPassword: password
+        signatureValidityMs: 300000
+        signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+        signatureAlgorithm: SHA256
+      body:
+        chat_history: '{{prompt}}'
+prompts:
+  - ' return this: {{prompt}}'
+tests:
+  - vars:
+      prompt: 'I hope our JKS signature works'
+    assert:
+      - type: contains
+        value: 'hello from JKS authenticated endpoint'

examples/provider-http/auth-signature-pfx/README.md

@@ -0,0 +1,148 @@
+# provider-http/auth-signature-pfx (HTTP provider with PFX certificate signature authentication)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/auth-signature-pfx
+cd provider-http/auth-signature-pfx
+```
+
+## Introduction
+
+This example demonstrates how to setup authentication with an HTTP provider using certificates for cryptographic signature validation. You can use either:
+
+1. A PFX certificate file (PKCS#12 format)
+2. Separate CRT and KEY files
+
+## Prerequisites
+
+- Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends July 30, 2026; Node.js 24 LTS recommended)
+- Either a PFX certificate file OR separate CRT and KEY files with a keypair for signing/verification
+
+## Setup
+
+### Installation
+
+1. Install dependencies:
+
+```bash
+npm install
+```
+
+2. **Create certificates** (if you don't have them):
+
+#### Option A: PFX Certificate
+
+```bash
+# First, create a private key and certificate
+openssl req -x509 -newkey rsa:2048 -keyout private.key -out certificate.crt \
+  -days 365 -nodes -subj "/CN=PromptFoo Test/O=Test/C=US"
+
+# Then, create a PFX file from the key and certificate
+openssl pkcs12 -export -out certificate.pfx -inkey private.key -in certificate.crt \
+  -passout pass:password
+
+# Clean up temporary files
+rm private.key certificate.crt
+```
+
+#### Option B: Separate CRT and KEY Files
+
+```bash
+# Create a private key and certificate (keep both files)
+openssl req -x509 -newkey rsa:2048 -keyout private.key -out certificate.crt \
+  -days 365 -nodes -subj "/CN=PromptFoo Test/O=Test/C=US"
+
+# No cleanup needed - both files are used directly
+```
+
+3. Start the server:
+
+```bash
+npm start
+```
+
+## Configuration
+
+The example includes two configuration files demonstrating different certificate formats:
+
+### Option A: PFX Certificate (`promptfooconfig.yaml`)
+
+```yaml
+signatureAuth:
+  type: pfx
+  pfxPath: ./certificate.pfx
+  pfxPassword: password
+  signatureAlgorithm: SHA256
+  signatureValidityMs: 300000
+  signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+```
+
+### Option B: Separate CRT/KEY Files (`promptfooconfig-crt-key.yaml`)
+
+```yaml
+signatureAuth:
+  type: pfx
+  certPath: ./certificate.crt
+  keyPath: ./private.key
+  signatureAlgorithm: SHA256
+  signatureValidityMs: 300000
+  signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+```
+
+**Important**: In production, use environment variables for passwords and secure key management practices.
+
+## Running Tests
+
+Note, for this example to work, you will need to set the environment variable `NODE_TLS_REJECT_UNAUTHORIZED=0`, as this cert is self-signed.
+
+```bash
+# Run test cases with PFX certificate
+NODE_TLS_REJECT_UNAUTHORIZED=0 promptfoo eval --no-cache
+
+# Or run with separate CRT/KEY files
+NODE_TLS_REJECT_UNAUTHORIZED=0 promptfoo eval -c promptfooconfig-crt-key.yaml --no-cache
+
+# View results
+promptfoo view
+```
+
+**IMPORTANT**: Be sure to run with `--no-cache` when testing! Otherwise it may cache responses from good signatures.
+
+## How it Works
+
+1. The server loads the certificate (either from PFX file or separate CRT/KEY files) and extracts the public key for signature verification
+2. The promptfoo HTTP provider uses the same certificate to generate signatures
+3. Incoming requests must include signature headers (`signature`, `timestamp`, `client-id`)
+4. The server validates the timestamp and verifies the signature using the public key
+5. Only requests with valid signatures are processed
+
+## Environment Variables
+
+This example uses hardcoded values for simplicity. In production, you should use:
+
+- `PROMPTFOO_PFX_PASSWORD` - Password for the PFX certificate file (when using PFX option)
+
+## About Certificate Formats
+
+### PFX/PKCS#12
+
+PFX (Personal Information Exchange) is a binary format for storing cryptographic objects. It's commonly used on Windows systems and can contain:
+
+- Private keys
+- Public key certificates
+- Certificate chains
+- Other cryptographic data
+
+This format is password-protected and provides a convenient way to transport certificates and private keys together.
+
+### Separate CRT/KEY Files
+
+Alternatively, you can use separate certificate and key files:
+
+- **CRT file**: Contains the public certificate in PEM format
+- **KEY file**: Contains the private key in PEM format
+
+This approach is common in Unix/Linux environments and provides flexibility in managing certificates and keys separately.
+
+Both formats are supported by the promptfoo HTTP provider for cryptographic signature generation and verification.

examples/provider-http/auth-signature-pfx/app.js

@@ -0,0 +1,146 @@
+const express = require('express');
+const https = require('https');
+const fs = require('fs');
+const crypto = require('crypto');
+const pem = require('pem');
+const rateLimit = require('express-rate-limit');
+
+const app = express();
+app.use(express.json());
+
+const chatRateLimiter = rateLimit({
+  windowMs: 60 * 1000,
+  max: 30,
+  standardHeaders: true,
+  legacyHeaders: false,
+});
+
+// Add signature validation configuration for PFX
+const SIGNATURE_CONFIG = {
+  pfxPath: './certificate.pfx',
+  pfxPassword: 'password', // In real apps, use environment variables
+  signatureHeader: 'signature',
+  timestampHeader: 'timestamp',
+  clientIdHeader: 'client-id',
+  signatureValidityMs: 300000, // 5 minutes
+  signatureDataTemplate: 'promptfoo-app{{timestamp}}',
+  signatureAlgorithm: 'SHA256',
+};
+
+// PFX certificate configuration for HTTPS
+const HTTPS_OPTIONS = {
+  pfx: fs.readFileSync(SIGNATURE_CONFIG.pfxPath),
+  passphrase: SIGNATURE_CONFIG.pfxPassword,
+};
+
+// Load PFX certificate and extract public key
+let publicKey;
+
+async function loadPfxCertificate() {
+  return new Promise((resolve, reject) => {
+    pem.readPkcs12(
+      SIGNATURE_CONFIG.pfxPath,
+      { p12Password: SIGNATURE_CONFIG.pfxPassword },
+      (err, result) => {
+        if (err) {
+          reject(new Error(`Error reading PKCS12/PFX: ${err.message}`));
+          return;
+        }
+
+        try {
+          // Create public key from the certificate
+          publicKey = crypto.createPublicKey(result.cert);
+          console.log(
+            'Successfully loaded PFX certificate and extracted public key using pem library',
+          );
+          resolve();
+        } catch (error) {
+          reject(new Error(`Error creating public key from certificate: ${error.message}`));
+        }
+      },
+    );
+  });
+}
+
+// Signature validation middleware
+function validateSignature(req, res, next) {
+  try {
+    const signature = req.headers[SIGNATURE_CONFIG.signatureHeader];
+    const timestamp = req.headers[SIGNATURE_CONFIG.timestampHeader];
+    const clientId = req.headers[SIGNATURE_CONFIG.clientIdHeader];
+
+    // Check if all required headers are present
+    if (!signature || !timestamp || !clientId) {
+      console.warn('Request rejected: Missing signature headers');
+      return res.status(401).json({ error: 'Missing signature headers' });
+    }
+
+    // Check timestamp validity
+    const now = Date.now();
+    const requestTime = Number.parseInt(timestamp, 10);
+
+    if (Number.isNaN(requestTime) || now - requestTime > SIGNATURE_CONFIG.signatureValidityMs) {
+      console.warn('Request rejected: Signature expired or invalid timestamp');
+      return res.status(401).json({ error: 'Signature expired or invalid timestamp' });
+    }
+
+    // Generate signature data using the template
+    const signatureData = SIGNATURE_CONFIG.signatureDataTemplate.replace(
+      '{{timestamp}}',
+      timestamp,
+    );
+
+    // Verify signature using the public key from PFX
+    const verify = crypto.createVerify(SIGNATURE_CONFIG.signatureAlgorithm);
+    verify.update(signatureData);
+    const isValid = verify.verify(publicKey, signature, 'base64');
+
+    if (!isValid) {
+      console.warn('Request rejected: Invalid signature');
+      return res.status(401).json({ error: 'Invalid signature' });
+    }
+
+    console.log('PFX signature checks out... continuing');
+    next();
+  } catch (error) {
+    console.error('Error validating signature:', error);
+    return res.status(500).json({ error: 'Error validating signature' });
+  }
+}
+
+app.post('/chat', chatRateLimiter, validateSignature, async (req, res) => {
+  try {
+    return res.json({ message: 'hello from PFX authenticated endpoint' });
+  } catch (error) {
+    console.error('Error processing chat request:', error);
+    return res.status(500).json({ error: error.message });
+  }
+});
+
+const PORT = process.env.PORT || 2347;
+
+// Initialize the server
+async function startServer() {
+  try {
+    // Load PFX certificate first
+    await loadPfxCertificate();
+
+    // Create HTTPS server with PFX certificate
+    https.createServer(HTTPS_OPTIONS, app).listen(PORT, (error) => {
+      if (error) {
+        console.error(`Failed to start HTTPS server: ${error.message}`);
+        process.exit(1);
+        return;
+      }
+      console.info(`PFX HTTPS server is running on port ${PORT}`);
+      console.info('Server is using PFX certificate for SSL/TLS and signature validation');
+    });
+  } catch (error) {
+    console.error('Error loading PFX certificate:', error.message);
+    console.error('Make sure certificate.pfx exists and password is correct');
+    process.exit(1);
+  }
+}
+
+// Start the server
+startServer();

examples/provider-http/auth-signature-pfx/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "http-provider-auth-signature-pfx",
+  "version": "1.0.0",
+  "description": "Example HTTP provider with PFX certificate signature authentication",
+  "main": "app.js",
+  "scripts": {
+    "start": "node app.js",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "dependencies": {
+    "express": "^5.2.1",
+    "express-rate-limit": "^8.2.1",
+    "pem": "^1.15.1",
+    "promptfoo": "^0.121.0"
+  },
+  "keywords": [
+    "promptfoo",
+    "authentication",
+    "pfx",
+    "pkcs12",
+    "certificates",
+    "signature"
+  ],
+  "author": "",
+  "license": "MIT"
+}

examples/provider-http/auth-signature-pfx/promptfooconfig-crt-key.yaml

@@ -0,0 +1,30 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Separate CRT/KEY Certificate Signature Authentication
+targets:
+  - id: http
+    config:
+      url: https://localhost:2347/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+        'client-id': 'promptfoo-app'
+        'timestamp': '{{signatureTimestamp}}'
+        'signature': '{{signature}}'
+      signatureAuth:
+        type: pfx
+        # Can either be relative to the promptfoo.yaml file, or absolute paths
+        certPath: certificate.crt
+        keyPath: private.key
+        signatureValidityMs: 300000
+        signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+        signatureAlgorithm: SHA256
+      body:
+        chat_history: '{{prompt}}'
+prompts:
+  - ' return this: {{prompt}}'
+tests:
+  - vars:
+      prompt: 'I hope our separate CRT/KEY signature works'
+    assert:
+      - type: contains
+        value: 'hello from PFX authenticated endpoint'

examples/provider-http/auth-signature-pfx/promptfooconfig.yaml

@@ -0,0 +1,29 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: PFX Certificate Signature Authentication
+targets:
+  - id: http
+    config:
+      url: https://localhost:2347/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+        'client-id': 'promptfoo-app'
+        'timestamp': '{{signatureTimestamp}}'
+        'signature': '{{signature}}'
+      signatureAuth:
+        type: pfx
+        pfxPath: ./certificate.pfx
+        pfxPassword: 'password'
+        signatureValidityMs: 300000
+        signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+        signatureAlgorithm: SHA256
+      body:
+        chat_history: '{{prompt}}'
+prompts:
+  - ' return this: {{prompt}}'
+tests:
+  - vars:
+      prompt: 'I hope our PFX signature works'
+    assert:
+      - type: contains
+        value: 'hello from PFX authenticated endpoint'

examples/provider-http/auth-signature/README.md

@@ -0,0 +1,40 @@
+# provider-http/auth-signature (Setting up an HTTP provider with cryptographically signed requests)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/auth-signature
+cd provider-http/auth-signature
+```
+
+## Introduction
+
+This example demonstrates how to setup authentication with an http provider using a signed authentication mechanism
+
+## Setup
+
+### Installation
+
+1. Install dependencies:
+
+```bash
+npm install
+```
+
+2. Start the server:
+
+```bash
+npm start
+```
+
+## Running Tests
+
+```bash
+# Run test cases
+promptfoo eval --no-cache
+
+# View results
+promptfoo view
+```
+
+IMPORTANT: be sure to run with --no-cache when testing! Otherwise it may cache responses from good signatures.

examples/provider-http/auth-signature/app.js

@@ -0,0 +1,91 @@
+const express = require('express');
+const crypto = require('crypto');
+const fs = require('fs');
+const rateLimit = require('express-rate-limit');
+
+const app = express();
+app.use(express.json());
+
+const chatRateLimiter = rateLimit({
+  windowMs: 60 * 1000,
+  max: 30,
+  standardHeaders: true,
+  legacyHeaders: false,
+});
+
+// Add signature validation configuration
+const SIGNATURE_CONFIG = {
+  publicKeyPath: './public_key.pem',
+  signatureHeader: 'signature',
+  timestampHeader: 'timestamp',
+  clientIdHeader: 'client-id',
+  signatureValidityMs: 300000, // 5 minutes
+  signatureDataTemplate: 'promptfoo-app{{timestamp}}',
+  signatureAlgorithm: 'SHA256',
+};
+
+// Signature validation middleware
+function validateSignature(req, res, next) {
+  try {
+    const signature = req.headers[SIGNATURE_CONFIG.signatureHeader];
+    const timestamp = req.headers[SIGNATURE_CONFIG.timestampHeader];
+    const clientId = req.headers[SIGNATURE_CONFIG.clientIdHeader];
+
+    // Check if all required headers are present
+    if (!signature || !timestamp || !clientId) {
+      console.warn('Request rejected: Missing signature headers');
+      return res.status(401).json({ error: 'Missing signature headers' });
+    }
+
+    // Check timestamp validity
+    const now = Date.now();
+    const requestTime = Number.parseInt(timestamp, 10);
+
+    if (Number.isNaN(requestTime) || now - requestTime > SIGNATURE_CONFIG.signatureValidityMs) {
+      console.warn('Request rejected: Signature expired or invalid timestamp');
+      return res.status(401).json({ error: 'Signature expired or invalid timestamp' });
+    }
+
+    // Generate signature data using the template
+    const signatureData = SIGNATURE_CONFIG.signatureDataTemplate.replace(
+      '{{timestamp}}',
+      timestamp,
+    );
+
+    // Verify signature
+    const publicKey = fs.readFileSync(SIGNATURE_CONFIG.publicKeyPath, 'utf8');
+    const verify = crypto.createVerify(SIGNATURE_CONFIG.signatureAlgorithm);
+    verify.update(signatureData);
+    const isValid = verify.verify(publicKey, signature, 'base64');
+
+    if (!isValid) {
+      console.warn('Request rejected: Invalid signature');
+      return res.status(401).json({ error: 'Invalid signature' });
+    }
+
+    console.log('Signature checks out... continuing');
+    next();
+  } catch (error) {
+    console.error('Error validating signature:', error);
+    return res.status(500).json({ error: 'Error validating signature' });
+  }
+}
+
+app.post('/chat', chatRateLimiter, validateSignature, async (req, res) => {
+  try {
+    return res.json({ message: 'hello' });
+  } catch (error) {
+    console.error('Error processing chat request:', error);
+    return res.status(500).json({ error: error.message });
+  }
+});
+
+const PORT = process.env.PORT || 2345;
+app.listen(PORT, (error) => {
+  if (error) {
+    console.error(`Failed to start server: ${error.message}`);
+    process.exit(1);
+    return;
+  }
+  console.info(`Server is running on port ${PORT}`);
+});

examples/provider-http/auth-signature/package.json

@@ -0,0 +1,17 @@
+{
+  "name": "http-provider-auth-signature",
+  "version": "1.0.0",
+  "description": "Example HTTP provider with signature authentication",
+  "main": "app.js",
+  "scripts": {
+    "start": "node app.js",
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "author": "Promptfoo",
+  "license": "MIT",
+  "dependencies": {
+    "express": "^5.2.1",
+    "express-rate-limit": "^8.2.1",
+    "promptfoo": "^0.121.0"
+  }
+}

examples/provider-http/auth-signature/private_key.pem

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCFGIEV83eqJboz
+tv8zy+L3cK4iEb1EBlm4RQp2NKNkpnYgyAt2/2oZc444Zsymu9vAFSm0BmQW6ocr
+fYnqel9fNihbYzcozyEV0MXTTmMD1MnPtCBQ7Wuc/fZvBQ/c6ATMez4vwLPe7A2+
+TuD7uJCrADBRxoPlCmDogimrOLYVvfzSlfFlO6LiorJIaMBJ6l3uoxtTRGkOazeE
+XJ6/62fep6MlHW84Aq22xv4dYXqL1giDMxaBz5GwDZQVaZe21g+lqKyX6ylmb4YJ
+8W0cyMKzdfOQN3sVK4FuyxTn5KI0BF/1Ph0iBOeSWE3IheiUTN2+YCIi2AosMw9o
+rseUIp1BAgMBAAECggEAAOHPBoiM3s7N3hvfXcYtzzqq8howqUEVhSLH//fiAVUX
+wVMyEuwnFYYY3NzUNw1Vq9SNaIkGTsnnyBer5fxA9448qZENw4fOjf4f4YJKe7ME
+Ugu9OcFpDir3YCL7c5kCDHeLS6EOUB2RCxWAvjT/Qx2G3x6JWeeTXtcbVEGAS21A
+IPglNVuGTWPCbdKffGs6QAGxkV9s8qigzHfwHSA4BH3ZvLYq61B8rMVh/l+a4ruy
+aysRUXI+bd2DN+ORLiP+RGktHGTJw2Foz/OSsKdqZrc6mVVqMlp1pJ8Yck9VHNt1
+kfnjWJtvrZNqVZcLj3KE6a9uT3KW2LRBUyg1VP0ewQKBgQC6vRMfhlAt/h7vBTO2
+iSeDsUGSyDi0n0fFsROPKk6GwMi51ZEhOVaQfz+P+XskXtAc7KShvW1SBtBKA01i
+2/gEE3/nZMk5RvWDsI4Yw6bOC9hHjNn1XQEBUk2Oi42P8aIvhVA8tXqxspOPXtiD
+hSRjlXgA0ziTEbaFZy3+l1hp2QKBgQC2dgM0yqTtk1YXmQimgth2fdsz01m19MqR
+NBqMG81Spn6z7r6fPt/9Ck6XvS+p3nwbBEr6EEhjWq+V3iwUacuIe+SuRRIgv39l
+jNUZqpryzyC6arNDcwPMSzUvFfARjt22kNzKUu497BdQYvhbiOwlKhRWoDwTH7sy
+Iy2WLp+FqQKBgCIRJE/3/OCnH5WTaV+/ncnUqJXSmSW6eSmDqIHRwgmrWfMtFxDs
+mPI7hKkLZn+4HFdqhI5NNIhmXdFi0NdcMd7sf8UDCgK9A0VHDGVQLDoixw4mkAzH
+LsvC7As5QlYkSuZId97bbMrGPU1GjFFSFNVmC0J7RjLuZFHqBOYRTjvhAoGBAJp/
+YQWhlXfJqzt+DJIZ9zqWJTdD/hGRfrjm+peqrvgOHPk07lofPkCgKp5XxnU6+7FQ
+uD/366OdVVI8duyuDHa8GY3q1IfAPxp43rTF/kAdXOQcl0BEsnGZOSZumAH2DAyb
+qyjygeMS90bNFulDDloFwIT9VwEGfFbe7KnfPppJAoGAQRRS/mgm9uqSG8uVlEU/
+1Oi5Rnes6MdSL6p/3O97usmyNfleHnD4YlkIYAtF8vza5Qv99sVvI8U3H0/lAQkj
+JUNdpxbus/rOJ28C4kYlxHidB7oewFyvcEM3+iuiHPQMFOZdcM2pFwLisUPdBprl
+ZPSwP2+wP10Y06mxGd3yBsQ=
+-----END PRIVATE KEY-----

examples/provider-http/auth-signature/promptfooconfig.yaml

@@ -0,0 +1,27 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Auth Signature Test
+targets:
+  - id: http
+    config:
+      url: http://localhost:2345/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+        'client-id': 'promptfoo-app'
+        'timestamp': '{{signatureTimestamp}}'
+        'signature': '{{signature}}'
+      signatureAuth:
+        privateKeyPath: ./private_key.pem
+        signatureValidityMs: 300000
+        signatureDataTemplate: 'promptfoo-app{{signatureTimestamp}}'
+        signatureAlgorithm: SHA256
+      body:
+        chat_history: '{{prompt}}'
+prompts:
+  - ' return this: {{prompt}}'
+tests:
+  - vars:
+      prompt: 'I hope our signature works'
+    assert:
+      - type: contains
+        value: 'hello'

examples/provider-http/auth-signature/public_key.pem

@@ -0,0 +1,9 @@
+-----BEGIN PUBLIC KEY-----
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAhRiBFfN3qiW6M7b/M8vi
+93CuIhG9RAZZuEUKdjSjZKZ2IMgLdv9qGXOOOGbMprvbwBUptAZkFuqHK32J6npf
+XzYoW2M3KM8hFdDF005jA9TJz7QgUO1rnP32bwUP3OgEzHs+L8Cz3uwNvk7g+7iQ
+qwAwUcaD5Qpg6IIpqzi2Fb380pXxZTui4qKySGjASepd7qMbU0RpDms3hFyev+tn
+3qejJR1vOAKttsb+HWF6i9YIgzMWgc+RsA2UFWmXttYPpaisl+spZm+GCfFtHMjC
+s3XzkDd7FSuBbssU5+SiNARf9T4dIgTnklhNyIXolEzdvmAiItgKLDMPaK7HlCKd
+QQIDAQAB
+-----END PUBLIC KEY-----

examples/provider-http/basic/README.md

@@ -0,0 +1,71 @@
+# provider-http/basic (HTTP Provider Example)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/basic
+cd provider-http/basic
+```
+
+This example demonstrates how to configure and use HTTP providers with promptfoo to integrate with external API endpoints.
+
+## Quick Start
+
+1. Visit the [HTTP Provider Generator](/http-provider-generator) to automatically generate configurations based on your HTTP endpoint.
+2. Enter your request configuration and sample response.
+3. Copy the generated configuration to your promptfoo config file.
+
+## Example Configuration
+
+Here's a basic example configuration for an HTTP provider that sends prompts to an API endpoint:
+
+```yaml
+providers:
+  - id: https://api.example.com/chat
+    config:
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        messages:
+          - role: user
+            content: '{{prompt}}'
+      transformResponse: json.choices[0].message.content
+```
+
+## Configuration Options
+
+- `method`: HTTP method (GET, POST, PUT, etc.)
+- `headers`: Request headers (Content-Type, Authorization, etc.)
+- `body`: Request body (supports template variables like `{{prompt}}`)
+- `queryParams`: URL query parameters
+- `transformResponse`: JavaScript expression to extract the response
+
+## Usage Steps
+
+1. Copy the example configuration above
+2. Modify the configuration for your endpoint:
+   - Update the URL
+   - Adjust headers (add authentication if needed)
+   - Modify the request body structure
+   - Update the transformResponse to match your API's response format
+3. Test the configuration using the [HTTP Provider Generator](/http-provider-generator)
+4. Save the working configuration to your promptfoo config file
+
+## Testing Locally
+
+To test your configuration:
+
+1. Create a promptfooconfig.yaml file with your configuration
+2. Run the evaluation:
+   ```bash
+   promptfoo eval
+   ```
+3. View the results:
+   ```bash
+   promptfoo view
+   ```
+
+For development and testing, you can use services like [webhook.site](https://webhook.site) to create test endpoints.
+
+For more detailed information, see the [HTTP Provider documentation](/docs/providers/http).

examples/provider-http/basic/promptfooconfig.yaml

@@ -0,0 +1,25 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'HTTP provider test'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+  - 'Write a very concise, funny tweet about {{topic}}'
+
+providers:
+  - id: https://webhook.site/3d47d2dd-d450-4869-b80d-41862e577e45
+    config:
+      method: 'POST'
+      headers:
+        'Content-Type': 'application/json'
+      body:
+        'prompt': '{{prompt}}'
+        'model':
+          '{{model}}'
+          #
+      # Parse OpenAI response
+      transformResponse: 'json.choices[0].message.content'
+
+tests:
+  - vars:
+      model: gpt-5
+      topic: bananas

examples/provider-http/streaming/README.md

@@ -0,0 +1,50 @@
+# provider-http/streaming (HTTP Provider Streaming Example)
+
+This example shows how to use OpenAI's streaming API via HTTP provider.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/streaming
+cd provider-http/streaming
+```
+
+⚠️ **Streaming is not recommended for evaluations**
+
+Promptfoo supports streaming HTTP targets, but evals wait for full responses before scoring. That means:
+
+- No progressive display during evals
+- Extra parsing complexity for streaming formats (SSE/chunked)
+- Similar end-to-end latency vs. non-streaming
+
+## Environment Variables
+
+Required:
+
+- `OPENAI_API_KEY` - Your OpenAI API key from `https://platform.openai.com/api-keys`
+
+You can set it in your shell or in a project-level `.env` file (recommended):
+
+```bash
+export OPENAI_API_KEY="your-openai-api-key"
+# or in .env
+OPENAI_API_KEY=your-openai-api-key
+```
+
+## Quick Start
+
+1. Set your API key (or ensure `.env` is populated)
+
+2. Run the evaluation (recommended):
+
+   ```bash
+   npx promptfoo@latest eval -c examples/provider-http/streaming/promptfooconfig.yaml
+   ```
+
+3. View results (optional):
+
+   ```bash
+   npx promptfoo@latest view
+   ```
+
+For more HTTP provider configuration options, see the docs: `https://promptfoo.dev/docs/providers/http`.

examples/provider-http/streaming/promptfooconfig.yaml

@@ -0,0 +1,67 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: OpenAI HTTP provider streaming example
+
+# This example shows how to use OpenAI's streaming API via HTTP provider, but this
+# approach is NOT RECOMMENDED. Promptfoo tool waits for the
+# entire response to complete anyway, so you get no streaming benefits.
+#
+# 🎯 RECOMMENDED: Use the built-in OpenAI provider instead: openai:responses:gpt-5-mini
+# - Simpler configuration
+# - Automatic environment variable handling
+# - Better error handling
+# - Optimized performance
+
+prompts:
+  - 'Write a haiku about {{topic}}'
+  - 'Explain {{topic}} in one sentence'
+
+providers:
+  - id: https://api.openai.com/v1/responses
+    config:
+      method: 'POST'
+      headers:
+        'Content-Type': 'application/json'
+        'Authorization': 'Bearer {{env.OPENAI_API_KEY}}'
+      body:
+        model: 'gpt-5-nano'
+        input:
+          - role: 'user'
+            content: '{{prompt}}'
+        stream: true # Demonstrates streaming (no eval benefit)
+      transformResponse: |
+        (json, text) => {
+          // If non-streaming JSON with output_text/response arrives, return it
+          if (json && (json.output_text || json.response)) {
+            return json.output_text || json.response;
+          }
+          // Otherwise parse SSE text and accumulate delta chunks
+          let out = '';
+          for (const line of String(text || '').split('\n')) {
+            const trimmed = line.trim();
+            if (!trimmed.startsWith('data: ')) continue;
+            try {
+              const evt = JSON.parse(trimmed.slice(6));
+              if (evt && evt.type === 'response.output_text.delta' && typeof evt.delta === 'string') {
+                out += evt.delta;
+              }
+            } catch {}
+          }
+          return out.trim();
+        }
+
+defaultTest:
+  assert:
+    # Ensure we get some content
+    - type: javascript
+      value: 'output && output.length > 0'
+    # Verify it's a string
+    - type: javascript
+      value: 'typeof output === "string"'
+
+tests:
+  - vars:
+      topic: 'mountains'
+  - vars:
+      topic: 'artificial intelligence'
+  - vars:
+      topic: 'ocean waves'

examples/provider-http/tls/.gitignore

@@ -0,0 +1,26 @@
+# Certificates and keys
+certs/
+*.pem
+*.pfx
+*.p12
+*.key
+*.crt
+*.cer
+*.csr
+
+# Environment files
+.env
+.env.local
+.env.*.local
+
+# Test outputs
+output/
+results/
+*.log
+
+# Node modules (if running mock server locally)
+node_modules/
+
+# OS files
+.DS_Store
+Thumbs.db 

examples/provider-http/tls/README.md

@@ -0,0 +1,181 @@
+# provider-http/tls (HTTP Provider with TLS Certificates)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/tls
+cd provider-http/tls
+```
+
+This example demonstrates how to configure the HTTP provider with TLS/SSL certificates for mutual TLS (mTLS) authentication.
+
+## Overview
+
+The HTTP provider supports multiple certificate formats for mutual TLS authentication:
+
+1. **PEM (Separate cert/key files)**: Traditional format with separate certificate and key files
+2. **PEM with Encrypted Key**: PEM format where the private key is password-protected
+3. **PFX/PKCS#12**: Combined certificate bundle format
+
+Each format can be provided via:
+
+- **File Path**: Reference files on disk
+- **Inline Content**: Embed certificate content directly (base64 for binary formats)
+- **Environment Variables**: Load from environment variables
+
+## PEM Certificate Options
+
+### Using Unencrypted PEM Files
+
+The simplest approach - separate certificate and key files:
+
+```yaml
+tls:
+  certPath: '/path/to/client-cert.pem'
+  keyPath: '/path/to/client-key.pem'
+```
+
+### Using Encrypted PEM Private Key
+
+When your private key is password-protected (starts with `BEGIN ENCRYPTED PRIVATE KEY`):
+
+```yaml
+tls:
+  certPath: '/path/to/client-cert.pem'
+  keyPath: '/path/to/client-key-encrypted.pem'
+  passphrase: 'your-key-password'
+```
+
+### Using Inline PEM Content
+
+Embed certificates directly in your configuration:
+
+```yaml
+tls:
+  cert: |
+    -----BEGIN CERTIFICATE-----
+    MIIDxTCCAq2gAwIBAgIJAL...
+    -----END CERTIFICATE-----
+  key: |
+    -----BEGIN PRIVATE KEY-----
+    MIIEvQIBADANBgkqhkiG9w0B...
+    -----END PRIVATE KEY-----
+```
+
+## PFX Certificate Options
+
+### Using File Path
+
+Reference a PFX file on the filesystem:
+
+```yaml
+tls:
+  pfxPath: '/path/to/certificate.pfx'
+  passphrase: 'your-passphrase'
+```
+
+### Using Inline Base64 Content
+
+Embed the certificate directly in your configuration:
+
+```yaml
+tls:
+  pfx: 'MIIJKQIBAzCCCO8GCSqGSIb3DQEHA...' # Base64-encoded PFX
+  passphrase: 'your-passphrase'
+```
+
+### Using Environment Variables
+
+Store sensitive certificates in environment variables:
+
+```yaml
+tls:
+  pfx: '{{env.PFX_CERTIFICATE_BASE64}}'
+  passphrase: '{{env.PFX_PASSPHRASE}}'
+```
+
+## Converting PFX to Base64
+
+To use inline PFX certificates, you need to convert your PFX file to base64:
+
+### Linux/Mac
+
+```bash
+base64 -i certificate.pfx -o certificate.b64
+```
+
+### Windows
+
+```cmd
+certutil -encode certificate.pfx certificate.b64
+```
+
+Then copy the content (excluding the BEGIN/END headers) to use as the `pfx` value.
+
+## Security Considerations
+
+1. **Never commit certificates to version control**: Use environment variables or external secret management
+2. **Protect your private keys**: Ensure PFX files have appropriate file permissions
+3. **Use strong passphrases**: Always protect PFX files with strong passphrases
+4. **Certificate validation**: Keep `rejectUnauthorized: true` in production
+
+## Running the Example
+
+1. Replace the sample certificate values with your actual certificates
+2. Set the required environment variables:
+   ```bash
+   export PFX_PASSPHRASE="your-passphrase"
+   export PFX_CERTIFICATE_BASE64="your-base64-cert"
+   ```
+3. Run the evaluation:
+   ```bash
+   promptfoo eval
+   ```
+
+## TLS Configuration Options
+
+| Option               | Description                                               |
+| -------------------- | --------------------------------------------------------- |
+| `cert`               | Inline PEM certificate content                            |
+| `certPath`           | Path to PEM certificate file                              |
+| `key`                | Inline PEM private key content                            |
+| `keyPath`            | Path to PEM private key file                              |
+| `pfx`                | Inline PFX certificate (base64-encoded string or Buffer)  |
+| `pfxPath`            | Path to PFX file on disk                                  |
+| `passphrase`         | Password for encrypted PEM private key or PFX certificate |
+| `ca`                 | CA certificate content for server verification            |
+| `caPath`             | Path to CA certificate file                               |
+| `rejectUnauthorized` | Verify server certificates (always `true` in production)  |
+| `minVersion`         | Minimum TLS version (e.g., 'TLSv1.2')                     |
+| `maxVersion`         | Maximum TLS version (e.g., 'TLSv1.3')                     |
+| `ciphers`            | Cipher suite specification                                |
+
+## Troubleshooting
+
+### Invalid PFX Format
+
+If you get an error about invalid PFX format:
+
+- Ensure the base64 encoding is correct
+- Verify the passphrase is correct
+- Check that the PFX file is not corrupted
+
+### Connection Refused
+
+If the connection is refused:
+
+- Verify the server requires client certificates
+- Ensure the certificate is valid and not expired
+- Check that the certificate is trusted by the server
+
+### Certificate Verification Failed
+
+If certificate verification fails:
+
+- Add the server's CA certificate using `ca` or `caPath`
+- For development only: set `rejectUnauthorized: false` (never in production)
+
+## Related Documentation
+
+- [HTTP Provider Documentation](https://promptfoo.com/docs/providers/http)
+- [TLS/HTTPS Configuration](https://promptfoo.com/docs/providers/http#tlshttps-configuration)

examples/provider-http/tls/env.example

@@ -0,0 +1,4 @@
+# HTTP Provider TLS Example Environment Variables
+# Mock server settings (for local testing)
+PORT=8443
+REQUIRE_CLIENT_CERT=true 

examples/provider-http/tls/generate-test-certs.sh

@@ -0,0 +1,129 @@
+#!/bin/bash
+
+# Generate self-signed certificates for testing TLS configurations
+# WARNING: These certificates are for testing only, not for production!
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${GREEN}🔐 Generating test certificates for TLS example${NC}"
+echo -e "${YELLOW}⚠️  WARNING: These certificates are for testing only!${NC}\n"
+
+# Create certs directory
+mkdir -p certs
+cd certs
+
+# Certificate configuration
+DAYS_VALID=365
+KEY_SIZE=2048
+COUNTRY="US"
+STATE="California"
+LOCALITY="San Francisco"
+ORG="Promptfoo Test"
+OU="Development"
+
+# Generate CA private key
+echo -e "${GREEN}1. Generating CA private key...${NC}"
+openssl genrsa -out ca-key.pem $KEY_SIZE
+
+# Generate CA certificate
+echo -e "${GREEN}2. Generating CA certificate...${NC}"
+openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days $DAYS_VALID \
+  -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG/OU=$OU/CN=Promptfoo Test CA"
+
+# Generate server private key
+echo -e "${GREEN}3. Generating server private key...${NC}"
+openssl genrsa -out server-key.pem $KEY_SIZE
+
+# Generate server certificate signing request
+echo -e "${GREEN}4. Generating server CSR...${NC}"
+cat >server.cnf <<EOF
+[req]
+distinguished_name = req_distinguished_name
+req_extensions = v3_req
+prompt = no
+
+[req_distinguished_name]
+C = $COUNTRY
+ST = $STATE
+L = $LOCALITY
+O = $ORG
+OU = $OU
+CN = localhost
+
+[v3_req]
+keyUsage = keyEncipherment, dataEncipherment
+extendedKeyUsage = serverAuth
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost
+DNS.2 = *.localhost
+IP.1 = 127.0.0.1
+IP.2 = ::1
+EOF
+
+openssl req -new -key server-key.pem -out server.csr -config server.cnf
+
+# Sign server certificate with CA
+echo -e "${GREEN}5. Signing server certificate with CA...${NC}"
+openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
+  -CAcreateserial -out server-cert.pem -days $DAYS_VALID \
+  -extensions v3_req -extfile server.cnf
+
+# Generate client private key
+echo -e "${GREEN}6. Generating client private key...${NC}"
+openssl genrsa -out client-key.pem $KEY_SIZE
+
+# Generate client certificate signing request
+echo -e "${GREEN}7. Generating client CSR...${NC}"
+openssl req -new -key client-key.pem -out client.csr \
+  -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG/OU=$OU/CN=Test Client"
+
+# Sign client certificate with CA
+echo -e "${GREEN}8. Signing client certificate with CA...${NC}"
+openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem \
+  -CAcreateserial -out client-cert.pem -days $DAYS_VALID
+
+# Generate PFX bundle from client cert and key
+echo -e "${GREEN}9. Creating PFX bundle...${NC}"
+openssl pkcs12 -export -out client.pfx \
+  -inkey client-key.pem -in client-cert.pem \
+  -certfile ca-cert.pem -password pass:testpassword
+
+# Generate encrypted client private key (for testing passphrase with PEM)
+echo -e "${GREEN}10. Creating encrypted client private key...${NC}"
+openssl rsa -aes256 -in client-key.pem -out client-key-encrypted.pem \
+  -passout pass:testpassword
+
+# Clean up temporary files
+rm -f *.csr *.cnf *.srl
+
+# Display certificate information
+echo -e "\n${GREEN}✅ Certificates generated successfully!${NC}"
+echo -e "\nGenerated files:"
+echo "  📄 ca-cert.pem              - Certificate Authority certificate"
+echo "  🔑 ca-key.pem               - CA private key (keep secure!)"
+echo "  📄 server-cert.pem          - Server certificate"
+echo "  🔑 server-key.pem           - Server private key"
+echo "  📄 client-cert.pem          - Client certificate"
+echo "  🔑 client-key.pem           - Client private key (unencrypted)"
+echo "  🔑 client-key-encrypted.pem - Client private key (encrypted, password: testpassword)"
+echo "  📦 client.pfx               - Client certificate bundle (password: testpassword)"
+
+echo -e "\n${YELLOW}Certificate Details:${NC}"
+echo "CA Certificate:"
+openssl x509 -in ca-cert.pem -noout -subject -dates | sed 's/^/  /'
+echo -e "\nServer Certificate:"
+openssl x509 -in server-cert.pem -noout -subject -dates | sed 's/^/  /'
+echo -e "\nClient Certificate:"
+openssl x509 -in client-cert.pem -noout -subject -dates | sed 's/^/  /'
+
+echo -e "\n${GREEN}You can now test the TLS configuration with:${NC}"
+echo "  1. Start the mock server: node ../mock-server.js"
+echo "  2. Run the evaluation: npm run local -- eval -c ../promptfooconfig-mock.yaml"

examples/provider-http/tls/mock-server.js

@@ -0,0 +1,120 @@
+/**
+ * Mock HTTPS Server for testing TLS certificate configurations
+ *
+ * This server demonstrates:
+ * - HTTPS with self-signed certificates
+ * - Client certificate verification (mTLS)
+ * - Custom TLS options
+ */
+
+import fs from 'node:fs';
+import https from 'node:https';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+
+// Configuration
+const PORT = process.env.PORT || 8443;
+const REQUIRE_CLIENT_CERT = process.env.REQUIRE_CLIENT_CERT === 'true';
+
+// Server options
+const serverOptions = {
+  // Server certificate and key (self-signed for testing)
+  key: fs.readFileSync(path.join(__dirname, 'certs', 'server-key.pem')),
+  cert: fs.readFileSync(path.join(__dirname, 'certs', 'server-cert.pem')),
+};
+
+// Add client certificate verification for mTLS
+if (REQUIRE_CLIENT_CERT) {
+  serverOptions.requestCert = true;
+  serverOptions.rejectUnauthorized = true;
+  serverOptions.ca = fs.readFileSync(path.join(__dirname, 'certs', 'ca-cert.pem'));
+  console.log('📋 Client certificate verification enabled (mTLS)');
+}
+
+// Create HTTPS server
+const server = https.createServer(serverOptions, (req, res) => {
+  let body = '';
+
+  req.on('data', (chunk) => {
+    body += chunk.toString();
+  });
+
+  req.on('end', () => {
+    console.log(`📥 ${req.method} ${req.url}`);
+    console.log('Headers:', req.headers);
+
+    if (body) {
+      try {
+        const parsed = JSON.parse(body);
+        console.log('Body:', parsed);
+      } catch {
+        console.log('Body (raw):', body);
+      }
+    }
+
+    // Check for client certificate
+    if (req.socket.getPeerCertificate) {
+      const cert = req.socket.getPeerCertificate();
+      if (cert && cert.subject) {
+        console.log('✅ Client certificate:', cert.subject);
+      }
+    }
+
+    // Simulate API response
+    const response = {
+      result: `Mock response for: ${body ? JSON.parse(body).prompt || JSON.parse(body).message || JSON.parse(body).input || 'no prompt' : 'no body'}`,
+      timestamp: new Date().toISOString(),
+      tls: {
+        cipher: req.socket.getCipher(),
+        protocol: req.socket.getProtocol(),
+        clientCert: req.socket.authorized ? 'verified' : 'none',
+      },
+    };
+
+    // For OpenAI-style responses
+    if (req.url.includes('/chat/completions')) {
+      response.choices = [
+        {
+          message: {
+            content: response.result,
+            role: 'assistant',
+          },
+          finish_reason: 'stop',
+        },
+      ];
+    }
+
+    res.writeHead(200, { 'Content-Type': 'application/json' });
+    res.end(JSON.stringify(response));
+  });
+});
+
+// Start server
+server.listen(PORT, () => {
+  console.log(`🚀 Mock HTTPS server running on https://localhost:${PORT}`);
+  console.log(`🔐 TLS Configuration:`);
+  console.log(`   - Protocol versions: TLS 1.2+`);
+  console.log(`   - Client certificates: ${REQUIRE_CLIENT_CERT ? 'Required' : 'Not required'}`);
+  console.log(`\n📝 Test with:`);
+  console.log(`   curl -k https://localhost:${PORT}/test`);
+  if (REQUIRE_CLIENT_CERT) {
+    console.log(
+      `   curl --cert certs/client-cert.pem --key certs/client-key.pem --cacert certs/ca-cert.pem https://localhost:${PORT}/test`,
+    );
+  }
+});
+
+// Handle errors
+server.on('tlsClientError', (err, socket) => {
+  console.error('❌ TLS Client Error:', err.message);
+});
+
+process.on('SIGINT', () => {
+  console.log('\n👋 Shutting down server...');
+  server.close(() => {
+    process.exit(0);
+  });
+});

examples/provider-http/tls/promptfooconfig-mock.yaml

@@ -0,0 +1,147 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Test TLS with mock server
+
+prompts:
+  - 'Tell me a fun fact about {{animal}}'
+
+providers:
+  # Test with self-signed certificate (dev mode)
+  # Expected to fail, the mock server expects a client certificate
+  - id: http
+    label: Mock Server (Basic)
+    config:
+      url: https://localhost:8443/api/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # Accept self-signed certificate for local testing
+        rejectUnauthorized: false
+      transformResponse: 'json.result'
+      maxRetries: 0
+
+  # Test with CA certificate verification
+  # Expected to fail, the mock server expects a client certificate
+  - id: http
+    label: Mock Server (CA Verified)
+    config:
+      url: https://localhost:8443/api/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # Verify server certificate with CA
+        caPath: ./certs/ca-cert.pem
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+  # Test with mutual TLS
+  # Expected to pass, the mock server requires a client certificate
+  - id: http
+    label: Mock Server (mTLS)
+    config:
+      url: https://localhost:8443/api/secure
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate for authentication
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+  # Test with mutual TLS using PFX format
+  # Expected to pass, using combined certificate/key file with passphrase
+  - id: http
+    label: Mock Server (mTLS with PFX)
+    config:
+      url: https://localhost:8443/api/secure
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate and key in PFX format
+        pfxPath: ./certs/client.pfx
+        passphrase: 'testpassword'
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+tests:
+  - vars:
+      animal: penguins
+    assert:
+      - type: contains-any
+        value:
+          - penguin
+          - Mock response
+      - type: javascript
+        value: |
+          // Check that output is not empty
+          if (!output || output.trim().length === 0) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'Response is empty'
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response is not empty'
+          };
+
+  - vars:
+      animal: dolphins
+    assert:
+      - type: javascript
+        value: |
+          // Check that we got a response
+          if (!output) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'No response received'
+            };
+          }
+          // Verify it contains expected mock response pattern
+          const containsMock = output.includes('Mock response');
+          return {
+            pass: containsMock,
+            score: containsMock ? 1 : 0,
+            reason: containsMock ? 'Got mock response' : 'Unexpected response format'
+          };
+
+  - vars:
+      animal: octopuses
+    assert:
+      - type: javascript
+        value: |
+          // Check minimum length of 10 characters
+          if (!output || output.length < 10) {
+            return {
+              pass: false,
+              score: 0,
+              reason: `Response too short (${output ? output.length : 0}/10 characters)`
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response meets minimum length requirement'
+          };

examples/provider-http/tls/promptfooconfig.yaml

@@ -0,0 +1,166 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: HTTP provider with TLS certificates
+
+prompts:
+  - 'Explain {{topic}} in simple terms'
+
+providers:
+  # Basic configuration with CA certificate
+  - id: http
+    label: HTTPS with CA cert
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ env.API_KEY }}
+      body:
+        model: gpt-4o-mini
+        messages:
+          - role: system
+            content: You are a helpful assistant that explains complex topics simply
+          - role: user
+            content: '{{prompt}}'
+        temperature: 0.7
+      tls:
+        # CA certificate for verifying server certificates
+        caPath: ./certs/ca-cert.pem
+        rejectUnauthorized: true
+      transformResponse: 'json.choices[0].message.content'
+      maxRetries: 0
+
+  # Mutual TLS (client certificate authentication)
+  - id: http
+    label: HTTPS with mTLS
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+        max_tokens: 150
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate for authentication
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        # Optional: Override server name for SNI
+        servername: secure-api.example.com
+      transformResponse: 'json.result'
+      maxRetries: 0
+
+  # PFX/PKCS12 certificate bundle
+  - id: http
+    label: HTTPS with PFX
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        X-API-Key: '{{ env.API_KEY }}'
+      body:
+        input: '{{prompt}}'
+      tls:
+        # PFX bundle containing cert and key
+        pfxPath: ./certs/client.pfx
+        passphrase: testpassword
+        rejectUnauthorized: true
+      transformResponse: 'json.output'
+      maxRetries: 0
+
+  # Development configuration (self-signed certificates)
+  - id: http
+    label: HTTPS dev mode
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # WARNING: Only for development!
+        rejectUnauthorized: false
+      transformResponse: 'json.response'
+      maxRetries: 0
+
+  # Advanced TLS configuration
+  - id: http
+    label: HTTPS with advanced TLS
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ env.API_KEY }}
+      body:
+        query: '{{prompt}}'
+      tls:
+        caPath: ./certs/ca-cert.pem
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        servername: high-security.example.com
+        # Restrict to strong ciphers
+        ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256'
+        # Enforce modern TLS versions
+        minVersion: 'TLSv1.2'
+        maxVersion: 'TLSv1.3'
+      transformResponse: 'json.data'
+      maxRetries: 0
+
+defaultTest:
+  assert:
+    - type: not-empty
+      value: 'Response should not be empty'
+
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: contains-any
+        value:
+          - quantum
+          - qubit
+          - superposition
+          - entanglement
+      - type: min-length
+        value: 50
+
+  - vars:
+      topic: blockchain technology
+    assert:
+      - type: contains-any
+        value:
+          - blockchain
+          - distributed
+          - ledger
+          - cryptocurrency
+      - type: javascript
+        value: |
+          if (!output || output.length < 50) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'Response too short'
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response is detailed enough'
+          };
+
+  - vars:
+      topic: machine learning
+    assert:
+      - type: llm-rubric
+        value: |
+          The explanation should:
+          1. Be understandable to a non-technical audience
+          2. Include at least one real-world example
+          3. Avoid excessive jargon
+          Rate the response on a scale of 0-1 for clarity and completeness