chore: import upstream snapshot with attribution

examples/provider-http/tls/.gitignore

@@ -0,0 +1,26 @@
+# Certificates and keys
+certs/
+*.pem
+*.pfx
+*.p12
+*.key
+*.crt
+*.cer
+*.csr
+
+# Environment files
+.env
+.env.local
+.env.*.local
+
+# Test outputs
+output/
+results/
+*.log
+
+# Node modules (if running mock server locally)
+node_modules/
+
+# OS files
+.DS_Store
+Thumbs.db 

examples/provider-http/tls/README.md

@@ -0,0 +1,181 @@
+# provider-http/tls (HTTP Provider with TLS Certificates)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-http/tls
+cd provider-http/tls
+```
+
+This example demonstrates how to configure the HTTP provider with TLS/SSL certificates for mutual TLS (mTLS) authentication.
+
+## Overview
+
+The HTTP provider supports multiple certificate formats for mutual TLS authentication:
+
+1. **PEM (Separate cert/key files)**: Traditional format with separate certificate and key files
+2. **PEM with Encrypted Key**: PEM format where the private key is password-protected
+3. **PFX/PKCS#12**: Combined certificate bundle format
+
+Each format can be provided via:
+
+- **File Path**: Reference files on disk
+- **Inline Content**: Embed certificate content directly (base64 for binary formats)
+- **Environment Variables**: Load from environment variables
+
+## PEM Certificate Options
+
+### Using Unencrypted PEM Files
+
+The simplest approach - separate certificate and key files:
+
+```yaml
+tls:
+  certPath: '/path/to/client-cert.pem'
+  keyPath: '/path/to/client-key.pem'
+```
+
+### Using Encrypted PEM Private Key
+
+When your private key is password-protected (starts with `BEGIN ENCRYPTED PRIVATE KEY`):
+
+```yaml
+tls:
+  certPath: '/path/to/client-cert.pem'
+  keyPath: '/path/to/client-key-encrypted.pem'
+  passphrase: 'your-key-password'
+```
+
+### Using Inline PEM Content
+
+Embed certificates directly in your configuration:
+
+```yaml
+tls:
+  cert: |
+    -----BEGIN CERTIFICATE-----
+    MIIDxTCCAq2gAwIBAgIJAL...
+    -----END CERTIFICATE-----
+  key: |
+    -----BEGIN PRIVATE KEY-----
+    MIIEvQIBADANBgkqhkiG9w0B...
+    -----END PRIVATE KEY-----
+```
+
+## PFX Certificate Options
+
+### Using File Path
+
+Reference a PFX file on the filesystem:
+
+```yaml
+tls:
+  pfxPath: '/path/to/certificate.pfx'
+  passphrase: 'your-passphrase'
+```
+
+### Using Inline Base64 Content
+
+Embed the certificate directly in your configuration:
+
+```yaml
+tls:
+  pfx: 'MIIJKQIBAzCCCO8GCSqGSIb3DQEHA...' # Base64-encoded PFX
+  passphrase: 'your-passphrase'
+```
+
+### Using Environment Variables
+
+Store sensitive certificates in environment variables:
+
+```yaml
+tls:
+  pfx: '{{env.PFX_CERTIFICATE_BASE64}}'
+  passphrase: '{{env.PFX_PASSPHRASE}}'
+```
+
+## Converting PFX to Base64
+
+To use inline PFX certificates, you need to convert your PFX file to base64:
+
+### Linux/Mac
+
+```bash
+base64 -i certificate.pfx -o certificate.b64
+```
+
+### Windows
+
+```cmd
+certutil -encode certificate.pfx certificate.b64
+```
+
+Then copy the content (excluding the BEGIN/END headers) to use as the `pfx` value.
+
+## Security Considerations
+
+1. **Never commit certificates to version control**: Use environment variables or external secret management
+2. **Protect your private keys**: Ensure PFX files have appropriate file permissions
+3. **Use strong passphrases**: Always protect PFX files with strong passphrases
+4. **Certificate validation**: Keep `rejectUnauthorized: true` in production
+
+## Running the Example
+
+1. Replace the sample certificate values with your actual certificates
+2. Set the required environment variables:
+   ```bash
+   export PFX_PASSPHRASE="your-passphrase"
+   export PFX_CERTIFICATE_BASE64="your-base64-cert"
+   ```
+3. Run the evaluation:
+   ```bash
+   promptfoo eval
+   ```
+
+## TLS Configuration Options
+
+| Option               | Description                                               |
+| -------------------- | --------------------------------------------------------- |
+| `cert`               | Inline PEM certificate content                            |
+| `certPath`           | Path to PEM certificate file                              |
+| `key`                | Inline PEM private key content                            |
+| `keyPath`            | Path to PEM private key file                              |
+| `pfx`                | Inline PFX certificate (base64-encoded string or Buffer)  |
+| `pfxPath`            | Path to PFX file on disk                                  |
+| `passphrase`         | Password for encrypted PEM private key or PFX certificate |
+| `ca`                 | CA certificate content for server verification            |
+| `caPath`             | Path to CA certificate file                               |
+| `rejectUnauthorized` | Verify server certificates (always `true` in production)  |
+| `minVersion`         | Minimum TLS version (e.g., 'TLSv1.2')                     |
+| `maxVersion`         | Maximum TLS version (e.g., 'TLSv1.3')                     |
+| `ciphers`            | Cipher suite specification                                |
+
+## Troubleshooting
+
+### Invalid PFX Format
+
+If you get an error about invalid PFX format:
+
+- Ensure the base64 encoding is correct
+- Verify the passphrase is correct
+- Check that the PFX file is not corrupted
+
+### Connection Refused
+
+If the connection is refused:
+
+- Verify the server requires client certificates
+- Ensure the certificate is valid and not expired
+- Check that the certificate is trusted by the server
+
+### Certificate Verification Failed
+
+If certificate verification fails:
+
+- Add the server's CA certificate using `ca` or `caPath`
+- For development only: set `rejectUnauthorized: false` (never in production)
+
+## Related Documentation
+
+- [HTTP Provider Documentation](https://promptfoo.com/docs/providers/http)
+- [TLS/HTTPS Configuration](https://promptfoo.com/docs/providers/http#tlshttps-configuration)

examples/provider-http/tls/env.example

@@ -0,0 +1,4 @@
+# HTTP Provider TLS Example Environment Variables
+# Mock server settings (for local testing)
+PORT=8443
+REQUIRE_CLIENT_CERT=true 

examples/provider-http/tls/generate-test-certs.sh

@@ -0,0 +1,129 @@
+#!/bin/bash
+
+# Generate self-signed certificates for testing TLS configurations
+# WARNING: These certificates are for testing only, not for production!
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${GREEN}🔐 Generating test certificates for TLS example${NC}"
+echo -e "${YELLOW}⚠️  WARNING: These certificates are for testing only!${NC}\n"
+
+# Create certs directory
+mkdir -p certs
+cd certs
+
+# Certificate configuration
+DAYS_VALID=365
+KEY_SIZE=2048
+COUNTRY="US"
+STATE="California"
+LOCALITY="San Francisco"
+ORG="Promptfoo Test"
+OU="Development"
+
+# Generate CA private key
+echo -e "${GREEN}1. Generating CA private key...${NC}"
+openssl genrsa -out ca-key.pem $KEY_SIZE
+
+# Generate CA certificate
+echo -e "${GREEN}2. Generating CA certificate...${NC}"
+openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days $DAYS_VALID \
+  -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG/OU=$OU/CN=Promptfoo Test CA"
+
+# Generate server private key
+echo -e "${GREEN}3. Generating server private key...${NC}"
+openssl genrsa -out server-key.pem $KEY_SIZE
+
+# Generate server certificate signing request
+echo -e "${GREEN}4. Generating server CSR...${NC}"
+cat >server.cnf <<EOF
+[req]
+distinguished_name = req_distinguished_name
+req_extensions = v3_req
+prompt = no
+
+[req_distinguished_name]
+C = $COUNTRY
+ST = $STATE
+L = $LOCALITY
+O = $ORG
+OU = $OU
+CN = localhost
+
+[v3_req]
+keyUsage = keyEncipherment, dataEncipherment
+extendedKeyUsage = serverAuth
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = localhost
+DNS.2 = *.localhost
+IP.1 = 127.0.0.1
+IP.2 = ::1
+EOF
+
+openssl req -new -key server-key.pem -out server.csr -config server.cnf
+
+# Sign server certificate with CA
+echo -e "${GREEN}5. Signing server certificate with CA...${NC}"
+openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
+  -CAcreateserial -out server-cert.pem -days $DAYS_VALID \
+  -extensions v3_req -extfile server.cnf
+
+# Generate client private key
+echo -e "${GREEN}6. Generating client private key...${NC}"
+openssl genrsa -out client-key.pem $KEY_SIZE
+
+# Generate client certificate signing request
+echo -e "${GREEN}7. Generating client CSR...${NC}"
+openssl req -new -key client-key.pem -out client.csr \
+  -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG/OU=$OU/CN=Test Client"
+
+# Sign client certificate with CA
+echo -e "${GREEN}8. Signing client certificate with CA...${NC}"
+openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem \
+  -CAcreateserial -out client-cert.pem -days $DAYS_VALID
+
+# Generate PFX bundle from client cert and key
+echo -e "${GREEN}9. Creating PFX bundle...${NC}"
+openssl pkcs12 -export -out client.pfx \
+  -inkey client-key.pem -in client-cert.pem \
+  -certfile ca-cert.pem -password pass:testpassword
+
+# Generate encrypted client private key (for testing passphrase with PEM)
+echo -e "${GREEN}10. Creating encrypted client private key...${NC}"
+openssl rsa -aes256 -in client-key.pem -out client-key-encrypted.pem \
+  -passout pass:testpassword
+
+# Clean up temporary files
+rm -f *.csr *.cnf *.srl
+
+# Display certificate information
+echo -e "\n${GREEN}✅ Certificates generated successfully!${NC}"
+echo -e "\nGenerated files:"
+echo "  📄 ca-cert.pem              - Certificate Authority certificate"
+echo "  🔑 ca-key.pem               - CA private key (keep secure!)"
+echo "  📄 server-cert.pem          - Server certificate"
+echo "  🔑 server-key.pem           - Server private key"
+echo "  📄 client-cert.pem          - Client certificate"
+echo "  🔑 client-key.pem           - Client private key (unencrypted)"
+echo "  🔑 client-key-encrypted.pem - Client private key (encrypted, password: testpassword)"
+echo "  📦 client.pfx               - Client certificate bundle (password: testpassword)"
+
+echo -e "\n${YELLOW}Certificate Details:${NC}"
+echo "CA Certificate:"
+openssl x509 -in ca-cert.pem -noout -subject -dates | sed 's/^/  /'
+echo -e "\nServer Certificate:"
+openssl x509 -in server-cert.pem -noout -subject -dates | sed 's/^/  /'
+echo -e "\nClient Certificate:"
+openssl x509 -in client-cert.pem -noout -subject -dates | sed 's/^/  /'
+
+echo -e "\n${GREEN}You can now test the TLS configuration with:${NC}"
+echo "  1. Start the mock server: node ../mock-server.js"
+echo "  2. Run the evaluation: npm run local -- eval -c ../promptfooconfig-mock.yaml"

examples/provider-http/tls/mock-server.js

@@ -0,0 +1,120 @@
+/**
+ * Mock HTTPS Server for testing TLS certificate configurations
+ *
+ * This server demonstrates:
+ * - HTTPS with self-signed certificates
+ * - Client certificate verification (mTLS)
+ * - Custom TLS options
+ */
+
+import fs from 'node:fs';
+import https from 'node:https';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+
+// Configuration
+const PORT = process.env.PORT || 8443;
+const REQUIRE_CLIENT_CERT = process.env.REQUIRE_CLIENT_CERT === 'true';
+
+// Server options
+const serverOptions = {
+  // Server certificate and key (self-signed for testing)
+  key: fs.readFileSync(path.join(__dirname, 'certs', 'server-key.pem')),
+  cert: fs.readFileSync(path.join(__dirname, 'certs', 'server-cert.pem')),
+};
+
+// Add client certificate verification for mTLS
+if (REQUIRE_CLIENT_CERT) {
+  serverOptions.requestCert = true;
+  serverOptions.rejectUnauthorized = true;
+  serverOptions.ca = fs.readFileSync(path.join(__dirname, 'certs', 'ca-cert.pem'));
+  console.log('📋 Client certificate verification enabled (mTLS)');
+}
+
+// Create HTTPS server
+const server = https.createServer(serverOptions, (req, res) => {
+  let body = '';
+
+  req.on('data', (chunk) => {
+    body += chunk.toString();
+  });
+
+  req.on('end', () => {
+    console.log(`📥 ${req.method} ${req.url}`);
+    console.log('Headers:', req.headers);
+
+    if (body) {
+      try {
+        const parsed = JSON.parse(body);
+        console.log('Body:', parsed);
+      } catch {
+        console.log('Body (raw):', body);
+      }
+    }
+
+    // Check for client certificate
+    if (req.socket.getPeerCertificate) {
+      const cert = req.socket.getPeerCertificate();
+      if (cert && cert.subject) {
+        console.log('✅ Client certificate:', cert.subject);
+      }
+    }
+
+    // Simulate API response
+    const response = {
+      result: `Mock response for: ${body ? JSON.parse(body).prompt || JSON.parse(body).message || JSON.parse(body).input || 'no prompt' : 'no body'}`,
+      timestamp: new Date().toISOString(),
+      tls: {
+        cipher: req.socket.getCipher(),
+        protocol: req.socket.getProtocol(),
+        clientCert: req.socket.authorized ? 'verified' : 'none',
+      },
+    };
+
+    // For OpenAI-style responses
+    if (req.url.includes('/chat/completions')) {
+      response.choices = [
+        {
+          message: {
+            content: response.result,
+            role: 'assistant',
+          },
+          finish_reason: 'stop',
+        },
+      ];
+    }
+
+    res.writeHead(200, { 'Content-Type': 'application/json' });
+    res.end(JSON.stringify(response));
+  });
+});
+
+// Start server
+server.listen(PORT, () => {
+  console.log(`🚀 Mock HTTPS server running on https://localhost:${PORT}`);
+  console.log(`🔐 TLS Configuration:`);
+  console.log(`   - Protocol versions: TLS 1.2+`);
+  console.log(`   - Client certificates: ${REQUIRE_CLIENT_CERT ? 'Required' : 'Not required'}`);
+  console.log(`\n📝 Test with:`);
+  console.log(`   curl -k https://localhost:${PORT}/test`);
+  if (REQUIRE_CLIENT_CERT) {
+    console.log(
+      `   curl --cert certs/client-cert.pem --key certs/client-key.pem --cacert certs/ca-cert.pem https://localhost:${PORT}/test`,
+    );
+  }
+});
+
+// Handle errors
+server.on('tlsClientError', (err, socket) => {
+  console.error('❌ TLS Client Error:', err.message);
+});
+
+process.on('SIGINT', () => {
+  console.log('\n👋 Shutting down server...');
+  server.close(() => {
+    process.exit(0);
+  });
+});

examples/provider-http/tls/promptfooconfig-mock.yaml

@@ -0,0 +1,147 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Test TLS with mock server
+
+prompts:
+  - 'Tell me a fun fact about {{animal}}'
+
+providers:
+  # Test with self-signed certificate (dev mode)
+  # Expected to fail, the mock server expects a client certificate
+  - id: http
+    label: Mock Server (Basic)
+    config:
+      url: https://localhost:8443/api/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # Accept self-signed certificate for local testing
+        rejectUnauthorized: false
+      transformResponse: 'json.result'
+      maxRetries: 0
+
+  # Test with CA certificate verification
+  # Expected to fail, the mock server expects a client certificate
+  - id: http
+    label: Mock Server (CA Verified)
+    config:
+      url: https://localhost:8443/api/chat
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # Verify server certificate with CA
+        caPath: ./certs/ca-cert.pem
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+  # Test with mutual TLS
+  # Expected to pass, the mock server requires a client certificate
+  - id: http
+    label: Mock Server (mTLS)
+    config:
+      url: https://localhost:8443/api/secure
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate for authentication
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+  # Test with mutual TLS using PFX format
+  # Expected to pass, using combined certificate/key file with passphrase
+  - id: http
+    label: Mock Server (mTLS with PFX)
+    config:
+      url: https://localhost:8443/api/secure
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate and key in PFX format
+        pfxPath: ./certs/client.pfx
+        passphrase: 'testpassword'
+        rejectUnauthorized: true
+        servername: localhost
+      transformResponse: 'json.result'
+      maxRetries: 0
+tests:
+  - vars:
+      animal: penguins
+    assert:
+      - type: contains-any
+        value:
+          - penguin
+          - Mock response
+      - type: javascript
+        value: |
+          // Check that output is not empty
+          if (!output || output.trim().length === 0) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'Response is empty'
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response is not empty'
+          };
+
+  - vars:
+      animal: dolphins
+    assert:
+      - type: javascript
+        value: |
+          // Check that we got a response
+          if (!output) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'No response received'
+            };
+          }
+          // Verify it contains expected mock response pattern
+          const containsMock = output.includes('Mock response');
+          return {
+            pass: containsMock,
+            score: containsMock ? 1 : 0,
+            reason: containsMock ? 'Got mock response' : 'Unexpected response format'
+          };
+
+  - vars:
+      animal: octopuses
+    assert:
+      - type: javascript
+        value: |
+          // Check minimum length of 10 characters
+          if (!output || output.length < 10) {
+            return {
+              pass: false,
+              score: 0,
+              reason: `Response too short (${output ? output.length : 0}/10 characters)`
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response meets minimum length requirement'
+          };

examples/provider-http/tls/promptfooconfig.yaml

@@ -0,0 +1,166 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: HTTP provider with TLS certificates
+
+prompts:
+  - 'Explain {{topic}} in simple terms'
+
+providers:
+  # Basic configuration with CA certificate
+  - id: http
+    label: HTTPS with CA cert
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ env.API_KEY }}
+      body:
+        model: gpt-4o-mini
+        messages:
+          - role: system
+            content: You are a helpful assistant that explains complex topics simply
+          - role: user
+            content: '{{prompt}}'
+        temperature: 0.7
+      tls:
+        # CA certificate for verifying server certificates
+        caPath: ./certs/ca-cert.pem
+        rejectUnauthorized: true
+      transformResponse: 'json.choices[0].message.content'
+      maxRetries: 0
+
+  # Mutual TLS (client certificate authentication)
+  - id: http
+    label: HTTPS with mTLS
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        prompt: '{{prompt}}'
+        max_tokens: 150
+      tls:
+        # CA for verifying server
+        caPath: ./certs/ca-cert.pem
+        # Client certificate for authentication
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        # Optional: Override server name for SNI
+        servername: secure-api.example.com
+      transformResponse: 'json.result'
+      maxRetries: 0
+
+  # PFX/PKCS12 certificate bundle
+  - id: http
+    label: HTTPS with PFX
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        X-API-Key: '{{ env.API_KEY }}'
+      body:
+        input: '{{prompt}}'
+      tls:
+        # PFX bundle containing cert and key
+        pfxPath: ./certs/client.pfx
+        passphrase: testpassword
+        rejectUnauthorized: true
+      transformResponse: 'json.output'
+      maxRetries: 0
+
+  # Development configuration (self-signed certificates)
+  - id: http
+    label: HTTPS dev mode
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+      body:
+        message: '{{prompt}}'
+      tls:
+        # WARNING: Only for development!
+        rejectUnauthorized: false
+      transformResponse: 'json.response'
+      maxRetries: 0
+
+  # Advanced TLS configuration
+  - id: http
+    label: HTTPS with advanced TLS
+    config:
+      url: https://localhost:8443/chat/completions
+      method: POST
+      headers:
+        Content-Type: application/json
+        Authorization: Bearer {{ env.API_KEY }}
+      body:
+        query: '{{prompt}}'
+      tls:
+        caPath: ./certs/ca-cert.pem
+        certPath: ./certs/client-cert.pem
+        keyPath: ./certs/client-key.pem
+        rejectUnauthorized: true
+        servername: high-security.example.com
+        # Restrict to strong ciphers
+        ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256'
+        # Enforce modern TLS versions
+        minVersion: 'TLSv1.2'
+        maxVersion: 'TLSv1.3'
+      transformResponse: 'json.data'
+      maxRetries: 0
+
+defaultTest:
+  assert:
+    - type: not-empty
+      value: 'Response should not be empty'
+
+tests:
+  - vars:
+      topic: quantum computing
+    assert:
+      - type: contains-any
+        value:
+          - quantum
+          - qubit
+          - superposition
+          - entanglement
+      - type: min-length
+        value: 50
+
+  - vars:
+      topic: blockchain technology
+    assert:
+      - type: contains-any
+        value:
+          - blockchain
+          - distributed
+          - ledger
+          - cryptocurrency
+      - type: javascript
+        value: |
+          if (!output || output.length < 50) {
+            return {
+              pass: false,
+              score: 0,
+              reason: 'Response too short'
+            };
+          }
+          return {
+            pass: true,
+            score: 1,
+            reason: 'Response is detailed enough'
+          };
+
+  - vars:
+      topic: machine learning
+    assert:
+      - type: llm-rubric
+        value: |
+          The explanation should:
+          1. Be understandable to a non-technical audience
+          2. Include at least one real-world example
+          3. Avoid excessive jargon
+          Rate the response on a scale of 0-1 for clarity and completeness