chore: import upstream snapshot with attribution

examples/config-websockets/streaming/README.md

@@ -0,0 +1,125 @@
+# config-websockets/streaming (WebSocket Streaming)
+
+This example shows how to configure a websocket application that streams its responses. It includes a small Node.js server that exposes two WebSocket endpoints:
+
+- A non-streaming endpoint (`/ws`) that returns a single message when the model finishes.
+- A streaming endpoint (`/ws-stream`) that sends incremental deltas and a final message.
+
+You’ll run the server locally and use promptfoo’s eval command to test the quality of the application.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example config-websockets/streaming
+cd config-websockets/streaming
+```
+
+## What’s in this folder
+
+- `promptfooconfig.yaml` – Configures a target pointing at the local WebSocket server using the streaming endpoint
+- `server/` – Minimal Express + WebSocket server that calls the OpenAI Responses API and exposes the two endpoints
+
+## Prerequisites
+
+- Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends July 30, 2026; Node.js 24 LTS recommended)
+- An OpenAI API key set as `OPENAI_API_KEY`
+
+## 1) Start the local WebSocket server
+
+From this directory:
+
+```bash
+cd server
+npm install
+
+# Option A: set environment variables in your shell
+export OPENAI_API_KEY=your_key_here
+# Optional:
+# export CHATBOT_MODEL=gpt-4.1-mini  # defaults to gpt-4.1-mini
+# export PORT=3300                   # defaults to 3300
+
+# Start the server
+npm start
+```
+
+You should see the server listening at `http://localhost:3300`.
+
+Health check:
+
+```bash
+curl http://localhost:3300/health
+# {"status":"ok"}
+```
+
+WebSocket Endpoints:
+
+- `ws://localhost:3300/ws` – non-streaming
+- `ws://localhost:3300/ws-stream` – streaming (sends `delta` updates and a final `message`)
+
+## 2) How the WebSocket configuration works
+
+In `promptfooconfig.yaml`, the websocket endpoint is configured under the websocket endpoint id:
+
+```yaml
+- id: 'ws://localhost:3300/ws-stream'
+```
+
+The target configuration uses the streamResponse function `streamResponse(accumulator, data, context?)` to decide when to stop and what to return.
+
+## Server Response Format
+
+The server three types of messages:
+
+1. `delta` messages that include a partial response
+2. `message` messages that include the finalized response in full
+3. `error` messages that indicate an error occurred
+
+Example of a successful message stream:
+
+```json
+{"type":"delta","message":"Part of a thought"}
+{"type":"message","message":"Part of a thought, now the thought is completed"}
+```
+
+The streamResponse function includes logic for handling these different cases. Note: the `delta` case is the fallback, which returns false for the second item in the tuple to indicate the response is not yet complete:
+
+```yaml
+- id: 'ws://localhost:3300/ws-stream'
+  config:
+    messageTemplate: '{"input": {{prompt | dump}}}'
+    streamResponse: |
+      (accumulator, event, context) => {
+        const { message, type } = JSON.parse(event.data);
+        if (type === 'message') { return [{ output: message }, true]; }
+        if (type === 'error')   { return [{ error: message }, true]; }
+        return [{output: message}, false];
+      }
+```
+
+Tip: If you need to concatenate partials for UX, you can return an accumulator object with the concatenated value on `delta` frames and only return `true` when you receive the final message.
+
+## 3) Run the evaluation
+
+With the server running, open a new terminal at this example directory and run:
+
+```bash
+promptfoo eval
+```
+
+This will evaluate the test cases against the streaming WebSocket endpoint.
+
+View results in the browser UI:
+
+```bash
+promptfoo view
+```
+
+## Troubleshooting
+
+- If requests fail immediately, ensure `OPENAI_API_KEY` is set in the environment where the server is running.
+- If the client can’t connect, verify the server is listening on the expected port (`PORT`, defaults to 3300) and that you’re using the correct `ws://` URL.
+- For streaming behavior, watch the server logs and confirm you’re receiving `delta` events followed by a final `message`.
+
+## Cleanup
+
+Stop the server with `Ctrl+C` in its terminal.

examples/config-websockets/streaming/promptfooconfig.yaml

@@ -0,0 +1,31 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Websocket target with streaming response'
+
+prompts:
+  - 'Rephrase this in {{language}}: {{body}}'
+  - 'Translate this to conversational {{language}}: {{body}}'
+
+targets:
+  - id: 'ws://localhost:3300/ws-stream'
+    config:
+      messageTemplate: '{"input": {{prompt | dump}}}'
+      streamResponse: |-
+        (accumulator, event, context) => {
+          const { message, type } = JSON.parse(event.data);
+          if(type === "message"){return [{output: message},true]}
+          if(type === "error"){return [{error: message},true]}
+          return [{output: message}, false];}
+
+tests:
+  - vars:
+      language: French
+      body: Hello world
+  - vars:
+      language: French
+      body: I'm hungry
+  - vars:
+      language: Pirate
+      body: Hello world
+  - vars:
+      language: Pirate
+      body: I'm hungry

examples/config-websockets/streaming/server/README.md

@@ -0,0 +1,101 @@
+# config-websockets/streaming/server (OpenAI WebSocket Server)
+
+Simple Node.js server using Express and native WebSockets that exposes two real-time endpoints to interact with the OpenAI Responses API.
+
+## Requirements
+
+- Node.js >= 18.17
+- An OpenAI API key
+
+## Setup
+
+1. Install dependencies:
+
+```bash
+npm install
+```
+
+2. Configure environment:
+
+- set `OPENAI_API_KEY` in your environment or use .env file
+
+```bash
+cp env.example .env
+# edit .env and set OPENAI_API_KEY
+```
+
+3. Start the server (defaults to port 3300):
+
+```bash
+npm start
+```
+
+You can also run in dev mode with automatic restarts:
+
+```bash
+npm run dev
+```
+
+## HTTP
+
+- Health check: `GET /health` → `{ "status": "ok" }`
+
+## Real-time (WebSocket)
+
+Two WebSocket upgrade paths are provided:
+
+- `/ws` — non-streaming. Emits a single `response` when the OpenAI request completes.
+- `/ws-stream` — streaming. Emits incremental `delta` and `message` events, then `done`.
+
+Both endpoints accept the same request payload (model is configured via env):
+
+```json
+{ "input": "Hello there!" }
+```
+
+Model is read from `CHATBOT_MODEL` env var and defaults to `gpt-4.1-mini`.
+
+### Client examples
+
+Non-streaming (`/ws`):
+
+```js
+const ws = new WebSocket('ws://localhost:3300/ws');
+
+ws.onopen = () => {
+  ws.send(JSON.stringify({ input: 'Hello there!' }));
+};
+
+ws.onmessage = (event) => {
+  const msg = JSON.parse(event.data);
+  // msg.type: 'ready' | 'response' | 'done' | 'error'
+  console.log(msg);
+};
+
+ws.onerror = (err) => console.error('ws error', err);
+```
+
+Streaming (`/ws-stream`):
+
+```js
+const ws = new WebSocket('ws://localhost:3300/ws-stream');
+
+ws.onopen = () => {
+  ws.send(JSON.stringify({ input: 'Stream this please' }));
+};
+
+ws.onmessage = (event) => {
+  const msg = JSON.parse(event.data);
+  // msg.type: 'ready' | 'delta' | 'message' | 'done' | 'error'
+  if (msg.type === 'delta') process.stdout.write(msg.message || '');
+  else console.log(msg);
+};
+
+ws.onerror = (err) => console.error('ws error', err);
+```
+
+## Notes
+
+- Configure port via `PORT` in `.env`.
+- Configure model via `CHATBOT_MODEL` in `.env` (default: `gpt-4.1-mini`).
+- Health route is `GET /health`. WebSocket upgrade paths are `/ws` and `/ws-stream`.

examples/config-websockets/streaming/server/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "openai-ws-server",
+  "version": "0.1.0",
+  "license": "MIT",
+  "private": true,
+  "type": "module",
+  "description": "Express server with WebSocket endpoint that calls OpenAI Responses API",
+  "engines": {
+    "node": ">=18.20.8"
+  },
+  "scripts": {
+    "start": "node server.js",
+    "dev": "nodemon --watch server.js server.js",
+    "lint": "echo \"No linter configured\""
+  },
+  "dependencies": {
+    "cors": "^2.8.6",
+    "dotenv": "^17.2.3",
+    "express": "^5.2.1",
+    "openai": "^6.37.0",
+    "ws": "^8.19.0"
+  },
+  "devDependencies": {
+    "nodemon": "^3.1.11"
+  }
+}

examples/config-websockets/streaming/server/server.js

@@ -0,0 +1,196 @@
+import http from 'http';
+
+import cors from 'cors';
+import dotenv from 'dotenv';
+import express from 'express';
+import OpenAI from 'openai';
+import WebSocket, { WebSocketServer } from 'ws';
+
+dotenv.config();
+
+function validateEnvironmentVariables() {
+  const missing = [];
+  if (!process.env.OPENAI_API_KEY) {
+    missing.push('OPENAI_API_KEY');
+  }
+  if (missing.length > 0) {
+    // eslint-disable-next-line no-console
+    console.error(`Missing required environment variables: ${missing.join(', ')}`);
+    process.exit(1);
+  }
+}
+
+validateEnvironmentVariables();
+
+const PORT = Number(process.env.PORT || 3300);
+const app = express();
+const server = http.createServer(app);
+const wss = new WebSocketServer({ noServer: true });
+const wssStream = new WebSocketServer({ noServer: true });
+const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+const DEFAULT_MODEL = 'gpt-4.1-mini';
+const CHATBOT_MODEL =
+  process.env.CHATBOT_MODEL && process.env.CHATBOT_MODEL.trim().length > 0
+    ? process.env.CHATBOT_MODEL
+    : DEFAULT_MODEL;
+
+app.use(cors());
+app.use(express.json());
+
+app.get('/health', (_req, res) => {
+  res.json({ status: 'ok' });
+});
+
+// WebSocket: /ws (non-streaming)
+wss.on('connection', (ws, request) => {
+  const clientAddress = request.socket.remoteAddress;
+  // eslint-disable-next-line no-console
+  console.log(`WS client connected to /ws${clientAddress ? `: ${clientAddress}` : ''}`);
+
+  ws.on('message', async (raw) => {
+    let payload;
+    try {
+      payload = JSON.parse(raw.toString());
+    } catch {
+      ws.send(JSON.stringify({ type: 'error', error: 'Invalid JSON payload' }));
+      return;
+    }
+
+    const { input } = payload || {};
+    if (typeof input !== 'string' || input.trim().length === 0) {
+      ws.send(
+        JSON.stringify({
+          type: 'error',
+          error: 'Missing "input" (non-empty string) in payload',
+        }),
+      );
+      return;
+    }
+
+    try {
+      const response = await openai.responses.create({
+        model: CHATBOT_MODEL,
+        input,
+      });
+      const text = response.output[0].content[0].text;
+      ws.send(JSON.stringify({ type: 'message', message: text, raw: response }));
+    } catch (err) {
+      console.error(err);
+      const message = err && err.message ? err.message : 'OpenAI request failed';
+      ws.send(JSON.stringify({ type: 'error', error: message }));
+    }
+  });
+
+  ws.on('close', () => {
+    // eslint-disable-next-line no-console
+    console.log('WS client disconnected from /ws');
+  });
+});
+
+// WebSocket: /ws-stream (streaming)
+wssStream.on('connection', (ws, request) => {
+  const clientAddress = request.socket.remoteAddress;
+  // eslint-disable-next-line no-console
+  console.log(`WS client connected to /ws-stream${clientAddress ? `: ${clientAddress}` : ''}`);
+
+  if (ws.readyState === WebSocket.OPEN) {
+    ws.send(JSON.stringify({ type: 'ready', message: 'Connected to /ws-stream' }));
+  }
+
+  ws.on('message', async (raw) => {
+    let payload;
+    try {
+      payload = JSON.parse(raw.toString());
+    } catch {
+      console.error('Invalid JSON payload', raw);
+      ws.send(JSON.stringify({ type: 'error', message: 'Invalid JSON payload' }));
+      return;
+    }
+
+    const { input } = payload || {};
+    if (typeof input !== 'string' || input.trim().length === 0) {
+      ws.send(
+        JSON.stringify({
+          type: 'error',
+          error: 'Missing "input" (non-empty string) in payload',
+        }),
+      );
+      return;
+    }
+
+    try {
+      const stream = await openai.responses.stream(
+        {
+          model: CHATBOT_MODEL,
+          input,
+        },
+        { stream: true },
+      );
+
+      stream.on('response.output_text.delta', (delta) => {
+        if (ws.readyState === WebSocket.OPEN) {
+          ws.send(JSON.stringify({ type: 'delta', message: delta.delta }));
+        }
+      });
+
+      stream.on('response.output_text.done', ({ text }) => {
+        if (ws.readyState === WebSocket.OPEN) {
+          ws.send(JSON.stringify({ type: 'message', message: text }));
+        }
+      });
+
+      stream.on('error', (err) => {
+        if (ws.readyState === WebSocket.OPEN) {
+          ws.send(
+            JSON.stringify({
+              type: 'error',
+              message: err && err.message ? err.message : 'Unknown streaming error',
+            }),
+          );
+        }
+      });
+
+      await stream.done();
+      // Debug: mark stream completion
+      if (ws.readyState === WebSocket.OPEN) {
+        ws.send(JSON.stringify({ type: 'done' }));
+      }
+    } catch (err) {
+      const message = err && err.message ? err.message : 'OpenAI request failed';
+      if (ws.readyState === WebSocket.OPEN) {
+        ws.send(JSON.stringify({ type: 'error', message: message }));
+      }
+    }
+  });
+
+  ws.on('close', () => {
+    // eslint-disable-next-line no-console
+    console.log('WS client disconnected from /ws-stream');
+  });
+});
+
+server.on('upgrade', (request, socket, head) => {
+  const url = request.url || '';
+  if (url === '/ws' || url.startsWith('/ws?')) {
+    wss.handleUpgrade(request, socket, head, (ws) => {
+      wss.emit('connection', ws, request);
+    });
+  } else if (url === '/ws-stream' || url.startsWith('/ws-stream?')) {
+    wssStream.handleUpgrade(request, socket, head, (ws) => {
+      wssStream.emit('connection', ws, request);
+    });
+  } else {
+    socket.destroy();
+  }
+});
+
+server.listen(PORT, () => {
+  // eslint-disable-next-line no-console
+  console.log(`Server listening on http://localhost:${PORT}`);
+});
+
+process.on('SIGINT', () => {
+  // eslint-disable-next-line no-console
+  console.log('\nGracefully shutting down...');
+  server.close(() => process.exit(0));
+});