chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,89 @@
+# openai-chatkit (OpenAI ChatKit Workflow Evaluation)
+
+Evaluate ChatKit workflows built with OpenAI's Agent Builder using browser automation.
+
+## Quick Start
+
+```bash
+npx promptfoo@latest init --example openai-chatkit
+cd openai-chatkit
+```
+
+## Prerequisites
+
+1. Install Playwright browser:
+
+```bash
+npx playwright install chromium
+```
+
+2. Get your workflow ID from Agent Builder:
+   - Open [platform.openai.com](https://platform.openai.com) > Agent Builder
+   - Click **Publish** on your workflow
+   - Copy the workflow ID (e.g., `wf_68ffb83dbfc88190a38103c2bb9f421003f913035dbdb131`)
+
+3. Update `promptfooconfig.yaml` with your workflow ID
+
+## Environment Variables
+
+- `OPENAI_API_KEY` - Your OpenAI API key
+
+## Run the Eval
+
+```bash
+npx promptfoo@latest eval --max-concurrency 4
+```
+
+View results:
+
+```bash
+npx promptfoo@latest view
+```
+
+## Multi-Turn Conversations
+
+For workflows that ask follow-up questions, enable stateful mode:
+
+```yaml
+providers:
+  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
+    config:
+      stateful: true
+
+tests:
+  - vars:
+      message: 'I want to plan a party'
+  - vars:
+      message: 'For 20 people with a $500 budget'
+```
+
+Run with `--max-concurrency 1` for stateful mode.
+
+## Using with Simulated User
+
+Test multi-turn conversations automatically:
+
+```yaml
+providers:
+  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
+    config:
+      stateful: true
+
+defaultTest:
+  provider:
+    id: 'promptfoo:simulated-user'
+    config:
+      maxTurns: 5
+
+tests:
+  - vars:
+      instructions: |
+        You are planning a birthday party.
+        Answer questions naturally and provide details when asked.
+```
+
+## Learn More
+
+- [ChatKit Provider Documentation](https://www.promptfoo.dev/docs/providers/openai-chatkit/)
+- [Simulated User Provider](https://www.promptfoo.dev/docs/providers/simulated-user/)
+- [OpenAI Agent Builder](https://platform.openai.com/docs/guides/chatkit)
@@ -0,0 +1,35 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Evaluate ChatKit workflows from OpenAI Agent Builder
+
+prompts:
+  - '{{message}}'
+
+providers:
+  # Replace with your workflow ID from Agent Builder
+  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID_HERE
+    label: my-workflow
+    config:
+      timeout: 120000
+      poolSize: 4
+
+tests:
+  - description: Basic greeting
+    vars:
+      message: Hello, how can you help me today?
+    assert:
+      - type: llm-rubric
+        value: Response is helpful and relevant
+
+  - description: Task request
+    vars:
+      message: I need help planning a project
+    assert:
+      - type: contains-any
+        value: ['help', 'plan', 'project', 'assist']
+
+  - description: Question handling
+    vars:
+      message: What information do you need from me?
+    assert:
+      - type: llm-rubric
+        value: Response asks clarifying questions or explains capabilities