chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,62 @@
+# compare-openai-models (OpenAI Model Comparison)
+
+This example compares OpenAI's `gpt-5.4` with `gpt-5.4-mini` across various riddles and reasoning tasks.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example compare-openai-models
+cd compare-openai-models
+```
+
+## Quick Start
+
+1. Initialize this example by running:
+
+   ```bash
+   npx promptfoo@latest init --example compare-openai-models
+   ```
+
+2. Navigate to the newly created `compare-openai-models` directory:
+
+   ```bash
+   cd compare-openai-models
+   ```
+
+3. Set an OpenAI API key directly in your environment:
+
+   ```bash
+   export OPENAI_API_KEY="your_openai_api_key"
+   ```
+
+   Alternatively, you can set the API key in a `.env` file:
+
+   ```bash
+   OPENAI_API_KEY=your_openai_api_key
+   ```
+
+4. Run the evaluation with:
+
+   ```bash
+   npx promptfoo@latest eval --no-cache
+   ```
+
+   Note: the `--no-cache` flag is required because the example uses a [latency assertion](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#latency) which does not support caching.
+
+5. View the results:
+
+   ```bash
+   npx promptfoo@latest view
+   ```
+
+   The expected output will include the responses from both models for the provided riddles, allowing you to compare their performance side by side.
+
+## What this example demonstrates
+
+This example compares OpenAI's GPT-5.4 with GPT-5.4 Mini across various riddles and puzzles. It demonstrates:
+
+- **Model comparison**: Side-by-side evaluation of `gpt-5.4` vs `gpt-5.4-mini`
+- **Cost and latency assertions**: Ensuring responses meet performance thresholds
+- **Content validation**: Using `contains` assertions to verify specific answers
+- **LLM-based grading**: Using `llm-rubric` assertions for nuanced evaluation criteria
+- **Diverse test cases**: A variety of riddles testing different reasoning capabilities
@@ -0,0 +1,79 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Comparing OpenAI flagship models performance on riddles
+
+prompts:
+  - 'Solve this riddle: {{riddle}}'
+
+providers:
+  - openai:chat:gpt-5.4
+  - openai:chat:gpt-5.4-mini
+
+defaultTest:
+  assert:
+    # Inference should always cost less than this (USD)
+    - type: cost
+      threshold: 0.01
+    # Inference should always be faster than this (milliseconds)
+    - type: latency
+      threshold: 10000
+
+tests:
+  - vars:
+      riddle: 'I speak without a mouth and hear without ears. I have no body, but I come alive with wind. What am I?'
+    assert:
+      # Make sure the LLM output contains this word
+      - type: contains
+        value: echo
+      # Use model-graded assertions to enforce free-form instructions
+      - type: llm-rubric
+        value: Do not apologize
+  - vars:
+      riddle: "You see a boat filled with people. It has not sunk, but when you look again you don't see a single person on the boat. Why?"
+    assert:
+      - type: llm-rubric
+        value: explains that there are no single people (they are all married)
+  - vars:
+      riddle: 'The more of this there is, the less you see. What is it?'
+    assert:
+      - type: contains
+        value: darkness
+  - vars:
+      riddle: >-
+        I have keys but no locks. I have space but no room. You can enter, but
+        can't go outside. What am I?
+  - vars:
+      riddle: >-
+        I am not alive, but I grow; I don't have lungs, but I need air; I don't
+        have a mouth, but water kills me. What am I?
+  - vars:
+      riddle: What can travel around the world while staying in a corner?
+  - vars:
+      riddle: Forward I am heavy, but backward I am not. What am I?
+  - vars:
+      riddle: >-
+        The person who makes it, sells it. The person who buys it, never uses
+        it. The person who uses it, doesn't know they're using it. What is it?
+  - vars:
+      riddle: I can be cracked, made, told, and played. What am I?
+  - vars:
+      riddle: What has keys but can't open locks?
+  - vars:
+      riddle: >-
+        I'm light as a feather, yet the strongest person can't hold me for much
+        more than a minute. What am I?
+  - vars:
+      riddle: >-
+        I can fly without wings, I can cry without eyes. Whenever I go, darkness
+        follows me. What am I?
+  - vars:
+      riddle: >-
+        I am taken from a mine, and shut up in a wooden case, from which I am
+        never released, and yet I am used by almost every person. What am I?
+  - vars:
+      riddle: >-
+        David's father has three sons: Snap, Crackle, and _____? What is the
+        name of the third son?
+  - vars:
+      riddle: >-
+        I am light as a feather, but even the world's strongest man couldn't
+        hold me for much longer than a minute. What am I?