chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,26 @@
+# eval-self-grading (Self Grading)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example eval-self-grading
+cd eval-self-grading
+```
+
+## Usage
+
+This example shows how you can have an LLM grade its own output according to predefined expectations.
+
+The configuration is provided in `promptfooconfig.yaml`.
+
+Run:
+
+```bash
+promptfoo eval
+```
+
+You can also define the tests in a CSV file:
+
+```bash
+promptfoo eval --tests tests.csv
+```
@@ -0,0 +1,21 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Automatic response evaluation using LLM rubric scoring
+
+prompts:
+  - file://prompts.txt
+providers:
+  - openai:chat:gpt-5.4
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: Do not mention that you are an AI or chat assistant
+    - type: javascript
+      # Shorter is better
+      value: Math.max(0, Math.min(1, 1 - (output.length - 100) / 900));
+tests:
+  - vars:
+      name: Bob
+      question: Can you help me find a specific product on your website?
+  - vars:
+      name: Jane
+      question: Do you have any promotions or discounts currently available?
@@ -0,0 +1,5 @@
+You're an ecommerce chat assistant for a shoe company.
+Answer this user's question: {{name}}: "{{question}}"
+---
+You're a smart, bubbly customer service rep for a shoe company.
+Answer this user's question concisely: {{name}}: "{{question}}"
@@ -0,0 +1,11 @@
+name,question,__expected
+Bob,Can you help me find a specific product on your website?,grade:Do not mention that you are an AI or chat assistant
+Jane,Do you have any promotions or discounts currently available?,grade:Do not mention that you are an AI or chat assistant
+Dave,What are your shipping and return policies?,grade:Do not mention that you are an AI or chat assistant
+Jim,Can you provide more information about the product specifications or features?,grade:Do not mention that you are an AI or chat assistant
+Alice,Can you recommend products that are similar to what I've been looking at?,grade:Do not mention that you are an AI or chat assistant
+Sophie,Do you have any recommendations for products that are currently popular or trending?,grade:Do not mention that you are an AI or chat assistant
+Ben,Can you check the availability of a product at a specific store location?,grade:Do not mention that you are an AI or chat assistant
+Jessie,How can I track my order after it has been shipped?,grade:Do not mention that you are an AI or chat assistant
+Kim,What payment methods do you accept?,grade:Do not mention that you are an AI or chat assistant
+Emily,Can you help me with a problem I'm having with my account or order?,grade:Do not mention that you are an AI or chat assistant