chore: import upstream snapshot with attribution

2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,27 @@
+# provider-fireworks (Fireworks AI)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example provider-fireworks
+cd provider-fireworks
+```
+
+## Usage
+
+Set your `FIREWORKS_API_KEY` environment variable. You can get a key from [fireworks.ai](https://fireworks.ai), sign in, open **Settings -> API Keys**, and create one.
+
+Then run:
+
+```bash
+promptfoo eval
+```
+
+View the results with `promptfoo view`.
+
+## What this shows
+
+- Three Fireworks **serverless** chat models (`gpt-oss-120b`, `deepseek-v4-pro`, `kimi-k2p6`) compared on a summarisation task. These are reasoning models, so `max_tokens` is set high enough to leave room for hidden reasoning tokens plus the visible answer.
+- A `similar` assertion graded by a Fireworks **embedding** model via the `fireworks:embedding:` prefix.
+
+Models rotate in and out of the serverless tier — if a model 404s, pick a current one from the [serverless catalogue](https://fireworks.ai/models?deployment=serverless).
@@ -0,0 +1,62 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Fireworks AI Provider Demo'
+
+prompts:
+  - 'In one sentence, summarise: {{passage}}'
+
+# These are Fireworks serverless models, so a `FIREWORKS_API_KEY` is all you
+# need to run the example. The current serverless catalogue lives at
+# https://fireworks.ai/models?deployment=serverless — swap in any other
+# serverless id if these rotate out. Fireworks's flagship models are reasoning
+# models, so `max_tokens` is set high enough to leave room for the hidden
+# reasoning tokens plus the visible answer.
+providers:
+  - id: fireworks:accounts/fireworks/models/gpt-oss-120b
+    label: 'GPT-OSS 120B'
+    config:
+      temperature: 0.2
+      max_tokens: 1024
+  - id: fireworks:accounts/fireworks/models/deepseek-v4-pro
+    label: 'DeepSeek V4 Pro'
+    config:
+      temperature: 0.2
+      max_tokens: 1024
+  - id: fireworks:accounts/fireworks/models/kimi-k2p6
+    label: 'Kimi K2.6'
+    config:
+      temperature: 0.2
+      max_tokens: 1024
+
+# Fireworks also serves embedding models on the same key. Here a `similar`
+# assertion is graded by the serverless Qwen3 embedding model via the
+# `fireworks:embedding:` prefix.
+defaultTest:
+  options:
+    provider:
+      embedding:
+        id: fireworks:embedding:accounts/fireworks/models/qwen3-embedding-8b
+
+tests:
+  - vars:
+      passage: |
+        On the surface, the Antikythera mechanism was a corroded lump of bronze pulled from a Roman-era
+        shipwreck off Greece in 1901. Decades of x-ray and tomographic analysis revealed it to be an
+        intricate analogue computer, with dozens of geared dials that tracked the Sun and Moon, predicted
+        eclipses, and modelled the irregular motion of the lunar orbit using an epicyclic gear train.
+    assert:
+      - type: icontains
+        value: Antikythera
+      - type: icontains-any
+        value: [mechanism, computer, gears, astronomical]
+      - type: similar
+        value: an ancient Greek geared device that predicted astronomical events and eclipses
+        threshold: 0.4
+  - vars:
+      passage: |
+        Edsger Dijkstra argued that programs should be derived from their specifications through a
+        chain of small, verifiable steps. He took particular issue with the unrestricted use of GOTO,
+        which he believed encouraged jumping around in code in ways that made formal reasoning about
+        program correctness much harder than it needed to be.
+    assert:
+      - type: icontains-any
+        value: [Dijkstra, GOTO, correctness, specification]