wehub-resource-sync
0d3cb498a3
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
89 lines
2.4 KiB
Markdown
89 lines
2.4 KiB
Markdown
# Examples
|
|
|
|
Self-contained example configurations for `npx promptfoo@latest init --example <name>`.
|
|
|
|
## CRITICAL: Test with Local Build
|
|
|
|
```bash
|
|
# Correct - tests your changes
|
|
npm run local -- eval -c examples/my-example/promptfooconfig.yaml
|
|
|
|
# Wrong - tests published version
|
|
npx promptfoo@latest eval
|
|
```
|
|
|
|
## Example Structure
|
|
|
|
Each example needs:
|
|
|
|
1. Directory with clear name
|
|
2. `README.md` starting with `# folder-name (Human Readable Name)`
|
|
3. `promptfooconfig.yaml` with schema reference
|
|
4. Instructions using `npx promptfoo@latest init --example <name>`
|
|
|
|
## Configuration Format
|
|
|
|
```yaml
|
|
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
|
description: Short description (3-10 words)
|
|
prompts:
|
|
- ...
|
|
providers:
|
|
- ...
|
|
tests:
|
|
- ...
|
|
```
|
|
|
|
**Field order:** description, env, prompts, providers, defaultTest, scenarios, tests
|
|
|
|
## Environment Variables in Configs
|
|
|
|
Use Nunjucks template syntax to reference environment variables:
|
|
|
|
```yaml
|
|
# ✅ CORRECT - Nunjucks template syntax
|
|
accountId: '{{env.CLOUDFLARE_ACCOUNT_ID}}'
|
|
apiKey: '{{env.OPENAI_API_KEY}}'
|
|
|
|
# ❌ WRONG - Shell syntax doesn't work in YAML configs
|
|
accountId: ${CLOUDFLARE_ACCOUNT_ID}
|
|
apiKey: $OPENAI_API_KEY
|
|
```
|
|
|
|
Note: Quotes around `'{{env.VAR}}'` are required in YAML to prevent parsing issues.
|
|
|
|
## Model Selection
|
|
|
|
Use current model identifiers (see `site/docs/providers/` for full list):
|
|
|
|
- OpenAI: `openai:chat:gpt-5.6`, `openai:responses:gpt-5.6-sol`, `openai:responses:gpt-5.6-terra`, `openai:responses:gpt-5.6-luna`, `openai:chat:gpt-5.4-mini`
|
|
- Anthropic: `anthropic:messages:claude-sonnet-4-6`, `anthropic:messages:claude-haiku-4-5-20251001`
|
|
- Google: `google:gemini-3.1-pro-preview`, `google:gemini-2.5-flash`
|
|
|
|
## Guidelines
|
|
|
|
- Keep examples simple while demonstrating the concept
|
|
- Use `file://` prefix for external files
|
|
- Make test cases engaging, not boring
|
|
- Document required environment variables in README
|
|
- Test examples before submitting
|
|
- **If adding example for a new provider**, verify docs exist at `site/docs/providers/<provider>.md`
|
|
|
|
## Chat Format Prompts
|
|
|
|
Some providers require specific message structures (e.g., exactly one system + one user message). Use a JSON file instead of inline YAML:
|
|
|
|
```json
|
|
// prompts/chat.json
|
|
[
|
|
{ "role": "system", "content": "You are a helpful assistant." },
|
|
{ "role": "user", "content": "Help with: {{topic}}" }
|
|
]
|
|
```
|
|
|
|
```yaml
|
|
# promptfooconfig.yaml
|
|
prompts:
|
|
- file://prompts/chat.json
|
|
```
|