wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
46 lines
1.4 KiB
YAML
46 lines
1.4 KiB
YAML
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
|
description: Reproduce Mistral Magistral AIME2024 benchmark
|
|
|
|
prompts:
|
|
- |
|
|
Solve this AIME mathematical problem step by step.
|
|
|
|
Problem: {{question}}
|
|
|
|
Think through this carefully and provide your final answer as a 3-digit integer (000-999).
|
|
End with: "Therefore, the answer is [your answer]."
|
|
|
|
providers:
|
|
- id: mistral:magistral-medium-latest
|
|
label: Magistral Medium
|
|
config:
|
|
temperature: 0.7
|
|
top_p: 0.95
|
|
max_tokens: 40960
|
|
# The `magistral-small-latest` alias now resolves to Mistral Small 4, so use the
|
|
# canonical id. It contrasts a general hybrid model against native-reasoning Magistral.
|
|
- id: mistral:mistral-small-latest
|
|
label: Mistral Small 4
|
|
config:
|
|
temperature: 0.7
|
|
top_p: 0.95
|
|
max_tokens: 40960
|
|
|
|
tests:
|
|
- huggingface://datasets/sea-snell/aime-2024?split=test
|
|
|
|
defaultTest:
|
|
assert:
|
|
- type: llm-rubric
|
|
value: |
|
|
Evaluate this mathematical solution to an AIME competition problem.
|
|
|
|
The correct answer is: {{answer}}
|
|
|
|
Grade as PASS if and only if:
|
|
1. The response shows clear step-by-step mathematical reasoning
|
|
2. The final answer presented equals {{answer}} exactly
|
|
3. The mathematical work supports the conclusion
|
|
|
|
Grade as FAIL if the final answer is incorrect, regardless of the reasoning quality.
|