wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
63 lines
2.7 KiB
YAML
63 lines
2.7 KiB
YAML
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
|
description: Advanced reasoning capabilities testing
|
|
|
|
prompts:
|
|
- '{{prompt}}'
|
|
|
|
providers:
|
|
- id: hyperbolic:deepseek-ai/DeepSeek-V3
|
|
label: DeepSeek V3
|
|
config:
|
|
temperature: 0.1
|
|
max_tokens: 2048
|
|
|
|
- id: hyperbolic:meta-llama/Meta-Llama-3.1-70B-Instruct
|
|
label: Llama 3.1-70B
|
|
config:
|
|
temperature: 0.3
|
|
max_tokens: 1024
|
|
|
|
tests:
|
|
# Classic puzzle with a twist
|
|
- vars:
|
|
prompt: 'Solve this step by step: A time-traveling farmer needs to transport a robot fox, a cyborg chicken, and quantum grain across a dimensional rift. The portal can only carry the farmer and one item. If left alone, the robot fox hacks the cyborg chicken, and the cyborg chicken deletes the quantum grain. How can the farmer transport all three safely across dimensions?'
|
|
assert:
|
|
- type: contains-all
|
|
value: ['farmer', 'chicken', 'fox', 'grain']
|
|
- type: llm-rubric
|
|
value: 'The solution correctly solves the river crossing puzzle with proper step-by-step reasoning'
|
|
|
|
# Tricky weighing problem
|
|
- vars:
|
|
prompt: 'Solve this step by step: You have 8 identical magic orbs, but one contains a tiny dragon that makes it slightly heavier. Using an enchanted balance scale only twice, how do you find the orb with the dragon?'
|
|
assert:
|
|
- type: llm-rubric
|
|
value: 'The solution correctly identifies how to find the heavier orb in exactly 2 weighings with clear logic'
|
|
|
|
# Mathematical brain teaser
|
|
- vars:
|
|
prompt: 'Explain this using mathematical proof: A mathematician walks into a café and claims that 0.999... (repeating) equals exactly 1. The barista is skeptical. How would you convince the barista using mathematical proof?'
|
|
assert:
|
|
- type: llm-rubric
|
|
value: 'The explanation provides a clear and convincing mathematical proof that 0.999... = 1'
|
|
|
|
# Word problem with a catch
|
|
- vars:
|
|
prompt: 'Solve this step by step: At a magical sports shop, a enchanted bat and a golden ball cost $1.10 total. The bat costs exactly $1 more than the ball. What does each item cost? (Warning: your first instinct might be wrong!)'
|
|
assert:
|
|
- type: contains
|
|
value: '$0.05'
|
|
- type: contains
|
|
value: '$1.05'
|
|
|
|
# Programming puzzle
|
|
- vars:
|
|
prompt: 'Write a recursive function to calculate the nth Fibonacci number, then explain why it would make a computer cry and how to make it happy again with optimization.'
|
|
assert:
|
|
- type: contains
|
|
value: 'def'
|
|
- type: contains-any
|
|
value: ['O(2^n)', 'exponential']
|
|
- type: contains-any
|
|
value: ['memoization', 'dynamic programming', 'iterative']
|