chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,109 @@
|
||||
# provider-cerebras (Cerebras Example (High-Performance LLM Inference))
|
||||
|
||||
This example demonstrates how to use the Cerebras provider with promptfoo to evaluate Cerebras Inference API models, which offer high-performance inference for Llama and other LLM models.
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example provider-cerebras
|
||||
cd provider-cerebras
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### API Key Setup
|
||||
|
||||
1. Sign up for an account at [Cerebras AI](https://console.cerebras.ai/)
|
||||
2. Navigate to your account settings to generate an API key
|
||||
3. Set your Cerebras API key as an environment variable:
|
||||
|
||||
```bash
|
||||
export CEREBRAS_API_KEY="your-api-key-here"
|
||||
```
|
||||
|
||||
Alternatively, you can add it to your `.env` file:
|
||||
|
||||
```text
|
||||
CEREBRAS_API_KEY=your-api-key-here
|
||||
```
|
||||
|
||||
## Example Configurations
|
||||
|
||||
This repository contains three example configurations demonstrating different Cerebras features:
|
||||
|
||||
### 1. Basic Model Evaluation (`promptfooconfig.yaml`)
|
||||
|
||||
This configuration evaluates two Cerebras models on their ability to explain complex concepts in simple terms.
|
||||
|
||||
```bash
|
||||
promptfoo eval
|
||||
```
|
||||
|
||||
**Expected output:** You'll see a comparison of how each model explains concepts from different domains, with metrics on clarity, accuracy, and response time.
|
||||
|
||||
### 2. Structured Outputs (`promptfooconfig-structured.yaml`)
|
||||
|
||||
The structured output example demonstrates Cerebras's JSON schema enforcement capabilities, ensuring the model returns consistent, structured recipe data with proper types and required fields.
|
||||
|
||||
```bash
|
||||
promptfoo eval -c promptfooconfig-structured.yaml
|
||||
```
|
||||
|
||||
**Expected output:** You'll receive structured JSON outputs for different recipes, with consistent fields like cuisine type, difficulty level, ingredients, and cooking instructions - all following the defined schema.
|
||||
|
||||
Example output:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Traditional Pasta Carbonara",
|
||||
"cuisine": "Italian",
|
||||
"difficulty": "medium",
|
||||
"prepTime": 15,
|
||||
"cookTime": 20,
|
||||
"ingredients": [
|
||||
{ "name": "spaghetti", "amount": "400g" },
|
||||
{ "name": "pancetta", "amount": "150g" },
|
||||
{ "name": "eggs", "amount": "3 large" },
|
||||
{ "name": "parmesan cheese", "amount": "50g" }
|
||||
],
|
||||
"instructions": [
|
||||
"Bring a large pot of salted water to boil",
|
||||
"Cook spaghetti according to package instructions",
|
||||
"In a separate pan, cook pancetta until crispy",
|
||||
"In a bowl, whisk eggs and grated parmesan cheese",
|
||||
"Drain pasta, reserving some pasta water",
|
||||
"Toss hot pasta with pancetta, then quickly mix in egg mixture",
|
||||
"Add pasta water as needed to create a silky sauce"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Tool Use (`promptfooconfig-tools.yaml`)
|
||||
|
||||
The tool use example demonstrates Cerebras's function calling capabilities with a calculator tool that the model can use to solve math problems.
|
||||
|
||||
```bash
|
||||
promptfoo eval -c promptfooconfig-tools.yaml
|
||||
```
|
||||
|
||||
**Expected output:** The model will use the calculator tool to solve math problems and provide step-by-step explanations of the solution process. For example, when given "15 × 7", it will calculate 105 and explain multiplication concepts.
|
||||
|
||||
## Model Capabilities
|
||||
|
||||
Cerebras supports several powerful models:
|
||||
|
||||
- `llama-4-scout-17b-16e-instruct` - Llama 4 Scout 17B model with 16 expert MoE (featured in examples)
|
||||
- `llama3.1-8b` - Llama 3.1 8B model
|
||||
- `llama-3.3-70b` - Llama 3.3 70B model
|
||||
- `deepSeek-r1-distill-llama-70B` (private preview)
|
||||
|
||||
## Pricing & Usage
|
||||
|
||||
Cerebras Inference API offers competitive pricing compared to other inference services. Check the [official pricing page](https://docs.cerebras.ai) for the most current rates. Usage is billed based on input and output tokens.
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Cerebras Provider Documentation](https://promptfoo.dev/docs/providers/cerebras)
|
||||
- [Cerebras API Reference](https://docs.cerebras.ai/)
|
||||
- [Cerebras Structured Outputs Guide](https://docs.cerebras.ai/capabilities/structured-outputs/)
|
||||
- [Cerebras Tool Use Guide](https://docs.cerebras.ai/capabilities/tool-use/)
|
||||
@@ -0,0 +1,68 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Schema-enforced JSON recipe generation with Llama 4
|
||||
prompts:
|
||||
- file://structured_prompts.txt
|
||||
providers:
|
||||
- id: cerebras:llama-4-scout-17b-16e-instruct
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_completion_tokens: 1024
|
||||
response_format:
|
||||
type: 'json_schema'
|
||||
json_schema:
|
||||
name: 'recipe_schema'
|
||||
strict: true
|
||||
schema:
|
||||
type: 'object'
|
||||
properties:
|
||||
name: { 'type': 'string' }
|
||||
cuisine: { 'type': 'string' }
|
||||
difficulty:
|
||||
type: 'string'
|
||||
enum: ['easy', 'medium', 'hard']
|
||||
prepTime: { 'type': 'integer' }
|
||||
cookTime: { 'type': 'integer' }
|
||||
ingredients:
|
||||
type: 'array'
|
||||
items:
|
||||
type: 'object'
|
||||
properties:
|
||||
name: { 'type': 'string' }
|
||||
amount: { 'type': 'string' }
|
||||
required: ['name', 'amount']
|
||||
instructions:
|
||||
type: 'array'
|
||||
items: { 'type': 'string' }
|
||||
required: ['name', 'cuisine', 'difficulty', 'ingredients', 'instructions']
|
||||
additionalProperties: false
|
||||
tests:
|
||||
- vars:
|
||||
cuisine: Italian
|
||||
dish: Cacio e Pepe
|
||||
assert:
|
||||
- type: json-schema-valid
|
||||
value: |
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["name", "cuisine", "difficulty", "ingredients", "instructions"]
|
||||
}
|
||||
- vars:
|
||||
cuisine: Japanese
|
||||
dish: Okonomiyaki
|
||||
assert:
|
||||
- type: contains-json-property
|
||||
value: ['ingredients', 'instructions']
|
||||
- type: javascript
|
||||
value: |
|
||||
return output.ingredients.length >= 5 ? 'pass' : 'fail: needs at least 5 ingredients'
|
||||
- vars:
|
||||
cuisine: Mexican
|
||||
dish: Chiles en Nogada
|
||||
assert:
|
||||
- type: contains-json-property
|
||||
value: ['difficulty']
|
||||
- type: javascript
|
||||
value: |
|
||||
return output.instructions.length >= 3 ? 'pass' : 'fail: needs detailed instructions'
|
||||
outputs:
|
||||
- type: json
|
||||
@@ -0,0 +1,49 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Function calling with Llama 4 for mathematical problem solving
|
||||
prompts:
|
||||
- file://tool_prompts.txt
|
||||
providers:
|
||||
- id: cerebras:llama-4-scout-17b-16e-instruct
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_completion_tokens: 1024
|
||||
tools:
|
||||
- type: 'function'
|
||||
function:
|
||||
name: 'calculate'
|
||||
description: 'A calculator that can perform basic arithmetic operations'
|
||||
parameters:
|
||||
type: 'object'
|
||||
properties:
|
||||
expression:
|
||||
type: 'string'
|
||||
description: 'The mathematical expression to evaluate'
|
||||
required: ['expression']
|
||||
strict: true
|
||||
tests:
|
||||
- vars:
|
||||
problem: '15 × 7'
|
||||
explanation: 'the product of fifteen and seven'
|
||||
assert:
|
||||
- type: contains
|
||||
value: '105'
|
||||
- type: contains-any
|
||||
value: ['multiply', 'multiplication', 'multiplying', 'product']
|
||||
- vars:
|
||||
problem: '(42 × 3) ÷ 6'
|
||||
explanation: 'forty-two times three, divided by six'
|
||||
assert:
|
||||
- type: contains
|
||||
value: '21'
|
||||
- type: contains-any
|
||||
value: ['order of operations', 'PEMDAS', 'parentheses', 'brackets']
|
||||
- vars:
|
||||
problem: '√(144) + 25²'
|
||||
explanation: 'the square root of one hundred forty-four plus twenty-five squared'
|
||||
assert:
|
||||
- type: contains
|
||||
value: '637'
|
||||
- type: contains-any
|
||||
value: ['square root', 'exponent', 'power', 'squared']
|
||||
outputs:
|
||||
- type: text
|
||||
@@ -0,0 +1,42 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Evaluating Llama models for educational concept explanations
|
||||
prompts:
|
||||
- file://prompts.txt
|
||||
providers:
|
||||
- id: cerebras:llama-4-scout-17b-16e-instruct
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_completion_tokens: 1024
|
||||
- id: cerebras:llama-3.3-70b
|
||||
config:
|
||||
temperature: 0.5
|
||||
max_completion_tokens: 1024
|
||||
tests:
|
||||
- vars:
|
||||
topic: quantum computing
|
||||
concept: quantum entanglement
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['particles', 'correlation', 'distance', 'spooky action', 'Einstein']
|
||||
- vars:
|
||||
topic: artificial intelligence
|
||||
concept: transformer architecture
|
||||
assert:
|
||||
- type: contains-any
|
||||
value:
|
||||
['attention mechanism', 'self-attention', 'parallel processing', 'encoder', 'decoder']
|
||||
- vars:
|
||||
topic: astrophysics
|
||||
concept: black hole information paradox
|
||||
assert:
|
||||
- type: contains-any
|
||||
value:
|
||||
[
|
||||
'Hawking radiation',
|
||||
'event horizon',
|
||||
'quantum mechanics',
|
||||
'information loss',
|
||||
'firewall',
|
||||
]
|
||||
outputs:
|
||||
- type: csv
|
||||
@@ -0,0 +1,5 @@
|
||||
You are an expert in {{topic}} with exceptional teaching skills. Your task is to explain the concept of {{concept}} in simple terms that a high school student could understand.
|
||||
|
||||
Keep your explanation under 300 words and include 2-3 real-world examples or analogies that make this concept intuitive and memorable. Avoid overly technical jargon, and when you must use specialized terms, define them clearly.
|
||||
|
||||
Begin with a captivating introduction that sparks curiosity, follow with the main explanation using everyday language, and conclude with why understanding this concept matters in both academic and practical contexts.
|
||||
@@ -0,0 +1,7 @@
|
||||
You are a professional chef with expertise in various cuisines.
|
||||
|
||||
Create a {{cuisine}} recipe for {{dish}}.
|
||||
|
||||
Include a name for the recipe, the cuisine type, difficulty level (easy, medium, or hard), preparation time in minutes, cooking time in minutes, a list of ingredients with amounts, and step-by-step cooking instructions.
|
||||
|
||||
Your response should be structured as valid JSON.
|
||||
@@ -0,0 +1,7 @@
|
||||
You are a helpful math tutor with access to a calculator tool.
|
||||
|
||||
Calculate {{problem}} ({{explanation}}).
|
||||
|
||||
First, use your calculator tool to solve the problem. Then, briefly explain how to solve this type of problem step by step, as if you were teaching a student.
|
||||
|
||||
Make sure to show your final answer and check if it's correct.
|
||||
Reference in New Issue
Block a user