chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# provider-nscale (Nscale Example (Cost-Effective AI Inference))
|
||||
|
||||
This example demonstrates how to use the Nscale provider with promptfoo to evaluate Nscale Serverless Inference API models, which offer cost-effective, high-performance AI inference with zero rate limits.
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example provider-nscale
|
||||
cd provider-nscale
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Authentication Setup
|
||||
|
||||
1. Sign up for an account at [Nscale](https://nscale.com/)
|
||||
2. Navigate to your account settings
|
||||
3. Go to "Service Tokens" section
|
||||
4. Generate a service token and set it as an environment variable:
|
||||
|
||||
```bash
|
||||
export NSCALE_SERVICE_TOKEN="your-service-token-here"
|
||||
```
|
||||
|
||||
Alternatively, you can add it to your `.env` file:
|
||||
|
||||
```env
|
||||
NSCALE_SERVICE_TOKEN=your-service-token-here
|
||||
```
|
||||
|
||||
## Example Configuration
|
||||
|
||||
This repository contains an example configuration demonstrating Nscale's capabilities:
|
||||
|
||||
### Basic Model Evaluation (`promptfooconfig.yaml`)
|
||||
|
||||
This configuration evaluates two popular Nscale models on their ability to solve reasoning problems and generate creative content.
|
||||
|
||||
```bash
|
||||
promptfoo eval
|
||||
```
|
||||
|
||||
**Expected output:** You'll see a comparison of how each model handles different types of tasks, with metrics on accuracy, creativity, and response quality.
|
||||
|
||||
### Image Generation Evaluation (`image-promptfooconfig.yaml`)
|
||||
|
||||
This configuration compares Nscale's image generation models on various prompts to evaluate their quality and consistency.
|
||||
|
||||
```bash
|
||||
promptfoo eval -c image-promptfooconfig.yaml
|
||||
```
|
||||
|
||||
**Expected output:** You'll see generated images from different models (Flux.1 Schnell, SDXL Lightning, Stable Diffusion XL) for comparison across various image types including landscapes, futuristic scenes, portraits, and abstract art.
|
||||
|
||||
## Model Capabilities
|
||||
|
||||
Nscale supports many popular models with competitive pricing:
|
||||
|
||||
**Text Generation Models:**
|
||||
|
||||
- `openai/gpt-oss-120b` - OpenAI's 120B open-weight model
|
||||
- `openai/gpt-oss-20b` - OpenAI's 20B model
|
||||
- `qwen/qwen-3-235b-a22b-instruct` - Qwen 3 235B model
|
||||
- `qwen/qwen-3-235b-a22b-instruct-2507` - Qwen 3 235B Instruct 2507
|
||||
- `qwen/qwen-3-4b-thinking-2507` - Qwen 3 4B Thinking 2507
|
||||
- `qwen/qwen-3-8b` - Qwen 3 8B model
|
||||
- `qwen/qwen-3-14b` - Qwen 3 14B model
|
||||
- `qwen/qwen-3-32b` - Qwen 3 32B model
|
||||
- `qwen/qwen-2.5-coder-3b-instruct` - Qwen 2.5 Coder 3B Instruct
|
||||
- `qwen/qwen-2.5-coder-7b-instruct` - Qwen 2.5 Coder 7B Instruct
|
||||
- `qwen/qwen-2.5-coder-32b-instruct` - Qwen 2.5 Coder 32B Instruct
|
||||
- `qwen/qwq-32b` - Qwen QwQ 32B model
|
||||
- `meta/llama-3.3-70b-instruct` - Meta's Llama 3.3 70B model
|
||||
- `meta/llama-3.1-8b-instruct` - Meta's Llama 3.1 8B model
|
||||
- `meta/llama-4-scout-17b-16e-instruct` - Llama 4 Scout 17B model (Image-Text-to-Text)
|
||||
- `deepseek/deepseek-r1-distill-llama-70b` - DeepSeek R1 Distill Llama 70B
|
||||
- `deepseek/deepseek-r1-distill-llama-8b` - DeepSeek R1 Distill Llama 8B
|
||||
- `deepseek/deepseek-r1-distill-qwen-1.5b` - DeepSeek R1 Distill Qwen 1.5B
|
||||
- `deepseek/deepseek-r1-distill-qwen-7b` - DeepSeek R1 Distill Qwen 7B
|
||||
- `deepseek/deepseek-r1-distill-qwen-14b` - DeepSeek R1 Distill Qwen 14B
|
||||
- `deepseek/deepseek-r1-distill-qwen-32b` - DeepSeek R1 Distill Qwen 32B
|
||||
- `mistral/devstral-small-2505` - Mistral's Devstral Small model
|
||||
- `mistral/mixtral-8x22b-instruct-v0.1` - Mixtral 8x22B Instruct
|
||||
|
||||
**Embedding Models:**
|
||||
|
||||
- `Qwen/Qwen3-Embedding-8B` - Qwen 3 8B Embedding model
|
||||
|
||||
**Text-to-Image Models:**
|
||||
|
||||
- `BlackForestLabs/FLUX.1-schnell` - Flux.1 Schnell image generation model
|
||||
- `stabilityai/stable-diffusion-xl-base-1.0` - Stable Diffusion XL 1.0
|
||||
- `ByteDance/SDXL-Lightning-4step` - SDXL Lightning 4-step
|
||||
- `ByteDance/SDXL-Lightning-8step` - SDXL Lightning 8-step
|
||||
|
||||
## Pricing & Usage
|
||||
|
||||
Nscale offers highly competitive pricing with up to 80% cost savings compared to other providers:
|
||||
|
||||
- **Text Generation:** Starting from $0.01 input / $0.03 output per 1M tokens
|
||||
- **Image Generation:** Starting from $0.0008 per mega-pixel
|
||||
- **Zero rate limits** and **no cold starts**
|
||||
|
||||
Check the [official pricing page](https://docs.nscale.com/pricing) for the most current rates.
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Nscale Provider Documentation](https://promptfoo.dev/docs/providers/nscale)
|
||||
- [Nscale API Reference](https://docs.nscale.com/)
|
||||
- [Nscale Serverless Inference](https://nscale.com/serverless)
|
||||
- [Nscale Model Marketplace](https://nscale.com/models)
|
||||
@@ -0,0 +1,50 @@
|
||||
description: 'Nscale Image Generation Example - comparing different image models'
|
||||
|
||||
providers:
|
||||
- id: nscale:image:BlackForestLabs/FLUX.1-schnell
|
||||
config:
|
||||
size: '1024x1024'
|
||||
n: 1
|
||||
- id: nscale:image:ByteDance/SDXL-Lightning-4step
|
||||
config:
|
||||
size: '1024x1024'
|
||||
n: 1
|
||||
- id: nscale:image:stabilityai/stable-diffusion-xl-base-1.0
|
||||
config:
|
||||
size: '1024x1024'
|
||||
n: 1
|
||||
|
||||
prompts:
|
||||
- 'A serene landscape with mountains and a lake at sunset'
|
||||
- 'A futuristic cityscape with flying cars and neon lights'
|
||||
- 'A close-up portrait of a wise old wizard with a long beard'
|
||||
- 'An abstract geometric pattern in vibrant colors'
|
||||
|
||||
tests:
|
||||
- description: 'Nature scene generation'
|
||||
vars:
|
||||
prompt: 'A serene landscape with mountains and a lake at sunset'
|
||||
assert:
|
||||
- type: not-empty
|
||||
value: 'Image should be generated'
|
||||
|
||||
- description: 'Futuristic scene generation'
|
||||
vars:
|
||||
prompt: 'A futuristic cityscape with flying cars and neon lights'
|
||||
assert:
|
||||
- type: not-empty
|
||||
value: 'Image should be generated'
|
||||
|
||||
- description: 'Portrait generation'
|
||||
vars:
|
||||
prompt: 'A close-up portrait of a wise old wizard with a long beard'
|
||||
assert:
|
||||
- type: not-empty
|
||||
value: 'Image should be generated'
|
||||
|
||||
- description: 'Abstract art generation'
|
||||
vars:
|
||||
prompt: 'An abstract geometric pattern in vibrant colors'
|
||||
assert:
|
||||
- type: not-empty
|
||||
value: 'Image should be generated'
|
||||
@@ -0,0 +1,48 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: Evaluating Nscale models for reasoning and creative tasks
|
||||
prompts:
|
||||
- file://prompts.txt
|
||||
providers:
|
||||
- id: nscale:openai/gpt-oss-120b
|
||||
config:
|
||||
temperature: 0.7
|
||||
max_tokens: 1024
|
||||
- id: nscale:meta/llama-3.3-70b-instruct
|
||||
config:
|
||||
temperature: 0.5
|
||||
max_tokens: 1024
|
||||
tests:
|
||||
- vars:
|
||||
task_type: mathematical reasoning
|
||||
task_description: 'Solve this problem step by step: A train leaves Station A at 2:00 PM traveling at 60 mph toward Station B, which is 180 miles away. Another train leaves Station B at 2:30 PM traveling at 80 mph toward Station A. At what time will the two trains meet?'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['3:48', '3:48 PM', 'distance', 'relative speed', 'combined speed']
|
||||
- type: llm-rubric
|
||||
value: 'The response should show clear step-by-step mathematical reasoning and arrive at the correct answer of 3:48 PM.'
|
||||
- vars:
|
||||
task_type: creative writing
|
||||
task_description: 'Write a short story (2-3 paragraphs) about a librarian who discovers that books in their library are portals to different worlds. Include vivid descriptions and an intriguing plot twist.'
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['portal', 'world', 'librarian', 'books', 'discovery']
|
||||
- type: llm-rubric
|
||||
value: 'The story should be creative, well-structured, include vivid descriptions, and have an interesting plot twist.'
|
||||
- vars:
|
||||
task_type: logical reasoning
|
||||
task_description: "Three friends - Alice, Bob, and Charlie - each have a different favorite color (red, blue, green) and a different pet (cat, dog, bird). Given the clues: 1) Alice doesn't like red, 2) The person who likes blue has a cat, 3) Charlie has a bird, 4) Bob doesn't like green. Determine who has which pet and favorite color."
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['Alice', 'Bob', 'Charlie', 'blue', 'cat', 'bird']
|
||||
- type: llm-rubric
|
||||
value: 'The response should systematically work through the logical constraints and arrive at the correct solution.'
|
||||
- vars:
|
||||
task_type: scientific explanation
|
||||
task_description: "Explain the concept of photosynthesis in a way that a 12-year-old could understand, including why it's important for life on Earth."
|
||||
assert:
|
||||
- type: contains-any
|
||||
value: ['sunlight', 'carbon dioxide', 'oxygen', 'glucose', 'chlorophyll', 'plants']
|
||||
- type: llm-rubric
|
||||
value: 'The explanation should be age-appropriate, scientifically accurate, and clearly explain the importance of photosynthesis.'
|
||||
outputs:
|
||||
- type: csv
|
||||
@@ -0,0 +1,13 @@
|
||||
You are an expert assistant that provides clear, accurate, and helpful responses.
|
||||
|
||||
Please analyze the following {{task_type}} task and provide a comprehensive response:
|
||||
|
||||
{{task_description}}
|
||||
|
||||
Consider the following aspects:
|
||||
1. Key concepts and principles involved
|
||||
2. Step-by-step reasoning or approach
|
||||
3. Practical implications or applications
|
||||
4. Any potential challenges or considerations
|
||||
|
||||
Provide your response in a clear, structured format.
|
||||
Reference in New Issue
Block a user