Smoke Tests

Curated smoke tests that verify Cline works correctly with LLM providers.

Purpose

These tests catch regressions in:

Tool execution (read, write, edit files)
Provider response parsing
Tool chaining (multiple operations)
Basic code generation

Quick Start

# One-time auth setup
cline auth

# Run tests (3 trials by default)
npm run eval:smoke:run

Commands

Command	What it does
`npm run eval:smoke:run`	Run tests (uses installed CLI)

Options

# Run specific scenario
npm run eval:smoke:run -- --scenario 01-create-file

# Run with fewer trials (faster)
npm run eval:smoke:run -- --trials 1

# Run with specific model (overrides any per-scenario models)
npm run eval:smoke:run -- --model claude-sonnet-4-5-20250929

Authentication

Interactive (recommended for local dev)

cline auth

With API key (for automation)

cline auth -p cline -k "$CLINE_API_KEY" -m anthropic/claude-sonnet-4.5

Scenarios

ID	Name	What it tests
01-create-file	Create a simple file	`write_to_file`
02-edit-file	Edit existing file	`replace_in_file`
03-read-summarize	Read and summarize	`read_file`
04-multi-file	Create multiple files	Multiple tool calls
05-typescript-function	Generate TypeScript	Code generation
06-apply-patch	Edit file (GPT-5)	`apply_patch` tool, native tool calling
07-edit-gemini	Edit file (Gemini)	Gemini model variant

Per-Scenario Models

Scenarios can specify their own model(s) via the models field in config.json. This is useful for testing model-specific code paths like apply_patch (GPT-5 only).

If you pass --model, it overrides any per-scenario models list.

Examples:

# Run apply_patch scenario with its default model (GPT-5)
npm run eval:smoke:run -- --scenario 06-apply-patch

# Force that scenario to use a specific model
npm run eval:smoke:run -- --scenario 06-apply-patch --model openai/gpt-4o

Metrics

pass@k: Probability at least 1 of k trials succeeds
pass^k: Probability ALL k trials succeed (reliability)

Shows pass@1 when trials < 3, pass@3 otherwise.

Adding New Scenarios

Create directory: scenarios/<name>/

Add config.json:

{
  "name": "Human-readable name",
  "description": "What this tests",
  "prompt": "The task prompt for Cline",
  "expectedFiles": ["file1.txt"],
  "expectedContent": [
    { "file": "file1.txt", "contains": "expected text" }
  ],
  "timeout": 60
}

(Optional) Add template/ directory with starting files

CI Integration

Smoke test CI is temporarily disabled while the build step is repointed at the SDK CLI. The remaining root script runs the scenarios against whichever cline binary is already on $PATH.

Required Secrets

CLINE_API_KEY - Cline API key, needed when CI is re-enabled

Viewing Results

Local summaries are written under evals/smoke-tests/results/latest/

TODO

Native tool calling tests: Add CLI support for native_tool_call_enabled setting, then create a scenario that tests Claude 4 with native tool calling enabled (currently only GPT-5 models automatically use native tools via the Responses API)