chore: import upstream snapshot with attribution
Voice Workbench / headless workbench (mocked backends) (push) Has been cancelled
Voice Workbench / real acoustic lane (nightly, provisioned only) (push) Has been cancelled
ci / test (push) Has been cancelled
ci / lint-and-format (push) Has been cancelled
ci / build (push) Has been cancelled
ci / dev-startup (push) Has been cancelled
gitleaks / gitleaks (push) Has been cancelled
Markdown Links / Relative Markdown Links (push) Has been cancelled
Quality (Extended) / Homepage Build (PR smoke) (push) Has been cancelled
Quality (Extended) / Comment-only diff guard (push) Has been cancelled
Quality (Extended) / Format + Type Safety Ratchet (push) Has been cancelled
Quality (Extended) / Develop Gate (secret scan + UI determinism) (push) Has been cancelled
Quality (Extended) / Develop Gate (lint) (push) Has been cancelled
Chat shell gestures / Chat shell gesture + parity e2e (push) Has been cancelled
Cloud Gateway Discord / Test (push) Has been cancelled
Benchmark Bridge Tests / benchmark (bunx @biomejs/biome check packages/lifeops-bench/src, benchmark-lint) (push) Has been cancelled
Benchmark Bridge Tests / benchmark (bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests, benchmark-tests) (push) Has been cancelled
Build Agent Image / build-and-push (push) Has been cancelled
Dev Smoke / bun run dev onboarding chat (push) Has been cancelled
Dev Smoke / Vite HMR dependency-level smoke (push) Has been cancelled
Electrobun Submodule Guard / electrobun gitlink is fetchable (push) Has been cancelled
Publish @elizaos/example-code / check_npm (push) Has been cancelled
Publish @elizaos/example-code / publish_npm (push) Has been cancelled
Publish @elizaos/plugin-elizacloud / verify_version (push) Has been cancelled
Publish @elizaos/plugin-elizacloud / publish_npm (push) Has been cancelled
Sandbox Live Smoke / Sandbox live smoke (push) Has been cancelled
Snap Build & Test / Build Snap (amd64) (push) Has been cancelled
Snap Build & Test / Build Snap (arm64) (push) Has been cancelled
Test Packaging / elizaos CLI global-install smoke (node + bun) (push) Has been cancelled
Cloud Gateway Webhook / Test (push) Has been cancelled
Cloud Tests / lint-and-types (push) Has been cancelled
Cloud Tests / unit-tests (push) Has been cancelled
Cloud Tests / integration-tests (push) Has been cancelled
Cloud Tests / e2e-tests (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Deploy Apps Worker (Product 2) / Determine environment (push) Has been cancelled
Deploy Apps Worker (Product 2) / Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }}) (push) Has been cancelled
Deploy Eliza Provisioning Worker / Determine environment (push) Has been cancelled
Deploy Eliza Provisioning Worker / Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }}) (push) Has been cancelled
Dev Smoke / Classify changed paths (push) Has been cancelled
supply-chain / sbom (push) Has been cancelled
supply-chain / vulnerability-scan (push) Has been cancelled
Build, Push & Deploy to Phala Cloud / build-and-push (push) Has been cancelled
Test Packaging / Validate Packaging Configs (push) Has been cancelled
Test Packaging / Build & Test PyPI Package (push) Has been cancelled
Test Packaging / PyPI on Python ${{ matrix.python }} (push) Has been cancelled
Test Packaging / Pack & Test JS Tarballs (push) Has been cancelled
UI Fixture E2E / ui-fixture-e2e (push) Has been cancelled
UI Fixture E2E / fixture-e2e (push) Has been cancelled
UI Story Gate / story-gate (push) Has been cancelled
vault-ci / test (macos-latest) (push) Has been cancelled
vault-ci / test (ubuntu-latest) (push) Has been cancelled
vault-ci / test (windows-latest) (push) Has been cancelled
vault-ci / app-core wiring tests (push) Has been cancelled
verify-patches / verify patches/CHECKSUMS.sha256 (push) Has been cancelled
Voice Benchmark Smoke / voice-emotion fixture smoke (push) Has been cancelled
Voice Benchmark Smoke / voiceagentbench fixture smoke (push) Has been cancelled
Voice Benchmark Smoke / voicebench-quality unit smoke (push) Has been cancelled
Voice Benchmark Smoke / voicebench TypeScript unit (no audio) (push) Has been cancelled
Voice Benchmark Smoke / voice bench smoke summary (push) Has been cancelled
Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Has been cancelled
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Has been cancelled
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Has been cancelled
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Has been cancelled
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:43:05 +08:00
commit 426e9eeabd
41828 changed files with 9656266 additions and 0 deletions
@@ -0,0 +1,144 @@
/**
* GEPA / bootstrap-fewshot run for the CONTEXTUAL view evaluator's `view_context`
* prompt against a local eliza-1 model via the eliza llama.cpp fork's
* `llama-server` (NOT Ollama — see lib/llamacpp.ts). Optimizes the situation→view
* INSTRUCTION the evaluator uses, scored by view-id match (scoreViewSelection),
* schema-constrained (mirrors production guided decode). Persist the winning
* instruction to <state>/optimized-prompts/view_context/ and the evaluator
* auto-loads it via resolveOptimizedPromptForRuntime(runtime,"view_context",base).
*
* Start a server (one model per server), then run:
* LLAMACPP_URL=http://127.0.0.1:8080 LABEL=eliza-1-2b \
* bun run plugins/plugin-training/scripts/gepa-view-context.ts
*
* runNativeBackend is not used here; the best prompt is written to a temp dir for
* inspection. Promote into the live store deliberately (never from a test).
*
* Last local sweep (eliza-1 via llama.cpp, view-id match over the 23-row dataset):
* eliza-1-2b usable default tier
* eliza-1-4b larger local tier; bootstrap demos may help
*/
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { evaluatePromotion } from "../src/core/promotion-gate.js";
import {
createPromptScorer,
runBootstrapFewshot,
runGepa,
scoreViewSelection,
} from "../src/optimizers/index.js";
import type { OptimizationExample } from "../src/optimizers/types.js";
import { LLAMACPP_URL, llamacppAdapter } from "./lib/llamacpp.js";
const LABEL = process.env.LABEL ?? "llamacpp";
const DATASET = join(
dirname(fileURLToPath(import.meta.url)),
"..",
"src",
"optimizers",
"__fixtures__",
"view-context.jsonl",
);
const TMP_OUT = "/tmp/gepa-view-context";
const VIEW_IDS = [
"calendar",
"inbox",
"wallet",
"finances",
"todos",
"goals",
"health",
"documents",
"relationships",
"focus",
"none",
];
const SCHEMA = {
type: "object",
properties: {
viewId: { type: "string", enum: VIEW_IDS },
reason: { type: "string" },
},
required: ["viewId"],
};
function load(): OptimizationExample[] {
return readFileSync(DATASET, "utf8")
.split("\n")
.map((l) => l.trim())
.filter(Boolean)
.map((l) => {
const r = JSON.parse(l) as {
request: { messages: Array<{ content: string }> };
response: { text: string };
};
return {
input: { user: r.request.messages.at(-1)?.content ?? "" },
expectedOutput: r.response.text,
};
});
}
// Deliberately generic baseline so GEPA/bootstrap have headroom to discover the
// situation→view mapping. The schema already constrains the output shape.
const BASELINE =
"Decide whether opening one app view would help the user, and which. Return JSON {viewId, reason}.";
async function main() {
mkdirSync(TMP_OUT, { recursive: true });
const dataset = load();
const adapter = llamacppAdapter(SCHEMA);
console.log(`[${LABEL}] dataset: ${dataset.length} rows | ${LLAMACPP_URL}`);
const scorer = createPromptScorer(adapter, {
compare: scoreViewSelection,
maxTokens: 60,
});
const baseline = await scorer(BASELINE, dataset);
const boot = await runBootstrapFewshot({
baselinePrompt: BASELINE,
dataset,
scorer,
llm: adapter,
options: { k: 6, rankByScorer: true },
});
const gepa = await runGepa({
baselinePrompt: BASELINE,
dataset,
scorer,
llm: adapter,
options: { population: 8, generations: 5, scoringSubset: dataset.length },
});
console.log(
`[${LABEL}] baseline=${baseline.toFixed(3)} bootstrap=${boot.score.toFixed(3)} gepa=${gepa.score.toFixed(3)}`,
);
const best = [
{ name: "bootstrap", score: boot.score, prompt: boot.optimizedPrompt },
{ name: "gepa", score: gepa.score, prompt: gepa.optimizedPrompt },
].sort((a, b) => b.score - a.score)[0];
// Regression gate (#8797): an optimized artifact may only be promoted when it
// beats the baseline by more than scoring noise. Reuse the canonical
// variance-aware promotion gate so a noisy single run can never silently
// regress the production `view_context` prompt.
const decision = await evaluatePromotion({
incumbentPrompt: BASELINE,
candidatePrompt: best.prompt,
dataset,
scorer,
});
const out = join(TMP_OUT, `${LABEL.replace(/[^a-z0-9]+/gi, "_")}.json`);
writeFileSync(out, JSON.stringify({ ...best, baseline, decision }, null, 2));
console.log(
`[${LABEL}] best candidate: ${best.name} ${best.score.toFixed(3)} | gate: ${decision.promote ? "PROMOTE" : "REJECT"} (${decision.reason}) → ${out}`,
);
if (!decision.promote) {
console.log(
`[${LABEL}] candidate did not beat baseline by the noise margin — keeping baseline.`,
);
}
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
@@ -0,0 +1,69 @@
// Subscription-only live-model lane for the LifeOps GEPA seed runner
// (#11384, unblocked by #10757).
//
// Hosts with no Cerebras/Anthropic API key but a Claude subscription can run
// the per-capability GEPA loop through the sanctioned `claude --print` binary.
// This reuses plugin-cli-inference's hardened `ClaudeCli` (isolated tmp cwd,
// env filtering, SOC2 binary allowlist, stderr redaction) rather than
// re-implementing the spawn, and adapts it to the `EvalModelClient` shape the
// optimizer + judge consume.
//
// Knobs (mirroring the runtime plugin):
// ELIZA_CLI_MODEL model id passed to `claude --model` (default haiku)
// ELIZA_CLI_TIMEOUT_MS per-call timeout (default 240 s)
//
// The CLI does not expose temperature / max-token controls, so those request
// fields are accepted and ignored — completions for these tasks are short and
// instruction-bounded.
import { existsSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
import { ClaudeCli } from "../../../plugin-cli-inference/src/claude-cli.ts";
import type { EvalModelClient } from "../../src/core/cerebras-eval-model.ts";
const DEFAULT_CLI_MODEL = "claude-haiku-4-5-20251001";
const DEFAULT_CLI_TIMEOUT_MS = 240_000;
export function cliCredentialsPresent(): boolean {
return existsSync(join(homedir(), ".claude", ".credentials.json"));
}
export function resolveCliModel(): string {
const explicit = process.env.ELIZA_CLI_MODEL?.trim();
return explicit && explicit.length > 0 ? explicit : DEFAULT_CLI_MODEL;
}
function resolveCliTimeoutMs(): number {
const raw = process.env.ELIZA_CLI_TIMEOUT_MS?.trim();
const parsed = raw ? Number.parseInt(raw, 10) : Number.NaN;
return Number.isFinite(parsed) && parsed > 0
? parsed
: DEFAULT_CLI_TIMEOUT_MS;
}
/**
* `EvalModelClient` backed by `claude --print` (subscription credentials,
* read by the CLI itself from `~/.claude/.credentials.json` — never injected
* into the child env).
*/
export function getCliModelClient(): EvalModelClient {
if (!cliCredentialsPresent()) {
throw new Error(
"[cli-model] ~/.claude/.credentials.json not found — the cli provider " +
"needs a logged-in claude CLI (run `claude login`) or use " +
"TRAIN_MODEL_PROVIDER=cerebras|anthropic with an API key.",
);
}
const cli = new ClaudeCli({
model: resolveCliModel(),
timeoutMs: resolveCliTimeoutMs(),
});
return async (req) => {
const text = await cli.generate({
system: req.systemPrompt,
prompt: req.prompt,
});
return { text };
};
}
@@ -0,0 +1,58 @@
/**
* Shared llama.cpp (eliza-fork `llama-server`) adapter for the local prompt-
* optimization measurement scripts. We use llama.cpp — NOT Ollama — because the
* production local-inference path is the eliza llama.cpp fork. That fork carries
* the Gemma 4 chat-template, long-context, and separate-drafter MTP support used
* by the active eliza-1 text tiers.
*
* Start a server first, e.g.:
* BC=plugins/plugin-local-inference/native/llama.cpp/build-cuda
* LD_LIBRARY_PATH="$BC/bin" "$BC/bin/llama-server" \
* -m /path/eliza-1-2b-128k.gguf --host 127.0.0.1 --port 8080 -ngl 99 -c 8192 --jinja
*
* Then point a script at it via LLAMACPP_URL (default http://127.0.0.1:8080).
* One server serves ONE model, so these scripts test whichever GGUF is loaded
* (label it via LABEL=...).
*
* Decoding is schema-constrained (OpenAI `response_format: json_schema`, mirrors
* production guided decode). `chat_template_kwargs.enable_thinking=false` is
* included for llama.cpp builds/models that expose thinking controls; these are
* cheap structured classifications, so private reasoning tokens are not useful.
*/
import type { LlmAdapter } from "../../src/optimizers/types.js";
export const LLAMACPP_URL = process.env.LLAMACPP_URL ?? "http://127.0.0.1:8080";
export function llamacppAdapter(
schema: object,
url: string = LLAMACPP_URL,
): LlmAdapter {
return {
async complete({ system, user, temperature, maxTokens }) {
const res = await fetch(`${url}/v1/chat/completions`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
messages: [
...(system ? [{ role: "system", content: system }] : []),
{ role: "user", content: user },
],
response_format: {
type: "json_schema",
json_schema: { name: "out", schema, strict: true },
},
chat_template_kwargs: { enable_thinking: false },
temperature: temperature ?? 0,
max_tokens: maxTokens ?? 80,
}),
});
if (!res.ok) {
throw new Error(`llama-server ${res.status}: ${await res.text()}`);
}
const data = (await res.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
return data.choices?.[0]?.message?.content ?? "";
},
};
}
@@ -0,0 +1,209 @@
#!/usr/bin/env bun
/**
* LifeOps per-capability GEPA loop (#8795).
*
* This is the file-backed CLI entrypoint around `triggerTraining`: it loads
* recorded real/scenario trajectories, privacy-filters and buckets them through
* the normal training orchestrator, runs the native GEPA backend for one
* LifeOps task, and persists through the existing promotion gate.
*
* Dry run:
* bun run --cwd plugins/plugin-training lifeops:gepa -- \
* --trajectories ../../reports/scenarios/run/trajectories --task calendar_extract --dry-run
*
* Live GEPA/promotion run:
* TRAIN_MODEL_PROVIDER=cerebras CEREBRAS_API_KEY=... \
* bun run --cwd plugins/plugin-training lifeops:gepa -- \
* --trajectories ../../reports/scenarios/run/trajectories --task calendar_extract
*/
import { existsSync, statSync } from "node:fs";
import { readdir, readFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { parseArgs } from "node:util";
import type { Trajectory } from "@elizaos/agent";
import { OptimizedPromptService } from "@elizaos/core";
import { triggerTraining } from "../src/core/training-orchestrator.js";
import {
LIFEOPS_TRAINING_TASKS,
type TrajectoryTrainingTask,
} from "../src/core/trajectory-task-datasets.js";
type CliArgs = {
dryRun: boolean;
stateDir?: string;
task: TrajectoryTrainingTask;
trajectories: string;
};
const LIFEOPS_TASK_SET = new Set<string>(LIFEOPS_TRAINING_TASKS);
function parseCliArgs(argv: string[]): CliArgs | "help" {
const { values } = parseArgs({
args: argv,
options: {
trajectories: { type: "string" },
task: { type: "string" },
"state-dir": { type: "string" },
"dry-run": { type: "boolean" },
help: { type: "boolean" },
},
allowPositionals: false,
});
if (values.help) return "help";
const trajectories = values.trajectories?.trim();
if (!trajectories) {
throw new Error("--trajectories <file-or-dir> is required");
}
const task = values.task?.trim();
if (!task || !LIFEOPS_TASK_SET.has(task)) {
throw new Error(
`--task must be one of: ${LIFEOPS_TRAINING_TASKS.join(", ")}`,
);
}
return {
dryRun: values["dry-run"] ?? false,
stateDir: values["state-dir"],
task: task as TrajectoryTrainingTask,
trajectories,
};
}
async function listJsonFiles(inputPath: string): Promise<string[]> {
const resolved = resolve(process.cwd(), inputPath);
if (!existsSync(resolved)) {
throw new Error(`trajectory path does not exist: ${resolved}`);
}
const stat = statSync(resolved);
if (stat.isFile()) return [resolved];
if (!stat.isDirectory()) {
throw new Error(
`trajectory path is neither file nor directory: ${resolved}`,
);
}
const out: string[] = [];
const stack = [resolved];
while (stack.length > 0) {
const dir = stack.pop();
if (!dir) continue;
for (const entry of await readdir(dir, { withFileTypes: true })) {
const fullPath = join(dir, entry.name);
if (entry.isDirectory()) stack.push(fullPath);
else if (entry.isFile() && entry.name.endsWith(".json"))
out.push(fullPath);
}
}
out.sort();
return out;
}
function isTrajectory(value: unknown): value is Trajectory {
return (
Boolean(value) &&
typeof value === "object" &&
typeof (value as { trajectoryId?: unknown }).trajectoryId === "string"
);
}
async function readTrajectories(inputPath: string): Promise<Trajectory[]> {
const files = await listJsonFiles(inputPath);
const trajectories: Trajectory[] = [];
for (const file of files) {
const parsed = JSON.parse(await readFile(file, "utf8")) as unknown;
if (Array.isArray(parsed)) {
for (const entry of parsed) {
if (isTrajectory(entry)) trajectories.push(entry);
}
} else if (isTrajectory(parsed)) {
trajectories.push(parsed);
}
}
return trajectories;
}
function makeTrajectoryService(trajectories: Trajectory[]) {
const byId = new Map(
trajectories.map((trajectory, index) => [
trajectory.trajectoryId || `trajectory-${index}`,
trajectory,
]),
);
return {
async listTrajectories() {
return {
trajectories: [...byId.keys()].map((id) => ({ id })),
};
},
async getTrajectoryDetail(id: string) {
return byId.get(id) ?? null;
},
};
}
async function main(): Promise<number> {
const parsed = parseCliArgs(process.argv.slice(2));
if (parsed === "help") {
process.stdout.write(
`Usage: bun run lifeops:gepa -- --trajectories <file-or-dir> --task <lifeops-task> [--dry-run] [--state-dir <dir>]\n`,
);
return 0;
}
const trajectories = await readTrajectories(parsed.trajectories);
if (trajectories.length === 0) {
throw new Error("no recorded trajectory JSON files were found");
}
const resolvedStateDir = parsed.stateDir
? resolve(process.cwd(), parsed.stateDir)
: undefined;
if (resolvedStateDir) {
process.env.TRAINING_STATE_DIR = resolvedStateDir;
process.env.ELIZA_STATE_DIR = resolvedStateDir;
}
const optimizedPromptService = new OptimizedPromptService();
if (resolvedStateDir) {
optimizedPromptService.setStoreRoot(
join(resolvedStateDir, "optimized-prompts"),
);
}
const trajectoryService = makeTrajectoryService(trajectories);
const runtime = {
getService(name: string) {
if (name === "trajectories") return trajectoryService;
if (name === "optimized_prompt") return optimizedPromptService;
return null;
},
logger: {
info: (message: string) => process.stdout.write(`${message}\n`),
warn: (message: string) => process.stderr.write(`${message}\n`),
error: (message: string) => process.stderr.write(`${message}\n`),
},
};
const record = await triggerTraining(runtime, {
backend: "native",
dryRun: parsed.dryRun,
source: "manual",
task: parsed.task,
trajectoryLimit: trajectories.length,
});
process.stdout.write(`${JSON.stringify(record, null, 2)}\n`);
return record.status === "failed" ? 1 : 0;
}
main()
.then((code) => {
process.exitCode = code;
})
.catch((error) => {
process.stderr.write(
`[lifeops-gepa-loop] ${error instanceof Error ? error.message : String(error)}\n`,
);
process.exitCode = 1;
});
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,269 @@
#!/usr/bin/env bun
/**
* Batch trajectory-quality review (#8795) — CLI.
*
* Reads recorded trajectory JSON (same corpus `lifeops:gepa` consumes),
* samples N model calls per LifeOps capability, scores each against the
* per-capability rubric with the configured eval model (deterministic JSON
* `{score, reason}`, fail-closed parse), and writes a per-capability
* scoreboard as JSON + markdown (mean/min/max, worst samples with reasons +
* source file paths for hand review).
*
* Keyless dry run (lists what WOULD be scored, no model calls):
* bun run --cwd plugins/plugin-training trajectories:review -- \
* --trajectories ../../reports/scenarios/run/trajectories --dry-run
*
* Live review (eval backend configured like the other training scripts —
* EVAL_MODEL_PROVIDER=cerebras CEREBRAS_API_KEY=..., or
* EVAL_MODEL_PROVIDER=anthropic ANTHROPIC_API_KEY=...):
* bun run --cwd plugins/plugin-training trajectories:review -- \
* --trajectories ../../reports/scenarios/run/trajectories \
* --samples 5 --out reports/trajectory-quality
*/
import { existsSync, statSync } from "node:fs";
import { mkdir, readdir, readFile, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { parseArgs } from "node:util";
import type { Trajectory } from "@elizaos/agent";
import {
buildReview,
collectQualitySamples,
judgeSamples,
renderReviewMarkdown,
type TrajectoryQualitySample,
} from "../src/core/trajectory-quality-review.js";
import {
LIFEOPS_TRAINING_TASKS,
type LifeOpsTrainingTask,
} from "../src/core/trajectory-task-datasets.js";
type CliArgs = {
trajectories: string;
samples: number;
out: string;
task?: LifeOpsTrainingTask;
dryRun: boolean;
};
const LIFEOPS_TASK_SET = new Set<string>(LIFEOPS_TRAINING_TASKS);
function parseCliArgs(argv: string[]): CliArgs | "help" {
const { values } = parseArgs({
args: argv,
options: {
trajectories: { type: "string" },
samples: { type: "string" },
out: { type: "string" },
task: { type: "string" },
"dry-run": { type: "boolean" },
help: { type: "boolean" },
},
allowPositionals: false,
});
if (values.help) return "help";
const trajectories = values.trajectories?.trim();
if (!trajectories) {
throw new Error("--trajectories <file-or-dir> is required");
}
const samples = Number.parseInt(values.samples ?? "5", 10);
if (!Number.isFinite(samples) || samples <= 0) {
throw new Error("--samples must be a positive integer");
}
const task = values.task?.trim();
if (task && !LIFEOPS_TASK_SET.has(task)) {
throw new Error(
`--task must be one of: ${LIFEOPS_TRAINING_TASKS.join(", ")}`,
);
}
return {
trajectories,
samples,
out: values.out?.trim() || "trajectory-quality-review",
task: task ? (task as LifeOpsTrainingTask) : undefined,
dryRun: values["dry-run"] ?? false,
};
}
async function listJsonFiles(inputPath: string): Promise<string[]> {
const resolved = resolve(process.cwd(), inputPath);
if (!existsSync(resolved)) {
throw new Error(`trajectory path does not exist: ${resolved}`);
}
const stat = statSync(resolved);
if (stat.isFile()) return [resolved];
if (!stat.isDirectory()) {
throw new Error(
`trajectory path is neither file nor directory: ${resolved}`,
);
}
const out: string[] = [];
const stack = [resolved];
while (stack.length > 0) {
const dir = stack.pop();
if (!dir) continue;
for (const entry of await readdir(dir, { withFileTypes: true })) {
const fullPath = join(dir, entry.name);
if (entry.isDirectory()) stack.push(fullPath);
else if (entry.isFile() && entry.name.endsWith(".json"))
out.push(fullPath);
}
}
out.sort();
return out;
}
function isTrajectory(value: unknown): value is Trajectory {
return (
Boolean(value) &&
typeof value === "object" &&
typeof (value as { trajectoryId?: unknown }).trajectoryId === "string"
);
}
async function readTrajectories(inputPath: string): Promise<{
trajectories: Trajectory[];
sourcePathByTrajectoryId: Map<string, string>;
}> {
const files = await listJsonFiles(inputPath);
const trajectories: Trajectory[] = [];
const sourcePathByTrajectoryId = new Map<string, string>();
for (const file of files) {
const parsed = JSON.parse(await readFile(file, "utf8")) as unknown;
const entries = Array.isArray(parsed) ? parsed : [parsed];
for (const entry of entries) {
if (!isTrajectory(entry)) continue;
trajectories.push(entry);
sourcePathByTrajectoryId.set(String(entry.trajectoryId), file);
}
}
return { trajectories, sourcePathByTrajectoryId };
}
/**
* Eval-model judge, configured exactly like the other plugin-training
* scripts: `EVAL_MODEL_PROVIDER` (cerebras default, anthropic supported)
* via the shared lifeops-eval-model helper. Imported dynamically because the
* helper lives in plugin-personal-assistant's test tree, outside this
* package's emit rootDir (same pattern as src/cli/train.ts).
*/
async function makeJudge(): Promise<{
judge: (prompt: string) => Promise<string>;
description: string;
}> {
interface EvalModelModule {
getEvalModelClient(): (req: {
prompt: string;
systemPrompt?: string;
temperature?: number;
maxTokens?: number;
}) => Promise<{ text: string }>;
}
const helperPath =
"../../plugin-personal-assistant/test/helpers/lifeops-eval-model.ts";
const helperModule: EvalModelModule = await import(helperPath);
const client = helperModule.getEvalModelClient();
const provider =
process.env.EVAL_MODEL_PROVIDER?.trim() ||
process.env.EVAL_PROVIDER?.trim() ||
"cerebras";
return {
judge: async (prompt: string) => {
const response = await client({ prompt, temperature: 0, maxTokens: 700 });
return response.text;
},
description: `eval provider ${provider} (EVAL_MODEL_PROVIDER)`,
};
}
function describeSamplePlan(
samplesByTask: Record<LifeOpsTrainingTask, TrajectoryQualitySample[]>,
): string {
const lines: string[] = [];
for (const [task, samples] of Object.entries(samplesByTask)) {
lines.push(` ${task}: ${samples.length} sample(s)`);
for (const sample of samples) {
lines.push(
` - ${sample.trajectoryId} / ${sample.callId}${sample.sourcePath ? ` (${sample.sourcePath})` : ""}`,
);
}
}
return lines.join("\n");
}
async function main(): Promise<number> {
const parsed = parseCliArgs(process.argv.slice(2));
if (parsed === "help") {
process.stdout.write(
"Usage: bun run trajectories:review -- --trajectories <file-or-dir> [--samples <n>] [--task <lifeops-task>] [--out <dir>] [--dry-run]\n",
);
return 0;
}
const { trajectories, sourcePathByTrajectoryId } = await readTrajectories(
parsed.trajectories,
);
if (trajectories.length === 0) {
throw new Error("no recorded trajectory JSON files were found");
}
const tasks = parsed.task ? [parsed.task] : LIFEOPS_TRAINING_TASKS;
const samplesByTask = collectQualitySamples(trajectories, {
samplesPerTask: parsed.samples,
tasks,
sourcePathByTrajectoryId,
});
const sampled = Object.values(samplesByTask).reduce(
(sum, samples) => sum + samples.length,
0,
);
if (parsed.dryRun) {
process.stdout.write(
`[trajectory-quality-review] dry run — would judge ${sampled} sample(s) across ${tasks.length} capability(ies):\n${describeSamplePlan(samplesByTask)}\n`,
);
return 0;
}
if (sampled === 0) {
throw new Error(
"no LifeOps samples found in the corpus — nothing to judge (run with --dry-run to inspect bucketing)",
);
}
const { judge, description } = await makeJudge();
const { judged, failed } = await judgeSamples(samplesByTask, judge);
const review = buildReview({
judgeModel: description,
samplesPerTask: parsed.samples,
sampled,
judged,
failed,
});
const outDir = resolve(process.cwd(), parsed.out);
await mkdir(outDir, { recursive: true });
const jsonPath = join(outDir, "trajectory-quality-review.json");
const markdownPath = join(outDir, "trajectory-quality-review.md");
await writeFile(jsonPath, `${JSON.stringify(review, null, 2)}\n`, "utf8");
await writeFile(markdownPath, renderReviewMarkdown(review), "utf8");
process.stdout.write(
`[trajectory-quality-review] judged ${judged.length}/${sampled} sample(s) (${failed.length} failed judgment(s))\n` +
`[trajectory-quality-review] scoreboard → ${jsonPath}\n` +
`[trajectory-quality-review] markdown → ${markdownPath}\n`,
);
// Fail-closed: unparseable judge output means the review is incomplete.
return failed.length > 0 ? 1 : 0;
}
main()
.then((code) => {
process.exitCode = code;
})
.catch((error) => {
process.stderr.write(
`[trajectory-quality-review] ${error instanceof Error ? error.message : String(error)}\n`,
);
process.exitCode = 1;
});
@@ -0,0 +1,151 @@
/**
* Grid-builder + accuracy-gate coverage for the view-switching verification
* harness, over a hand-built fixture that deliberately omits one
* view×language×modality combination so the grid must report an "absent" cell
* rather than inventing a status. Pure and deterministic — no model, no server.
*/
import { describe, expect, it } from "vitest";
import {
buildResultGrid,
evaluateAccuracyGate,
type GridRow,
} from "./verify-view-switching.ts";
// Hand-built fixture spanning 2 views × 2 languages × 2 modalities with mixed
// pass/fail, deliberately omitting one combination so the grid must report an
// "absent" cell rather than inventing a status.
const ROWS: GridRow[] = [
{ view: "calendar", language: "en", modality: "text", landedOk: true },
{ view: "calendar", language: "en", modality: "voice", landedOk: false },
{ view: "calendar", language: "es", modality: "text", landedOk: true },
// calendar / es / voice intentionally omitted -> absent
{ view: "wallet", language: "en", modality: "text", landedOk: false },
{ view: "wallet", language: "en", modality: "voice", landedOk: true },
{ view: "wallet", language: "es", modality: "text", landedOk: true },
{ view: "wallet", language: "es", modality: "voice", landedOk: true },
];
describe("buildResultGrid", () => {
it("cross-tabulates rows into a per-(view, language, modality) grid", () => {
const result = buildResultGrid(ROWS);
expect(result.views).toEqual(["calendar", "wallet"]);
expect(result.languages).toEqual(["en", "es"]);
expect(result.modalities).toEqual(["text", "voice"]);
// pass / fail / absent are each represented in the nested grid.
expect(result.grid.calendar.en.text).toBe("pass");
expect(result.grid.calendar.en.voice).toBe("fail");
expect(result.grid.calendar.es.text).toBe("pass");
expect(result.grid.calendar.es.voice).toBe("absent");
expect(result.grid.wallet.en.text).toBe("fail");
expect(result.grid.wallet.en.voice).toBe("pass");
expect(result.grid.wallet.es.text).toBe("pass");
expect(result.grid.wallet.es.voice).toBe("pass");
});
it("emits one flat cell per (view × language × modality) combination", () => {
const result = buildResultGrid(ROWS);
// 2 views × 2 languages × 2 modalities = 8 cells, even with one absent.
expect(result.cells).toHaveLength(8);
const cell = (view: string, language: string, modality: string) =>
result.cells.find(
(c) =>
c.view === view && c.language === language && c.modality === modality,
);
expect(cell("calendar", "es", "voice")?.status).toBe("absent");
expect(cell("calendar", "en", "text")?.status).toBe("pass");
expect(cell("wallet", "en", "text")?.status).toBe("fail");
// The flat cells and the nested grid agree everywhere.
for (const c of result.cells) {
expect(result.grid[c.view][c.language][c.modality]).toBe(c.status);
}
});
it("marks a combination as fail when any covering row fails (fail dominates)", () => {
const result = buildResultGrid([
{ view: "inbox", language: "en", modality: "text", landedOk: true },
{ view: "inbox", language: "en", modality: "text", landedOk: false },
]);
expect(result.grid.inbox.en.text).toBe("fail");
expect(result.cells).toHaveLength(1);
});
});
describe("evaluateAccuracyGate", () => {
const summary = {
total: 24,
landedAccuracy: 0.4,
directAccuracy: 0.5,
negativeControlPrecision: 1,
};
it("fails when landedAccuracy is below the floor", () => {
const gate = evaluateAccuracyGate(summary, { minLandedAccuracy: 0.8 });
expect(gate.pass).toBe(false);
expect(gate.failures).toHaveLength(1);
expect(gate.failures[0]).toMatch(
/landedAccuracy 40\.0% below floor 80\.0%/,
);
});
it("passes when landedAccuracy is exactly at the floor", () => {
const gate = evaluateAccuracyGate(
{ ...summary, landedAccuracy: 0.8 },
{ minLandedAccuracy: 0.8 },
);
expect(gate.pass).toBe(true);
expect(gate.failures).toEqual([]);
});
it("passes when landedAccuracy is above the floor", () => {
const gate = evaluateAccuracyGate(
{ ...summary, landedAccuracy: 0.95 },
{ minLandedAccuracy: 0.8 },
);
expect(gate.pass).toBe(true);
expect(gate.failures).toEqual([]);
});
it("passes with no floors configured (opt-in)", () => {
const gate = evaluateAccuracyGate(summary, {});
expect(gate.pass).toBe(true);
expect(gate.failures).toEqual([]);
});
it("gates per-kind direct accuracy and negative-control precision independently", () => {
const gate = evaluateAccuracyGate(summary, {
minDirectAccuracy: 0.9,
minNegativeControlPrecision: 0.9,
});
// direct 0.5 < 0.9 fails; negative-control 1.0 >= 0.9 passes.
expect(gate.pass).toBe(false);
expect(gate.failures).toHaveLength(1);
expect(gate.failures[0]).toMatch(
/directAccuracy 50\.0% below floor 90\.0%/,
);
});
it("collects every failing floor", () => {
const gate = evaluateAccuracyGate(
{ total: 24, landedAccuracy: 0.4, directAccuracy: 0.3 },
{ minLandedAccuracy: 0.8, minDirectAccuracy: 0.9 },
);
expect(gate.pass).toBe(false);
expect(gate.failures).toHaveLength(2);
});
it("flags a gated metric that has no value rather than silently passing", () => {
const gate = evaluateAccuracyGate(
{ total: 24, landedAccuracy: 0.95 },
{ minDirectAccuracy: 0.9 },
);
expect(gate.pass).toBe(false);
expect(gate.failures[0]).toMatch(/directAccuracy: no value/);
});
});
@@ -0,0 +1,578 @@
/**
* Comprehensive view-switching verification harness.
*
* Runs a fixed matrix of natural-language navigation prompts through an
* OpenAI-compatible chat-completions endpoint (local llama.cpp `llama-server`
* OR a cloud provider), constrains the model to the same planner schema the
* runtime uses ({action, view}), then mirrors `runViewsShow`'s end-to-end
* landing logic — the deterministic `resolveIntentView` override corrects a
* wrong/missing model `view` for the known domain surfaces — and scores:
*
* - actionOk : did the model pick the right action (VIEWS vs REPLY)?
* - rawViewOk : did the model's raw `view` param match expected?
* - landedOk : did the user END UP on the expected view (with correction)?
*
* Usage:
* MODEL_URL=http://127.0.0.1:8081/v1 MODEL_LABEL=eliza-1-2b \
* bun run plugins/plugin-training/scripts/verify-view-switching.ts
* MODEL_URL=https://api.anthropic.com/... MODEL_KEY=sk-... MODEL_NAME=claude-... \
* MODEL_LABEL=cloud bun run plugins/plugin-training/scripts/verify-view-switching.ts
*
* Writes a JSON + HTML report under output/view-switching-verify/.
*/
import { mkdirSync, writeFileSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { resolveIntentView } from "../../plugin-app-control/src/actions/views-show.ts";
import { extractPlannerView } from "../src/optimizers/scoring.ts";
// Navigable view ids exposed to the planner (domain surfaces + common builtins).
const VIEW_IDS = [
"chat",
"settings",
"calendar",
"inbox",
"wallet",
"finances",
"focus",
"goals",
"health",
"todos",
"documents",
"relationships",
"task-coordinator",
"help",
"character",
"automations",
"none",
] as const;
// The set the deterministic resolveIntentView override knows about — landing on
// these is auto-corrected even when the model picks a wrong view name (this is
// exactly what runViewsShow does at views-show.ts:372-380). Used to mirror
// end-to-end landing.
const REGISTERED = new Set(VIEW_IDS.filter((v) => v !== "none"));
// Input modality the prompt arrives over. Today every case is typed `text`;
// `voice` exists so transcribed/dictated prompts can be tracked separately in
// the grid without changing the scoring path.
type Modality = "text" | "voice";
interface Case {
prompt: string;
expected: string; // expected landed view id, or "none" for no-nav
kind: "direct" | "passive" | "contextual" | "multilingual" | "negative";
// Grid axes. `view` defaults to `expected`, `language` to "en", `modality` to
// "text" when omitted; multilingual cases carry an explicit BCP-47 language.
view?: string;
language?: string;
modality?: Modality;
}
const CASES: Case[] = [
// direct "open X"
{ prompt: "open settings", expected: "settings", kind: "direct" },
{ prompt: "go to my calendar", expected: "calendar", kind: "direct" },
{ prompt: "open my inbox", expected: "inbox", kind: "direct" },
{ prompt: "show my wallet", expected: "wallet", kind: "direct" },
{ prompt: "open my todos", expected: "todos", kind: "direct" },
{ prompt: "take me to my documents", expected: "documents", kind: "direct" },
{ prompt: "open my goals", expected: "goals", kind: "direct" },
// passive intent (no show-verb)
{
prompt: "what's on my schedule today",
expected: "calendar",
kind: "passive",
},
{ prompt: "check my messages", expected: "inbox", kind: "passive" },
{ prompt: "my crypto balance", expected: "wallet", kind: "passive" },
{
prompt: "how much did i spend on subscriptions",
expected: "finances",
kind: "passive",
},
{
prompt: "i need to focus and block distractions",
expected: "focus",
kind: "passive",
},
{ prompt: "how did i sleep last night", expected: "health", kind: "passive" },
{
prompt: "who do i know at acme corp",
expected: "relationships",
kind: "passive",
},
{ prompt: "change my preferences", expected: "settings", kind: "passive" },
// contextual (situation implies a view)
{
prompt: "i need to fix the login bug in my app",
expected: "task-coordinator",
kind: "contextual",
},
{
prompt: "let's build a new feature for my app",
expected: "task-coordinator",
kind: "contextual",
},
// multilingual
{
prompt: "muéstrame mi calendario",
expected: "calendar",
kind: "multilingual",
language: "es",
},
{
prompt: "abre mi correo",
expected: "inbox",
kind: "multilingual",
language: "es",
},
{
prompt: "我的钱包",
expected: "wallet",
kind: "multilingual",
language: "zh",
},
// negatives (must NOT navigate)
{
prompt: "what's the weather like today",
expected: "none",
kind: "negative",
},
{ prompt: "tell me a joke", expected: "none", kind: "negative" },
{
prompt: "what is the capital of France",
expected: "none",
kind: "negative",
},
];
// Resolve the grid axes for a case, applying the documented defaults.
function caseView(c: Case): string {
return c.view ?? c.expected;
}
function caseLanguage(c: Case): string {
return c.language ?? "en";
}
function caseModality(c: Case): Modality {
return c.modality ?? "text";
}
const SYSTEM_PROMPT = [
"You route a user's chat message to an app view, or reply normally.",
`Available views: ${VIEW_IDS.filter((v) => v !== "none").join(", ")}.`,
"If the message asks to open/show/go to a view, or the situation clearly calls for one, respond with action VIEWS and the best matching view id.",
'If it\'s small talk, a general question, or no view clearly helps, respond with action REPLY and view "none".',
'Respond ONLY as compact JSON: {"action": "VIEWS" or "REPLY", "view": "<one listed view id or none>"}.',
].join("\n");
const PLANNER_SCHEMA = {
type: "object",
properties: {
action: { type: "string", enum: ["VIEWS", "REPLY"] },
view: { type: "string", enum: [...VIEW_IDS] },
},
required: ["action", "view"],
additionalProperties: false,
};
const MODEL_URL = process.env.MODEL_URL ?? "http://127.0.0.1:8081/v1";
const MODEL_LABEL = process.env.MODEL_LABEL ?? "local";
const MODEL_NAME = process.env.MODEL_NAME ?? "eliza-1";
const MODEL_KEY = process.env.MODEL_KEY ?? "sk-no-key";
async function postChat(
prompt: string,
structured: boolean,
): Promise<Response> {
const body: Record<string, unknown> = {
model: MODEL_NAME,
messages: [
{ role: "system", content: SYSTEM_PROMPT },
{ role: "user", content: prompt },
],
temperature: 0,
max_tokens: 60,
chat_template_kwargs: { enable_thinking: false },
};
if (structured) {
body.response_format = {
type: "json_schema",
json_schema: { name: "planner", schema: PLANNER_SCHEMA, strict: true },
};
}
return fetch(`${MODEL_URL}/chat/completions`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${MODEL_KEY}`,
},
body: JSON.stringify(body),
});
}
async function callModel(
prompt: string,
): Promise<{ action: string; view: string; raw: string }> {
let res = await postChat(prompt, true);
// Some providers reject json_schema response_format — retry unconstrained and
// rely on the system-prompt JSON instruction + loose extraction.
if (!res.ok && (res.status === 400 || res.status === 422)) {
res = await postChat(prompt, false);
}
if (!res.ok) throw new Error(`HTTP ${res.status} ${await res.text()}`);
const data = await res.json();
const content = data?.choices?.[0]?.message?.content ?? "";
let action = "REPLY";
let view = "none";
try {
const parsed = JSON.parse(content);
action = String(parsed.action ?? "REPLY").toUpperCase();
view = String(parsed.view ?? "none").toLowerCase();
} catch {
const ev = extractPlannerView(content);
if (ev) {
view = ev;
action = "VIEWS";
}
}
return { action, view, raw: content };
}
// Mirror the full 3-stage cascade end-to-end landing:
// 1. EARLY hook (viewCommandShortcutEvaluator): a rigid matchViewCommand hit
// FORCES the VIEWS action regardless of what the model chose — so explicit
// commands land deterministically, model strength irrelevant.
// 2. ACTION: if the model engaged VIEWS, land on resolveIntentView(prompt) (it
// wraps matchViewCommand) or the model's view param.
// Contextual intent with no rigid match that the model REPLYs to would be
// caught by the POST evaluator (small model) — not simulated here.
function landedView(prompt: string, action: string, modelView: string): string {
// EARLY hook fires on any deterministic resolveIntentView match (rigid
// command OR passive keyword intent), forcing VIEWS model-independently.
const deterministic = resolveIntentView(prompt);
if (deterministic && REGISTERED.has(deterministic)) return deterministic;
if (action !== "VIEWS") return "none";
if (REGISTERED.has(modelView)) return modelView;
return "none";
}
export type GridStatus = "pass" | "fail" | "absent";
// Minimal row shape the grid cross-tabulates over. The verify harness rows are
// a superset of this; tests build the fixture directly from this interface.
export interface GridRow {
view: string;
language: string;
modality: string;
landedOk: boolean;
}
export interface GridCell {
view: string;
language: string;
modality: string;
status: GridStatus;
}
export interface ResultGrid {
views: string[];
languages: string[];
modalities: string[];
// grid[view][language][modality] -> "pass" | "fail" | "absent". A combo with
// no row is "absent"; a combo with any failing row is "fail" (fail dominates).
grid: Record<string, Record<string, Record<string, GridStatus>>>;
cells: GridCell[];
}
// Pure, model-free cross-tabulation of result rows into a per-(view, language,
// modality) pass/fail grid. Every (view × language × modality) combination that
// appears across the axes gets a cell: "absent" when no row covers it, "fail"
// when any covering row failed, otherwise "pass".
export function buildResultGrid(rows: GridRow[]): ResultGrid {
const uniqueSorted = (values: string[]): string[] =>
[...new Set(values)].sort();
const views = uniqueSorted(rows.map((r) => r.view));
const languages = uniqueSorted(rows.map((r) => r.language));
const modalities = uniqueSorted(rows.map((r) => r.modality));
const grid: Record<string, Record<string, Record<string, GridStatus>>> = {};
const cells: GridCell[] = [];
for (const view of views) {
grid[view] = {};
for (const language of languages) {
grid[view][language] = {};
for (const modality of modalities) {
const covering = rows.filter(
(r) =>
r.view === view &&
r.language === language &&
r.modality === modality,
);
const status: GridStatus =
covering.length === 0
? "absent"
: covering.every((r) => r.landedOk)
? "pass"
: "fail";
grid[view][language][modality] = status;
cells.push({ view, language, modality, status });
}
}
}
return { views, languages, modalities, grid, cells };
}
// Accuracy summary the gate scores against. `landedAccuracy` is always present;
// `directAccuracy` (direct "open X" commands) and `negativeControlPrecision`
// (negatives that correctly did NOT navigate) are cheaply derivable from the
// run and let the gate guard the two most regression-prone axes.
export interface AccuracySummary {
total: number;
landedAccuracy: number;
directAccuracy?: number;
negativeControlPrecision?: number;
}
// Floors are opt-in: a metric is only gated when its floor is a finite number.
export interface AccuracyFloors {
minLandedAccuracy?: number;
minDirectAccuracy?: number;
minNegativeControlPrecision?: number;
}
// Pure, model-free accuracy gate. Compares the run's accuracy metrics against
// the provided floors and returns whether every gated metric cleared its floor,
// plus a human-readable failure list. A metric with no floor (or a non-finite
// floor) is not gated. With no floors at all the gate always passes (opt-in).
export function evaluateAccuracyGate(
summary: AccuracySummary,
floors: AccuracyFloors,
): { pass: boolean; failures: string[] } {
const failures: string[] = [];
const check = (
label: string,
actual: number | undefined,
floor: number | undefined,
): void => {
if (floor === undefined || !Number.isFinite(floor)) return;
if (actual === undefined || !Number.isFinite(actual)) {
failures.push(`${label}: no value to compare against floor ${floor}`);
return;
}
if (actual < floor) {
failures.push(
`${label} ${(actual * 100).toFixed(1)}% below floor ${(floor * 100).toFixed(1)}%`,
);
}
};
check("landedAccuracy", summary.landedAccuracy, floors.minLandedAccuracy);
check("directAccuracy", summary.directAccuracy, floors.minDirectAccuracy);
check(
"negativeControlPrecision",
summary.negativeControlPrecision,
floors.minNegativeControlPrecision,
);
return { pass: failures.length === 0, failures };
}
// Read opt-in accuracy floors from the environment. A metric is only gated when
// its env var parses to a finite number.
function readAccuracyFloors(env: NodeJS.ProcessEnv): AccuracyFloors {
const num = (raw: string | undefined): number | undefined => {
if (raw === undefined || raw.trim() === "") return undefined;
const parsed = Number(raw);
return Number.isFinite(parsed) ? parsed : undefined;
};
return {
minLandedAccuracy: num(env.MIN_LANDED_ACCURACY),
minDirectAccuracy: num(env.MIN_DIRECT_ACCURACY),
minNegativeControlPrecision: num(env.MIN_NEGATIVE_CONTROL_PRECISION),
};
}
// True when at least one floor is configured.
function anyFloorConfigured(floors: AccuracyFloors): boolean {
return (
floors.minLandedAccuracy !== undefined ||
floors.minDirectAccuracy !== undefined ||
floors.minNegativeControlPrecision !== undefined
);
}
async function main() {
console.log(
`[verify] model=${MODEL_LABEL} url=${MODEL_URL} name=${MODEL_NAME}`,
);
const rows: Array<{
c: Case;
action: string;
modelView: string;
landed: string;
actionOk: boolean;
rawViewOk: boolean;
landedOk: boolean;
err?: string;
}> = [];
for (const c of CASES) {
try {
const { action, view } = await callModel(c.prompt);
const landed = landedView(c.prompt, action, view);
const expectNav = c.expected !== "none";
const actionOk = expectNav ? action === "VIEWS" : action === "REPLY";
const rawViewOk = expectNav
? view === c.expected
: action === "REPLY" || view === "none";
const landedOk = landed === c.expected;
rows.push({
c,
action,
modelView: view,
landed,
actionOk,
rawViewOk,
landedOk,
});
console.log(
` [${landedOk ? "PASS" : "FAIL"}] ${c.kind.padEnd(12)} "${c.prompt}" → action=${action} view=${view} landed=${landed} (want ${c.expected})`,
);
} catch (e) {
rows.push({
c,
action: "ERR",
modelView: "",
landed: "",
actionOk: false,
rawViewOk: false,
landedOk: false,
err: String(e),
});
console.log(` [ERR ] ${c.prompt}: ${e}`);
}
}
const n = rows.length;
const sum = (k: "actionOk" | "rawViewOk" | "landedOk") =>
rows.filter((r) => r[k]).length;
// Per-kind accuracy on the two most regression-prone axes: direct "open X"
// commands (must land) and negatives (must NOT navigate).
const accuracyOver = (predicate: (r: (typeof rows)[number]) => boolean) => {
const subset = rows.filter(predicate);
return subset.length === 0
? undefined
: subset.filter((r) => r.landedOk).length / subset.length;
};
const directAccuracy = accuracyOver((r) => r.c.kind === "direct");
const negativeControlPrecision = accuracyOver((r) => r.c.kind === "negative");
const summary = {
model: MODEL_LABEL,
modelName: MODEL_NAME,
url: MODEL_URL,
total: n,
actionAccuracy: sum("actionOk") / n,
rawViewAccuracy: sum("rawViewOk") / n,
landedAccuracy: sum("landedOk") / n,
directAccuracy,
negativeControlPrecision,
};
console.log(
`\n[verify] ${MODEL_LABEL}: action ${sum("actionOk")}/${n} | rawView ${sum("rawViewOk")}/${n} | LANDED ${sum("landedOk")}/${n}`,
);
const grid = buildResultGrid(
rows.map((r) => ({
view: caseView(r.c),
language: caseLanguage(r.c),
modality: caseModality(r.c),
landedOk: r.landedOk,
})),
);
const outDir = path.join(process.cwd(), "output", "view-switching-verify");
mkdirSync(outDir, { recursive: true });
const stamp = MODEL_LABEL.replace(/[^a-z0-9_-]/gi, "_");
writeFileSync(
path.join(outDir, `report-${stamp}.json`),
JSON.stringify(
{
summary,
rows: rows.map((r) => ({
...r.c,
action: r.action,
modelView: r.modelView,
landed: r.landed,
actionOk: r.actionOk,
rawViewOk: r.rawViewOk,
landedOk: r.landedOk,
err: r.err,
})),
grid,
},
null,
2,
),
);
const gridCols = grid.languages.flatMap((language) =>
grid.modalities.map((modality) => ({ language, modality })),
);
const gridSym: Record<GridStatus, string> = {
pass: "✓",
fail: "✗",
absent: "·",
};
const gridTable = `<h2>Grid — view × language × modality</h2>
<table class=grid><tr><th>view</th>${gridCols.map((col) => `<th>${col.language}·${col.modality}</th>`).join("")}</tr>
${grid.views
.map(
(view) =>
`<tr><td>${view}</td>${gridCols
.map((col) => {
const s = grid.grid[view][col.language][col.modality];
return `<td class=${s} title="${view} / ${col.language} / ${col.modality}: ${s}">${gridSym[s]}</td>`;
})
.join("")}</tr>`,
)
.join("\n")}
</table>`;
const html = `<!doctype html><meta charset=utf8><title>view-switching ${MODEL_LABEL}</title>
<style>body{font:14px system-ui;margin:24px;background:#111;color:#eee}table{border-collapse:collapse;width:100%;margin-bottom:24px}td,th{border:1px solid #333;padding:6px 8px;text-align:left}.pass{color:#3c3}.fail{color:#f55}.absent{color:#777}table.grid td{text-align:center}h1{font-size:18px}h2{font-size:15px}code{background:#222;padding:1px 4px;border-radius:3px}</style>
<h1>View-switching verification — ${MODEL_LABEL} (${MODEL_NAME})</h1>
<p>Landed: <b>${sum("landedOk")}/${n}</b> (${(summary.landedAccuracy * 100).toFixed(0)}%) · Action: ${sum("actionOk")}/${n} · Raw view: ${sum("rawViewOk")}/${n}</p>
${gridTable}
<table><tr><th>kind</th><th>prompt</th><th>expected</th><th>action</th><th>model view</th><th>landed</th><th>result</th></tr>
${rows.map((r) => `<tr><td>${r.c.kind}</td><td><code>${r.c.prompt}</code></td><td>${r.c.expected}</td><td>${r.action}</td><td>${r.modelView}</td><td>${r.landed}</td><td class=${r.landedOk ? "pass" : "fail"}>${r.landedOk ? "PASS" : "FAIL"}${r.err ? ` ${r.err}` : ""}</td></tr>`).join("\n")}
</table>`;
writeFileSync(path.join(outDir, `report-${stamp}.html`), html);
console.log(
`[verify] wrote output/view-switching-verify/report-${stamp}.{json,html}`,
);
// Opt-in accuracy gate. A model regression must fail the script — but only
// when a floor is configured AND a model endpoint was actually reached (the
// run produced at least one non-error result). Ad-hoc runs with no floor, and
// runs that never reached an endpoint, never fail here.
const floors = readAccuracyFloors(process.env);
if (anyFloorConfigured(floors)) {
const reached = rows.some((r) => r.action !== "ERR");
if (!reached) {
console.warn(
"[verify] accuracy floors set but no model endpoint was reached — skipping gate",
);
return;
}
const gate = evaluateAccuracyGate(summary, floors);
if (!gate.pass) {
console.error(
`[verify] accuracy gate FAILED:\n${gate.failures.map((f) => ` - ${f}`).join("\n")}`,
);
process.exit(1);
}
console.log("[verify] accuracy gate passed");
}
}
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
main().catch((e) => {
console.error(e);
process.exit(1);
});
}