chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,692 @@
|
||||
---
|
||||
title: Adding New Converter Target Providers
|
||||
category: architecture
|
||||
tags: [converter, target-provider, plugin-conversion, multi-platform, pattern]
|
||||
created: 2026-02-23
|
||||
last_refreshed: 2026-06-23
|
||||
severity: medium
|
||||
component: converter-cli
|
||||
problem_type: architecture_pattern
|
||||
root_cause: architectural_pattern
|
||||
---
|
||||
|
||||
# Adding New Converter Target Providers
|
||||
|
||||
## Problem
|
||||
|
||||
When adding support for a new AI platform, the converter CLI architecture requires consistent implementation across types, converters, writers, CLI integration, and tests. Without documented patterns and learnings, new targets take longer to implement and risk architectural inconsistency.
|
||||
|
||||
## Solution
|
||||
|
||||
The compound-engineering-plugin uses a proven **6-phase target provider pattern** that has been applied across converter targets this repo maintains. Some older converter modules remain in tests or cleanup paths for compatibility, but new user-facing installs should prefer native package/plugin mechanisms when the harness supports them.
|
||||
|
||||
1. **OpenCode** (primary target, reference implementation)
|
||||
2. **Codex** (second target, established pattern)
|
||||
3. **Pi** (MCPorter ecosystem)
|
||||
4. **Antigravity CLI** (content transformation patterns)
|
||||
5. **Compatibility converter modules** such as Copilot, Droid, and Kiro (kept where regression coverage or cleanup support still matters)
|
||||
6. **Historical removed targets** such as Windsurf, OpenClaw, Qwen, and Gemini CLI (use as archived lessons only)
|
||||
|
||||
Each implementation follows this architecture precisely, ensuring consistency and maintainability.
|
||||
|
||||
## Architecture: The 6-Phase Pattern
|
||||
|
||||
### Phase 1: Type Definitions (`src/types/{target}.ts`)
|
||||
|
||||
**Purpose:** Define TypeScript types for the intermediate bundle format
|
||||
|
||||
**Key Pattern:**
|
||||
|
||||
```typescript
|
||||
// Exported bundle type used by converter and writer
|
||||
export type {TargetName}Bundle = {
|
||||
// Component arrays matching the target format
|
||||
agents?: {TargetName}Agent[]
|
||||
commands?: {TargetName}Command[]
|
||||
skillDirs?: {TargetName}SkillDir[]
|
||||
mcpServers?: Record<string, {TargetName}McpServer>
|
||||
// Target-specific fields
|
||||
setup?: string // Instructions file content
|
||||
}
|
||||
|
||||
// Individual component types
|
||||
export type {TargetName}Agent = {
|
||||
name: string
|
||||
content: string // Full file content (with frontmatter if applicable)
|
||||
category?: string // e.g., "agent", "rule", "playbook"
|
||||
meta?: Record<string, unknown> // Target-specific metadata
|
||||
}
|
||||
```
|
||||
|
||||
**Key Learnings:**
|
||||
|
||||
- Always include a `content` field (full file text) rather than decomposed fields — it's simpler and matches how files are written
|
||||
- Use intermediate types for complex sections to make section building independently testable
|
||||
- Avoid target-specific fields in the base bundle unless essential — aim for shared structure across targets
|
||||
- Include a `category` field if the target has file-type variants (agents vs. commands vs. rules)
|
||||
|
||||
**Reference Implementations:**
|
||||
- OpenCode: `src/types/opencode.ts` (command + agent split)
|
||||
- Codex: `src/types/codex.ts` (agents plus optional copied skills)
|
||||
- Pi: `src/types/pi.ts` (plugin/extension output)
|
||||
- Antigravity: `src/types/antigravity.ts` (extension-style output)
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Converter (`src/converters/claude-to-{target}.ts`)
|
||||
|
||||
**Purpose:** Transform Claude Code plugin format → target-specific bundle format
|
||||
|
||||
**Key Pattern:**
|
||||
|
||||
```typescript
|
||||
export type ClaudeTo{Target}Options = ClaudeToOpenCodeOptions // Reuse common options
|
||||
|
||||
export function convertClaudeTo{Target}(
|
||||
plugin: ClaudePlugin,
|
||||
_options: ClaudeTo{Target}Options,
|
||||
): {Target}Bundle {
|
||||
// Pre-scan: build maps for cross-reference resolution (agents, commands)
|
||||
// Needed if target requires deduplication or reference tracking
|
||||
const refMap: Record<string, string> = {}
|
||||
for (const agent of plugin.agents) {
|
||||
refMap[normalize(agent.name)] = macroName(agent.name)
|
||||
}
|
||||
|
||||
// Phase 1: Convert agents
|
||||
const agents = plugin.agents.map(a => convert{Target}Agent(a, usedNames, refMap))
|
||||
|
||||
// Phase 2: Convert commands (may depend on agent names for dedup)
|
||||
const commands = plugin.commands.map(c => convert{Target}Command(c, usedNames, refMap))
|
||||
|
||||
// Phase 3: Handle skills (usually pass-through, sometimes conversion)
|
||||
const skillDirs = plugin.skills.map(s => ({ name: s.name, sourceDir: s.sourceDir }))
|
||||
|
||||
// Phase 4: Convert MCP servers (target-specific prefixing/type mapping)
|
||||
const mcpConfig = convertMcpServers(plugin.mcpServers)
|
||||
|
||||
// Phase 5: Warn on unsupported features
|
||||
if (plugin.hooks && Object.keys(plugin.hooks.hooks).length > 0) {
|
||||
console.warn("Warning: {Target} does not support hooks. Hooks were skipped.")
|
||||
}
|
||||
|
||||
return { agents, commands, skillDirs, mcpConfig }
|
||||
}
|
||||
```
|
||||
|
||||
**Content Transformation (`transformContentFor{Target}`):**
|
||||
|
||||
Applied to both agent bodies and command bodies to rewrite paths, command references, and agent mentions:
|
||||
|
||||
```typescript
|
||||
export function transformContentFor{Target}(body: string): string {
|
||||
let result = body
|
||||
|
||||
// 1. Rewrite paths (.claude/ → .github/, ~/.claude/ → ~/.{target}/)
|
||||
result = result
|
||||
.replace(/~\/\.claude\//g, `~/.${targetDir}/`)
|
||||
.replace(/\.claude\//g, `.${targetDir}/`)
|
||||
|
||||
// 2. Transform Task agent calls (to natural language)
|
||||
const taskPattern = /Task\s+([a-z][a-z0-9-]*)\(([^)]+)\)/gm
|
||||
result = result.replace(taskPattern, (_match, agentName: string, args: string) => {
|
||||
const skillName = normalize(agentName)
|
||||
return `Use the ${skillName} skill to: ${args.trim()}`
|
||||
})
|
||||
|
||||
// 3. Flatten slash commands (/workflows:plan → /plan)
|
||||
const slashPattern = /(?<![:\w])\/([a-z][a-z0-9_:-]*?)(?=[\s,."')\]}`]|$)/gi
|
||||
result = result.replace(slashPattern, (match, commandName: string) => {
|
||||
if (commandName.includes("/")) return match // Skip file paths
|
||||
const normalized = normalize(commandName)
|
||||
return `/${normalized}`
|
||||
})
|
||||
|
||||
// 4. Transform @agent-name references
|
||||
const agentPattern = /@([a-z][a-z0-9-]*-(?:agent|reviewer|analyst|...))/gi
|
||||
result = result.replace(agentPattern, (_match, agentName: string) => {
|
||||
return `the ${normalize(agentName)} agent` // or "rule", "playbook", etc.
|
||||
})
|
||||
|
||||
// 5. Remove examples (if target doesn't support them)
|
||||
result = result.replace(/<examples>[\s\S]*?<\/examples>/g, "")
|
||||
|
||||
return result
|
||||
}
|
||||
```
|
||||
|
||||
**Deduplication Pattern (`uniqueName`):**
|
||||
|
||||
Used when target has flat namespaces or when name collisions occur:
|
||||
|
||||
```typescript
|
||||
function uniqueName(base: string, used: Set<string>): string {
|
||||
if (!used.has(base)) {
|
||||
used.add(base)
|
||||
return base
|
||||
}
|
||||
let index = 2
|
||||
while (used.has(`${base}-${index}`)) {
|
||||
index += 1
|
||||
}
|
||||
const name = `${base}-${index}`
|
||||
used.add(name)
|
||||
return name
|
||||
}
|
||||
|
||||
function normalizeName(value: string): string {
|
||||
const trimmed = value.trim()
|
||||
if (!trimmed) return "item"
|
||||
const normalized = trimmed
|
||||
.toLowerCase()
|
||||
.replace(/[\\/]+/g, "-")
|
||||
.replace(/[:\s]+/g, "-")
|
||||
.replace(/[^a-z0-9_-]+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^-+|-+$/g, "")
|
||||
return normalized || "item"
|
||||
}
|
||||
|
||||
// Flatten: drops namespace prefix (workflows:plan → plan)
|
||||
function flattenCommandName(name: string): string {
|
||||
const normalized = normalizeName(name)
|
||||
return normalized.replace(/^[a-z]+-/, "") // Drop prefix before first dash
|
||||
}
|
||||
```
|
||||
|
||||
**Key Learnings:**
|
||||
|
||||
1. **Pre-scan for cross-references** — If target requires reference names (macros, URIs, IDs), build a map before conversion to avoid name collisions and enable deduplication.
|
||||
|
||||
2. **Content transformation is fragile** — Test extensively. Patterns that work for slash commands might false-match on file paths. Use negative lookahead to skip `/etc`, `/usr`, `/var`, etc.
|
||||
|
||||
3. **Simplify heuristics, trust structural mapping** — Don't try to parse agent body for "You are..." or "NEVER do..." patterns. Instead, map agent.description → Overview, agent.body → Procedure, agent.capabilities → Specifications. Heuristics fail on edge cases and are hard to test.
|
||||
|
||||
4. **Normalize early and consistently** — Use the same `normalizeName()` function throughout. Inconsistent normalization causes deduplication bugs.
|
||||
|
||||
5. **MCP servers need target-specific handling:**
|
||||
- **OpenCode:** Merge into `opencode.json` (preserve user keys)
|
||||
- **Pi:** Emit extension/package metadata without requiring CE-owned subagent extensions
|
||||
- **Antigravity:** Use `serverUrl` (not `url`) for remote MCP servers; emit `mcp_config.json` at plugin root
|
||||
|
||||
6. **Warn on unsupported features** — Hooks and target-incompatible MCP types should emit to stderr and continue conversion.
|
||||
|
||||
**Reference Implementations:**
|
||||
- OpenCode: `src/converters/claude-to-opencode.ts` (most comprehensive)
|
||||
- Codex: `src/converters/claude-to-codex.ts` (native-plugin-compatible default with legacy include-skills path)
|
||||
- Pi: `src/converters/claude-to-pi.ts` (Pi plugin metadata)
|
||||
- Antigravity: `src/converters/claude-to-antigravity.ts` (Antigravity plugin output)
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Writer (`src/targets/{target}.ts`)
|
||||
|
||||
**Purpose:** Write converted bundle to disk in target-specific directory structure
|
||||
|
||||
**Key Pattern:**
|
||||
|
||||
```typescript
|
||||
export async function write{Target}Bundle(outputRoot: string, bundle: {Target}Bundle): Promise<void> {
|
||||
const paths = resolve{Target}Paths(outputRoot)
|
||||
await ensureDir(paths.root)
|
||||
|
||||
// Write each component type
|
||||
if (bundle.agents?.length > 0) {
|
||||
const agentsDir = path.join(paths.root, "agents")
|
||||
for (const agent of bundle.agents) {
|
||||
await writeText(path.join(agentsDir, `${agent.name}.ext`), agent.content + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
if (bundle.commands?.length > 0) {
|
||||
const commandsDir = path.join(paths.root, "commands")
|
||||
for (const command of bundle.commands) {
|
||||
await writeText(path.join(commandsDir, `${command.name}.ext`), command.content + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Copy skills (pass-through case)
|
||||
if (bundle.skillDirs?.length > 0) {
|
||||
const skillsDir = path.join(paths.root, "skills")
|
||||
for (const skill of bundle.skillDirs) {
|
||||
await copyDir(skill.sourceDir, path.join(skillsDir, skill.name))
|
||||
}
|
||||
}
|
||||
|
||||
// Write generated skills (converted from commands)
|
||||
if (bundle.generatedSkills?.length > 0) {
|
||||
const skillsDir = path.join(paths.root, "skills")
|
||||
for (const skill of bundle.generatedSkills) {
|
||||
await writeText(path.join(skillsDir, skill.name, "SKILL.md"), skill.content + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Write MCP config (target-specific location and format)
|
||||
if (bundle.mcpServers && Object.keys(bundle.mcpServers).length > 0) {
|
||||
const mcpPath = path.join(paths.root, "mcp.json") // or copilot-mcp-config.json, etc.
|
||||
const backupPath = await backupFile(mcpPath)
|
||||
if (backupPath) {
|
||||
console.log(`Backed up existing MCP config to ${backupPath}`)
|
||||
}
|
||||
await writeJson(mcpPath, { mcpServers: bundle.mcpServers })
|
||||
}
|
||||
|
||||
// Write instructions or setup guides
|
||||
if (bundle.setupInstructions) {
|
||||
const setupPath = path.join(paths.root, "setup-instructions.md")
|
||||
await writeText(setupPath, bundle.setupInstructions + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
// Avoid double-nesting (.target/.target/)
|
||||
function resolve{Target}Paths(outputRoot: string) {
|
||||
const base = path.basename(outputRoot)
|
||||
// If already pointing at .target, write directly into it
|
||||
if (base === ".target") {
|
||||
return { root: outputRoot }
|
||||
}
|
||||
// Otherwise nest under .target
|
||||
return { root: path.join(outputRoot, ".target") }
|
||||
}
|
||||
```
|
||||
|
||||
**Backup Pattern (MCP configs only):**
|
||||
|
||||
MCP configs are often pre-existing and user-edited. Backup before overwrite:
|
||||
|
||||
```typescript
|
||||
// From src/utils/files.ts
|
||||
export async function backupFile(filePath: string): Promise<string | null> {
|
||||
if (!existsSync(filePath)) return null
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, "-")
|
||||
const dirname = path.dirname(filePath)
|
||||
const basename = path.basename(filePath)
|
||||
const ext = path.extname(basename)
|
||||
const name = basename.slice(0, -ext.length)
|
||||
const backupPath = path.join(dirname, `${name}.${timestamp}${ext}`)
|
||||
await copyFile(filePath, backupPath)
|
||||
return backupPath
|
||||
}
|
||||
```
|
||||
|
||||
**Key Learnings:**
|
||||
|
||||
1. **Always check for double-nesting** — If output root is already `.target`, don't nest again. Pattern:
|
||||
```typescript
|
||||
if (path.basename(outputRoot) === ".target") {
|
||||
return { root: outputRoot } // Write directly
|
||||
}
|
||||
return { root: path.join(outputRoot, ".target") } // Nest
|
||||
```
|
||||
|
||||
2. **Use `writeText` and `writeJson` helpers** — These handle directory creation and line endings consistently
|
||||
|
||||
3. **Backup MCP configs before overwriting** — MCP JSON files are often hand-edited. Always backup with timestamp.
|
||||
|
||||
4. **Empty bundles should succeed gracefully** — Don't fail if a component array is empty. Many plugins may have no commands or no skills.
|
||||
|
||||
5. **File extensions matter** — Match target conventions exactly:
|
||||
- OpenCode: `.md` for commands
|
||||
- Codex/Antigravity/Pi: preserve the target's native skill or extension filenames exactly
|
||||
|
||||
6. **Permissions for sensitive files** — MCP config with API keys should use `0o600`:
|
||||
```typescript
|
||||
await writeJson(mcpPath, config, { mode: 0o600 })
|
||||
```
|
||||
|
||||
**Reference Implementations:**
|
||||
- OpenCode: `src/targets/opencode.ts` (merge-preserving workspace/global writer)
|
||||
- Codex: `src/targets/codex.ts` (Codex home writer with optional legacy skill output)
|
||||
- Pi: `src/targets/pi.ts` (Pi plugin output)
|
||||
- Antigravity: `src/targets/antigravity.ts` (Antigravity plugin output)
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: CLI Wiring
|
||||
|
||||
**File: `src/targets/index.ts`**
|
||||
|
||||
Register the new target in the global target registry:
|
||||
|
||||
```typescript
|
||||
import { convertClaudeTo{Target} } from "../converters/claude-to-{target}"
|
||||
import { write{Target}Bundle } from "./{target}"
|
||||
import type { {Target}Bundle } from "../types/{target}"
|
||||
|
||||
export const targets: Record<string, TargetHandler<any>> = {
|
||||
// ... existing targets ...
|
||||
{target}: {
|
||||
name: "{target}",
|
||||
implemented: true,
|
||||
convert: convertClaudeTo{Target} as TargetHandler<{Target}Bundle>["convert"],
|
||||
write: write{Target}Bundle as TargetHandler<{Target}Bundle>["write"],
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**File: `src/commands/convert.ts` and `src/commands/install.ts`**
|
||||
|
||||
Add output root resolution:
|
||||
|
||||
```typescript
|
||||
// In resolveTargetOutputRoot()
|
||||
if (targetName === "{target}") {
|
||||
return path.join(outputRoot, ".{target}")
|
||||
}
|
||||
|
||||
// Update --to flag description
|
||||
const toDescription = "Target format (opencode | codex | pi | antigravity | all)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Personal Sync Support (Historical/Optional)
|
||||
|
||||
The current target registry does not have a separate `src/sync/` layer. Earlier versions used sync modules for personal skills and MCP servers; if a future target reintroduces that behavior, keep it separate from conversion and treat it as optional platform glue.
|
||||
|
||||
If the target supports syncing personal skills and MCP servers, the shape looks like:
|
||||
|
||||
```typescript
|
||||
export async function syncTo{Target}(outputRoot: string): Promise<void> {
|
||||
const personalSkillsDir = path.join(expandHome("~/.claude/skills"))
|
||||
const personalSettings = loadSettings(expandHome("~/.claude/settings.json"))
|
||||
|
||||
const skillsDest = path.join(outputRoot, ".{target}", "skills")
|
||||
await ensureDir(skillsDest)
|
||||
|
||||
// Symlink personal skills
|
||||
if (existsSync(personalSkillsDir)) {
|
||||
const skills = readdirSync(personalSkillsDir)
|
||||
for (const skill of skills) {
|
||||
if (!isValidSkillName(skill)) continue
|
||||
const source = path.join(personalSkillsDir, skill)
|
||||
const dest = path.join(skillsDest, skill)
|
||||
await forceSymlink(source, dest)
|
||||
}
|
||||
}
|
||||
|
||||
// Merge MCP servers if applicable
|
||||
if (personalSettings.mcpServers) {
|
||||
const mcpPath = path.join(outputRoot, ".{target}", "mcp.json")
|
||||
const existing = readJson(mcpPath) || {}
|
||||
const merged = {
|
||||
...existing,
|
||||
mcpServers: {
|
||||
...existing.mcpServers,
|
||||
...personalSettings.mcpServers,
|
||||
},
|
||||
}
|
||||
await writeJson(mcpPath, merged, { mode: 0o600 })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then wire the optional command path in the same explicit-target style as `src/commands/convert.ts`:
|
||||
|
||||
```typescript
|
||||
// Add to validTargets array
|
||||
const validTargets = ["opencode", "codex", "pi", "antigravity", "{target}"] as const
|
||||
|
||||
// In resolveOutputRoot()
|
||||
case "{target}":
|
||||
return path.join(process.cwd(), ".{target}")
|
||||
|
||||
// In main switch
|
||||
case "{target}":
|
||||
await syncTo{Target}(outputRoot)
|
||||
break
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Tests
|
||||
|
||||
**File: `tests/{target}-converter.test.ts`**
|
||||
|
||||
Test converter using inline `ClaudePlugin` fixtures:
|
||||
|
||||
```typescript
|
||||
describe("convertClaudeTo{Target}", () => {
|
||||
it("converts agents to {target} format", () => {
|
||||
const plugin: ClaudePlugin = {
|
||||
name: "test",
|
||||
agents: [
|
||||
{
|
||||
name: "test-agent",
|
||||
description: "Test description",
|
||||
body: "Test body",
|
||||
capabilities: ["Cap 1", "Cap 2"],
|
||||
},
|
||||
],
|
||||
commands: [],
|
||||
skills: [],
|
||||
}
|
||||
|
||||
const bundle = convertClaudeTo{Target}(plugin, {})
|
||||
|
||||
expect(bundle.agents).toHaveLength(1)
|
||||
expect(bundle.agents[0].name).toBe("test-agent")
|
||||
expect(bundle.agents[0].content).toContain("Test description")
|
||||
})
|
||||
|
||||
it("normalizes agent names", () => {
|
||||
const plugin: ClaudePlugin = {
|
||||
name: "test",
|
||||
agents: [
|
||||
{ name: "Test Agent", description: "", body: "", capabilities: [] },
|
||||
],
|
||||
commands: [],
|
||||
skills: [],
|
||||
}
|
||||
|
||||
const bundle = convertClaudeTo{Target}(plugin, {})
|
||||
expect(bundle.agents[0].name).toBe("test-agent")
|
||||
})
|
||||
|
||||
it("deduplicates colliding names", () => {
|
||||
const plugin: ClaudePlugin = {
|
||||
name: "test",
|
||||
agents: [
|
||||
{ name: "Agent Name", description: "", body: "", capabilities: [] },
|
||||
{ name: "Agent Name", description: "", body: "", capabilities: [] },
|
||||
],
|
||||
commands: [],
|
||||
skills: [],
|
||||
}
|
||||
|
||||
const bundle = convertClaudeTo{Target}(plugin, {})
|
||||
expect(bundle.agents.map(a => a.name)).toEqual(["agent-name", "agent-name-2"])
|
||||
})
|
||||
|
||||
it("transforms content paths (.claude → .{target})", () => {
|
||||
const result = transformContentFor{Target}("See ~/.claude/config")
|
||||
expect(result).toContain("~/.{target}/config")
|
||||
})
|
||||
|
||||
it("warns when hooks are present", () => {
|
||||
const spy = jest.spyOn(console, "warn")
|
||||
const plugin: ClaudePlugin = {
|
||||
name: "test",
|
||||
agents: [],
|
||||
commands: [],
|
||||
skills: [],
|
||||
hooks: { hooks: { "file:save": "test" } },
|
||||
}
|
||||
|
||||
convertClaudeTo{Target}(plugin, {})
|
||||
expect(spy).toHaveBeenCalledWith(expect.stringContaining("hooks"))
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
**File: `tests/{target}-writer.test.ts`**
|
||||
|
||||
Test writer using temp directories (from `tmp` package):
|
||||
|
||||
```typescript
|
||||
describe("write{Target}Bundle", () => {
|
||||
it("writes agents to {target} format", async () => {
|
||||
const tmpDir = await tmp.dir()
|
||||
const bundle: {Target}Bundle = {
|
||||
agents: [{ name: "test", content: "# Test\nBody" }],
|
||||
commands: [],
|
||||
skillDirs: [],
|
||||
}
|
||||
|
||||
await write{Target}Bundle(tmpDir.path, bundle)
|
||||
|
||||
const written = readFileSync(path.join(tmpDir.path, ".{target}", "agents", "test.ext"), "utf-8")
|
||||
expect(written).toContain("# Test")
|
||||
})
|
||||
|
||||
it("does not double-nest when output root is .{target}", async () => {
|
||||
const tmpDir = await tmp.dir()
|
||||
const targetDir = path.join(tmpDir.path, ".{target}")
|
||||
await ensureDir(targetDir)
|
||||
|
||||
const bundle: {Target}Bundle = {
|
||||
agents: [{ name: "test", content: "# Test" }],
|
||||
commands: [],
|
||||
skillDirs: [],
|
||||
}
|
||||
|
||||
await write{Target}Bundle(targetDir, bundle)
|
||||
|
||||
// Should write to targetDir directly, not targetDir/.{target}
|
||||
const written = path.join(targetDir, "agents", "test.ext")
|
||||
expect(existsSync(written)).toBe(true)
|
||||
})
|
||||
|
||||
it("backs up existing MCP config", async () => {
|
||||
const tmpDir = await tmp.dir()
|
||||
const mcpPath = path.join(tmpDir.path, ".{target}", "mcp.json")
|
||||
await ensureDir(path.dirname(mcpPath))
|
||||
await writeJson(mcpPath, { existing: true })
|
||||
|
||||
const bundle: {Target}Bundle = {
|
||||
agents: [],
|
||||
commands: [],
|
||||
skillDirs: [],
|
||||
mcpServers: { "test": { command: "test" } },
|
||||
}
|
||||
|
||||
await write{Target}Bundle(tmpDir.path, bundle)
|
||||
|
||||
// Backup should exist
|
||||
const backups = readdirSync(path.dirname(mcpPath)).filter(f => f.includes("mcp") && f.includes("-"))
|
||||
expect(backups.length).toBeGreaterThan(0)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
**Key Testing Patterns:**
|
||||
|
||||
- Test normalization, deduplication, content transformation separately
|
||||
- Use inline plugin fixtures (not file-based)
|
||||
- For writer tests, use temp directories and verify file existence
|
||||
- Test edge cases: empty names, empty bodies, special characters
|
||||
- Test error handling: missing files, permission issues
|
||||
|
||||
---
|
||||
|
||||
## Documentation Requirements
|
||||
|
||||
**File: `docs/specs/{target}.md`**
|
||||
|
||||
Document the target format specification:
|
||||
|
||||
- Last verified date (link to official docs)
|
||||
- Config file locations (project-level vs. user-level)
|
||||
- Agent/command/skill format with field descriptions
|
||||
- MCP configuration structure
|
||||
- Character limits (if any)
|
||||
- Example file
|
||||
|
||||
**File: `README.md`**
|
||||
|
||||
Add to supported targets list and include usage examples.
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls and Solutions
|
||||
|
||||
| Pitfall | Solution |
|
||||
|---------|----------|
|
||||
| **Double-nesting** (`.target/.target/`) | Check `path.basename(outputRoot)` before nesting |
|
||||
| **Inconsistent name normalization** | Use single `normalizeName()` function everywhere |
|
||||
| **Fragile content transformation** | Test regex patterns against edge cases (file paths, URLs) |
|
||||
| **Heuristic section extraction fails** | Use structural mapping (description → Overview, body → Procedure) instead |
|
||||
| **MCP config overwrites user edits** | Always backup with timestamp before overwriting |
|
||||
| **Skill body not loaded** | Verify `ClaudeSkill` has `skillPath` field for file reading |
|
||||
| **Missing deduplication** | Build `usedNames` set before conversion, pass to each converter |
|
||||
| **Unsupported features cause silent loss** | Always warn to stderr (hooks, incompatible MCP types, etc.) |
|
||||
| **Test isolation failures** | Use unique temp directories per test, clean up afterward |
|
||||
| **Command namespace collisions after flattening** | Use `uniqueName()` with deduplication, test multiple collisions |
|
||||
|
||||
---
|
||||
|
||||
## Checklist for Adding a New Target
|
||||
|
||||
Use this checklist when adding a new target provider:
|
||||
|
||||
### Implementation
|
||||
- [ ] Create `src/types/{target}.ts` with bundle and component types
|
||||
- [ ] Implement `src/converters/claude-to-{target}.ts` with converter and content transformer
|
||||
- [ ] Implement `src/targets/{target}.ts` with writer
|
||||
- [ ] Register target in `src/targets/index.ts`
|
||||
- [ ] Update `src/commands/convert.ts` (add output root resolution, update help text)
|
||||
- [ ] Update `src/commands/install.ts` (same as convert.ts)
|
||||
- [ ] (Optional/historical) Add a separate sync command only if the target truly needs personal-skill synchronization outside conversion
|
||||
|
||||
### Testing
|
||||
- [ ] Create `tests/{target}-converter.test.ts` with converter tests
|
||||
- [ ] Create `tests/{target}-writer.test.ts` with writer tests
|
||||
- [ ] (Optional) Create `tests/sync-{target}.test.ts` with sync tests
|
||||
- [ ] Run full test suite: `bun test`
|
||||
- [ ] Manual test: `bun run src/index.ts convert --to {target} .`
|
||||
|
||||
### Documentation
|
||||
- [ ] Create `docs/specs/{target}.md` with format specification
|
||||
- [ ] Update `README.md` with target in list and usage examples
|
||||
- [ ] Do not hand-add release notes; release automation owns GitHub release notes and release-owned versions
|
||||
|
||||
### Version Bumping
|
||||
- [ ] Use a conventional `feat:` or `fix:` title so release automation can infer the right bump
|
||||
- [ ] Do not hand-start or hand-bump release-owned version lines in `package.json` or plugin manifests
|
||||
- [ ] Run `bun run release:validate` if component counts or descriptions changed
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
### Implementation Examples
|
||||
|
||||
**Reference implementations by priority (easiest to hardest):**
|
||||
|
||||
1. **OpenCode** (`src/targets/opencode.ts`, `src/converters/claude-to-opencode.ts`) — Most comprehensive, handles command structure and config merging
|
||||
2. **Codex** (`src/targets/codex.ts`, `src/converters/claude-to-codex.ts`) — Native-plugin-compatible default plus legacy include-skills path
|
||||
3. **Pi** (`src/targets/pi.ts`, `src/converters/claude-to-pi.ts`) — Plugin metadata and Pi extension output
|
||||
4. **Antigravity** (`src/targets/antigravity.ts`, `src/converters/claude-to-antigravity.ts`) — Antigravity plugin output
|
||||
|
||||
### Key Utilities
|
||||
|
||||
- `src/utils/frontmatter.ts` — `formatFrontmatter()` and `parseFrontmatter()`
|
||||
- `src/utils/files.ts` — `writeText()`, `writeJson()`, `copyDir()`, `backupFile()`, `ensureDir()`
|
||||
- `src/utils/resolve-home.ts` — `expandHome()` for `~/.{target}` path resolution
|
||||
|
||||
### Existing Tests
|
||||
|
||||
- `tests/opencode-writer.test.ts` — Writer tests with temp directories and merge behavior
|
||||
- `tests/codex-writer.test.ts` — Codex writer layout and cleanup behavior
|
||||
- `tests/antigravity-writer.test.ts` — Antigravity writer output
|
||||
- `tests/antigravity-converter.test.ts` — Antigravity converter tests
|
||||
- `tests/kiro-writer.test.ts` — Compatibility writer coverage for a historical target
|
||||
|
||||
---
|
||||
|
||||
## Related Files
|
||||
|
||||
- `.claude-plugin/plugin.json` — Version and component counts
|
||||
- `CHANGELOG.md` — Pointer to canonical GitHub release history
|
||||
- `README.md` — Usage examples for all targets
|
||||
- `docs/solutions/plugin-versioning-requirements.md` — Checklist for releases
|
||||
@@ -0,0 +1,452 @@
|
||||
# Building Agent-Friendly CLIs: Practical Principles
|
||||
|
||||
CLIs are a natural fit for agents — text in, text out, composable by design. They're also more practical than MCP for most developer-facing agent work: LLMs already know common CLI tools from training data, so there's no schema overhead. An MCP server can burn tens of thousands of tokens just loading its tool definitions before a single question is asked, while a CLI call costs only the command and its output. MCP earns its complexity when agents need per-user auth and structured governance, but for the tools developers build and use day-to-day, a well-designed CLI is faster, cheaper, and more reliable.
|
||||
|
||||
The details still trip agents up, though: interactive prompts they can't answer, help pages with no examples, error messages that say "invalid input" and nothing else, output that buries useful data in formatting. As agents become real consumers of developer tooling, CLI design needs to account for them explicitly.
|
||||
|
||||
This guide synthesizes ideas from Anthropic's tool-design guidance, the Command Line Interface Guidelines project, CLI-Anything, and practitioner experience into **7 practical principles** for evaluating whether a CLI is merely usable by agents or genuinely well-optimized for them.
|
||||
|
||||
This is not a generic CLI style guide. It is a rubric for CLIs that are intended to work well with AI agents.
|
||||
|
||||
---
|
||||
|
||||
## How to Use This Rubric
|
||||
|
||||
This guide is intentionally opinionated, but it is **not pass/fail**.
|
||||
|
||||
Use each finding to classify the CLI along three levels:
|
||||
|
||||
| Level | Meaning | Typical impact on agents |
|
||||
|---|---|---|
|
||||
| Blocker | Prevents reliable agent use | Hangs, requires human intervention, or makes output hard to recover from |
|
||||
| Friction | Agents can use it, but inefficiently or unreliably | More retries, wasted tokens, brittle parsing, extra tool calls |
|
||||
| Optimization | Improves speed, cost, and robustness | Better agent throughput, lower token cost, fewer corrective loops |
|
||||
|
||||
In practice, you should evaluate commands by **command type**, not only at the CLI level:
|
||||
|
||||
| Command type | Most important principles |
|
||||
|---|---|
|
||||
| Read/query commands | Structured output, bounded output, composability |
|
||||
| Mutating commands | Non-interactive execution, actionable errors, safety, idempotence where feasible |
|
||||
| Streaming/logging commands | Filtering, truncation controls, clean stderr/stdout behavior |
|
||||
| Interactive/bootstrap commands | Automation escape hatch, `--no-input`, scriptable alternatives |
|
||||
| Bulk/export commands | Pagination, range selection, machine-readable output |
|
||||
|
||||
This keeps the rubric practical. For example, idempotence is critical for many mutating commands, but not every `tail -f`-style command needs to satisfy it.
|
||||
|
||||
---
|
||||
|
||||
## The 7 Principles
|
||||
|
||||
| # | Principle | Why it matters |
|
||||
|---|-----------|---------------|
|
||||
| 1 | Non-interactive by default for automation paths | Agents cannot reliably answer prompts or navigate TUI flows |
|
||||
| 2 | Structured, parseable output | Agents need stable data contracts, not presentation formatting |
|
||||
| 3 | Progressive help discovery | Agents explore tools incrementally and benefit from concrete examples |
|
||||
| 4 | Fail fast with actionable errors | Agents recover well when errors tell them exactly how to correct course |
|
||||
| 5 | Safe retries and explicit mutation boundaries | Agents retry, resume, and recover; commands must not make that dangerous |
|
||||
| 6 | Composable and predictable command structure | Agents chain commands and depend on consistent affordances |
|
||||
| 7 | Bounded, high-signal responses | Extra output consumes context, time, and tool budget |
|
||||
|
||||
---
|
||||
|
||||
## 1. Non-Interactive by Default for Automation Paths
|
||||
|
||||
**The principle:** Any command an agent might reasonably automate should be invocable without prompts. Interactive mode can still exist, but it should be a convenience layer, not the only path.
|
||||
|
||||
This principle is strongly supported by the CLI Guidelines project: if stdin is not a TTY, the command should not prompt, and `--no-input` should disable prompting entirely. The broader inference from agent-tooling guidance is straightforward: tools that pause for human intervention are poor fits for autonomous execution.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
# Human at a terminal (TTY detected) — prompts fill in missing inputs
|
||||
$ blog-cli publish
|
||||
? Status? (use arrow keys)
|
||||
draft
|
||||
> published
|
||||
scheduled
|
||||
? Status? published
|
||||
? Path to content: my-post.md
|
||||
Published "My Post" to personal
|
||||
|
||||
# Agent or script (no TTY, or --no-input) — flags only, no prompts
|
||||
$ blog-cli publish --content my-post.md --yes
|
||||
Published "My Post" to personal (post_id: post_8k3m)
|
||||
```
|
||||
|
||||
- `Blocker`: a common automation command cannot run without a prompt
|
||||
- `Friction`: some prompts can be bypassed, but behavior is inconsistent across subcommands
|
||||
- `Optimization`: every automation path supports explicit flags and a global non-interactive mode
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Support `--no-input` or `--non-interactive`
|
||||
- Detect TTY vs non-TTY and never prompt when stdin is not interactive
|
||||
- Support `--yes` / `--force` for confirmation bypass where appropriate
|
||||
- Accept structured input via flags, files, or stdin
|
||||
|
||||
**Evaluation goal:** verify that commands never hang waiting for input in non-interactive execution.
|
||||
|
||||
**One practical check (POSIX shell + Python 3 example):**
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
import subprocess, sys
|
||||
|
||||
cmd = ["blog-cli", "publish", "--content", "my-post.md"]
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
print("exit:", result.returncode)
|
||||
print("PASS: command exited without hanging")
|
||||
except subprocess.TimeoutExpired:
|
||||
print("FAIL: command hung waiting for input")
|
||||
sys.exit(1)
|
||||
PY
|
||||
```
|
||||
|
||||
Adapt the mechanism to your environment. The important part is the test purpose: **detach stdin and enforce a timeout**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Structured, Parseable Output
|
||||
|
||||
**The principle:** Commands that return data should expose a stable machine-readable representation and predictable process semantics.
|
||||
|
||||
Anthropic explicitly recommends returning meaningful context from tools and optimizing tool responses for token efficiency. CLIG explicitly recommends `--json`, clean stdout/stderr separation, and suppressing presentation formatting in non-TTY contexts. This document extends that guidance into a CLI-evaluation rule for agent use.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
# Human-readable
|
||||
$ blog-cli publish --content my-post.md
|
||||
Published "My Post" to personal
|
||||
URL: https://personal.blog.dev/my-post
|
||||
Post ID: post_8k3m
|
||||
|
||||
# Machine-readable
|
||||
$ blog-cli publish --content my-post.md --json
|
||||
{"title":"My Post","url":"https://personal.blog.dev/my-post","post_id":"post_8k3m","status":"published"}
|
||||
```
|
||||
|
||||
- `Blocker`: output is only prose, tables, or ANSI-heavy formatting with no stable parse path
|
||||
- `Friction`: some commands support structured output, but coverage is inconsistent or stderr/stdout are mixed
|
||||
- `Optimization`: all data-bearing commands expose a stable machine-readable mode with useful identifiers
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Support `--json` or another clearly documented machine-readable format on data-bearing commands
|
||||
- Use exit code `0` for success and non-zero for failure
|
||||
- Write result data to stdout and diagnostics/logs/errors to stderr
|
||||
- Return meaningful fields such as names, URLs, status, and IDs
|
||||
- Suppress color, spinners, and decorative output when not attached to a TTY
|
||||
|
||||
**Evaluation goal:** verify that structured output is valid, stable enough to parse, and cleanly separated from diagnostics.
|
||||
|
||||
**One practical check (POSIX shell + Python 3 example):**
|
||||
|
||||
```bash
|
||||
blog-cli publish --content my-post.md --json 2>stderr.txt | python3 -c '
|
||||
import json, sys
|
||||
data = json.load(sys.stdin)
|
||||
required = ["title", "url", "post_id", "status"]
|
||||
missing = [field for field in required if field not in data]
|
||||
sys.exit(1 if missing else 0)
|
||||
'
|
||||
echo "json-valid: $?"
|
||||
test ! -s stderr.txt
|
||||
echo "stderr-empty-on-success: $?"
|
||||
rm -f stderr.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Progressive Help Discovery
|
||||
|
||||
**The principle:** Agents rarely learn a CLI from one giant document. They probe top-level help, then subcommand help, then examples. Help should support that workflow.
|
||||
|
||||
CLIG directly recommends concise help, examples, subcommand help, and linking to deeper docs. Anthropic separately shows that precise tool descriptions and examples materially improve tool-use behavior. The inference here is that CLI help should be designed as layered runtime documentation.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
$ blog-cli --help
|
||||
Usage: blog-cli <command>
|
||||
|
||||
Commands:
|
||||
publish Publish content
|
||||
posts List and manage posts
|
||||
|
||||
$ blog-cli publish --help
|
||||
Publish a markdown file to your blog.
|
||||
|
||||
Options:
|
||||
--content Path to markdown file
|
||||
--status Post status (draft, published, scheduled; default: published)
|
||||
--yes Skip confirmation prompt
|
||||
--json Output as JSON
|
||||
--dry-run Preview without publishing
|
||||
|
||||
Examples:
|
||||
blog-cli publish --content my-post.md
|
||||
blog-cli publish --content my-post.md --status draft
|
||||
blog-cli publish --content my-post.md --dry-run
|
||||
```
|
||||
|
||||
- `Blocker`: subcommands are hard to discover or `--help` is missing/incomplete
|
||||
- `Friction`: help exists but omits concrete invocation patterns or required argument guidance
|
||||
- `Optimization`: help is layered, concise, example-driven, and points to deeper docs when needed
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Top-level help lists commands clearly
|
||||
- Subcommand help includes synopsis, required inputs, key flags, and at least one concrete example for non-trivial commands
|
||||
- Common flags appear near the top
|
||||
- Deeper docs are linked from help where helpful
|
||||
|
||||
**Evaluation goal:** verify that an agent can discover how to invoke a command without leaving the CLI or reading the source code.
|
||||
|
||||
**A better check than `grep example`:**
|
||||
|
||||
For each important subcommand, inspect whether help includes all four of:
|
||||
|
||||
1. A one-line purpose
|
||||
2. A concrete invocation pattern
|
||||
3. Required arguments or required flags
|
||||
4. The most important modifiers or safety flags
|
||||
|
||||
If one of those is missing, treat it as `Friction`. If several are missing, treat it as a `Blocker` for discoverability.
|
||||
|
||||
---
|
||||
|
||||
## 4. Fail Fast with Actionable Errors
|
||||
|
||||
**The principle:** When a command fails, the error should help the agent fix the next attempt.
|
||||
|
||||
This is directly supported by Anthropic's guidance: error responses should communicate specific, actionable improvements rather than opaque codes or tracebacks. CLIG also recommends clear error handling and concise output.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
# Bad
|
||||
$ blog-cli publish
|
||||
Error: missing required arguments
|
||||
|
||||
# Better
|
||||
$ blog-cli publish
|
||||
Error: --content is required.
|
||||
Usage: blog-cli publish --content <file> [--status <status>]
|
||||
Available statuses: draft, published, scheduled
|
||||
Example: blog-cli publish --content my-post.md
|
||||
```
|
||||
|
||||
- `Blocker`: failures are vague, silent, or buried in stack traces
|
||||
- `Friction`: errors mention what failed but not how to correct it
|
||||
- `Optimization`: errors include the correction path, valid values, and nearby examples
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Include the correct syntax or usage pattern
|
||||
- Suggest valid values when validation fails
|
||||
- Validate early, before side effects
|
||||
- Prefer actionable text over raw tracebacks by default
|
||||
|
||||
**Evaluation goal:** verify that a failed invocation tells the next caller how to succeed.
|
||||
|
||||
**One practical check:**
|
||||
|
||||
```bash
|
||||
error_output=$(blog-cli publish 2>&1 >/dev/null)
|
||||
exit_code=$?
|
||||
printf '%s\n' "$error_output"
|
||||
echo "exit=$exit_code"
|
||||
```
|
||||
|
||||
Assess the error against these questions:
|
||||
|
||||
- Does it say what was wrong?
|
||||
- Does it show the correct invocation shape?
|
||||
- Does it suggest valid values or next steps?
|
||||
|
||||
If the answer is only yes to the first question, that is usually `Friction`, not `Optimization`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Safe Retries and Explicit Mutation Boundaries
|
||||
|
||||
**The principle:** Agents retry, resume, and sometimes replay commands. Mutating commands should make that safe when possible, and dangerous mutations should be explicit.
|
||||
|
||||
This section intentionally goes beyond the sources a bit. Anthropic emphasizes clear boundaries, careful tool selection, and annotations for destructive tools; CLIG emphasizes confirmations, `--force`, and `--dry-run`. From an agent-readiness perspective, the practical synthesis is: retries must be safe enough that automation is not reckless.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
# Repeating the same command does not create duplicate work
|
||||
$ blog-cli publish --content my-post.md
|
||||
Published "My Post" to personal (post_id: post_8k3m)
|
||||
|
||||
$ blog-cli publish --content my-post.md
|
||||
Already published "My Post" to personal, no changes (post_id: post_8k3m)
|
||||
|
||||
# Dangerous mutation is explicit
|
||||
$ blog-cli posts delete --slug my-post --confirm
|
||||
```
|
||||
|
||||
- `Blocker`: retrying a mutating command can easily duplicate or corrupt state with no warning
|
||||
- `Friction`: destructive commands are scriptable but offer little preview or state feedback
|
||||
- `Optimization`: retries are safe where feasible, and destructive intent is explicit and inspectable
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Provide `--dry-run` for consequential mutations where feasible
|
||||
- Use explicit destructive flags for dangerous operations
|
||||
- Return enough state in success output to verify what happened
|
||||
- Make duplicate application a no-op or clearly detectable when the domain allows it
|
||||
|
||||
Important scoping note:
|
||||
|
||||
- For **create/update/deploy/apply** commands, idempotence or duplicate detection is usually high-value
|
||||
- For **append/send/trigger/run-now** commands, exact idempotence may be impossible; in those cases, the CLI should at least make mutation boundaries explicit and return audit-friendly identifiers
|
||||
|
||||
**Evaluation goal:** verify that retrying or re-running a command is not surprisingly dangerous.
|
||||
|
||||
**Practical checks:**
|
||||
|
||||
- Run the same low-risk mutating command twice and compare outcomes
|
||||
- Check whether destructive commands expose preview, confirmation-bypass, or explicit-danger affordances
|
||||
- Check whether success output includes identifiers that let an agent determine whether it repeated work
|
||||
|
||||
---
|
||||
|
||||
## 6. Composable and Predictable Command Structure
|
||||
|
||||
**The principle:** Agents solve tasks by chaining commands. They benefit from CLIs that accept stdin, produce clean stdout, and use predictable naming and subcommand structure.
|
||||
|
||||
CLIG strongly supports composition: support stdin/stdout, `-` for pipes, clean stderr separation, and order-independent argument handling where possible. Anthropic separately recommends choosing thoughtful, composable tools instead of forcing agents through many low-level steps. The practical synthesis for CLI evaluation is consistency plus pipeability.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
cat posts.json | blog-cli posts import --stdin
|
||||
blog-cli posts list --json | blog-cli posts validate --stdin
|
||||
blog-cli posts list --status draft --limit 5 --json | jq -r '.[].title'
|
||||
```
|
||||
|
||||
- `Blocker`: commands cannot participate in pipelines or have inconsistent invocation structure
|
||||
- `Friction`: some commands are pipeable, but naming and structure vary unpredictably
|
||||
- `Optimization`: the CLI is easy to chain because inputs, outputs, and subcommand patterns are regular
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Accept input via flags, files, or stdin where that materially helps automation
|
||||
- Support `-` as a stdin/stdout alias when file paths are involved
|
||||
- Keep command structures consistent across related resources
|
||||
- Prefer flags for ambiguous multi-field operations; reserve positional arguments for familiar, conventional cases
|
||||
- Avoid requiring users to remember arbitrary ordering rules for flags and subcommands
|
||||
|
||||
**Evaluation goal:** verify that commands can be chained without brittle adapters or special-case knowledge.
|
||||
|
||||
**Practical checks:**
|
||||
|
||||
- Can a command consume stdin or `-` when input logically comes from another command?
|
||||
- Can output from a data command be piped into another tool without stripping logs or ANSI codes?
|
||||
- Do related commands use similar verb/resource patterns?
|
||||
|
||||
This is a better evaluation axis than requiring a specific grammar such as `resource verb` for every CLI.
|
||||
|
||||
---
|
||||
|
||||
## 7. Bounded, High-Signal Responses
|
||||
|
||||
**The principle:** Agents pay a real cost for every extra line of output. Large outputs are sometimes justified, but the CLI should make narrow, relevant responses the default path.
|
||||
|
||||
This is directly aligned with Anthropic's token-efficiency guidance: use pagination, filtering, truncation, and sensible defaults for large responses, and steer agents toward narrowing strategies. This document adds a practical optimization stance for CLIs: a command may be usable while still being wasteful.
|
||||
|
||||
**What good looks like:**
|
||||
|
||||
```bash
|
||||
# Broad but bounded
|
||||
$ blog-cli posts list --limit 25
|
||||
Showing 25 of 312 posts
|
||||
To narrow results: blog-cli posts list --status published --since 7d --limit 10
|
||||
|
||||
# More precise
|
||||
$ blog-cli posts list --tag javascript --status published --since 30d --limit 10 --json
|
||||
```
|
||||
|
||||
- `Blocker`: a routine query command dumps huge output by default with no narrowing controls
|
||||
- `Friction`: narrowing exists, but defaults are too broad or truncation provides no guidance
|
||||
- `Optimization`: defaults are bounded, filters are obvious, and truncation teaches the next better query
|
||||
|
||||
Recommended traits:
|
||||
|
||||
- Support filtering, pagination, range selection, and limits on potentially large result sets
|
||||
- Provide concise vs detailed response modes where helpful
|
||||
- When truncating, explain how to narrow or page the query
|
||||
- Return semantic identifiers and summaries before raw detail
|
||||
|
||||
On thresholds:
|
||||
|
||||
- A default response comfortably under a few hundred lines is often a strong optimization for agents
|
||||
- A larger default is not automatically wrong if the command is inherently export-oriented or the data volume is intrinsic
|
||||
- For evaluation, prefer asking whether the default is **proportionate to the common task** rather than treating any fixed line count as a hard fail
|
||||
|
||||
**Evaluation goal:** verify that agents can get relevant answers without first paying for an unnecessary data dump.
|
||||
|
||||
**Practical checks:**
|
||||
|
||||
- Compare default output to filtered output and check whether narrowing materially reduces volume
|
||||
- Check whether the command exposes `--limit`, filters, time bounds, selectors, or pagination
|
||||
- If default output is large, check whether the command is explicitly an export/bulk command rather than a routine query surface
|
||||
|
||||
As a heuristic, treat a default output above roughly 500 lines as a likely `Friction` signal unless the command is explicitly bulk-oriented and documented as such.
|
||||
|
||||
---
|
||||
|
||||
## Quick Assessment Checklist
|
||||
|
||||
Use this to evaluate a CLI quickly without pretending every issue is binary:
|
||||
|
||||
| # | Check | What you are testing | Typical severity if missing |
|
||||
|---|-------|----------------------|-----------------------------|
|
||||
| 1 | Non-interactive path | Can the command run with stdin detached and no prompt? | `Blocker` |
|
||||
| 2 | Structured output | Can agents get machine-readable output without scraping prose? | `Blocker` or `Friction` |
|
||||
| 3 | Discoverable help | Can an agent find the invocation shape from `--help` alone? | `Friction` |
|
||||
| 4 | Actionable errors | Does failure teach the next correct invocation? | `Friction` |
|
||||
| 5 | Safe mutation boundaries | Are retries, destructive actions, and previews handled explicitly? | `Blocker` or `Friction` |
|
||||
| 6 | Composition | Can the command participate in pipelines cleanly? | `Friction` |
|
||||
| 7 | Bounded output | Are defaults reasonably scoped for common agent tasks? | `Friction` or `Optimization` |
|
||||
|
||||
---
|
||||
|
||||
## Recommended Evaluation Flow
|
||||
|
||||
When assessing a real CLI, review it in this order:
|
||||
|
||||
1. Pick representative commands by type: one read command, one mutating command, one bulk/logging command, and any intentionally interactive workflow.
|
||||
2. Check for automation blockers first: prompts, unusable help, prose-only output, mixed stdout/stderr.
|
||||
3. Check recovery quality next: error messages, validation, stable identifiers, repeatability.
|
||||
4. Check optimization last: narrowing defaults, concise modes, consistent structure, pipeability.
|
||||
|
||||
This avoids over-penalizing a CLI for missing optimizations before confirming whether agents can use it at all.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary sources
|
||||
|
||||
- [Writing effective tools for agents — Anthropic Engineering](https://www.anthropic.com/engineering/writing-tools-for-agents) — Primary source for tool design guidance around meaningful context, token efficiency, actionable errors, and evaluation-driven optimization.
|
||||
- [Command Line Interface Guidelines](https://clig.dev/) — Primary source for CLI behavior around help, stdout/stderr separation, interactivity, arguments/flags, and composability.
|
||||
- [CLI-Anything](https://clianything.org/) — Useful agent-CLI reference point emphasizing self-description, composability, JSON output, and deterministic behavior. Best treated as a practitioner framework, not a standards source.
|
||||
|
||||
### Additional references
|
||||
|
||||
- [Why CLI is the New MCP — OneUptime](https://oneuptime.com/blog/post/2026-02-03-cli-is-the-new-mcp/view) — Opinionated ecosystem commentary on why CLI remains a strong agent integration surface.
|
||||
- [How to Write a Good Spec for AI Agents — Addy Osmani](https://addyosmani.com/blog/good-spec/) — Relevant to layered documentation and context budgeting, but not a primary source for CLI-specific guidance.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: "Separate host-native browser capabilities from portable fallbacks"
|
||||
date: 2026-07-09
|
||||
category: architecture-patterns
|
||||
module: skills/ce-test-browser
|
||||
problem_type: architecture_pattern
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "A cross-platform skill needs browser automation across app and CLI harnesses"
|
||||
- "An unattended workflow still runs inside a harness with an observable browser"
|
||||
- "A host-native browser exposes an API whose implementation name resembles a prohibited standalone tool"
|
||||
tags:
|
||||
- skill-design
|
||||
- browser-automation
|
||||
- host-native
|
||||
- agent-browser
|
||||
- pipeline
|
||||
- cross-platform
|
||||
---
|
||||
|
||||
# Separate host-native browser capabilities from portable fallbacks
|
||||
|
||||
## Context
|
||||
|
||||
`ce-test-browser` historically mandated `agent-browser` to prevent agents from drifting into standalone Playwright, Puppeteer, or arbitrary browser integrations. That guard also prohibited browser surfaces embedded in or directly owned by app harnesses, even when those surfaces provided the complete testing contract and a better integrated experience.
|
||||
|
||||
The categorical rule also coupled unrelated decisions: `mode:pipeline` meant both "do not block on questions" and "hide the browser." An LFG run inside an app harness is unattended, but its integrated browser can remain observable without interrupting automation.
|
||||
|
||||
## Guidance
|
||||
|
||||
Treat browser choices as three distinct layers:
|
||||
|
||||
1. Prefer a browser surface embedded in or directly owned by the active harness when it supports local navigation, rendered and interactive state inspection, interactions, screenshots, and console-error inspection (`skills/ce-test-browser/SKILL.md:20`). A separately configured browser extension or integration does not qualify as host-native.
|
||||
2. Fall back to the portable `agent-browser` CLI when no qualifying host-native capability exists (`skills/ce-test-browser/SKILL.md:21`).
|
||||
3. Continue prohibiting standalone Playwright, Puppeteer, separately configured browser extensions or MCPs, and ad hoc browser automation (`skills/ce-test-browser/SKILL.md:22`).
|
||||
|
||||
An API named Playwright inside the selected host-native browser is still part of the host capability; implementation vocabulary does not turn it into a standalone Playwright substitution (`skills/ce-test-browser/SKILL.md:22`).
|
||||
|
||||
Select one driver before testing and keep its session, element references, screenshots, and authentication state for the entire run. Permit fallback only during initialization, before the first route is tested (`skills/ce-test-browser/SKILL.md:24`). Put fallback-specific commands and troubleshooting in a conditional reference so host-native runs do not carry irrelevant CLI instructions (`skills/ce-test-browser/references/agent-browser-driver.md:3`).
|
||||
|
||||
Keep orchestration policy independent from driver selection and visibility. Headless or pipeline mode means no blocking questions; it does not require hiding a host-native browser. Integrated browsers can remain visible and non-blocking, while a CLI fallback can run headless (`skills/ce-test-browser/references/pipeline-orchestration.md:3`, `skills/ce-test-browser/references/pipeline-orchestration.md:7`).
|
||||
|
||||
## Why This Matters
|
||||
|
||||
This preserves the original quality guard without handicapping app harnesses. CLI environments retain one portable fallback, while app environments can use the browser surface their harness is designed to control. Separating orchestration, visibility, and driver choice also prevents future mode flags from silently changing capabilities they do not own.
|
||||
|
||||
The rule is testable as a contract rather than a preference. The regression test pins native-first selection, fallback behavior, embedded-Playwright classification, one-driver ownership, pipeline observability, and user-documentation parity (`tests/ce-test-browser-driver-policy.test.ts:9`).
|
||||
|
||||
## When to Apply
|
||||
|
||||
- A skill runs across both app and CLI harnesses.
|
||||
- A host provides an embedded or directly owned capability that is materially different from installing or separately configuring another tool.
|
||||
- A portability fallback remains valuable but should not override a better native experience.
|
||||
- An unattended mode controls questions or error handling but does not inherently require invisible execution.
|
||||
|
||||
## Examples
|
||||
|
||||
Before:
|
||||
|
||||
```markdown
|
||||
Always use agent-browser. Do not use any built-in browser-control tool.
|
||||
Pipeline mode defaults every run to hidden/headless execution.
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```markdown
|
||||
Prefer a qualifying host-native integrated browser. Otherwise fall back to
|
||||
agent-browser. Never introduce a third standalone automation stack.
|
||||
|
||||
Pipeline mode suppresses blocking questions; it does not change driver
|
||||
selection or force an integrated browser to be hidden.
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/compound-refresh-skill-improvements.md` — related capability-first skill-writing guidance; low overlap because it addresses question tools and autonomous maintenance rather than browser-driver selection.
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: A correctness cache needs a COMPLETE, schema-derived invalidation input set
|
||||
date: 2026-06-29
|
||||
category: docs/solutions/best-practices/
|
||||
module: repo-grounding-cache
|
||||
problem_type: best_practice
|
||||
component: tooling
|
||||
severity: high
|
||||
applies_when:
|
||||
- Building an optimization cache whose stale result could change a downstream decision or output
|
||||
- Invalidating a cache by checking whether a set of "input" files changed
|
||||
- Deciding what counts as a cache-busting change vs. an ignorable one
|
||||
tags: [cache, invalidation, correctness, freshness, git, stale-data]
|
||||
---
|
||||
|
||||
# A correctness cache needs a COMPLETE, schema-derived invalidation input set
|
||||
|
||||
## Context
|
||||
|
||||
We cached a question-agnostic "project profile" (stack, deps, license, conventions, topology) keyed by git `<root-sha>/<head-sha>`, and reused it only when the working tree was "clean enough." The cardinal rule: the cache is an optimization that must **never** serve a stale profile that changes a skill's output. "Clean enough" was implemented as a delta check — reuse the entry unless a **profile-input** path is dirty (`git status --porcelain`). The whole correctness of the scheme rests on the *profile-input set* being complete.
|
||||
|
||||
The first version's input set was a hand-picked, JS/Python/Go/Ruby-centric allowlist of manifest filenames. Adversarial review found whole ecosystems missing — `.NET` (`*.csproj`, `*.sln`), Swift/iOS (`Package.swift`, `Podfile`), Deno, modern Python (`uv.lock`, `pdm.lock`), C/C++, Gradle version catalogs. In any of those repos, editing a manifest at an unchanged HEAD would **not** invalidate, and the cache would serve a profile with the old stack/deps — a silent cardinal-rule break.
|
||||
|
||||
## Guidance
|
||||
|
||||
When a cache must never change an output, treat the **invalidation input set as a completeness obligation**, not a convenience list:
|
||||
|
||||
1. **Derive the input set from the cached schema's actual sources**, conservatively, as a *superset*. Every field the cache stores must trace to the files that produce it; if a file feeds the schema, a change to it must invalidate. Over-matching costs a re-derive (cheap); under-matching serves stale (a correctness break).
|
||||
2. **Span ecosystems, not just the ones in front of you.** A hardcoded allowlist will omit the language you don't use today. Use suffix matching for project-file families (`.csproj`/`.fsproj`/`.sln`) and cover the mainstream package managers, deploy descriptors, and CI configs.
|
||||
3. **Catch *new* (untracked) inputs.** `git status --porcelain --untracked-files=all` surfaces a newly-added manifest as `??`; without `--untracked-files=all` git collapses a fully-untracked new directory to `?? dir/` and hides the manifest inside.
|
||||
4. **Count both endpoints of a rename.** `R old -> new` must invalidate on the *source* too — a profile input renamed away (`package.json -> pkg.json`) otherwise drops its invalidation signal.
|
||||
5. **Don't cache cheap-to-recompute, churny, correctness-sensitive data at all.** The `docs/solutions/` index is re-globbed fresh every run rather than cached: a directory listing is ~free, the consuming match reads files fresh anyway, and caching it risked serving a stale match (e.g. missing a just-written learning). Caching it AND invalidating on every write would defeat the cache exactly in the compounding loop. Glob-fresh wins over both alternatives.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
A cache that "usually" invalidates is worse than no cache: it is correct in testing (you test the ecosystems you use) and silently wrong in production for someone else's stack. The failure is invisible — there's no error, just a stale answer fed into a decision. The completeness of the input set is the single load-bearing guarantee; everything else (keying, atomic writes, TTL) is secondary to it.
|
||||
|
||||
The general principle: **bias every ambiguous case toward over-invalidation.** A needless re-derive is a few seconds; a served-stale profile is a wrong verdict, plan, or review.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- The cached value, if stale, would change a downstream decision (a verdict, a plan, a generated artifact) — not just a perf metric.
|
||||
- You invalidate by "did these inputs change?" rather than by content hash.
|
||||
- The set of inputs spans formats/ecosystems you can't fully enumerate from the current repo.
|
||||
|
||||
For a pure latency cache where a stale value is merely slower-correct (not wrong), this rigor is overkill — bound it by TTL and move on.
|
||||
|
||||
## Examples
|
||||
|
||||
```python
|
||||
# Under-complete (silently serves stale in a .NET / Swift / Deno repo):
|
||||
_MANIFEST = {"package.json", "go.mod", "Cargo.toml", "Gemfile", "pyproject.toml", ...}
|
||||
def is_input(p): return os.path.basename(p) in _MANIFEST
|
||||
|
||||
# Complete superset: span ecosystems + suffix-match project files + untracked + rename
|
||||
_MANIFEST = { ...JS, Go, Rust, Ruby, Python(+uv/pdm), PHP, JVM(+catalogs),
|
||||
Swift/iOS(Package.swift, Podfile), .NET(packages.config, *.props),
|
||||
Deno, C/C++, Haskell... }
|
||||
_PROJECT_SUFFIXES = (".csproj", ".fsproj", ".vbproj", ".sln", ".cabal")
|
||||
def is_input(p):
|
||||
b = os.path.basename(p)
|
||||
return (b in _MANIFEST or b.endswith(_PROJECT_SUFFIXES)
|
||||
or (("/" not in p) and b in _ROOT_DOCS)
|
||||
or p.startswith((".cursor/", ".github/workflows/", ".circleci/")))
|
||||
# git status --porcelain --untracked-files=all -> catches new ?? manifests
|
||||
# rename "old -> new" -> append BOTH paths
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/cross-skill-shared-cache-primitive.md` — the cache this rule shipped on
|
||||
- `docs/solutions/best-practices/predictable-tmp-cache-ownership-check.md` — a separate safety property of the same cache
|
||||
- AGENTS.md "Shared Repo-Grounding Profile Cache"
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: "End-to-end learnings from running the full CE pipeline on a substantial feature"
|
||||
date: 2026-04-17
|
||||
category: best-practices
|
||||
module: compound-engineering
|
||||
problem_type: best_practice
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Running ce-brainstorm -> ce-plan -> ce-work -> ce-code-review on any non-trivial feature (more than ~1 unit of implementation work)
|
||||
- Orchestrating the full compound-engineering pipeline end-to-end in a single session
|
||||
- Deciding when to insert document-review passes between pipeline stages
|
||||
- Any feature that introduces a new user-facing flow, especially bulk actions or single-keystroke commitments
|
||||
- Any time a research agent returns a confident architectural recommendation that would add a stage, schema field, or module
|
||||
tags: [compound-engineering, ce-pipeline, ce-brainstorm, ce-plan, ce-work, ce-review, document-review, workflow, hitl, pipeline-discipline]
|
||||
---
|
||||
|
||||
# End-to-end learnings from running the full CE pipeline on a substantial feature
|
||||
|
||||
## Context
|
||||
|
||||
The compound-engineering pipeline is designed as a sequence of progressively more expensive stages: `ce-brainstorm` -> `ce-doc-review` -> `ce-plan` -> `ce-doc-review` -> `ce-work` -> `ce-code-review` -> `ce-resolve-pr-feedback`. Each stage operates on a different artifact (requirements doc, plan doc, diff, PR) and applies a different lens (exploration, critique, execution, synthesis, defense).
|
||||
|
||||
It is tempting, on a substantial feature, to collapse this sequence — jump from a rough idea to implementation, or skip document-review because the plan "looks right." A recent session ran the full pipeline end-to-end on a non-trivial feature: redesigning the Interactive mode of `ce-code-review` with a per-finding walk-through, a compact bulk-action preview, a four-option routing model, and defer-to-tracker integration.
|
||||
|
||||
The cross-cutting insight from that run is that **the pipeline itself compounds**. Issues that would have been cheap to fix at brainstorm time became expensive in PR review; issues document-review caught at plan time would have corrupted implementation if they had slipped through. Each stage catches a different class of problem, and each cheaper stage eliminates issues before they become expensive ones downstream. The value of running the pipeline in full isn't process-for-its-own-sake — it is that the stages are not redundant. They find different things.
|
||||
|
||||
This document codifies the concrete patterns that surfaced repeatedly so future runs — by humans or agents — inherit the lessons instead of rediscovering them.
|
||||
|
||||
---
|
||||
|
||||
## Guidance
|
||||
|
||||
### 1. Sample actual evidence before accepting research-agent claims
|
||||
|
||||
Research agents and sub-agents return confident conclusions. Treat those conclusions as hypotheses, not facts, whenever an architectural decision rides on them. "Did you check?" is the correct response to any recommendation framed as "our analysis shows..." when the downstream cost of being wrong is a new stage, a new schema, or a new module.
|
||||
|
||||
The concrete practice:
|
||||
|
||||
- When a research agent recommends a structural intervention (new stage, new field, new module), name the specific artifacts the claim is derived from.
|
||||
- Sample 10-20 real artifacts across the relevant axes.
|
||||
- Compare what the sampled evidence actually shows to what the research claim asserts.
|
||||
- Update the intervention to match the evidence, not the claim.
|
||||
|
||||
Sampled evidence is often directionally correct but mechanistically wrong — and the mechanism is what determines the fix.
|
||||
|
||||
### 2. Run document-review after brainstorm AND after plan
|
||||
|
||||
Document-review is not a single gate. It operates differently on requirements (is this the right problem, framed coherently?) than on plans (does this design hold together, and does it contradict its own scope?). Skipping either application is a different failure mode:
|
||||
|
||||
- Skipping post-brainstorm doc-review: you plan the wrong thing.
|
||||
- Skipping post-plan doc-review: you implement a plan with internal contradictions.
|
||||
|
||||
Multiple doc-review personas routinely catch architectural contradictions — a unit that adds a schema field the plan's own scope boundary forbade, a feature whose framing undermines its stated goal. These are cheap catches at plan time, expensive in implementation, and nearly unfixable in PR review.
|
||||
|
||||
### 3. Treat "trust the agent" UX options as rubber-stamp vectors
|
||||
|
||||
Any feature that offers a single-keystroke commit-a-lot action is a rubber-stamping risk, regardless of how well it is labeled. If the redesign's goal is *reducing* rubber-stamping, any such action needs a visible plan the user can inspect before executing.
|
||||
|
||||
The pattern:
|
||||
|
||||
- Compact preview grouped by action class (Applying / Filing / Skipping).
|
||||
- Proceed / Cancel gate before execution.
|
||||
- Preview is cheap to render and hard to misuse.
|
||||
|
||||
This is the right surface for *reviewing a pre-computed plan*. It is explicitly the wrong surface for *per-item decisions* — a numbered list with per-item options looks efficient at low volume and collapses working memory at high volume.
|
||||
|
||||
### 4. Distinguish bulk-preview ergonomics from per-item walk-through ergonomics
|
||||
|
||||
Two different review modalities with different affordances:
|
||||
|
||||
| Modality | Good for | Bad for |
|
||||
|---|---|---|
|
||||
| Bulk preview grouped by action | Reviewing a pre-computed plan | Making per-item decisions |
|
||||
| Per-item walk-through | Making per-item decisions | Reviewing dozens of items at once |
|
||||
|
||||
Mixing the two — a numbered list with per-row options — feels dense and efficient until volume hits. Then it breaks. Decide which modality each surface is, and commit.
|
||||
|
||||
### 5. Treat tool/platform caps as structural constraints
|
||||
|
||||
Cross-platform tool limits (e.g., the 4-option cap on `AskUserQuestion`) are not annoyances to route around. They force design decisions. Collapsing a 5-option set into 4 + a follow-up question is architecturally different from a 5-option set. Accept the cap early and design for it; do not fight it in implementation and pay for it later.
|
||||
|
||||
### 6. Never conflate two semantic meanings in one flag
|
||||
|
||||
Flag names that read sensibly in one callsite can be silently wrong in another. The symptom is a flag whose definition ("is X available?") is consistent, but whose *use* answers two different questions ("can we invoke X?" vs. "should we offer X as an option?"). One flag cannot answer both correctly.
|
||||
|
||||
When a flag's meaning depends on the caller, split it (see Example 2 below).
|
||||
|
||||
This pattern recurs in the codebase. Prior instances surfaced during the `batch_confirm` collapse in document-review (session history) — a three-tier routing was collapsed to two because the middle tier conflated "high confidence in the fix" with "needs user judgment." And in the signal-word tightening for plan deepening, where "strengthen" / "confidence gaps" as standalone trigger words conflated targeted-edit intent with holistic-deepening intent, producing false positives until tightened to require "deepen" explicitly.
|
||||
|
||||
### 7. Contract tests assert structure, not prose
|
||||
|
||||
A contract test that pins exact wording becomes a tax on future copy improvements. Every wording refinement breaks the test even though the contract is intact. The philosophy is "regression guard, not authoring ossification."
|
||||
|
||||
Assert: file existence, required section headings, required tokens, regex on distinguishing words. Do not assert: sentence-level wording, punctuation, or phrasing that copy editors will legitimately touch. This parallels the structural-evaluation practice used in skill-creator evals, where assertion names map to concrete fields in the output JSON (`overlap_detected`, `update_not_create`) rather than subjective prose judgments.
|
||||
|
||||
### 8. Don't cite external plugins or tools in durable artifacts
|
||||
|
||||
External references may be useful *in dialogue* during brainstorming — "plugin X's review flow does Y, what if we did Z?" — but should not appear in requirements docs, plan docs, PR descriptions, or commit messages. Artifacts need to stand on their own.
|
||||
|
||||
- Dialogue: "X's design is interesting because..."
|
||||
- Artifact: re-frame the same insight in self-contained terms that do not depend on the reader knowing X.
|
||||
|
||||
The cost of violating this is low-visibility: the artifact reads fine today, but a future reader (or re-user of the pattern) hits an unexplained proper noun with no resolution path.
|
||||
|
||||
### 9. Skill bodies are product code — author them accordingly
|
||||
|
||||
Skills are the instruction substrate for future dispatch. Violations in a skill being shipped propagate into every future invocation. The authoring rules that apply to agent definitions apply equally to skill bodies:
|
||||
|
||||
- Third-person agent voice ("What should the agent do?", not "What should I do?").
|
||||
- Front-load distinguishing words so truncated labels remain differentiable.
|
||||
- Rationale discipline: conditional and late-sequence blocks must explain *why*, not just *what*, because agents landing mid-skill need the reasoning to route correctly.
|
||||
|
||||
### 10. Each pipeline stage catches a different class of issue
|
||||
|
||||
Don't skip stages because "the previous one looked fine." The value distribution across stages:
|
||||
|
||||
| Stage | Catches | Relative cost to fix |
|
||||
|---|---|---|
|
||||
| Brainstorm | Wrong problem, wrong framing | Cheapest |
|
||||
| Doc-review (requirements) | Incoherent requirements, missing constraints | Cheap |
|
||||
| Plan | Wrong design | Medium |
|
||||
| Doc-review (plan) | Self-contradicting plan, scope violations | Medium |
|
||||
| Work | Execution bugs | Expensive |
|
||||
| ce-code-review | Scope drift in implementation | Expensive |
|
||||
| PR review | Subtle semantic conflations (flags, schema, contracts) | Most expensive |
|
||||
|
||||
The stages are not redundant. Each catches things the others structurally cannot.
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Cheaper stages eliminate expensive bugs.** The `sink_available` conflation (Example 2) was caught in PR review; had it shipped, it would have been a user-visible bug in an interactive flow. A hypothetical new "Stage 5b synthesis-time rewrite pass" would have added a persistent stage and per-finding model dispatch to the pipeline had it not been caught at plan time by sampling real artifacts instead of accepting a research claim.
|
||||
- **Document-review finds contradictions authors miss.** The plan draft contained a unit that added a new field to merged findings — a schema change that contradicted the plan's own "no changes to the findings schema" scope boundary. The authors did not see this; multiple doc-review personas did. (session history: this same pattern appears across testing-addressed-gate, universal-planning, and the deepen-plan work — adversarial and scope-guardian reviewers consistently catch scope contradictions.)
|
||||
- **Rubber-stamping risk is invisible without a preview gate.** A compact preview is cheap to implement and hard to misuse. Its absence is invisible until an interactive flow has been rubber-stamped in production. This was the exact failure mode in an earlier LFG-autopilot session where 6 of 7 reviewers scored just below the 80 threshold on legitimately fixable issues and were auto-suppressed.
|
||||
- **Contract tests that ossify prose become a hidden tax on iteration.** Every future wording improvement triggers a false-positive test break, which trains contributors to either skip wording improvements or mechanically update tests without thinking. Neither is the intended outcome.
|
||||
- **Pipelines compound only if run in full.** Running brainstorm-then-work is not compound engineering. It is ad-hoc engineering with extra syntax. The compounding effect comes from stages catching each other's misses.
|
||||
|
||||
---
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Running `ce-brainstorm` -> `ce-plan` -> `ce-work` -> `ce-code-review` on any non-trivial feature (more than ~1 unit of implementation work).
|
||||
- Any feature that introduces a new user-facing flow, especially one with bulk actions, routing decisions, or single-keystroke commitments.
|
||||
- Any time a research agent or sub-agent returns a confident architectural recommendation that would add a stage, a schema field, or a module.
|
||||
- Any PR whose scope boundary is explicitly stated ("no changes to X schema", "no new stages") — doc-review both the requirements and the plan before implementation starts.
|
||||
- Any contract test or snapshot test being written against generated documentation.
|
||||
- Any flag whose name could plausibly answer more than one question.
|
||||
- Any skill body being authored or revised.
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Sampling-over-assumption (Stage 5b → shared-template upgrade)
|
||||
|
||||
**Before** — a research agent asserted "personas will not reliably produce R22-R25 framing." The plan drafted a new Stage 5b synthesis-time rewrite pass to enforce framing post-hoc via a new per-finding model dispatch.
|
||||
|
||||
**Intervention** — user pushback: "are you sure?" Sampled 15+ real review artifacts across 5 personas.
|
||||
|
||||
**Sampled finding** — the research was directionally correct but mechanistically wrong. The actual issues were:
|
||||
|
||||
- Null `why_it_matters` fields in `adversarial` and `api-contract` personas.
|
||||
- Code-structure-first framing (vs. impact-first) in `correctness` and `maintainability` personas.
|
||||
|
||||
**After** — intervention changed from "new per-finding model-dispatch stage" to "one-file shared-template upgrade" (`references/subagent-template.md`). Smaller surface area, cheaper to implement, targets the actual failure modes. No new stage, no recurring per-review model cost.
|
||||
|
||||
This mirrors a prior pattern (session history): in the `feat/plan-review-personas` work, a model-tiering assumption ("Codex probably ignores the `sonnet` param") was challenged with "are you sure other platforms ignore it?" Checking the converter code revealed `model: sonnet` was already propagated to all targets, flipping the design from Claude-Code-only to universal.
|
||||
|
||||
### Example 2: The `sink_available` split
|
||||
|
||||
**Before** — one flag, used in two places with two different meanings:
|
||||
|
||||
```
|
||||
# Detection output
|
||||
{ tracker_name, confidence, sink_available }
|
||||
|
||||
# sink_available definition: "the detected tracker can be invoked"
|
||||
|
||||
# Callsite A — label logic
|
||||
if confidence == "high" and sink_available:
|
||||
label = f"File a {tracker_name} ticket..."
|
||||
else:
|
||||
label = "File a ticket..." # generic
|
||||
|
||||
# Callsite B — no-sink suppression (subtly wrong)
|
||||
if not sink_available:
|
||||
omit_option_C()
|
||||
# Question really being answered: "should we offer Defer at all?"
|
||||
# which is NOT the same as "can we invoke the named tracker?"
|
||||
```
|
||||
|
||||
The bug: when `sink_available = false` for the named tracker but GitHub Issues via `gh` or the harness task primitive *would* work, Callsite B silently drops Defer even though a fallback sink is available.
|
||||
|
||||
**After** — two flags, one meaning each:
|
||||
|
||||
```
|
||||
# Detection output
|
||||
{ tracker_name, confidence, named_sink_available, any_sink_available }
|
||||
|
||||
# named_sink_available — the specifically-named tracker is invokable
|
||||
# any_sink_available — any tier in the fallback chain works
|
||||
|
||||
# Callsite A — label logic uses the narrow flag
|
||||
if confidence == "high" and named_sink_available:
|
||||
label = f"File a {tracker_name} ticket..."
|
||||
elif any_sink_available:
|
||||
label = "File a ticket..." # generic, fallback works
|
||||
# else: option omitted
|
||||
|
||||
# Callsite B — suppression uses the broad flag
|
||||
if not any_sink_available:
|
||||
omit_option_C()
|
||||
```
|
||||
|
||||
The two callsites now answer their respective questions correctly. A repo with no documented tracker but working `gh` correctly offers Defer with a generic label instead of silently suppressing.
|
||||
|
||||
### Example 3: Structural-vs-prose contract test assertion
|
||||
|
||||
**Before:**
|
||||
|
||||
```
|
||||
def test_release_notes_contract():
|
||||
doc = (root / "RELEASE_NOTES.md").read_text()
|
||||
assert "only when one or more fixes landed" in doc
|
||||
assert "applied during the review" in doc
|
||||
```
|
||||
|
||||
Every rephrase of either sentence breaks the test, even when the contract is intact.
|
||||
|
||||
**After:**
|
||||
|
||||
```
|
||||
def test_release_notes_contract():
|
||||
doc_path = root / "RELEASE_NOTES.md"
|
||||
assert doc_path.exists(), "release notes file must be generated"
|
||||
|
||||
doc = doc_path.read_text()
|
||||
|
||||
# Required sections (structural landmarks)
|
||||
assert "## Fixes applied" in doc
|
||||
assert "## Findings deferred" in doc
|
||||
|
||||
# Required distinguishing tokens
|
||||
assert re.search(r"\bfix(es)?\b.*\bland", doc, re.I), \
|
||||
"must describe fixes landing"
|
||||
assert re.search(r"\bdefer(red)?\b", doc, re.I), \
|
||||
"must describe deferrals"
|
||||
```
|
||||
|
||||
Structural landmarks (file exists, section exists, token present) are the contract. Sentence-level wording is not. This matches the structural-evaluation style used in skill-creator evals, where assertion names map to concrete fields in output JSON (`overlap_detected`, `update_not_create`).
|
||||
|
||||
### Example 4: Preview gate for bulk "trust the agent" action
|
||||
|
||||
**Before** — an LFG-style routing option executes the full bulk plan on one keystroke. Looks efficient; is a rubber-stamp vector.
|
||||
|
||||
**After** — LFG presents a compact preview grouped by action class, then gates execution behind explicit Proceed/Cancel:
|
||||
|
||||
```
|
||||
Review plan:
|
||||
|
||||
Applying (3):
|
||||
- src/auth.ts:44 fix stale session on logout
|
||||
- src/auth.ts:112 null-check refresh token
|
||||
- src/api.ts:87 handle 429 retry-after
|
||||
|
||||
Filing (2):
|
||||
- src/ui/modal.tsx:23 a11y focus trap (defer)
|
||||
- src/db/migrate.ts:9 idempotency audit (defer)
|
||||
|
||||
Skipping (1):
|
||||
- docs/README.md:4 prose nit
|
||||
|
||||
[Proceed] [Cancel]
|
||||
```
|
||||
|
||||
The plan is visible. Rubber-stamping is now an explicit, informed act rather than a side effect of UI design.
|
||||
|
||||
### Example 5: External plugin references stay in dialogue
|
||||
|
||||
**Dialogue (acceptable):** "Plugin X's review flow groups findings by file, which works well for their navigation-driven use case. What if we grouped by action class instead, since our Interactive mode is decision-driven?"
|
||||
|
||||
**Artifact (acceptable):** "Findings are grouped by action class (Applying / Filing / Skipping) because Interactive mode is decision-driven: the user's question at this surface is 'what is about to happen?', not 'where in the tree am I?'."
|
||||
|
||||
**Artifact (not acceptable):** "Findings are grouped by action class, similar to plugin X's review flow but adapted for our decision-driven Interactive mode."
|
||||
|
||||
The artifact version stands on its own without the external reference. A future reader does not need to know X to understand the design. *(auto memory [claude]: this rule was applied throughout the ce-code-review redesign session — the requirements doc, plan, and PR description all re-framed externally-inspired patterns in self-contained terms.)*
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- [research-agent-pipeline-separation.md](../skill-design/research-agent-pipeline-separation.md) — Establishes the brainstorm / plan / work stage separation. This learning extends downstream to doc-review, ce-code-review, and resolve-pr-feedback, and focuses on what issues surface at each stage rather than what research dispatches.
|
||||
- [compound-refresh-skill-improvements.md](../skill-design/compound-refresh-skill-improvements.md) — The 6-item skill review checklist is a natural companion for review-time prevention rules, particularly around cross-phase consistency and blind-user-question avoidance.
|
||||
- [beta-promotion-orchestration-contract.md](../skill-design/beta-promotion-orchestration-contract.md) — Contract-tests-enforce-orchestration-assumptions pattern for the ce-code-review surface; direct prior art for structural assertion philosophy.
|
||||
- [git-workflow-skills-need-explicit-state-machines.md](../skill-design/git-workflow-skills-need-explicit-state-machines.md) — Methodologically aligned ("state machine over prose" ≈ "structural assertions over prose"), different domain.
|
||||
- [pass-paths-not-content-to-subagents.md](../skill-design/pass-paths-not-content-to-subagents.md) — Companion for any subagent-template changes, particularly around instruction phrasing.
|
||||
- [codex-delegation-best-practices.md](codex-delegation-best-practices.md) — Canonical example of sampling-evidence-over-assumption at depth (6 evaluation iterations, empirical token measurement).
|
||||
@@ -0,0 +1,204 @@
|
||||
---
|
||||
title: "Codex Delegation Best Practices"
|
||||
date: 2026-04-01
|
||||
category: best-practices
|
||||
module: "Codex delegation / skill design"
|
||||
problem_type: convention
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Designing delegation to external models (Codex, future delegates) in orchestrator skills
|
||||
- Authoring or editing SKILL.md files where token cost matters
|
||||
- Choosing whether to delegate plan execution or implement directly
|
||||
- Writing delegation prompts for secondary agents
|
||||
tags:
|
||||
- codex-delegation
|
||||
- token-economics
|
||||
- skill-design
|
||||
- batching
|
||||
- orchestration-cost
|
||||
- prompt-engineering
|
||||
---
|
||||
|
||||
# Codex Delegation Best Practices
|
||||
|
||||
## Context
|
||||
|
||||
> **Note:** This is a retrospective. The experimental delegation skill it studied (`ce-work-beta`) has since been removed from the plugin. The findings below are preserved as general guidance for designing external-model delegation in any orchestrator skill, not as documentation for a live feature.
|
||||
|
||||
Over six iterations of evaluation building Codex delegation into an experimental `ce-work` delegation mode, we collected quantitative data on the token economics of orchestrating work between Claude Code (the orchestrator) and Codex (the delegated executor). The core question: when does delegating plan units to Codex actually save Claude tokens, and what architectural patterns control the cost?
|
||||
|
||||
The delegation model: the delegating skill receives a plan with N implementation units, then decides whether to execute them directly (standard mode) or delegate them to Codex via `codex exec`. Delegation has a fixed orchestration overhead per batch (prompt file write, codex exec invocation, result classification, commit) of approximately 4-5k Claude tokens. Each unit of code Claude does not write saves roughly 3-5k tokens. The crossover depends on how many units are batched per delegation call.
|
||||
|
||||
The evaluation spanned iterations 1-6, testing small (1-2 units), medium (4 units), large (7 units), and extra-large (10 units) plans in both delegation and standard modes, with real code implementation and test verification in isolated worktrees.
|
||||
|
||||
---
|
||||
|
||||
## Guidance
|
||||
|
||||
### Token Economics
|
||||
|
||||
Delegation has a fixed orchestration cost per batch (~4-5k Claude tokens for prompt generation, codex exec, result classification, and commit) and a variable savings per unit (~3-5k Claude tokens of code-writing avoided). The crossover depends on how many units are batched per call.
|
||||
|
||||
**Crossover by plan size:**
|
||||
|
||||
| Plan size | Units | Delegate tokens | Standard tokens | Overhead | Verdict |
|
||||
|-----------|-------|----------------|-----------------|----------|---------|
|
||||
| Small (bug fix) | 1 | 51k | 38k | +34% | Not worth it for token savings |
|
||||
| Small (new feature) | 1 | 63k | 42k | +50% | Not worth it for token savings |
|
||||
| Medium | 4 | 54k | 53k | +2% | Marginal |
|
||||
| Large | 7 | 62k | 62k | +1% | Break-even |
|
||||
| Extra-large | 10 | 54k | 62k* | **-13%** | Delegation is cheaper |
|
||||
|
||||
*Standard mode extrapolated from 7-unit baseline. The XL delegate cost (54k) is lower than the 7-unit standard cost (62k) because orchestration is amortized over more units per batch.
|
||||
|
||||
**How it scales:** Each additional unit in a batch saves ~3-5k Claude tokens while adding zero orchestration cost. The orchestration is per-batch, not per-unit. A 10-unit plan in 2 batches costs ~8-10k in orchestration regardless of whether those batches contain 5 units or 50 lines of code each.
|
||||
|
||||
**The crossover point is ~5-7 units.** Below that, orchestration overhead dominates. Above it, code-writing savings dominate. Users may still choose delegation below the crossover for cost arbitrage (Codex tokens are cheaper than Claude tokens) or coding preference.
|
||||
|
||||
**Wall clock time cost:** Delegation is 1.7-2.2x slower due to codex exec latency:
|
||||
|
||||
| Plan size | Delegate time | Standard time | Slowdown |
|
||||
|-----------|---------------|---------------|----------|
|
||||
| Medium (4 units) | 353s | 188s | 1.9x |
|
||||
| Large (7 units) | 569s | 254s | 2.2x |
|
||||
| Extra-large (10 units) | 574s | ~300s* | ~1.9x |
|
||||
|
||||
**Test coverage cost:** Without explicit testing guidance in the prompt, Codex produces 15-43% fewer tests than Claude. Adding the `<testing>` section to the prompt closed this gap by ~35% on large plans (see Prompt Engineering section below).
|
||||
|
||||
**Evolution across iterations:**
|
||||
|
||||
| Iteration | Architecture | Medium delegate tokens | Change |
|
||||
|-----------|-------------|----------------------|--------|
|
||||
| 3 | Per-unit loop, all content in SKILL.md body (776 lines) | 58k | Baseline |
|
||||
| 4 | Added optimizations to body (~810 lines) | 79k | +38% (worse — body growth overwhelmed savings) |
|
||||
| 5 | Extracted to reference file, batched model (514 lines) | 61k | -23% from iter-4, back to baseline |
|
||||
| 6 | Added `<testing>` to prompt | 54k | -7% (with better test quality) |
|
||||
|
||||
The key lesson from iteration 4: adding content to the skill body increases cost on every tool call. Optimizations that save a few tool calls but add 50+ lines to the body can be net negative.
|
||||
|
||||
### Skill Body Size is the Multiplicative Cost Driver
|
||||
|
||||
The dominant formula:
|
||||
|
||||
```
|
||||
total_token_cost ~ skill_body_lines x tokens_per_line x num_tool_calls
|
||||
```
|
||||
|
||||
Reducing tool calls helps linearly. Reducing skill body size helps **multiplicatively** because it affects every remaining tool call for the entire session. In iteration 4, adding optimization instructions directly to the SKILL.md body caused a net token *increase* despite the optimizations being structurally sound — the larger body cost more on every subsequent tool call than the optimizations saved.
|
||||
|
||||
**Threshold rule:** Move content to a reference file if it exceeds ~50 lines AND is only used in a minority of invocations. Keep always-needed content in the body.
|
||||
|
||||
### Architecture Patterns That Reduce Cost (Ranked by Impact)
|
||||
|
||||
**1. Extract conditional content to reference files.**
|
||||
Moving delegation-specific content (~250 lines) from the SKILL.md body to `references/codex-delegation-workflow.md` shrank the skill from 776 to 514 lines. This saved ~15k Claude tokens per non-delegation run — a 34% body reduction affecting every tool call. The reference is loaded once, only when delegation is active.
|
||||
|
||||
**2. Batch execution over per-unit execution.**
|
||||
Sending all units (or groups of roughly 5) in a single `codex exec` call reduces orchestration from O(N) to O(ceil(N/batch_size)). For a 10-unit plan: 2 batches x ~4-5k = 8-10k orchestration vs 10 x 4-5k = 40-50k with per-unit delegation.
|
||||
|
||||
**3. Delegate the verify/test-fix loop to Codex.**
|
||||
In the original design, Codex wrote code and the orchestrator independently ran tests to verify. This doubled the verification cost — Claude re-ran the same tests Codex already ran, adding a tool call per batch and classification logic for "completed but verify failed" (a 6th signal in the result table). Moving verification into the delegation prompt ("run tests, fix failures, do not report completed unless tests pass") eliminates that round-trip.
|
||||
|
||||
The safety net is the circuit breaker, not the orchestrator re-running tests. If Codex reports "completed" but the code is actually broken, the failure surfaces at one of three catch points: (1) the result schema — Codex reports "failed" or "partial" when it cannot get tests to pass, triggering rollback; (2) the circuit breaker — 3 consecutive failures disable delegation and fall back to standard mode where Claude implements with full Phase 2 testing guidance; (3) Phase 3 quality check — the full test suite runs before shipping regardless of execution mode. The orchestrator does not need to independently verify each batch because these layered catches prevent bad code from shipping. This is the key design insight: trust the delegate's self-report, protect against systematic failure with the circuit breaker, and verify the whole at the end.
|
||||
|
||||
**4. Cache pre-delegation checks.**
|
||||
Environment guard, CLI availability, and consent checks run once before the first batch, not per-unit or per-batch. These don't change mid-execution.
|
||||
|
||||
**5. Batch scratch cleanup.**
|
||||
Clean up `.context/` delegation artifacts at end-of-plan, not per-unit. Fewer tool calls, same outcome.
|
||||
|
||||
### Plan Quality Enables Good Delegation Decisions
|
||||
|
||||
Every delegation decision — whether to delegate, how to batch, what to include in the prompt — depends on what the plan file provides. The orchestrator can only be as smart as the plan it reads.
|
||||
|
||||
| Plan signal | What it enables |
|
||||
|-------------|----------------|
|
||||
| Unit count and scope | The crossover decision (5-7 unit threshold) |
|
||||
| File lists per unit | "Don't split units that share files" batching rule |
|
||||
| Test scenarios per unit | Forwarded to Codex via the `<testing>` prompt section; thin plan scenarios produce thin Codex tests regardless of prompt engineering |
|
||||
| Verification commands | Become the `<verify>` section; missing verification means Codex cannot confirm its own work |
|
||||
| Triviality signals (Goal, Approach) | Whether delegation is considered at all ("config change" vs "recursive validation engine") |
|
||||
| Dependencies between units | Batch boundary decisions for plans >5 units |
|
||||
|
||||
A well-structured ce-plan output provides all of these. A hand-written requirements doc or TODO list may provide few or none — the delegation logic still works (the skill handles non-standard plans), but the decisions are less informed. For example, without explicit file lists, the batching rule cannot check for shared files; without test scenarios, the Codex prompt's `<testing>` section has nothing to supplement.
|
||||
|
||||
This does not mean delegation requires ce-plan output. It means the quality of delegation improves proportionally with the structure of the plan. Users who invest in structured plans get smarter delegation decisions. Users with lightweight plans get delegation that works but makes conservative choices (e.g., single-batch everything, generic test guidance).
|
||||
|
||||
### Prompt Engineering for Delegation Quality
|
||||
|
||||
Without explicit testing guidance, Codex produces 15-43% fewer tests than Claude. Three prompt additions close this gap:
|
||||
|
||||
**`<testing>` section** — Include Test Scenario Completeness guidance (happy path, edge cases, error paths, integration). This improved Codex test output by ~35% on large plans. Codex implements what the prompt asks; it does not infer quality standards from context.
|
||||
|
||||
**Combined `<verify>` command** — Require running ALL test files in a single command, not per-file. Per-file verification misses cross-file contamination — observed in eval when mocked `globalThis.fetch` in one test file leaked into integration tests running in the same bun process.
|
||||
|
||||
**Light system-wide check** — "If your changes touch callbacks, middleware, or event handlers, verify the interaction chain end-to-end." One sentence that catches architectural issues Codex would otherwise miss.
|
||||
|
||||
### Batching Strategy
|
||||
|
||||
Delegate all units in one batch. If the plan exceeds 5 units, split into batches of roughly 5 — never splitting units that share files. Skip delegation entirely if every unit is trivial.
|
||||
|
||||
Between batches: report progress and continue immediately unless the user intervenes. The checkpoint exists so the user *can* steer, not so they *must*.
|
||||
|
||||
### User Choice Matters
|
||||
|
||||
Users may prefer delegation even when it is not optimal for Claude token savings:
|
||||
|
||||
- **Cost arbitrage** — Codex tokens may be cheaper on their usage plan
|
||||
- **Coding preference** — they may prefer Codex's implementation style for certain tasks
|
||||
- **Usage conservation** — they may want to conserve Claude Code usage specifically
|
||||
|
||||
The `work_delegate_decision` setting (`auto`/`ask`) supports this. In `ask` mode, the skill presents a recommendation with rationale but lets the user override. When recommending against delegation: "Codex delegation active, but these are small changes where the cost of delegating outweighs having Claude Code do them." The user can still choose "Delegate to Codex anyway."
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The naive assumption — that offloading work to a secondary agent always saves the orchestrator tokens — is wrong for small workloads and only becomes true past a specific threshold. Without this data, skill authors will either avoid delegation entirely (missing savings on large plans) or apply it universally (wasting tokens on small plans). The 5-7 unit crossover, derived from six evaluation iterations with real token counts, provides a concrete decision boundary.
|
||||
|
||||
The discovery that skill body size is a multiplicative cost driver changes how skills should be authored across the entire plugin. Every line in a SKILL.md body is paid for on every tool call in the session. This makes "extract rarely-used content to reference files" one of the highest-leverage optimizations available to skill authors, and it reframes the instinct to add helpful content to a skill body as a potential anti-pattern when that content is conditional.
|
||||
|
||||
---
|
||||
|
||||
## When to Apply
|
||||
|
||||
- **Designing delegation in any orchestrator skill:** Use the 5-7 unit crossover as the threshold. Below it, prefer direct execution unless the user explicitly requests delegation.
|
||||
- **Authoring or editing any SKILL.md:** Audit for conditional content blocks exceeding ~50 lines. If they apply to a minority of invocations, extract to reference files.
|
||||
- **Adding optimization or guidance content to a skill:** Measure whether the added body size costs more per-call than the optimization saves. If content is only relevant to a specific execution path, it belongs in a reference file.
|
||||
- **Writing delegation prompts:** Include explicit testing completeness guidance and require unified test execution. Do not assume the delegated agent will infer quality standards.
|
||||
- **Choosing batch sizes:** Use batches of up to roughly 5 units, never splitting units that share files.
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
**Skill body size impact — iteration 4 regression:**
|
||||
|
||||
Iteration 3: SKILL.md at 776 lines. Medium plan (4 units) delegated cost 58k Claude tokens.
|
||||
Iteration 4: Added optimization content to body, SKILL.md grew to ~810 lines. Same plan cost 79k tokens (+38%) despite fewer tool calls. The optimization content was sound but the body growth overwhelmed the savings.
|
||||
Iteration 5: Extracted delegation to reference file, SKILL.md back to 514 lines. Same plan cost 61k tokens — back to iter-3 levels with more features.
|
||||
|
||||
**Delegation decision examples:**
|
||||
|
||||
3-unit plan, all implementation:
|
||||
> Standard mode recommended. These 3 units are below the efficiency threshold. Direct execution uses fewer Claude tokens.
|
||||
|
||||
8-unit plan, mixed implementation and tests:
|
||||
> Delegate. Batch into [units 1-5] and [units 6-8], keeping shared-file units together. Pre-delegation checks run once. Progress reported between batches.
|
||||
|
||||
4-unit plan, all config/renames:
|
||||
> Skip delegation. All units are trivial — orchestration overhead exceeds any benefit.
|
||||
|
||||
4-unit plan, user explicitly requests delegation:
|
||||
> Delegate despite marginal economics. User preference is respected. One batch, standard flow.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- [Codex delegation requirements](../../brainstorms/2026-03-31-codex-delegation-requirements.md) — origin requirements defining the delegation flow
|
||||
- [Codex delegation implementation plan](../../plans/2026-03-31-001-feat-codex-delegation-plan.md) — implementation plan with prompt template and circuit breaker design
|
||||
- [Pass paths not content to subagents](../skill-design/pass-paths-not-content-to-subagents.md) — foundational token efficiency pattern for multi-agent orchestration
|
||||
- [Script-first skill architecture](../skill-design/script-first-skill-architecture.md) — complementary token reduction pattern (60-75% savings by moving processing to scripts)
|
||||
- [Agent-friendly CLI principles](../agent-friendly-cli-principles.md) — CLI design principles relevant to how `codex exec` is consumed
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
title: Conditional visual aids in generated documents and PR descriptions
|
||||
date: 2026-03-29
|
||||
category: best-practices
|
||||
module: compound-engineering plugin skills
|
||||
problem_type: design_pattern
|
||||
component: documentation
|
||||
symptoms:
|
||||
- "Generated documents and PR descriptions lack visual aids that would improve comprehension of complex workflows and relationships"
|
||||
- "No consistent criteria for when to include mermaid diagrams vs ASCII art vs markdown tables"
|
||||
- "Dense prose obscures architectural relationships that a diagram would clarify instantly"
|
||||
- "Downstream consumers recreate visuals from scratch because upstream documents did not include them"
|
||||
root_cause: inadequate_documentation
|
||||
resolution_type: documentation_update
|
||||
severity: low
|
||||
tags:
|
||||
- visual-aids
|
||||
- mermaid
|
||||
- ascii-diagrams
|
||||
- markdown-tables
|
||||
- pr-descriptions
|
||||
- skill-design
|
||||
- document-generation
|
||||
---
|
||||
|
||||
# Conditional visual aids in generated documents and PR descriptions
|
||||
|
||||
## Problem
|
||||
|
||||
AI-generated documents and PR descriptions default to prose-only output, even when the content -- multi-step workflows, behavioral mode comparisons, multi-participant interactions, dependency structures -- would be understood significantly faster with a visual aid. The gap is not "no diagrams." The gap is that there is no principled framework for deciding when a visual aid earns its place, which format to use, and how to calibrate for different output surfaces.
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Readers mentally reconstruct workflows, dependency graphs, or mode differences from dense prose paragraphs
|
||||
- Downstream consumers (ce-plan reading a brainstorm, reviewers reading a PR) create their own visual aids from scratch because the upstream document didn't include them
|
||||
- Plans with 5+ implementation units and non-linear dependencies force readers to scan every unit's Dependencies field to reconstruct the execution graph
|
||||
- System-Wide Impact sections naming multiple interacting surfaces read as a wall of prose when a component diagram would take seconds to scan
|
||||
- PR descriptions for architecturally significant changes are text-only even though they were built from plans that contained visual aids
|
||||
- Simple, linear documents include diagrams that add no comprehension value beyond restating the prose
|
||||
|
||||
---
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
- **Always adding diagrams** -- treating visual aids as mandatory by depth classification, document length, or PR size produces noise. Reflexive diagram inclusion trains readers to skip them.
|
||||
- **Never adding diagrams** -- prose-only output fails when content has branching flows, mode comparisons, or multi-participant interactions. Downstream consumers end up building the visuals themselves.
|
||||
- **Wrong diagram type for the content** -- using a mermaid flow diagram when the value is in rich annotations within each step (CLI commands, decision logic) produces a diagram that strips out the useful detail.
|
||||
- **Wrong abstraction level for the surface** -- code-level detail in a brainstorm diagram is premature. Product-level user flows in a plan's Technical Design section miss the point. Oversized diagrams in a PR description slow down reviewers.
|
||||
- **Size/depth as the trigger** -- gating visual aids on "Standard" or "Deep" depth classification, or on PR line count, produces false positives (long but simple docs get unwanted diagrams) and false negatives (short but complex docs get none).
|
||||
|
||||
---
|
||||
|
||||
## Solution: The Conditional Visual Aid Pattern
|
||||
|
||||
Visual aids are conditional on **content patterns** -- what the content describes -- not on document size, depth classification, or surface type alone. Include a visual aid when the content would be significantly easier to understand with one; skip it when prose already communicates the concept clearly.
|
||||
|
||||
### 1. Content-Pattern Triggers (Not Size/Depth Triggers)
|
||||
|
||||
Whether to include a visual aid depends on WHAT the content describes, not HOW MUCH content there is. A Lightweight brainstorm about a complex workflow may warrant a diagram; a Deep brainstorm about a straightforward feature may not.
|
||||
|
||||
| Content describes... | Visual aid type | Notes |
|
||||
|---|---|---|
|
||||
| Multi-step workflow or process with branching | Flow diagram (mermaid or ASCII) | Shows sequence, branches, decision points |
|
||||
| 3+ behavioral modes, variants, or states | Comparison table (markdown) | Shows how modes differ across dimensions |
|
||||
| 3+ interacting participants (roles, components, services) | Relationship/interaction diagram (mermaid or ASCII) | Shows who talks to whom and in what order |
|
||||
| Multiple competing approaches or alternatives | Comparison table (markdown) | Structured side-by-side evaluation |
|
||||
| 4+ units/stages with non-linear dependencies | Dependency graph (mermaid) | Shows parallelism, fan-in/fan-out, blocking order |
|
||||
| Data pipeline or transformation chain | Data flow sketch (mermaid or ASCII) | Shows input/output transformations |
|
||||
| State-heavy lifecycle | State diagram (mermaid) | Shows transitions and guards |
|
||||
| Before/after performance or behavioral changes | Comparison table (markdown) | Structured quantitative comparison |
|
||||
|
||||
**Why content patterns beat size thresholds:** Size correlates weakly with structural complexity. A 200-line brainstorm about a simple CRUD feature is structurally simple. A 50-line brainstorm about a multi-actor authorization workflow is structurally complex. Pattern-based triggers correctly distinguish these; size-based triggers don't.
|
||||
|
||||
**Universal skip criteria:**
|
||||
- Prose already communicates the concept clearly
|
||||
- Diagram would just restate content in visual form without adding comprehension value
|
||||
- Content is simple and linear with no multi-step flows, mode comparisons, or multi-participant interactions
|
||||
- Visual describes detail at the wrong abstraction level for the surface
|
||||
- Three or fewer items in a straight chain -- text is sufficient
|
||||
- Diagram would be 3 nodes or fewer -- it adds ceremony without comprehension benefit
|
||||
|
||||
### 2. Which Visual Aid to Choose
|
||||
|
||||
```
|
||||
+---------------------------+
|
||||
| Does the content warrant |
|
||||
| a visual aid at all? |
|
||||
+-------------+-------------+
|
||||
|
|
||||
+--------+--------+
|
||||
| |
|
||||
No Yes
|
||||
| |
|
||||
Skip entirely What kind of content?
|
||||
|
|
||||
+--------------------+--------------------+
|
||||
| | |
|
||||
Flows/sequences Comparisons/data Relationships
|
||||
| | |
|
||||
+-----+-----+ Markdown table +-----+-----+
|
||||
| | | |
|
||||
Annotation Simple flow Simple graph Complex
|
||||
density high? (5-15 nodes) (5-15 nodes) spatial
|
||||
| | | layout
|
||||
| Mermaid Mermaid |
|
||||
ASCII ASCII
|
||||
```
|
||||
|
||||
**Mermaid diagrams (default for most flow and relationship content)**
|
||||
|
||||
- Best for: simple flows (5-15 nodes), dependency graphs, sequence diagrams, state diagrams, component diagrams
|
||||
- Strengths: renders as SVG in GitHub; source text readable as fallback in email, Slack, terminal, diff views; standardized syntax; easy to maintain
|
||||
- Limitations: poor at rich in-box annotations; node labels must be concise; awkward for multi-line content within a node
|
||||
- Use `TB` (top-to-bottom) direction for narrow rendering in both SVG and source fallback
|
||||
|
||||
**ASCII/box-drawing diagrams (when annotation density is high)**
|
||||
|
||||
- Best for: annotated flows with CLI commands, decision logic, file paths at each step; multi-column spatial arrangements; layouts where the value is in *annotations within steps*, not just the flow between them
|
||||
- Strengths: renders identically everywhere (no renderer dependency); more expressive for in-box content
|
||||
- Constraints: 80-column max for terminal and diff view compatibility; use vertical stacking to fit
|
||||
- Choose over mermaid when: the diagram's value comes from what's written inside each box, not from the graph shape
|
||||
|
||||
**Markdown tables (structured comparison data)**
|
||||
|
||||
- Best for: mode/variant comparisons (3+ modes), before/after data, decision matrices, approach evaluations, trade-off summaries
|
||||
- Strengths: wrap naturally in renderers; universally supported; dense information in scannable form
|
||||
- Choose for any structured data that maps inputs to outputs or compares items across dimensions
|
||||
|
||||
### 3. Surface-Specific Calibration
|
||||
|
||||
Each output surface has different reading patterns. The trigger bar and diagram density must adjust.
|
||||
|
||||
| Surface | Reading pattern | Trigger bar | Abstraction level | Typical diagram size |
|
||||
|---|---|---|---|---|
|
||||
| Requirements (ce-brainstorm) | Studied deeply | Standard | Conceptual/product-level: user flows, information flows, mode comparisons | 5-20 nodes |
|
||||
| Plan -- Technical Design (ce-plan 3.4) | Studied deeply | Work-characteristic-driven | Solution architecture: component interactions, data flow, state machines | 5-15 nodes |
|
||||
| Plan -- Readability (ce-plan 4.4) | Studied deeply | Standard | Document structure: unit dependencies, impact surfaces, mode overviews | 5-15 nodes |
|
||||
| PR description (git-commit-push-pr) | Scanned quickly | High | Change impact: what changed architecturally, what flows differently | 5-10 nodes |
|
||||
|
||||
Key distinctions:
|
||||
- **Brainstorm**: conceptual level only. No implementation architecture, data schemas, or code structure.
|
||||
- **Plan Technical Design vs. Plan Readability**: Section 3.4 diagrams describe *what's being built*. Section 4.4 diagrams help readers *comprehend the plan document itself*. These are complementary, not overlapping.
|
||||
- **PR description**: highest bar. Only include when the change involves structural complexity a reviewer would struggle to reconstruct from prose alone. Derived from the branch diff, not from upstream plan/brainstorm artifacts.
|
||||
|
||||
### 4. Layout and Cross-Device Optimization
|
||||
|
||||
**TB direction for mermaid.** Top-to-bottom diagrams stay narrow in both rendered SVG and source text fallback. This matters for:
|
||||
- GitHub's PR description view (limited horizontal space)
|
||||
- Side-by-side diff views (source text appears as code block)
|
||||
- Email/Slack notifications (source text is all that renders)
|
||||
|
||||
**80-column max for ASCII.** Terminal windows, diff views, and email clients clip or wrap beyond 80 columns. Use vertical stacking to fit complex content within column limits.
|
||||
|
||||
**Proportionality: 5-15 nodes typical.** Every node should earn its place:
|
||||
- Simple 5-step workflow -> 5-10 nodes
|
||||
- Complex workflow with decision branches -> 15-20 nodes if every node earns its place
|
||||
- PR descriptions trend smaller (5-10 nodes); brainstorms and plans can trend larger
|
||||
- Exceeding 15 should be because the content genuinely has that many meaningful steps
|
||||
|
||||
**Mermaid source as text fallback.** Many consumers first encounter generated documents through contexts that don't render mermaid:
|
||||
- Email notifications of PR descriptions
|
||||
- Slack link previews
|
||||
- Terminal diff views and `git log` output
|
||||
- RSS readers
|
||||
Source text must be readable as text. TB direction and concise node labels help.
|
||||
|
||||
**Inline placement at point of relevance.** Always place visual aids where they help comprehension:
|
||||
- Workflow diagram after Problem Frame, not in a "Diagrams" appendix
|
||||
- Dependency graph before or after Implementation Units heading
|
||||
- Comparison table within the section discussing modes or alternatives
|
||||
- A separate "Diagrams" section invites diagrams for diagrams' sake
|
||||
- Exception: substantial flows (>10 nodes) may warrant their own heading near the point of relevance
|
||||
|
||||
---
|
||||
|
||||
## Why This Works
|
||||
|
||||
The conditional, content-pattern-based approach ties the inclusion decision to an observable property of the content itself, not to a proxy metric. This produces correct decisions at both ends: a short brainstorm about a complex multi-actor workflow gets a diagram (trigger matches); a long brainstorm about a straightforward feature does not (no trigger matches).
|
||||
|
||||
Surface-specific calibration ensures the same core principle -- "include when content patterns warrant it" -- adapts to consumption context. The trigger bar rises and diagram sizes shrink as reading pattern shifts from deep study to quick scanning.
|
||||
|
||||
Self-contained format selection per skill (rather than cross-references) keeps skills independently functional while shared structural patterns (When to include / When to skip / Format selection / Prose-is-authoritative) maintain consistency.
|
||||
|
||||
The prose-is-authoritative invariant resolves the trust problem: when diagram and prose disagree, prose governs. No ambiguity for reviewers or implementers.
|
||||
|
||||
---
|
||||
|
||||
## Prevention
|
||||
|
||||
Concrete guidance for any skill that generates documents with visual aids:
|
||||
|
||||
1. **Use content-pattern triggers, not size/depth gates.** Define an explicit "When to include" table mapping content patterns to visual aid types. Never gate on depth classification or line count.
|
||||
|
||||
2. **Define explicit skip criteria.** Every "When to include" needs a "When to skip." Include at minimum: prose already clear, diagram would restate without value, content is simple/linear, visual is at wrong abstraction level.
|
||||
|
||||
3. **Make format selection self-contained per skill.** Each skill contains its own format guidance (mermaid, ASCII, markdown tables) with surface-appropriate calibration. Don't cross-reference other skills' guidance.
|
||||
|
||||
4. **Calibrate to the surface's reading pattern.** Define trigger bar relative to consumption context. Studied surfaces get standard bar; scanned surfaces get higher bar with smaller diagrams.
|
||||
|
||||
5. **Specify the abstraction level.** State what detail level belongs in visual aids for this surface. "Conceptual level only -- not implementation architecture" is the brainstorm example.
|
||||
|
||||
6. **Enforce prose-is-authoritative.** State that when visual aid and prose disagree, prose governs. Cross-skill invariant.
|
||||
|
||||
7. **Require post-generation accuracy check.** After generating any visual aid, verify it matches surrounding content -- correct sequence, no missing branches, no merged steps, no omitted participants.
|
||||
|
||||
8. **Use TB direction for mermaid, 80-column max for ASCII.** Layout constraints for cross-device compatibility.
|
||||
|
||||
9. **Place inline at point of relevance.** Never create a separate "Diagrams" section.
|
||||
|
||||
10. **Keep diagrams proportionate.** Every node earns its place. 5-15 nodes typical. Exceed 15 only for genuinely complex content.
|
||||
|
||||
---
|
||||
|
||||
## Related Issues
|
||||
|
||||
- `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines.md` -- related but distinct: covers git-commit-push-pr state machine correctness, not output content quality
|
||||
- GitHub issue #44 -- mermaid dark mode rendering, relevant when considering diagram styling
|
||||
- PR #437 -- ce-brainstorm visual aids implementation
|
||||
- PR #440 -- ce-plan visual aids implementation
|
||||
- `docs/plans/2026-03-29-003-feat-pr-description-visual-aids-plan.md` -- git-commit-push-pr visual aids plan
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: A predictable-path cache in shared /tmp is a prompt-injection vector — ownership-check reads
|
||||
date: 2026-06-29
|
||||
category: docs/solutions/best-practices/
|
||||
module: repo-grounding-cache
|
||||
problem_type: best_practice
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Writing a cache or scratch file to a world-shared location (/tmp) at a predictable path
|
||||
- The cached content is later fed into an LLM/agent context
|
||||
- Running on a multi-user host where a local co-tenant could pre-create files
|
||||
tags: [security, prompt-injection, cache, tmp, file-ownership, shared-host]
|
||||
---
|
||||
|
||||
# A predictable-path cache in shared /tmp is a prompt-injection vector — ownership-check reads
|
||||
|
||||
## Context
|
||||
|
||||
The repo-grounding cache stored profiles at `/tmp/compound-engineering/repo-profile/<root-sha>/<head-sha>.json`. Both SHAs are knowable for any public or shared repo (the root commit and HEAD), and `/tmp/compound-engineering/` is world-traversable. Security review found: on a multi-user host, a local co-tenant can **pre-create** that exact path and plant a `<head-sha>.json` that satisfies every validity gate (it sets `head_sha` to the victim's HEAD, the current schema version, and a `profile` object — cleanliness is checked against the victim's working tree, not the file's authenticity). The victim's skill then prints `HIT` and feeds the attacker's JSON into the agent as the "project profile" — attacker-controlled text into the LLM context, i.e. **indirect prompt injection**.
|
||||
|
||||
Impact is calibrated (needs a local co-tenant + predictable SHAs; payload is text, not code-exec or secret disclosure), but the injection elevation is why it is worth fixing rather than dismissing as "just repo metadata."
|
||||
|
||||
## Guidance
|
||||
|
||||
When a cache/scratch file in shared `/tmp` will be **read back into an agent's context**, do not trust it by path + content gates alone — verify it is **yours**:
|
||||
|
||||
- **Reject any cache file not owned by the current user.** After opening, `os.fstat(fd).st_uid != os.geteuid()` -> treat as a miss and re-derive. Check via the *opened fd* (`fstat`), not a pre-open `stat`, so it also defeats a symlink an attacker planted pointing at a file they own. Guard the check where `geteuid` is unavailable (non-POSIX) — the shared-`/tmp` threat doesn't apply there.
|
||||
- This composes with the cache's existing principle that it is **never a correctness dependency**: a rejected entry simply degrades to "derive fresh," never blocks.
|
||||
- Write side is already safe if you use `tempfile.mkstemp` (`O_EXCL`, mode `0600`) + `os.replace` (atomic). The exposure is purely on the *read* path.
|
||||
|
||||
Alternatives considered and why ownership-check won: per-uid namespacing the cache root (`/tmp/compound-engineering-$(id -u)/...`) also works but deviates from the project's `/tmp/compound-engineering/` convention and the deliberate choice of `/tmp` over `$TMPDIR` for user-inspectability. The fstat-on-read check is minimal, keeps the path convention, and closes both the planted-file and planted-symlink cases.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Predictable paths in shared `/tmp` are a classic local attack surface, and the usual framing ("it's just a cache, low impact") misses the new twist: **anything fed into an LLM context is an injection sink.** A cache of benign-looking metadata becomes an attacker-controlled-text channel into the model. The data being "non-sensitive" does not bound the risk when the data is *instructions-adjacent*.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any agent/skill that reads a `/tmp` (or other shared-dir) file at a guessable path into model context.
|
||||
- Caches keyed by values an attacker can compute (commit SHAs, repo names, usernames).
|
||||
|
||||
Not needed for per-run `mktemp -d` scratch with an unguessable path consumed only within the same process, or for files never surfaced to the model.
|
||||
|
||||
## Examples
|
||||
|
||||
```python
|
||||
# Vulnerable: gates check authenticity-irrelevant facts (head_sha/schema/cleanliness),
|
||||
# never WHO wrote the file.
|
||||
with open(path) as f:
|
||||
doc = json.load(f)
|
||||
# ... print("HIT"); print(doc["profile"]) # attacker text -> agent context
|
||||
|
||||
# Fixed: reject a file we don't own (defeats planted file AND planted symlink via fstat-on-fd).
|
||||
with open(path) as f:
|
||||
geteuid = getattr(os, "geteuid", None)
|
||||
if geteuid is not None and os.fstat(f.fileno()).st_uid != geteuid():
|
||||
return miss()
|
||||
doc = json.load(f)
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/cross-skill-shared-cache-primitive.md` — the cache this hardened
|
||||
- `docs/solutions/best-practices/cache-invalidation-input-set-completeness.md` — the cache's correctness (separate) property
|
||||
- AGENTS.md "Scratch Space" (the `/tmp/compound-engineering/` convention)
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: "Prefer Python over bash for multi-step pipeline scripts"
|
||||
date: 2026-04-09
|
||||
last_refreshed: 2026-06-20
|
||||
category: best-practices
|
||||
module: "skill scripting / historical ce-demo-reel"
|
||||
problem_type: tooling_decision
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Script orchestrates 2+ external CLI tools (ffmpeg, curl, silicon, vhs)
|
||||
- Script needs retry logic or graceful degradation on tool failure
|
||||
- Script will run on macOS where bash 3.2 is the default
|
||||
- Script needs to be tested from a non-shell test runner (Bun, Jest, pytest)
|
||||
- Script has conditional failure paths where some errors should be caught and others should abort
|
||||
tags:
|
||||
- bash-vs-python
|
||||
- pipeline-scripts
|
||||
- skill-scripting
|
||||
- set-e-footguns
|
||||
- error-handling
|
||||
- ce-demo-reel
|
||||
---
|
||||
|
||||
# Prefer Python over bash for multi-step pipeline scripts
|
||||
|
||||
## Context
|
||||
|
||||
When building the now-removed `ce-demo-reel` skill, the initial implementation used a bash script (`capture-evidence.sh`) to orchestrate ffmpeg stitching, frame normalization, and catbox.moe upload. Over 4 review rounds, the script hit 4 distinct bug classes that are inherent to bash's execution model rather than simple coding mistakes.
|
||||
|
||||
The skill is gone, but the lesson survives for any future multi-step skill script that coordinates external CLIs.
|
||||
|
||||
## Guidance
|
||||
|
||||
Use Python for agent pipeline scripts that chain multiple CLI tools with error handling. Bash `set -euo pipefail` works for simple sequential scripts but becomes a footgun when you need controlled failure paths.
|
||||
|
||||
**Python subprocess model (explicit error handling):**
|
||||
```python
|
||||
result = subprocess.run(
|
||||
["curl", "-s", "-F", f"fileToUpload=@{file_path}", url],
|
||||
capture_output=True, text=True, timeout=30, check=False
|
||||
)
|
||||
if result.returncode != 0:
|
||||
# Retry logic runs normally
|
||||
attempts += 1
|
||||
continue
|
||||
```
|
||||
|
||||
**Python timeout handling (explicit catch):**
|
||||
```python
|
||||
try:
|
||||
result = subprocess.run(cmd, timeout=60)
|
||||
except subprocess.TimeoutExpired:
|
||||
# Controlled failure, not a crash
|
||||
return subprocess.CompletedProcess(cmd, returncode=1, stdout="", stderr="Timed out")
|
||||
```
|
||||
|
||||
**Bash equivalent (the footgun):**
|
||||
```bash
|
||||
set -euo pipefail
|
||||
|
||||
# Exits the entire script before retry logic runs
|
||||
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint")
|
||||
# Never reaches here on curl failure
|
||||
|
||||
# Workaround: || true on every line that might fail
|
||||
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint") || true
|
||||
# Works but fragile and easy to forget
|
||||
```
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Agent pipeline scripts run in environments the skill author does not control: different macOS versions (bash 3.2 vs 5.x), CI containers, worktrees. Each bash portability issue requires a non-obvious workaround that reviewers must catch. Python's subprocess model makes error handling explicit and testable rather than implicit and version-dependent.
|
||||
|
||||
The 4 bugs found were not unusual. They are the predictable consequence of using bash for scripts that exceed its sweet spot.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Use Python when:
|
||||
- The script orchestrates 2+ external CLI tools
|
||||
- The script needs retry logic or graceful degradation on tool failure
|
||||
- The script will run on macOS where bash 3.2 is the default
|
||||
- The script needs to be tested from a non-shell test runner
|
||||
- The script has more than ~3 subcommands
|
||||
|
||||
Bash is still the right choice when:
|
||||
- Simple sequential scripts with no error recovery (set -e is fine)
|
||||
- One-liner wrappers around a single tool
|
||||
- Scripts using only POSIX features with no array manipulation
|
||||
- Git hooks and CI steps where the only failure mode is "abort the pipeline"
|
||||
|
||||
## Examples
|
||||
|
||||
**Before (bash, 4 bugs across 4 review rounds):**
|
||||
|
||||
| Bug | Cause | Workaround needed |
|
||||
|---|---|---|
|
||||
| `url=$(curl ...)` exits on network failure | `set -e` + command substitution | `\|\| true` on every line |
|
||||
| `${array[-1]}` fails | Bash 3.2 lacks negative indexing | `${array[${#array[@]}-1]}` |
|
||||
| Frame reduction keeps all frames for n=3,4 | Integer math: `step=(n-1)/2` with min 1 | Minimum step of 2 |
|
||||
| `command -v ffmpeg` in Bun tests | `command` is a shell builtin, not spawnable | Use `which` instead |
|
||||
|
||||
**After (Python, all 4 bug classes eliminated):**
|
||||
|
||||
```python
|
||||
# Negative indexing just works
|
||||
last = frames[-1]
|
||||
|
||||
# Timeout handling is explicit
|
||||
try:
|
||||
result = subprocess.run(cmd, timeout=30)
|
||||
except subprocess.TimeoutExpired:
|
||||
return None
|
||||
|
||||
# Tool detection is a regular function
|
||||
if not shutil.which("ffmpeg"):
|
||||
sys.exit("ffmpeg not found")
|
||||
|
||||
# Math is straightforward
|
||||
step = max(2, (len(frames) - 1) // 2)
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/script-first-skill-architecture.md`: covers when to use scripts vs agent logic (complementary: that doc answers "should a script do this?", this doc answers "which language?")
|
||||
- `docs/solutions/agent-friendly-cli-principles.md`: CLI design from the consumer side (overlaps on exit code and stderr patterns)
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: "A preserve-user-content guard must cover every destructive path to the target, not just the main one"
|
||||
date: 2026-07-09
|
||||
category: docs/solutions/best-practices/
|
||||
module: src/targets
|
||||
problem_type: best_practice
|
||||
component: tooling
|
||||
severity: high
|
||||
applies_when:
|
||||
- "Adding a guard so an installer/writer stops clobbering user-managed files (symlink overrides, hand-authored dirs)"
|
||||
- "A target writer removes-then-rewrites content and more than one function can delete or move that same path"
|
||||
- "Reviewing a fix that protects a resource against one operation when sibling operations touch it too"
|
||||
tags: [pi-writer, symlink, user-content, install, choke-point, adversarial-review, converters]
|
||||
---
|
||||
|
||||
# A preserve-user-content guard must cover every destructive path to the target, not just the main one
|
||||
|
||||
## Context
|
||||
|
||||
The Pi writer (`src/targets/pi.ts`) clobbered user-managed symlink overrides on every `install ... --to pi` (issue #1048): the main skill/agent loop did `fs.rm(target, { recursive: true, force: true })` then copied fresh upstream content, deactivating a symlink that pointed at a user's fork. PR #1089 fixed the main loop by `lstat`-ing the target first and skipping both the `rm` and the copy for symlinks and unmanaged dirs.
|
||||
|
||||
That fix was correct but **incomplete**. The Pi writer has more than one code path that destroys a target skill/agent path:
|
||||
|
||||
1. the main write loop (`cleanupCurrentManagedSkillDir` / `cleanupCurrentManagedAgentFile`) — guarded by the PR;
|
||||
2. the removed-entry sweeps (`cleanupRemovedSkills` / `cleanupRemovedAgents`) — also guarded by the PR;
|
||||
3. the **legacy-artifact sweep** (`cleanupKnownLegacyPiArtifacts` → `moveLegacyArtifactToBackup`), which `fs.rename`s CE-owned legacy skill/agent paths into `legacy-backup/` — **missed**.
|
||||
|
||||
The gap was not academic: the issue's own repro skill, `ce-session-inventory`, is a legacy-only skill (dropped from the bundle, still listed in `EXTRA_LEGACY_ARTIFACTS_BY_PLUGIN["compound-engineering"]`). Because it is absent from the bundle it never reaches the guarded main loop — it flows straight to the legacy sweep, which followed the symlink and renamed it away, *while the run had already logged that the symlink was preserved*. The headline scenario of the bug was still broken after the "fix," and an adversarial reviewer (Codex, P1 on the PR) is what surfaced it.
|
||||
|
||||
## Guidance
|
||||
|
||||
When you add a guard to preserve user content against a destructive writer, treat the **set of destructive paths to that target as a completeness obligation**, the same way a correctness cache treats its invalidation-input set (see Related). Guarding the one path in front of you is a partial fix by default.
|
||||
|
||||
1. **Enumerate every function that can `rm`, `rename`, overwrite, or move the protected target — then guard all of them.** In the Pi writer that meant three call sites, not one. Grep the module for `fs.rm`, `fs.rename`, `fs.unlink`, `writeText`/copy-into-place against the same directory root.
|
||||
2. **Prefer guarding the single destructive choke point over each call site.** The follow-up fix put the `lstat` check inside `moveLegacyArtifactToBackup` (`src/targets/pi.ts:561`), the one function all five legacy sweeps (skills, agents, prompts, extensions, mcporter) funnel through — reusing the existing `isPreservedSymlink` helper (`src/targets/pi.ts:434`) and covering kinds the reviewer didn't even name, for free. A choke-point guard cannot be forgotten by a future sibling call the way a per-call-site guard can.
|
||||
3. **Order the guard before any existence check that follows the link.** `pathExists` (stat/access) follows a symlink and reports a dangling link as absent; `lstat` sees the link node. Check `isPreservedSymlink` *before* `pathExists` so live and dangling user symlinks are both preserved.
|
||||
4. **Distrust a fix whose own reproduction case routes through the unguarded path.** If the bug report's example still traverses code you didn't touch, the fix is unverified — reproduce the exact reported case, don't just test the path you changed.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Silent user-data loss is the worst failure class for an installer: no error, and the run actively logs reassurance ("existing user-managed symlink (not overwritten)") while a different code path deletes the same override moments later. A partial guard is arguably worse than no guard because the warning manufactures false confidence.
|
||||
|
||||
The failure mode is structural, not a one-off: a resource protected against operation A is still exposed to sibling operations B and C that touch it independently. This plugin has the exact shape waiting in two more writers — `src/targets/codex.ts` (`cleanupCurrentManagedSkillDir`) and `src/targets/managed-artifacts.ts` (`cleanupCurrentManagedDirectory`, used by the OpenCode writer) share the unconditional rm-then-copy pattern and would clobber symlinks the same way. Naming them keeps the completeness obligation visible for the next fix.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- You are adding "skip if the user owns this path" logic to any target writer or installer.
|
||||
- A resource is mutated by more than one function and you are protecting it against one of them.
|
||||
- You are reviewing a preserve-user-content or don't-clobber fix — audit for sibling destructive paths and for whether the original repro actually exercises the changed code.
|
||||
|
||||
## Examples
|
||||
|
||||
Guarding the choke point instead of the call site — one edit covers every legacy sweep:
|
||||
|
||||
```ts
|
||||
// src/targets/pi.ts — moveLegacyArtifactToBackup
|
||||
async function moveLegacyArtifactToBackup(managedDir, kind, artifactPath) {
|
||||
// Checked before pathExists (which follows the link) so a user symlink is
|
||||
// preserved whether it is live or dangling.
|
||||
if (await isPreservedSymlink(artifactPath)) return
|
||||
if (!(await pathExists(artifactPath))) return
|
||||
// ... ensureDir + fs.rename(artifactPath, backupPath)
|
||||
}
|
||||
```
|
||||
|
||||
Testing the exact repro, not just the changed path. The regression test symlinks `ce-session-inventory` (a legacy-only skill that reaches *only* the legacy sweep) and asserts ownership would have fired absent the guard, so it fails loudly if the fingerprint ever drifts instead of passing for the wrong reason:
|
||||
|
||||
```ts
|
||||
// Guard against silent false-green: if the fingerprint stops matching,
|
||||
// ownership goes false and the sweep skips for the wrong reason.
|
||||
expect(await isLegacySkillArtifactOwned(symlinkPath, "ce-session-inventory")).toBe(true)
|
||||
await writePiBundle(outputRoot, bundle) // bundle omits the skill -> legacy sweep only
|
||||
expect((await fs.lstat(symlinkPath)).isSymbolicLink()).toBe(true)
|
||||
expect(await exists(path.join(outputRoot, "compound-engineering", "legacy-backup"))).toBe(false)
|
||||
```
|
||||
|
||||
That false-green assertion earned its keep on the first run: an em-dash mismatch in the fork's description made ownership return `false`, which the assertion caught — otherwise the test would have "passed" without exercising the guard at all.
|
||||
|
||||
## Related
|
||||
- [A correctness cache needs a COMPLETE, schema-derived invalidation input set](cache-invalidation-input-set-completeness.md) — sibling pattern: a safety-relevant set (there, invalidation inputs) that must be complete, with the gap found by adversarial review.
|
||||
- Issue #1048 (repro) and PR #1089 (merged), which fixed both the main-loop clobber and — as a follow-up commit on the same branch — the legacy-sweep gap described here.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: Codex native skills, legacy prompts, and converter entry points
|
||||
category: architecture
|
||||
tags: [codex, converter, skills, prompts, workflows, deprecation]
|
||||
created: 2026-03-15
|
||||
last_refreshed: 2026-06-20
|
||||
severity: medium
|
||||
component: codex-target
|
||||
problem_type: convention
|
||||
root_cause: outdated_target_model
|
||||
---
|
||||
|
||||
# Codex native skills, legacy prompts, and converter entry points
|
||||
|
||||
## Problem
|
||||
|
||||
The Codex target used to treat Compound workflow entrypoints as converter-generated skill/prompt artifacts. That made sense when Codex lacked a native plugin install path and the plugin still shipped standalone agents, but it is no longer the primary model.
|
||||
|
||||
The current Compound Engineering package is root-native and skills-only:
|
||||
|
||||
- Codex native install reads `.codex-plugin/plugin.json`.
|
||||
- The Codex manifest declares `skills: "./skills/"`.
|
||||
- User-facing skills are root directories like `skills/ce-plan`, `skills/ce-work`, and `skills/ce-code-review`.
|
||||
- Specialist review/research behavior lives in skill-local prompt assets under `references/agents/` or `references/personas/`.
|
||||
- There are 0 standalone CE agents in the plugin surface.
|
||||
|
||||
The old copied-skill/prompt-wrapper model still matters for legacy cleanup and for `--to codex --codex-include-skills` style converter tests, but it should not be documented as the normal install path.
|
||||
|
||||
## Current Codex model
|
||||
|
||||
### Native install is the source of skills
|
||||
|
||||
For users, Codex skills come from the native plugin install:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "compound-engineering",
|
||||
"skills": "./skills/"
|
||||
}
|
||||
```
|
||||
|
||||
The Bun converter's default `--to codex` behavior is intentionally not a second skill installer. It suppresses skills, prompts, command-skills, and MCP so the native plugin install remains the sole source for those artifact types.
|
||||
|
||||
### Converter default mode is compatibility-only
|
||||
|
||||
`convertClaudeToCodex()` still converts formal Claude agents to Codex custom-agent TOML because older plugin shapes and fixtures may contain agents. For the current Compound Engineering package, `plugin.agents` is empty, so default Codex conversion emits no standalone agents.
|
||||
|
||||
Default mode passes `externallyManagedSkillNames` to the writer so cleanup does not mistake natively installed skills for stale converter-owned artifacts.
|
||||
|
||||
### Full converter mode is legacy
|
||||
|
||||
When `codexIncludeSkills` is enabled, the converter can still emit copied skills and generated prompts. That mode exists for compatibility and tests, not for the current recommended install documentation.
|
||||
|
||||
In that mode:
|
||||
|
||||
- Deprecated `workflows:*` aliases are filtered or canonicalized.
|
||||
- Command prompts delegate to generated command-skills.
|
||||
- `transformContentForCodex()` rewrites known slash references while preserving unknown routes, URLs, and application paths.
|
||||
|
||||
## Rewrite and cleanup rules
|
||||
|
||||
When maintaining Codex conversion code:
|
||||
|
||||
- Do not reintroduce prompt wrappers for current native CE skills.
|
||||
- Keep unknown slash references unchanged in copied skill content; otherwise the converter can corrupt URLs or app routes.
|
||||
- Treat old `ce:*` and `workflows:*` names as legacy artifacts for cleanup and reference rewriting only.
|
||||
- Keep native skill names hyphenated (`ce-plan`, not `ce:plan`) because current source directories and frontmatter use hyphenated names.
|
||||
- Preserve `externallyManagedSkillNames` in default mode so re-running `install --to codex` cannot sweep active native skills into backup.
|
||||
|
||||
## Prevention
|
||||
|
||||
Before changing the Codex converter again:
|
||||
|
||||
1. Decide whether the target behavior belongs to native plugin install, default compatibility conversion, or full legacy conversion.
|
||||
2. Verify whether the artifact surface is a skill, prompt, custom agent, or cleanup-only legacy path.
|
||||
3. Add tests for copied skill content when changing full converter mode.
|
||||
4. Add cleanup tests when changing stale prompt or old `ce:*` ownership detection.
|
||||
5. Keep README language focused on native Codex plugin install, not converter-generated prompts.
|
||||
|
||||
## Related files
|
||||
|
||||
- `src/converters/claude-to-codex.ts`
|
||||
- `src/targets/codex.ts`
|
||||
- `src/types/codex.ts`
|
||||
- `src/utils/codex-content.ts`
|
||||
- `src/utils/legacy-cleanup.ts`
|
||||
- `tests/codex-converter.test.ts`
|
||||
- `tests/codex-writer.test.ts`
|
||||
- `tests/legacy-cleanup.test.ts`
|
||||
- `.codex-plugin/plugin.json`
|
||||
- `README.md`
|
||||
- `skills/ce-brainstorm/SKILL.md`
|
||||
- `skills/ce-plan/SKILL.md`
|
||||
- `docs/solutions/integrations/native-plugin-install-strategy.md`
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: Verify a new target platform's plugin format against the CLI binary, not its docs
|
||||
date: 2026-06-23
|
||||
category: conventions
|
||||
module: converters
|
||||
problem_type: convention
|
||||
component: converter-cli
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Adding a new converter or install Target for an agent platform
|
||||
- The platform's official docs render client-side or are otherwise not machine-readable
|
||||
- Establishing the ground-truth plugin layout before writing converter or writer code
|
||||
- Migrating an existing Target to a successor platform (e.g. Gemini CLI to Antigravity)
|
||||
tags: [antigravity, agy, new-target, plugin-format, empirical-verification, converter]
|
||||
---
|
||||
|
||||
# Verify a new target platform's plugin format against the CLI binary, not its docs
|
||||
|
||||
## Context
|
||||
|
||||
Adding a new converter Target (see [adding-converter-target-providers.md](../adding-converter-target-providers.md)
|
||||
for the structural 6-phase checklist) assumes you know the target's plugin format: its
|
||||
manifest schema, directory layout, MCP/hook config shape, and install command. The usual
|
||||
source for that is the platform's documentation.
|
||||
|
||||
When targeting **Antigravity CLI** (`agy`, Google's successor to the retired consumer Gemini
|
||||
CLI) this failed: `antigravity.google/docs` renders as a client-side app, so `WebFetch`
|
||||
returned only the page title — no schema, no commands, no field names. Two independent
|
||||
research agents, working only from docs and training data, disagreed on concrete facts (e.g.
|
||||
whether the interactive tool was `ask_user`, and whether remote MCP servers used `url` or
|
||||
`serverUrl`). Building a converter on either guess would have shipped a format that `agy`
|
||||
silently rejects or mis-reads.
|
||||
|
||||
## Guidance
|
||||
|
||||
**When a target platform ships a CLI, treat the installed binary as the authoritative format
|
||||
spec and probe it empirically before writing converter code.** Run a throwaway fixture plugin
|
||||
through the CLI's own validate/install/list commands and read what it accepts, transforms, and
|
||||
stores. Capture the findings in a `docs/specs/<target>.md` spec, then build the converter
|
||||
against the spec — not against the docs.
|
||||
|
||||
The probing loop that worked for `agy` v1.0.10:
|
||||
|
||||
1. **Start minimal, let the validator teach you the schema.** A `plugin.json` of just
|
||||
`{ "name", "version" }` passed `agy plugin validate`; the validator's per-section output
|
||||
(`skills`, `agents`, `commands`, `mcpServers`, `hooks` — each `processed` or
|
||||
`skipped (not found)`) revealed the full component surface without any docs.
|
||||
2. **Add one candidate component at a time** and re-validate to learn each one's expected
|
||||
location and form (`agents/<n>.md`, `commands/<n>.{toml,md}` reported as *"converted to
|
||||
skills"*, root `mcp_config.json`, root `hooks.json`).
|
||||
3. **Let validator error messages settle field-name disputes.** Feeding a remote MCP server
|
||||
`{ "url": ... }` produced `must have either command or serverUrl` — definitively resolving
|
||||
`serverUrl` over `url`/`httpUrl`, which docs and agents had guessed wrong.
|
||||
4. **`install` then `list --json`, then `uninstall`** to learn the install model and storage
|
||||
without leaving residue: `agy plugin install <dir>` requires a **local directory** (no
|
||||
install-from-URL), and installed plugins live in an internal registry
|
||||
(`agy plugin list --json` shows `source ∈ {antigravity, gemini-cli, claude}`), not a
|
||||
readable `plugins/` tree.
|
||||
5. **Mine the binary and its bundled assets** for what probing can't surface
|
||||
(`~/.gemini/antigravity-cli/builtin/skills/.../cli.md` documented `/permissions` and the
|
||||
`toolPermission` setting).
|
||||
|
||||
**Defer, don't guess, on anything the probe can't confirm.** The per-event `hooks.json` schema
|
||||
was not establishable from a fixture, so the converter emits only the `{ hooks: {...} }`
|
||||
container and the spec records the gap — rather than emitting an unverified per-event shape.
|
||||
Equally, where the human operator has direct live knowledge the probe lacks, prefer it: the
|
||||
`ask_question` tool name was confirmed by the user from live `agy` usage after binary
|
||||
string-inspection came up empty.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
A converter writes files another tool ingests. A wrong field name, manifest location, or
|
||||
install assumption fails **silently** — the target skips the section or rejects the bundle with
|
||||
no error in our pipeline — and ships broken output to every user of that target. Docs are a
|
||||
weaker oracle than the binary for three reasons seen here: they can be unfetchable
|
||||
(client-rendered), they lag the CLI's actual behavior, and an LLM filling the gap from training
|
||||
priors produces confident, wrong specifics. The CLI binary is the artifact users actually run,
|
||||
so its acceptance behavior is the only spec that can't be stale or hallucinated. The cost is
|
||||
low (a few `validate`/`install` cycles against a fixture) and the spec it produces is reusable
|
||||
by every later converter change.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any time you add or significantly change a converter/install Target and the platform exposes
|
||||
a CLI you can run locally.
|
||||
- Especially when the platform's docs are client-rendered, sparse, brand-new, or in flux.
|
||||
- When migrating a Target to a successor platform — do not assume format continuity. Antigravity
|
||||
inherited Gemini's *models* and reads `GEMINI.md`, but its plugin format, install model, MCP
|
||||
field names, and permission model all differ.
|
||||
- Skip (or lean lighter) only when the platform publishes a machine-readable schema you can
|
||||
validate against directly, or ships an official converter/import you can diff against.
|
||||
|
||||
## Examples
|
||||
|
||||
**Antigravity facts established by probing (full record in [docs/specs/antigravity.md](../../specs/antigravity.md)):**
|
||||
|
||||
| Question | Docs/agent guess | Probe result |
|
||||
| --- | --- | --- |
|
||||
| Remote MCP field | `url` / `httpUrl` | **`serverUrl`** (validator: "must have either command or serverUrl") |
|
||||
| Install source | repo URL | **local directory** with a root `plugin.json` (`agy plugin install <dir>`) |
|
||||
| Commands | a command primitive | **converted to skills on install** |
|
||||
| Interactive tool | `ask_user` | **`ask_question`** (confirmed in live use; absent from binary strings) |
|
||||
| Minimal manifest | unknown | **`{ name, version }`** |
|
||||
|
||||
**Converter consequence:** `src/converters/claude-to-antigravity.ts` maps the Claude remote-MCP
|
||||
`url` to `serverUrl`, and `src/targets/antigravity.ts` emits the verified bundle layout. The
|
||||
spec drove these mappings; the docs could not have.
|
||||
|
||||
**Packaging consequence — committed `.agy/` bundle with a symlink:** because `agy plugin install`
|
||||
resolves component dirs relative to the `plugin.json` location, this plugin ships a committed
|
||||
`.agy/` directory holding `plugin.json` plus a `skills -> ../skills` symlink (verified:
|
||||
`agy plugin validate ./.agy` processes all skills through the symlink). This keeps the repo root
|
||||
uncluttered while reusing the canonical root `skills/` rather than duplicating it — and lets users
|
||||
`git clone … && agy plugin install ./compound-engineering-plugin/.agy`. (See also the per-platform
|
||||
choices in [native-plugin-install-strategy.md](../integrations/native-plugin-install-strategy.md).)
|
||||
|
||||
Related: GitHub issue #911 (Transition to Antigravity CLI); plan
|
||||
`docs/plans/2026-06-22-001-feat-antigravity-target-remove-gemini-plan.md`.
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
title: "Branch-based plugin install and testing for Claude Code plugins"
|
||||
date: 2026-03-26
|
||||
problem_type: developer_experience
|
||||
category: developer-experience
|
||||
component: development_workflow
|
||||
root_cause: missing_workflow_step
|
||||
resolution_type: workflow_improvement
|
||||
severity: medium
|
||||
tags:
|
||||
- cli
|
||||
- plugin-install
|
||||
- branch-testing
|
||||
- developer-experience
|
||||
- git-clone
|
||||
- plugin-path
|
||||
symptoms:
|
||||
- "No way to install or test a Claude Code plugin from a specific git branch"
|
||||
- "install command always cloned the default branch from GitHub"
|
||||
- "claude --plugin-dir only accepts a local filesystem path with no branch support"
|
||||
- "Developers had to manually checkout branches to test others' plugin changes"
|
||||
root_cause_detail: "The CLI lacked any mechanism to target a specific git branch when installing or testing plugins. Claude Code's --plugin-dir flag only accepts local paths, and the install command had no --branch option."
|
||||
solution_summary: "Added a new plugin-path subcommand that clones a specific branch to a deterministic cache path (~/.cache/compound-engineering/branches/) and outputs it for use with claude --plugin-dir. Also added a --branch flag to the install command for non-Claude targets."
|
||||
key_insight: "Worktree-based development means multiple branches are active simultaneously and the repo root checkout can't serve as a reliable plugin source. A deterministic cache path based on the sanitized branch name enables branch-specific plugin testing without disrupting any checkout, and re-runs update in place via git fetch + reset --hard."
|
||||
files_changed:
|
||||
- src/commands/plugin-path.ts
|
||||
- src/commands/install.ts
|
||||
- src/index.ts
|
||||
- tests/plugin-path.test.ts
|
||||
- tests/cli.test.ts
|
||||
verification_steps:
|
||||
- "Run bun test to confirm all tests pass including 5 new plugin-path tests and 1 new CLI test"
|
||||
- "Test plugin-path subcommand outputs correct deterministic cache path for a given branch"
|
||||
- "Test install --branch flag clones from the specified branch for non-Claude targets"
|
||||
- "Verify re-running plugin-path on same branch updates via fetch+reset rather than re-cloning"
|
||||
related_docs:
|
||||
- docs/solutions/adding-converter-target-providers.md
|
||||
- docs/solutions/plugin-versioning-requirements.md
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The compound-engineering plugin CLI's `install` command always cloned the default branch from GitHub, and Claude Code's `--plugin-dir` flag only accepts local filesystem paths. Developers who wanted to test a plugin from a specific git branch had to manually check out that branch in their local repo, disrupting their working tree.
|
||||
|
||||
This is especially painful in worktree-based workflows where a single local checkout root can only represent one branch at a time. Two concrete scenarios:
|
||||
|
||||
- **Cross-repo**: You're working in a different project and want to use a CE branch as your plugin. Without this, you'd have to switch the CE repo's checkout — which is likely WIP on something else.
|
||||
- **Same-repo**: You're working on CE itself — `feat/feature-2` in your main checkout, `feat/feature-1` in a worktree. You want to test feature-1's plugin while continuing to develop feature-2. The main checkout can't serve both purposes.
|
||||
|
||||
Note: the `--branch` flag works with pushed branches (those available on the remote). For unpushed local worktree branches, developers can point `--plugin-dir` directly at the worktree root (e.g., `claude --plugin-dir /path/to/compound-engineering-plugin-worktree`).
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Running `bunx compound-engineering install <plugin>` always fetched the default branch regardless of what branch contained the changes under review.
|
||||
- `claude --plugin-dir` required a local path, so there was no way to point it at a remote branch without a manual `git clone` or `git checkout`.
|
||||
- Developers testing PR branches had to stash or commit their local work, switch branches, test, then switch back -- a disruptive and error-prone workflow.
|
||||
- In worktree-based workflows, the repo root checkout always points to one branch, not every worktree branch being developed. Developers working on multiple branches simultaneously had no ergonomic way to install from a specific worktree's branch.
|
||||
- No scripting path existed to spin up a branch-specific plugin directory for automated testing.
|
||||
|
||||
---
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
- **Using `/tmp/` for cloned branches** was rejected because temporary directories are cleared on reboot, forcing a full re-clone every session and losing the fast-update path.
|
||||
- **Random temp directory names** (e.g., `mktemp -d`) were rejected because they cause directory proliferation and make it impossible to re-run the same command and update in place.
|
||||
- **Extending `claude --plugin-dir` itself** was not an option -- that flag is owned by Claude Code and only accepts local filesystem paths; the solution had to live in the plugin CLI layer.
|
||||
- **Symlinking the bundled plugin** would not help because the bundled copy is always pinned to the installed CLI version, not an arbitrary remote branch.
|
||||
- **Naive branch sanitization** (`replace(/[^a-zA-Z0-9._-]/g, "-")`) collapsed distinct branches to the same cache path (e.g., `feat/foo-bar` and `feat-foo/bar` both became `feat-foo-bar`). An escape-then-replace scheme (`~` → `~~`, `/` → `~`) was attempted next but was still not injective — `feat~~foo` and `feat~//foo` both produced `feat~~~~foo`. The correct insight was that `~` is illegal in git branch names (`git-check-ref-format` reserves it for reflog notation), so a simple `/` → `~` replacement is injective without any escape step.
|
||||
|
||||
---
|
||||
|
||||
## Solution
|
||||
|
||||
Two complementary features were added:
|
||||
|
||||
### 1. New `plugin-path` command (for Claude Code)
|
||||
|
||||
Clones a branch to a deterministic cache directory and prints the path for use with `claude --plugin-dir`.
|
||||
|
||||
```bash
|
||||
bun run src/index.ts plugin-path compound-engineering --branch feat/new-agents
|
||||
# Output: claude --plugin-dir ~/.cache/compound-engineering/branches/compound-engineering-feat~new-agents
|
||||
```
|
||||
|
||||
Key implementation details in `src/commands/plugin-path.ts`:
|
||||
|
||||
- Cache path: `~/.cache/compound-engineering/branches/<plugin>-<sanitized-branch>/`
|
||||
- Branch sanitization: `/` → `~`, then strip remaining non-`[a-zA-Z0-9._~-]` chars. This is injective because `~` is illegal in git branch names (`git-check-ref-format` reserves it for reflog notation), so no valid branch input contains `~` and the mapping is 1:1.
|
||||
- First run: `git clone --depth 1 --branch <name> <source> <dest>`
|
||||
- Re-run: `git fetch origin <branch>` + `git reset --hard origin/<branch>`
|
||||
|
||||
### 2. `--branch` flag on `install` command (for converter-backed targets)
|
||||
|
||||
Threads a branch name through the full resolution chain so `install` clones from the specified branch instead of the default.
|
||||
|
||||
```bash
|
||||
bun run src/index.ts install compound-engineering --to opencode --branch feat/new-agents
|
||||
```
|
||||
|
||||
Changes in `src/commands/install.ts`:
|
||||
|
||||
- When `--branch` is provided, skips bundled plugin lookup (user explicitly wants a remote version)
|
||||
- Threaded through `resolvePluginPath` -> `resolveGitHubPluginPath` -> `cloneGitHubRepo`
|
||||
- `cloneGitHubRepo` conditionally adds `--branch <name>` to `git clone --depth 1`
|
||||
|
||||
### Key difference between the two
|
||||
|
||||
`plugin-path` caches the checkout in `~/.cache/` for reuse across sessions. `install --branch` uses an ephemeral temp directory that's cleaned up after the install completes -- it only needs the clone long enough to read and convert the plugin.
|
||||
|
||||
---
|
||||
|
||||
## Why This Works
|
||||
|
||||
The root issue was a missing indirection layer: the CLI assumed "install" always means "use the default branch," and Claude Code assumes "plugin directory" always means "a path that already exists locally." The solution bridges that gap by:
|
||||
|
||||
- **Deterministic cache paths** mean the same branch always maps to the same directory. No proliferation, no ambiguity.
|
||||
- **Fetch + hard reset on re-run** keeps the cached checkout current without requiring a full re-clone, making iteration fast.
|
||||
- **`~/.cache/`** follows XDG conventions, persists across reboots, and is understood by users and tooling as a safe-to-delete cache layer.
|
||||
- **The `COMPOUND_PLUGIN_GITHUB_SOURCE` env var** works with both features, allowing tests to use local git repos and avoiding network dependency.
|
||||
|
||||
---
|
||||
|
||||
## Prevention
|
||||
|
||||
- **Test coverage**: `tests/plugin-path.test.ts` (6 tests: clone-to-cache, slash sanitization, update-on-rerun, slash-placement collision resistance, nonexistent branch error, nonexistent plugin error) and `tests/cli.test.ts` (1 test: install --branch clones specific branch). All tests use local git repos via `COMPOUND_PLUGIN_GITHUB_SOURCE`.
|
||||
- **Cache directory convention**: Any future features that need ephemeral or semi-persistent clones should use `~/.cache/compound-engineering/<purpose>/` with deterministic, sanitized subdirectory names. Avoid `/tmp/` for anything that benefits from surviving a reboot.
|
||||
- **Branch sanitization**: Always sanitize branch names before using them in filesystem paths. Using `~` as the slash replacement is injective because `~` is illegal in git branch names (`git-check-ref-format`). A naive `replace(/[^a-zA-Z0-9._-]/g, "-")` is insufficient because it collapses branches like `feat/foo-bar` and `feat-foo/bar` into the same path.
|
||||
- **Resolution chain threading**: When adding new resolution strategies to the CLI, thread optional parameters through the full `resolvePluginPath -> resolveGitHubPluginPath -> cloneGitHubRepo` chain rather than branching at the top level. This keeps the resolution logic composable.
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: "Colon-namespaced skill names break filesystem paths on Windows"
|
||||
date: 2026-03-26
|
||||
last_refreshed: 2026-06-20
|
||||
category: integration-issues
|
||||
module: cli-converter
|
||||
problem_type: integration_issue
|
||||
component: tooling
|
||||
symptoms:
|
||||
- "ENOTDIR error when running bun convert on Windows"
|
||||
- "mkdir fails with '.config\\opencode\\skills\\ce:brainstorm'"
|
||||
- "All target writers (opencode, codex, copilot, etc.) produce colon paths"
|
||||
root_cause: config_error
|
||||
resolution_type: code_fix
|
||||
severity: high
|
||||
related_issues:
|
||||
- "https://github.com/EveryInc/compound-engineering-plugin/issues/366"
|
||||
related_components:
|
||||
- targets
|
||||
- sync
|
||||
- converters
|
||||
tags:
|
||||
- windows
|
||||
- cross-platform
|
||||
- path-sanitization
|
||||
- skill-names
|
||||
- colons
|
||||
---
|
||||
|
||||
# Colon-namespaced skill names break filesystem paths on Windows
|
||||
|
||||
## Problem
|
||||
|
||||
Earlier plugin versions allowed skill names containing colons (e.g., `ce:brainstorm`, `ce:plan`) to flow directly into target writer paths. Colons are illegal in Windows filenames, causing `ENOTDIR` errors during `bun convert` or `bun install`.
|
||||
|
||||
## Symptoms
|
||||
|
||||
```
|
||||
{ [Error: ENOTDIR: not a directory, mkdir '.config\opencode\skills\ce:brainstorm']
|
||||
code: 'ENOTDIR',
|
||||
path: '.config\\opencode\\skills\\ce:brainstorm',
|
||||
syscall: 'mkdir',
|
||||
errno: -20 }
|
||||
```
|
||||
|
||||
This affected every target present at the time because all used `skill.name` directly in `path.join()` calls. Current CE source skills are hyphenated (`ce-brainstorm`, `ce-plan`), but the sanitizer still matters for compatibility fixtures, imported third-party plugins, and legacy artifact cleanup.
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
Using `/` (forward slash) as the replacement character was initially considered — turning `ce:brainstorm` into nested directories `ce/brainstorm/`. This was rejected because:
|
||||
|
||||
1. It introduces unnecessary directory nesting for what's fundamentally a character-replacement problem
|
||||
2. The `isValidSkillName` and `validatePathSafe` functions reject `/` and `\`, so sanitized names would fail existing validation
|
||||
3. The source directories already use hyphens (`skills/ce-brainstorm/`), so the output should match
|
||||
|
||||
## Solution
|
||||
|
||||
Added `sanitizePathName()` in `src/utils/files.ts` that replaces colons with hyphens:
|
||||
|
||||
```typescript
|
||||
export function sanitizePathName(name: string): string {
|
||||
return name.replace(/:/g, "-")
|
||||
}
|
||||
```
|
||||
|
||||
Applied across two layers:
|
||||
|
||||
### Layer 1: Target writers
|
||||
|
||||
Every target writer wraps skill/agent names with `sanitizePathName()` when constructing output paths:
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
await copyDir(skill.sourceDir, path.join(skillsRoot, skill.name))
|
||||
|
||||
// After
|
||||
await copyDir(skill.sourceDir, path.join(skillsRoot, sanitizePathName(skill.name)))
|
||||
```
|
||||
|
||||
Currently applied in the maintained target writers and managed-artifact cleanup path. When this fix was first written, a separate `src/sync/` directory also held path-construction logic that needed the same treatment; that layer has since been consolidated into target writers.
|
||||
|
||||
### Layer 2: Converter dedupe sets and manifests
|
||||
|
||||
Sanitizing paths in writers created a secondary bug: converter dedupe logic used unsanitized names, so a pass-through skill `ce:plan` and a generated skill normalizing to `ce-plan` wouldn't detect the collision — both would write to `skills/ce-plan/` on disk.
|
||||
|
||||
Fixed in converters that maintain dedupe sets — currently `src/converters/claude-to-copilot.ts`:
|
||||
|
||||
- `usedSkillNames.add(sanitizePathName(skill.name))` instead of raw `skill.name`
|
||||
|
||||
Any future converter that maintains a name-collision set or emits a manifest must apply the same sanitization so the in-memory set matches the on-disk paths.
|
||||
|
||||
## Why This Works
|
||||
|
||||
The core issue was a mismatch between the logical name domain (where older plugin data used colons as namespace separators) and the filesystem domain (where colons are illegal on Windows). The fix sanitizes at the boundary: legacy/imported names can keep colons in data structures, but paths use hyphens. Current CE source directories and frontmatter use hyphenated names directly (`skills/ce-brainstorm/`, `name: ce-brainstorm`), so the sanitizer is now primarily a compatibility guard.
|
||||
|
||||
## Prevention
|
||||
|
||||
### 1. Collision detection test
|
||||
|
||||
A test in `tests/path-sanitization.test.ts` loads the real compound-engineering plugin and verifies no two skill or agent names collide after sanitization:
|
||||
|
||||
```typescript
|
||||
test("no two skill names collide after sanitization", async () => {
|
||||
const plugin = await loadClaudePlugin(pluginRoot)
|
||||
const sanitized = plugin.skills.map((skill) => sanitizePathName(skill.name))
|
||||
const unique = new Set(sanitized)
|
||||
expect(unique.size).toBe(sanitized.length)
|
||||
})
|
||||
```
|
||||
|
||||
### 2. When adding names to filesystem paths
|
||||
|
||||
Always use `sanitizePathName()` when constructing output paths from skill, agent, or component names. Never pass `skill.name` or `agent.name` directly to `path.join()` in target writers or managed artifact paths.
|
||||
|
||||
### 3. When building dedupe sets in converters
|
||||
|
||||
If a converter reserves names for collision detection, the reserved names must be sanitized to match what the writer will produce on disk. Raw names in the set + normalized names from generators = missed collisions.
|
||||
|
||||
### 4. Inconsistency with `resolveCommandPath`
|
||||
|
||||
Note that `resolveCommandPath` (used for commands) converts colons to nested directories (`ce:plan` -> `ce/plan.md`), while `sanitizePathName` (used for skills/agents and compatibility artifact paths) converts to hyphens (`ce:plan` -> `ce-plan`). This is intentional — commands and skills are different surfaces with different resolution patterns. If a new component type is added, decide which pattern fits and document the choice.
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: "Cross-platform model field normalization for target converters"
|
||||
date: 2026-03-29
|
||||
category: integration-issues
|
||||
module: src/converters
|
||||
problem_type: integration_issue
|
||||
component: tooling
|
||||
symptoms:
|
||||
- "Target platforms received raw Claude model aliases (e.g., 'sonnet') they could not resolve"
|
||||
- "Qwen converter mapped model aliases to wrong canonical names (claude-sonnet instead of claude-sonnet-4-6)"
|
||||
- "OpenClaw and Copilot passed through unnormalized model values in formats the target could not use"
|
||||
- "Duplicated CLAUDE_FAMILY_ALIASES and normalizeModel logic across converters with divergent alias values"
|
||||
root_cause: config_error
|
||||
resolution_type: code_fix
|
||||
severity: medium
|
||||
tags:
|
||||
- model-normalization
|
||||
- converters
|
||||
- cross-platform
|
||||
- opencode
|
||||
- qwen
|
||||
- droid
|
||||
- copilot
|
||||
- openclaw
|
||||
- codex
|
||||
---
|
||||
|
||||
# Cross-platform model field normalization for target converters
|
||||
|
||||
## Problem
|
||||
|
||||
Claude Code uses bare model aliases (`model: sonnet`, `model: haiku`, `model: opus`) in agent and command frontmatter. Each target platform expects a different format for the model field, but the converters handled this inconsistently — some passed through raw values, others had duplicated normalization logic with wrong alias mappings.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- OpenClaw passed `model: sonnet` through raw — invalid on a platform expecting `anthropic/claude-sonnet-4-6`
|
||||
- Qwen mapped `sonnet` to `anthropic/claude-sonnet` instead of `anthropic/claude-sonnet-4-6` (wrong alias in its local copy of `CLAUDE_FAMILY_ALIASES`)
|
||||
- Copilot passed through raw Claude model IDs like `claude-sonnet-4-20250514` — Copilot uses display-name format ("Claude Opus 4.5"), not model IDs
|
||||
- Codex emitted no model field — correct behavior, but accidental (no deliberate handling)
|
||||
- Droid passed through as-is — correct behavior, but undocumented as intentional
|
||||
- Two copies of `CLAUDE_FAMILY_ALIASES` existed in OpenCode and Qwen converters with divergent values
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
- **Passing model through as-is**: works for Droid (Factory natively resolves bare aliases), breaks OpenClaw/Qwen/OpenCode
|
||||
- **Mapping bare aliases to incomplete model names**: Qwen's `sonnet` -> `claude-sonnet` was wrong; correct is `claude-sonnet-4-6`
|
||||
- **Assuming all targets want the same model format**: each platform has fundamentally different expectations
|
||||
- **Assuming Codex skills support model overrides in frontmatter**: they don't — confirmed by the Rust source `SkillFrontmatter` struct which only has `name` and `description`
|
||||
- **Initial assumption that Qwen should drop model entirely**: wrong — Qwen is multi-provider and supports Anthropic models via `settings.json` with `anthropic` provider config
|
||||
- **Initial assumption that Copilot doesn't support models**: wrong — Copilot supports multi-model including Claude, but the exact format is uncertain (display names vs model IDs)
|
||||
|
||||
## Solution
|
||||
|
||||
Created `src/utils/model.ts` with shared normalization utilities:
|
||||
|
||||
```typescript
|
||||
// Single source of truth for bare Claude family aliases
|
||||
export const CLAUDE_FAMILY_ALIASES: Record<string, string> = {
|
||||
haiku: "claude-haiku-4-5",
|
||||
sonnet: "claude-sonnet-4-6",
|
||||
opus: "claude-opus-4-6",
|
||||
}
|
||||
|
||||
// Resolve bare alias without provider prefix (used by Droid)
|
||||
export function resolveClaudeFamilyAlias(model: string): string
|
||||
|
||||
// Add provider prefix based on naming conventions
|
||||
export function addProviderPrefix(model: string): string
|
||||
|
||||
// Combined: resolve + prefix (used by OpenCode, Qwen, OpenClaw)
|
||||
export function normalizeModelWithProvider(model: string): string
|
||||
```
|
||||
|
||||
Each converter uses the appropriate shared utility:
|
||||
|
||||
| Target | Behavior | Output for `model: sonnet` |
|
||||
|--------|----------|----------------------------|
|
||||
| OpenCode | Resolve alias + add provider prefix | `anthropic/claude-sonnet-4-6` |
|
||||
| Droid | Pass through as-is | `sonnet` |
|
||||
| Copilot | Drop entirely | (omitted) |
|
||||
| Codex | Drop entirely | (omitted) |
|
||||
|
||||
> **Note:** This doc was written when the converter set also included Qwen and OpenClaw, both of which used the "Resolve alias + add provider prefix" behavior. Both have since been removed in favor of native plugin install — see `docs/solutions/integrations/native-plugin-install-strategy.md`. The pattern still applies to any future multi-provider target with the `provider/model-id` format.
|
||||
|
||||
---
|
||||
|
||||
## Why This Works
|
||||
|
||||
Each platform has fundamentally different model handling requirements:
|
||||
|
||||
**Platforms that normalize (OpenCode, Qwen, OpenClaw):** These are multi-provider platforms that support Anthropic, OpenAI, Google, and other model providers. They need provider-prefixed IDs like `anthropic/claude-sonnet-4-6` to route requests to the correct backend. The `normalizeModelWithProvider` function resolves bare aliases and adds the appropriate prefix.
|
||||
|
||||
**Droid (Factory) — pass-through:** Factory is multi-provider but natively resolves Claude's bare aliases (`sonnet`, `opus`, `haiku`) internally. Pass-through is correct and simpler than normalizing to a format Factory would also accept but doesn't require. Factory also accepts full dated model IDs like `claude-sonnet-4-5-20250929` and non-Anthropic models prefixed with `custom:`.
|
||||
|
||||
**Copilot — drop:** Copilot supports a `model` field in `.agent.md` frontmatter (documented in `docs/specs/copilot.md`), but the expected values are Copilot-specific display names like "Claude Opus 4.5" — not Claude model IDs like `claude-sonnet-4-20250514` or bare aliases like `sonnet`. Passing through Claude-specific values would emit a field Copilot can't use. Unlike Droid (which natively resolves `sonnet`), Copilot has no documented resolution for Claude model IDs. Dropping is safer: the spec says "If unset, inherits the default model."
|
||||
|
||||
**Codex — drop:** Codex skill frontmatter (`SKILL.md`) only supports `name` and `description` fields. This was confirmed by examining the Rust source code (`SkillFrontmatter` struct in `codex-rs/core-skills/src/loader.rs`). Model selection in Codex is global via `config.toml` or runtime `/model` command, not per-skill.
|
||||
|
||||
---
|
||||
|
||||
## Target platform model field reference
|
||||
|
||||
This reference captures research findings as of 2026-03-29. Targets marked **(removed)** below no longer have custom Bun converters — they rely on native plugin install. The research is preserved as a future reference if those targets re-enter the converter set.
|
||||
|
||||
### OpenCode
|
||||
- **Model format:** `provider/model-id` (e.g., `anthropic/claude-sonnet-4-6`)
|
||||
- **Provider prefixes:** `anthropic/`, `openai/`, `google/`
|
||||
- **Docs:** Agents defined in `.opencode/agents/*.md`
|
||||
|
||||
### Qwen (removed)
|
||||
- **Model format:** `provider/model-id` (e.g., `anthropic/claude-sonnet-4-6`)
|
||||
- **Multi-provider:** Yes — supports Anthropic, OpenAI, Google GenAI via `settings.json`
|
||||
- **Configuration example:** `"anthropic": [{"id": "claude-sonnet-4-20250514", "name": "Claude Sonnet 4", "envKey": "ANTHROPIC_API_KEY"}]`
|
||||
- **Common misconception:** Qwen is NOT limited to its own foundation model
|
||||
|
||||
### Droid (Factory)
|
||||
- **Model format:** Bare names (`sonnet`, `claude-sonnet-4-5-20250929`) or `custom:<model>` for BYOK
|
||||
- **Native alias resolution:** Factory resolves `sonnet`, `opus`, `haiku` internally
|
||||
- **Multi-provider:** Yes — supports Anthropic, OpenAI, Google, and Factory's own `droid-core`
|
||||
- **Docs:** Custom droids defined in `.factory/droids/*.md`
|
||||
|
||||
### Copilot
|
||||
- **Model format:** Display names (e.g., "Claude Opus 4.5", "GPT-5.2"), possibly array syntax `model: ['Claude Opus 4.5', 'GPT-5.2']`
|
||||
- **Multi-provider:** Yes — supports Claude and GPT models
|
||||
- **Current converter behavior:** Drop (Claude model IDs don't map to Copilot's expected format)
|
||||
- **Note:** Spec says "may be ignored on github.com" — model selection works in IDE but may not apply on the GitHub web platform
|
||||
- **Docs:** Agents defined in `.github/agents/*.agent.md`
|
||||
|
||||
### OpenClaw (removed)
|
||||
- **Model format:** `provider/model-id` (same as OpenCode)
|
||||
- **Docs:** Skills defined in `skills/*/SKILL.md`
|
||||
|
||||
### Codex
|
||||
- **Model field in skill frontmatter:** NOT SUPPORTED
|
||||
- **Supported frontmatter fields:** `name`, `description` only
|
||||
- **Model configuration:** Global `config.toml` (`model = "gpt-5.4"`) or runtime `/model` command
|
||||
- **Valid model IDs (as of 2026-03):** `gpt-5.4` (flagship), `gpt-5.4-mini` (fast), `gpt-5.3-codex` (coding-specialized)
|
||||
- **Deprecated:** `codex-mini-latest` (removed Feb 2026)
|
||||
- **Docs:** Skills defined in `.codex/skills/*/SKILL.md` or `.agents/skills/*/SKILL.md`
|
||||
|
||||
---
|
||||
|
||||
## Prevention
|
||||
|
||||
1. **Research before implementing:** When adding a new converter target, research its model field format with external documentation before assuming pass-through or copying from another converter. The format varies significantly between platforms.
|
||||
|
||||
2. **Single source of truth:** The `CLAUDE_FAMILY_ALIASES` map in `src/utils/model.ts` is the canonical alias map. Update it there — not in individual converters — when new Claude model generations are released.
|
||||
|
||||
3. **Test coverage:** Run `bun test` after model-related changes. The test suite covers model handling across all converters (`tests/model-utils.test.ts` plus each converter's test file).
|
||||
|
||||
4. **Don't assume format from the field name:** A `model` field in frontmatter doesn't mean the format is the same across platforms. OpenCode wants `anthropic/claude-sonnet-4-6`, Factory wants `sonnet`, Copilot wants "Claude Sonnet 4", and Codex doesn't support the field at all.
|
||||
|
||||
5. **When in doubt, drop:** If you can't confidently produce the target's expected format, omit the field rather than emitting a potentially invalid value. Most platforms fall back to a sensible default when model is unset.
|
||||
|
||||
## Related Issues
|
||||
|
||||
- `docs/solutions/adding-converter-target-providers.md` — Converter architecture doc; should be updated to reference model normalization as part of the conversion pattern
|
||||
- `docs/solutions/integrations/colon-namespaced-names-break-windows-paths.md` — Structural analog: same pattern of per-target boundary normalization
|
||||
- `docs/specs/codex.md` — Platform spec (last verified 2026-01-21); confirms skill frontmatter limitations
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: "Use native Kimi plugin manifests instead of a converter target"
|
||||
date: 2026-06-24
|
||||
category: integrations
|
||||
module: installer
|
||||
problem_type: tooling_decision
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "Adding support for a coding-agent platform that can install plugin manifests directly"
|
||||
- "A proposed converter target duplicates a platform-native plugin or marketplace flow"
|
||||
- "Release automation must keep platform manifests in parity with canonical plugin metadata"
|
||||
related_components:
|
||||
- release-metadata
|
||||
- release-please
|
||||
- native-plugin-install
|
||||
tags:
|
||||
- kimi
|
||||
- native-plugins
|
||||
- marketplace
|
||||
- release-validation
|
||||
- converter-targets
|
||||
---
|
||||
|
||||
# Use native Kimi plugin manifests instead of a converter target
|
||||
|
||||
## Problem
|
||||
|
||||
Kimi Code support can look like a normal new target provider at first: add `--to kimi`, write a converter, and emit a Kimi-specific output tree. That is the wrong first move when the platform already has a native plugin manifest and custom marketplace contract.
|
||||
|
||||
This came up when [PR #997](https://github.com/EveryInc/compound-engineering-plugin/pull/997) by [@mastepanoski](https://github.com/mastepanoski) proposed Kimi support as a converter target. The useful signal was the demand for Kimi support; the implementation shape duplicated a native install surface.
|
||||
|
||||
## Decision
|
||||
|
||||
Prefer Kimi's native plugin metadata over a converter target:
|
||||
|
||||
- Commit `.kimi-plugin/plugin.json` with the Kimi manifest fields that describe this plugin, including `interface`, `skills`, `sessionStart.skill`, and `skillInstructions`.
|
||||
- Commit `.kimi-plugin/marketplace.json` using Kimi marketplace schema version `2`.
|
||||
- Keep `.kimi-plugin/plugin.json` in the root release component so release automation bumps it with the canonical plugin version.
|
||||
- Treat `.kimi-plugin/marketplace.json` as static catalog metadata, and validate it instead of version-bumping it.
|
||||
- Do not add `src/converters/claude-to-kimi.ts`, `src/targets/kimi.ts`, or a `--to kimi` CLI target unless Kimi later documents a separate generated output format that cannot be represented by the native manifest.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Native plugin support is a distribution contract, not a format-conversion problem. A converter target would create another generated install path to document, test, version, and clean up, while Kimi users would still need the manifest and marketplace metadata for the normal install flow.
|
||||
|
||||
The correct support surface is therefore:
|
||||
|
||||
- platform metadata: `.kimi-plugin/plugin.json` and `.kimi-plugin/marketplace.json`
|
||||
- release metadata: version parity, required fields, skill path checks, and marketplace schema validation
|
||||
- user docs: install and local-development instructions for Kimi's native plugin flow
|
||||
- target spec docs: a short spec explaining which Kimi fields are used and which unsupported runtime fields are intentionally absent
|
||||
|
||||
## Implementation Pattern
|
||||
|
||||
When adding a platform with a native plugin surface, wire it like a first-class release surface:
|
||||
|
||||
1. Add the native manifest and marketplace/catalog files expected by the platform.
|
||||
2. Put the release-owned manifest file in `.github/release-please-config.json` as an extra file for the canonical component.
|
||||
3. Exclude static marketplace files from release component ownership when they do not carry a version.
|
||||
4. Add release validation that rejects missing manifests, version drift, missing declared asset paths, marketplace schema drift, and marketplace plugin-list drift.
|
||||
5. Update README install docs and `docs/specs/` so future contributors know this is native metadata, not a converter target.
|
||||
|
||||
For Kimi specifically, validate at least:
|
||||
|
||||
- `.kimi-plugin/plugin.json` exists
|
||||
- `name`, `version`, `description`, and `skills` are non-empty
|
||||
- `skills` points at an existing directory in the repo
|
||||
- the Kimi manifest version equals the root plugin/package version
|
||||
- `.kimi-plugin/marketplace.json` has schema `version: "2"`
|
||||
- marketplace plugin IDs match the Claude marketplace plugin IDs
|
||||
- each marketplace entry has a non-empty `source`
|
||||
- root-local marketplace sources such as `"."` or `"./"` are rejected because they are only local-development placeholders
|
||||
|
||||
## Warning Signs
|
||||
|
||||
Reconsider a proposed new target provider when:
|
||||
|
||||
- the platform docs describe a `plugin.json`-style manifest in the source repo
|
||||
- the platform supports a custom marketplace or catalog pointing at repository/plugin sources
|
||||
- the target would mostly copy existing skills and docs without meaningful tool, permission, hook, or model conversion
|
||||
- install docs would need to tell users to run this repo's converter instead of the platform's documented plugin install path
|
||||
|
||||
Those are signs the platform support belongs in native metadata and release validation.
|
||||
|
||||
## Related
|
||||
|
||||
- [Native plugin install strategy](./native-plugin-install-strategy.md)
|
||||
- [Plugin versioning requirements](../plugin-versioning-requirements.md)
|
||||
- [Adding converter target providers](../adding-converter-target-providers.md)
|
||||
- [PR #997: original Kimi support proposal](https://github.com/EveryInc/compound-engineering-plugin/pull/997)
|
||||
- [PR #998: native Kimi plugin support](https://github.com/EveryInc/compound-engineering-plugin/pull/998)
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
title: "Native plugin install strategy for supported harnesses"
|
||||
date: 2026-06-19
|
||||
last_updated: 2026-06-30
|
||||
category: integrations
|
||||
module: installer
|
||||
problem_type: integration_decision
|
||||
component: installer
|
||||
symptoms:
|
||||
- "Formal standalone agent definitions are unevenly supported across coding-agent harnesses"
|
||||
- "Custom Bun installs create extra update and cleanup behavior for users"
|
||||
- "OpenCode and Pi can load skills directly from a git-backed package/plugin shape"
|
||||
root_cause: evolving_platform_install_surfaces
|
||||
resolution_type: install_strategy
|
||||
severity: medium
|
||||
tags:
|
||||
- install-strategy
|
||||
- native-plugins
|
||||
- cursor
|
||||
- codex
|
||||
- copilot
|
||||
- droid
|
||||
- qwen
|
||||
- kimi
|
||||
- antigravity
|
||||
- opencode
|
||||
- pi
|
||||
- cline
|
||||
---
|
||||
|
||||
# Native Plugin Install Strategy
|
||||
|
||||
Last verified: 2026-06-20
|
||||
|
||||
Compound Engineering now treats the plugin as a self-contained skills package. Specialist reviewer and researcher behavior lives in skill-local prompt assets under `references/agents/` or `references/personas/`, and skills seed generic subagents with those files when the current harness exposes a subagent primitive. There are no formal standalone CE agents in the plugin surface.
|
||||
|
||||
The install strategy follows from that: prefer each harness's native plugin/package mechanism, avoid generated agent installs, and keep the Bun converter as repo tooling rather than the user-facing installer.
|
||||
|
||||
## Summary
|
||||
|
||||
| Harness | Current install path | Bun CLI needed? | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Claude Code | Native plugin marketplace using `.claude-plugin/marketplace.json` and `.claude-plugin/plugin.json` | No | Claude remains the source plugin format. |
|
||||
| Codex | Native Codex plugin install from a custom marketplace pointing at this repository root | No | Codex App users add the marketplace manually with no sparse path; Codex CLI users register the repo and install through `/plugins`. Skill-local personas avoid the old custom-agent copy step. |
|
||||
| Cursor | Native Cursor Plugin Marketplace using `.cursor-plugin/marketplace.json` and `.cursor-plugin/plugin.json` | No | Users install from Cursor Agent chat with `/add-plugin compound-engineering` or marketplace search. |
|
||||
| GitHub Copilot CLI | Native plugin marketplace using the existing Claude plugin metadata | No | Copilot translates the Claude plugin metadata itself. |
|
||||
| Factory Droid | Native plugin marketplace pointed at the CE GitHub repository | No | Droid translates Claude Code plugins automatically. |
|
||||
| Qwen Code | Native extension install from the CE GitHub repository and existing Claude plugin metadata | No | Qwen translates Claude Code extensions automatically. |
|
||||
| Kimi Code CLI | Native plugin install from this repository using `.kimi-plugin/plugin.json` | No | Kimi can install directly from the GitHub repo and can browse the committed `.kimi-plugin/marketplace.json` custom catalog. |
|
||||
| OpenCode | Git-backed OpenCode plugin entry in `opencode.json` | No | `.opencode/plugins/compound-engineering.js` registers the CE skills directory directly. |
|
||||
| Pi | Git-backed Pi package install from this repository | No | Root `package.json` exposes `.pi/extensions/compound-engineering.ts` and the CE skills directory. `pi-ask-user` is a recommended companion for richer prompts. |
|
||||
| Antigravity CLI | Native plugin install from root `plugin.json` + `skills/`, or bundled `.agy/` entry point | No | `agy plugin install https://github.com/EveryInc/compound-engineering-plugin` for one-command remote install. `.agy/plugin.json` symlinks to the root manifest; `.agy/skills` symlinks to `skills/`. |
|
||||
| Cline | Native skills install via `.cline/scripts/install-skills.sh` | No | Symlinks invocable CE skills into `~/.cline/skills/` or `.cline/skills/`, skipping manual-only skills (`disable-model-invocation: true`). Enable Skills in the Cline extension settings. |
|
||||
|
||||
Kiro is no longer a documented CE install target. Historical converter and cleanup code may remain for regression coverage or old artifact handling, but user-facing install docs should not advertise Kiro.
|
||||
|
||||
## OpenCode
|
||||
|
||||
OpenCode can load plugins from git package entries in `opencode.json`. CE ships `.opencode/plugins/compound-engineering.js`, which resolves the repository's `skills` directory and appends it to OpenCode's skill paths.
|
||||
|
||||
Recommended config:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugin": ["compound-engineering@git+https://github.com/EveryInc/compound-engineering-plugin.git"]
|
||||
}
|
||||
```
|
||||
|
||||
For local development, point OpenCode at this checkout:
|
||||
|
||||
```json
|
||||
{
|
||||
"plugin": ["/path/to/compound-engineering-plugin/.opencode/plugins/compound-engineering.js"]
|
||||
}
|
||||
```
|
||||
|
||||
This replaces the old custom OpenCode Bun install path for normal CE users. The converter can still exist as development or compatibility tooling, but it is not the primary install story.
|
||||
|
||||
## Pi
|
||||
|
||||
Pi can install packages from git repositories. CE exposes a Pi package through root `package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"pi": {
|
||||
"extensions": ["./.pi/extensions/compound-engineering.ts"],
|
||||
"skills": ["./skills"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
pi install git:github.com/EveryInc/compound-engineering-plugin
|
||||
```
|
||||
|
||||
Recommended companion:
|
||||
|
||||
```bash
|
||||
pi install npm:pi-subagents
|
||||
pi install npm:pi-ask-user
|
||||
```
|
||||
|
||||
`pi-subagents` is required for CE workflows that dispatch reviewer, research, or implementation subagents. `pi-ask-user` is only for richer blocking question UX.
|
||||
|
||||
For local development:
|
||||
|
||||
```bash
|
||||
pi -e /path/to/compound-engineering-plugin
|
||||
```
|
||||
|
||||
## Antigravity CLI
|
||||
|
||||
Antigravity installs plugins from a local directory or a remote Git URL when the source root contains `plugin.json` and `skills/`. CE publishes both at the repository root.
|
||||
|
||||
Recommended one-command install:
|
||||
|
||||
```bash
|
||||
agy plugin install https://github.com/EveryInc/compound-engineering-plugin
|
||||
```
|
||||
|
||||
Local checkout:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/EveryInc/compound-engineering-plugin
|
||||
agy plugin install ./compound-engineering-plugin
|
||||
```
|
||||
|
||||
The committed `.agy/` bundle remains for explicit local installs (`agy plugin install ./compound-engineering-plugin/.agy`). Its `plugin.json` symlinks to the root manifest and `skills` symlinks to `../skills`.
|
||||
|
||||
`agy` still reads `GEMINI.md` as workspace context. See `.agy/INSTALL.md` for pinning, validation, and uninstall.
|
||||
|
||||
## Cline
|
||||
|
||||
Cline discovers skills from `~/.cline/skills/` (global) or `.cline/skills/` (project). CE ships `.cline/scripts/install-skills.sh`, which symlinks each directory under this repository's `skills/` into the chosen destination.
|
||||
|
||||
Recommended global install:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/EveryInc/compound-engineering-plugin
|
||||
./compound-engineering-plugin/.cline/scripts/install-skills.sh --global
|
||||
```
|
||||
|
||||
For local development from a checkout:
|
||||
|
||||
```bash
|
||||
/path/to/compound-engineering-plugin/.cline/scripts/install-skills.sh --global
|
||||
```
|
||||
|
||||
Enable **Settings -> Features -> Enable Skills** in the Cline extension, then start a new task. The install script skips manual-only skills marked `disable-model-invocation: true` in frontmatter (for example `lfg`, `ce-dogfood`, `ce-polish`) because Cline auto-activates skills from description matching alone. Pass `--include-manual` to link those skills for slash-command use, accepting that Cline may still auto-activate them. CE does not ship a separate Cline CLI `AgentPlugin` entry point; skills are the install surface.
|
||||
|
||||
## Kimi Code CLI
|
||||
|
||||
Kimi Code CLI has a native plugin surface, so CE should not maintain a Kimi converter target for normal installation. The root `.kimi-plugin/plugin.json` declares the CE skills directory with `skills: "./skills/"` and carries display metadata through Kimi's `interface` object.
|
||||
|
||||
Direct install:
|
||||
|
||||
```text
|
||||
/plugins install https://github.com/EveryInc/compound-engineering-plugin
|
||||
```
|
||||
|
||||
Marketplace install:
|
||||
|
||||
```text
|
||||
/plugins marketplace https://raw.githubusercontent.com/EveryInc/compound-engineering-plugin/main/.kimi-plugin/marketplace.json
|
||||
```
|
||||
|
||||
The Kimi marketplace catalog uses schema version `"2"` and entries with `id` plus `source`. It has no release-owned marketplace version, so release automation bumps only `.kimi-plugin/plugin.json` through the root `compound-engineering` component. `bun run release:validate` enforces Kimi manifest and marketplace parity with the Claude source manifest/catalog.
|
||||
|
||||
## Bun Package Posture
|
||||
|
||||
The root package remains useful for:
|
||||
|
||||
- Repo development scripts and tests.
|
||||
- OpenCode package metadata (`main`).
|
||||
- Pi package metadata (`pi` field).
|
||||
- Shared converter code and regression tests for historical or fixture targets.
|
||||
|
||||
It is not a public npm installer. Release automation should not publish `@every-env/compound-plugin`, and README install instructions should not rely on `bunx`.
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: OpenCode converter emits a temperature that Sonnet 5 / Opus 4.8 reject
|
||||
module: src/converters/claude-to-opencode.ts
|
||||
date: 2026-07-08
|
||||
problem_type: integration_issue
|
||||
component: tooling
|
||||
severity: high
|
||||
symptoms:
|
||||
- Converted OpenCode primary agent fails at runtime with HTTP 400 from Anthropic
|
||||
- Breakage appears only after bumping the Claude family alias map to claude-sonnet-5 / claude-opus-4-8
|
||||
- inferTemperature (CLI default) emits a temperature (0.1-0.6) on every converted agent
|
||||
- Primary agents pinned to claude-sonnet-5 or claude-opus-4-8 reject non-default temperature/top_p/top_k
|
||||
- The same conversion worked previously under claude-sonnet-4-6, which accepts temperature
|
||||
root_cause: config_error
|
||||
resolution_type: code_fix
|
||||
related_components:
|
||||
- src/utils/model.ts
|
||||
- tests/model-utils.test.ts
|
||||
- tests/converter.test.ts
|
||||
tags:
|
||||
- opencode
|
||||
- converter
|
||||
- temperature
|
||||
- sampling-params
|
||||
- claude-sonnet-5
|
||||
- claude-opus-4-8
|
||||
- model-alias
|
||||
---
|
||||
|
||||
# OpenCode converter emits a temperature that Sonnet 5 / Opus 4.8 reject
|
||||
|
||||
## Problem
|
||||
|
||||
Bumping the `sonnet`/`opus` aliases to a newer Claude generation (Sonnet 5, Opus 4.8) made the OpenCode converter emit an inferred `temperature` into primary-agent configs for models that reject non-default sampling params, producing configs the target runtime rejects with HTTP 400.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Converted OpenCode primary-agent configs carried an explicit `temperature` for agents pinned to Sonnet 5 (and Opus 4.7/4.8).
|
||||
- The generated config triggers an HTTP 400 from Anthropic at runtime because Sonnet 5 / Opus 4.7+ reject non-default `temperature`/`top_p`/`top_k`.
|
||||
- No test failed — no existing test exercised a primary OpenCode agent on a new-generation model, so the breakage was invisible in the suite.
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
The regression is easy to miss because cause and effect live in different files with no obvious link:
|
||||
|
||||
- The triggering change — bumping the `sonnet` alias to Sonnet 5 — looks purely mechanical: a single string swap in an alias map (`CLAUDE_FAMILY_ALIASES` in `src/utils/model.ts`). Nothing at the edit site hints that temperature emission is affected.
|
||||
- The failing behavior lives elsewhere: the converter's `inferTemperature` emission is in `src/converters/claude-to-opencode.ts`. Reviewing the alias bump in isolation gives no signal.
|
||||
- No existing test exercised a primary OpenCode agent on a new-generation model, so the suite stayed green.
|
||||
|
||||
A naive fix would over- or under-reach:
|
||||
|
||||
- Dropping `temperature` for **all** agents over-reaches — models that still accept sampling params (Sonnet 4, Haiku) would lose a valid inferred temperature.
|
||||
- String-matching `"sonnet"` under-reaches and misclassifies — it would wrongly suppress temperature for older accepting Sonnets (`claude-sonnet-4-20250514`) while missing rejecting Opus generations (`claude-opus-4-7`, `claude-opus-4-8`) entirely.
|
||||
|
||||
## Solution
|
||||
|
||||
Introduce a precise, canonical-ID-based predicate for "this model rejects sampling params," and gate the converter's temperature emission on it.
|
||||
|
||||
Added `rejectsSamplingParams` in `src/utils/model.ts`, backed by a set of canonical IDs. It resolves bare aliases via `resolveClaudeFamilyAlias` and strips the `anthropic/` provider prefix, so every spelling of the same model matches:
|
||||
|
||||
```ts
|
||||
const SAMPLING_PARAM_REJECTING_MODELS: ReadonlySet<string> = new Set([
|
||||
"claude-sonnet-5",
|
||||
"claude-opus-4-7",
|
||||
"claude-opus-4-8",
|
||||
])
|
||||
|
||||
export function rejectsSamplingParams(model: string): boolean {
|
||||
const canonical = resolveClaudeFamilyAlias(model).replace(/^anthropic\//, "")
|
||||
return SAMPLING_PARAM_REJECTING_MODELS.has(canonical)
|
||||
}
|
||||
```
|
||||
|
||||
This matches `sonnet`, `claude-sonnet-5`, and `anthropic/claude-sonnet-5`; it does **not** match `claude-sonnet-4-20250514` or `haiku`.
|
||||
|
||||
In `convertAgent` (`src/converters/claude-to-opencode.ts`), the temperature emission is gated so it is skipped only when a rejecting model was actually written to the config. The converter writes `model` only for primary agents, so the `frontmatter.model !== undefined` guard scopes this to primary agents; subagents (no model written — they inherit the parent session's model) keep existing behavior, out of scope because the runtime model is unknown at convert time.
|
||||
|
||||
Before:
|
||||
|
||||
```ts
|
||||
if (options.inferTemperature) {
|
||||
const temperature = inferTemperature(agent)
|
||||
if (temperature !== undefined) {
|
||||
frontmatter.temperature = temperature
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```ts
|
||||
if (options.inferTemperature) {
|
||||
const temperature = inferTemperature(agent)
|
||||
const modelRejectsTemperature =
|
||||
frontmatter.model !== undefined &&
|
||||
typeof agent.model === "string" &&
|
||||
rejectsSamplingParams(agent.model)
|
||||
if (temperature !== undefined && !modelRejectsTemperature) {
|
||||
frontmatter.temperature = temperature
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Why This Works
|
||||
|
||||
Sonnet 5 and Opus 4.7/4.8 return HTTP 400 for any non-default `temperature`/`top_p`/`top_k` (per Anthropic's Sonnet 5 and Opus 4.8 migration notes). The converter only writes `model` into the config for primary agents, so tying suppression to "we wrote a rejecting model" (`frontmatter.model !== undefined && rejectsSamplingParams(agent.model)`) targets exactly the case that would fail at runtime — a primary agent pinned to a rejecting model — without touching subagents or any model that still accepts sampling params.
|
||||
|
||||
## Prevention
|
||||
|
||||
The compounding lesson: **a model alias bump is never purely mechanical.** When you point an alias at a newer Claude generation, the model ID string is the smallest part of the change — audit every downstream emitter for API constraints that shifted with the generation. Newer generations (Sonnet 5+, Opus 4.7+) reject non-default sampling params, so any code path that emits `temperature`/`top_p`/`top_k` for a resolved Claude model must gate on model compatibility.
|
||||
|
||||
Concrete guardrails:
|
||||
|
||||
- Keep `SAMPLING_PARAM_REJECTING_MODELS` in sync with `CLAUDE_FAMILY_ALIASES` whenever a new generation is added — both live in `src/utils/model.ts` and are co-located intentionally so the two are edited together.
|
||||
- Test both directions so neither over- nor under-reach regresses: a rejecting model (temperature suppressed) and an accepting model (temperature still inferred). The tests added are `rejectsSamplingParams` unit tests in `tests/model-utils.test.ts` (alias resolution, the `anthropic/` prefix, and the accepting cases `claude-sonnet-4-20250514` / `haiku`), plus a converter test in `tests/converter.test.ts` asserting temperature is suppressed for a Sonnet 5 primary agent but still inferred (0.1) for a Haiku agent.
|
||||
- Verify bot-sourced API claims against the source of truth. An automated cross-model code-review bot (Codex) caught this before merge, but its factual claim about the HTTP 400 was confirmed against authoritative Anthropic migration docs before building the fix — the claim was true, but bot claims about API behavior must always be verified first.
|
||||
|
||||
## Related
|
||||
|
||||
- [`cross-platform-model-field-normalization.md`](cross-platform-model-field-normalization.md) — the sibling doc covering how bare Claude aliases resolve to provider-prefixed canonical IDs across target platforms. That doc owns the alias-to-provider normalization *mechanism* (`CLAUDE_FAMILY_ALIASES`, `normalizeModelWithProvider`); this doc covers a distinct downstream constraint (sampling-param API limits of newer generations). Both live in `src/utils/model.ts`.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Plugin Versioning and Documentation Requirements
|
||||
category: workflow
|
||||
tags: [versioning, changelog, readme, plugin, documentation]
|
||||
created: 2025-11-24
|
||||
date: 2026-03-17
|
||||
last_updated: 2026-06-23
|
||||
severity: process
|
||||
component: plugin-development
|
||||
---
|
||||
|
||||
# Plugin Versioning and Documentation Requirements
|
||||
|
||||
## Problem
|
||||
|
||||
When making changes to the compound-engineering plugin, documentation can get out of sync with the actual components (agents, commands, skills). This leads to confusion about what's included in each version and makes it difficult to track changes over time.
|
||||
|
||||
This document applies to release-owned plugin metadata and changelog surfaces for the `compound-engineering` plugin, not ordinary feature work.
|
||||
|
||||
The broader repo-level release model now lives in:
|
||||
|
||||
- `docs/solutions/workflow/manual-release-please-github-releases.md`
|
||||
|
||||
That doc covers the standing release PR, component ownership across the root `compound-engineering` package and the marketplace packages, and the GitHub Releases model for published release notes. This document stays narrower: it is the plugin-scoped reminder for contributors changing the root plugin surface.
|
||||
|
||||
## Solution
|
||||
|
||||
**Routine PRs should not cut plugin releases.**
|
||||
|
||||
Embedded plugin versions are release-owned metadata. Release automation prepares the next versions and changelog entries after deciding which merged changes ship together. Because multiple PRs may merge before release, contributors should not guess release versions inside individual PRs.
|
||||
|
||||
Contributors should:
|
||||
|
||||
1. **Avoid release bookkeeping in normal PRs**
|
||||
- Do not manually bump `package.json`, `.claude-plugin/plugin.json`, `.cursor-plugin/plugin.json`, `.codex-plugin/plugin.json`, or `.agy/plugin.json`
|
||||
- Do not manually bump the `compound-engineering` entry in `.claude-plugin/marketplace.json`
|
||||
- Do not cut release sections in the root `CHANGELOG.md`
|
||||
|
||||
2. **Keep substantive docs accurate**
|
||||
- Verify component counts match actual files
|
||||
- Verify agent/command/skill tables are accurate
|
||||
- Update descriptions if functionality changed
|
||||
- Run `bun run release:validate` when plugin inventories or release-owned descriptions may have changed
|
||||
|
||||
## Checklist for Plugin Changes
|
||||
|
||||
```markdown
|
||||
Before committing changes to compound-engineering plugin:
|
||||
|
||||
- [ ] No manual version bump in root package/plugin manifests
|
||||
- [ ] No manual version bump in the `compound-engineering` entry inside `.claude-plugin/marketplace.json`
|
||||
- [ ] No manual release section added to `CHANGELOG.md`
|
||||
- [ ] README.md component counts verified
|
||||
- [ ] README.md tables updated (if adding/removing/renaming)
|
||||
- [ ] plugin.json description updated (if component counts changed)
|
||||
- [ ] `bun run release:validate` passes
|
||||
```
|
||||
|
||||
## File Locations
|
||||
|
||||
- Plugin version is release-owned: `package.json`, `.claude-plugin/plugin.json`, `.cursor-plugin/plugin.json`, `.codex-plugin/plugin.json`, and `.agy/plugin.json`
|
||||
- Marketplace entry is release-owned: `.claude-plugin/marketplace.json`
|
||||
- Release notes are release-owned: GitHub release PRs and GitHub Releases
|
||||
- Readme: `README.md`
|
||||
|
||||
## Example Workflow
|
||||
|
||||
When adding, removing, or renaming a skill:
|
||||
|
||||
1. Create or remove the directory under `skills/`
|
||||
2. Update `README.md`
|
||||
3. Leave plugin version selection and canonical release-note generation to release automation
|
||||
4. Run `bun run release:validate`
|
||||
|
||||
## Prevention
|
||||
|
||||
This documentation serves as a reminder. When maintainers or agents work on this plugin, they should:
|
||||
|
||||
1. Check this doc before committing changes
|
||||
2. Follow the checklist above
|
||||
3. Do not guess release versions in feature PRs
|
||||
4. Refer to the repo-level release learning when the question is about batching, release PR behavior, or multi-component ownership rather than plugin-only bookkeeping
|
||||
|
||||
## Related Files
|
||||
|
||||
- `.claude-plugin/plugin.json`
|
||||
- `.cursor-plugin/plugin.json`
|
||||
- `.codex-plugin/plugin.json`
|
||||
- `.agy/plugin.json`
|
||||
- `README.md`
|
||||
- `package.json`
|
||||
- `CHANGELOG.md`
|
||||
- `docs/solutions/workflow/manual-release-please-github-releases.md`
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "$ARGUMENTS is reliably substituted inside SKILL.md only on Claude Code — reason over the user's prompt instead"
|
||||
date: 2026-06-26
|
||||
last_updated: 2026-07-12
|
||||
category: skill-design
|
||||
module: "skills (argument handling across harnesses)"
|
||||
problem_type: convention
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Authoring or reviewing a skill that needs the user's invocation arguments inside SKILL.md
|
||||
- A skill must work on more than one harness (Claude Code, Codex, Cursor, Gemini, Kiro)
|
||||
- "Scanning the prompt for a flag token (output:, mode:, delegate:) rather than only injecting a description"
|
||||
- Deciding whether to depend on the $ARGUMENTS substitution token in skill body prose
|
||||
tags:
|
||||
- skill-authoring
|
||||
- cross-harness
|
||||
- arguments
|
||||
- claude-arguments
|
||||
- prompt-reasoning
|
||||
- portability
|
||||
- flag-parsing
|
||||
related_components:
|
||||
- development_workflow
|
||||
- documentation
|
||||
---
|
||||
|
||||
# $ARGUMENTS is reliably substituted inside SKILL.md only on Claude Code — reason over the user's prompt instead
|
||||
|
||||
## Context
|
||||
|
||||
Skills routinely need the user's invocation input — both as free-form description and as flag tokens like `output:html`, `mode:headless`, `delegate:codex`. The established mechanism is the Claude Code `$ARGUMENTS` substitution: the harness replaces `$ARGUMENTS` in the skill body with the user's argument string before the model sees it. This plugin is authored once and converted to Codex, Cursor, Gemini, and Kiro, so the question is whether `$ARGUMENTS` ports.
|
||||
|
||||
It is a sibling of the bundled-script path problem (see `bundled-script-path-resolution-across-harnesses.md`): a Claude-Code-specific construct that looks portable in source but isn't guaranteed off Claude. It surfaced while reframing the output-format precedence in `ce-plan`/`ce-brainstorm`/`ce-ideate`: the resolution prose said "scan `$ARGUMENTS` for a token starting with `output:`," which is Claude-only phrasing for what is really "the user typed a format request in their prompt."
|
||||
|
||||
## Guidance
|
||||
|
||||
**`$ARGUMENTS` substitution inside a SKILL.md body is only confirmed on Claude Code.** Per the target specs in `docs/specs/`: Codex documents `$1`–`$9`, `$ARGUMENTS`, and named placeholders for **prompts** (skill-body behavior is not documented); Cursor documents only `$1`/`$2` for commands; Kiro lists `$ARGUMENTS` interpolation as **"Lost."** The converter rewrites the `argument-hint` frontmatter into an `## Arguments` section but does **not** rewrite inline body `$ARGUMENTS` (the OpenCode writer even emits it deliberately). So inline `$ARGUMENTS` in skill prose is a portability risk off Claude.
|
||||
|
||||
Separate the two uses — they take different fixes:
|
||||
|
||||
- **Reasoning / flag detection** ("scan `$ARGUMENTS` for `output:`/`mode:`"). The agent does not need the token here — the user's request is already in its context on every harness. Phrase it harness-neutrally: **reason over the user's prompt** for the intent, honoring both the shorthand token (`output:html`) and plain language ("make this a webpage"). Add the discriminating guard: a format/flag named as **subject matter** ("add an HTML export feature", "plan the CSV importer") is the work, not a flag — do not act on it.
|
||||
- **Input injection** (`<feature_description> #$ARGUMENTS </feature_description>`). On Claude this is *how* the description reaches the agent inline, but the token is not actually necessary — the user's request is redundantly present in the agent's context on every harness (Claude Code invokes a skill via the Skill tool, so the user's turn stays in the transcript; the OpenCode converter injects args through its own command stub; Codex/Gemini/Cursor load the skill mid-conversation). Two tiers of fix, in increasing cleanliness:
|
||||
- *Minimal (graceful degradation):* keep the token but pair it with a fallback — *"if this shows a literal `$ARGUMENTS`, the harness did not substitute it — use the user's actual request from the conversation."* Mirrors the pre-resolution-with-fallback pattern AGENTS.md prescribes for `${CLAUDE_PLUGIN_ROOT}`. Fixes the failure but leaves the Claude-only token in place with prose wrapped around it.
|
||||
- *Preferred (remove the token entirely):* replace the slot with a prose binding that reads the input from the invocation — e.g. "the **feature description** is the input this skill was invoked with, present in the current prompt or conversation." This removes the "was it substituted?" question by construction instead of papering over it. Three things must be preserved when you do this:
|
||||
1. **Named references.** Several skills use the injection tag as a *variable* elsewhere (`<input_document>`, `<bug_description>`, `{focus_hint}`), so define the name in the prose ("the rest of this skill refers to it as `<input_document>`") rather than orphaning those references.
|
||||
2. **Empty/clarify handling.** Route a missing input into the skill's own "ask the user" / "proceed open-ended" path rather than adding a competing one.
|
||||
3. **Caller-neutral semantics — the subtle one.** `$ARGUMENTS` is *whatever the skill was invoked with*, by **any** caller. Bind the input to "the input this skill was invoked with," **not** "the user's request" — because a skill is often invoked by *another skill* in `mode:pipeline` (e.g. `ce-babysit-pr` calls `ce-debug` passing failing jobs and log tails as the argument; `lfg` calls `ce-plan`/`ce-work` with a payload). A binding that says "read the user's request" makes a pipeline-delegated skill ignore the caller's payload and parse an empty input, silently breaking the autonomous path. This was caught in review on `ce-debug`'s binding: the first-pass rewrite narrowed the input to "the user," and the fix was to phrase it as the invocation input from the user *or* a calling skill. The prose-logic phrasing "the arguments you were invoked with" was already caller-neutral; only the injection-slot bindings needed this widening.
|
||||
|
||||
This was applied to every injection-slot skill in #1110 (open as of this writing).
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The failure is **recoverable and loud**, which is why this is `medium`, not `high`. Because the user's input is redundantly present in the conversation, a capable agent on Cursor/Kiro that meets a literal `$ARGUMENTS` routes around it and uses the real request; and if it does misfire, the output is visibly wrong (`$ARGUMENTS` echoed, or "planning $ARGUMENTS") and the user re-prompts. Contrast the silent-failure class — bundled-script `exit 127`, an empty `${CLAUDE_PLUGIN_ROOT}` — where nothing announces the break.
|
||||
|
||||
The quieter, higher-value risk is **flag-scanning**: an unsubstituted token means the scan finds no flag and falls to defaults — harmless for `output:` (defaults to `md`), but a missed `mode:` could skip an intended mode without any visible signal. Reasoning over the prompt removes that failure mode entirely.
|
||||
|
||||
Dominant harnesses (Claude native, Codex documented for prompts) are fine; the genuinely-exposed slice is Cursor/Kiro users invoking a flag-scanning skill on a weaker model. Worth fixing for robustness, not worth treating as a blocker.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any skill that reads invocation arguments inside SKILL.md — especially one that **scans for flag tokens** rather than only injecting a description.
|
||||
- When generalizing a Claude-only mechanism for cross-harness use: prefer describing the **capability** ("the user's prompt for this run") over naming the **token** (`$ARGUMENTS`), consistent with AGENTS.md "Platform-Specific Variables in Skills."
|
||||
|
||||
## Examples
|
||||
|
||||
Flag detection — before (Claude-only) vs after (harness-neutral), from the output-format resolution tier:
|
||||
|
||||
```text
|
||||
# Before
|
||||
1. CLI arg. Scan $ARGUMENTS for a token starting with the literal prefix
|
||||
output:. If found, strip it from arguments before treating the remainder
|
||||
as the feature description.
|
||||
|
||||
# After
|
||||
1. In-prompt request. Reason over the user's prompt for this run for a request
|
||||
about this document's output format, expressed either as the output:
|
||||
shorthand or in plain language ("make the plan a webpage"). Ignore the
|
||||
output: token when reading the rest of the prompt as the feature
|
||||
description. Distinguish a request about the document's format from a format
|
||||
named as subject matter: "add an HTML export feature" is the work, not a
|
||||
doc-format request — do not switch on it.
|
||||
```
|
||||
|
||||
Injection point — two fixes. Minimal (keep the token, add a fallback):
|
||||
|
||||
```text
|
||||
<feature_description> #$ARGUMENTS </feature_description>
|
||||
|
||||
(If the block above shows a literal "$ARGUMENTS", your harness did not
|
||||
substitute it — use the user's actual request from the conversation.)
|
||||
```
|
||||
|
||||
Preferred (remove the token; bind the input in prose, preserving the named
|
||||
reference downstream logic uses):
|
||||
|
||||
```text
|
||||
The input document for this run is the input this skill was invoked with —
|
||||
present in the current prompt or conversation, whether the user provided it
|
||||
directly or a calling skill passed it (e.g. in mode:pipeline). The rest of this
|
||||
skill refers to it as <input_document>; if nothing was provided, treat
|
||||
<input_document> as blank.
|
||||
```
|
||||
|
||||
**Status / scope note (updated 2026-07-12):** both items the original capture deferred are now done, and the fix landed cleaner than the deferred plan.
|
||||
|
||||
- **The broad sweep is complete, via removal rather than fallback.** Every `$ARGUMENTS` reference was removed from all skill *bodies* — five prose-logic references reworded to "the arguments you were invoked with," and ten input-injection slots converted to conversation-sourced prose bindings (preserving named references and each skill's empty/clarify handling). Done in #1110 (open as of this writing). The OpenCode command-stub converter still emits its own `$ARGUMENTS` (`src/converters/claude-to-opencode.ts`) and is untouched — removal applies to skill bodies, not the converter's generated command entry.
|
||||
- **The empirical question is settled by the 2-harness probe the original note called for.** The *current* skill bodies (no `$ARGUMENTS`) were injected into fresh Claude and Codex subagents across five input-ingestion cases — explicit request, empty/bare invocation, `mode:` token stripping, blank-input discovery, and subject identification — covering `ce-plan`, `ce-work`, and `ce-pov`. Result: **5/5 on both hosts**, every run confirming no `$ARGUMENTS` reliance. Codex derived the input from the conversation, stripped mode tokens, and hit the ask/discovery paths identically to Claude. So "reason over the prompt / read from the conversation" is verified cross-host, not just reasoned from specs.
|
||||
|
||||
Still open (low-priority): a portability note in AGENTS.md's "Platform-Specific Variables in Skills" section — `$ARGUMENTS` is still not called out there by name. See `bundled-script-path-resolution-across-harnesses.md` and `codex-skill-prompt-entrypoints.md` for adjacent cross-harness skill-portability findings.
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Authoring "Make It Automatic" auto-invoke guidance for CE skills
|
||||
date: 2026-07-12
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: convention
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Adding a "Make It Automatic" / "Make Capture Automatic" section to a skill's docs page
|
||||
- Writing a standing instruction that tells an agent to auto-invoke a skill from AGENTS.md/CLAUDE.md
|
||||
- Wiring a mutating skill (ce-simplify-code, ce-compound) to run on its own without user prompting
|
||||
tags:
|
||||
- auto-invoke
|
||||
- standing-instruction
|
||||
- ce-simplify-code
|
||||
- ce-compound
|
||||
- defense-in-depth
|
||||
- cross-harness
|
||||
related_components:
|
||||
- ce-simplify-code
|
||||
- ce-compound
|
||||
---
|
||||
|
||||
# Authoring "Make It Automatic" auto-invoke guidance for CE skills
|
||||
|
||||
## Context
|
||||
|
||||
Several CE skills deliver their value at a moment that's easy to forget — `ce-simplify-code` right after a feature settles, `ce-compound` right after a fix verifies. Users bridge that gap by adding a standing instruction to their agent-instructions file (`AGENTS.md`/`CLAUDE.md`, or a global `~/.claude/CLAUDE.md` / `~/.codex/AGENTS.md`) so the agent invokes the skill on its own. A naive instruction does more harm than good in two concrete ways: it wastes cycles (a homegrown "always simplify after changes" rule ran `ce-simplify-code`'s three reviewer subagents on documentation-only diffs, which yield nothing), and it fires at the wrong moment (mid-build, where a simplification pass rewrites code still being shaped).
|
||||
|
||||
`ce-compound` established the pattern with its "Make Capture Automatic" section (PR #1110). This learning generalizes how to author these sections for any CE skill, based on adding one to `ce-simplify-code` (PR #1113), and pairs it with the skill-side self-guard that makes the whole thing robust.
|
||||
|
||||
## Guidance
|
||||
|
||||
### 1. Two layers, different jobs — duplicate the no-yield boundary, keep cost policy in the caller
|
||||
|
||||
Put the skip logic in **both** the standing instruction and the skill, but split responsibility:
|
||||
|
||||
- **The standing instruction carries the full activation gate**: timing, the no-yield exclusions, the size floor, and a not-already-run guard. This is the cheap layer — not invoking the skill at all beats invoking it only to have it bail.
|
||||
- **The skill self-guards on the no-yield boundary only** — if the resolved scope has no substantive change of the kind the skill acts on, it short-circuits before spending subagents. Crucially, the skill gates on the **kind** of change, **never on size**: an explicit user-named scope on a small function is authoritative and must still run. The numeric size floor is a *cost policy* that belongs only in the caller/standing instruction, not baked into the skill where it would override a legitimate explicit invocation.
|
||||
|
||||
`ce-simplify-code` had no self-guard before this: `SKILL.md` Step 1 resolved any non-empty scope and Step 2 unconditionally dispatched three reviewers, so a markdown-only diff burned three dispatches. The added preflight (`skills/ce-simplify-code/SKILL.md` Step 1) short-circuits on a no-code scope and narrows to code files on a mixed diff.
|
||||
|
||||
### 2. Anchor timing to a completion boundary, not per-edit
|
||||
|
||||
The instruction must fire when a unit of work has *settled* — "when you finish a coherent unit of work / before you review, commit, or hand it off" — with an explicit negative: **not after every individual edit or intermediate fix while still building**. Firing per-edit makes the agent rewrite code it's still shaping, which is worse than not running at all. Vague wording like "after significant changes" with no boundary and no numeric floor lets eager agents fire constantly.
|
||||
|
||||
### 3. Present offer-first and auto-run as peer variants — don't reflexively recommend the "safe" one
|
||||
|
||||
When a skill is safe by construction — behavior-preserving, refuses to weaken tests/types, never strips a safety check, verifies before finishing, and lands edits on a branch the user reviews before commit — auto-run is **not** the reckless option and offer-first is **not** the safe one. Frame the choice as *interruption preference, not risk*, and let the reader pick. Reflexively stamping offer-first "recommended" over-weights a risk the skill design already handles.
|
||||
|
||||
### 4. Cross-harness phrasing rules
|
||||
|
||||
These sections are read by whatever agent the user runs (Claude Code, Codex, Gemini, Cursor):
|
||||
|
||||
- **"invoke the `<skill>` skill"**, never "run `/<skill>`" — the slash-command form is not reliably agent-callable across harnesses; reference the capability, not the keystroke.
|
||||
- **"before review, commit, or handoff"**, not "at the end of the session" — an agent can't reliably detect session end but does know an imminent workflow boundary.
|
||||
- Key the eligibility on **"substantive human-authored code"**, not a filename allowlist — tests, migrations, and code-bearing config carry real yield, and a mixed code+docs diff still qualifies (the skill scopes to the code).
|
||||
- Exclusions are the **load-bearing** part and must be hard negatives: documentation-/Markdown-only, formatting/lint-only, dependency/lockfile, generated/vendored, other purely mechanical churn.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The failure this prevents is concrete: three reviewer subagents dispatched against a diff that can only come back empty, repeated on every docs commit, plus mid-build churn that fights the user. The two-layer split means the common automatic path is cheap (instruction gates before invoking) *and* direct invocation is safe (skill bails on a no-yield scope no matter how it was reached). Keeping the size floor out of the skill preserves the authority of an explicit `/ce-simplify-code` on a small change. Getting the cross-harness phrasing right is what makes a single authored section work on every harness the plugin targets.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Apply when adding a "Make It Automatic" section to any CE skill's docs page, or when wiring a mutating skill to auto-invoke. The pattern assumes a skill whose value is concentrated at a completion boundary and whose reviewers/agents have a no-yield input class worth excluding. A skill that is cheap, always-relevant, or has no distinct no-yield class needs neither the exclusions nor the self-guard.
|
||||
|
||||
## Examples
|
||||
|
||||
The self-guard is verifiable: a cross-host routing eval (5 resolved-scope scenarios × Claude + Codex, fresh subagents reading the on-disk `SKILL.md`) scored 10/10 — docs-only and lockfile/generated scopes short-circuited, a mixed diff narrowed to its code file, and both a ~5-line explicit-scope case and a normal code diff ran, confirming the guard keys on change kind, not size.
|
||||
|
||||
Standing-instruction shape (auto-run variant, from `docs/skills/ce-simplify-code.md`):
|
||||
|
||||
> When you finish a coherent unit of work — a feature is complete, or you're wrapping up to open a PR — and before you review, commit, or hand it off, automatically invoke the `ce-simplify-code` skill on the changed code. Do this at that completion checkpoint only, not after every individual edit or intermediate fix while you're still building. Run it only when the accumulated diff has at least 10 substantive code lines and the skill hasn't already run since the last code edit. Never run it for documentation- or Markdown-only changes; formatting-, lint-, or dependency/lockfile-only changes; generated or vendored files; other purely mechanical changes; or code you've said to keep as written.
|
||||
|
||||
Skill-side preflight (from `skills/ce-simplify-code/SKILL.md` Step 1): if the resolved scope contains no substantive human-authored code, stop with a one-line "nothing to simplify" note; on a mixed diff, narrow to the code files. Gates on kind of change only, never size.
|
||||
|
||||
## See Also
|
||||
|
||||
- [`portable-agent-skill-authoring.md`](./portable-agent-skill-authoring.md) — the canonical cross-model/cross-harness authoring guide these phrasing rules instantiate
|
||||
- [`discoverability-check-for-documented-solutions.md`](./discoverability-check-for-documented-solutions.md) — the sibling pattern of a skill making a small, principled edit to an instruction file
|
||||
- [`post-menu-routing-belongs-inline.md`](./post-menu-routing-belongs-inline.md) — related SKILL.md authoring-placement discipline
|
||||
- `ce-compound`'s "Make Capture Automatic" section (`docs/skills/ce-compound.md`, PR #1110) — the pattern this generalizes
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
title: "Beta-to-stable skill promotion: wiring legacy cleanup so the stale beta dir is actually swept"
|
||||
category: skill-design
|
||||
date: 2026-06-28
|
||||
module: legacy-cleanup
|
||||
problem_type: convention
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "promoting a skill from beta (ce-X-beta) to stable (ce-X) and registering the old dir for legacy flat-install cleanup"
|
||||
tags:
|
||||
- skill-promotion
|
||||
- legacy-cleanup
|
||||
- beta-rename
|
||||
- stale-skill-dirs
|
||||
- flat-install
|
||||
related:
|
||||
- docs/solutions/skill-design/beta-skills-framework.md
|
||||
- docs/solutions/skill-design/beta-promotion-orchestration-contract.md
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
When a compound-engineering skill graduates from beta to stable, its directory is renamed `ce-X-beta` -> `ce-X`. Users who installed the plugin before the rename will have a stale `ce-X-beta/` directory sitting next to the new `ce-X/` in their flat-install layout. `src/utils/legacy-cleanup.ts` is supposed to sweep that stale dir on the next upgrade.
|
||||
|
||||
The trap: the two edits that *look* sufficient — adding the old name to `STALE_SKILL_DIRS` and adding a `LEGACY_SKILL_DESCRIPTION_ALIASES` entry — are a **silent no-op** on their own. The existing precedent (`ce-polish-beta`) shipped with exactly that incomplete wiring, so its cleanup never ran, and there was no error, log, or failing test to reveal it. The `ce-dogfood-beta -> ce-dogfood` promotion would have inherited the same dead code by copying the precedent.
|
||||
|
||||
## Guidance
|
||||
|
||||
A correct beta-to-stable rename requires **four edits across two files plus a regression test**. The load-bearing one is easy to miss.
|
||||
|
||||
### How the mechanism works
|
||||
|
||||
`cleanupStaleSkillDirs(skillsRoot)` iterates `STALE_SKILL_DIRS` and, for each name, calls:
|
||||
|
||||
```ts
|
||||
isLegacyPluginOwned(targetPath, skills.get(name), null)
|
||||
```
|
||||
|
||||
`isLegacyPluginOwned` guards early:
|
||||
|
||||
```ts
|
||||
if (!expectedDescription) return false
|
||||
```
|
||||
|
||||
Only *after* that guard does it read the on-disk `SKILL.md` description and compare it against `expectedDescription` **plus** any `LEGACY_SKILL_DESCRIPTION_ALIASES[basename]` entries. So if `expectedDescription` is `undefined`, the alias list is never consulted.
|
||||
|
||||
`expectedDescription` comes from the `skills` map seeded in `loadLegacyFingerprints()`. For each `STALE_SKILL_DIRS` name:
|
||||
|
||||
```ts
|
||||
const currentPath = skillIndex.get(currentSkillNameForLegacy(name))
|
||||
if (currentPath) {
|
||||
// seed = the currently-shipping skill's CURRENT description
|
||||
} else if (LEGACY_ONLY_SKILL_DESCRIPTIONS[name]) {
|
||||
// seed = hardcoded last-shipped description (for FULLY-RETIRED skills, no replacement)
|
||||
} else {
|
||||
// seed = undefined -> isLegacyPluginOwned bails immediately
|
||||
}
|
||||
```
|
||||
|
||||
`currentSkillNameForLegacy` has explicit `case`s for skills renamed to a *different* name, and a default that returns any `ce-`-prefixed name unchanged:
|
||||
|
||||
```ts
|
||||
default:
|
||||
return legacyName.startsWith("ce-") ? legacyName : `ce-${legacyName}`
|
||||
```
|
||||
|
||||
So `currentSkillNameForLegacy("ce-dogfood-beta")` returns `"ce-dogfood-beta"`. After the rename that name is no longer in the skill index, the seed stays `undefined`, the alias is never reached, and nothing is swept.
|
||||
|
||||
### The correct wiring
|
||||
|
||||
**1 — `STALE_SKILL_DIRS` (`src/utils/legacy-cleanup.ts`)** — register the old name:
|
||||
|
||||
```ts
|
||||
// ce-dogfood-beta -> ce-dogfood (promoted to stable)
|
||||
"ce-dogfood-beta",
|
||||
```
|
||||
|
||||
**2 (load-bearing) — `currentSkillNameForLegacy` (`src/utils/legacy-cleanup.ts`)** — map the beta name to the shipping stable name so the seed resolves to a real description:
|
||||
|
||||
```ts
|
||||
case "ce-polish-beta":
|
||||
return "ce-polish"
|
||||
case "ce-dogfood-beta":
|
||||
return "ce-dogfood"
|
||||
```
|
||||
|
||||
**3 — `LEGACY_SKILL_DESCRIPTION_ALIASES` (`src/utils/legacy-cleanup.ts`)** — the seed is now the *new* stable description, but the stale dir on disk still carries the *old* beta description. Add the **verbatim last-shipped beta `description:`** so the old-on-disk file matches:
|
||||
|
||||
```ts
|
||||
"ce-dogfood-beta": [
|
||||
"[BETA] Hands-off end-to-end branch dogfood pass with browser testing, auto-fixes, regression tests, and fix commits.",
|
||||
],
|
||||
```
|
||||
|
||||
**4 — `EXTRA_LEGACY_ARTIFACTS_BY_PLUGIN["compound-engineering"]` (`src/data/plugin-legacy-artifacts.ts`)** — register the name with the universal artifact sweeper:
|
||||
|
||||
```ts
|
||||
"ce-dogfood-beta",
|
||||
```
|
||||
|
||||
**5 — Regression test (`tests/legacy-cleanup.test.ts`)** — mirror the existing "removes ce-review and ce-document-review (renamed skills)" test, but create the stale dir with its **old beta description** (the realistic upgrade state), not the current stable one. A test seeded with the current description would pass for the wrong reason and would not catch a missing step 2:
|
||||
|
||||
```ts
|
||||
test("removes promoted-from-beta skill dirs via their last-shipped beta description (ce-dogfood-beta, ce-polish-beta)", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "cleanup-beta-promoted-"))
|
||||
await createDir(
|
||||
path.join(root, "ce-dogfood-beta"),
|
||||
skillContent(
|
||||
"ce-dogfood-beta",
|
||||
"[BETA] Hands-off end-to-end branch dogfood pass with browser testing, auto-fixes, regression tests, and fix commits.",
|
||||
),
|
||||
)
|
||||
await createDir(
|
||||
path.join(root, "ce-polish-beta"),
|
||||
skillContent(
|
||||
"ce-polish-beta",
|
||||
"Start the dev server, open the feature in a browser, and iterate on improvements together. Manual invocation only — type /ce-polish to run it.",
|
||||
),
|
||||
)
|
||||
|
||||
const removed = await cleanupStaleSkillDirs(root)
|
||||
|
||||
expect(removed).toBe(2)
|
||||
expect(await exists(path.join(root, "ce-dogfood-beta"))).toBe(false)
|
||||
expect(await exists(path.join(root, "ce-polish-beta"))).toBe(false)
|
||||
})
|
||||
```
|
||||
|
||||
`cleanupStaleSkillDirs(root)` takes a single argument; it loads the fingerprint map internally.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Cleanup is the only thing stopping stale beta dirs from accumulating in users' flat installs after an upgrade. A lingering `ce-dogfood-beta/` next to `ce-dogfood/` means two skill definitions for the same concept — listing conflicts and extra tokens loaded on every invocation. The failure mode is silent: no error, no log, the dir just stays.
|
||||
|
||||
The `ce-polish-beta` cleanup had been shipped live and broken. The existing suite never caught it because the only rename tests (`ce-review -> ce-code-review`, `ce-document-review -> ce-doc-review`) all use names with explicit `currentSkillNameForLegacy` cases — so they pass through step 2 by construction. Any beta promotion that copies the `ce-polish-beta` pattern as a template inherits the same no-op. This wiring was fixed for both `ce-dogfood-beta` and `ce-polish-beta` in the dogfood promotion.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Apply this every time a skill is promoted from beta to stable (`ce-X-beta` renamed to `ce-X`). Land all four edits plus the test in the same PR that performs the rename — not a follow-up. It does **not** apply to fully-retired skills with no replacement; those seed their fingerprint via `LEGACY_ONLY_SKILL_DESCRIPTIONS` instead of the `currentSkillNameForLegacy` path.
|
||||
|
||||
## Examples
|
||||
|
||||
### Before — the incomplete `ce-polish-beta` pattern (silent no-op)
|
||||
|
||||
```ts
|
||||
// src/utils/legacy-cleanup.ts
|
||||
export const STALE_SKILL_DIRS = [
|
||||
"ce-polish-beta", // step 1: present
|
||||
]
|
||||
// step 2: MISSING — no currentSkillNameForLegacy case, so the default returns
|
||||
// "ce-polish-beta" (not a shipping skill) -> seed undefined -> isLegacyPluginOwned
|
||||
// bails before the alias is read -> nothing swept
|
||||
const LEGACY_SKILL_DESCRIPTION_ALIASES = {
|
||||
"ce-polish-beta": [ /* present, but unreachable */ ],
|
||||
}
|
||||
```
|
||||
|
||||
Steps 1, 3, 4 present; step 2 missing; step 5 absent. Stale `ce-polish-beta/` dirs survive upgrades forever. No error, no test failure.
|
||||
|
||||
### After — complete wiring
|
||||
|
||||
Add the `currentSkillNameForLegacy` case (step 2) so the seed resolves to the stable skill's current description, keep the alias (step 3) so the old-on-disk description still matches, and add the regression test (step 5) that proves the sweep fires. Result: `bun test` 1600 pass / 0 fail; `bun run release:validate` in sync; stale beta dirs swept on the first upgrade after the rename.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: "Beta-to-stable promotions must update orchestration callers atomically"
|
||||
category: skill-design
|
||||
date: 2026-03-23
|
||||
module: skills
|
||||
component: SKILL.md
|
||||
tags:
|
||||
- skill-design
|
||||
- beta-testing
|
||||
- rollout-safety
|
||||
- orchestration
|
||||
severity: medium
|
||||
description: "When promoting a beta skill to stable, update all orchestration callers in the same PR so they pass correct mode flags instead of inheriting defaults."
|
||||
related:
|
||||
- docs/solutions/skill-design/beta-skills-framework.md
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
When a beta skill introduces new invocation semantics (e.g., explicit mode flags), promoting it over its stable counterpart without updating orchestration callers causes those callers to silently inherit the wrong default behavior.
|
||||
|
||||
## Solution
|
||||
|
||||
Treat promotion as an orchestration contract change, not a file rename.
|
||||
|
||||
1. Replace the stable skill with the promoted content
|
||||
2. Update every workflow that invokes the skill in the same PR
|
||||
3. Hardcode the intended mode at each callsite instead of relying on the default
|
||||
4. Add or update contract tests so the orchestration assumptions are executable
|
||||
|
||||
## Applied: ce-review-beta -> ce-code-review (2026-03-24)
|
||||
|
||||
This pattern was applied when promoting the review beta (`ce-review-beta`, tracked as the legacy artifact `ce:review-beta`/`ce-review-beta` in the cleanup registry) into the stable `ce-code-review` skill. The caller contract at the time:
|
||||
|
||||
- `lfg` -> `/ce-code-review mode:autofix` (enforced by `tests/review-skill-contract.test.ts`)
|
||||
`slfg` has since been removed, but the durable rule remains: every orchestrator that invokes the promoted skill must pass the intended mode explicitly and have contract coverage where the caller still exists.
|
||||
|
||||
## Prevention
|
||||
|
||||
- When a beta skill changes invocation semantics, its promotion plan must include caller updates as a first-class implementation unit
|
||||
- Promotion PRs should be atomic: promote the skill and update orchestrators in the same branch
|
||||
- Add contract coverage for the promoted callsites so future refactors cannot silently drop required mode flags
|
||||
- Do not rely on “remembering later” for orchestration mode changes; encode them in docs, plans, and tests
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "Beta skills framework: parallel skills with -beta suffix for safe rollouts"
|
||||
category: skill-design
|
||||
date: 2026-03-17
|
||||
module: skills
|
||||
component: SKILL.md
|
||||
tags:
|
||||
- skill-design
|
||||
- beta-testing
|
||||
- skill-versioning
|
||||
- rollout-safety
|
||||
severity: medium
|
||||
description: "Pattern for trialing new skill versions alongside stable ones using a -beta suffix. Covers naming, plan file naming, internal references, and promotion path."
|
||||
related:
|
||||
- docs/solutions/skill-design/compound-refresh-skill-improvements.md
|
||||
- docs/solutions/skill-design/beta-promotion-orchestration-contract.md
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
Core workflow skills like `ce-plan` are deeply chained (`ce-brainstorm` -> `ce-plan` -> `ce-work`) and orchestrated by `lfg`. Rewriting these skills risks breaking the entire workflow for all users simultaneously. There was no mechanism to let users trial new skill versions alongside stable ones.
|
||||
|
||||
Alternatives considered and rejected:
|
||||
- **Beta gate in SKILL.md** with config-driven routing (`beta: true` in `compound-engineering.local.md`): relies on prompt-level conditional routing which risks instruction blending, requires setup integration, and adds complexity to the skill files themselves.
|
||||
- **Pure router SKILL.md** with both versions in `references/`: adds file-read penalty and refactors stable skills unnecessarily.
|
||||
- **Separate beta plugin**: heavy infrastructure for a temporary need.
|
||||
|
||||
## Solution
|
||||
|
||||
### Parallel skills with `-beta` suffix
|
||||
|
||||
Create separate skill directories alongside the stable ones. Each beta skill is a fully independent copy with its own frontmatter, instructions, and internal references.
|
||||
|
||||
```
|
||||
skills/
|
||||
├── ce-plan/SKILL.md # Stable (unchanged)
|
||||
└── ce-plan-beta/SKILL.md # New version
|
||||
```
|
||||
|
||||
### Naming and frontmatter conventions
|
||||
|
||||
- **Directory**: `<skill-name>-beta/`
|
||||
- **Frontmatter name**: `<skill-name>-beta` (e.g., `ce-plan-beta`)
|
||||
- **Description**: Write the intended stable description, then prefix with `[BETA]`. This ensures promotion is a simple prefix removal rather than a rewrite.
|
||||
- **`disable-model-invocation: true`**: Prevents the model from auto-triggering the beta skill. Users invoke it manually with the slash command. Remove this field when promoting to stable.
|
||||
- **Plan files**: Use `-beta-plan.md` suffix (e.g., `2026-03-17-001-feat-auth-flow-beta-plan.md`) to avoid clobbering stable plan files
|
||||
|
||||
### Internal references
|
||||
|
||||
Beta skills must reference other beta skills by their beta names. For example, if both `ce-plan` and `ce-code-review` have beta versions:
|
||||
- `ce-plan-beta` references `ce-code-review-beta` (not `ce-code-review`)
|
||||
- `ce-code-review-beta` references `ce-plan-beta` (not `ce-plan`)
|
||||
|
||||
### What doesn't change
|
||||
|
||||
- Stable skills are completely untouched
|
||||
- `lfg` orchestration continues to use stable skills — no modification needed
|
||||
- `ce-brainstorm` still hands off to stable `ce-plan` — no modification needed
|
||||
- `ce-work` consumes plan files from either version (reads the file, doesn't care which skill wrote it)
|
||||
|
||||
### Tradeoffs
|
||||
|
||||
**Simplicity over seamless integration.** Beta skills exist as standalone, manually-invoked skills. They won't be auto-triggered by `ce-brainstorm` handoffs or `lfg` orchestration without further surgery to those skills, which isn't worth the complexity for a trial period.
|
||||
|
||||
**Intended usage pattern:** A user can run `/ce-plan` for the stable output, then run `/ce-plan-beta` on the same input to compare the two plan documents side by side. The `-beta-plan.md` suffix ensures both outputs coexist in `docs/plans/` without collision.
|
||||
|
||||
## Promotion path
|
||||
|
||||
When the beta version is validated:
|
||||
|
||||
1. Replace stable `SKILL.md` content with beta skill content
|
||||
2. Restore stable frontmatter: remove `[BETA]` prefix from description, restore stable `name:`
|
||||
3. Remove `disable-model-invocation: true` so the model can auto-trigger it
|
||||
4. Update all internal references back to stable names
|
||||
5. Restore stable plan file naming (remove `-beta` from the convention)
|
||||
6. Rename the beta skill directory to the stable name (`ce-X-beta` -> `ce-X`)
|
||||
7. Register the retired `ce-X-beta` name for legacy flat-install cleanup so upgrading users don't keep a stale duplicate dir. This is more than adding the name to `STALE_SKILL_DIRS` — the sweep silently no-ops without the `currentSkillNameForLegacy` case that maps the beta name to the stable one. See [beta-promotion-cleanup-registry-wiring.md](./beta-promotion-cleanup-registry-wiring.md) for the complete four-edit-plus-test wiring.
|
||||
8. Update README.md: remove from Beta Skills section, verify counts
|
||||
9. Verify `lfg` works with the promoted skill
|
||||
10. Verify `ce-work` consumes plans from the promoted skill
|
||||
|
||||
If the beta skill changed its invocation contract, promotion must also update all orchestration callers in the same PR instead of relying on the stable default behavior. See [beta-promotion-orchestration-contract.md](./beta-promotion-orchestration-contract.md) for the concrete review-skill example.
|
||||
|
||||
## Validation
|
||||
|
||||
After creating a beta skill, search its SKILL.md for references to the stable skill name it replaces. Any occurrence of the stable name without `-beta` is a missed rename — it would cause output collisions or route to the wrong skill.
|
||||
|
||||
Check for:
|
||||
- **Output file paths** that use the stable naming convention instead of the `-beta` variant
|
||||
- **Cross-skill references** that point to stable skill names instead of beta counterparts
|
||||
- **User-facing text** (questions, confirmations) that mentions stable paths or names
|
||||
|
||||
## Prevention
|
||||
|
||||
- When adding a beta skill, always use the `-beta` suffix consistently in directory name, frontmatter name, description, plan file naming, and all internal skill-to-skill references
|
||||
- After creating a beta skill, run the validation checks above to catch missed renames in file paths, user-facing text, and cross-skill references
|
||||
- Always test that stable skills are completely unaffected by the beta skill's existence
|
||||
- Keep beta and stable plan file suffixes distinct so outputs can coexist for comparison
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
title: "Reference bundled skill files by tier: relative for reads, SKILL_DIR anchor for executed scripts"
|
||||
date: 2026-06-26
|
||||
category: skill-design
|
||||
module: "skills (bundled-script invocation across harnesses)"
|
||||
problem_type: convention
|
||||
component: tooling
|
||||
severity: high
|
||||
applies_when:
|
||||
- Authoring or reviewing a skill that executes a bundled script via the Bash tool
|
||||
- A skill must work on more than one harness (Claude Code, Codex, Cursor, Gemini)
|
||||
- "Choosing between a bare relative path, ${CLAUDE_SKILL_DIR}, and a model-filled SKILL_DIR anchor"
|
||||
- Generalizing an empirical harness finding into a broad authoring rule
|
||||
tags:
|
||||
- bundled-scripts
|
||||
- path-resolution
|
||||
- skill-authoring
|
||||
- cross-harness
|
||||
- skill-dir
|
||||
- bash-tool
|
||||
- claude-skill-dir
|
||||
- empirical-validation
|
||||
related_components:
|
||||
- development_workflow
|
||||
- documentation
|
||||
---
|
||||
|
||||
# Reference bundled skill files by tier: relative for reads, SKILL_DIR anchor for executed scripts
|
||||
|
||||
## Context
|
||||
|
||||
Skills bundle files the agent uses at runtime: reference docs, schemas, and `scripts/`. Getting paths to those files to resolve correctly across harnesses (Claude Code, Codex, Cursor) has been a recurring bug class — see closed issues #764 (`ce-worktree`), #811 (`ce-code-review`), #898 (`ce-compound`), all "script path resolved against the project root, not the skill dir," plus the still-open #944 ("reconcile contradictory AGENTS.md guidance on bare relative paths") and #949 (a live prose-reference variant).
|
||||
|
||||
The trigger for this learning was a wrong turn. An empirical finding — *the Bash tool's working directory is the user's project root, not the skill directory, on Claude Code, Codex, and Cursor* — was over-generalized into "bare relative paths like `bash scripts/x.sh` are broken, so anchor every invocation." That conclusion was codified and acted on before it was checked against the cross-tool skill spec or any independent implementation. It turned out to be wrong on the strong form: bare relative paths work fine in practice. The correction produced the tiered model below, and a transferable lesson about validating findings before codifying them.
|
||||
|
||||
## Guidance
|
||||
|
||||
Pick the reference form by **what the agent does with the file**, in three tiers.
|
||||
|
||||
**Tier 1 — Read-time file references (the agent *reads* a co-located file into context, e.g. `references/*.md`).** Bare relative path from the skill root, no anchor — the skill loader resolves these against the skill directory on every major harness (the form AGENTS.md Tier 1 codifies). The line vs Tier 2: reading a reference *into context* is Tier 1; the moment the file is used in an *action the agent performs* (copy it, pass it as an argument, execute it) it becomes Tier 2 and takes the cue. Open caveat (#949): if a `Read references/X` is ever observed to resolve against the project CWD and miss on a target, treat that read as Tier 2 and add the "from this skill's directory" cue.
|
||||
|
||||
```
|
||||
Read `references/schema.yaml` and validate frontmatter against it.
|
||||
```
|
||||
|
||||
**Tier 2 — Prose pointers to a bundled file the agent acts on but does *not* execute** (a template to copy, a file to inspect). Bare relative path **plus an explicit "from this skill's directory" cue**, so the agent resolves it against the skill dir rather than the project CWD.
|
||||
|
||||
```
|
||||
Copy `scripts/hook.sh` from this skill's directory into `.claude/hooks/`.
|
||||
```
|
||||
|
||||
**Tier 3 — Executed shell commands** (fenced ```bash``` blocks *or* inline `bash …` / `python …` the agent runs through the Bash tool). Use the **model-filled `SKILL_DIR` anchor**, set inline in the same command (shell state does not persist between separate Bash-tool calls):
|
||||
|
||||
```bash
|
||||
SKILL_DIR="<absolute path of the directory containing the SKILL.md you just read>"
|
||||
bash "$SKILL_DIR/scripts/measure.sh" "$ARG"
|
||||
```
|
||||
|
||||
This is the conservative **house default** for executed shell — not because bare relative "fails," but because it bakes resolution into the command so a fenced block copied verbatim into a Bash call cannot miss, regardless of harness or model version.
|
||||
|
||||
Two adjacent rules:
|
||||
|
||||
- **Avoid `${CLAUDE_SKILL_DIR}`.** It is a Claude-Code-only SKILL.md *content* substitution (not an env var) and is empty on every other host. A `${CLAUDE_SKILL_DIR}`-guarded call's `then` branch then silently never fires off-Claude — a genuine silent skip. This plugin ships as a *native* Codex plugin (the converter is not in that path), so a Claude-only mechanism is a footgun, not a neutral fallback. The model-filled `SKILL_DIR` anchor works on every host with no downside. (Narrow exception: behavior that is genuinely Claude-only and will never run elsewhere — essentially never, given the cross-host install model.)
|
||||
- **A script that needs its *own* directory** (to open a sibling file from inside the process) derives it from `BASH_SOURCE`, not `SKILL_DIR` — `SKILL_DIR` is the orchestrator's shell variable and is not exported to the child process.
|
||||
|
||||
### Where this is codified
|
||||
|
||||
`AGENTS.md` > "Platform-Specific Variables in Skills" codifies this three-tier model as the repo's authoring rule; this learning is the rationale and worked examples behind it. A few legacy `${CLAUDE_SKILL_DIR}` uses are still being migrated to the anchor (e.g. `ce-compound`'s `validate-frontmatter.py`), tracked by #944. Note `tests/skill-conventions.test.ts` enforces an existence guard only when a skill-dir *platform var* is used — it does not forbid the model-filled `SKILL_DIR` anchor (which uses no platform var), so anchor-based skills pass it.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
There are two payoffs, and the second is the larger one.
|
||||
|
||||
First-order: correctness. A bundled script that silently no-ops on a non-Claude host, or resolves the wrong path, fails quietly in a user's environment and is hard to catch in review. The tier rules remove that class of failure while keeping the simple cases simple.
|
||||
|
||||
Second-order — the meta-lesson: **a single empirical finding is not an authoring rule until it is validated against the spec and independent implementations.** "The shell's CWD is the project root" is a true, narrow *harness fact*. It was inflated into "bare relative is broken" without checking the actual *resolution mechanism* — which is the agent, not the shell. The agentskills.io spec says it directly: relative script paths work because "the agent resolves these paths automatically." Conflating "where the shell starts" with "who resolves the path" produced guidance that diverged from the ecosystem and added unnecessary machinery. Distinguish the harness fact from the resolution mechanism, and confirm any broad rule against the cross-tool spec plus two or three real skills before codifying it. Vendor docs alone do not settle a cross-harness question — they are platform-centric by construction (e.g., Anthropic's Claude Code docs recommend `${CLAUDE_SKILL_DIR}`, which is exactly the non-portable form to avoid here).
|
||||
|
||||
The evidence that corrected the over-generalization, for the record: the agentskills.io spec ships bare `bash scripts/validate.sh` as its canonical example; obra/superpowers' `brainstorming` skill runs bare `scripts/start-server.sh` with no anchor across four named platforms; its `subagent-driven-development` skill pairs a bare relative path with a "from this skill's directory" cue (Tier 2); mattpocock/skills sidesteps in-place execution entirely (copies a hook to `.claude/hooks/`, or ships a `.template.sh` referenced in prose); and `last30days` adopts the explicit `SKILL_DIR` anchor for its critical multi-host engine — *after* a path-resolution regression. The tiers reconcile all of these: relative is the ecosystem norm and works via agent resolution; the anchor is the determinism upgrade reserved for executed shell.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Whenever a SKILL.md or a reference file tells the agent to run a bundled script through the Bash tool (Tier 3).
|
||||
- When adding files under a skill's `scripts/` that are *executed* rather than *read*.
|
||||
- When reviewing or migrating a skill that uses `${CLAUDE_SKILL_DIR}` guards — check whether they silently no-op off-Claude and move executed-shell calls to the anchor.
|
||||
- When weighing "inline logic" vs. "bundled script" for portability — the tiers remove the old fear that bundled scripts are inherently fragile cross-harness; just give executed scripts Tier-3 treatment.
|
||||
- Do **not** apply the anchor to Tier 1 or Tier 2 references — it adds noise without benefit there.
|
||||
|
||||
## Examples
|
||||
|
||||
Tier 3 — executed script, before and after:
|
||||
|
||||
```bash
|
||||
# Before: bare relative in a fenced block. Works via agent resolution, but a
|
||||
# verbatim-copied block resolves against the project root and misses.
|
||||
bash scripts/measure.sh "$TARGET"
|
||||
```
|
||||
|
||||
```bash
|
||||
# After: model-filled anchor, deterministic regardless of harness/model.
|
||||
SKILL_DIR="<absolute path of the directory containing the SKILL.md you just read>"
|
||||
bash "$SKILL_DIR/scripts/measure.sh" "$TARGET"
|
||||
```
|
||||
|
||||
Anti-pattern — the `${CLAUDE_SKILL_DIR}` existence guard (silently no-ops on Codex/Cursor):
|
||||
|
||||
```bash
|
||||
# AVOID — `then` branch never fires off-Claude, where ${CLAUDE_SKILL_DIR} is unset.
|
||||
if [ -n "${CLAUDE_SKILL_DIR}" ] && [ -f "${CLAUDE_SKILL_DIR}/scripts/x.sh" ]; then
|
||||
bash "${CLAUDE_SKILL_DIR}/scripts/x.sh"
|
||||
else
|
||||
echo "script unavailable"
|
||||
fi
|
||||
```
|
||||
|
||||
Tier 1 — read-time reference, already correct, no anchor needed:
|
||||
|
||||
```
|
||||
Read `references/schema.yaml` and apply its field definitions.
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [`script-first-skill-architecture.md`](script-first-skill-architecture.md) — *whether* to offload work to a bundled script (token cost). This doc is the companion *how to invoke it* once you have. Low overlap, complementary.
|
||||
- [`pass-paths-not-content-to-subagents.md`](pass-paths-not-content-to-subagents.md) — path-passing for orchestrator->subagent dispatch; a different "paths" problem (token efficiency, not CWD resolution).
|
||||
- [`../best-practices/prefer-python-over-bash-for-pipeline-scripts.md`](../best-practices/prefer-python-over-bash-for-pipeline-scripts.md) — which language to write a bundled script in.
|
||||
- `AGENTS.md` > "Platform-Specific Variables in Skills" — codifies this three-tier model as the repo's authoring rule; this doc is its rationale and worked examples. Its Tier 1 (read-time references) and Tier 2 (prose pointers + cue) address the prose-reference bug class in #949.
|
||||
- Issues: #944 (open — audit/reconcile bundled-script invocation guidance; this learning informs it), #949 (open — live Tier-2 prose-reference miss), #943/#898/#811/#764 (closed — the Tier-3 origin bug class).
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "ce-doc-review calibration patterns: tier classification, chain grouping, and FYI routing"
|
||||
date: 2026-04-19
|
||||
category: skill-design
|
||||
module: compound-engineering / ce-doc-review
|
||||
problem_type: design_pattern
|
||||
component: tooling
|
||||
severity: medium
|
||||
tags:
|
||||
- ce-doc-review
|
||||
- autofix-classification
|
||||
- synthesis-pipeline
|
||||
- persona-calibration
|
||||
- premise-dependency
|
||||
- fyi-routing
|
||||
- calibration
|
||||
applies_when:
|
||||
- Changing persona confidence calibration in the doc-review skill-local personas under `skills/ce-doc-review/references/personas/`
|
||||
- Modifying the synthesis pipeline in `skills/ce-doc-review/references/synthesis-and-presentation.md`
|
||||
- Adjusting the subagent template's output contract in `references/subagent-template.md`
|
||||
- Adding or modifying seeded test fixtures under `tests/fixtures/ce-doc-review/`
|
||||
- Debugging why a finding landed in a different tier than expected
|
||||
---
|
||||
|
||||
# ce-doc-review calibration patterns
|
||||
|
||||
Calibration work on ce-doc-review (PR #601 series, 2026-04-18 and -19) surfaced several non-obvious patterns in how the synthesis pipeline classifies findings. These patterns are durable: they will re-surface any time personas or synthesis guidance are retuned. Future contributors changing calibration should expect them and not "fix" them as bugs.
|
||||
|
||||
## Tier classification is context-sensitive, not purely formal
|
||||
|
||||
The naive read of the tier spec says `safe_auto` = "one clear correct fix, applied silently." In practice, the same shape of finding can legitimately land in different tiers depending on scope and verifiability. Two recurring patterns:
|
||||
|
||||
### External stale cross-reference → gated_auto (not safe_auto)
|
||||
|
||||
When the document says `see Unit 7` and Unit 7 doesn't exist in the same document, that's an **internal** stale cross-reference — coherence can verify from the document text alone and apply `safe_auto`. When the document says `see docs/guides/keyboard-nav.md Section 4` and that file isn't verifiable from the document content, that's an **external** cross-reference; applying "delete this reference" silently risks masking a legitimate external doc. The reviewer should route these to `gated_auto` with a "verify before applying" fix, not `safe_auto`.
|
||||
|
||||
Observed in: feature-plan fixture runs. The external cross-ref landed at P2 as `gated_auto` with the fix "Verify docs/guides/keyboard-nav.md exists... If stale, either remove the reference or replace with inline guidance."
|
||||
|
||||
### Multi-surface terminology drift → gated_auto (not safe_auto)
|
||||
|
||||
When two synonyms appear in prose only (`data store` / `database`), `safe_auto` normalizes correctly. When the drift crosses surfaces — UI copy, aria-labels, toast messages, analytics events, file names, code identifiers — the fix's scope exceeds prose normalization and warrants user confirmation. Security-adjacent terminology (`token` / `credential` / `secret` / `API key`) carries different semantic weight and should also route to `gated_auto` with a glossary-fix recommendation.
|
||||
|
||||
Observed in: auth-plan fixture runs (security-lens escalated), feature-plan fixture runs (UI-surface escalated).
|
||||
|
||||
**Do not tighten coherence's `safe_auto` guidance to force these into `safe_auto`.** The reclassification is reviewer judgment doing useful work.
|
||||
|
||||
## Premise-dependency chains have scope hierarchy
|
||||
|
||||
Synthesis step 3.5c groups manual findings whose fixes cascade from a single premise challenge. When multiple premise-level candidates surface, they may be **peer roots** (independent premises at different scopes) or **nested** (one premise's resolution moots the other). The decision rules:
|
||||
|
||||
### Peer vs nested — mechanical test, not example-based
|
||||
|
||||
> "Two candidate roots are peers when accepting root A's proposed fix would not resolve root B's concern (and vice versa). They are nested when one root's fix would moot the other — in which case the subsumed candidate becomes a dependent of the surviving root."
|
||||
|
||||
Apply symmetrically: check both directions before deciding. Example-based teaching ("e.g., 'drop the alias'") overfits to specific vocabulary; a mechanical decision test generalizes across domains.
|
||||
|
||||
### Surviving root under nested — scope dominates confidence
|
||||
|
||||
When nested, the surviving root is the one whose fix moots the other — **not** the higher-confidence candidate. In a rename plan, the broader-scope "rename premise unsupported" root dominates the higher-confidence "alias machinery unjustified" candidate, because rejecting the rename moots the alias entirely, while rejecting the alias still leaves the rename standing. Earlier synthesis picked the higher-confidence candidate as root, which stranded the broader-scope premise's natural dependents as independent findings.
|
||||
|
||||
Confidence is for tie-breaking *among peers*, not for deciding which of two nested candidates dominates.
|
||||
|
||||
### Multi-root requires explicit elevation
|
||||
|
||||
Synthesis defaults to picking a single root when multiple candidates match. A phrase like "typically 0–2 roots surface per review" anchors the synthesizer to elevate only one. Explicit guidance to elevate ALL matching candidates (subject only to the peer-vs-nested test) is needed. The criteria themselves are the filter — no numerical cap on roots.
|
||||
|
||||
## FYI routing requires band + template-level anchoring
|
||||
|
||||
Advisory observations with no articulable consequence need somewhere to land, or they get either promoted above the gate (appearing as real decisions) or suppressed entirely. The FYI bucket gives them a home, but it stays empty unless two changes are made together:
|
||||
|
||||
1. **Per-persona advisory band** tailored to each persona's scope. Each of the 7 skill-local personas needs its own band; a single template-level rule doesn't override persona-specific calibrations.
|
||||
2. **Template-level advisory rule** in `subagent-template.md`'s output-contract using the "what actually breaks if we don't fix this?" heuristic. Anchors the scoring decision when a persona's own rubric doesn't make the band's applicability obvious.
|
||||
|
||||
Either alone is insufficient. Persona bands without the template rule produce inconsistent results across personas; the template rule without per-persona bands has nothing to calibrate against.
|
||||
|
||||
> **Scoring model note:** This pattern predates the anchored-rubric migration. The original calibration used continuous float bands; scoring is now an anchored rubric (discrete `0/25/50/75/100`, with FYI = anchor `50`). See [confidence-anchored-scoring.md](./confidence-anchored-scoring.md) for the canonical scoring model. The band-plus-template structural insight above is independent of the numeric scale.
|
||||
|
||||
## Schema compliance requires inline enum callouts, not just `{schema}` injection
|
||||
|
||||
The subagent template injects the full JSON schema into each persona's prompt. Schema conformance nonetheless broke on longer personas (adversarial at 89 lines, scope-guardian at 54 lines) — severity emitted as `"high"/"medium"/"low"` instead of `P0/P1/P2/P3`, evidence as strings instead of arrays.
|
||||
|
||||
The fix that worked: a **"Schema conformance — hard constraints"** block at the top of the output contract prose, naming the exact enum values and forbidding common deviations. Schema injection alone gets pushed down in attention by dense persona rubrics; inline enum callouts anchor them at the top of the output contract and survive longer prompts.
|
||||
|
||||
A severity translation rule ("if your persona's prose discusses 'critical/important/low-signal', map to P0/P1/P2/P3 at emit time") prevents informal priority language in persona rubrics from leaking into JSON output.
|
||||
|
||||
## Coverage/rendering count invariants need a single source of truth
|
||||
|
||||
Early chain runs reported coverage count (`1 root with 6 dependents`) that didn't match the rendered output (5 dependents shown). The spec didn't name which step's count was authoritative (candidate count from Step 2, post-safeguard from Step 3, or post-cap from Step 4), so the orchestrator used different values for coverage and rendering.
|
||||
|
||||
**Invariant to preserve:** the `dependents` array populated in the final annotation step (after all filtering) is the single source of truth for both coverage and rendering. A finding appearing in a root's `dependents` array must appear nested under that root in presentation and must NOT appear at its own severity position. Coverage count equals the length of the `dependents` array.
|
||||
|
||||
Any future pipeline change that adds filtering or reorganization steps must re-state which post-step snapshot is authoritative.
|
||||
|
||||
## Reviewer variance is inherent; single runs aren't baselines
|
||||
|
||||
Across 7+ runs on the rename fixture, the same document produced user-engagement counts of 0, 1, 2, 3 for `safe_auto` applied and 14, 19, 6, 12, 8, 6 for total user decisions. Calibration work reduced but did not eliminate variance. Primary variance sources:
|
||||
|
||||
- **Adversarial reviewer activation** — the activation signals (requirement count, architectural decisions, high-stakes domain) produce non-deterministic decisions at borderline documents
|
||||
- **Root selection when multiple candidates exist** — even with scope-dominance guidance, the synthesizer's root choice varies across runs
|
||||
- **Confidence calibration on borderline findings** — the same finding lands in FYI on one run and manual on the next, because the reviewer's anchor choice flips at the boundary across runs
|
||||
|
||||
**Testing implication:** validate calibration changes against multiple runs, not single samples. A single "bad" run is likely noise; a pattern across 3+ runs is signal. Seeded fixtures document expected tier distributions as targets, not as pass/fail assertions.
|
||||
|
||||
## Related documentation
|
||||
|
||||
- `skills/ce-doc-review/references/synthesis-and-presentation.md` — canonical synthesis pipeline spec, including 3.5c premise-dependency chain linking
|
||||
- `skills/ce-doc-review/references/subagent-template.md` — output contract with schema conformance block and advisory routing rule
|
||||
- `skills/ce-doc-review/references/personas/` — the 7 doc-review persona prompts (`coherence-reviewer.md`, `feasibility-reviewer.md`, `design-lens-reviewer.md`, `security-lens-reviewer.md`, `scope-guardian-reviewer.md`, `product-lens-reviewer.md`, `adversarial-document-reviewer.md`) with their confidence calibration bands
|
||||
- `tests/fixtures/ce-doc-review/` — three seeded fixtures (rename, auth, feature) for manual calibration testing; see each fixture's header comment for its specific seed map
|
||||
- `docs/solutions/developer-experience/branch-based-plugin-install-and-testing.md` — how to run the skill from a branch checkout for testing
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Public skills must use the ce- prefix; enforce it in tests, not just prose
|
||||
date: 2026-05-01
|
||||
last_refreshed: 2026-06-20
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: convention
|
||||
component: root plugin
|
||||
severity: low
|
||||
applies_when:
|
||||
- Adding a new skill directory under `skills/`
|
||||
- Authoring or reviewing a PR that introduces a new public plugin component
|
||||
- Deciding whether a new specialist prompt should be a public skill or a skill-local reference asset
|
||||
tags:
|
||||
- naming-convention
|
||||
- ce-prefix
|
||||
- skill-authoring
|
||||
- test-enforcement
|
||||
- plugin-conventions
|
||||
related:
|
||||
- docs/solutions/skill-design/beta-skills-framework.md
|
||||
related_pr: https://github.com/EveryInc/compound-engineering-plugin/pull/747
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
`AGENTS.md` stated that public Compound Engineering components use the `ce-` prefix to make ownership clear. But the rule was prose-only, and legacy skills sat unprefixed in the same directory as their `ce-`-prefixed siblings. The combination — a soft rule plus visible exceptions — let a new skill (`riffrec-feedback-analysis`) ship in PR #747 without the prefix. The user caught it post-merge of the first commit, requiring a rename commit on the same PR.
|
||||
|
||||
That skill is now `ce-riffrec-feedback-analysis`; `lfg` remains the sole intentional public-skill exemption.
|
||||
|
||||
The repo no longer ships standalone CE agents. Specialist reviewer, researcher, and helper behavior lives as skill-local prompt assets under `skills/*/references/agents/` or `skills/*/references/personas/`. Those internal filenames should be descriptive for the owning skill, not treated as public plugin component names.
|
||||
|
||||
## Root cause
|
||||
|
||||
Two layered problems:
|
||||
|
||||
1. **The rule was unenforced.** Nothing in CI or the test suite failed when a non-`ce-` public skill was added.
|
||||
2. **The exception list was implicit.** Legacy skills predated the rule. Without an explicit allowlist, "predates the rule" looked identical to "the rule does not apply" when reading the filesystem.
|
||||
|
||||
## Solution
|
||||
|
||||
Make the public-skill rule mechanically enforced and pin exceptions explicitly.
|
||||
|
||||
### 1. Test enforcement
|
||||
|
||||
Enforcement lives in `tests/frontmatter.test.ts`, which walks root `skills/` directories and asserts the prefix on the directory name and frontmatter `name`. Exemptions are explicit:
|
||||
|
||||
```ts
|
||||
const SKILL_PREFIX_ALLOWLIST = new Set([
|
||||
// lfg ships as the public command `/lfg` (see README.md).
|
||||
"lfg",
|
||||
])
|
||||
```
|
||||
|
||||
The test also verifies each skill frontmatter `name` matches its parent directory and uses only lowercase letters, numbers, and hyphens. That protects Pi and other native plugin loaders that reject punctuation-heavy names.
|
||||
|
||||
### 2. Strengthened prose
|
||||
|
||||
`AGENTS.md` documents the naming rule and points authors at the test. Prose alone would not have prevented the original mistake, but pairing it with a CI check gives a single internally consistent story.
|
||||
|
||||
### 3. Internal prompt assets stay internal
|
||||
|
||||
Do not use this rule as a reason to prefix every internal persona file. After the agentless restructure, names like `learnings-researcher.md` and `coherence-reviewer.md` are intentionally scoped by their owning skill directory. The public namespace is the skill name; the internal filename is just a prompt asset.
|
||||
|
||||
## Prevention
|
||||
|
||||
For any plugin convention that is prose-only, ask:
|
||||
|
||||
- Is there at least one visible counterexample in the codebase that an author could mistake for permission?
|
||||
- Is there a mechanical check that would fail on violation?
|
||||
|
||||
If the answer to the first is yes and the second is no, the convention will eventually be violated. Add a test with a hard-coded allowlist, or migrate the legacy exceptions so the rule is universal.
|
||||
|
||||
## Related
|
||||
|
||||
- `AGENTS.md` — Naming Convention section documents the rule and allowlist.
|
||||
- `tests/frontmatter.test.ts` — public-skill prefix enforcement.
|
||||
- PR #747 — the original mistake and the rename + enforcement that came with it.
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
title: "ce-compound-refresh skill redesign for autonomous maintenance without live user context"
|
||||
category: skill-design
|
||||
date: 2026-03-13
|
||||
module: skills/ce-compound-refresh
|
||||
component: SKILL.md
|
||||
tags:
|
||||
- skill-design
|
||||
- compound-refresh
|
||||
- maintenance-workflow
|
||||
- drift-classification
|
||||
- subagent-architecture
|
||||
- platform-agnostic
|
||||
severity: medium
|
||||
description: "Redesign ce-compound-refresh to handle autonomous drift triage, in-skill replacement via subagents, and smart scoping without relying on live problem-solving context that ce-compound expects."
|
||||
related:
|
||||
- docs/solutions/plugin-versioning-requirements.md
|
||||
- https://github.com/EveryInc/compound-engineering-plugin/pull/260
|
||||
- https://github.com/EveryInc/compound-engineering-plugin/issues/204
|
||||
- https://github.com/EveryInc/compound-engineering-plugin/issues/221
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
The initial `ce-compound-refresh` skill had several design issues discovered during real-world testing:
|
||||
|
||||
1. Interactive questions never triggered the proper tool (AskUserQuestion) because the instruction used a weak "when available" qualifier
|
||||
2. Auto-delete criteria contradicted a "always ask before deleting" rule in a later phase
|
||||
3. Broad scope (9+ docs) asked the user to choose an area blindly without providing analysis
|
||||
4. The Replace flow tried to hand off to `ce-compound`, which expects fresh problem-solving context the user doesn't have months later
|
||||
5. Subagents used shell commands for file existence checks, triggering permission prompts
|
||||
6. No way to run the skill unattended (e.g., on a schedule) — every run required user interaction
|
||||
|
||||
## Root Cause
|
||||
|
||||
Five independent design issues, each with a distinct root cause:
|
||||
|
||||
1. **Hardcoded tool name with escape hatch.** Saying "Use AskUserQuestion when available" gave the model permission to skip the tool and just output text. Also non-portable to Codex and other platforms.
|
||||
2. **Contradictory rules across phases.** Phase 2 defined auto-delete criteria. Phase 3 said "always ask before deleting" with no exception. The model followed Phase 3.
|
||||
3. **Question before evidence.** The skill prompted scope selection before gathering any information about which areas were most stale or interconnected.
|
||||
4. **Unsatisfied precondition in cross-skill handoff.** `ce-compound` expects a recently solved problem with fresh context. A maintenance refresh has investigation evidence instead — equivalent data, different shape.
|
||||
5. **No tool preference guidance for subagents.** Without explicit instruction, subagents defaulted to bash for file operations.
|
||||
6. **Interactive-only design.** Every phase assumed a user was present. No way to run autonomously for scheduled maintenance or hands-off sweeps.
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. Platform-agnostic interactive questions
|
||||
|
||||
Reference "the platform's interactive question tool" as the concept, with concrete examples:
|
||||
|
||||
```markdown
|
||||
Ask questions **one at a time** — use the platform's interactive question tool
|
||||
(e.g. `AskUserQuestion` in Claude Code, `request_user_input` in Codex) and
|
||||
**stop to wait for the answer** before continuing.
|
||||
```
|
||||
|
||||
The "stop to wait" language removes the escape hatch. The examples help each platform's model select the right tool.
|
||||
|
||||
### 2. Auto-delete exemption for unambiguous cases
|
||||
|
||||
Phase 3 now defers to Phase 2's auto-delete criteria:
|
||||
|
||||
```markdown
|
||||
You are about to Delete a document **and** the evidence is not unambiguous
|
||||
(see auto-delete criteria in Phase 2). When auto-delete criteria are met,
|
||||
proceed without asking.
|
||||
```
|
||||
|
||||
### 3. Smart triage for broad scope
|
||||
|
||||
When 9+ candidate docs are found, triage before asking:
|
||||
|
||||
1. **Inventory** — read frontmatter, group by module/component/category
|
||||
2. **Impact clustering** — dense clusters of interconnected learnings + pattern docs are higher-impact than isolated docs
|
||||
3. **Spot-check drift** — check whether primary referenced files still exist
|
||||
4. **Recommend** — present the highest-impact cluster with rationale
|
||||
|
||||
Key insight: "code changed recently" is NOT a reliable staleness signal. Missing references in a high-impact cluster is the strongest signal.
|
||||
|
||||
### 4. Replacement subagents instead of ce-compound handoff
|
||||
|
||||
By the time a Replace is identified, Phase 1 investigation has already gathered the evidence that `ce-compound` would research:
|
||||
- The old learning's claims
|
||||
- What the current code actually does
|
||||
- Where and why the drift occurred
|
||||
|
||||
A replacement subagent writes the successor directly using `ce-compound`'s document format (frontmatter, problem, root cause, solution, prevention). Run sequentially — one at a time — because each may read significant code.
|
||||
|
||||
When evidence is insufficient (e.g., entire subsystem replaced, new architecture too complex to understand from investigation alone), mark as stale and recommend `ce-compound` after the user's next encounter with that area.
|
||||
|
||||
### 5. Dedicated file tools over shell commands
|
||||
|
||||
Added to subagent strategy:
|
||||
|
||||
```markdown
|
||||
Subagents should use dedicated file search and read tools for investigation —
|
||||
not shell commands. This avoids unnecessary permission prompts and is more
|
||||
reliable across platforms.
|
||||
```
|
||||
|
||||
### 6. Headless mode for scheduled/unattended runs
|
||||
|
||||
Added `mode:headless` argument support so the skill can run without user interaction (e.g., on a schedule, in CI, or when the user just wants a hands-off sweep).
|
||||
|
||||
Key design decisions:
|
||||
- **Explicit opt-in only.** `mode:headless` must be in the arguments. Auto-detection based on tool availability was rejected because a user in an interactive agent without a question tool (e.g., Cursor, Windsurf) is still interactive — they just use plain-text replies.
|
||||
- **Conservative confidence.** Borderline cases that would get a user question in interactive mode get marked stale in headless mode. Err toward stale-marking over incorrect action.
|
||||
- **Detailed report as deliverable.** Since no user was present, the output report includes full rationale for each action so a human can review after the fact.
|
||||
- **Process everything.** No scope narrowing questions — if no scope hint provided, process all docs. For broad scope, process clusters in impact order without asking.
|
||||
|
||||
## Prevention
|
||||
|
||||
### Skill review checklist additions
|
||||
|
||||
These five patterns should be checked during any skill review:
|
||||
|
||||
1. **No hardcoded tool names** — All tool references use capability-first language with platform examples and a plain-text fallback
|
||||
2. **No contradictory rules across phases** — Trace each action type through all phases; verify absolute language ("always," "never") is not contradicted elsewhere
|
||||
3. **No blind user questions** — Every question presented to the user is informed by evidence the agent gathered first
|
||||
4. **No unsatisfied cross-skill preconditions** — Every skill handoff verifies the target skill's preconditions are met by the calling context
|
||||
5. **No shell commands for file operations in subagents** — Subagent instructions explicitly prefer dedicated tools over shell commands
|
||||
6. **Headless mode for long-running skills** — Any skill that could run unattended should support an explicit opt-in mode with conservative confidence and detailed reporting
|
||||
|
||||
### Key anti-patterns
|
||||
|
||||
| Anti-pattern | Better pattern |
|
||||
|---|---|
|
||||
| "Use the AskUserQuestion tool when available" | "Use the platform's interactive question tool (e.g. AskUserQuestion in Claude Code, request_user_input in Codex)" |
|
||||
| Defining auto-delete conditions, then "always ask before deleting" | Single-source-of-truth: define the rule once, reference it elsewhere |
|
||||
| "Which area should we review?" before any investigation | Triage first, recommend with evidence, let user confirm or redirect |
|
||||
| "Create a successor learning through ce-compound" during a refresh | Replacement subagent writes directly using gathered evidence |
|
||||
| No tool guidance for subagents | "Use dedicated file search and read tools, not shell commands" |
|
||||
| Auto-detecting "no question tool = headless" | Explicit `mode:headless` argument — interactive agents without question tools are still interactive |
|
||||
|
||||
## Cross-References
|
||||
|
||||
- **PR #260**: The PR containing all these improvements
|
||||
- **Issue #204**: Platform-agnostic tool references (AskUserQuestion dependency)
|
||||
- **Issue #221**: Motivating issue for maintenance at scale
|
||||
- **PR #242**: ce-audit (detection counterpart, closed)
|
||||
- **PR #150**: Established subagent context-isolation pattern
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
title: "ce-doc-review confidence scoring: anchored rubric over continuous floats"
|
||||
date: 2026-04-21
|
||||
category: skill-design
|
||||
module: compound-engineering / ce-doc-review
|
||||
problem_type: design_pattern
|
||||
component: tooling
|
||||
severity: medium
|
||||
tags:
|
||||
- ce-doc-review
|
||||
- scoring
|
||||
- calibration
|
||||
- personas
|
||||
- persona-rubric
|
||||
---
|
||||
|
||||
# ce-doc-review confidence scoring: anchored rubric over continuous floats
|
||||
|
||||
## Problem
|
||||
|
||||
Persona-based document review originally used a continuous `confidence` field (0.0 to 1.0) that synthesis compared against per-severity numeric gates (0.50 / 0.60 / 0.65 / 0.75) and a 0.40 FYI floor. In practice the continuous scale invited false precision: personas clustered on round values (0.60, 0.65, 0.72, 0.80, 0.85), and gate boundaries created coin-flip bands where trivial score shifts moved findings in and out of the actionable tier. The personas were not genuinely differentiating 0.65 from 0.72; the model cannot calibrate self-reported confidence at that granularity.
|
||||
|
||||
Symptoms surfaced in review output:
|
||||
|
||||
- Single personas filing 3+ findings all rated 0.68-0.72, all variants of the same root premise
|
||||
- Findings at 0.65 admitted into the actionable tier on noise, not signal
|
||||
- Residual concerns and deferred questions near-duplicated findings already surfaced, indicating the persona's own ordering did not distinguish "raise this" from "note this"
|
||||
|
||||
## Reference pattern: Anthropic's anchored rubric
|
||||
|
||||
Anthropic's official code-review plugin (`anthropics/claude-plugins-official/plugins/code-review/commands/code-review.md`) solves the calibration problem with 5 discrete anchors (`0`, `25`, `50`, `75`, `100`) each tied to a behavioral criterion the model can honestly self-apply:
|
||||
|
||||
- `0` — false positive or pre-existing issue
|
||||
- `25` — might be real but couldn't verify; stylistic-not-in-CLAUDE.md
|
||||
- `50` — verified real but nitpick / not very important
|
||||
- `75` — double-checked, will hit in practice, directly impacts functionality
|
||||
- `100` — confirmed, evidence directly confirms, will happen frequently
|
||||
|
||||
The rubric is passed verbatim to a separate scoring agent. Filter threshold: `>= 80`.
|
||||
|
||||
## Solution adopted for ce-doc-review
|
||||
|
||||
Port the structural techniques — anchored rubric, verbatim persona-facing text, explicit false-positive catalog — and tune the filter threshold for document-review economics. The doc-review threshold is `>= 50`, not Anthropic's `>= 80`.
|
||||
|
||||
### Anchor-to-route mapping
|
||||
|
||||
| Anchor | Route |
|
||||
|--------|-------|
|
||||
| `0`, `25` | Dropped silently (counted in Coverage only) |
|
||||
| `50` | FYI subsection (surface-only, no forced decision) |
|
||||
| `75`, `100` | Actionable tier, classified by `autofix_class` |
|
||||
|
||||
Cross-persona corroboration promotes one anchor step (`50 → 75`, `75 → 100`, `100 → 100`). This replaces the prior `+0.10` numeric boost.
|
||||
|
||||
Within-severity sort: anchor descending, then document order as the deterministic final tiebreak.
|
||||
|
||||
### Files
|
||||
|
||||
- `skills/ce-doc-review/references/findings-schema.json` — `confidence` is an integer enum `[0, 25, 50, 75, 100]` with behavioral definitions embedded in the `description` field
|
||||
- `skills/ce-doc-review/references/subagent-template.md` — the rubric section personas see verbatim, plus the consolidated false-positive catalog
|
||||
- `skills/ce-doc-review/references/synthesis-and-presentation.md` — anchor-based gate in 3.2, anchor-step promotion in 3.4, anchor-sorted ordering in 3.8, anchor+autofix routing in 3.7
|
||||
- `skills/ce-doc-review/references/personas/*.md` — the 7 doc-review persona prompts; each carries a persona-specific calibration section that maps domain criteria to the shared anchors
|
||||
- `tests/pipeline-review-contract.test.ts` — contract tests that assert the schema enforces discrete anchors and the template embeds the rubric
|
||||
|
||||
## Why the threshold diverges from Anthropic
|
||||
|
||||
Code review and document review have different economics. Anthropic's `>= 80` filter is load-bearing for code review because of three constraints that do not apply to doc review:
|
||||
|
||||
1. **Code review has a linter backstop.** CI runs linters, typecheckers, and tests. The LLM reviewer is a second layer on top of automated tooling, and a second layer only adds value by being *more selective*. If automation already catches the 50-75 tier, the LLM surfacing it again is noise.
|
||||
2. **Code review is high-frequency and publicly visible.** Every surfaced finding becomes a PR comment. A reviewer who cries wolf 5 times gets muted. Precision dominates recall.
|
||||
3. **Code claims are ground-truth verifiable.** "The code does X" can be proven or refuted by reading it. A 75 in code review often means "I couldn't verify" — which means waiting for someone who can.
|
||||
|
||||
Document review inverts all three:
|
||||
|
||||
1. **Doc review IS the backstop.** There is no linter that catches a plan's premise gaps or scope drift. A missed finding in the plan derails implementation weeks later.
|
||||
2. **Doc review is low-frequency and private.** One review per plan, not per PR. Surfaced findings are dismissed with a keystroke via the routing menu; they are not public commentary.
|
||||
3. **Premise claims have a natural confidence ceiling.** "Is the motivation valid?" and "does this scope match the goal?" cannot be verified against ground truth. Personas working in strategy, premise, and adversarial domains (product-lens, adversarial) legitimately cap at anchors 50-75 because full verification is not possible from document text alone. A `>= 80` filter would silence those personas.
|
||||
|
||||
Filter at `>= 50` for doc review; let the routing menu handle volume. Dismissing a surfaced finding is cheap; missing a real concern is expensive.
|
||||
|
||||
## When to port this pattern
|
||||
|
||||
- Other persona-based review skills with similar economics (no linter backstop, one-shot consumption, dismissal cheap via routing). Default threshold for such skills: `>= 50`.
|
||||
- Any scoring workflow where the model is asked to self-report confidence on a continuous scale and clustering on round numbers is observed.
|
||||
|
||||
## When NOT to port directly
|
||||
|
||||
- Code review workflows have linter backstops and public-comment costs. Port the rubric structure, but tune the threshold higher (`>= 75`). See the "ce-code-review migration" section below for the completed port.
|
||||
- High-throughput pipelines where the `25` anchor ("couldn't verify") represents most findings. Dropping everything below `50` may be too aggressive; consider surfacing `25` as "needs human triage" instead.
|
||||
|
||||
## Migration history
|
||||
|
||||
Landed in a single atomic change because the schema, template, synthesis, rendering, personas, and tests are coupled — a partial migration would have failed validation at every boundary. The schema change is the load-bearing commit; the persona updates and test updates consume it.
|
||||
|
||||
## Evaluation
|
||||
|
||||
After the migration, an A/B evaluation compared baseline (continuous float) against treatment (anchored integer rubric) across four documents spanning size and type: a 7KB in-repo plan, a 63KB in-repo plan, a 27KB external-repo plan, and a 10KB in-repo brainstorm. Both versions were executed by orchestrator subagents reading their matching skill snapshot as prompt material, dispatching all 7 personas, and emitting the Phase 4 headless envelope. The workspace, per-run envelopes, and timing data live under `.context/compound-engineering/ce-doc-review-eval/` during the evaluation.
|
||||
|
||||
### Confirmed effects
|
||||
|
||||
- **Score dispersion collapsed.** Baseline produced 7-12 distinct float values per document (typical: 0.45, 0.50, 0.55, 0.65, 0.72, 0.80, 0.85) — the exact false-precision clustering the migration targeted. Treatment concentrated on 2-3 anchors per document. Anchors `0` and `25` were never emitted by any persona, which matches the template's "suppress silently" instruction for those tiers.
|
||||
- **Cross-persona +1 anchor promotion fires as specified.** Observed on cli-printing-press plan (security-lens + feasibility promoting an IP-range-check finding to anchor 100) and interactive-judgment plan (product-lens + adversarial promoting a premise finding to anchor 100).
|
||||
- **Chain linking, safe_auto silent-apply, FYI routing, and per-persona redundancy collapse** all exercised correctly on at least one run.
|
||||
- **The `>= 50` threshold is load-bearing on large plans.** On cli-printing-press, baseline's graduated per-severity gates admitted 13 Decisions; treatment admitted 21. Inspection of the delta confirmed the new findings were genuine concerns the old gates' coin-flip behavior at boundaries was suppressing — not noise. The migration doc's prediction that "missing a real concern is expensive" held in practice.
|
||||
|
||||
### Anchor-75 calibration boundary discovered
|
||||
|
||||
The evaluation surfaced a boundary issue: on large plans, personas emitted anchor 75 for premise-strength concerns ("motivation is thin," "premise is unconvincing") whose "will be hit in practice" claim was the reviewer's reading, not a concrete downstream outcome. This inflated the actionable tier with strength-of-argument critique that was more appropriately observational.
|
||||
|
||||
The subagent template's anchor 75 bullet was refined with a calibration paragraph:
|
||||
|
||||
> **Anchor `75` requires naming a concrete downstream consequence someone will hit** — a wrong deploy order, an unimplementable step, a contract mismatch, missing evidence that blocks a decision. Strength-of-argument concerns ("motivation is thin," "premise is unconvincing," "a different reader might disagree") do not meet this bar on their own — they are advisory observations and land at anchor `50` unless they also name the specific downstream outcome the reader hits.
|
||||
|
||||
The test the template adds: *"will a competent implementer or reader concretely encounter this, or is this my opinion about the document's strength?"* The former is `75`; the latter is `50`.
|
||||
|
||||
Re-evaluation with the tightened criterion shifted cli-printing-press from 21 Decisions/4 FYI to 10 Decisions/23 FYI — premise-strength concerns moved to observational routing. The change was *not* a blanket suppression of premise findings: on interactive-judgment plan, the premise challenge survived the tightening and got cross-persona-promoted to anchor 100, because its concrete consequence was explicit ("8-unit redesign creates maintenance debt across three reference files if the premise is wrong"). The refinement distinguishes grounded premise challenges from hand-wavy framing critique — which is the exact precision the rubric was meant to have from the start.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Small corpus.** Four documents is enough to confirm macro patterns (clustering, severity inflation, feature coverage) but not to tune threshold values or anchor boundaries at finer granularity.
|
||||
- **Harness drift between iterations.** Iteration-1 orchestrators dispatched parallel persona subagents; iteration-2 orchestrators executed personas inline (nested Agent tool unavailable in that session). This affected side metrics (proposed-fix count on cli-printing-press iteration-2 dropped 15 → 4, likely harness-driven rather than tweak-driven) but did not obscure the tweak's core effect, which was large-magnitude.
|
||||
- **No absolute-calibration ground truth.** The evaluation measured the migration's stated failure modes disappearing. Whether an anchor-75 finding literally hits 75% of the time remains unmeasured; no labeled doc-review corpus exists.
|
||||
|
||||
## ce-code-review migration (2026-04-21)
|
||||
|
||||
Ported the same anchored-rubric structure into `ce-code-review` and bundled it with three additional code-review-specific precision controls. The two skills now share calibration discipline but diverge on threshold and on how independent verification is implemented.
|
||||
|
||||
### Threshold: `>= 75` (not `>= 50` like ce-doc-review, not `>= 80` like Anthropic)
|
||||
|
||||
ce-code-review uses anchor 75 as the gate. P0 findings escape at anchor 50.
|
||||
|
||||
`>= 75` matches the ce-doc-review choice of using the anchor itself as the threshold (no awkward middle-bucket gap). At `>= 75`, anchors 75 ("real, will hit in practice") and 100 ("verifiable from code alone") survive; anchors 0/25/50 are dropped. Anthropic's `>= 80` under a discrete `{0,25,50,75,100}` scale would collapse to "anchor 100 only," which is too narrow — it would silence findings where personas can construct the trace but cannot literally read the bug off the code.
|
||||
|
||||
The threshold divergence from ce-doc-review (`>= 50`) is correct for the same reasons documented in the "Why the threshold diverges from Anthropic" section above, applied in reverse: code review HAS a linter backstop, IS publicly visible, and code claims ARE ground-truth verifiable. Code review wants narrow precision; doc review wants broad surfacing.
|
||||
|
||||
### Validation pass (Stage 5b): the deferred follow-up, now landed
|
||||
|
||||
The ce-doc-review plan deferred a "neutral-scorer second pass" to a follow-up plan. ce-code-review implements it as **Stage 5b**: an independent validator sub-agent per surviving finding, mode-conditional dispatch, and a 15-finding budget cap.
|
||||
|
||||
- **Why now for code review, not doc review:** code review has externalizing modes (autofix applies fixes, headless returns findings to programmatic callers) where false positives have real cost — wrong fixes get committed, downstream automation acts on bad signal. Doc review's worst case is a noisy report a user dismisses with a keystroke; code review's worst case is a wrong-fix PR getting merged.
|
||||
- **Mode-conditional dispatch:** validation runs in `headless`, `autofix`, and the interactive LFG/File-tickets routing paths. It is skipped in interactive walk-through (the human is the per-finding validator) and report-only (nothing is being externalized). This scopes cost to the cases where false positives have real cost.
|
||||
- **Per-finding parallel dispatch, not batched:** independence is the design point. A single batched validator looking at all findings together pattern-matches across them and recreates the persona-bias problem we are escaping. Per-file batching is left as a future optimization for reviews with many findings clustered in few files.
|
||||
- **No `validated` field on findings:** an early plan added a `validated: boolean` field; it was removed during planning. Surviving findings post-validation are validated by definition (rejected ones are dropped); in modes where validation does not run, the run's mode tells consumers everything they need. A field constant within any mode does no work.
|
||||
- **Conservative failure mode:** validator timeout, malformed output, or dispatch error → drop the finding. Unverified findings should not externalize.
|
||||
|
||||
The validator's protocol is `{ "validated": true | false, "reason": "<one sentence>" }` answering three questions: is the issue real, is it introduced by THIS diff, and is it not handled elsewhere. Template: `references/validator-template.md`.
|
||||
|
||||
### Mode-aware false-positive demotion
|
||||
|
||||
ce-code-review's broader persona surface (~14 reviewers vs ce-doc-review's 7) means more weak general-quality signal. Stricter precision in externalizing modes was already accomplished by the higher threshold; for interactive mode, a different policy: route weak findings to existing soft buckets (`testing_gaps`, `residual_risks`, `advisory`) rather than suppress.
|
||||
|
||||
The demotion rule is intentionally narrow: severity P2 or P3, `autofix_class` advisory, contributing reviewer is `testing` or `maintainability`. Headless and autofix suppress these entirely; interactive and report-only demote them to soft buckets where they remain visible without competing for primary-findings attention.
|
||||
|
||||
This is the "tier the precision bar by mode" framing. Synthesis owns it; personas don't change what they flag based on mode.
|
||||
|
||||
### Lint-ignore suppression
|
||||
|
||||
Code carrying an explicit lint disable comment for the rule a reviewer is about to flag (`eslint-disable-next-line no-unused-vars`, `# rubocop:disable Style/...`, `# noqa: E501`, etc.) — suppress unless the suppression itself violates a project-standards rule. The author already chose to suppress; re-flagging via a different reviewer creates noise and ignores their decision.
|
||||
|
||||
This is the only entirely new false-positive category in ce-code-review's catalog; the rest were ported from the existing pre-anchor catalog.
|
||||
|
||||
### PR-mode skip-condition pre-check
|
||||
|
||||
Before running the full review on a PR, a single `gh pr view` call probes for skip conditions:
|
||||
- Closed or merged PR
|
||||
- Draft PR
|
||||
- Trivial automated PR (conservative `chore(deps)` / `build(deps)` / release-bump pattern with empty body)
|
||||
- Already has a ce-code-review report comment
|
||||
|
||||
Skip cleanly without dispatching reviewers. Standalone branch and `base:` modes always run — the skip-check is PR-mode only. Already-reviewed detection deliberately ignores commits-since-comment; the escape hatch for "I want to re-review after pushing more commits" is branch mode or `base:` mode, both of which bypass the skip-check entirely.
|
||||
|
||||
This avoids the wasted multi-agent review cost on PRs that should not be reviewed (closed, draft, dependabot-style, or already-reviewed). It is the cheapest mechanism in this migration and disproportionately valuable for any team that runs the skill against arbitrary PR queues.
|
||||
|
||||
### Files
|
||||
|
||||
- `skills/ce-code-review/references/findings-schema.json` — `confidence` is integer enum `[0, 25, 50, 75, 100]` with code-review-specific behavioral definitions in the description; `_meta.confidence_anchors` and `_meta.confidence_thresholds` document the anchors and `>= 75` gate
|
||||
- `skills/ce-code-review/references/subagent-template.md` — verbatim 5-anchor rubric with code-review framing, expanded false-positive catalog including lint-ignore rule, hard schema-conformance constraints rejecting floats
|
||||
- `skills/ce-code-review/references/validator-template.md` — Stage 5b validator subagent prompt
|
||||
- `skills/ce-code-review/SKILL.md` — Stage 5 anchor gate and one-anchor promotion (replaces `+0.10`), Stage 5 step 7c mode-aware demotion, Stage 5b validation pass with budget cap, Stage 1 PR-mode skip-condition pre-check, After-Review options B and C invoke validation before externalizing
|
||||
- `skills/ce-code-review/references/personas/*.md` — the code-review reviewer personas updated from float bands to anchored language, preserving each persona's specific calibration signal
|
||||
- `skills/ce-code-review/references/review-output-template.md` — Confidence column renders as integer (`75`, `100`), not float
|
||||
- `tests/review-skill-contract.test.ts` — schema, synthesis, validation pass, skip-conditions, mode-aware demotion, and per-persona anchored-language assertions
|
||||
|
||||
### When to apply this combined pattern to a new skill
|
||||
|
||||
Apply the full bundle (anchored rubric + validation pass + mode-aware demotion + skip-conditions + lint-ignore) when **all** of:
|
||||
1. The skill is a multi-persona review workflow producing structured findings.
|
||||
2. The skill has externalizing modes — outputs that get acted on without further human review (PR comments, autofix, downstream automation, headless callers).
|
||||
3. The skill is invoked frequently enough that wasted runs are visible (skip-conditions are pure win in this case; modest cost in low-volume cases).
|
||||
|
||||
Apply only the **anchored rubric** (the ce-doc-review subset) when:
|
||||
- The skill is single-shot or dismissal is cheap via UI/menu — validation pass adds cost without protecting anything that wasn't already going to be triaged by a human.
|
||||
- The skill operates on premise/strategy claims that lack ground-truth verification — anchor 100 is unreachable; threshold should be `>= 50`.
|
||||
|
||||
Skip the entire pattern when:
|
||||
- The skill produces a single value, not a population of findings.
|
||||
- The skill operates on user input where the user IS the source of truth (e.g., interactive Q&A skills).
|
||||
|
||||
### Migration history (ce-code-review)
|
||||
|
||||
Landed in a single PR with anchored rubric, validation pass, skip-conditions, mode-aware demotion, lint-ignore suppression, and persona sweep all together. The schema change is the load-bearing commit; subagent template, synthesis, and persona updates consume it. Branch: `refactor/ce-code-review-precision-and-validation`. The plan with full decision rationale lives at `docs/plans/2026-04-21-002-refactor-ce-code-review-precision-and-validation-plan.md`.
|
||||
|
||||
## Deferred follow-ups
|
||||
|
||||
- **PR inline comment posting mode for ce-code-review.** Anthropic's plugin posts findings as inline GitHub PR comments via `mcp__github_inline_comment__create_inline_comment` with full-SHA link discipline and committable suggestion blocks. ce-code-review currently has no PR-comment mode at all (terminal output, fixer auto-apply, or headless return only). Real workflow gap; deferred because it is a substantial new mode (link format, suggestion-block handling, deduplication semantics, tracker integration overlap).
|
||||
- **Per-file validator batching.** When real-world reviews routinely surface many findings clustered in few files (large refactors), a per-file validator that reads the file once and evaluates all findings against it could meaningfully reduce cost while preserving cross-file independence. Implement when data shows the saving matters.
|
||||
- **Haiku-tier orchestrator-side checks.** ce-code-review currently uses sonnet for all subagent dispatch including the cheap PR skip-condition probe. Push obvious cheap checks (skip-conditions, standards path discovery) to haiku.
|
||||
- **Re-evaluate which always-on personas earn their noise.** ce-code-review keeps `testing` and `maintainability` always-on with mode-aware demotion as the safety valve. If real review runs show the demotion is firing constantly, consider making them conditional rather than always-on.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Cross-harness/cross-model skills drive agent tool calls, not slash commands — describe the capability, verify it live"
|
||||
category: skill-design
|
||||
date: 2026-07-11
|
||||
module: skills/ce-babysit-pr
|
||||
problem_type: design_pattern
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "Authoring a skill, once, for multiple agent harnesses and models where the skill needs the agent to invoke a capability (schedule/loop, ask the user, invoke a sub-skill, run background work, drive a browser)"
|
||||
- "Deciding whether to name a specific tool/command in skill prose or describe the capability"
|
||||
- "About to assume a particular tool or slash command exists or behaves a certain way on Codex/Grok/Cursor/Claude Code"
|
||||
- "A skill's capability works on the harness/model you authored it in but fails on another"
|
||||
- "A skill can invoke a slash-command-like affordance interactively but not from within its own execution"
|
||||
tags:
|
||||
- cross-harness
|
||||
- cross-model
|
||||
- capability-over-tool
|
||||
- tool-vs-slash-command
|
||||
- empirical-verification
|
||||
- portable-skills
|
||||
- orchestration
|
||||
- ce-babysit-pr
|
||||
related_components:
|
||||
- development_workflow
|
||||
- tooling
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
A skill authored once for many harnesses (Claude Code, Codex, Grok, Cursor, …) frequently needs the *agent* to invoke a harness capability — schedule a re-invocation, ask the user a blocking question, invoke another skill, run a background process, drive a browser. The tempting design is to hardcode the mechanism per harness: "on Codex call X, on Grok call Y." That path is brittle in two directions at once — **cross-harness** (tool names and availability differ) and **cross-model** (even on one harness, models differ in what they'll reach for and how a tool behaves). It also silently confuses two different things: what a *skill* can drive versus what a *user* can do.
|
||||
|
||||
This crystallized while designing `ce-babysit-pr`'s self-sustaining loop, where a plan full of per-harness scheduler assumptions turned out to be partly wrong once tested — but the lesson is general to any capability a skill needs invoked.
|
||||
|
||||
## Guidance
|
||||
|
||||
**1. The reliable unit a skill can drive is an agent *tool call*, not a user affordance.** Skill prose steers the agent; the agent invokes *tools*. It cannot press keys or type a slash command, so a user-typed command (e.g. Cursor's `/loop`) is **not skill-invocable** — verified live: the agent reported `/loop` "only loads instructions into context." The exception is a slash command the harness *also* exposes as a tool (Claude Code's `Skill` tool can invoke `/loop`, so that one counts). Design for tool calls; treat any "have the skill run /command" step as a smell to verify.
|
||||
|
||||
**2. The tool surface varies by harness *and* model, and agents reach for the *simplest sufficient* tool, not the fanciest.** Given a capability need with the mechanism unspecified, fresh agents on all four harnesses built a plain background shell loop — **none reached for a first-class scheduler tool**, even Grok, whose `scheduler_create` (durable, agent-callable) was right there; it explicitly skipped it as overkill. So designing around a specific "correct" tool is doubly wrong: it may not exist on another harness, and even where it does, the model won't necessarily pick it.
|
||||
|
||||
**3. Therefore: name the known tool as a short-circuit, but describe the *capability* as the portable fallback.** For a recognized harness, state the specific agent tool for an instant, unambiguous pick. Underneath it, describe what the tool must *do* ("a way to run a background process and be woken when it emits a line, without ending your turn"; "the platform's blocking-question capability"), so an agent whose tool is absent, renamed, or newer can still satisfy the need — and degrade explicitly when nothing fits. Both, not either: the named tool is speed, the capability description is robustness.
|
||||
|
||||
**4. Verify per-harness/per-model tool behavior *empirically*, with live agents — not from your authoring runtime.** Assumptions baked from one runtime ship wrong. Two live checks (fresh agents per harness, dispatched via orchestration) corrected real errors before they landed: Codex's CLI exposes **no** scheduler tool and a detached `nohup` is *reaped* the instant the tool call ends (only a runtime-owned handle survives); Cursor's `/loop` is not skill-invocable; Grok's `scheduler_create` is durable and agent-callable but goes unused. This operationalizes the "portable agent skill authoring" decentering principle: when a design rests on per-harness/per-model tool behavior, prove it with agents on those targets.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The failure mode is expensive because it is invisible in-repo: confidently-wrong per-harness prose (a background process that silently dies, a slash command the skill can't trigger, a tool that isn't there) passes every unit test and static check — it only fails at *runtime*, on the harness/model you didn't author in. No amount of `bun test` catches it. Describing capabilities plus verifying with live agents is the only guard, and it applies to every capability a portable skill invokes, not just scheduling.
|
||||
|
||||
It also collapses complexity. Reframing `ce-babysit-pr` from "detect the harness → call its scheduler → guard a bundled driver script against nesting" to "describe the watch intent, let the agent build the loop with whatever it has" deleted a driver script, a tool-tier matrix, and a sentinel guard — because it stopped fighting what agents already do well.
|
||||
|
||||
One boundary the same experiments surfaced: agents pick the *simplest* sufficient tool, which for a trivial task is a dumb shell command. When the per-invocation work is actually agent *reasoning* (invoke a sub-skill, judge feedback), the prose must say so, or the agent takes the shell shortcut that can't do the reasoning.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any skill authored once for multiple harnesses/models that needs the agent to invoke a capability — scheduling/looping, blocking questions, sub-skill invocation, background work, browser or MCP tools.
|
||||
- Whenever prose is about to say "call tool X" or "run /command" for a cross-harness action — first ask whether describing the capability works, keep the named tool only as a short-circuit, and verify X on the target harness+model.
|
||||
- Not needed for a single-harness skill, or for a capability already proven on the target runtime.
|
||||
|
||||
## Examples
|
||||
|
||||
**Slash command vs tool (the sharp distinction):** "have the skill run `/loop`" is not portable — a skill can't type it, and on Cursor it isn't agent-invocable at all. "Use whatever agent tool re-invokes work on a cadence (Claude Code: `ScheduleWakeup` / the `Skill`-tool-invoked `/loop`; Grok: `scheduler_create`; else a background process the agent runs)" is portable, because it targets tool calls and names tools only as examples.
|
||||
|
||||
**Capability over tool (asking the user):** prose that says `AskUserQuestion` breaks off-Claude. `ce-babysit-pr` instead says "use the platform's blocking-question tool" and lists `AskUserQuestion` / `request_user_input` / `ask_question` / `ask_user` as examples with a chat fallback — one capability, many tools.
|
||||
|
||||
**Live verification (the reusable technique):** publish a controllable external artifact (an [ht-ml.app](https://ht-ml.app) page), dispatch a fresh agent per harness with intent-only instructions to watch it and react to a change, then change it and confirm each caught the change unattended with an unguessable value. Proof beats assumption for anything that varies by harness or model — and it is how "Codex `nohup` is reaped" and "Cursor `/loop` isn't skill-invocable" were caught before shipping.
|
||||
|
||||
## Related
|
||||
|
||||
- [Watch-loop skills need a blocked-external terminal state for fork-PR CI approval gates](./watch-loops-need-a-blocked-external-terminal-state.md) — a sibling `ce-babysit-pr` learning; the motivating example (its self-sustaining loop) is where this general principle surfaced.
|
||||
- [Bundled script path resolution across harnesses](./bundled-script-path-resolution-across-harnesses.md), [`arguments` token is Claude-only in skill bodies](./arguments-token-is-claude-only-in-skill-bodies.md) — sibling cross-harness-portability learnings; same "don't assume your runtime is universal" root.
|
||||
- Design artifact: `docs/plans/2026-07-11-001-feat-babysit-self-initiating-loop-plan.md`.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: Building a shared cached primitive across self-contained skills
|
||||
date: 2026-06-29
|
||||
category: docs/solutions/skill-design/
|
||||
module: repo-grounding-cache
|
||||
problem_type: architecture_pattern
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Multiple skills independently re-derive the same expensive, question-agnostic data
|
||||
- You want one shared cache/helper/persona reused across skills that cannot import each other
|
||||
- Adding a cross-skill optimization where the plugin's self-contained-skill rule forbids shared modules
|
||||
tags: [cross-skill, cache, skill-design, duplication, parity-test, skill-creator-eval]
|
||||
---
|
||||
|
||||
# Building a shared cached primitive across self-contained skills
|
||||
|
||||
## Context
|
||||
|
||||
Repo-grounding skills (`ce-pov`, `ce-plan`, `ce-optimize`, `ce-ideate`, `ce-brainstorm`, `ce-code-review`, `ce-compound`, `ce-debug`) each independently re-derived the same question-agnostic "project profile" (stack, deps, conventions, structure) on every run — an expensive grounding pass repeated per skill and per invocation. We wanted one shared, cached primitive. But the plugin's hard constraint (AGENTS.md "File References in Skills") forbids cross-skill imports: the converter copies each skill directory as an isolated unit, so a skill may only reference files inside its own tree. There is no shared-module mechanism.
|
||||
|
||||
## Guidance
|
||||
|
||||
Build the "shared" primitive as **byte-duplicated assets plus a parity test**, not a shared import:
|
||||
|
||||
1. **Duplicate the assets into every consuming skill.** Here that was three files per skill: a protocol/schema reference (`references/repo-profile-cache.md`), a bundled helper script (`scripts/repo-profile-cache.py`), and a derivation persona (`references/agents/repo-profiler.md`). Each consumer carries identical copies.
|
||||
2. **Guard file drift with a `tests/` byte-identity test** — declare the asset filenames x the consumer-skill list and assert each copy equals the first (mirror `tests/compound-support-files.test.ts`). Adding a consumer = drop in the copies + add its name to the test's `CONSUMER_SKILLS`.
|
||||
3. **Invoke the bundled script via the `SKILL_DIR` anchor**, never `${CLAUDE_SKILL_DIR}` (see `docs/solutions/skill-design/bundled-script-path-resolution-across-harnesses.md`).
|
||||
4. **Put deterministic work in the script, judgment in the persona.** The Python helper does git keying + validity + atomic read/write (unit-testable); the LLM persona derives the profile only on a miss.
|
||||
5. **Share state through a single OS-temp location, keyed by content/identity**, so any skill reads what another wrote (`/tmp/compound-engineering/repo-profile/<root-sha>/<head-sha>.json`).
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The parity test guards **file** drift but **not integration** drift: a consumer can carry byte-identical assets yet wire the grounding phase wrong (skip the cache, or — worse — skip the still-required question-specific grounding). Two layers are needed:
|
||||
|
||||
- **Parity test** (`bun test`) — proves the duplicated files are identical across skills.
|
||||
- **Per-consumer `skill-creator` grounding-phase eval** — proves each skill actually *uses* the primitive correctly: takes the agnostic slice from the cache AND keeps its question-specific work fresh. This is the only check that catches integration drift, and it is manual (not run by CI).
|
||||
|
||||
One more unguarded seam: **per-skill field reads** (e.g. a SKILL.md that reads `conventions.testing` from the profile JSON) are not byte-duplicated, so renaming a schema field passes the parity test and the version bump yet silently breaks consumers — document a "grep the consumers for the field" step in the schema-change checklist.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Several skills re-derive the same stable, question-agnostic data and you want to compute it once.
|
||||
- The shared thing is genuinely reusable across skills (not one skill's private concern).
|
||||
- You can express the shared contract as a small, self-contained asset set (reference + script + persona) rather than a code dependency.
|
||||
|
||||
Do **not** reach for this when only one skill needs the data, or when the "shared" data is actually question-specific per skill (then there is nothing stable to cache).
|
||||
|
||||
## Examples
|
||||
|
||||
The validation layering that made the difference:
|
||||
|
||||
```
|
||||
tests/repo-profile-cache-parity.test.ts # FILE drift: 3 assets x 8 skills byte-identical
|
||||
skill-creator evals (per consumer) # INTEGRATION drift: agnostic-from-cache AND
|
||||
# question-specific-still-fresh, per skill
|
||||
```
|
||||
|
||||
The grounding-phase eval pattern is cheap and high-signal precisely because the cache logic lives at the *front* of each skill — you can drive a fresh subagent through just the grounding phase (cache HIT/MISS/NO-CACHE) and observe its decisions without running the whole skill. That pattern caught real wiring issues on every batch it ran.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/bundled-script-path-resolution-across-harnesses.md` — the `SKILL_DIR` anchor used to invoke the shared script
|
||||
- `docs/solutions/skill-design/script-first-skill-architecture.md` — deterministic-script / model-presents split
|
||||
- `docs/solutions/best-practices/cache-invalidation-input-set-completeness.md` — the cardinal rule for the cache this pattern shipped
|
||||
- `docs/solutions/skill-design/paired-old-vs-new-injection-skill-evals.md` — generalizes the field-rename parity gap noted here into a rule: prove the anti-drift test fails on one-sided drift before trusting it.
|
||||
- AGENTS.md "Shared Repo-Grounding Profile Cache" and "File References in Skills"
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: Discoverability check for documented solutions in project instruction files
|
||||
date: 2026-03-30
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: convention
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Adding a post-write verification step to a knowledge-compounding skill
|
||||
- Ensuring documented knowledge is discoverable by agents in fresh sessions
|
||||
- Designing skills that may modify project instruction files
|
||||
- Onboarding a new agent platform that reads its own instruction file
|
||||
tags:
|
||||
- discoverability
|
||||
- ce-compound
|
||||
- ce-compound-refresh
|
||||
- instruction-files
|
||||
- skill-design
|
||||
- knowledge-compounding
|
||||
---
|
||||
|
||||
# Discoverability check for documented solutions in project instruction files
|
||||
|
||||
## Context
|
||||
|
||||
Knowledge stores — structured directories of solutions, patterns, and learnings — only compound value when agents can find them. A project might accumulate dozens of well-categorized documents under `docs/solutions/` with YAML frontmatter, category directories, and searchable fields, yet agents in fresh sessions, different tools, or collaborators without the originating plugin would never know to look there.
|
||||
|
||||
The root cause: project instruction files (`AGENTS.md`, `CLAUDE.md`, `.cursorrules`, etc.) are the universal discovery surface. Every agent platform reads them on session start. If the instruction file doesn't mention the knowledge store, the agent has no reason to search for it — and no way to know what structure to expect if it stumbled upon it accidentally.
|
||||
|
||||
This gap becomes more costly as the knowledge store grows. Each undiscovered solution means an agent re-derives something already documented, wastes tokens on exploration, or arrives at a contradictory approach because it never found the prior decision.
|
||||
|
||||
## Guidance
|
||||
|
||||
After writing or updating a knowledge store entry, verify that the project's root instruction files give agents enough information to discover and use the store. The check has three parts:
|
||||
|
||||
**1. Identify the substantive instruction file.**
|
||||
|
||||
Projects often have multiple instruction files where one is a shim that delegates to another (e.g., `CLAUDE.md` containing only `@AGENTS.md`). Target the file with actual content, not the shim.
|
||||
|
||||
**2. Semantically assess discoverability — not string presence.**
|
||||
|
||||
An agent reading the instruction file should be able to answer three questions:
|
||||
- Does a searchable knowledge store exist in this project?
|
||||
- What is its structure (location, categories, metadata format)?
|
||||
- When should I search it?
|
||||
|
||||
This is a semantic check, not a grep for a path string. A file might mention `docs/solutions/` in a directory tree without conveying that it's searchable or when to use it. Conversely, a file might describe the knowledge store without using the exact directory path.
|
||||
|
||||
**3. Draft the smallest effective addition.**
|
||||
|
||||
If discoverability is missing, the addition should be minimal and stylistically consistent:
|
||||
|
||||
- Prefer augmenting an existing section (directory listing, architecture description) over adding a new headed section
|
||||
- Match the file's existing density and tone — a terse file gets a terse addition
|
||||
- Use informational tone, not imperative — describe what exists and when it's relevant, rather than issuing commands
|
||||
|
||||
**4. Gate on user consent.**
|
||||
|
||||
Never edit instruction files without asking. In interactive mode, present the proposed change and ask for approval using the platform's question tool. In automated or autofix mode, surface the recommendation without applying it.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Without discoverability, a knowledge store has zero value outside the session that wrote it. The entire premise of compounding knowledge is that future sessions build on past ones. If future sessions can't find the store, every session starts from scratch.
|
||||
|
||||
The cost is proportional to the store's size: a project with 50 documented solutions where agents never search wastes more effort than one with 3. The waste is silent — no error, no warning, just redundant work and occasionally contradictory decisions.
|
||||
|
||||
Keeping the addition minimal and informational avoids a secondary problem: imperative directives like "always search the knowledge store before implementing" cause agents to perform redundant reads when the active workflow already includes a dedicated search step. The instruction file should make the store discoverable, not mandate a specific workflow around it.
|
||||
|
||||
The semantic approach (assessing whether an agent would discover the store) rather than syntactic matching (grepping for a path) avoids both false positives (path appears in a tree but conveys nothing about searchability) and false negatives (description uses different phrasing but fully communicates the store's purpose).
|
||||
|
||||
## When to Apply
|
||||
|
||||
- **After creating a knowledge store for the first time** — the most critical moment, since no prior session has had reason to mention it
|
||||
- **After writing or refreshing a learning** in an existing store — the check is cheap and catches instruction files that were refactored or regenerated without the discoverability note
|
||||
- **When onboarding a new agent platform** — if the project adds `.cursorrules` alongside existing `AGENTS.md`, the new file needs the same discoverability affordance
|
||||
- **When instruction files are substantially rewritten** — reorganization can drop a previously-present mention
|
||||
|
||||
The check is unnecessary when:
|
||||
- The instruction file was just verified in the current session
|
||||
- The knowledge store is part of a plugin that injects its own discovery mechanism (the plugin's agents already know where to look)
|
||||
|
||||
## Examples
|
||||
|
||||
**Existing directory listing — add a single line:**
|
||||
|
||||
Before:
|
||||
```
|
||||
src/ Application source code
|
||||
tests/ Test suite and fixtures
|
||||
docs/ Project documentation
|
||||
scripts/ Build and deploy scripts
|
||||
```
|
||||
|
||||
After:
|
||||
```
|
||||
src/ Application source code
|
||||
tests/ Test suite and fixtures
|
||||
docs/ Project documentation
|
||||
docs/solutions/ Categorized solutions with YAML frontmatter; relevant when implementing or debugging in areas with prior decisions
|
||||
scripts/ Build and deploy scripts
|
||||
```
|
||||
|
||||
One line, matches the existing style, communicates all three things: the store exists, it's structured, and when to use it.
|
||||
|
||||
---
|
||||
|
||||
**No natural insertion point — small headed section:**
|
||||
|
||||
Before:
|
||||
```markdown
|
||||
# Project Instructions
|
||||
|
||||
Use TypeScript strict mode. Run `npm test` before committing.
|
||||
Prefer composition over inheritance.
|
||||
```
|
||||
|
||||
After:
|
||||
```markdown
|
||||
# Project Instructions
|
||||
|
||||
Use TypeScript strict mode. Run `npm test` before committing.
|
||||
Prefer composition over inheritance.
|
||||
|
||||
## Knowledge Store
|
||||
|
||||
`docs/solutions/` contains categorized solution documents with YAML frontmatter
|
||||
(category, severity, tags). Searching this directory is useful when implementing
|
||||
features or debugging issues in areas where prior decisions have been recorded.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Shim file — skip it:**
|
||||
|
||||
```markdown
|
||||
@AGENTS.md
|
||||
```
|
||||
|
||||
This file delegates entirely to `AGENTS.md`. The discoverability note belongs in `AGENTS.md`, not here. Adding content to a shim file defeats its purpose.
|
||||
|
||||
## Related
|
||||
|
||||
- [#111](https://github.com/EveryInc/compound-engineering-plugin/issues/111) — Enhancement: Add project scaffolding for `docs/solutions/` schema + agentic feedback loops. The discoverability check is a lighter-weight partial solution to this issue's "medium-term" suggestion of making ce-compound check for scaffolding.
|
||||
- [#171](https://github.com/EveryInc/compound-engineering-plugin/issues/171) — Closed-Loop Self-Improvement System. The discoverability check helps close part of this loop by ensuring agents can find `docs/solutions/` content.
|
||||
- `docs/solutions/skill-design/compound-refresh-skill-improvements.md` — Documents the ce-compound-refresh skill redesign. The discoverability check adds a new step to that skill's workflow.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "Validate skill judgment changes with a fake-CLI harness and a discriminating fixture"
|
||||
category: skill-design
|
||||
module: ce-resolve-pr-feedback
|
||||
problem_type: best_practice
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Changing the judgment or decision logic of a skill that calls an external service (gh, git, an HTTP API)
|
||||
- Needing evidence that a prose/behavior change to a skill actually improves outcomes, not just that it runs
|
||||
- Comparing two versions of a skill (new vs committed baseline) on the same task
|
||||
tags:
|
||||
- skill-eval
|
||||
- testing
|
||||
- fake-cli
|
||||
- fixture-design
|
||||
- ce-resolve-pr-feedback
|
||||
- skill-design
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
`ce-resolve-pr-feedback` was changed so the orchestrator judges every PR finding centrally (a "legitimacy gate") before dispatching fixer subagents, instead of each finding being judged-and-fixed by an isolated per-thread subagent. The motivating risk: a confidently-wrong code-review bot getting its finding blindly accepted and "fixed," introducing a bug the original code didn't have.
|
||||
|
||||
The change was prose-only (skill instructions), so the validation question was behavioral: *does the new design actually accept fewer wrong findings than the old one?* The skill talks to GitHub via `gh` and mutates a repo via `git`, so it cannot be unit-tested in the ordinary sense, and dispatching the skill inside the authoring session would run the cached pre-edit copy (Claude Code caches plugin skills at session start).
|
||||
|
||||
## Guidance
|
||||
|
||||
Build a **fake-CLI harness** plus a **discriminating fixture**, then run the skill new-vs-old over several reps and grade objective outcomes.
|
||||
|
||||
**1. Mock at the CLI boundary, not the network.** The skill's only external touchpoints are `gh` and `git`. Put a fake `gh` executable first on `PATH` that dispatches on argv and returns canned fixture JSON (matching the real output shape after `gh api graphql --slurp`), and logs every mutation (replies, resolves) to a file. Run inside a throwaway `git init` repo with a local bare remote so `git push` is a harmless no-op. This drives the skill's *real* bundled scripts (`get-pr-comments`, `reply-to-pr-thread`, …) unchanged — you mock what they call, not what they are. No network, no auth, no real PR.
|
||||
|
||||
**2. Observe outcomes through the side effects you already have.** Tag a baseline commit (`eval-base`) before the run; the grader diffs against that tag (not `HEAD`, because the skill commits its own fixes on top). Grade three channels: the `git diff` of the work-tree, the mutation log (what replies it posted), and the run's summary. Key metric: a binary `BLIND_ACCEPT=yes|no` per rep.
|
||||
|
||||
**3. The fixture must be *discriminating*, or the eval proves nothing.** A finding that is disprovable by reading the one file it points at is too easy — every design catches it, including the one you're trying to show is worse. Construct findings whose disproof lives **outside the referenced file**, so an isolated, narrowly-scoped agent is tempted to "fix" while a design with broader context debunks it. The sharpest case is a *systematically-wrong cluster*: several individually-plausible findings from one source, all false for the same reason (a shared invariant the referenced files don't reveal).
|
||||
|
||||
**4. Run new-vs-old over several reps and inspect the mechanism, not just the count.** Skill judgment is non-deterministic. A small-N ratio is suggestive; what makes it convincing is reading the failing reps and confirming they fail *the predicted way*.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The first fixture (a single bogus "null deref" finding, disprovable by a guard three lines up in the same file) showed **0/4 blind-acceptance on both** the new and old designs — i.e., it could not tell them apart. It would have let us "confirm" the change with no evidence, or wrongly conclude it did nothing.
|
||||
|
||||
The discriminating cluster fixture (3 plausible findings that `req.body.amount` is unvalidated, all false because a shared `validateAmount` middleware in a *different* file guards every route) separated them:
|
||||
|
||||
| Design | Blind-accepted ≥1 false finding | Per-rep |
|
||||
|--------|-------------------------------|---------|
|
||||
| New (central gate) | **0/4** | 6,6,6,5* |
|
||||
| Old (per-thread judge+fix) | **2/4** | 6,6,2,2 |
|
||||
|
||||
\*the one new-design "miss" was a grader phrasing strictness, not blind acceptance.
|
||||
|
||||
The mechanism check confirmed it: old-design failures were the exact predicted pathology — each isolated agent read only its own handler file, never saw the shared middleware, added a redundant guard, and replied `Addressed:`. The insight isn't "old is always wrong" (it was right 2/4) — it's that the old design's correctness *depends on whether an isolated agent happens to read the right file*, while the gate makes it reliable. That distinction is invisible to an easy fixture.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any change to a skill's decision/judgment logic where "it runs" is not the same as "it decides better."
|
||||
- Especially when the skill hits an external service: mock the CLI/tool boundary so the real bundled scripts still execute.
|
||||
- Skip the harness for mechanical skill changes (parsing, output paths, anything a normal test exercises) — those run current source and don't need it.
|
||||
|
||||
## Examples
|
||||
|
||||
Fake `gh` dispatch (sketch): match argv → `repo view`/`pr view` return identity; `api graphql` inspects the `-f query=` body to return `threads/comments/reviews.json` or log a `REPLY`/`RESOLVE` mutation. The reply/resolve scripts call this fake and append to `mutations.log`.
|
||||
|
||||
Discriminating fixture shape:
|
||||
- `src/handlers.js` — three handlers use `req.body.amount` raw (the referenced lines; look naive in isolation).
|
||||
- `src/routes.js` + `src/middleware.js` — a `validateAmount` middleware wired to every route guarantees `amount` is a positive finite number. **The disproof lives here, not in `handlers.js`.**
|
||||
- one genuine bug (`src/math.js` divide-by-zero) as a control, so a design can't "win" by skipping everything.
|
||||
|
||||
Grader (objective): `handlers.js` unchanged vs `eval-base` AND each cluster thread replied `Not addressing`/`Declined` (never `Addressed:`) → `BLIND_ACCEPT=no`; `math.js` modified + `Addressed` → control passed.
|
||||
|
||||
The full harness lives under `.context/compound-engineering/ce-resolve-pr-feedback-eval/` (fake-bin/gh, two fixtures, `run-eval.sh`, `batch.sh`, graders). Related: [pass-paths-not-content-to-subagents](pass-paths-not-content-to-subagents.md), the in-session plugin-skill caching note in [bundled-script-path-resolution-across-harnesses](bundled-script-path-resolution-across-harnesses.md), and the sibling prose-injection variant [paired-old-vs-new-injection-skill-evals](paired-old-vs-new-injection-skill-evals.md) (injects SKILL.md excerpts into blind subagents instead of mocking a CLI boundary).
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
title: Frontier-model skill modernization methodology
|
||||
date: 2026-06-10
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: design_pattern
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "Reviewing or modernizing an existing skill against current frontier-model prompting guidance"
|
||||
- "Deciding whether to compress enumerated judgment examples or keep protocol text verbatim in SKILL.md"
|
||||
- "Designing model-tier vocabulary and degradation rules for sub-agent fleets"
|
||||
- "Extracting load-bearing skill content into reference files with inline load stubs"
|
||||
- "Verifying skill prose changes with injected-subagent evals instead of cached plugin dispatch"
|
||||
tags:
|
||||
- skill-design
|
||||
- prompting-guidance
|
||||
- frontier-models
|
||||
- model-tiers
|
||||
- context-window
|
||||
- reference-extraction
|
||||
- load-reliability
|
||||
- subagent-evals
|
||||
---
|
||||
|
||||
# Frontier-Model Skill Modernization Methodology
|
||||
|
||||
## Context
|
||||
|
||||
We modernized the `ce-ideate` skill (a ~13-agent ideation orchestrator) against the Claude Fable 5 prompting guide and the Claude prompting best-practices doc, then verified the result with a transcript-graded eval. The review surfaced a repeatable methodology for bringing any orchestration-heavy skill up to frontier-model standards: the skill went from 424 to 372 lines (-12%) while *gaining* capability (model tiering, file-based data flow, ceiling-raising dispatch mechanics), and the eval passed 6/8 mechanical assertions with 0 failures. This doc generalizes that sequence so the next skill review (ce-plan, ce-brainstorm, ce-code-review, ...) starts from the playbook instead of rediscovering it.
|
||||
|
||||
---
|
||||
|
||||
## Guidance
|
||||
|
||||
Run the steps in order. Each has a named test or rule — apply the test, don't improvise the judgment.
|
||||
|
||||
### 1. Audit: classify every prescriptive block as PROTOCOL or JUDGMENT
|
||||
|
||||
Read the skill top to bottom and tag each instruction block:
|
||||
|
||||
- **PROTOCOL** — *what to do*: output-format resolution order, cache file shapes, scratch paths, read budgets, agent counts, checkpoint mechanics. Unambiguous, costs a strong model nothing, and the workflow mechanically breaks without it. **Keep at full prescription.** (`git-workflow-skills-need-explicit-state-machines.md` is the canonical example of protocol content that regressed whenever it was softened to prose.)
|
||||
- **JUDGMENT** — *how to think*: enumerated example lists, multi-row sample classification tables, multi-paragraph elaborations of a single principle. A frontier model already has the capability; the prescription only narrows it. **Candidate for compression.**
|
||||
|
||||
The test: *would a strong model behave correctly given only the principle?* If yes, it's JUDGMENT. If the skill produces wrong file paths, wrong agent counts, or broken handoffs without it, it's PROTOCOL.
|
||||
|
||||
This refines (not replaces) the three-level prescription model in the plugin's AGENTS.md Skill Design Principles — hard rules / strong guidance / trust. Protocol maps to hard rules; the protocol-vs-judgment test decides which of the other two levels a block deserves.
|
||||
|
||||
### 2. Establish the orchestrator-model floor before cutting anything
|
||||
|
||||
Pruning JUDGMENT prescription is only safe if the skill realistically runs on frontier models. State the floor argument explicitly in your review notes. For ce-ideate: anyone launching a 13-agent workflow on any platform picks a frontier model, so the floor holds. If a skill plausibly runs on small models (e.g., a lightweight formatting skill), keep more scaffolding. The Fable guide's warning is the anchor: "Skills developed for prior models are often too prescriptive for Claude Fable 5 and can degrade output quality."
|
||||
|
||||
### 3. Prune: compress each JUDGMENT block to principle + ONE contrast pair
|
||||
|
||||
The compression rule: replace the enumeration with the underlying principle and a single minimal contrast pair that makes the boundary unmistakable. Example from ce-ideate: a list of vague-phrase examples became "`browser sniff` is identifiable, `quick wins` is not — vagueness is about referent, not length." One pair carries the distinction; seven rows of table did not carry more. Also deduplicate: triplicated boilerplate becomes one full copy + pointers. This matches the broader principle: prefer principles + a named test over enumerated specifics — specifics drift. (auto memory [claude])
|
||||
|
||||
### 4. Tier: define cost tiers semantically, once, and reference by name
|
||||
|
||||
Define three tiers in one place in SKILL.md; everywhere else refers to the tier name, never a model name:
|
||||
|
||||
- **Extraction tier** — cheapest capable model. Scouts, retrieval, quoting.
|
||||
- **Generation tier** — mid-tier model. Evidence-driven generation, mechanical verification.
|
||||
- **Ceiling tier** — *inherit the orchestrator's model by omitting the model parameter*. Never name a model for the ceiling.
|
||||
|
||||
Rules that travel with the tiers:
|
||||
|
||||
- Per-platform model hints follow the plugin's existing platform-enumeration pattern (the same shape used for blocking-question tools); never pin other vendors' model names in pass-through skill content — naming drifts faster than release cadence. Note: the converter does propagate `model:` params to all targets (see `best-practices/ce-pipeline-end-to-end-learnings.md`), so tier hints are not Claude-only decoration.
|
||||
- **Degradation rule**: when the platform's subagent primitive lacks per-agent model selection, dispatch everything on the inherited model and keep read budgets/output caps — cost control comes from structure, not tiering. Write this rule into the skill; it fired correctly in our eval (the harness had no nested dispatch).
|
||||
- **Architecture principle**: separate evidence-gathering (cheap extraction scouts producing quote+pointer dossiers) from ceiling reasoning (strong model only at choke points: ceiling framing, cross-cutting synthesis, final arbitration). This is cheaper *and* better grounded than a uniform fleet fed a thin summary.
|
||||
|
||||
### 5. Optimize context: extract only conditional/late content, and move bulk data to files
|
||||
|
||||
Two independent levers:
|
||||
|
||||
- **Reference extraction pays only for CONDITIONAL or LATE-SEQUENCE content.** Early unconditional content gains nothing — it would be read at start and carried anyway, plus a read round-trip. The test: *how many turns of other work happen before this content executes, and might it never execute?* ce-ideate's Phase 2 (~100 lines, ~22% of the file, runs after 5-8 turns of grounding) qualified; Phase 0 gating did not.
|
||||
- **Data flows usually dominate prose.** Measure both: 5 scouts × 150-line dossiers ≈ 10k tokens carried every subsequent turn if returned inline — more than the entire SKILL.md (~6k). Fix: subagents write outputs to scratch files (`/tmp/compound-engineering/<skill>/<run-id>/...`), return a 3-5-line gist; downstream agents receive paths and read the files themselves. This extends the established path-passing pattern (`skill-design/pass-paths-not-content-to-subagents.md`) with the gist refinement: the orchestrator keeps just enough orientation to route, never the bulk.
|
||||
|
||||
### 6. Load-stub design: make extracted references information-asymmetric
|
||||
|
||||
A soft pointer ("see references/X.md for details") gets skipped. When extracting load-bearing content, the inline stub must satisfy all five properties:
|
||||
|
||||
1. **Load-instruction-only** — no spec, no contract, nothing to improvise from. Converts "should load" into "cannot proceed without loading."
|
||||
2. **Names exactly what the reference contains** and states those details appear nowhere in the main body.
|
||||
3. **Names the failure mode of skipping** in the skill's own terms (e.g., "improvising produces unverifiable candidates — the precise failure this skill exists to prevent").
|
||||
4. **Closes inline-information leaks** — any number or detail that remains inline for other reasons gets explicitly disclaimed ("the fleet counts in Phase 0.6 are cost transparency, not the dispatch spec").
|
||||
5. **Pre-empts rationalizations** ("'Quickly' means smaller volume targets, not skipping the reference").
|
||||
|
||||
Defense in depth: anchor downstream phases on *different* reference files (rejection criteria, section contract) so a skipped load fails visibly, not silently.
|
||||
|
||||
This is the complement of `skill-design/post-menu-routing-belongs-inline.md`: inline the content when it is always-on; use an information-asymmetric stub when it is genuinely conditional or late-sequence but must load when its branch fires.
|
||||
|
||||
### 7. Eval verification: fresh subagent, mechanical transcript grading
|
||||
|
||||
- **Bypass the cache.** Plugin skill/agent definitions cache at session start; typed invocation tests the stale copy. Instead, spawn a fresh subagent told to read the skill source from disk and follow it.
|
||||
- **Grade from the transcript, not the self-report.** Parse the JSONL into a tool-call timeline and assert mechanically: Read-event ordering against generation checkpoints (e.g., the extracted-reference Read landing after scout writes and before the candidates checkpoint); zero orchestrator Reads of bulk data files; filesystem artifacts present with correct names and the full section contract.
|
||||
- **Know the harness limits and record them.** An eval subagent without nested dispatch *cannot* verify dispatch payload shape or fleet tiering — mark those assertions "not testable," don't fudge them. It *does* verify load ordering, file contracts, volume/format overrides, and degradation-rule behavior. Closing the gap requires a main-session run.
|
||||
- Errors encountered while following the skill during the eval are findings about the skill, not noise. (auto memory [claude])
|
||||
|
||||
### 8. Ceiling mechanics: explicitly request above-and-beyond behavior in dispatches
|
||||
|
||||
Floor-guarding (basis requirements, rejection criteria) prevents bad output; it does not produce ambitious output. From the best-practices doc:
|
||||
|
||||
- **Ambition charter**, included verbatim in every generation dispatch: intent framing (why the output matters), warm-up framing ("your first few ideas are warm-up; keep only those that earn their place after the non-obvious ones exist"), and an anti-genericness test ("if it would appear in a generic listicle, sharpen or drop").
|
||||
- **Fresh-context verifiers over self-critique** (per the Fable guide): the orchestrator grading its own synthesis is anchored; a verifier that never saw generation, prompted to *refute*, is not.
|
||||
- **Dispatch payload structure**: XML tags (`<grounding> <constraints> <background> <task>`); longform shared material first, task last (documented long-context gain); byte-identical shared prefix across parallel dispatches for prompt-cache reuse; constraint-vs-background made mechanical by tags rather than prose.
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Skills written for prior model generations accumulate two opposite debts simultaneously: too much JUDGMENT prescription (which the Fable guide warns actively degrades frontier-model output) and too little PROTOCOL infrastructure for cost, context, and verification. A naive "shorten it" pass cuts the wrong things; a naive "harden it" pass bloats the wrong things. The PROTOCOL/JUDGMENT split plus the ordered sequence resolves the tension: prune where the model is strong, prescribe where the workflow is mechanical.
|
||||
|
||||
Measured outcomes from the ce-ideate application:
|
||||
|
||||
- SKILL.md: 424 → 372 lines (-12%) while adding model tiering, file-based dossier flow, the information-asymmetric stub, and the ambition charter. ~16 lines recovered from three judgment-prescription cuts alone, with no behavior loss in the verification run.
|
||||
- Context math: inlined dossiers would have cost ~10k tokens carried every turn — more than the whole SKILL.md (~6k). The file+gist pattern removed that entirely from the orchestrator's window.
|
||||
- Eval: 6/8 mechanical assertions passed, 2 correctly reported untestable (nested-dispatch assertions in a dispatch-less harness), 0 failures. The degradation rule fired as designed. Degraded inline run: 14 minutes, 208k tokens.
|
||||
- Cost architecture: cheap extraction scouts feeding quote+pointer dossiers to ceiling-tier choke points was both cheaper and better grounded than a uniform inherited-model fleet fed a thin summary.
|
||||
|
||||
---
|
||||
|
||||
## When to Apply
|
||||
|
||||
Run this sequence when reviewing a skill that matches one or more of:
|
||||
|
||||
- **Multi-agent orchestration skills** — anything dispatching subagent fleets (the tiering, file-flow, and dispatch-payload steps only matter here).
|
||||
- **Skills over ~300 lines** — large enough that conditional/late-sequence extraction and judgment pruning have measurable payoff.
|
||||
- **Skills written before frontier models** (or before the current Fable guide) — likely over-prescribed on judgment, under-built on protocol.
|
||||
- **Skills inlining bulk data into dispatch prompts or return values** — any place subagent output re-enters the orchestrator's window as content rather than a path.
|
||||
- **Skills with soft "see reference" pointers guarding load-bearing content** — apply step 6 even without the rest.
|
||||
|
||||
Skip or scale down when: the skill is short and unconditional (extraction won't pay), or it plausibly runs on non-frontier models (the step-2 floor fails — keep the scaffolding). Always pair structural changes with the step-7 eval; never ship on the agent's self-report.
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
**1. Judgment enumeration → principle + one contrast pair**
|
||||
|
||||
Before (enumerated list of vague phrases plus a 7-row sample classification table):
|
||||
|
||||
```
|
||||
Vague subjects include: "quick wins", "low-hanging fruit", "improvements",
|
||||
"polish", "cleanup", "things to fix", ...
|
||||
[+ 7-row table classifying sample subjects as vague/identifiable]
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```
|
||||
A subject is workable when it names an identifiable referent:
|
||||
`browser sniff` is identifiable, `quick wins` is not — vagueness is
|
||||
about referent, not length.
|
||||
```
|
||||
|
||||
**2. Inline bulk data → file + gist**
|
||||
|
||||
Before (scout returns its full dossier; orchestrator carries it forever):
|
||||
|
||||
```
|
||||
Return your complete evidence dossier (~150 lines of quotes + pointers)
|
||||
in your final message.
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```
|
||||
Write your dossier to /tmp/compound-engineering/<skill>/<run-id>/evidence-<axis-slug>.md.
|
||||
Return only a 3-5 line gist plus the file path. Downstream agents read
|
||||
the file themselves; the orchestrator never does.
|
||||
```
|
||||
|
||||
**3. Soft pointer → information-asymmetric stub**
|
||||
|
||||
Before:
|
||||
|
||||
```
|
||||
Phase 2: Divergent ideation. See references/divergent-ideation.md for details
|
||||
on the fleet structure. Dispatch the agents and collect candidates.
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```
|
||||
Phase 2: Read references/divergent-ideation.md now. It contains the fleet
|
||||
spec, per-agent dispatch contract, and volume targets — none of which appear
|
||||
in this main body. Dispatch prompts cannot be correctly constructed without
|
||||
it; improvising them produces unverifiable candidates — the precise failure
|
||||
this skill exists to prevent. The fleet counts in Phase 0.6 are cost
|
||||
transparency, not the dispatch spec. "Quickly" means smaller volume targets,
|
||||
not skipping the reference.
|
||||
```
|
||||
|
||||
The before version leaves enough inline (phase name, "dispatch the agents") to improvise from; the after version makes proceeding without the read impossible, names the skip-failure, closes the leak, and pre-empts the "we're in a hurry" rationalization.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/pass-paths-not-content-to-subagents.md` — established precedent for path-passing to subagents; step 5 extends it with the gist refinement.
|
||||
- `docs/solutions/skill-design/post-menu-routing-belongs-inline.md` — the complementary lever for the same load-reliability failure: inline always-on content; load-stub conditional content (step 6).
|
||||
- `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines.md` — canonical example of PROTOCOL content that must keep full prescription (step 1).
|
||||
- `docs/solutions/skill-design/script-first-skill-architecture.md` — complementary token-optimization pattern (bundled scripts instead of model-context work).
|
||||
- `docs/solutions/skill-design/safe-auto-rubric-calibration.md` — earlier eval-methodology precedent (fixture-based grading, variance awareness) consistent with step 7.
|
||||
- `docs/solutions/skill-design/paired-old-vs-new-injection-skill-evals.md` — sharpens step 7's fresh-subagent grading into a controlled old-vs-new blind A/B that separates demonstrated improvement from no-regression.
|
||||
- `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings.md` — evidence that `model:` params propagate to all conversion targets (step 4).
|
||||
- Plugin `AGENTS.md` → Skill Design Principles — the prescription-calibration framework this methodology refines; and the conditional/late-sequence extraction rule step 5 operationalizes.
|
||||
- GitHub issues #714 and #374 — historical reference-load failures in the same family step 6 addresses.
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
title: "Git workflow skills need explicit state machines for branch, push, and PR state"
|
||||
category: skill-design
|
||||
date: 2026-03-27
|
||||
last_refreshed: 2026-07-12
|
||||
module: skills/ce-commit and ce-commit-push-pr
|
||||
problem_type: architecture_pattern
|
||||
component: tooling
|
||||
symptoms:
|
||||
- Detached HEAD could fall through to invalid push or PR paths
|
||||
- Untracked-only work could be misclassified as a clean working tree
|
||||
- PR detection could select the wrong PR or mis-handle the no-PR case
|
||||
- Default-branch flows could attempt invalid "open a PR from the default branch" behavior
|
||||
root_cause: missing_workflow_step
|
||||
resolution_type: workflow_improvement
|
||||
severity: high
|
||||
tags:
|
||||
- git-workflows
|
||||
- skill-design
|
||||
- state-machine
|
||||
- detached-head
|
||||
- gh-cli
|
||||
- pr-detection
|
||||
- default-branch
|
||||
---
|
||||
|
||||
# Git workflow skills need explicit state machines for branch, push, and PR state
|
||||
|
||||
## Problem
|
||||
|
||||
The `ce-commit` and `ce-commit-push-pr` skills had accumulated branch-state and PR-state bugs because they described Git flow in broad prose instead of modeling the workflow as a sequence of explicit state checks. Small wording changes kept introducing regressions around detached HEAD, untracked files, upstream detection, default-branch pushes, and PR lookup.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- `git push -u origin HEAD` could be reached from detached HEAD, where Git rejects the push because `HEAD` is not a branch ref
|
||||
- A repo with only untracked files could be treated as "nothing changed" because `git diff HEAD` is empty for untracked files
|
||||
- A no-PR branch could trigger an error path that looked like a fatal failure instead of an expected "no PR for this branch" state
|
||||
- `gh pr list --head "<branch>"` could match an unrelated PR from another fork with the same branch name
|
||||
- Clean-working-tree flows on the default branch could push default-branch commits and then try to open a PR from the default branch to itself
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
- Using a single early `git branch --show-current` result and referring back to it later. Once the workflow creates a branch, the earlier value is stale.
|
||||
- Using `git diff HEAD` as the definition of "has changes." It does not account for untracked files.
|
||||
- Treating every non-zero exit from a PR-detection command as a fatal failure. "No PR for this branch" is often a normal branch state.
|
||||
- Flip-flopping between `gh pr view` and `gh pr list` without writing down the tradeoff. `gh pr view` is current-branch-aware but exits 1 on the normal no-PR state; `gh pr list --head <branch>` has clean exit-0-`[]` = "no PR" semantics but filters by branch name only. The fix is not to pick one silently — it is to document the tradeoff and the branch-name-collision caveat (see §4) so the choice stops regressing.
|
||||
- Adding a "clean working tree" fast path before re-checking whether the current branch was still the default branch. That let the workflow skip the feature-branch safety gate and head straight toward invalid push/PR transitions.
|
||||
|
||||
## Solution
|
||||
|
||||
Treat the skill as a small state machine. For each transition, run the command that answers the next question directly, then branch on that result instead of carrying state forward in prose.
|
||||
|
||||
### 1. Use `git status` as the source of truth for working-tree cleanliness
|
||||
|
||||
Use the `git status` result from Step 1 to decide whether the tree is clean. This covers staged, modified, and untracked files.
|
||||
|
||||
```text
|
||||
Clean working tree:
|
||||
- no staged files
|
||||
- no modified files
|
||||
- no untracked files
|
||||
```
|
||||
|
||||
Do not use `git diff HEAD` as the cleanliness check.
|
||||
|
||||
### 2. Re-read branch state after every branch-changing transition
|
||||
|
||||
When the workflow starts in detached HEAD:
|
||||
|
||||
```bash
|
||||
git branch --show-current
|
||||
git checkout -b <branch-name>
|
||||
git branch --show-current
|
||||
```
|
||||
|
||||
In `ce-commit-push-pr`, create that branch automatically: the user invoked a commit/push/PR workflow, and later push/PR steps require a branch-backed ref. The second `git branch --show-current` is not redundant. It converts "the skill thinks it created branch X" into "Git says the current branch is X."
|
||||
|
||||
Apply the same pattern before default-branch safety checks:
|
||||
|
||||
```bash
|
||||
git branch --show-current
|
||||
```
|
||||
|
||||
Run it again at the moment the decision is needed. Do not rely on a branch value captured earlier in the workflow.
|
||||
|
||||
### 3. Split "upstream exists" from "there are unpushed commits"
|
||||
|
||||
Check upstream existence first:
|
||||
|
||||
```bash
|
||||
git rev-parse --abbrev-ref --symbolic-full-name @{u}
|
||||
```
|
||||
|
||||
Only if that succeeds, check for unpushed commits:
|
||||
|
||||
```bash
|
||||
git log <upstream>..HEAD --oneline
|
||||
```
|
||||
|
||||
This avoids conflating "no upstream configured yet" with "nothing to push."
|
||||
|
||||
### 4. Detect an existing PR with `gh pr list`, and read its exit status as state
|
||||
|
||||
For "does this branch already have a PR?" use a command that separates "no PR" from "lookup failed":
|
||||
|
||||
```bash
|
||||
gh pr list --head <branch> --state open --json number,url,title,body,state,headRefName,headRepositoryOwner
|
||||
```
|
||||
|
||||
Interpret the result as a state check:
|
||||
|
||||
- Exit 0 with `[]` -> no open PR for this branch (proceed to creation)
|
||||
- Exit 0 with entries -> a PR exists; pick the entry whose `headRepositoryOwner`/`headRefName` match the current head, not index 0. If several entries share the branch name from different owners and none is confirmably yours, treat it as ambiguous and stop rather than act on someone else's PR
|
||||
- Non-zero exit -> `gh` is missing, unauthenticated, or offline: PR state is **unknown**, never "no PR". Resolve auth/connectivity before creating, so a lookup failure cannot cause a duplicate PR.
|
||||
|
||||
Pass the branch **name only** — `gh pr list --head` does not accept `<owner>:<branch>` syntax (it silently returns `[]` for it, which reads as "no PR" and opens a duplicate). On a fork checkout the PR lives on the base repo, so target the base via `gh`'s default-repo resolution or `-R <base-owner>/<repo>`. Skip this check entirely on detached HEAD (empty branch): `gh pr list` with an empty `--head` drops the filter and lists unrelated PRs.
|
||||
|
||||
**Known tradeoff (branch-name collision).** `gh pr list --head <branch>` filters by head-branch *name*, so in a busy multi-fork repo two open PRs from different owners can share a branch name, and this command cannot disambiguate them the way `gh pr view` (current-branch-aware) can. It is chosen anyway for its clean exit semantics: `gh pr view` exits 1 on the normal no-PR state, conflating "no PR" with a real failure — harder to interpret, and fatal if the check ever runs at skill *load* time (see [no-load-time-pre-resolution-for-fallible-context.md](no-load-time-pre-resolution-for-fallible-context.md)). Bound the collision by re-verifying immediately before `gh pr create` and inspecting the returned entry rather than assuming the first match is yours.
|
||||
|
||||
### 5. Keep the default-branch safety gate ahead of push/PR transitions
|
||||
|
||||
If the current branch is `main`, `master`, or the resolved default branch, and the workflow is about to push or create a PR:
|
||||
|
||||
- create a feature branch first and re-read the branch name
|
||||
- ask only when unpushed local commits create a real carry-forward decision, such as preserving them on the feature branch vs starting from the fresh remote base
|
||||
- if the safe branch transition cannot be completed in `ce-commit-push-pr`, stop rather than trying to open a PR from the default branch
|
||||
|
||||
This prevents "push default branch, then attempt impossible PR flow" behavior.
|
||||
|
||||
## Why This Works
|
||||
|
||||
Git workflows look linear in prose but are actually stateful. Detached HEAD, missing upstreams, untracked files, and existing-vs-missing PRs are all separate dimensions of state. The bug pattern was always the same: the skill would observe one dimension once, then assume it remained true after a later transition.
|
||||
|
||||
The fix is not more prose. The fix is explicit re-checks at each transition boundary:
|
||||
|
||||
- branch state after branch creation
|
||||
- cleanliness from `git status`, not a partial diff
|
||||
- upstream existence before unpushed-commit checks
|
||||
- PR existence via `gh pr list --head <branch>`, reading exit-0-`[]` = none vs non-zero = unknown (with the branch-name-collision caveat in §4)
|
||||
- default-branch safety before any push/PR transition
|
||||
|
||||
This turns a brittle narrative into a deterministic control flow with a small number of clear state transitions.
|
||||
|
||||
## Edge Cases We Hit While Fixing This
|
||||
|
||||
These were not hypothetical concerns. Each one showed up while revising `ce-commit` and `ce-commit-push-pr`, and several "fixes" introduced a new bug one step later in the flow.
|
||||
|
||||
### 1. Detached HEAD can reappear as a later bug even after it seems "handled"
|
||||
|
||||
An early version only guarded detached HEAD in the PR-detection step. That looked fine until the workflow added a "clean working tree" shortcut before PR detection. In detached HEAD with committed local work, that shortcut could jump directly to push logic and hit:
|
||||
|
||||
```bash
|
||||
git push -u origin HEAD
|
||||
```
|
||||
|
||||
which fails because detached HEAD is not a branch ref.
|
||||
|
||||
Learning: detached HEAD must be handled before any later shortcut can skip around it.
|
||||
|
||||
### 2. Creating a branch is not enough; the skill must re-read which branch Git says is current
|
||||
|
||||
Another revision created a branch from detached HEAD but still described later steps as using "the branch name from Step 1." If Step 1 originally ran in detached HEAD, that earlier branch value was empty. Later PR detection could still use the stale empty value.
|
||||
|
||||
Learning: after `git checkout -b <branch-name>`, run `git branch --show-current` again and treat that output as the only trusted branch name.
|
||||
|
||||
### 3. Bare branch-name PR lookup fixed one problem and created another
|
||||
|
||||
We switched from `gh pr view` to:
|
||||
|
||||
```bash
|
||||
gh pr list --head "<branch>" --json url,title,state --jq '.[0] // empty'
|
||||
```
|
||||
|
||||
because `gh pr view` was surfacing a non-zero exit when no PR existed. That improved the no-PR path, but it introduced a correctness problem: `gh pr list --head` matches on branch name only, and GitHub CLI does not support `<owner>:<branch>` syntax for that flag. In a multi-fork repo, another person's PR can reuse the same branch name.
|
||||
|
||||
Learning: this is a genuine tradeoff, not a settled winner. The skills ultimately kept `gh pr list --head <branch> --state open` for its clean exit semantics (exit-0-`[]` = no PR vs non-zero = unknown) — which also matters because `gh pr view`'s exit-1-on-no-PR is fatal if the check ever runs at skill *load* time — and bound the branch-name collision by targeting the base repo and re-verifying before `gh pr create` (see §4). Whatever you pick, write the tradeoff down so it stops regressing.
|
||||
|
||||
### 4. "No PR" is not an error in the workflow, even if the CLI exits non-zero
|
||||
|
||||
The original reason for changing away from `gh pr view` was that a branch with no PR looked like a command failure. But for this workflow, "no PR yet" is often the expected state and should lead to creation logic, not stop the skill.
|
||||
|
||||
Learning: document expected non-zero exits as state transitions, not generic failures.
|
||||
|
||||
### 5. `git diff HEAD` misses one of the most common commit cases: untracked files
|
||||
|
||||
At one point the skill used `git diff HEAD` to decide whether work existed. In a repo with only a newly created file, `git diff HEAD` is empty even though `git status` shows `?? file`.
|
||||
|
||||
Learning: untracked-only work is a first-class case. Use `git status` as the cleanliness check.
|
||||
|
||||
### 6. "No upstream" and "nothing to push" are different states
|
||||
|
||||
An early shortcut treated an error from `git log @{u}..HEAD` as "nothing to push." That is wrong on a new feature branch with local commits but no upstream yet. The branch still needs its first push.
|
||||
|
||||
Learning: first check whether an upstream exists, then check whether there are unpushed commits.
|
||||
|
||||
### 7. Default-branch safety can be bypassed by a convenience shortcut
|
||||
|
||||
Another revision added a clean-working-tree shortcut that said "if there are unpushed commits, skip commit and continue to push." That worked on feature branches but accidentally skipped the normal "don't work directly on main/default branch" safety gate. The result was: push default-branch commits, then head toward PR creation.
|
||||
|
||||
Learning: every path that can lead to push or PR creation must pass through a default-branch safety check.
|
||||
|
||||
### 8. Declining feature-branch creation on the default branch must stop the PR workflow
|
||||
|
||||
One fix asked the user whether to create a feature branch first when clean-tree logic found unpushed default-branch commits. But if the user declined, the workflow still continued to push and then attempt PR creation. That leads to an impossible "open a PR from the default branch to itself" situation.
|
||||
|
||||
Learning: in `ce-commit-push-pr`, declining feature-branch creation on the default branch is a stop condition, not a continue condition.
|
||||
|
||||
### 9. Clean-working-tree shortcuts interact with branch safety, PR state, and upstream state all at once
|
||||
|
||||
The hardest bugs came from the "no local edits, but there may still be work to do" path. That single branch of logic had to answer all of these:
|
||||
|
||||
- Is the current branch detached?
|
||||
- Is the current branch the default branch?
|
||||
- Does the branch have an upstream?
|
||||
- Are there unpushed commits?
|
||||
- Does a PR already exist?
|
||||
|
||||
Missing any one of those checks produced a new bug.
|
||||
|
||||
Learning: clean-working-tree shortcuts are the highest-risk part of Git workflow skills because they combine the most state dimensions at once.
|
||||
|
||||
### 10. Git workflow skills are unusually prone to whack-a-mole regressions
|
||||
|
||||
The meta-pattern across all these fixes was:
|
||||
|
||||
1. Improve one failure mode
|
||||
2. Reveal that another state transition was only implicitly modeled
|
||||
3. Add a new branch in the prose
|
||||
4. Discover that the new branch skipped a previously safe checkpoint
|
||||
|
||||
Learning: these skills should be designed and reviewed like tiny state machines, not as narrative instructions. Any change to one state transition should trigger a walkthrough of all adjacent states before considering the skill fixed.
|
||||
|
||||
## Prevention
|
||||
|
||||
- For Git/GitHub skills, treat workflow design as a state machine, not as a linear checklist.
|
||||
- Re-run the command that answers the current question at the point of decision. Do not rely on values gathered earlier if a mutating command may have changed them.
|
||||
- Use `git status` for "is there local work?" and reserve `git diff` for describing content, not determining whether work exists.
|
||||
- Model expected non-zero CLI exits explicitly when they represent state, such as `gh pr view` on a branch with no PR.
|
||||
- When a tool visually highlights non-zero exits as failures, capture the exit code yourself for expected state probes so correct logic does not still look broken to the user.
|
||||
- Know the PR-detection tradeoff and document your choice. `gh pr list --head <branch>` has clean exit semantics (exit-0-`[]` = no PR, non-zero = unknown) but filters by branch name only; `gh pr view` is current-branch-aware but exits 1 on the normal no-PR state (and is fatal at skill load time). The current skills use `gh pr list` — base-repo-targeted, branch name only — and bound the multi-fork branch-name collision by re-verifying before `gh pr create`.
|
||||
- Keep default-branch safety checks in every path that can lead to push or PR creation, including "clean working tree but unpushed commits" shortcuts.
|
||||
- When editing skill logic, manually walk these cases before considering the change complete:
|
||||
- detached HEAD with uncommitted changes
|
||||
- detached HEAD with committed but unpushed work
|
||||
- untracked-only files
|
||||
- feature branch with no upstream
|
||||
- feature branch with upstream and no PR
|
||||
- feature branch with upstream and an existing PR
|
||||
- default branch with unpushed commits
|
||||
- non-`main` default branch names such as `develop` or `trunk`
|
||||
|
||||
## Related Issues
|
||||
|
||||
- [no-load-time-pre-resolution-for-fallible-context.md](no-load-time-pre-resolution-for-fallible-context.md) — why the PR check moved to `gh pr list` and out of load-time pre-resolution; the source of the current §4 decision.
|
||||
- [script-first-skill-architecture.md](script-first-skill-architecture.md)
|
||||
- [pass-paths-not-content-to-subagents.md](pass-paths-not-content-to-subagents.md)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Never use load-time `!`cmd`` pre-resolution in SKILL.md for fallible or context commands — gather at runtime with shell-neutral argv calls
|
||||
date: 2026-07-12
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: best_practice
|
||||
component: development_workflow
|
||||
severity: high
|
||||
applies_when:
|
||||
- Authoring or reviewing a skill that gathers git/gh or other environment context inside SKILL.md
|
||||
- Tempted to use Claude Code's `!`cmd`` load-time pre-resolution to inline command output into skill text
|
||||
- A skill must work across harnesses (Claude Code, Codex, Cursor, Gemini, Grok) and shells (POSIX sh and Windows PowerShell 5.1)
|
||||
- Gathering a value whose non-zero exit is a normal expected state (no PR yet, no origin/HEAD, detached HEAD, unborn repo)
|
||||
- Deciding whether a command belongs in load-time pre-resolution versus runtime control flow
|
||||
tags:
|
||||
- skill-authoring
|
||||
- cross-harness
|
||||
- cross-shell
|
||||
- claude-only-mechanism
|
||||
- load-time-resolution
|
||||
- runtime-gather
|
||||
- powershell
|
||||
- argv-commands
|
||||
- portability
|
||||
---
|
||||
|
||||
# Don't pre-resolve fallible context with Claude-only `!` load-time commands — gather it at runtime as shell-neutral argv calls
|
||||
|
||||
## Context
|
||||
|
||||
Claude Code's `SKILL.md` format supports **load-time pre-resolution**: a line containing `` !`cmd` `` runs `cmd` when the skill *loads* and inlines the command's stdout into the loaded skill text before the model ever reads it. It looks like a free way to hand the agent pre-computed context — repo root, default branch, current branch — with zero runtime tool calls. Across this plugin it was used exactly that way: **16 lines in 8 skills, every one a git context command** — `git rev-parse --show-toplevel` (×6), `git rev-parse --abbrev-ref` (×2), and `git status` / `git diff HEAD` / `git branch --show-current` / `git log` (×2 each, in the two commit skills). A separate fallible command, `gh pr view` for existing-PR detection, sat in `ce-commit-push-pr`'s runtime *fallback block* rather than a `!` line at the time of this change — but it exits 1 on the normal no-PR state for the same reason, and a `!`gh pr view`` line was the original trigger that started this whole effort.
|
||||
|
||||
Two hard facts make the construct unsafe for that use:
|
||||
|
||||
1. **Claude-Code-only.** On Codex, Cursor, Gemini, and Grok the `` !`cmd` `` line is inert literal text — the command never runs, so any skill that depends on the inlined value is already broken off-Claude. This is the same class of Claude-only-in-skill-body mechanism as `$ARGUMENTS` (see the sibling learning cross-referenced below).
|
||||
2. **On Claude Code, a non-zero exit ABORTS skill load** with a user-facing error. The skill does not degrade — it fails to open.
|
||||
|
||||
The trap is the intersection: for git/gh context, a **non-zero exit is the normal state**, not an error. No PR yet, no `origin/HEAD` set, detached HEAD, an unborn repo, not a git repo, or a missing/unauthenticated `gh` all exit non-zero. The archetype is a branch with no PR: an existing-PR check via `gh pr view` exits 1 on that ordinary pre-create state — fatal inside a `!` line (where it originally lived — the bug that started this effort) and misleading even at runtime, since exit 1 conflates "no PR" with a real failure.
|
||||
|
||||
## Guidance
|
||||
|
||||
**Never place a command whose non-zero exit is a *normal* state into a `` !`` `` pre-resolution line. In this plugin, ban `` !`cmd` `` pre-resolution for context-gathering entirely.** The right distinction is **control-flow vs. precondition**: a command whose failure the agent should *interpret* (branch it on) is control flow and must be gathered at runtime; only a genuine hard precondition could ever justify aborting, and none of the git/gh context here is one.
|
||||
|
||||
Gather context at **runtime** instead, as shell-neutral **argv-style commands** — the program and its arguments only, **one per tool call**. Do not join them with `;`, `&&`, `||`, pipes, `$(...)`, or redirects like `2>/dev/null`. A lone `git …` / `gh …` invocation is a single external-program call with no shell operators, so it parses **identically** under POSIX sh and Windows PowerShell 5.1 — that is what makes it dual-shell safe. The agent reads each command's **exit status as data** (a non-zero exit is a state to interpret: no PR, no `origin/HEAD`, detached HEAD), not as a load-time abort.
|
||||
|
||||
**Existing-PR detection: use `gh pr list`, not `gh pr view`.**
|
||||
- `gh pr view` exits 1 when the branch has no PR — the normal pre-create state — which conflates "no PR" with a real failure.
|
||||
- `gh pr list --head <branch> --state open --json number,url,title,body,state,headRefName,headRepositoryOwner` exits **0** and returns `[]` when there is no PR. Only an exit-0 `[]` means "no open PR." A **non-zero** exit means `gh` is missing, unauthenticated, or offline — treat PR state as **unknown**, never as "none."
|
||||
- **`--head` gotcha:** `gh pr list --head` does **not** accept `<owner>:<branch>` syntax (confirmed on gh 2.96.0: the flag help reads `Filter by head branch ("<owner>:<branch>" syntax not supported)`). Passing `owner:branch` silently returns `[]` → a false "no PR" → a **duplicate PR**. Pass the **branch name only**. Fork PRs live on the *base* repo, so target it via `gh`'s default-repo resolution or `-R <base-owner>/<repo>`.
|
||||
- **Select the returned entry by head owner, not index 0.** Since `--head` filters by branch *name* only, a base repo with multiple forks can return several entries sharing the branch name — pick the one whose `headRepositoryOwner`/`headRefName` match the current head (which is why those fields are in the `--json`), and stop as ambiguous rather than assume the first match is yours.
|
||||
- **Empty branch (detached HEAD):** skip the PR check entirely — `gh pr list` with an empty `--head` drops the filter and lists *unrelated* PRs.
|
||||
|
||||
**Demote every gathered value to a stale hint.** Context gathered up front is a snapshot; re-verify immediately before any consequential action: re-read the current branch before `git push`, and **always** re-run the existing-PR check before `gh pr create` (not only when the first check came back unknown), since a PR can appear between gather and create.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
This is a genuine dilemma with **no solution inside the `` !`` `` line**:
|
||||
|
||||
- The **guarded** form `` !`… 2>/dev/null || echo SENTINEL` `` exits 0 on macOS/Linux but **fails to parse under Windows PowerShell 5.1**: PowerShell has no `||`/`&&`, `/dev/null` resolves to a literal path (`D:\dev\null`), and there is no `true`. That broke skill *load* on PowerShell — issue #1066.
|
||||
- The **bare** form parses everywhere but aborts load on any legitimate non-zero exit.
|
||||
|
||||
There is no single command string that both (a) exits 0 on the expected-failure states **and** (b) parses under both POSIX sh and PowerShell 5.1. You cannot win inside the `!` line.
|
||||
|
||||
The history is a recurring-footgun saga. Claude's permission checker forced ever-narrower guard shapes over many iterations — `case`/`esac` (#699); `[A] && B || C` "ambiguous syntax" (#701/#710); nested `$()` quoted strings (#709); `;` "Unhandled node type", pipes, parameter expansion (#758/#934). Then PR #1078 **stripped** the `2>/dev/null || echo` guards across 8 skills to fix the PowerShell load break (#1066) — which reintroduced the **bare-form load-abort**, and the very first fallible bare command (`gh pr view` on a branch with no PR) aborted load again. This branch's fix (unmerged at capture) ends the cycle by removing the construct rather than chasing a portable guard that provably does not exist.
|
||||
|
||||
The cost is not a wrong value — it is the **skill failing to load at all** for the user, with a user-facing error, on the ordinary path. And the runtime replacement must itself stay shell-neutral: a POSIX-only *runtime* gather (a fenced `2>/dev/null || echo` block, or a compound table command) merely moves the same #1066 PowerShell break from load time to mid-skill.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Authoring or reviewing **any skill distributed across harnesses** (Claude Code, Codex, Cursor, Gemini, Grok) that reaches for `` !`cmd` `` load-time pre-resolution, or for a shell context-gather that uses compound operators.
|
||||
- **Especially git/gh state** — repo root, default branch, whether a PR exists — where non-zero exit is the *normal* state, so pre-resolution aborts on the common path.
|
||||
- Generalize the rule: prefer describing the **capability** to gather at runtime over baking a Claude-only, fail-closed pre-resolution into the skill body.
|
||||
|
||||
## Examples
|
||||
|
||||
**Removed `` !`` `` pre-resolution → runtime resolution.** The simple repo-root consumers no longer inline the value at load; they resolve it at runtime and only when needed:
|
||||
|
||||
```text
|
||||
# Before (in a skill body — aborts load when not in a git repo)
|
||||
Repo root: !`git rev-parse --show-toplevel`
|
||||
|
||||
# After (ce-commit-push-pr, Step 4 concept-teaching gate)
|
||||
Use the repo root gathered in Context (resolving it with
|
||||
`git rev-parse --show-toplevel` if you don't already have it) …
|
||||
```
|
||||
|
||||
**Removed POSIX fallback block → argv-style command table.** The two commit skills replaced a single fenced POSIX bash gather (with `2>/dev/null`, `||`, `;`, `$()`) with a table of single argv commands, run one per tool call, each exit status read as control flow. From `ce-commit-push-pr`'s `## Context`:
|
||||
|
||||
```text
|
||||
| Command | Purpose | Non-zero exit / empty output means |
|
||||
| --- | --- | --- |
|
||||
| `git rev-parse --show-toplevel` | Repo root | Not a git repository — report and stop |
|
||||
| `git branch --show-current` | Current branch | Empty = detached HEAD |
|
||||
| `git rev-parse --abbrev-ref origin/HEAD` | Remote default | No origin/HEAD set — resolve per Step 1 |
|
||||
| `gh pr list --head <branch> --state open --json number,url,title,body,state,headRefName,headRepositoryOwner`
|
||||
| Open PR for this branch | Exit 0 with `[]` = no open PR; non-zero = unknown, never "no PR" |
|
||||
```
|
||||
|
||||
with the load-bearing instruction above it: *"run each command below as its **own** shell tool call … Do **not** join them with `;`, `&&`, `||`, pipes, `$(...)`, or redirects like `2>/dev/null`."*
|
||||
|
||||
**`gh pr view` → `gh pr list`.** Existing-PR detection switched from a command that exits 1 on the normal state to one that exits 0 and returns `[]`:
|
||||
|
||||
```text
|
||||
# Before — exits 1 when the branch has no PR (the normal pre-create state)
|
||||
gh pr view
|
||||
|
||||
# After — exits 0, returns [] when there is no PR; branch name only (no owner:branch);
|
||||
# select the entry by head owner, not index 0
|
||||
gh pr list --head <branch> --state open --json number,url,title,body,state,headRefName,headRepositoryOwner
|
||||
```
|
||||
|
||||
**Test enforcement.** `tests/skill-shell-safety.test.ts` was rewritten from per-pattern "bare-form" checks into (a) a **total ban** on `` !`cmd` `` in any skill file, (b) a per-command **load-abort catalog** naming each historical command and the state that would abort load, and (c) an **argv-only regression guard** on the two commit skills' `## Context` sections — no fenced shell block, no compound operators — so a POSIX-only runtime gather can't quietly reintroduce the #1066 break at runtime. `AGENTS.md` was updated to stop recommending `!` pre-resolution and document the runtime-gather pattern. `bun test` → 1930 pass, 0 fail; a Grok cross-model review (SHIP-WITH-FIXES) caught the `--head owner:branch` bug and 6 other findings, all applied.
|
||||
|
||||
**Sibling learnings:**
|
||||
- `docs/solutions/skill-design/arguments-token-is-claude-only-in-skill-bodies.md` — `$ARGUMENTS` is the same class of Claude-only-in-skill-body mechanism (reliably substituted only on Claude Code). The difference in failure mode is instructive: an unsubstituted `$ARGUMENTS` fails *loud and recoverable* (the agent sees a literal token and can route around it), whereas a fallible `` !`cmd` `` fails *closed* — the skill never loads.
|
||||
- `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines.md` — models the same two commit skills as state machines; its PR-detection transition (§4) uses the `gh pr list` decision established here, and its "gh pr view vs gh pr list" tradeoff was reconciled to match this learning.
|
||||
@@ -0,0 +1,163 @@
|
||||
---
|
||||
title: "Prove a skill prose change moved behavior with paired old-vs-new blind injection"
|
||||
date: 2026-07-01
|
||||
category: skill-design
|
||||
module: compound-engineering-plugin skill evaluation
|
||||
problem_type: design_pattern
|
||||
component: testing_framework
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Validating a prose/behavior edit to a SKILL.md, agent persona, or prompt
|
||||
- Needing to tell "demonstrated improvement" apart from "no regression"
|
||||
- One skill's output is consumed or gated by another skill or a test
|
||||
- Adding or renaming a field in a cross-skill output contract
|
||||
related_components:
|
||||
- tooling
|
||||
- development_workflow
|
||||
tags:
|
||||
- skill-evals
|
||||
- paired-injection
|
||||
- behavior-verification
|
||||
- cross-skill-contract
|
||||
- anti-drift-test
|
||||
- frontier-model
|
||||
- named-fields
|
||||
---
|
||||
|
||||
# Prove a skill prose change moved behavior with paired old-vs-new blind injection
|
||||
|
||||
## Context
|
||||
|
||||
Editing the prose of an agent skill (a `SKILL.md`, an embedded persona, or any behavior-shaping prompt) is cheap; *knowing whether the edit actually changed agent behavior* is not. Prose changes look self-evidently good in a diff, so they ship on the author's intuition. Two failure modes hide in that gap:
|
||||
|
||||
- **The change is a no-op at the current model tier.** A frontier model may already do the "new" thing by default, so the added rule buys nothing you can observe — you shipped tokens, not behavior.
|
||||
- **The change creates a cross-skill contract that no test enforces.** One skill's output is supposed to be gated by another skill (or a test), but the two ends can drift independently and every existing test stays green.
|
||||
|
||||
This methodology came out of validating a set of prose edits that added a *behavior-verification evidence* contract across four orchestration skills (`ce-work`, `ce-debug`, `ce-plan`, `lfg`): one producer skill emits verification evidence, and a downstream orchestrator gates shipping on that evidence. The goal was to prove — not assume — that the edits changed behavior, and to leave behind a test that fails when the contract is broken.
|
||||
|
||||
## Guidance
|
||||
|
||||
Validate behavior-changing prose edits with four complementary techniques. The first two establish *whether* the edit does anything; the last two make a *cross-skill* contract durable.
|
||||
|
||||
### 1. Paired old-vs-new blind injection
|
||||
|
||||
Extract the **actual** pre-change excerpt of the changed section from `git HEAD~1` and the post-change excerpt from the working tree — the real bytes, not a paraphrase. Dispatch two subagents:
|
||||
|
||||
- one seeded with the OLD excerpt, one with the NEW excerpt;
|
||||
- **both blind** to which version they hold and blind to the expected answer;
|
||||
- **identical** realistic scenario given to each;
|
||||
- each must return a **concrete decision plus an artifact** — the ordered steps it would take, the structured return object it would emit, the next pipeline action it would invoke — not a vague opinion.
|
||||
|
||||
Then compare the two artifacts.
|
||||
|
||||
- If both produce the **same correct** decision, you have proven **no-regression only** — the new prose did not break the behavior. That is a real result, but it is not evidence of improvement.
|
||||
- To claim **improvement**, you need a scenario where the OLD excerpt *fails* and the NEW excerpt *succeeds*. Design for that discriminating case explicitly; if you cannot find one, the honest conclusion is that the change is not an improvement at this tier (see technique 2).
|
||||
|
||||
Complement the paired runs with **new-only restraint negatives**: scenarios that prove the new rule does *not* over-fire. If you add a rule "emit an execution note when X," run a plain case with no X and confirm the new prose still omits the note; if you tighten a testing gate, run a pure-config task and confirm it still takes the legitimate no-test exception. A rule that fires on everything is as broken as one that fires on nothing.
|
||||
|
||||
### 2. Read a non-discriminating result honestly
|
||||
|
||||
At a capable model tier, many prose rules buy **determinism, weaker-model insurance, and run-to-run variance reduction** — not a behavior flip. When a paired eval comes out non-discriminating (old and new both do the right thing, and the old-prose agent even *justifies* the right thing with its own reasoning), do not relabel it as proof of improvement. Record it as: "already emergent at this tier; the rule locks in determinism and protects weaker models." Separate the changes that genuinely discriminate from the ones that only harden — both are worth shipping, but only the former is a behavior change you can demonstrate.
|
||||
|
||||
### 3. Standardize the field NAME, not just the information
|
||||
|
||||
When one skill's output must be consumed or gated by another skill or a test, the value of the edit is giving the consumer a **stable token to match on**. A capable producer already surfaces the relevant information — but as improvised free-text under an inconsistent name, which nothing downstream can key on. Change the contract to emit a **named field with named subfields**, and write the consumer's gate against that name. "Prompt the producer to consider X" is not testable; "the producer emits `X` and the consumer requires `X` when a condition holds" is mechanically checkable.
|
||||
|
||||
### 4. Prove the parity test fails on one-sided drift
|
||||
|
||||
Prose-presence tests (`SKILL.md` `.toContain("field_name")`) guard each skill **in isolation** — both stay green if the producer renames the field but the consumer's gate is not updated, or vice versa. To guard the *contract*, write a structural **parity test** that:
|
||||
|
||||
- **scopes** its assertions to the *owning section* of each file (the producer's return block, the consumer's gate block) via string-slice anchors, not "anywhere in the file" — an unscoped match passes on an incidental mention elsewhere; and
|
||||
- **cross-checks a shared facts map** so both ends are asserted to name the *same* facts (the producer's backtick token `existing_tests_inspected` matched against the consumer's prose "existing tests inspected").
|
||||
|
||||
Then do the step that is almost always skipped: **inject one-sided drift and watch the test fail.** Rename the field on the producer side only, run the parity test, confirm it goes red, and restore. A parity test you have never seen fail on injected drift is not known to work — it may be asserting something trivially true.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Prose diffs are persuasive and unfalsifiable by inspection.** Reading a "better" rule tells you nothing about whether the model's output changes. Blind paired injection is the cheapest way to convert a hunch into evidence.
|
||||
- **Blindness removes the two biggest confounds.** An agent told "you have the new, improved version" will rationalize a better answer; an agent told the expected answer will pattern-match to it. Withholding both makes the decision reflect the prose, not the framing.
|
||||
- **Honest non-discriminating results prevent overclaiming.** Shipping a rule as a "behavior fix" when it is really weaker-model insurance pollutes the record and inflates confidence in prose as a behavior lever.
|
||||
- **Named fields are the unit of cross-skill testability.** Gates and tests key on tokens. Without a stable name, the contract lives only in the model's judgment and silently rots.
|
||||
- **Parity tests are a common false comfort.** A test that has never been observed to fail may be asserting a tautology. Injected-drift verification is what separates a real guard from decoration.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Apply the full methodology when:
|
||||
|
||||
- The edit changes the **prose/behavior** of a skill, agent persona, or prompt (not mechanical code that a normal unit test already exercises).
|
||||
- The change is intended to **flip a decision** the agent makes, or to **add/rename a field** in an output contract.
|
||||
- The output of one skill is **consumed or gated by another skill or a test** (apply techniques 3 and 4).
|
||||
|
||||
Scale down when:
|
||||
|
||||
- The change is a **pure no-op cleanup** (typo, formatting) — no eval needed.
|
||||
- The change only touches a **single skill in isolation** with no downstream consumer — techniques 1 and 2 suffice; skip the parity test.
|
||||
|
||||
Note on tooling: use a skill-authoring/eval harness that **injects the excerpt into a fresh subagent's prompt at dispatch time**, so each run reads the current source. Do not iterate by dispatching the already-loaded plugin skill in the same session — cached skill definitions run pre-edit content.
|
||||
|
||||
## Examples
|
||||
|
||||
### Paired-injection dispatch shape
|
||||
|
||||
```
|
||||
# Extract the real bytes of the changed section, both sides.
|
||||
git show HEAD~1:skills/ce-work/SKILL.md # -> OLD excerpt of the Return-to-Caller block
|
||||
# working tree # -> NEW excerpt of the same block
|
||||
|
||||
Subagent A (blind): [OLD excerpt] + [identical scenario] -> "return the structured object you would emit"
|
||||
Subagent B (blind): [NEW excerpt] + [identical scenario] -> "return the structured object you would emit"
|
||||
|
||||
Neither subagent is told which version it holds or what the expected answer is.
|
||||
```
|
||||
|
||||
Reading the result:
|
||||
|
||||
- Both emit the same correct object -> NO-REGRESSION proven. Not improvement.
|
||||
- Old fails / new succeeds -> IMPROVEMENT proven (this is the case you must design for).
|
||||
- Old already succeeds on its own reasoning -> NON-DISCRIMINATING: rule buys determinism / weaker-model insurance, not a behavior flip. Say so.
|
||||
|
||||
Restraint negative (new-only): give the NEW excerpt a plain task with no triggering
|
||||
condition and confirm it does NOT emit the new field / note (proves the rule doesn't over-fire).
|
||||
|
||||
### Discriminating vs hardening, from one real change set
|
||||
|
||||
- **Discriminating (behavior flipped):** old prose let the pipeline ship after a producer returned with no evidence field ("gate satisfied -> proceed to simplify/ship"); new prose retried the producer once, then STOPPED BLOCKED. Old vs new produced different next pipeline actions -> genuine improvement.
|
||||
- **Hardening only (non-discriminating):** an "update the stale test in place, do not add a duplicate" guardrail — both old- and new-prose agents chose update-in-place, the old one reasoning "two contradictory assertions can't both be green." Ship it for determinism, but do not call it a behavior fix.
|
||||
|
||||
### Parity test with injected-drift proof
|
||||
|
||||
```ts
|
||||
// Scope to the OWNING section, then cross-check a shared facts map.
|
||||
const EVIDENCE_FACTS = {
|
||||
existing_tests_inspected: "existing tests inspected", // producer token -> consumer prose
|
||||
tests_added_or_changed: "tests added/changed",
|
||||
behavior_changed: "behavior_change: true",
|
||||
};
|
||||
|
||||
const producerBlock = slice(ceWorkSrc, "## Return-to-Caller Mode", "Engine selection ("); // owning section
|
||||
const consumerGate = slice(lfgSrc, "2. Invoke the `ce-work`", "3. Invoke the `ce-simplify-code`");
|
||||
|
||||
for (const [token, prose] of Object.entries(EVIDENCE_FACTS)) {
|
||||
expect(producerBlock).toContain(token); // producer names the field
|
||||
expect(consumerGate).toContain(prose); // consumer gate names the same fact
|
||||
}
|
||||
```
|
||||
|
||||
Proving it works (the step people skip):
|
||||
|
||||
```
|
||||
1. Rename `existing_tests_inspected` -> `existing_tests_reviewed` in the PRODUCER only.
|
||||
2. Run the parity test -> MUST go red (consumer still says "existing tests inspected").
|
||||
3. Restore the original name.
|
||||
```
|
||||
|
||||
If step 2 does not turn the test red, the parity test is not actually guarding the contract — fix the scoping or the facts map before trusting it.
|
||||
|
||||
## Related
|
||||
|
||||
- [fake-cli-harness-for-skill-judgment-evals](fake-cli-harness-for-skill-judgment-evals.md) — sibling method: new-vs-old skill comparison over a *discriminating* fixture. Same "an easy fixture proves nothing" insight; different mechanism (mocks a CLI boundary rather than injecting SKILL.md excerpts into blind subagents).
|
||||
- [frontier-model-skill-modernization-methodology](frontier-model-skill-modernization-methodology.md) — parent eval methodology; its "fresh subagent, bypass the plugin cache, mechanical transcript grading" step is what paired old-vs-new injection refines into a controlled A/B.
|
||||
- [safe-auto-rubric-calibration](safe-auto-rubric-calibration.md) — prior art for technique 2: frames a shipped prose change as "mostly a determinism patch, not a rate increase," and argues for measuring variance, not just outcome shift.
|
||||
- [cross-skill-shared-cache-primitive](cross-skill-shared-cache-primitive.md) — origin of the field-name contract and the parity-drift gap: renaming a schema field can pass a byte-identity parity test yet silently break per-skill consumers. Technique 4 is the mitigation.
|
||||
- [ce-doc-review-calibration-patterns](ce-doc-review-calibration-patterns.md) — reinforces technique 2: skill judgment is non-deterministic, so grade across reps rather than trusting a single run.
|
||||
- Source: PR [#1054](https://github.com/EveryInc/compound-engineering-plugin/pull/1054) "fix(testing): require behavior verification evidence" (HEAD `3923315a`), plus the follow-up parity test in `tests/pipeline-review-contract.test.ts`.
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "Pass paths, not content, when dispatching sub-agents"
|
||||
category: skill-design
|
||||
problem_type: design_pattern
|
||||
component: tooling
|
||||
root_cause: inadequate_documentation
|
||||
resolution_type: workflow_improvement
|
||||
severity: medium
|
||||
tags: [orchestration, subagent, token-efficiency, skill-design, multi-agent]
|
||||
date: 2026-03-26
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
When orchestrating sub-agents that need codebase reference material (config files, standards docs, etc.), passing full file contents in the sub-agent prompt bloats context and makes the orchestrator do expensive upfront work that may go unused.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Orchestrator skill reads multiple files, concatenates their contents into a block (e.g., `<standards>` with full CLAUDE.md/AGENTS.md content), and injects it into the sub-agent prompt
|
||||
- Sub-agent receives all content regardless of how much is relevant to its specific task
|
||||
- In repos with directory-scoped config files, the orchestrator must discover and read every file before invoking a single sub-agent
|
||||
- Sub-agent prompts grow linearly with the number of reference files, even when the agent needs only specific sections
|
||||
|
||||
## What Didn't Work
|
||||
|
||||
Having the orchestrator read all relevant file contents and pass them in a content block. This was the initial approach for the `ce-project-standards-reviewer` agent in ce-code-review: Stage 3b collected all CLAUDE.md/AGENTS.md content into a `<standards>` block passed in the sub-agent prompt.
|
||||
|
||||
Problems:
|
||||
- Orchestrator did expensive read work that may be partially wasted
|
||||
- Sub-agent prompt inflated with content it may not fully use
|
||||
- Scales poorly as the number of directory-scoped config files grows
|
||||
- Sub-agent loses agency to decide what's relevant
|
||||
|
||||
## Solution
|
||||
|
||||
Separate discovery (cheap) from reading (expensive). The orchestrator discovers file paths via glob or search, passes a path list, and the sub-agent reads only the files and sections it needs.
|
||||
|
||||
**Pattern from Anthropic's code-review command:**
|
||||
|
||||
> "Use another Haiku agent to give you a list of file paths to (but not the contents of) any relevant CLAUDE.md files from the codebase: the root CLAUDE.md file (if one exists), as well as any CLAUDE.md files in the directories whose files the pull request modified"
|
||||
|
||||
The reviewing agents then receive those paths and read the files themselves.
|
||||
|
||||
**How we applied it in ce-code-review:**
|
||||
|
||||
1. Stage 3b: orchestrator globs for CLAUDE.md/AGENTS.md paths in changed directories, emits a `<standards-paths>` block
|
||||
2. Sub-agent prompt: `ce-project-standards-reviewer` reads the listed files itself, targeting sections relevant to the changed file types
|
||||
3. Standalone fallback: if no `<standards-paths>` block is present, the agent discovers paths independently
|
||||
|
||||
**General template:**
|
||||
|
||||
```
|
||||
Orchestrator:
|
||||
1. Discover paths (glob/search) -> emit <reference-paths> block
|
||||
2. Pass path list to sub-agent
|
||||
|
||||
Sub-agent:
|
||||
1. If <reference-paths> present, read listed files
|
||||
2. If absent, discover paths independently (standalone fallback)
|
||||
3. Read only sections relevant to the specific task
|
||||
```
|
||||
|
||||
## Why This Works
|
||||
|
||||
Discovery is cheap; reading and processing file contents is expensive. The sub-agent is closer to the task (it knows what it's reviewing) and is better positioned to decide which sections of which files are relevant. This is lazy evaluation applied to agent orchestration: don't pay the cost of reading until you know you need the content.
|
||||
|
||||
## Prevention
|
||||
|
||||
When designing orchestrator skills that invoke sub-agents needing repo reference material:
|
||||
|
||||
1. **Default to path-passing.** Orchestrator discovers paths, sub-agent reads content.
|
||||
2. **Include a standalone fallback.** If the paths block is absent, the sub-agent discovers paths on its own. This enables both orchestrated and standalone invocation.
|
||||
3. **Content-passing is acceptable when:** the reference material is small, static, and guaranteed to be fully consumed by every invocation (e.g., a JSON schema under 50 lines that the sub-agent always needs in full).
|
||||
4. **Signal to refactor:** if you catch an orchestrator reading file contents before invoking sub-agents, treat it as a candidate for the path-passing pattern.
|
||||
|
||||
## Instruction phrasing matters more than meta-rules
|
||||
|
||||
Empirical testing showed that how the skill phrases a search instruction has a dramatic effect on tool call count. For the same task (find ancestor CLAUDE.md/AGENTS.md files for changed paths):
|
||||
|
||||
| Instruction phrasing | Claude Code tool calls | Codex shell commands |
|
||||
|---|---|---|
|
||||
| "for each changed file, walk its ancestor directories and check for X at each level" | 14 | 2 |
|
||||
| "find all X in the repo, then filter to ancestors of changed files" | 2 | 2 |
|
||||
|
||||
The "per-item walk" phrasing caused Claude Code to glob each directory level individually. The "bulk find, then filter" phrasing produced two globs total. Codex was resilient to both phrasings (it wrote a Python script to batch the work either way).
|
||||
|
||||
When in doubt about whether an instruction phrasing is efficient, test it empirically before committing. Both `claude -p` and `codex exec` support JSON output that reveals tool call counts:
|
||||
|
||||
```bash
|
||||
# Claude Code: stream-json + verbose shows each tool call
|
||||
claude -p "instruction here" --output-format stream-json --verbose 2>/dev/null > out.jsonl
|
||||
|
||||
# Codex: --json shows command_execution events
|
||||
codex exec --json --full-auto "instruction here" > out.jsonl
|
||||
```
|
||||
|
||||
This is worth doing for orchestration-heavy skills where instructions drive search or file discovery — a small phrasing change can produce a large difference in tool calls, latency, and token cost. Not every instruction needs benchmarking, but when the skill will run on every review or every plan, the cost compounds.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/compound-refresh-skill-improvements.md` — establishes "no shell commands for file operations in subagents"; complementary pattern about letting sub-agents use appropriate tools rather than orchestrating reads on their behalf
|
||||
- `docs/solutions/skill-design/script-first-skill-architecture.md` — complementary pattern: scripts pre-process large datasets so orchestrators don't load raw data
|
||||
- `docs/solutions/agent-friendly-cli-principles.md` — Principle #7 (Bounded, High-Signal Responses) reinforces that agents pay real cost for extra output; paths are bounded, content is not
|
||||
@@ -0,0 +1,434 @@
|
||||
---
|
||||
title: Portable agent skill authoring across models and harnesses
|
||||
date: 2026-07-11
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: best_practice
|
||||
component: development_workflow
|
||||
severity: high
|
||||
applies_when:
|
||||
- Creating or materially revising a skill that is distributed to multiple agent models or harnesses
|
||||
- Reviewing skill prose for cross-model behavior, harness portability, authority, or over-prompting
|
||||
- Choosing deterministic checks and targeted reasoning evals for a skill change
|
||||
tags:
|
||||
- skill-design
|
||||
- cross-model
|
||||
- cross-harness
|
||||
- prompting-guidance
|
||||
- protocol
|
||||
- judgment
|
||||
- skill-eval
|
||||
---
|
||||
|
||||
# Portable Agent Skill Authoring
|
||||
|
||||
Use this guide when creating or materially revising a skill that must work across models and agent harnesses.
|
||||
|
||||
The governing idea is simple:
|
||||
|
||||
> Start with the outcome and intent. Add only the smallest protocol needed to protect that outcome across runtimes.
|
||||
|
||||
You are always authoring from inside one model and one harness. Treat that runtime as one data point, not as the definition of how agents behave.
|
||||
|
||||
This guide is not a template whose every section must appear in every skill. A small skill may need only an outcome, a completion condition, and one boundary. Add the rest only when the skill's risk, observed behavior, or downstream contract justifies it.
|
||||
|
||||
## Author in this order
|
||||
|
||||
| Layer | What belongs there | When to include it |
|
||||
|---|---|---|
|
||||
| Outcome spine | Result or decision, next consumer, done condition, and non-obvious intent | Always first; a small skill may express it in one sentence |
|
||||
| Hard protocol | Falsifiable scope, gates, state, evidence, coverage, authority, and failure behavior | Only when omission can produce a wrong path or unsafe action |
|
||||
| Load-bearing workflow | Sequence whose order materially changes correctness | Only for invariant ordering |
|
||||
| Useful context | Domain facts, schemas, examples, specialist payloads, and late routes | Conditionally, when it can change judgment |
|
||||
| Adapters and techniques | Harness capability detection, verified tool adapters, path mechanics, and optional methods | As defaults or heuristics, never as the portable core |
|
||||
|
||||
The minimal form is the outcome spine plus only the protocol this skill needs, ending in completion or an explicit blocker.
|
||||
|
||||
Prefer small units of weaker-model insurance. Put one threshold, enum, count, quantifier, or gate beside the action it protects. Do not add a paragraph of defensive workflow when one falsifiable rule closes the observed gap.
|
||||
|
||||
If a capable model's output becomes worse after adding prose, remove judgment guidance and non-load-bearing steps first. Do not respond to lost reasoning quality by stacking more protocol.
|
||||
|
||||
### Every instruction must earn its cost
|
||||
|
||||
Always-loaded prose compounds across the workflow. Keep an instruction when it adds falsifiable protocol, counters a demonstrated model or harness tendency, or supplies domain knowledge that can materially change a decision. Vague effort or quality language does not earn that cost by itself.
|
||||
|
||||
Prefer an observable rule over a qualitative exhortation:
|
||||
|
||||
| Instead of | State what the instruction must change |
|
||||
|---|---|
|
||||
| "Be thorough." | "Check every changed execution path and report any path you could not verify." |
|
||||
| "Produce high-quality work." | "The handoff must name the decision, supporting evidence, unresolved risk, and next owner." |
|
||||
| "Be concise." | "Lead with the outcome; omit details that would not change the reader's next decision." |
|
||||
|
||||
This reflects current vendor guidance, not a preference for terseness. [OpenAI's prompting guide](https://learn.chatgpt.com/docs/prompting) says a short prompt is often enough, recommends starting with the result, and adds process only when process matters. [Fable 5 guidance](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5) says brief instructions can replace behavior-by-behavior enumeration and warns that skills tuned for earlier models may be too prescriptive.
|
||||
|
||||
This is not a ban on effort cues. A targeted phrase may be useful when it counters a documented runtime behavior. For example, the [Opus 4.8 guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8) recommends an explicit careful-reasoning cue for multi-step work forced to low effort. Treat such wording as a model-behavior adapter: name the condition it addresses and verify the effect rather than promoting it to a universal quality slogan.
|
||||
|
||||
This is an admission principle, not a mandate to delete unfamiliar detail. A line that feels redundant may be targeted insurance for a more literal model or a different harness. Test that possibility before removing it.
|
||||
|
||||
## The portability problem
|
||||
|
||||
A portable skill operates across two execution axes under one authority overlay:
|
||||
|
||||
1. **Model behavior:** how literally the model follows scope, how much structure it invents, and how it handles ambiguity, effort, and delegation.
|
||||
2. **Harness mechanics:** which tools, paths, permissions, delegation primitives, and loading behavior are available.
|
||||
3. **Authority context:** system, harness, user, organization, and project instructions that constrain both axes.
|
||||
|
||||
Author against all three. Do not mistake behavior supplied by the current model or harness for behavior guaranteed by the skill.
|
||||
|
||||
## Your model is not a neutral author
|
||||
|
||||
Before changing a skill, state which model or capability tier and harness you are using. Then ask what that runtime may mask or exaggerate.
|
||||
|
||||
| Authoring reaction | Possible bias | Portability check |
|
||||
|---|---|---|
|
||||
| "This rule is redundant." | Your model supplies the behavior without prompting. | Does a more literal model still preserve the contract? |
|
||||
| "This needs more steps." | Your model or harness needs scaffolding another runtime does not. | Is the step protocol, or compensation for this runtime? |
|
||||
| "It worked in my test." | The harness supplied a tool, path, permission, or context. | What happens when that capability is absent? |
|
||||
| "This mechanic is broken." | You were asked to find changes and recognized a familiar failure pattern. | Can you reproduce it, or does the implementation already handle it? |
|
||||
| "This skill is missing X." | Review prompts bias agents toward additive recommendations. | What observable failure, unmet consumer contract, or material risk does X address? |
|
||||
|
||||
Use this decentering procedure:
|
||||
|
||||
1. State the current runtime.
|
||||
2. Name likely masking and compensation effects.
|
||||
3. Inspect the executed artifact, including referenced scripts, launchers, engines, and harness behavior.
|
||||
4. Separate confirmed failures from verification tasks and plausible enhancements.
|
||||
5. Test the smallest realistic behavioral floor before adding prose.
|
||||
|
||||
## Apply guidance presumptively, not mechanically
|
||||
|
||||
Classify guidance by strength:
|
||||
|
||||
| Strength | Meaning | Deviation rule |
|
||||
|---|---|---|
|
||||
| Invariant | Required for correctness, safety, or the artifact contract | Deviate only when the invariant does not apply or a higher-priority instruction conflicts |
|
||||
| Default | The best general choice, but not a theorem about every implementation | Override with a concrete local fact, named consequence, substitute safeguard, and verification |
|
||||
| Heuristic | A useful technique or diagnostic question | Apply only when useful |
|
||||
|
||||
Local evidence may override general guidance. Preference, confidence, and convenience may not.
|
||||
|
||||
When a material deviation is necessary, record it proportionally:
|
||||
|
||||
```text
|
||||
Guide deviation:
|
||||
- Rule and strength:
|
||||
- Local fact that makes the normal form unsuitable:
|
||||
- Failure, cost, or conflict the normal form would create here:
|
||||
- Chosen alternative and substitute safeguard:
|
||||
- Verification:
|
||||
```
|
||||
|
||||
"This skill is unusual" is not a deviation record. If the same justified exception recurs, update the guide instead of multiplying local carve-outs.
|
||||
|
||||
## Build the skill around an outcome spine
|
||||
|
||||
State these before workflow:
|
||||
|
||||
- **Result:** the artifact or decision the skill must produce.
|
||||
- **Next consumer:** the user, agent, skill, or system that uses it next.
|
||||
- **Done:** the observable completion condition.
|
||||
- **Intent:** only the non-obvious reason that could change the approach.
|
||||
|
||||
Intent is useful when it helps a capable model distinguish the real goal from ancillary instructions. Motivational rationale that does not change behavior is noise.
|
||||
|
||||
The protocol kernel begins with outcome and completion behavior. Add other fields only when they materially constrain the skill:
|
||||
|
||||
- Authority, when sources may conflict.
|
||||
- Boundaries, when scope or mutation is risky.
|
||||
- Decision state, when work persists or branches.
|
||||
- Act/ask rules, when ambiguity can change scope or authority.
|
||||
- Evidence rules, when claims need provenance.
|
||||
- Coverage floors, when missing a category silently makes the result incomplete.
|
||||
- Failure branches, when a missing capability could otherwise cause a silent skip.
|
||||
|
||||
If many invariants share one outcome, authority domain, mutable state, and definition of done, keep one skill with an invariant index and conditional expansions. Split when outcomes, triggers, authority domains, audiences, or lifecycles are independently meaningful. Do not reduce visible line count by creating a hidden cross-skill state machine.
|
||||
|
||||
## Make activation portable
|
||||
|
||||
The name and description are an activation contract. A correct body is useless if it never runs.
|
||||
|
||||
- Describe the user-visible job and the situations that should route to the skill.
|
||||
- Name the closest adjacent requests that belong elsewhere.
|
||||
- Preserve deliberate invocation as a fallback when automatic routing is unavailable.
|
||||
- Use capability language instead of relying on one harness's command syntax.
|
||||
- Do not stuff the workflow into frontmatter.
|
||||
|
||||
Evaluate activation separately from execution with a few positive triggers, adjacent negatives, and explicit invocations. A routing failure is not an execution failure.
|
||||
|
||||
## Separate protocol from judgment
|
||||
|
||||
For each prescriptive block, ask:
|
||||
|
||||
> If this instruction disappears, can the workflow produce a wrong path, state, count, gate, field, boundary, coverage floor, or handoff?
|
||||
|
||||
If yes, it is likely **protocol**. Keep it explicit and falsifiable.
|
||||
|
||||
If removal mainly gives a capable model more freedom to reason, it is likely **judgment**. First try deleting it. If the outcome spine already guides the work and the realistic floor does not drift, leave it out. If observed behavior shows the guidance is needed, compress it to the smallest principle or contrast pair that closes the gap.
|
||||
|
||||
| Usually protocol | Usually judgment |
|
||||
|---|---|
|
||||
| Output paths and stable file shapes | Long menus of possible reasoning approaches |
|
||||
| Stable fields, headings, and enums | Several examples proving the same distinction |
|
||||
| Ordering, state transitions, and gates | Multi-paragraph rationale after a clear rule |
|
||||
| Counts, thresholds, and scope quantifiers | Generic quality exhortations |
|
||||
| Permission and mutation boundaries | Step-by-step reasoning the model can choose itself |
|
||||
| Required coverage categories | Creative menus that supply inspiration only |
|
||||
| Failure and completion branches | Repetition without a demonstrated drift point |
|
||||
|
||||
A menu is not automatically judgment. If omitting one item silently drops required coverage, the menu is protocol. If omitting it only narrows creative range, it is judgment.
|
||||
|
||||
Mixed blocks must be decomposed before classification. Preserve the invariant skeleton, required fields, enums, and coverage. Compress or remove examples and rationale separately.
|
||||
|
||||
## Preserve literal scope locally
|
||||
|
||||
More literal models often lose a distant qualifier. Keep scope beside the action it governs:
|
||||
|
||||
- "For each candidate separately..."
|
||||
- "Return exactly three..."
|
||||
- "Do not change files outside..."
|
||||
- "Stop after the first confirmed blocker..."
|
||||
|
||||
Prefer a local quantifier or threshold over a general reminder elsewhere in the skill.
|
||||
|
||||
## Define completion, not effort
|
||||
|
||||
Avoid open-ended instructions such as "continue until good" or "be thorough." Define observable completion instead:
|
||||
|
||||
- required artifact exists;
|
||||
- mandatory fields are populated;
|
||||
- evidence or verification is recorded;
|
||||
- each route ends in a result, routed action, required question, or blocker;
|
||||
- no launch-blocking questions remain when readiness is claimed.
|
||||
|
||||
Do not request hidden reasoning or chain-of-thought. Ask for decisions, evidence, assumptions, material rejected alternatives, and next actions.
|
||||
|
||||
## Describe capabilities before tools
|
||||
|
||||
Tool calls are common in skills, but a named tool should not become the portable contract unless its exact semantics are load-bearing.
|
||||
|
||||
Write in this order:
|
||||
|
||||
1. State the required capability.
|
||||
2. State the observable success contract.
|
||||
3. State the acceptable degradation path.
|
||||
4. Name verified tools only as adapters, short-circuits, requirements for a load-bearing property, or non-exhaustive examples.
|
||||
|
||||
A skill drives agent-callable capabilities. A user affordance such as a slash command is not necessarily callable by the agent. Do not instruct the model to use one unless the harness exposes it as an agent-callable mechanism.
|
||||
|
||||
Preserve the semantic floor. If every iteration requires agent reasoning, sub-skill invocation, or a fresh judgment, a shell loop that only repeats the outer command is not an equivalent fallback.
|
||||
|
||||
Do not infer that a capability is unavailable from one missing binary, environment variable, or MCP server. Check the harness's available interfaces and degrade explicitly.
|
||||
|
||||
### Bundled files
|
||||
|
||||
Distinguish three path cases:
|
||||
|
||||
| Case | Rule |
|
||||
|---|---|
|
||||
| Read-time reference | Use a relative path from the skill root |
|
||||
| Prose pointer to a file the agent acts on | Use a relative path plus "from this skill's directory" |
|
||||
| Executed shell command | Use the repository's portable skill-directory anchor pattern |
|
||||
|
||||
Diagnose before rewriting a path. Trace the skill-to-launcher, shell-to-launcher, and engine-to-resource boundaries. An engine may already locate sibling resources through its own source path. A pattern that looks suspicious is not a defect until the failure is reproduced or a necessary failing path is identified.
|
||||
|
||||
## Make authority proportional to risk
|
||||
|
||||
Most read-only, single-shot, non-delegating skills need no authorization apparatus. Skip it.
|
||||
|
||||
For consequential workflows, distinguish:
|
||||
|
||||
- the action the user directly requested;
|
||||
- in-envelope actions that are necessary to complete it;
|
||||
- actions that remain outside the envelope;
|
||||
- higher-priority prohibitions that invocation cannot erase.
|
||||
|
||||
Invocation may satisfy a default confirmation requirement when the skill clearly names a bounded class of mutations as part of its job. It does not override system, organization, or user prohibitions.
|
||||
|
||||
Write the positive rule when invocation supplies authority:
|
||||
|
||||
```text
|
||||
Invoking this workflow authorizes the following in-envelope actions without
|
||||
per-action confirmation: [...]. It does not authorize: [...].
|
||||
```
|
||||
|
||||
For chained mutation workflows, carry authority as bounded data. Include the target, permitted action classes, exclusions, and whether authority is user-direct or inherited. Downstream skills may narrow inherited authority, never broaden it. If structured authority cannot travel, fall back to the harness confirmation default. A live user instruction can narrow or revoke the active envelope at any time.
|
||||
|
||||
## Load instructions when they can change behavior
|
||||
|
||||
Always-loaded skill prose remains in context throughout the workflow. Extract substantial content when it is conditional or late-sequence.
|
||||
|
||||
- Keep the outcome spine, protocol kernel, and load-bearing route inline.
|
||||
- Move large schemas, specialist prompts, examples, and route-specific instructions to references.
|
||||
- Keep the instruction to load the reference inline at the point of use.
|
||||
- Do not inline a summary complete enough to suppress loading the authoritative reference.
|
||||
- Pass large context to subagents by file path plus a short gist rather than duplicating it into prompts.
|
||||
|
||||
When delegation is used, each task needs a distinct scope, output contract, and synthesis owner. Use parallel work for genuinely independent questions, not as a reflex. A single capable model may be better when the work depends on one evolving context or requires tight synthesis.
|
||||
|
||||
Stable cross-skill fields, enums, and return statuses are protocols. Version or parity-test them when independently evolving skills depend on exact agreement.
|
||||
|
||||
## Diagnose before prescribing
|
||||
|
||||
A review agent is biased toward producing changes. Counter that bias directly.
|
||||
|
||||
### Suspected defects
|
||||
|
||||
A required correctness or protocol fix must cite one of:
|
||||
|
||||
- a reproduced failure;
|
||||
- the exact implementation path that necessarily fails.
|
||||
|
||||
If neither is available, return a verification task instead of a change prescription.
|
||||
|
||||
### Proposed additions
|
||||
|
||||
An addition must name:
|
||||
|
||||
- the observable consequence of its absence;
|
||||
- the unmet consumer contract or material risk;
|
||||
- the affected layer;
|
||||
- why the proposed mechanism is the smallest suitable one.
|
||||
|
||||
If the value is plausible but unverified, label it **Consider**, not **Change**.
|
||||
|
||||
Use three finding classes:
|
||||
|
||||
- **Change:** demonstrated gap with a supported smallest fix.
|
||||
- **Verify:** concrete risk that still needs reproduction or implementation tracing.
|
||||
- **Consider:** plausible enhancement whose value has not been demonstrated.
|
||||
|
||||
Do not solve a non-problem with a rewrite. Prefer an additive guard or explicit definition over replacing an implementation that already works.
|
||||
|
||||
## Evaluate proportionally
|
||||
|
||||
Mechanical checks belong in CI when they are deterministic and available to contributors:
|
||||
|
||||
- frontmatter and schema validation;
|
||||
- broken references and path checks;
|
||||
- duplicated-contract parity;
|
||||
- stable fields, headings, and enums;
|
||||
- script and fixture tests;
|
||||
- conversion and packaging invariants.
|
||||
|
||||
Behavioral agent reasoning evals are best-effort local evidence, not a mandatory exhaustive matrix. Use a small targeted fixture pack for the largest portability risks introduced by the change.
|
||||
|
||||
Prioritize:
|
||||
|
||||
1. **Weakest realistic layer:** does the minimum supported model or harness preserve the protocol?
|
||||
2. **Strong-model regression:** did added prose reduce reasoning quality, novelty, synthesis, or restraint?
|
||||
3. **Restraint:** does the agent avoid inventing defects, additions, authority machinery, or unrelated work?
|
||||
4. **Fresh downstream consumer:** can the next skill or agent use the output without clarification?
|
||||
5. **Activation:** do positive and adjacent-negative prompts route correctly?
|
||||
|
||||
Do not imply a full model-by-harness suite for every edit. Choose fixtures tied to the biggest gotchas in the change.
|
||||
|
||||
Use fresh context for behavioral prose evaluation. Some harnesses cache skill content at session start, so invoking the edited skill in the authoring session may test stale content.
|
||||
|
||||
For side-effecting skills, evaluate in layers:
|
||||
|
||||
1. Grade the intended and explicitly suppressed actions.
|
||||
2. Use fake boundaries, dry-run contracts, or mutation logs.
|
||||
3. Use an ephemeral external system if integration behavior matters.
|
||||
4. Use a live canary only when the remaining risk justifies it.
|
||||
|
||||
Verify load-bearing harness claims live on the runtimes that depend on them. Leave unverified claims as explicit verification tasks rather than universal assertions.
|
||||
|
||||
Read a tie honestly. If old and new prose both succeed on a strong model, the test shows no regression but not improvement. Test the claimed determinism or weaker-model insurance at the layer where it matters.
|
||||
|
||||
Measure the outcome the skill exists to improve, not proxy volume:
|
||||
|
||||
- creative work: grounded novelty, diversity of surviving decisions, and downstream usefulness;
|
||||
- planning: clarification burden and execution errors;
|
||||
- research: claim support and recall;
|
||||
- orchestration: correct routing, state, authority, and completion rather than tool-call count.
|
||||
|
||||
## Authoring checklist
|
||||
|
||||
### Outcome and restraint
|
||||
|
||||
- [ ] The outcome spine appears before workflow.
|
||||
- [ ] Non-obvious intent is included only when it changes the approach.
|
||||
- [ ] The skill stops at the minimal form unless evidence, risk, or a consumer contract justifies more.
|
||||
- [ ] Every route has a completion or blocker branch.
|
||||
- [ ] Generic quality exhortations and motivational rationale are absent.
|
||||
|
||||
### Protocol and judgment
|
||||
|
||||
- [ ] Protocol is explicit and falsifiable.
|
||||
- [ ] Judgment is deleted when the outcome already guides it.
|
||||
- [ ] Remaining judgment guidance is the smallest supported principle or contrast pair.
|
||||
- [ ] Required coverage menus and local quantifiers are preserved.
|
||||
- [ ] Mixed blocks were decomposed before classification.
|
||||
|
||||
### Runtime portability
|
||||
|
||||
- [ ] The current authoring model and harness are identified.
|
||||
- [ ] Model masking and compensation risks are stated.
|
||||
- [ ] Activation has positive, adjacent-negative, and explicit-invocation cases.
|
||||
- [ ] Capabilities and observable contracts precede named tools.
|
||||
- [ ] Missing capabilities degrade without silent skips.
|
||||
- [ ] Bundled execution paths are deterministic and were diagnosed before rewriting.
|
||||
|
||||
### Authority and delegation
|
||||
|
||||
- [ ] Read-only, non-delegating skills skip mutation-authority machinery.
|
||||
- [ ] Consequential workflows name their bounded mutation envelope and exclusions.
|
||||
- [ ] Higher-priority prohibitions remain intact.
|
||||
- [ ] Inherited authority is explicit and can only narrow.
|
||||
- [ ] Delegated tasks have distinct scopes, output contracts, and a synthesis owner.
|
||||
|
||||
### Evidence and evaluation
|
||||
|
||||
- [ ] Correctness fixes cite a reproduced failure or necessary failing path.
|
||||
- [ ] Additions cite an observable consequence, consumer contract, or material risk.
|
||||
- [ ] Unconfirmed defects are verification tasks.
|
||||
- [ ] Unproven enhancements are considerations.
|
||||
- [ ] The smallest supported change is preferred.
|
||||
- [ ] Mechanical contracts are tested deterministically.
|
||||
- [ ] Targeted behavioral fixtures cover the biggest portability risks.
|
||||
- [ ] Both weaker-model insurance and strong-model regression are considered.
|
||||
|
||||
## Compact review prompt
|
||||
|
||||
```text
|
||||
Review or author this skill for portability across models and agent harnesses.
|
||||
|
||||
Do not materialize every section of the guide. Start with the outcome spine:
|
||||
result, next consumer, done condition, and non-obvious intent when it changes
|
||||
the approach. Add only the protocol needed to protect that outcome.
|
||||
|
||||
1. State the current model or capability tier and harness. Name likely masking
|
||||
and compensation effects.
|
||||
2. Diagnose before prescribing. A correctness fix needs a reproduced failure or
|
||||
necessary failing path. An addition needs an observable consequence, unmet
|
||||
consumer contract, or material risk. Otherwise return Verify or Consider.
|
||||
3. Separate model behavior, harness mechanics, and authority context.
|
||||
4. Treat the name and description as an activation contract.
|
||||
5. Keep protocol explicit. Delete judgment guidance when the outcome is enough;
|
||||
otherwise use the smallest supported principle or contrast pair.
|
||||
6. Preserve local quantifiers, gates, stable fields, coverage floors, and
|
||||
completion branches.
|
||||
7. Describe capabilities and observable behavior before named tools. Preserve
|
||||
the semantic floor and define degradation.
|
||||
8. Add authority and delegation machinery only when the skill actually mutates
|
||||
or delegates consequential work.
|
||||
9. Use a small targeted evaluation set for the weakest realistic layer,
|
||||
strong-model regression, restraint, activation, and the next consumer.
|
||||
10. Choose the smallest supported change and record any material deviation.
|
||||
|
||||
Return the outcome spine, proposed skill or findings, intentionally inapplicable
|
||||
guide sections, Change/Verify/Consider findings, targeted tests, and unresolved
|
||||
decisions that would materially change the contract.
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
The principles above are model-neutral. Model-specific behavior examples should be rechecked as generations change.
|
||||
|
||||
- [OpenAI: Prompting](https://learn.chatgpt.com/docs/prompting)
|
||||
- [OpenAI: Model guidance](https://developers.openai.com/api/docs/guides/latest-model)
|
||||
- [OpenAI: Evals](https://developers.openai.com/api/docs/guides/evals)
|
||||
- [Anthropic: Prompt engineering overview](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview)
|
||||
- [Anthropic: Prompting Claude Opus 4.8](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8)
|
||||
- [Anthropic: Prompting Claude Fable 5](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5)
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: Always-on routing for interactive menus belongs inline in SKILL.md, not in references
|
||||
date: 2026-04-28
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: architecture_pattern
|
||||
component: ce-plan
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Authoring a skill that ends in an `AskUserQuestion`-style menu where the user picks the next action
|
||||
- Deciding whether per-option routing belongs in SKILL.md or in a reference file
|
||||
- Reviewing a skill where the agent renders a menu and stops at the user's selection without acting
|
||||
tags:
|
||||
- skill-design
|
||||
- menu-routing
|
||||
- skill-md-vs-references
|
||||
- ce-plan
|
||||
- extraction-rule
|
||||
- load-bearing-rules
|
||||
related_issue: https://github.com/EveryInc/compound-engineering-plugin/issues/714
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
`ce-plan` Phase 5.4 presented a four-option post-generation menu (`Start /ce-work`, `Create Issue`, `Open in Proof`, `Done for now`). The action that should fire when the user picked an option lived only in `references/plan-handoff.md`. The skill body said "Routing each selection ... lives in `references/plan-handoff.md` — follow it for every branch" plus a "Load `references/plan-handoff.md` now" instruction in 5.3.8.
|
||||
|
||||
In practice, agents rendered the menu, captured the user's selection, and stopped without firing the routed action. The user picked "Start `/ce-work` (Recommended)" and watched the agent acknowledge the choice in prose ("User picked Start /ce-work. Handing off — invoke `/ce-work` next") instead of programmatically invoking `ce-work`.
|
||||
|
||||
## Root Cause
|
||||
|
||||
Two failure modes compounded:
|
||||
|
||||
1. **The agent didn't load the reference.** SKILL.md content caches at session start; references load on demand. An agent that renders past the "Load `references/plan-handoff.md` now" instruction on the way to the menu has no per-option routing in its loaded context. The menu becomes a textual handoff with no associated action.
|
||||
2. **Even an agent that loaded the reference saw ambiguous language.** The reference said `**Start /ce-work** -> Call /ce-work with the plan path`. That doesn't name the platform's skill-invocation primitive. "Call /ce-work" can be read as "tell the user to type /ce-work in chat" rather than "fire the Skill tool now."
|
||||
|
||||
The plugin's own `AGENTS.md` "Extract Conditional and Late-Sequence Blocks" rule (under "Writing Skill Instructions") guides extraction: extract content that is *conditional or late-sequence and represents ~20%+ of the skill*. The bare per-option routing was late-sequence (only fires after Phase 5) but **not conditional** — option 1 always means "invoke ce-work," option 4 always means "end the turn." The always-on subset should not have been extracted.
|
||||
|
||||
The same AGENTS.md, in "Inline the Trigger, Not the Content," already articulates the underlying rule: a load-bearing instruction (one that MUST fire reliably) belongs inline at the top of its phase, because references load on demand and an agent that skipped one would stop or guess. The post-menu routing satisfies the load-bearing definition. Failing to apply this principle was the authoring mistake.
|
||||
|
||||
## Fix
|
||||
|
||||
1. Inline a `### Routing` block in SKILL.md Phase 5.4 with one explicit action per menu option. Use platform-explicit invocation language: "Invoke the `ce-work` skill via the platform's skill-invocation primitive (`Skill` in Claude Code, `Skill` in Codex, the equivalent on Gemini/Pi), passing the plan path as the skill argument. Do not merely tell the user to type `/ce-work` — fire the invocation now so the plan executes in this session."
|
||||
2. Mirror the same platform-explicit phrasing in `references/plan-handoff.md` so both surfaces converge. The reference still owns the elaborate sub-flows (Proof HITL state machine, Issue Creation tracker detection, post-HITL `ce-doc-review` resync, upload-failure fallback) — those are genuinely conditional and multi-step.
|
||||
3. Add a regression test (`tests/skills/ce-plan-handoff-routing.test.ts`) that fails if any of the four inline routing lines disappear, and specifically asserts that the `Start /ce-work` routing names the skill-invocation primitive and the plan path.
|
||||
|
||||
## Authoring Checklist for Future Skills
|
||||
|
||||
Before extracting a block to a reference file, ask:
|
||||
|
||||
- **Is the block always executed when this phase is reached?** If yes, lean toward inlining. References are for branches the agent enters only sometimes.
|
||||
- **Does the block carry routing for an interactive menu the skill renders?** If yes, the bare per-option action belongs inline. The elaborate sub-flow for each option (multi-status state machines, retry logic, downstream skill dispatch) can stay in a reference.
|
||||
- **Could an agent that skips the reference still complete the skill correctly?** If no — if the agent without the reference would stop or guess — the missing content is load-bearing and belongs inline.
|
||||
- **Is the language platform-explicit?** When a routing line says "Call /ce-work," ask whether an agent could read it as "tell the user" rather than "fire the tool." Name the platform primitive (Skill tool, skill-invocation primitive) and the argument shape (plan path, file path).
|
||||
- **Does the inline block command the agent to load and act, or does it summarize what the reference contains?** Inlining is two-sided. The firing imperative and the load instruction belong inline (the rest of this checklist). But a *paraphrase of the reference's substance* backfires the opposite way: it drifts from the reference (nothing tests the two copies against each other), and it suppresses the load — an agent that already has a workable inline summary judges it "has enough" and never opens the file, so the reference's templates and examples never reach it. Inline the trigger; keep the substance in the one reference that owns it. Test: if the inline text is complete enough to act on alone, the agent will, and the reference's nuance never lands. (See `AGENTS.md` → "Inline the Trigger, Not the Content.")
|
||||
|
||||
## Related Patterns
|
||||
|
||||
- `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines.md` — same family: skills that render decision points need their state transitions to be deterministic in the loaded context, not one reference-load away.
|
||||
- `docs/solutions/skill-design/confidence-anchored-scoring.md` — load-bearing scoring rubrics also belong inline in SKILL.md so they fire reliably across sessions.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: Research agent dispatch is intentionally separated across the skill pipeline
|
||||
date: 2026-04-05
|
||||
last_refreshed: 2026-06-20
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: architecture_pattern
|
||||
component: tooling
|
||||
severity: low
|
||||
applies_when:
|
||||
- Evaluating whether repo-research-analyst or learnings-researcher prompt assets in ce-plan duplicate work from ce-brainstorm or ce-work
|
||||
- Adding a new research prompt asset and deciding which pipeline stage should dispatch it
|
||||
- Considering pass-through optimizations like the Slack researcher pattern (commit f7a14b76)
|
||||
tags:
|
||||
- research-agent
|
||||
- pipeline
|
||||
- skill-design
|
||||
- deduplication
|
||||
- ce-plan
|
||||
- ce-brainstorm
|
||||
- ce-work
|
||||
---
|
||||
|
||||
# Research prompt dispatch is intentionally separated across the skill pipeline
|
||||
|
||||
## Context
|
||||
|
||||
After optimizing the Slack researcher prompt to avoid redundant work between ce-brainstorm and ce-plan (commit f7a14b76 on `tmchow/slack-analyst-agent`), a natural question arose: does the same duplication problem exist for `repo-research-analyst` and `learnings-researcher`? Both are skill-local prompt assets dispatched by ce-plan in Phase 1.1 on every run, regardless of whether ce-brainstorm produced an origin document.
|
||||
|
||||
Investigation confirmed no duplication exists. The three workflow stages operate on deliberately separated information types, and research prompt dispatch follows this separation cleanly.
|
||||
|
||||
## Guidance
|
||||
|
||||
The brainstorm -> plan -> work pipeline separates research by information type:
|
||||
|
||||
**ce-brainstorm** gathers *product context* (WHAT to build). It performs an inline "Existing Context Scan" -- surface-level file discovery focused on product questions. It does NOT dispatch `repo-research-analyst` or `learnings-researcher`. Its output is a requirements document covering product decisions, scope, and success criteria, intentionally excluding implementation details.
|
||||
|
||||
**ce-plan** gathers *implementation context* (HOW to build it). It ALWAYS reads `references/agents/repo-research-analyst.md` and `references/agents/learnings-researcher.md`, then uses those prompts to seed generic subagents in Phase 1.1. These produce: tech stack versions, architectural patterns, conventions, file paths, and institutional knowledge from `docs/solutions/`. This feeds the plan document's Context & Research, Patterns to Follow, Files, and Key Technical Decisions sections. The `repo-research-analyst` output also drives Phase 1.2 decisions about whether external research prompts are needed.
|
||||
|
||||
**ce-work** gathers NO research context independently. It reads the plan document and uses embedded research findings to guide implementation. For bare prompts (no plan), it does a lightweight inline scan -- no agent dispatch. The plan document IS the handoff mechanism from ce-plan's research to ce-work.
|
||||
|
||||
When ce-plan receives an origin document from ce-brainstorm, it reads it as primary input (Phase 0.3) but still runs its research prompts because they gather categorically different information.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Prevents false optimizations.** Without understanding the information type separation, a contributor might skip ce-plan's research prompts when a brainstorm document exists, breaking the plan's ability to produce implementation-ready guidance.
|
||||
- **Clarifies when pass-through optimizations ARE warranted.** The Slack researcher was a genuine redundancy: both ce-brainstorm and ce-plan dispatched the same prompt for overlapping information. The fix passed existing context so the subagent focuses on gaps. For `repo-research-analyst` and `learnings-researcher`, no such redundancy exists because only ce-plan dispatches them.
|
||||
- **Protects the plan document's role as the sole handoff artifact.** ce-work depends on the plan containing complete implementation context. If ce-plan's research agents are skipped, ce-work receives an incomplete plan and must improvise.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- When evaluating whether research prompt calls across pipeline stages are redundant -- check whether multiple stages dispatch the same prompt for overlapping information types.
|
||||
- When adding a new research prompt -- classify whether it gathers product context (brainstorm), implementation context (plan), or execution context (work), and dispatch it from the matching stage only.
|
||||
- When considering a pass-through optimization like the Slack pattern -- the prerequisite is that TWO stages independently dispatch the same prompt. If only one stage dispatches the prompt, no optimization is needed.
|
||||
|
||||
## Examples
|
||||
|
||||
**No optimization needed (this case):**
|
||||
ce-plan always calls `repo-research-analyst` even when a brainstorm document exists. Does ce-brainstorm also call it? No -- brainstorm only does an inline product-focused scan. The calls are not redundant; no change needed.
|
||||
|
||||
**Optimization warranted (Slack pattern):**
|
||||
Both ce-brainstorm and ce-plan dispatched the Slack researcher. Fix: when ce-plan finds Slack context in the origin document, pass it to the skill-local `slack-researcher` prompt asset so the subagent focuses on gaps. The prompt is still used -- it starts from a better baseline.
|
||||
|
||||
**Anti-pattern -- skipping agents incorrectly:**
|
||||
Removing `repo-research-analyst` from ce-plan when an origin document exists, reasoning "brainstorm already scanned the repo." The resulting plan lacks architectural patterns, file paths, and convention details. ce-work produces code that ignores existing patterns.
|
||||
|
||||
**Correct stage placement for a new agent:**
|
||||
A "dependency-analyzer" prompt that identifies library versions and compatibility constraints gathers implementation context (HOW). It belongs in ce-plan's Phase 1.1, not ce-brainstorm. ce-work will consume its findings via the plan document.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/solutions/skill-design/pass-paths-not-content-to-subagents.md` -- related agent dispatch optimization pattern (token efficiency, not deduplication)
|
||||
- `docs/solutions/skill-design/beta-skills-framework.md` -- documents the pipeline chain and the beta-skills rollout pattern that plugs into it
|
||||
- `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings.md` -- extends this framing downstream (document-review, ce-code-review, resolve-pr-feedback) with meta-observations from running the full pipeline end-to-end on a feature
|
||||
- Commit f7a14b76 on `tmchow/slack-analyst-agent` -- the Slack researcher pass-through optimization that prompted this analysis
|
||||
- GitHub issue #492 -- the historical `ce-repo-research-analyst` self-recursion bug (fixed, separate concern)
|
||||
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: "safe_auto rubric calibration: variance reduction beats safe_auto-rate-as-target"
|
||||
date: 2026-04-25
|
||||
last_updated: 2026-04-25
|
||||
category: skill-design
|
||||
module: compound-engineering / ce-code-review
|
||||
problem_type: design_pattern
|
||||
component: subagent-template
|
||||
severity: low
|
||||
tags:
|
||||
- ce-code-review
|
||||
- autofix-class
|
||||
- rubric
|
||||
- calibration
|
||||
- eval
|
||||
- eval-methodology
|
||||
- variance
|
||||
related_issue: EveryInc/compound-engineering-plugin#686
|
||||
related_pr: "PR #685 (suggested_fix push that this builds on)"
|
||||
---
|
||||
|
||||
# safe_auto rubric calibration: variance reduction beats safe_auto-rate-as-target
|
||||
|
||||
## TL;DR
|
||||
|
||||
Issue #686 hypothesized that personas were *under*-classifying findings as `safe_auto` and proposed tightening the rubric to push more findings into auto-apply. The 60-trial eval showed:
|
||||
|
||||
- The hypothesis doesn't hold for textbook cases. **6 of 9 fixture shapes** classify identically between baseline and tightened rubric (all `safe_auto` where mechanical, all `gated_auto` where contract-touching).
|
||||
- The real win is **variance reduction on ambiguous cases** — particularly orphan code without explicit "no callers" annotation, where the baseline rubric produces essentially random classification (manual / safe_auto / gated_auto across 4 trials on the same fixture).
|
||||
- The tightened rubric trades one stable disagreement: cross-file Rails service extraction goes from baseline `safe_auto` (4/4) to tightened `gated_auto` (6/7). Both classifications are internally defensible. Tightened is the more conservative reading and matches what a careful operator would want before an auto-apply.
|
||||
|
||||
The shipped change is mostly a determinism patch, not a safe_auto-rate increase. Two methodological lessons generalize beyond this calibration: **measure variance, not just classification-rate-shift**, and **a synthetic-fixture eval harness is the right tier between "ship and watch" and "stare at the diff"**. Both are written up in dedicated sections below.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
[`ce-code-review`'s subagent template](../../skills/ce-code-review/references/subagent-template.md) classifies each finding into one of four `autofix_class` buckets — `safe_auto`, `gated_auto`, `manual`, `advisory` — that govern downstream fixer dispatch. Headless mode auto-applies only `safe_auto`; everything else surfaces for user routing.
|
||||
|
||||
Issue #686 cited an incident pre-#685 ("8 findings ended up in tickets that should have been fixes") and inferred personas were too conservative on `safe_auto`, pushing genuinely-mechanical fixes into `gated_auto` or `manual`. PR #685 fixed the LFG defer-bias directly via `suggested_fix` propagation. #686 asked: should we also tighten the `safe_auto` boundary so more findings flow into auto-apply?
|
||||
|
||||
---
|
||||
|
||||
## What the eval probed
|
||||
|
||||
9 fixtures across distinct finding shapes, run on both the post-#685 baseline subagent template and a tightened version. Single-persona dispatches (correctness / maintainability / testing / security depending on fixture). 60 total trials across 5 iterations:
|
||||
|
||||
| Fixture | Shape | Persona |
|
||||
|---|---|---|
|
||||
| F1 | Nil guard inside internal helper | correctness |
|
||||
| F1b | Cart subtotal `min_by` semantic bug | correctness |
|
||||
| F2 | Off-by-one with parallel pattern in scope | correctness |
|
||||
| F3 | Dead code with explicit "no callers" comment | maintainability |
|
||||
| F3b | Orphan code with no explicit deadness signal | maintainability |
|
||||
| F4 | Local helper extraction within one class | maintainability |
|
||||
| F4b | Cross-file helper extraction | maintainability |
|
||||
| F5 | Missing test for new public method | testing |
|
||||
| F6 | Admin auth gate (negative control — should stay gated_auto) | security |
|
||||
|
||||
The tightened rubric added: a one-sentence "test" for `safe_auto` with explicit exclusion list (no contract / permission / signature change), four "boundary cases that feel risky but are safe_auto" examples, a symmetry-of-error opening sentence, and a "do not default to gated_auto" anti-pattern guard.
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
| Fixture | Baseline | Tightened | Delta |
|
||||
|---|---|---|---|
|
||||
| F1 | 3/3 safe_auto | 3/3 safe_auto | identical |
|
||||
| F1b | 3/3 safe_auto | 3/3 safe_auto | identical |
|
||||
| F2 | 3/3 safe_auto | 3/3 safe_auto | identical |
|
||||
| F3 | 3/3 safe_auto | 3/3 safe_auto | identical |
|
||||
| F4 | 2/3 safe_auto, 1/3 advisory | 3/3 safe_auto | tightened reduces variance |
|
||||
| **F3b** | **manual / safe_auto / gated_auto / safe_auto (4 trials, 3 different classes)** | **7/7 gated_auto** | **tightened dramatically reduces variance** |
|
||||
| F4b | 4/4 safe_auto | 6/7 gated_auto, 1/7 advisory | stable disagreement, opposite directions |
|
||||
| F5 | 3/3 safe_auto | 3/3 safe_auto | identical |
|
||||
| F6 (control) | 1/1 gated_auto | 1/1 gated_auto | identical (correctly stable) |
|
||||
|
||||
---
|
||||
|
||||
## Interpretation
|
||||
|
||||
### The hypothesis was approximately wrong, but the rubric tightening is approximately right anyway
|
||||
|
||||
The "personas under-classify safe_auto" hypothesis assumed personas were systematically conservative across the boundary. The data shows post-#685 personas already classify textbook mechanical cases (nil guards, off-by-ones with parallel patterns, explicit dead code, local helper extraction, missing tests for existing methods) as `safe_auto` — six of nine fixtures show no daylight between baseline and tightened.
|
||||
|
||||
What the rubric tightening actually does is reduce **variance** on cases where the rubric's previous wording was genuinely ambiguous. F3b is the headline: an orphan method without an explicit "no callers" comment. The baseline produced `manual`, `safe_auto`, and `gated_auto` across four trials on the same input — essentially random. The tightened rubric pins it to `gated_auto` deterministically by giving the persona a clearer test ("the surrounding refactor obviously displaces it" requires positive signal, which this fixture lacks).
|
||||
|
||||
Variance on classification is a real cost: ce-work's headless mode behaves differently across runs on identical inputs when the rubric is ambiguous. Determinism is more valuable than the specific classification chosen, as long as the classification is defensible.
|
||||
|
||||
### F4b is the one stable disagreement, and it's defensible either way
|
||||
|
||||
Cross-file extraction of two service objects with identical bodies: the baseline rubric's "extracting a duplicated helper" example matches, so 4/4 classify `safe_auto`. The tightened rubric's "naming or placement requires a design conversation" criterion catches Rails service-layering placement (base class vs concern vs module) and 6/7 classify `gated_auto`.
|
||||
|
||||
Both are internally consistent. The argument for `safe_auto` is "the consolidation is mechanical, the new module's name follows from the shared shape, both call sites update in lockstep within one diff." The argument for `gated_auto` is "in a Rails app, where a shared module lives is a real architectural decision the user should approve before it lands." Reasonable operators could prefer either.
|
||||
|
||||
The tightened rubric picks the conservative reading. That's a trade-off, not a regression: ce-work's headless will now flag cross-file extraction for user review instead of auto-applying it. For careful operators that's the right call; for autonomous bulk refactor flows it's modestly more friction.
|
||||
|
||||
### What the eval doesn't tell us
|
||||
|
||||
This was a single-persona, synthetic-fixture eval. Real reviews run multiple personas through synthesis with conservative tie-breaks; the persona-side classification I measured is one input. Synthesis-layer effects could amplify or dampen what the eval shows. A proper end-to-end test on a real branch with multi-persona dispatch would catch surprises.
|
||||
|
||||
The fixtures are also synthetic. The original "8 findings to tickets" incident might involve a finding shape I didn't probe. If the calibration ships and a similar incident recurs, that's evidence the rubric still has a gap and another iteration is warranted.
|
||||
|
||||
---
|
||||
|
||||
## What shipped
|
||||
|
||||
Two files changed:
|
||||
|
||||
1. **`subagent-template.md` (autofix_class decision guide, ~138-160).** Net +14 lines, −6 lines.
|
||||
- One-sentence "symmetry of wrong-side cost" framing at the top.
|
||||
- Replaced "without design judgment" with an operational test: one-sentence fix, no "depends on" clauses, no change to function signature / public-API contract / error contract / security posture / permission model.
|
||||
- Added a "Boundary cases that often feel risky but are still safe_auto" subsection covering nil guards, off-by-ones, dead code, helper extraction (with the cross-file discriminator that pins F4b to gated_auto when placement is design-shaped).
|
||||
- Added "do not default to gated_auto" parallel to the existing "do not default to advisory" anti-pattern guard.
|
||||
|
||||
2. **`findings-schema.json` (autofix_class field description).** Replaced terse "Reviewer's conservative recommendation" with an operational summary that mirrors the subagent-template wording.
|
||||
|
||||
---
|
||||
|
||||
## Methodological lesson 1: variance reduction beats classification-rate-shift
|
||||
|
||||
The headline lesson generalizes beyond `autofix_class`. When evaluating any persona-rubric change, **measure variance reduction on ambiguous fixtures first; treat classification-rate shifts on textbook fixtures as a noise-prone third-tier signal.**
|
||||
|
||||
### The hierarchy of evidence
|
||||
|
||||
1. **First-order signal — variance reduction on ambiguous fixtures.** Run each ambiguous cell at least N=3 trials per version, bumping to N=7+ if N=3 still looks noisy. Measure: how many distinct classifications does each version produce across trials on the same input? A baseline that emits 3 different classes across 4 trials, paired with a tightened version that pins to one class across 7 trials, is a clear win — independent of *which* class the tightened version chose.
|
||||
2. **Second-order signal — stable disagreements on boundary cases.** A cell where baseline gives `X` consistently and tightened gives `Y` consistently is a real trade-off, not noise. Both readings may be defensible; the question becomes "which is the right side to land on?" — a judgment call, but a legible one.
|
||||
3. **Third-order signal — classification-rate shifts on textbook cases.** This is the noisiest, lowest-value signal because synthetic textbook fixtures don't move on a well-tuned model. If your only "win" is rate-shifts on textbook cases, you are likely measuring noise.
|
||||
|
||||
### Why N=1 synthetic-fixture evals mislead
|
||||
|
||||
Persona dispatches over the same input can produce different classifications across runs because the rubric's prior wording was genuinely ambiguous, not because the model is broken. On synthetic fixtures the temptation to read N=1 is especially strong — the fixture *feels* deterministic, so one trial *feels* sufficient. It isn't.
|
||||
|
||||
In this calibration, two early N=1 reads produced two confidently-wrong conclusions in succession — first "the tightened rubric has no effect," then "the tightened rubric is causing a wrong-direction regression." Both reversed at N=3 and resolved cleanly only at N=4 to N=7 on the noisy cells.
|
||||
|
||||
The mechanism: F3b at the baseline samples from a tri-modal distribution {manual, safe_auto, gated_auto}. Two single-trial reads on the same prompt pair on the same fixture can therefore produce wildly different stories:
|
||||
|
||||
- (baseline=safe_auto, tightened=gated_auto) → "regression: tightening pushed a safe_auto into gated_auto"
|
||||
- (baseline=manual, tightened=gated_auto) → "improvement: tightening pulled a manual into gated_auto"
|
||||
- (baseline=gated_auto, tightened=gated_auto) → "no effect"
|
||||
|
||||
All three reads are sampled from the same prompt pair on the same fixture. Only the variance summary tells the truth: baseline is essentially random on this input; tightened is pinned. That's the win.
|
||||
|
||||
### Practical rules
|
||||
|
||||
- **Never trust N=1 on a synthetic fixture for a directional read.** Treat single-trial reads as "do the dispatches even run end-to-end?" smoke checks, not behavior measurements.
|
||||
- **N=3 is the floor; bump until variance stops moving.** If three trials disagree, run more trials *before* running more fixtures. The bottleneck for a confident read is depth on the noisy cell, not breadth across new cells.
|
||||
- **Aggregate variance explicitly in the summary table.** A row like `F3b: baseline manual / safe_auto / gated_auto / safe_auto (4 trials, 3 distinct classes)` tells the reader something a single-class summary cannot.
|
||||
- **Treat reduction in *number of distinct classes per cell* as the headline metric for prompt-tightening changes.** This is the determinism win, and it's what justifies the prompt's added token cost.
|
||||
- **Keep a negative control fixture** that should not move at all — if it moves under either version across trials, the rubric has a stability problem the calibration is masking.
|
||||
|
||||
### When the lens applies
|
||||
|
||||
Apply the variance-first lens when the eval is on synthetic fixtures (no ground-truth label), the rubric outputs into a small number of discrete buckets, or the change is motivated by an incident report claiming systematic mis-classification. The lens applies less when you have ground-truth labels (rate-shift against truth becomes meaningful) or when the rubric outputs free-text rather than a discrete bucket.
|
||||
|
||||
A related precedent in this repo: [`docs/solutions/skill-design/ce-doc-review-calibration-patterns.md`](./ce-doc-review-calibration-patterns.md) has a "Reviewer variance is inherent; single runs aren't baselines" section warning the same thing, scoped to ce-doc-review's tier classification. The principle generalizes: any persona-rubric eval needs N≥3 minimum on cells where the rubric is plausibly ambiguous.
|
||||
|
||||
---
|
||||
|
||||
## Methodological lesson 2: validating persona-rubric prompt changes before shipping
|
||||
|
||||
A reusable harness pattern for evaluating any subagent-template / persona-prompt change before merge.
|
||||
|
||||
### The gap this fills
|
||||
|
||||
There is a tier between "ship the prompt and watch real reviews" (slow, low signal, mixes in synthesis-layer effects) and "stare at the diff and reason about it" (no signal). The pattern below fills that tier with a lightweight, scriptable harness that holds everything constant except the prompt under test.
|
||||
|
||||
### Workspace pattern (reproduces the safe_auto eval; reusable as-is)
|
||||
|
||||
```
|
||||
/tmp/<eval-name>/
|
||||
fixtures/
|
||||
F<N>-<short-label>/
|
||||
fixture.json # id, intent, expected outcome, metric, persona
|
||||
diff.patch # the unified diff under review
|
||||
context/ # repo files visible as surrounding context (NOT in diff)
|
||||
files/ # post-change versions of touched files
|
||||
skill-snapshot/ # the BASELINE prompt(s), copied verbatim before any edits
|
||||
persona-runner-prompt.md
|
||||
iteration-1/
|
||||
F<N>-old_skill-trial-1/outputs/findings.json
|
||||
F<N>-with_skill-trial-1/outputs/findings.json
|
||||
...
|
||||
iteration-2/
|
||||
...
|
||||
```
|
||||
|
||||
The `persona-runner-prompt.md` defines a strict contract every dispatch obeys: (1) read exactly the four input paths (subagent template, persona profile, diff, context dir), (2) do not fall back to any other version of the prompt, (3) stay in persona, (4) write findings JSON to the specified `OUTPUT_PATH`, (5) no prose in the dispatch reply. This is what makes the workspace reproducible — every cell behaves identically except for the parameters you vary.
|
||||
|
||||
### Steps to apply
|
||||
|
||||
1. **Snapshot the baseline first.** Before editing the prompt, copy the current version to `skill-snapshot/`. Treat this directory as immutable for the duration of the eval.
|
||||
2. **Build a fixture matrix that spans the boundary, not just the easy cases.** Include textbook positives, textbook negatives, an explicit negative control that should not move, and at least two boundary cases that you genuinely cannot predict. Each fixture gets a tiny `fixture.json` documenting intent and expected outcome — this prevents post-hoc rationalization.
|
||||
3. **Spawn cells via parallel Agent dispatches.** Pass the four paths and a unique `OUTPUT_PATH` per cell. Use a simple naming scheme (`F3b-old_skill-trial-2`) so aggregation is `jq` over a glob.
|
||||
4. **Run multiple trials per cell.** Three is the practical minimum; bump to seven or more on cells that look noisy at N=3. (See the variance lesson above for the full argument.)
|
||||
5. **Aggregate with `jq` over the structured field under test** (e.g. `jq '.findings[0].autofix_class'` across iteration directories). Build a summary table indexed by fixture and prompt version.
|
||||
6. **Iterate, then re-snapshot if the prompt changes again.** Each iteration directory is a separate run; `iteration-N` lets you compare across prompt revisions without losing earlier data.
|
||||
|
||||
### Fixture matrix design (generalize the shape, not the content)
|
||||
|
||||
| Fixture role | What it probes | Example from this eval |
|
||||
|---|---|---|
|
||||
| Textbook positive | Should classify the "right" way under both versions | F1 nil guard inside internal helper |
|
||||
| Textbook negative | Should classify the "wrong-direction" way under both versions | F2 off-by-one with parallel pattern |
|
||||
| Explicit negative control | Must not move; if it moves, the prompt has a regression | F6 admin auth gate |
|
||||
| Ambiguous boundary | The reason the eval exists — outcome unknown a priori | F3b orphan code without "no callers" comment |
|
||||
| Stable disagreement candidate | Both versions defensible; you want to see the trade clearly | F4b cross-file Rails service extraction |
|
||||
|
||||
### Reproducibility
|
||||
|
||||
Workspace: `/tmp/safe-auto-eval/` (synthetic fixtures, snapshot baseline, persona-runner prompt, per-iteration outputs).
|
||||
|
||||
To re-run for a different rubric change:
|
||||
1. Snapshot the current `subagent-template.md` (or other persona prompt) to `/tmp/<eval-name>/skill-snapshot/`
|
||||
2. Reuse the persona-runner pattern in `/tmp/safe-auto-eval/persona-runner-prompt.md`
|
||||
3. Spawn one Agent dispatch per cell × trial, parameterized by SUBAGENT_TEMPLATE_PATH (current vs snapshot) + PERSONA_PATH + DIFF_PATH + FILES_DIR + CONTEXT_DIR
|
||||
4. Aggregate via `jq '.findings[0].<field>'` across iteration directories
|
||||
|
||||
The fixtures themselves (`/tmp/safe-auto-eval/fixtures/F{1,1b,2,3,3b,4,4b,5,6}/`) are kept for reproducibility but are not committed — they're synthetic eval scaffolding, not part of the plugin.
|
||||
|
||||
### When to apply this pattern
|
||||
|
||||
Use the harness when:
|
||||
|
||||
- A persona rubric, decision guide, or output-contract section is being edited, and the change is intended to alter classification behavior.
|
||||
- The rubric drives downstream automation (auto-apply gates, fixer dispatch, escalation routing) where wrong classification has real cost.
|
||||
- "Just ship it and watch" is too slow or too risky because the change touches headless or auto-apply paths.
|
||||
- A reported incident motivated the change and you want to validate the hypothesis before shipping (the safe_auto calibration is exactly this case — Issue #686 hypothesized under-classification; the eval falsified that hypothesis but surfaced a different real problem worth fixing).
|
||||
|
||||
Skip or downscale when the change is purely textual (typo, link fix), gated behind a feature flag with low cost-of-bad-ship, or when a real-branch test gives equally clean signal at similar cost (rare for persona-layer changes).
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- PR #685 — `fix(ce-code-review): replace LFG with best-judgment auto-resolve` (the suggested_fix push this builds on)
|
||||
- Issue #686 — the calibration request that prompted the eval
|
||||
- [`docs/solutions/skill-design/confidence-anchored-scoring.md`](./confidence-anchored-scoring.md) — prior art for eval-as-validation in this repo; established the A/B-against-baseline pattern this generalizes
|
||||
- [`docs/solutions/skill-design/ce-doc-review-calibration-patterns.md`](./ce-doc-review-calibration-patterns.md) — see the "Reviewer variance is inherent; single runs aren't baselines" section, which warns of the same N=1 trap scoped to ce-doc-review's tier classification
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: "Offload data processing to bundled scripts to reduce token consumption"
|
||||
category: "skill-design"
|
||||
date: "2026-03-17"
|
||||
tags:
|
||||
- token-optimization
|
||||
- skill-architecture
|
||||
- bundled-scripts
|
||||
- data-processing
|
||||
severity: "high"
|
||||
component: "skills"
|
||||
---
|
||||
|
||||
# Script-First Skill Architecture
|
||||
|
||||
When a skill processes large datasets (session transcripts, log files, configuration inventories), having the model do the processing is a token-expensive anti-pattern. Moving data processing into a bundled script and having the model present the results cuts token usage by 60-75%. (For which language to write that script in, see [prefer-python-over-bash-for-pipeline-scripts](../best-practices/prefer-python-over-bash-for-pipeline-scripts.md); this doc is about *whether* to offload, not *which language*.)
|
||||
|
||||
## Origin
|
||||
|
||||
Learned while building the `claude-permissions-optimizer` skill (since retired from the plugin in favor of `/less-permission-prompts`), which analyzed Claude Code session transcripts to find safe Bash commands to auto-allow. Initial iterations had the model reading JSONL session files, classifying commands against a 370-line reference doc, and normalizing patterns -- averaging 85-115k tokens per run. After moving all processing into the extraction script, runs dropped to ~40k tokens with equivalent output quality. The same pattern is live today in `skills/ce-compound/scripts/session-history/`, whose bundled `extract-metadata.py` / `extract-skeleton.py` scripts do session discovery and classification while the model only presents.
|
||||
|
||||
## The Anti-Pattern: Model-as-Processor
|
||||
|
||||
The default instinct when building a skill that touches data is to have the model read everything into context, parse it, classify it, and reason about it. This works for small inputs but scales terribly:
|
||||
|
||||
- Token usage grows linearly with data volume
|
||||
- Most tokens are spent on mechanical work (parsing JSON, matching patterns, counting frequencies)
|
||||
- Loading reference docs for classification rules inflates context further
|
||||
- The model's actual judgment contributes almost nothing to the classification output
|
||||
|
||||
## The Pattern: Script Produces, Model Presents
|
||||
|
||||
```
|
||||
skills/<skill-name>/
|
||||
SKILL.md # Instructions: run script, present output
|
||||
scripts/
|
||||
process.py # Does ALL data processing, outputs JSON
|
||||
```
|
||||
|
||||
1. **Script does all mechanical work.** Reading files, parsing structured formats, applying classification rules (regex, keyword lists), normalizing results, computing counts. Outputs pre-classified JSON to stdout.
|
||||
|
||||
2. **SKILL.md instructs presentation only.** Run the script, read the JSON, format it for the user. Explicitly prohibit re-classifying, re-parsing, or loading reference files.
|
||||
|
||||
3. **Single source of truth for rules.** Classification logic lives exclusively in the script. The SKILL.md references the script's output categories as given facts but does not define them.
|
||||
|
||||
## Token Impact
|
||||
|
||||
| Approach | Tokens | Reduction |
|
||||
|---|---|---|
|
||||
| Model does everything (read, parse, classify, present) | ~100k | baseline |
|
||||
| Added "do NOT grep session files" instruction | ~84k | 16% |
|
||||
| Script classifies; model still loads reference doc | ~38k | 62% |
|
||||
| Script classifies; model presents only | ~35k | 65% |
|
||||
|
||||
The biggest single win was moving classification into the script. The second was removing the instruction to load the reference file -- once the script handles classification, the reference file is maintenance documentation only.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Apply script-first architecture when a skill meets **any** of these:
|
||||
|
||||
- Processes more than ~50 items or reads files larger than a few KB
|
||||
- Classification rules are deterministic (regex, keyword lists, lookup tables)
|
||||
- Input data follows a consistent schema (JSONL, CSV, structured logs)
|
||||
- The skill runs frequently or feeds into further analysis
|
||||
|
||||
**Do not apply** when:
|
||||
- The skill's core value is the model's judgment (code review, architectural analysis)
|
||||
- Input is unstructured natural language
|
||||
- The dataset is small enough that processing costs are negligible
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
- **Instruction-only optimization.** Adding "don't do X" to SKILL.md without providing a script alternative. The model will find other token-expensive paths to the same result.
|
||||
|
||||
- **Hybrid classification.** Having the script classify some items and the model classify the rest. This still loads context and reference docs. Go all-in on the script. Items the script can't classify should be dropped as "unclassified," not handed to the model.
|
||||
|
||||
- **Dual rule definitions.** Classification rules in both the script AND the SKILL.md. They drift apart, the model may override the script's decisions, and tokens are wasted on re-evaluation. One source of truth.
|
||||
|
||||
## Checklist for Skill Authors
|
||||
|
||||
- [ ] Can the data processing be expressed as deterministic logic (regex, keyword matching, field checks)?
|
||||
- [ ] Script is the single owner of all classification rules
|
||||
- [ ] SKILL.md instructs the model to run the script as its first action
|
||||
- [ ] SKILL.md does not restate or duplicate the script's classification logic
|
||||
- [ ] Script output is structured JSON the model can present directly
|
||||
- [ ] Reference docs exist for maintainers but are never loaded at runtime
|
||||
- [ ] After building, verify the model is not doing any mechanical parsing or rule-application work
|
||||
|
||||
## Related
|
||||
|
||||
- [Reduce plugin context token usage](../../plans/2026-02-08-refactor-reduce-plugin-context-token-usage-plan.md) -- established the principle that descriptions are for discovery, detailed content belongs in the body
|
||||
- [Compound refresh skill improvements](compound-refresh-skill-improvements.md) -- patterns for autonomous skill execution and subagent architecture
|
||||
- [Beta skills framework](beta-skills-framework.md) -- skill organization and rollout conventions
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
title: A strong model can mask a defensive skill-prose fix — control confounds and guard both failure directions when evaluating
|
||||
date: 2026-07-09
|
||||
category: skill-design
|
||||
module: compound-engineering
|
||||
problem_type: best_practice
|
||||
component: ce-commit-push-pr
|
||||
severity: medium
|
||||
applies_when:
|
||||
- Improving an LLM-driven skill's prose (SKILL.md or a reference) and wanting to prove the change helps without regressing
|
||||
- An adversarial skill eval fails to reproduce the failure mode the change was meant to fix
|
||||
- A with-skill vs baseline eval ties on pass-rate and you must decide whether the change is worth keeping
|
||||
- Designing a skill eval that must not trade one failure mode for its opposite
|
||||
tags:
|
||||
- skill-design
|
||||
- skill-eval
|
||||
- skill-creator
|
||||
- eval-methodology
|
||||
- model-capability
|
||||
- confound-control
|
||||
- ce-commit-push-pr
|
||||
related_pr: https://github.com/EveryInc/compound-engineering-plugin/pull/1088
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
`ce-commit-push-pr` sized PR descriptions by diff shape — changed-line count, file extension, visual surface — and its evidence step auto-skipped anything it classed as docs, markdown, CI, or YAML as inert. The fix reframed sizing around **reviewer decision cost** (material claims + reviewer uncertainty + decision-changing evidence) and told the agent to classify files by runtime purpose, not extension.
|
||||
|
||||
The interesting part was not the edit — it was what a rigorous eval of the edit revealed, and how easily that eval could have lied. This documents the methodology, using the change as the worked example.
|
||||
|
||||
## Guidance
|
||||
|
||||
When evaluating a change to an LLM-driven skill's prose, hold four rules:
|
||||
|
||||
1. **Expect a strong model to mask a defensive fix.** A capable model (here, Opus 4.8) already reasons past literal, extension-based heuristics. Adversarial fixtures built specifically to trip the old "docs/YAML/CI → skip" instruction — a production Kubernetes manifest, a CI release workflow that dropped its test gate, a feature-flag flip — **could not reproduce the failure**. On every one, the baseline skill already treated the change as production behavior and surfaced the material claims. The misleading instruction was inert *for this model*.
|
||||
|
||||
2. **A tie on pass-rate can still hide a real improvement.** Because the baseline handled the adversarial cases, binary assertions tied at ~100% both arms. The genuine delta was qualitative: the new prose produced sharper risk-framing and better structure (a `## Risk / review notes` block naming that a replica cut and a memory cut *compound*; a per-parameter effect column). Grade the qualitative gap by reading outputs, not only by counting assertion passes — a non-discriminating assertion set is a measurement failure, not evidence of no effect.
|
||||
|
||||
3. **Control confounds the change doesn't touch.** The first eval round showed a spurious difference: one arm emitted a `## New concepts` section the other omitted. That section is governed by the concept-teaching gate — code the diff never touched — so the difference was run-to-run judgment variance on a borderline call, orthogonal to the change. Confirm suspected confounds mechanically (`git diff` proves the change doesn't touch the relevant path), then **neutralize them** (disable the gate in the fixtures via committed config in the base, so it's invisible to the diff) and re-run. An uncontrolled confound can manufacture or mask a delta.
|
||||
|
||||
4. **Guard both failure directions, not just the one you're fixing.** A change that fixes under-description can silently cause over-inflation. Include guardrail fixtures (a trivial dep bump, a large-but-mechanical rename) whose job is to fail if the new prose bloats simple diffs — not just target fixtures that prove the intended win. Here the guardrails held (trivial stayed a one-liner, a 6-file rename stayed ~250 chars), and a later verbosity check caught real length creep (deploy-YAML at 1904 chars) that a trim then cut −37% with no lost claims.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Without these rules the eval reaches the wrong verdict twice over. Rule 1 stops you concluding "the change did nothing" — its value is real but **defensive**: removing a misleading instruction matters most for the weaker harnesses the skill also ships to (this plugin is authored once and converted for Codex, Cursor, Gemini, etc.), even when the strongest model overrides it. Rule 2 stops the tied pass-rate from burying the enhancement. Rule 3 stops a confound from being read as signal. Rule 4 stops you shipping the opposite defect you just introduced.
|
||||
|
||||
The net decision was: keep the change. On a strong model it is an enhancement (good → better); across the harness matrix it is a cheap, safe defensive fix (~+1k tokens/PR) — not the "broken → fixed" rescue the hypothesis assumed.
|
||||
|
||||
## When to Apply
|
||||
|
||||
Any time you change skill *prose* (as opposed to a bundled script or parser, which `bun test` exercises directly) and want more than a vibe check. Skill-prose behavior can't be confirmed by reasoning about the diff, and — per this repo's convention — the plugin loader caches skill definitions at session start, so use `skill-creator`'s eval workflow (it injects the on-disk skill into a fresh subagent) rather than dispatching the cached skill in-session.
|
||||
|
||||
## Examples
|
||||
|
||||
The reusable shape is a **three-round eval**, each round answering the objection the previous round raised:
|
||||
|
||||
```
|
||||
Round 1 — baseline: old skill vs new, representative diff shapes, n=1.
|
||||
Establishes "no regression" and surfaces the first confound.
|
||||
Round 2 — controlled + adversarial: neutralize the confound (gate off in
|
||||
fixtures), add fixtures engineered to break the baseline
|
||||
(deploy YAML, CI workflow, flag flip) AND guardrail fixtures
|
||||
that fail on over-inflation. Reveals the baseline is strong.
|
||||
Round 3 — cost pass: re-run after a trim; confirm the length/verbosity
|
||||
cost came down (−19% total, deploy-YAML −37%) with no material
|
||||
claim lost.
|
||||
```
|
||||
|
||||
Confound neutralization, concretely — commit the gate-off config to the fixture's **base** so both branches share it and it never appears in the diff under test:
|
||||
|
||||
```bash
|
||||
git init -q -b main
|
||||
mkdir -p .compound-engineering
|
||||
printf 'pr_teaching_section: false\n' > .compound-engineering/config.local.yaml
|
||||
git add -A && git commit -q -m 'initial' # gate-off lives in base, invisible to the feature diff
|
||||
git checkout -q -b feature
|
||||
# ...make the change under test...
|
||||
```
|
||||
|
||||
Guardrail assertion, concretely — a fixture whose pass condition is *shortness*, so the eval fails loudly if the new prose bloats a simple change:
|
||||
|
||||
```python
|
||||
# mechanical rename across 6 files must NOT balloon
|
||||
assert body_chars <= 700 and file_or_bullet_refs <= 3, "over-inflated a mechanical diff"
|
||||
```
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
title: "Watch-loop skills need a blocked-external terminal state for fork-PR CI approval gates"
|
||||
category: skill-design
|
||||
date: 2026-07-11
|
||||
module: skills/ce-babysit-pr
|
||||
problem_type: design_pattern
|
||||
component: tooling
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "Building or reviewing a watch-loop skill that polls PR/CI status until merge-ready"
|
||||
- "A PR is a fork->upstream submission from a non-maintainer where CI requires a maintainer's approval to run"
|
||||
- "A status signal (e.g., all_checks_ok) is derived only from statusCheckRollup or check-runs"
|
||||
- "Classifying a stalled watch-loop into needs-human vs in-progress vs a new blocked-external state"
|
||||
- "Designing pipeline-mode termination behavior for an externally-gated, unbounded-timeline block"
|
||||
tags:
|
||||
- ce-babysit-pr
|
||||
- watch-loop
|
||||
- github-actions
|
||||
- fork-pr
|
||||
- ci-gating
|
||||
- blocked-external
|
||||
- pipeline-mode
|
||||
- false-green
|
||||
related_components:
|
||||
- development_workflow
|
||||
- tooling
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
`ce-babysit-pr` watches an open GitHub PR toward merge, driven by a deterministic snapshot engine (`skills/ce-babysit-pr/scripts/pr-snapshot`) that fetches both event streams — CI checks and review threads — on every tick and emits an actionable set the skill's prose acts on.
|
||||
|
||||
On `stablyai/orca#8238` (fork `tmchow/orca` -> upstream `stablyai/orca`), the loop reported `all_checks_ok: true` while the PR's real CI had never started. The single check the rollup could see (a lightweight `Track` job) had passed; the substantive workflows were sitting behind GitHub's fork-PR security gate, waiting for a base-repo maintainer to click "Approve and run workflows." Nothing in `gh pr view --json statusCheckRollup` reflected that — a workflow run that is `action_required`/`waiting` has not yet produced a check-run at all, so it is structurally absent from the rollup, not present-and-pending. The only surviving signal was `mergeStateStatus: UNSTABLE`, plus a manual `gh api repos/{owner}/{repo}/actions/runs?head_sha=...` query that showed the gated run.
|
||||
|
||||
This is a partial-truth API problem: `statusCheckRollup` answers "what do check-runs say," not "has CI actually run." A snapshot engine that treats the rollup as the complete picture will report green on a PR whose real CI is dormant. It is also the common open-source case: a contributor who is not a maintainer of the upstream repo cannot approve their own fork-PR workflow run — the block is on a third party, and the wait is unbounded (hours to days), so a naive loop either reports a false green or spins forever.
|
||||
|
||||
## Guidance
|
||||
|
||||
**Query a second, independent source for the fact the primary API can't see, and fold it into the "ok" computation rather than layering it on top.** `fetch_awaiting_approval(owner, name, head)` in `pr-snapshot` hits `repos/{owner}/{name}/actions/runs?head_sha=<head>` and counts runs with `status in (action_required, waiting)` or `conclusion == action_required` — best-effort, returning `0` on any API/permission failure rather than failing the tick. `diff()` folds that count into `all_checks_ok`:
|
||||
|
||||
```python
|
||||
awaiting_approval = cur.get("awaiting_approval", 0)
|
||||
all_checks_ok = checks_terminal and not has_failing and bool(cur["checks"]) and awaiting_approval == 0
|
||||
blocked_external = awaiting_approval > 0 and not has_failing and not actionable_threads
|
||||
```
|
||||
|
||||
`all_checks_ok` cannot go true while a workflow is gated, and `checks_awaiting_approval` / `blocked_external` are emitted as first-class fields alongside the actionable set — not inferred later from prose.
|
||||
|
||||
**Model the discovered state as its own terminal condition, not a variant of an existing one.** "Blocked on a third party neither the loop nor the user controls, for an unbounded time" is neither `needs-human` (the user *could* act — resolve a thread, fix code) nor transient `in-progress` (bounded, will resolve on its own soon). `ce-babysit-pr`'s Step 3 gives it a dedicated stop condition, "Blocked on external CI approval":
|
||||
|
||||
- Interactive: recommend stopping by default, report the wait as open-ended (hours to days, and review is often *also* gated on CI so there may be nothing to watch), give the exact resume command (`/ce-babysit-pr <url>`), and offer exactly **one** bounded alternative — poll at ~30-minute cadence, hard-capped at 24h, resuming full babysitting the moment CI clears.
|
||||
- Pipeline/unattended: don't ask, don't spin — return a `blocked-external` residual with the run URL and terminate.
|
||||
- **Never auto-approve the run.** That click is the maintainer's security gate; the skill treats it as out of scope for automation entirely.
|
||||
|
||||
**Gate on push-capability, not fork-status.** A PR from your own fork is still fully drivable — you can push fixes to the head. The distinction that matters is whether *this loop* can push to the PR's head ref, not whether the head repo happens to be a fork; read state from the base repo, push fixes to the head/fork.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
A watch loop over an external system's API inherits that API's blind spots. If the loop's "done"/"ok" signal is built directly from one endpoint's fields without checking whether that endpoint has a known gap, the loop will confidently report green on a red (or in this case, not-yet-run) reality — the worst failure mode for an autonomous monitor, because the false-positive is silent and looks identical to genuine success.
|
||||
|
||||
The fix generalizes past this one API: **don't let a single endpoint's completeness assumption become your loop's completeness assumption.** Cross-check with a second source when you know (or suspect) the primary source omits a state, and make the omission visible in the engine's boolean, not just left to a human noticing `mergeStateStatus` disagrees with the rollup.
|
||||
|
||||
The second half — modeling `blocked-external` as its own condition — matters because collapsing it into `needs-human` would tell the user "something needs your attention" when nothing does (they can't approve someone else's maintainer gate), and collapsing it into ordinary in-progress waiting would make the loop spin indefinitely (or the user assume it will resolve on the loop's normal cadence) on a wait that can run for days. A watch loop's stop-condition taxonomy needs a distinct bucket for "blocked on someone outside this conversation, unbounded," with its own handback shape (bounded-poll offer + resume command, no auto-resolution).
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Building or reviewing any polling/watch-loop engine (CI watchers, deploy monitors, review-status trackers) that derives an "ok"/"done" signal from a single external API's fields.
|
||||
- The external system has a known async-approval or moderation gate (fork-PR CI approval, app review, manual QA sign-off) where the gated item may not appear in the primary status feed until *after* approval.
|
||||
- Designing stop conditions for an autonomous loop: check whether "blocked on a third party, unbounded timeline, no one in this loop can act" is already collapsing into an existing bucket (`needs-human`, `in-progress`) rather than getting its own condition and handback UX.
|
||||
- Any handback path that could plausibly auto-approve, auto-retry, or auto-bypass a security/approval gate on the user's behalf — treat approval gates as categorically out of scope for automation, not just "risky."
|
||||
|
||||
## Examples
|
||||
|
||||
**Before:** `pr-snapshot` computed `all_checks_ok` solely from `statusCheckRollup`:
|
||||
|
||||
```python
|
||||
all_checks_ok = checks_terminal and not has_failing and bool(cur["checks"])
|
||||
```
|
||||
|
||||
On the gated fork PR, the rollup contained only the one ungated `Track` check (`COMPLETED`/`SUCCESS`), so `checks_terminal=True`, `has_failing=False`, `all_checks_ok=True` — reported ready while the substantive CI had never started. Nothing in the snapshot distinguished this from an actually-green PR, and `ce-babysit-pr` had no stop condition for it, so a "blocked on maintainer approval" PR would either be reported as merge-ready or fall through to the generic `needs-human` bucket with no bounded-wait guidance and no explicit refusal to auto-approve.
|
||||
|
||||
**After:** `fetch_awaiting_approval` queries the Actions runs API independently and `diff()` wires it into both the ok-signal and a dedicated flag:
|
||||
|
||||
```python
|
||||
awaiting_approval = cur.get("awaiting_approval", 0)
|
||||
all_checks_ok = checks_terminal and not has_failing and bool(cur["checks"]) and awaiting_approval == 0
|
||||
blocked_external = awaiting_approval > 0 and not has_failing and not actionable_threads
|
||||
```
|
||||
|
||||
`SKILL.md` Step 3 adds "Blocked on external CI approval" as its own stop condition with the interactive default-to-stop + bounded 30-min/24h-cap alternative + resume-command handback, and an explicit never-auto-approve rule. Verified live on `stablyai/orca#8238`: the snapshot returned `blocked_external: true` and `checks_awaiting_approval: 1`, `all_checks_ok: false`; the skill recommended stopping, offered the bounded watch, printed the resume command, and did not attempt to approve the run. The regression test `tests/ce-babysit-pr-snapshot.test.ts` ("a fork-PR workflow awaiting maintainer approval blocks 'all_checks_ok' and flags blocked_external") locks this in via `--fetch-file` with `awaiting_approval: 1`, asserting `checks_awaiting_approval === 1`, `has_failing_checks === false`, `all_checks_ok === false`, and `blocked_external === true`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Git workflow skills need explicit state machines for branch, push, and PR state](./git-workflow-skills-need-explicit-state-machines.md) — the same meta-pattern in a sibling git/gh skill family: an implicitly-assumed state (there, PR/branch existence and cleanliness) silently produces a wrong boolean instead of surfacing as an explicit state. This learning is that pattern applied to "a check-run existing at all," in a watch loop.
|
||||
- `docs/plans/pipeline-mode-contract-and-lfg-babysit-consolidation.md` — the originating design contract that defines *pipeline mode*, the *durable residual* (never blocking, never silently applying), and *terminate on a bound / return structured status*. The `blocked-external` handback here is a direct instantiation of those rules for the fork/CI-approval case.
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
title: "Manual release-please with GitHub Releases for plugin and marketplace releases"
|
||||
category: workflow
|
||||
date: 2026-03-17
|
||||
last_refreshed: 2026-06-23
|
||||
created: 2026-03-17
|
||||
severity: process
|
||||
component: release-automation
|
||||
tags:
|
||||
- release-please
|
||||
- github-releases
|
||||
- marketplace
|
||||
- plugin-versioning
|
||||
- ci
|
||||
- automation
|
||||
- release-process
|
||||
---
|
||||
|
||||
# Manual release-please with GitHub Releases for plugin and marketplace releases
|
||||
|
||||
## Problem
|
||||
|
||||
The repo had one automated release path for the npm CLI, but the actual release model was fragmented across root package metadata, plugin manifests, marketplace catalogs, and release-note surfaces. That made it easy for plugin manifests, marketplace metadata, and computed counts to drift out of sync.
|
||||
|
||||
## Solution
|
||||
|
||||
Use release-please manifest mode with one standing release PR and explicit component ownership.
|
||||
|
||||
Current components:
|
||||
|
||||
- `compound-engineering` package/plugin at root package path `.`
|
||||
- `marketplace` for `.claude-plugin/marketplace.json`
|
||||
- `cursor-marketplace` for `.cursor-plugin/marketplace.json`
|
||||
|
||||
The root `compound-engineering` package is now the plugin package. It owns the CLI/tooling code, root plugin manifests, native harness metadata, and `skills/`.
|
||||
|
||||
Key decisions:
|
||||
|
||||
- Keep release timing manual: the actual release happens when the generated release PR is merged.
|
||||
- Keep release PR maintenance automatic on pushes to `main`.
|
||||
- Use GitHub release PRs and GitHub Releases as the canonical release-notes surface.
|
||||
- Keep PR title scopes optional; use file paths to determine affected components.
|
||||
- Keep `AGENTS.md` canonical and `CLAUDE.md`/`GEMINI.md` as compatibility shims.
|
||||
|
||||
## Critical constraint discovered
|
||||
|
||||
Release-please does not allow package changelog paths that traverse upward with `..`. A multi-component repo cannot force subpackage release entries back into one shared root changelog file using `../../CHANGELOG.md` or `../CHANGELOG.md`.
|
||||
|
||||
The practical fix:
|
||||
|
||||
- Treat GitHub Releases as the canonical release-notes surface.
|
||||
- Keep root `CHANGELOG.md` as a pointer to GitHub Releases.
|
||||
- Validate `.github/release-please-config.json` in CI so unsupported changelog paths fail before the workflow reaches GitHub Actions.
|
||||
|
||||
## Resulting release process
|
||||
|
||||
1. Normal feature PRs merge to `main`.
|
||||
2. The `Release PR` workflow updates one standing release PR for the repo.
|
||||
3. Additional releasable merges accumulate into that release PR.
|
||||
4. Maintainers can inspect the standing release PR or run the manual preview flow.
|
||||
5. The actual release happens only when the generated release PR is merged.
|
||||
6. Component-specific release notes are published via GitHub Releases such as `compound-engineering-vX.Y.Z`, `marketplace-vX.Y.Z`, and `cursor-marketplace-vX.Y.Z`.
|
||||
|
||||
## Component rules
|
||||
|
||||
PR title determines release intent:
|
||||
|
||||
- `feat` -> minor
|
||||
- `fix`, `perf`, `revert` -> patch
|
||||
- `refactor` -> visible in release notes under `Refactoring`, but not release-driving unless breaking or explicitly overridden
|
||||
- `!` -> major; do not use without explicit maintainer confirmation
|
||||
|
||||
File paths determine component ownership:
|
||||
|
||||
| Component | Paths |
|
||||
|---|---|
|
||||
| `compound-engineering` | `skills/`, `src/`, `tests/`, `package.json`, root plugin manifests, `.opencode/`, `.pi/`, `.agy/plugin.json`, `README.md`, instruction shims |
|
||||
| `marketplace` | `.claude-plugin/marketplace.json` |
|
||||
| `cursor-marketplace` | `.cursor-plugin/marketplace.json` |
|
||||
|
||||
Docs-only, CI-only, and build-only changes are non-releasable unless their conventional type says otherwise and a releasable component path changed.
|
||||
|
||||
## Examples
|
||||
|
||||
### Plugin-only release
|
||||
|
||||
- A `fix:` PR changes `skills/ce-plan/SKILL.md`
|
||||
- `compound-engineering` bumps
|
||||
- marketplace versions remain untouched
|
||||
|
||||
### Root packaging release
|
||||
|
||||
- A `fix:` PR changes `.codex-plugin/plugin.json` or `.agy/plugin.json`
|
||||
- `compound-engineering` bumps because those files are root package/plugin extra-files
|
||||
- `bun run release:validate` must pass so all root package/plugin versions remain aligned
|
||||
|
||||
### Marketplace-only release
|
||||
|
||||
- A marketplace catalog entry changes in `.claude-plugin/marketplace.json`
|
||||
- `marketplace` bumps
|
||||
- plugin versions do not need to bump just because the catalog changed
|
||||
|
||||
## Release notes model
|
||||
|
||||
- Pending release state is visible in one standing release PR.
|
||||
- Published release history is canonical in GitHub Releases.
|
||||
- Root `CHANGELOG.md` is only a pointer to GitHub Releases and is not the canonical source for new releases.
|
||||
|
||||
## Key files
|
||||
|
||||
- `.github/release-please-config.json`
|
||||
- `.github/.release-please-manifest.json`
|
||||
- `.github/workflows/release-pr.yml`
|
||||
- `.github/workflows/release-preview.yml`
|
||||
- `.github/workflows/ci.yml`
|
||||
- `src/release/components.ts`
|
||||
- `src/release/metadata.ts`
|
||||
- `scripts/release/preview.ts`
|
||||
- `scripts/release/sync-metadata.ts`
|
||||
- `scripts/release/validate.ts`
|
||||
- `AGENTS.md`
|
||||
- `CLAUDE.md`
|
||||
- `GEMINI.md`
|
||||
|
||||
## Prevention
|
||||
|
||||
- Keep release authority in CI only.
|
||||
- Do not reintroduce local maintainer-only release flows or hand-managed version bumps.
|
||||
- Keep root package/plugin manifests aligned through release-please extra-files, not manual edits.
|
||||
- Do not try to force multi-component release notes back into one committed changelog file.
|
||||
- Run `bun run release:validate` whenever plugin inventories, release-owned descriptions, marketplace entries, or root plugin manifests may have changed.
|
||||
- Prefer maintained CI actions over custom validation when a generic concern does not need repo-specific logic.
|
||||
|
||||
## Validation checklist
|
||||
|
||||
Before merge:
|
||||
|
||||
- Confirm PR title passes semantic validation.
|
||||
- Run `bun test`.
|
||||
- Run `bun run release:validate`.
|
||||
- Run `bun run release:preview ...` for representative changed files when release-component selection is non-obvious.
|
||||
|
||||
Before merging a generated release PR:
|
||||
|
||||
- Verify untouched components are unchanged.
|
||||
- Verify marketplace components only bump for marketplace-level changes.
|
||||
- Verify root package/plugin extra-files share the same version.
|
||||
|
||||
After merging a generated release PR:
|
||||
|
||||
- Confirm no recursive follow-up release PR appears containing only generated churn.
|
||||
- Confirm the expected component GitHub Releases were created and release-owned metadata matches the released components.
|
||||
|
||||
## Related docs
|
||||
|
||||
- `docs/solutions/plugin-versioning-requirements.md`
|
||||
- `docs/solutions/adding-converter-target-providers.md`
|
||||
- `AGENTS.md`
|
||||
@@ -0,0 +1,153 @@
|
||||
---
|
||||
title: "Release-please version drift recovery"
|
||||
category: workflow
|
||||
date: 2026-04-24
|
||||
last_refreshed: 2026-06-20
|
||||
created: 2026-04-24
|
||||
severity: high
|
||||
component: release-automation
|
||||
problem_type: workflow_issue
|
||||
tags:
|
||||
- release-please
|
||||
- version-drift
|
||||
- plugin-versioning
|
||||
- recovery-playbook
|
||||
- extra-files
|
||||
---
|
||||
|
||||
# Release-please version drift recovery
|
||||
|
||||
## Problem
|
||||
|
||||
Manual edits to a release-managed version field cause drift that:
|
||||
|
||||
- Breaks `bun run release:validate` on PR CI.
|
||||
- Can cause version regression on the next release-please run if left uncorrected.
|
||||
- Is easy to introduce accidentally during a feature commit.
|
||||
- Has multiple recovery paths with different user-impact trade-offs.
|
||||
|
||||
This doc is the playbook when drift is detected. It exists because investigating from scratch takes significant effort and the wrong choice can make things worse.
|
||||
|
||||
## File relationship map
|
||||
|
||||
The current root-native repo has three release components. Release-please reads `.github/.release-please-manifest.json` and writes each package's configured `extra-files`.
|
||||
|
||||
```text
|
||||
.github/.release-please-manifest.json
|
||||
├── "." -> compound-engineering package/plugin (v = X.Y.Z)
|
||||
├── ".claude-plugin" -> Claude marketplace (v = M.N.O)
|
||||
└── ".cursor-plugin" -> Cursor marketplace (v = P.Q.R)
|
||||
|
||||
.github/release-please-config.json
|
||||
└── packages
|
||||
├── "." extra-files
|
||||
│ ├── package.json
|
||||
│ ├── .claude-plugin/plugin.json
|
||||
│ ├── .cursor-plugin/plugin.json
|
||||
│ ├── .codex-plugin/plugin.json
|
||||
│ └── gemini-extension.json
|
||||
├── ".claude-plugin" extra-files
|
||||
│ └── marketplace.json ($.metadata.version)
|
||||
└── ".cursor-plugin" extra-files
|
||||
└── marketplace.json ($.metadata.version)
|
||||
```
|
||||
|
||||
Key invariants:
|
||||
|
||||
- Every extra-file inside the root `.` component must share the same plugin version.
|
||||
- Marketplace components are independent; their metadata versions do not move with every plugin release.
|
||||
- `bun run release:validate` enforces root package/plugin version parity, marketplace parity, Codex manifest shape, and description sync.
|
||||
- The repo no longer has separate `cli`, `plugins/compound-engineering`, or `coding-tutor` release components.
|
||||
|
||||
## How release-please tracks versions
|
||||
|
||||
Release-please treats the manifest as the source of truth for "last released version per component." Extra-files are outputs that release-please writes during a release PR. Under normal operation, humans do not hand-edit the manifest or extra-files. Drift is the state where that guarantee has been violated.
|
||||
|
||||
## Drift detection
|
||||
|
||||
`bun run release:validate` runs on PRs and pushes to `main`. It fails when:
|
||||
|
||||
- Root package/plugin versions disagree across `package.json`, `.claude-plugin/plugin.json`, `.cursor-plugin/plugin.json`, `.codex-plugin/plugin.json`, and `gemini-extension.json`.
|
||||
- Marketplace plugin lists diverge across Claude, Cursor, and Codex marketplace metadata.
|
||||
- A Codex manifest is missing required fields or points at a missing `skills/` directory.
|
||||
- Release-owned descriptions drift across plugin manifests or marketplace entries.
|
||||
- `release-as` pins become stale relative to the base-branch manifest.
|
||||
|
||||
Important: a state where all extra-files agree at X.Y.Z but the manifest still says W.X.Y can pass some local checks and still cause the next release PR to regress versions. Check the manifest when investigating drift.
|
||||
|
||||
## Recovery decision tree
|
||||
|
||||
```text
|
||||
release:validate reports drift
|
||||
|
|
||||
v
|
||||
1. Identify the affected component:
|
||||
- "." root package/plugin
|
||||
- ".claude-plugin" marketplace
|
||||
- ".cursor-plugin" marketplace
|
||||
|
|
||||
v
|
||||
2. Compare:
|
||||
- extra-files vs each other within that component
|
||||
- extra-files vs .github/.release-please-manifest.json
|
||||
- any active release-as pins in .github/release-please-config.json
|
||||
|
|
||||
v
|
||||
3. Is anyone installed at the drifted higher version?
|
||||
- Yes or unknown -> forward-sync
|
||||
- Verified no -> backward-revert
|
||||
```
|
||||
|
||||
### Path A: Forward-sync
|
||||
|
||||
Use when any user may have installed the drifted version locally. For the root `.` component, update every root extra-file to the drifted higher version:
|
||||
|
||||
- `package.json`
|
||||
- `.claude-plugin/plugin.json`
|
||||
- `.cursor-plugin/plugin.json`
|
||||
- `.codex-plugin/plugin.json`
|
||||
- `gemini-extension.json`
|
||||
- `.github/.release-please-manifest.json` entry for `.`
|
||||
|
||||
For marketplace drift, sync the affected marketplace `marketplace.json` metadata version and matching manifest entry.
|
||||
|
||||
Why the manifest edit is necessary: without it, the next release-please run reads the stale last-released value and may write a lower next version to extra-files, regressing users at the forward-synced version.
|
||||
|
||||
### Path B: Backward-revert
|
||||
|
||||
Use only when you can verify no user is installed at the drifted version. Revert the drifted extra-file(s) down to the manifest value, leaving the manifest unchanged.
|
||||
|
||||
This is fewer files, but it risks user regression if verification was wrong. Default to Path A when in doubt.
|
||||
|
||||
### Path C: `release-as` pin
|
||||
|
||||
Use when you want release-please itself to drive the recovery via a normal release PR. Forward-sync extra-files up to the drifted version, add a temporary `"release-as": "<drifted+1>"` pin for the affected package, and let the release PR bump above the drifted value.
|
||||
|
||||
This has cleanup overhead: remove the pin after the release PR lands. Prefer Path A unless there is a specific reason the release PR should own the bump.
|
||||
|
||||
## Summary
|
||||
|
||||
| Path | Files changed | When to use | Risk |
|
||||
|---|---|---|---|
|
||||
| A -- forward-sync | Extra-files + manifest | Anyone might be at drifted version | Low if executed completely |
|
||||
| B -- backward-revert | Drifted extra-file(s) only | Verified no one has drifted version | User regression if wrong |
|
||||
| C -- `release-as` pin | Extra-files + config pin + later cleanup | Want release-please to drive recovery | Stale pin risk |
|
||||
|
||||
## Prevention
|
||||
|
||||
Direct-to-main merges are the root cause. They bypass PR CI, release validation, tests, and semantic title checks.
|
||||
|
||||
Branch protection on `main` is the enforcement. The `test` status check must be required before merge, and admin bypass should be reserved for true emergencies.
|
||||
|
||||
Optional guards:
|
||||
|
||||
- Dedicated CI detecting manual version bumps by non-release PRs.
|
||||
- A pre-commit or pre-push hook running `bun run release:validate`.
|
||||
|
||||
## Related docs
|
||||
|
||||
- `docs/solutions/workflow/manual-release-please-github-releases.md` -- big-picture release model.
|
||||
- `docs/solutions/plugin-versioning-requirements.md` -- plugin-scoped contributor rules.
|
||||
- `AGENTS.md` -- repo-level release versioning rules.
|
||||
- `.github/release-please-config.json` -- package and extra-file configuration.
|
||||
- `src/release/metadata.ts` -- metadata sync and validation implementation.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "Stale local base contamination in multi-session branch creation"
|
||||
category: workflow
|
||||
date: 2026-04-27
|
||||
created: 2026-04-27
|
||||
severity: medium
|
||||
component: ce-commit-push-pr
|
||||
problem_type: workflow_issue
|
||||
tags:
|
||||
- branching
|
||||
- multi-agent
|
||||
- multi-session
|
||||
- pre-push
|
||||
- stacked-prs
|
||||
- contamination
|
||||
---
|
||||
|
||||
# Stale local base contamination in multi-session branch creation
|
||||
|
||||
## Problem
|
||||
|
||||
When multiple agent sessions (Claude Code, Cursor, Codex, plus any humans) share one local clone, local `<default-branch>` can drift relative to its remote counterpart. Two specific drifts cause downstream pain:
|
||||
|
||||
1. **Local default behind remote.** Another session pushed and merged work; this session's local `main` doesn't know yet.
|
||||
2. **Local default ahead of remote with unpushed work.** Another session committed locally to `main`, or merged a feature branch into local `main`, before pushing — and never pushed those commits to `origin/main`.
|
||||
|
||||
When a session creates a feature branch from local `main` while drift type 2 holds, the new branch silently inherits the unpushed work. The eventual PR opens looking clean to the originating session but appears contaminated on GitHub. Resolving it requires force-push surgery during PR review.
|
||||
|
||||
This came in as [issue #707](https://github.com/EveryInc/compound-engineering-plugin/issues/707).
|
||||
|
||||
## Why post-facto detection is the wrong tool
|
||||
|
||||
The intuitive fix is to detect the contamination before pushing or before opening a PR. Two detection approaches were considered and rejected:
|
||||
|
||||
### Approach A: surface foreign commit authors
|
||||
|
||||
Read `git log <base>..HEAD --pretty=format:'%h %ae %s'` and warn when any commit's author email differs from `git config user.email`.
|
||||
|
||||
Catches the cross-author case (cherry-picks, teammate-authored work) but misses the dominant scenario: multi-agent setups where every session uses the same `user.email`. The check fires on intentional cherry-picks and stays silent on the actual contamination pattern.
|
||||
|
||||
### Approach B: cross-branch reachability
|
||||
|
||||
For each commit in `<base>..HEAD`, check whether it is reachable from any other `origin/*` ref. If yes, treat as suspect.
|
||||
|
||||
Authorship-agnostic, so it catches same-user contamination. But the signal it measures — "this commit is on another remote branch" — is the **defining characteristic** of stacked-PR workflows, where parent commits in the stack are intentionally shared with sibling branches. Tools like Graphite and git-spice rely on this. With GitHub-native stacked PRs moving toward general availability and likely broad adoption, the false-positive rate moves from "narrow population" to "majority of pushes for sophisticated users." The check would invert from useful signal to default noise.
|
||||
|
||||
You can patch around it (parse stack metadata from PR base refs) but the patches multiply with every adjacent workflow (first push before PR exists, multi-level stacks, fork-based stacks). Each patch is a heuristic that will be wrong somewhere.
|
||||
|
||||
## Solution
|
||||
|
||||
Prevent at branch creation rather than detect at push or PR time.
|
||||
|
||||
`ce-commit-push-pr` Step 4 — the branch-creation path used when the user invokes the skill while on the default branch with working-tree changes — was changed from:
|
||||
|
||||
```bash
|
||||
git checkout -b <branch-name>
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```bash
|
||||
git fetch --no-tags origin <base>
|
||||
git checkout -b <branch-name> origin/<base>
|
||||
```
|
||||
|
||||
with a graceful fallback to the local-base form when the fetch fails (offline, restricted network, expired auth). The fallback is documented to the user so they know base freshness was not verified.
|
||||
|
||||
This makes the skill's branch-creation path safe by construction:
|
||||
|
||||
- Drift type 1 (local behind remote): the new branch starts at fresh remote `<base>`, not stale local `<base>`.
|
||||
- Drift type 2 (local ahead of remote with unpushed work): unpushed local commits stay on local `<base>` (recoverable via reflog or branch ref); the new feature branch starts clean.
|
||||
|
||||
The principle generalizes cleanly to stacked PRs: when a user wants to stack on top of an open PR, the same `git fetch && git checkout -b <name> origin/<parent>` pattern works — `<parent>` is just a different ref. Nothing about prevention depends on detecting "is this commit suspicious."
|
||||
|
||||
## What this does not cover
|
||||
|
||||
- **Branches created outside the skill.** Users who run `git checkout -b` manually, or whose IDE creates branches without fetching, can still produce contaminated branches. The skill's path becomes safe; the user's general workflow is not. A pre-push hook (which the original reporter installed) covers this case — opt-in hooks remain a reasonable user-side mitigation.
|
||||
- **Already-contaminated branches.** Once a branch carries foreign commits, this change does nothing for it. Recovery is still manual: identify the foreign commits, drop them via interactive rebase or `git reset` to a clean base, force-push.
|
||||
- **Step 1 branch-creation paths with different semantics.** When the user is on the default branch with unpushed commits and asks to create a feature branch to "rescue" those commits, the desired behavior is to carry the local commits onto the new branch — opposite of the Step 4 case. Step 1's behavior is unchanged.
|
||||
|
||||
## User-side mitigations
|
||||
|
||||
For workflows where branch creation happens outside the skill, recommend:
|
||||
|
||||
- `git switch -c <name> origin/<base>` instead of `git checkout -b <name> <base>`
|
||||
- A `git config --global alias.nb '!f() { git fetch origin "${2:-main}" && git switch -c "$1" "origin/${2:-main}"; }; f'` style alias
|
||||
- An opt-in pre-push hook that compares HEAD's parent chain against `origin/<base>` for unexpected commits — useful for individual users, but not shipped from this plugin because the cost of getting stacked-PR semantics right in a hook outweighs the benefit at the plugin level
|
||||
|
||||
## Why we did not ship a detection check at all
|
||||
|
||||
The reporter framed their issue as "a pattern, not a request for a merge." Taking that at its word and acting on the structural signal — a real failure mode worth a permanent fix — produced this outcome:
|
||||
|
||||
- One small preventive change in the skill that is safe by construction
|
||||
- A documented pattern with rationale for future readers
|
||||
- No behavioral prompt added to a heavy-traffic skill
|
||||
- No detection heuristic that risks being obsoleted by stacked PRs
|
||||
|
||||
A detection check at push or PR time was not free even when scoped tightly: it adds a prompt to a frictionless workflow, false-positives on legitimate workflows that share commits across branches, and would require ongoing tuning as stacked-PR conventions evolve. Prevention at the right layer avoids all of that.
|
||||
Reference in New Issue
Block a user