chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,165 @@
|
||||
# Claude Code Skills Tests
|
||||
|
||||
Automated tests for superpowers skills using Claude Code CLI.
|
||||
|
||||
## Overview
|
||||
|
||||
This test suite verifies that skills are loaded correctly and Claude follows them as expected. Tests invoke Claude Code in headless mode (`claude -p`) and verify the behavior.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Claude Code CLI installed and in PATH (`claude --version` should work)
|
||||
- Local superpowers plugin installed (see main README for installation)
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Run all fast tests (recommended):
|
||||
```bash
|
||||
./run-skill-tests.sh
|
||||
```
|
||||
|
||||
### Run integration tests (slow, 10-30 minutes):
|
||||
```bash
|
||||
./run-skill-tests.sh --integration
|
||||
```
|
||||
|
||||
### Run specific test:
|
||||
```bash
|
||||
./run-skill-tests.sh --test test-subagent-driven-development.sh
|
||||
```
|
||||
|
||||
### Run with verbose output:
|
||||
```bash
|
||||
./run-skill-tests.sh --verbose
|
||||
```
|
||||
|
||||
### Set custom timeout:
|
||||
```bash
|
||||
./run-skill-tests.sh --timeout 1800 # 30 minutes for integration tests
|
||||
```
|
||||
|
||||
## Test Structure
|
||||
|
||||
### test-helpers.sh
|
||||
Common functions for skills testing:
|
||||
- `run_claude "prompt" [timeout]` - Run Claude with prompt
|
||||
- `assert_contains output pattern name` - Verify pattern exists
|
||||
- `assert_not_contains output pattern name` - Verify pattern absent
|
||||
- `assert_count output pattern count name` - Verify exact count
|
||||
- `assert_order output pattern_a pattern_b name` - Verify order
|
||||
- `create_test_project` - Create temp test directory
|
||||
- `create_test_plan project_dir` - Create sample plan file
|
||||
|
||||
### Test Files
|
||||
|
||||
Each test file:
|
||||
1. Sources `test-helpers.sh`
|
||||
2. Runs Claude Code with specific prompts
|
||||
3. Verifies expected behavior using assertions
|
||||
4. Returns 0 on success, non-zero on failure
|
||||
|
||||
## Example Test
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
echo "=== Test: My Skill ==="
|
||||
|
||||
# Ask Claude about the skill
|
||||
output=$(run_claude "What does the my-skill skill do?" 30)
|
||||
|
||||
# Verify response
|
||||
assert_contains "$output" "expected behavior" "Skill describes behavior"
|
||||
|
||||
echo "=== All tests passed ==="
|
||||
```
|
||||
|
||||
## Current Tests
|
||||
|
||||
### Fast Tests (run by default)
|
||||
|
||||
#### test-subagent-driven-development.sh
|
||||
Tests skill content and requirements (~2 minutes):
|
||||
- Skill loading and accessibility
|
||||
- Workflow ordering (spec compliance before code quality)
|
||||
- Self-review requirements documented
|
||||
- Plan reading efficiency documented
|
||||
- Spec compliance reviewer skepticism documented
|
||||
- Review loops documented
|
||||
- Task context provision documented
|
||||
|
||||
### Integration Tests (use --integration flag)
|
||||
|
||||
#### test-subagent-driven-development-integration.sh
|
||||
Full workflow execution test (~10-30 minutes):
|
||||
- Creates real test project with Node.js setup
|
||||
- Creates implementation plan with 2 tasks
|
||||
- Executes plan using subagent-driven-development
|
||||
- Verifies actual behaviors:
|
||||
- Plan read once at start (not per task)
|
||||
- Full task text provided in subagent prompts
|
||||
- Subagents perform self-review before reporting
|
||||
- Spec compliance review happens before code quality
|
||||
- Spec reviewer reads code independently
|
||||
- Working implementation is produced
|
||||
- Tests pass
|
||||
- Proper git commits created
|
||||
|
||||
**What it tests:**
|
||||
- The workflow actually works end-to-end
|
||||
- Our improvements are actually applied
|
||||
- Subagents follow the skill correctly
|
||||
- Final code is functional and tested
|
||||
|
||||
#### test-worktree-native-preference.sh
|
||||
RED-GREEN-REFACTOR validation for the using-git-worktrees skill (~5 minutes):
|
||||
- RED: skill without Step 1a — agent should use `git worktree add`
|
||||
- GREEN: skill with Step 1a — agent should use the native EnterWorktree tool
|
||||
- PRESSURE: same as GREEN under urgency framing with pre-existing `.worktrees/`
|
||||
- Drill scenario `worktree-creation-under-pressure.yaml` covers the PRESSURE phase only
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
1. Create new test file: `test-<skill-name>.sh`
|
||||
2. Source test-helpers.sh
|
||||
3. Write tests using `run_claude` and assertions
|
||||
4. Add to test list in `run-skill-tests.sh`
|
||||
5. Make executable: `chmod +x test-<skill-name>.sh`
|
||||
|
||||
## Timeout Considerations
|
||||
|
||||
- Default timeout: 5 minutes per test
|
||||
- Claude Code may take time to respond
|
||||
- Adjust with `--timeout` if needed
|
||||
- Tests should be focused to avoid long runs
|
||||
|
||||
## Debugging Failed Tests
|
||||
|
||||
With `--verbose`, you'll see full Claude output:
|
||||
```bash
|
||||
./run-skill-tests.sh --verbose --test test-subagent-driven-development.sh
|
||||
```
|
||||
|
||||
Without verbose, only failures show output.
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
To run in CI:
|
||||
```bash
|
||||
# Run with explicit timeout for CI environments
|
||||
./run-skill-tests.sh --timeout 900
|
||||
|
||||
# Exit code 0 = success, non-zero = failure
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Tests verify skill *instructions*, not full execution
|
||||
- Full workflow tests would be very slow
|
||||
- Focus on verifying key skill requirements
|
||||
- Tests should be deterministic
|
||||
- Avoid testing implementation details
|
||||
Executable
+168
@@ -0,0 +1,168 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Analyze token usage from Claude Code session transcripts.
|
||||
Breaks down usage by main session and individual subagents.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
|
||||
def analyze_main_session(filepath):
|
||||
"""Analyze a session file and return token usage broken down by agent."""
|
||||
main_usage = {
|
||||
'input_tokens': 0,
|
||||
'output_tokens': 0,
|
||||
'cache_creation': 0,
|
||||
'cache_read': 0,
|
||||
'messages': 0
|
||||
}
|
||||
|
||||
# Track usage per subagent
|
||||
subagent_usage = defaultdict(lambda: {
|
||||
'input_tokens': 0,
|
||||
'output_tokens': 0,
|
||||
'cache_creation': 0,
|
||||
'cache_read': 0,
|
||||
'messages': 0,
|
||||
'description': None
|
||||
})
|
||||
|
||||
with open(filepath, 'r') as f:
|
||||
for line in f:
|
||||
try:
|
||||
data = json.loads(line)
|
||||
|
||||
# Main session assistant messages
|
||||
if data.get('type') == 'assistant' and 'message' in data:
|
||||
main_usage['messages'] += 1
|
||||
msg_usage = data['message'].get('usage', {})
|
||||
main_usage['input_tokens'] += msg_usage.get('input_tokens', 0)
|
||||
main_usage['output_tokens'] += msg_usage.get('output_tokens', 0)
|
||||
main_usage['cache_creation'] += msg_usage.get('cache_creation_input_tokens', 0)
|
||||
main_usage['cache_read'] += msg_usage.get('cache_read_input_tokens', 0)
|
||||
|
||||
# Subagent tool results
|
||||
if data.get('type') == 'user' and 'toolUseResult' in data:
|
||||
result = data['toolUseResult']
|
||||
if 'usage' in result and 'agentId' in result:
|
||||
agent_id = result['agentId']
|
||||
usage = result['usage']
|
||||
|
||||
# Get description from prompt if available
|
||||
if subagent_usage[agent_id]['description'] is None:
|
||||
prompt = result.get('prompt', '')
|
||||
# Extract first line as description
|
||||
first_line = prompt.split('\n')[0] if prompt else f"agent-{agent_id}"
|
||||
if first_line.startswith('You are '):
|
||||
first_line = first_line[8:] # Remove "You are "
|
||||
subagent_usage[agent_id]['description'] = first_line[:60]
|
||||
|
||||
subagent_usage[agent_id]['messages'] += 1
|
||||
subagent_usage[agent_id]['input_tokens'] += usage.get('input_tokens', 0)
|
||||
subagent_usage[agent_id]['output_tokens'] += usage.get('output_tokens', 0)
|
||||
subagent_usage[agent_id]['cache_creation'] += usage.get('cache_creation_input_tokens', 0)
|
||||
subagent_usage[agent_id]['cache_read'] += usage.get('cache_read_input_tokens', 0)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return main_usage, dict(subagent_usage)
|
||||
|
||||
def format_tokens(n):
|
||||
"""Format token count with thousands separators."""
|
||||
return f"{n:,}"
|
||||
|
||||
def calculate_cost(usage, input_cost_per_m=3.0, output_cost_per_m=15.0):
|
||||
"""Calculate estimated cost in dollars."""
|
||||
total_input = usage['input_tokens'] + usage['cache_creation'] + usage['cache_read']
|
||||
input_cost = total_input * input_cost_per_m / 1_000_000
|
||||
output_cost = usage['output_tokens'] * output_cost_per_m / 1_000_000
|
||||
return input_cost + output_cost
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: analyze-token-usage.py <session-file.jsonl>")
|
||||
sys.exit(1)
|
||||
|
||||
main_session_file = sys.argv[1]
|
||||
|
||||
if not Path(main_session_file).exists():
|
||||
print(f"Error: Session file not found: {main_session_file}")
|
||||
sys.exit(1)
|
||||
|
||||
# Analyze the session
|
||||
main_usage, subagent_usage = analyze_main_session(main_session_file)
|
||||
|
||||
print("=" * 100)
|
||||
print("TOKEN USAGE ANALYSIS")
|
||||
print("=" * 100)
|
||||
print()
|
||||
|
||||
# Print breakdown
|
||||
print("Usage Breakdown:")
|
||||
print("-" * 100)
|
||||
print(f"{'Agent':<15} {'Description':<35} {'Msgs':>5} {'Input':>10} {'Output':>10} {'Cache':>10} {'Cost':>8}")
|
||||
print("-" * 100)
|
||||
|
||||
# Main session
|
||||
cost = calculate_cost(main_usage)
|
||||
print(f"{'main':<15} {'Main session (coordinator)':<35} "
|
||||
f"{main_usage['messages']:>5} "
|
||||
f"{format_tokens(main_usage['input_tokens']):>10} "
|
||||
f"{format_tokens(main_usage['output_tokens']):>10} "
|
||||
f"{format_tokens(main_usage['cache_read']):>10} "
|
||||
f"${cost:>7.2f}")
|
||||
|
||||
# Subagents (sorted by agent ID)
|
||||
for agent_id in sorted(subagent_usage.keys()):
|
||||
usage = subagent_usage[agent_id]
|
||||
cost = calculate_cost(usage)
|
||||
desc = usage['description'] or f"agent-{agent_id}"
|
||||
print(f"{agent_id:<15} {desc:<35} "
|
||||
f"{usage['messages']:>5} "
|
||||
f"{format_tokens(usage['input_tokens']):>10} "
|
||||
f"{format_tokens(usage['output_tokens']):>10} "
|
||||
f"{format_tokens(usage['cache_read']):>10} "
|
||||
f"${cost:>7.2f}")
|
||||
|
||||
print("-" * 100)
|
||||
|
||||
# Calculate totals
|
||||
total_usage = {
|
||||
'input_tokens': main_usage['input_tokens'],
|
||||
'output_tokens': main_usage['output_tokens'],
|
||||
'cache_creation': main_usage['cache_creation'],
|
||||
'cache_read': main_usage['cache_read'],
|
||||
'messages': main_usage['messages']
|
||||
}
|
||||
|
||||
for usage in subagent_usage.values():
|
||||
total_usage['input_tokens'] += usage['input_tokens']
|
||||
total_usage['output_tokens'] += usage['output_tokens']
|
||||
total_usage['cache_creation'] += usage['cache_creation']
|
||||
total_usage['cache_read'] += usage['cache_read']
|
||||
total_usage['messages'] += usage['messages']
|
||||
|
||||
total_input = total_usage['input_tokens'] + total_usage['cache_creation'] + total_usage['cache_read']
|
||||
total_tokens = total_input + total_usage['output_tokens']
|
||||
total_cost = calculate_cost(total_usage)
|
||||
|
||||
print()
|
||||
print("TOTALS:")
|
||||
print(f" Total messages: {format_tokens(total_usage['messages'])}")
|
||||
print(f" Input tokens: {format_tokens(total_usage['input_tokens'])}")
|
||||
print(f" Output tokens: {format_tokens(total_usage['output_tokens'])}")
|
||||
print(f" Cache creation tokens: {format_tokens(total_usage['cache_creation'])}")
|
||||
print(f" Cache read tokens: {format_tokens(total_usage['cache_read'])}")
|
||||
print()
|
||||
print(f" Total input (incl cache): {format_tokens(total_input)}")
|
||||
print(f" Total tokens: {format_tokens(total_tokens)}")
|
||||
print()
|
||||
print(f" Estimated cost: ${total_cost:.2f}")
|
||||
print(" (at $3/$15 per M tokens for input/output)")
|
||||
print()
|
||||
print("=" * 100)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+189
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test runner for Claude Code skills
|
||||
# Tests skills by invoking Claude Code CLI and verifying behavior
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
echo "========================================"
|
||||
echo " Claude Code Skills Test Suite"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Repository: $(cd ../.. && pwd)"
|
||||
echo "Test time: $(date)"
|
||||
echo "Claude version: $(claude --version 2>/dev/null || echo 'not found')"
|
||||
echo ""
|
||||
|
||||
# Check if Claude Code is available
|
||||
if ! command -v claude &> /dev/null; then
|
||||
echo "ERROR: Claude Code CLI not found"
|
||||
echo "Install Claude Code first: https://code.claude.com"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Parse command line arguments
|
||||
VERBOSE=false
|
||||
SPECIFIC_TEST=""
|
||||
TIMEOUT=600 # Default 10 minute timeout per test
|
||||
RUN_INTEGRATION=false
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--verbose|-v)
|
||||
VERBOSE=true
|
||||
shift
|
||||
;;
|
||||
--test|-t)
|
||||
SPECIFIC_TEST="$2"
|
||||
shift 2
|
||||
;;
|
||||
--timeout)
|
||||
TIMEOUT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--integration|-i)
|
||||
RUN_INTEGRATION=true
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [options]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --verbose, -v Show verbose output"
|
||||
echo " --test, -t NAME Run only the specified test"
|
||||
echo " --timeout SECONDS Set timeout per test (default: 300)"
|
||||
echo " --integration, -i Run integration tests (slow, 10-30 min)"
|
||||
echo " --help, -h Show this help"
|
||||
echo ""
|
||||
echo "Tests:"
|
||||
echo " test-subagent-driven-development.sh Test skill loading and requirements"
|
||||
echo ""
|
||||
echo "Integration Tests (use --integration):"
|
||||
echo " test-subagent-driven-development-integration.sh Full workflow execution"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# List of skill tests to run (fast unit tests)
|
||||
tests=(
|
||||
"test-worktree-path-policy.sh"
|
||||
"test-sdd-workspace.sh"
|
||||
"test-subagent-driven-development.sh"
|
||||
)
|
||||
|
||||
# Integration tests (slow, full execution)
|
||||
integration_tests=(
|
||||
"test-subagent-driven-development-integration.sh"
|
||||
)
|
||||
|
||||
# Add integration tests if requested
|
||||
if [ "$RUN_INTEGRATION" = true ]; then
|
||||
tests+=("${integration_tests[@]}")
|
||||
fi
|
||||
|
||||
# Filter to specific test if requested
|
||||
if [ -n "$SPECIFIC_TEST" ]; then
|
||||
tests=("$SPECIFIC_TEST")
|
||||
fi
|
||||
|
||||
# Track results
|
||||
passed=0
|
||||
failed=0
|
||||
skipped=0
|
||||
|
||||
# Run each test
|
||||
for test in "${tests[@]}"; do
|
||||
echo "----------------------------------------"
|
||||
echo "Running: $test"
|
||||
echo "----------------------------------------"
|
||||
|
||||
test_path="$SCRIPT_DIR/$test"
|
||||
|
||||
if [ ! -f "$test_path" ]; then
|
||||
echo " [SKIP] Test file not found: $test"
|
||||
skipped=$((skipped + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if [ ! -x "$test_path" ]; then
|
||||
echo " Making $test executable..."
|
||||
chmod +x "$test_path"
|
||||
fi
|
||||
|
||||
start_time=$(date +%s)
|
||||
|
||||
if [ "$VERBOSE" = true ]; then
|
||||
if timeout "$TIMEOUT" bash "$test_path"; then
|
||||
end_time=$(date +%s)
|
||||
duration=$((end_time - start_time))
|
||||
echo ""
|
||||
echo " [PASS] $test (${duration}s)"
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
exit_code=$?
|
||||
end_time=$(date +%s)
|
||||
duration=$((end_time - start_time))
|
||||
echo ""
|
||||
if [ $exit_code -eq 124 ]; then
|
||||
echo " [FAIL] $test (timeout after ${TIMEOUT}s)"
|
||||
else
|
||||
echo " [FAIL] $test (${duration}s)"
|
||||
fi
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
else
|
||||
# Capture output for non-verbose mode
|
||||
if output=$(timeout "$TIMEOUT" bash "$test_path" 2>&1); then
|
||||
end_time=$(date +%s)
|
||||
duration=$((end_time - start_time))
|
||||
echo " [PASS] (${duration}s)"
|
||||
passed=$((passed + 1))
|
||||
else
|
||||
exit_code=$?
|
||||
end_time=$(date +%s)
|
||||
duration=$((end_time - start_time))
|
||||
if [ $exit_code -eq 124 ]; then
|
||||
echo " [FAIL] (timeout after ${TIMEOUT}s)"
|
||||
else
|
||||
echo " [FAIL] (${duration}s)"
|
||||
fi
|
||||
echo ""
|
||||
echo " Output:"
|
||||
echo "$output" | sed 's/^/ /'
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
done
|
||||
|
||||
# Print summary
|
||||
echo "========================================"
|
||||
echo " Test Results Summary"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo " Passed: $passed"
|
||||
echo " Failed: $failed"
|
||||
echo " Skipped: $skipped"
|
||||
echo ""
|
||||
|
||||
if [ "$RUN_INTEGRATION" = false ] && [ ${#integration_tests[@]} -gt 0 ]; then
|
||||
echo "Note: Integration tests were not run (they take 10-30 minutes)."
|
||||
echo "Use --integration flag to run full workflow execution tests."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
if [ $failed -gt 0 ]; then
|
||||
echo "STATUS: FAILED"
|
||||
exit 1
|
||||
else
|
||||
echo "STATUS: PASSED"
|
||||
exit 0
|
||||
fi
|
||||
Executable
+202
@@ -0,0 +1,202 @@
|
||||
#!/usr/bin/env bash
|
||||
# Helper functions for Claude Code skill tests
|
||||
|
||||
# Run Claude Code with a prompt and capture output
|
||||
# Usage: run_claude "prompt text" [timeout_seconds] [allowed_tools]
|
||||
run_claude() {
|
||||
local prompt="$1"
|
||||
local timeout="${2:-60}"
|
||||
local allowed_tools="${3:-}"
|
||||
local output_file=$(mktemp)
|
||||
|
||||
# Build command as an argv array so timeout wraps claude directly.
|
||||
local cmd=(claude -p "$prompt")
|
||||
if [ -n "$allowed_tools" ]; then
|
||||
cmd+=(--allowed-tools="$allowed_tools")
|
||||
fi
|
||||
|
||||
# Run Claude in headless mode with timeout
|
||||
if timeout "$timeout" "${cmd[@]}" > "$output_file" 2>&1; then
|
||||
cat "$output_file"
|
||||
rm -f "$output_file"
|
||||
return 0
|
||||
else
|
||||
local exit_code=$?
|
||||
cat "$output_file" >&2
|
||||
rm -f "$output_file"
|
||||
return $exit_code
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if output contains a pattern
|
||||
# Usage: assert_contains "output" "pattern" "test name"
|
||||
assert_contains() {
|
||||
local output="$1"
|
||||
local pattern="$2"
|
||||
local test_name="${3:-test}"
|
||||
|
||||
if echo "$output" | grep -q "$pattern"; then
|
||||
echo " [PASS] $test_name"
|
||||
return 0
|
||||
else
|
||||
echo " [FAIL] $test_name"
|
||||
echo " Expected to find: $pattern"
|
||||
echo " In output:"
|
||||
echo "$output" | sed 's/^/ /'
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if output does NOT contain a pattern
|
||||
# Usage: assert_not_contains "output" "pattern" "test name"
|
||||
assert_not_contains() {
|
||||
local output="$1"
|
||||
local pattern="$2"
|
||||
local test_name="${3:-test}"
|
||||
|
||||
if echo "$output" | grep -q "$pattern"; then
|
||||
echo " [FAIL] $test_name"
|
||||
echo " Did not expect to find: $pattern"
|
||||
echo " In output:"
|
||||
echo "$output" | sed 's/^/ /'
|
||||
return 1
|
||||
else
|
||||
echo " [PASS] $test_name"
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if output matches a count
|
||||
# Usage: assert_count "output" "pattern" expected_count "test name"
|
||||
assert_count() {
|
||||
local output="$1"
|
||||
local pattern="$2"
|
||||
local expected="$3"
|
||||
local test_name="${4:-test}"
|
||||
|
||||
local actual=$(echo "$output" | grep -c "$pattern" || echo "0")
|
||||
|
||||
if [ "$actual" -eq "$expected" ]; then
|
||||
echo " [PASS] $test_name (found $actual instances)"
|
||||
return 0
|
||||
else
|
||||
echo " [FAIL] $test_name"
|
||||
echo " Expected $expected instances of: $pattern"
|
||||
echo " Found $actual instances"
|
||||
echo " In output:"
|
||||
echo "$output" | sed 's/^/ /'
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if pattern A appears before pattern B
|
||||
# Usage: assert_order "output" "pattern_a" "pattern_b" "test name"
|
||||
assert_order() {
|
||||
local output="$1"
|
||||
local pattern_a="$2"
|
||||
local pattern_b="$3"
|
||||
local test_name="${4:-test}"
|
||||
|
||||
# Get line numbers where patterns appear
|
||||
local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1)
|
||||
local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1)
|
||||
|
||||
if [ -z "$line_a" ]; then
|
||||
echo " [FAIL] $test_name: pattern A not found: $pattern_a"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ -z "$line_b" ]; then
|
||||
echo " [FAIL] $test_name: pattern B not found: $pattern_b"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ "$line_a" -lt "$line_b" ]; then
|
||||
echo " [PASS] $test_name (A at line $line_a, B at line $line_b)"
|
||||
return 0
|
||||
else
|
||||
echo " [FAIL] $test_name"
|
||||
echo " Expected '$pattern_a' before '$pattern_b'"
|
||||
echo " But found A at line $line_a, B at line $line_b"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Create a temporary test project directory
|
||||
# Usage: test_project=$(create_test_project)
|
||||
create_test_project() {
|
||||
local test_dir=$(mktemp -d)
|
||||
echo "$test_dir"
|
||||
}
|
||||
|
||||
# Cleanup test project
|
||||
# Usage: cleanup_test_project "$test_dir"
|
||||
cleanup_test_project() {
|
||||
local test_dir="$1"
|
||||
if [ -d "$test_dir" ]; then
|
||||
rm -rf "$test_dir"
|
||||
fi
|
||||
}
|
||||
|
||||
# Create a simple plan file for testing
|
||||
# Usage: create_test_plan "$project_dir" "$plan_name"
|
||||
create_test_plan() {
|
||||
local project_dir="$1"
|
||||
local plan_name="${2:-test-plan}"
|
||||
local plan_file="$project_dir/docs/superpowers/plans/$plan_name.md"
|
||||
|
||||
mkdir -p "$(dirname "$plan_file")"
|
||||
|
||||
cat > "$plan_file" <<'EOF'
|
||||
# Test Implementation Plan
|
||||
|
||||
## Task 1: Create Hello Function
|
||||
|
||||
Create a simple hello function that returns "Hello, World!".
|
||||
|
||||
**File:** `src/hello.js`
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
export function hello() {
|
||||
return "Hello, World!";
|
||||
}
|
||||
```
|
||||
|
||||
**Tests:** Write a test that verifies the function returns the expected string.
|
||||
|
||||
**Verification:** `npm test`
|
||||
|
||||
## Task 2: Create Goodbye Function
|
||||
|
||||
Create a goodbye function that takes a name and returns a goodbye message.
|
||||
|
||||
**File:** `src/goodbye.js`
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
export function goodbye(name) {
|
||||
return `Goodbye, ${name}!`;
|
||||
}
|
||||
```
|
||||
|
||||
**Tests:** Write tests for:
|
||||
- Default name
|
||||
- Custom name
|
||||
- Edge cases (empty string, null)
|
||||
|
||||
**Verification:** `npm test`
|
||||
EOF
|
||||
|
||||
echo "$plan_file"
|
||||
}
|
||||
|
||||
# Export functions for use in tests
|
||||
export -f run_claude
|
||||
export -f assert_contains
|
||||
export -f assert_not_contains
|
||||
export -f assert_count
|
||||
export -f assert_order
|
||||
export -f create_test_project
|
||||
export -f cleanup_test_project
|
||||
export -f create_test_plan
|
||||
Executable
+142
@@ -0,0 +1,142 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for the SDD workspace: scripts/sdd-workspace resolves a self-ignoring
|
||||
# working-tree directory for SDD artifacts, and the SDD scripts write into it.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
SDD_SCRIPTS="$REPO_ROOT/skills/subagent-driven-development/scripts"
|
||||
|
||||
FAILURES=0
|
||||
TEST_ROOT=""
|
||||
|
||||
pass() { echo " [PASS] $1"; }
|
||||
fail() {
|
||||
echo " [FAIL] $1"
|
||||
FAILURES=$((FAILURES + 1))
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "$TEST_ROOT" && -d "$TEST_ROOT" ]]; then
|
||||
rm -rf "$TEST_ROOT"
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
echo "=== Test: sdd-workspace ==="
|
||||
|
||||
TEST_ROOT="$(mktemp -d)"
|
||||
trap cleanup EXIT
|
||||
|
||||
# Resolve repo to its physical path so string comparisons match the
|
||||
# helper's output (git rev-parse --show-toplevel resolves symlinks; on
|
||||
# macOS mktemp lives under /var -> /private/var).
|
||||
git init -q -b main "$TEST_ROOT/repo"
|
||||
local repo
|
||||
repo="$(cd "$TEST_ROOT/repo" && git rev-parse --show-toplevel)"
|
||||
|
||||
local dir
|
||||
dir="$(cd "$repo" && "$SDD_SCRIPTS/sdd-workspace")"
|
||||
|
||||
if [[ "$dir" == "$repo/.superpowers/sdd" ]]; then
|
||||
pass "prints <repo-root>/.superpowers/sdd"
|
||||
else
|
||||
fail "prints <repo-root>/.superpowers/sdd"
|
||||
echo " got: $dir"
|
||||
fi
|
||||
|
||||
if [[ -f "$repo/.superpowers/sdd/.gitignore" && "$(cat "$repo/.superpowers/sdd/.gitignore")" == "*" ]]; then
|
||||
pass "self-ignoring .gitignore created with '*'"
|
||||
else
|
||||
fail "self-ignoring .gitignore created with '*'"
|
||||
fi
|
||||
|
||||
printf 'x\n' > "$repo/.superpowers/sdd/artifact.md"
|
||||
local status
|
||||
status="$(cd "$repo" && git status --porcelain)"
|
||||
if [[ -z "$status" ]]; then
|
||||
pass "workspace invisible to git status"
|
||||
else
|
||||
fail "workspace invisible to git status"
|
||||
echo " status: $status"
|
||||
fi
|
||||
|
||||
( cd "$repo" && git add -A )
|
||||
local staged
|
||||
staged="$(cd "$repo" && git diff --cached --name-only)"
|
||||
if [[ -z "$staged" ]]; then
|
||||
pass "git add -A does not stage the workspace"
|
||||
else
|
||||
fail "git add -A does not stage the workspace"
|
||||
echo " staged: $staged"
|
||||
fi
|
||||
|
||||
cat > "$repo/plan.md" <<'PLAN'
|
||||
# Plan
|
||||
|
||||
## Task 1: First thing
|
||||
|
||||
Do the first thing.
|
||||
PLAN
|
||||
|
||||
local brief_out brief_path
|
||||
brief_out="$(cd "$repo" && "$SDD_SCRIPTS/task-brief" plan.md 1)"
|
||||
brief_path="$(printf '%s\n' "$brief_out" | sed -n 's/^wrote \(.*\): [0-9][0-9]* lines$/\1/p')"
|
||||
case "$brief_path" in
|
||||
"$repo/.superpowers/sdd/"*) pass "task-brief writes its brief under the workspace" ;;
|
||||
*)
|
||||
fail "task-brief writes its brief under the workspace"
|
||||
echo " got: $brief_path"
|
||||
;;
|
||||
esac
|
||||
|
||||
local git_id=(-c user.email=t@example.com -c user.name=t -c commit.gpgsign=false)
|
||||
( cd "$repo" \
|
||||
&& git add plan.md \
|
||||
&& git "${git_id[@]}" commit -qm c1 \
|
||||
&& printf 'y\n' > f && git add f \
|
||||
&& git "${git_id[@]}" commit -qm c2 )
|
||||
local rp_out rp_path
|
||||
rp_out="$(cd "$repo" && "$SDD_SCRIPTS/review-package" HEAD~1 HEAD)"
|
||||
rp_path="$(printf '%s\n' "$rp_out" | sed -n 's/^wrote \(.*\): [0-9].*$/\1/p')"
|
||||
case "$rp_path" in
|
||||
"$repo/.superpowers/sdd/"*) pass "review-package writes its diff under the workspace" ;;
|
||||
*)
|
||||
fail "review-package writes its diff under the workspace"
|
||||
echo " got: $rp_path"
|
||||
;;
|
||||
esac
|
||||
|
||||
# --- Worktree isolation: a linked worktree resolves its own workspace ---
|
||||
local wt="$TEST_ROOT/wt"
|
||||
( cd "$repo" && git worktree add -q "$wt" -b wt-feature )
|
||||
local wt_root wt_dir
|
||||
wt_root="$(cd "$wt" && git rev-parse --show-toplevel)"
|
||||
wt_dir="$(cd "$wt" && "$SDD_SCRIPTS/sdd-workspace")"
|
||||
if [[ "$wt_dir" == "$wt_root/.superpowers/sdd" && "$wt_dir" != "$dir" ]]; then
|
||||
pass "linked worktree resolves its own distinct workspace"
|
||||
else
|
||||
fail "linked worktree resolves its own distinct workspace"
|
||||
echo " main: $dir"
|
||||
echo " wt: $wt_dir"
|
||||
fi
|
||||
|
||||
printf 'y\n' > "$wt/.superpowers/sdd/artifact.md"
|
||||
local wt_status
|
||||
wt_status="$(cd "$wt" && git status --porcelain)"
|
||||
if [[ -z "$wt_status" ]]; then
|
||||
pass "worktree workspace invisible to git status"
|
||||
else
|
||||
fail "worktree workspace invisible to git status"
|
||||
echo " status: $wt_status"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "$FAILURES" -ne 0 ]]; then
|
||||
echo "FAILED: $FAILURES assertion(s)."
|
||||
exit 1
|
||||
fi
|
||||
echo "PASS"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,333 @@
|
||||
#!/usr/bin/env bash
|
||||
# Integration Test: subagent-driven-development workflow
|
||||
# Actually executes a plan and verifies the new workflow behaviors
|
||||
#
|
||||
# Drill coverage: evals/scenarios/sdd-rejects-extra-features.yaml covers the
|
||||
# YAGNI enforcement subset (forbidden exports + reviewer-as-gate semantics)
|
||||
# and is stricter on that axis. This bash test additionally asserts:
|
||||
# - >=3 git commits (initial + per-task commits, exercising SDD's
|
||||
# commit-per-task workflow shape)
|
||||
# - >=2 Claude Code subagent dispatches via Agent or Task (drill only asserts >=1)
|
||||
# - Claude Code task-tracking tool usage (drill makes no assertion)
|
||||
# - test/math.test.js exists (drill relies on `npm test` succeeding)
|
||||
# - analyze-token-usage.py token-budget telemetry
|
||||
# Kept until those assertions are added to drill or explicitly retired.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
echo "========================================"
|
||||
echo " Integration Test: subagent-driven-development"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "This test executes a real plan using the skill and verifies:"
|
||||
echo " 1. Plan is read once (not per task)"
|
||||
echo " 2. Full task text provided to subagents"
|
||||
echo " 3. Subagents perform self-review"
|
||||
echo " 4. Spec compliance review before code quality"
|
||||
echo " 5. Review loops when issues found"
|
||||
echo " 6. Spec reviewer reads code independently"
|
||||
echo ""
|
||||
echo "WARNING: This test may take 10-30 minutes to complete."
|
||||
echo ""
|
||||
|
||||
# Create test project
|
||||
TEST_PROJECT=$(create_test_project)
|
||||
echo "Test project: $TEST_PROJECT"
|
||||
|
||||
# Trap to cleanup
|
||||
trap "cleanup_test_project $TEST_PROJECT" EXIT
|
||||
|
||||
# Set up minimal Node.js project
|
||||
cd "$TEST_PROJECT"
|
||||
|
||||
cat > package.json <<'EOF'
|
||||
{
|
||||
"name": "test-project",
|
||||
"version": "1.0.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "node --test"
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
mkdir -p src test docs/superpowers/plans
|
||||
|
||||
# Create a simple implementation plan
|
||||
cat > docs/superpowers/plans/implementation-plan.md <<'EOF'
|
||||
# Test Implementation Plan
|
||||
|
||||
This is a minimal plan to test the subagent-driven-development workflow.
|
||||
|
||||
## Task 1: Create Add Function
|
||||
|
||||
Create a function that adds two numbers.
|
||||
|
||||
**File:** `src/math.js`
|
||||
|
||||
**Requirements:**
|
||||
- Function named `add`
|
||||
- Takes two parameters: `a` and `b`
|
||||
- Returns the sum of `a` and `b`
|
||||
- Export the function
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
export function add(a, b) {
|
||||
return a + b;
|
||||
}
|
||||
```
|
||||
|
||||
**Tests:** Create `test/math.test.js` that verifies:
|
||||
- `add(2, 3)` returns `5`
|
||||
- `add(0, 0)` returns `0`
|
||||
- `add(-1, 1)` returns `0`
|
||||
|
||||
**Verification:** `npm test`
|
||||
|
||||
## Task 2: Create Multiply Function
|
||||
|
||||
Create a function that multiplies two numbers.
|
||||
|
||||
**File:** `src/math.js` (add to existing file)
|
||||
|
||||
**Requirements:**
|
||||
- Function named `multiply`
|
||||
- Takes two parameters: `a` and `b`
|
||||
- Returns the product of `a` and `b`
|
||||
- Export the function
|
||||
- DO NOT add any extra features (like power, divide, etc.)
|
||||
|
||||
**Implementation:**
|
||||
```javascript
|
||||
export function multiply(a, b) {
|
||||
return a * b;
|
||||
}
|
||||
```
|
||||
|
||||
**Tests:** Add to `test/math.test.js`:
|
||||
- `multiply(2, 3)` returns `6`
|
||||
- `multiply(0, 5)` returns `0`
|
||||
- `multiply(-2, 3)` returns `-6`
|
||||
|
||||
**Verification:** `npm test`
|
||||
EOF
|
||||
|
||||
# Initialize git repo
|
||||
git init --quiet
|
||||
git config user.email "test@test.com"
|
||||
git config user.name "Test User"
|
||||
git add .
|
||||
git commit -m "Initial commit" --quiet
|
||||
|
||||
echo ""
|
||||
echo "Project setup complete. Starting execution..."
|
||||
echo ""
|
||||
|
||||
# Run Claude with subagent-driven-development
|
||||
# Capture full output to analyze
|
||||
OUTPUT_FILE="$TEST_PROJECT/claude-output.txt"
|
||||
|
||||
# Create prompt file
|
||||
cat > "$TEST_PROJECT/prompt.txt" <<'EOF'
|
||||
I want you to execute the implementation plan at docs/superpowers/plans/implementation-plan.md using the subagent-driven-development skill.
|
||||
|
||||
IMPORTANT: Follow the skill exactly. I will be verifying that you:
|
||||
1. Read the plan once at the beginning
|
||||
2. Provide full task text to subagents (don't make them read files)
|
||||
3. Ensure subagents do self-review before reporting
|
||||
4. Run spec compliance review before code quality review
|
||||
5. Use review loops when issues are found
|
||||
|
||||
Begin now. Execute the plan.
|
||||
EOF
|
||||
|
||||
# Note: We use a longer timeout since this is integration testing
|
||||
# Use --allowed-tools to enable tool usage in headless mode
|
||||
PROMPT="Execute the implementation plan at docs/superpowers/plans/implementation-plan.md using the subagent-driven-development skill.
|
||||
|
||||
IMPORTANT: Follow the skill exactly. I will be verifying that you:
|
||||
1. Read the plan once at the beginning
|
||||
2. Provide full task text to subagents (don't make them read files)
|
||||
3. Ensure subagents do self-review before reporting
|
||||
4. Run spec compliance review before code quality review
|
||||
5. Use review loops when issues are found
|
||||
|
||||
Begin now. Execute the plan."
|
||||
|
||||
PLUGIN_DIR=$(cd "$SCRIPT_DIR/../.." && pwd)
|
||||
|
||||
# Run claude from inside the test project so its session JSONL lands in a
|
||||
# project-specific directory under ~/.claude/projects/, isolated from any
|
||||
# other concurrent claude sessions.
|
||||
echo "Running Claude (plugin-dir: $PLUGIN_DIR, cwd: $TEST_PROJECT)..."
|
||||
echo "================================================================================"
|
||||
cd "$TEST_PROJECT" && timeout 1800 claude -p "$PROMPT" --plugin-dir "$PLUGIN_DIR" --allowed-tools=all --permission-mode bypassPermissions 2>&1 | tee "$OUTPUT_FILE" || {
|
||||
echo ""
|
||||
echo "================================================================================"
|
||||
echo "EXECUTION FAILED (exit code: $?)"
|
||||
exit 1
|
||||
}
|
||||
echo "================================================================================"
|
||||
|
||||
echo ""
|
||||
echo "Execution complete. Analyzing results..."
|
||||
echo ""
|
||||
|
||||
# Find the session transcript. Because we ran claude from $TEST_PROJECT (a
|
||||
# unique tmp dir), its sessions live in their own ~/.claude/projects/ folder
|
||||
# and we can pick the most-recent one without racing other concurrent sessions.
|
||||
# Resolve the real path because macOS mktemp returns /var/... but claude
|
||||
# normalizes it to /private/var/... when naming the project dir.
|
||||
TEST_PROJECT_REAL=$(cd "$TEST_PROJECT" && pwd -P)
|
||||
# Claude normalizes the cwd to a directory name by replacing every non-alphanumeric
|
||||
# character with `-` (so `_`, `.`, `/` all become `-`).
|
||||
SESSION_DIR="$HOME/.claude/projects/$(echo "$TEST_PROJECT_REAL" | sed 's|[^a-zA-Z0-9]|-|g')"
|
||||
# `|| true` prevents pipefail killing the script if ls gets SIGPIPE'd by head.
|
||||
SESSION_FILE=$(ls -t "$SESSION_DIR"/*.jsonl 2>/dev/null | head -1 || true)
|
||||
|
||||
if [ -z "$SESSION_FILE" ]; then
|
||||
echo "ERROR: Could not find session transcript file"
|
||||
echo "Looked in: $SESSION_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Analyzing session transcript: $(basename "$SESSION_FILE")"
|
||||
echo ""
|
||||
|
||||
# Verification tests
|
||||
FAILED=0
|
||||
|
||||
echo "=== Verification Tests ==="
|
||||
echo ""
|
||||
|
||||
# Test 1: Skill was invoked
|
||||
echo "Test 1: Skill tool invoked..."
|
||||
if grep -q '"name":"Skill".*"skill":"superpowers:subagent-driven-development"' "$SESSION_FILE"; then
|
||||
echo " [PASS] subagent-driven-development skill was invoked"
|
||||
else
|
||||
echo " [FAIL] Skill was not invoked"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 2: Subagents were used (Agent / Task tool — name varies by harness version)
|
||||
echo "Test 2: Subagents dispatched..."
|
||||
task_count=$(grep -cE '"name":"(Agent|Task)"' "$SESSION_FILE" || echo "0")
|
||||
if [ "$task_count" -ge 2 ]; then
|
||||
echo " [PASS] $task_count subagents dispatched"
|
||||
else
|
||||
echo " [FAIL] Only $task_count subagent(s) dispatched (expected >= 2)"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 3: Claude Code task-tracking tool was used
|
||||
echo "Test 3: Task tracking..."
|
||||
todo_count=$(grep -cE '"name":"(TodoWrite|TaskCreate|TaskUpdate|TaskList|TaskGet)"' "$SESSION_FILE" || echo "0")
|
||||
if [ "$todo_count" -ge 1 ]; then
|
||||
echo " [PASS] Task tracking used $todo_count time(s)"
|
||||
else
|
||||
echo " [FAIL] No Claude Code task-tracking tool used"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 6: Implementation actually works
|
||||
echo "Test 6: Implementation verification..."
|
||||
if [ -f "$TEST_PROJECT/src/math.js" ]; then
|
||||
echo " [PASS] src/math.js created"
|
||||
|
||||
if grep -q "export function add" "$TEST_PROJECT/src/math.js"; then
|
||||
echo " [PASS] add function exists"
|
||||
else
|
||||
echo " [FAIL] add function missing"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
|
||||
if grep -q "export function multiply" "$TEST_PROJECT/src/math.js"; then
|
||||
echo " [PASS] multiply function exists"
|
||||
else
|
||||
echo " [FAIL] multiply function missing"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
else
|
||||
echo " [FAIL] src/math.js not created"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
|
||||
if [ -f "$TEST_PROJECT/test/math.test.js" ]; then
|
||||
echo " [PASS] test/math.test.js created"
|
||||
else
|
||||
echo " [FAIL] test/math.test.js not created"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
|
||||
# Try running tests
|
||||
if cd "$TEST_PROJECT" && npm test > test-output.txt 2>&1; then
|
||||
echo " [PASS] Tests pass"
|
||||
else
|
||||
echo " [FAIL] Tests failed"
|
||||
cat test-output.txt
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 7: Git commits show proper workflow
|
||||
echo "Test 7: Git commit history..."
|
||||
commit_count=$(git -C "$TEST_PROJECT" log --oneline | wc -l)
|
||||
if [ "$commit_count" -gt 2 ]; then # Initial + at least 2 task commits
|
||||
echo " [PASS] Multiple commits created ($commit_count total)"
|
||||
else
|
||||
echo " [FAIL] Too few commits ($commit_count, expected >2)"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Test 8: Check for extra features (spec compliance should catch)
|
||||
echo "Test 8: No extra features added (spec compliance)..."
|
||||
if grep -q "export function divide\|export function power\|export function subtract" "$TEST_PROJECT/src/math.js" 2>/dev/null; then
|
||||
echo " [WARN] Extra features found (spec review should have caught this)"
|
||||
# Not failing on this as it tests reviewer effectiveness
|
||||
else
|
||||
echo " [PASS] No extra features added"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Token Usage Analysis
|
||||
echo "========================================="
|
||||
echo " Token Usage Analysis"
|
||||
echo "========================================="
|
||||
echo ""
|
||||
python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"
|
||||
echo ""
|
||||
|
||||
# Summary
|
||||
echo "========================================"
|
||||
echo " Test Summary"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
|
||||
if [ $FAILED -eq 0 ]; then
|
||||
echo "STATUS: PASSED"
|
||||
echo "All verification tests passed!"
|
||||
echo ""
|
||||
echo "The subagent-driven-development skill correctly:"
|
||||
echo " ✓ Reads plan once at start"
|
||||
echo " ✓ Provides full task text to subagents"
|
||||
echo " ✓ Enforces self-review"
|
||||
echo " ✓ Runs spec compliance before code quality"
|
||||
echo " ✓ Spec reviewer verifies independently"
|
||||
echo " ✓ Produces working implementation"
|
||||
exit 0
|
||||
else
|
||||
echo "STATUS: FAILED"
|
||||
echo "Failed $FAILED verification tests"
|
||||
echo ""
|
||||
echo "Output saved to: $OUTPUT_FILE"
|
||||
echo ""
|
||||
echo "Review the output to see what went wrong."
|
||||
exit 1
|
||||
fi
|
||||
+179
@@ -0,0 +1,179 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test: subagent-driven-development skill
|
||||
# Verifies that the skill is loaded and follows correct workflow
|
||||
#
|
||||
# No drill coverage: this test asks the agent to *describe* SDD (string-
|
||||
# matches its verbal explanation against expected keywords like
|
||||
# "self-review", "skeptical", "worktree", "Step 1", "loop"). Drill scenarios
|
||||
# test behavior (real subagent dispatch, plan-following, review loops),
|
||||
# not description-recall. Kept by design.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
CLAUDE_PROMPT_TIMEOUT="${CLAUDE_PROMPT_TIMEOUT:-90}"
|
||||
|
||||
echo "=== Test: subagent-driven-development skill ==="
|
||||
echo ""
|
||||
|
||||
# Test 1: Verify skill can be loaded
|
||||
echo "Test 1: Skill loading..."
|
||||
|
||||
output=$(run_claude "What is the subagent-driven-development skill? Describe its key steps briefly." "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "subagent-driven-development\|Subagent-Driven Development\|Subagent Driven" "Skill is recognized"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "Load Plan\|read.*plan\|extract.*tasks" "Mentions loading plan"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 2: Verify skill describes correct workflow order
|
||||
echo "Test 2: Workflow ordering..."
|
||||
|
||||
output=$(run_claude "In the subagent-driven-development skill, what comes first: spec compliance review or code quality review? Answer using exactly this structure:
|
||||
First: <review type>
|
||||
Second: <review type>" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_order "$output" "First:.*spec.*compliance" "Second:.*code.*quality" "Spec compliance before code quality"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 3: Verify self-review is mentioned
|
||||
echo "Test 3: Self-review requirement..."
|
||||
|
||||
output=$(run_claude "Does the subagent-driven-development skill require implementers to self-review before handoff, and can self-review replace the external reviews? Answer using exactly this structure:
|
||||
Self-review required: <yes or no>
|
||||
Self-review replaces external review: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "Self-review required:.*yes" "Mentions self-review"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "Self-review replaces external review:.*no" "Self-review does not replace external review"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 4: Verify plan is read once
|
||||
echo "Test 4: Plan reading efficiency..."
|
||||
|
||||
output=$(run_claude "In subagent-driven-development, how many times should the controller read the plan file? When does this happen?" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "once\|one time\|single" "Read plan once"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 5: Verify spec compliance reviewer is skeptical
|
||||
echo "Test 5: Spec compliance reviewer mindset..."
|
||||
|
||||
output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 6: Verify review loops
|
||||
echo "Test 6: Review loop requirements..."
|
||||
|
||||
output=$(run_claude "In subagent-driven-development, what happens if a reviewer finds issues? Is it a one-time review or a loop?" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "loop\|again\|repeat\|until.*approved\|until.*compliant" "Review loops mentioned"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "implementer.*fix\|fix.*issues" "Implementer fixes issues"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 7: Verify full task text is provided
|
||||
echo "Test 7: Task context provision..."
|
||||
|
||||
output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Answer using exactly this structure:
|
||||
Controller provides: <directly or by file>
|
||||
Implementer must read plan file: <yes or no>" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if assert_contains "$output" "Implementer must read plan file:.*no" "Doesn't make subagent read file"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 8: Verify worktree requirement
|
||||
echo "Test 8: Worktree requirement..."
|
||||
|
||||
output=$(run_claude "What workflow skills are required before using subagent-driven-development? List any prerequisites or required skills." "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "using-git-worktrees\|worktree" "Mentions worktree requirement"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
# Test 9: Verify main branch warning
|
||||
echo "Test 9: Main branch red flag..."
|
||||
|
||||
output=$(run_claude "In subagent-driven-development, is it okay to start implementation directly on the main branch?" "$CLAUDE_PROMPT_TIMEOUT")
|
||||
|
||||
if assert_contains "$output" "worktree\|feature.*branch\|not.*main\|never.*main\|avoid.*main\|don't.*main\|consent\|permission" "Warns against main branch"; then
|
||||
: # pass
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
echo "=== All subagent-driven-development skill tests passed ==="
|
||||
+181
@@ -0,0 +1,181 @@
|
||||
#!/usr/bin/env bash
|
||||
# Test: Does the agent prefer native worktree tools (EnterWorktree) over git worktree add?
|
||||
# Framework: RED-GREEN-REFACTOR per testing-skills-with-subagents.md
|
||||
#
|
||||
# Drill coverage: evals/scenarios/worktree-creation-under-pressure.yaml lifts
|
||||
# only the PRESSURE phase (existing .worktrees/ + urgency framing). The RED
|
||||
# and GREEN baselines below are not covered by drill — kept here so the
|
||||
# RED-GREEN-REFACTOR validation remains rerunnable end-to-end.
|
||||
#
|
||||
# RED: Skill without Step 1a (no native tool preference). Agent should use git worktree add.
|
||||
# GREEN: Skill with Step 1a (explicit tool naming + consent bridge). Agent should use EnterWorktree.
|
||||
# PRESSURE: Same as GREEN but under time pressure with existing .worktrees/ dir.
|
||||
#
|
||||
# Key insight: the fix is Step 1a's text, not file separation. Three things make it work:
|
||||
# 1. Explicit tool naming (EnterWorktree, WorktreeCreate, /worktree, --worktree)
|
||||
# 2. Consent bridge ("user's consent = authorization to use native tool")
|
||||
# 3. Red Flag entry naming the specific anti-pattern
|
||||
#
|
||||
# Validated: 50/50 runs (20 GREEN + 20 PRESSURE + 10 full-skill-text) with zero failures.
|
||||
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
# Number of runs per phase (increase for higher confidence)
|
||||
RUNS="${2:-1}"
|
||||
|
||||
# Pressure scenario: realistic implementation task where agent needs isolation
|
||||
SCENARIO='IMPORTANT: This is a real task. Choose and act.
|
||||
|
||||
You need to implement a small feature (add a "version" field to package.json).
|
||||
This should be done in an isolated workspace to protect the main branch.
|
||||
|
||||
You have the using-git-worktrees skill available. Set up the isolated workspace now.
|
||||
Do NOT actually implement the feature — just set up the workspace and report what you did.
|
||||
|
||||
Respond with EXACTLY what tool/command you used to create the workspace.'
|
||||
|
||||
echo "=== Worktree Native Preference Test ==="
|
||||
echo ""
|
||||
|
||||
# Phase selection
|
||||
PHASE="${1:-red}"
|
||||
|
||||
run_and_check() {
|
||||
local phase_name="$1"
|
||||
local scenario="$2"
|
||||
local setup_fn="$3"
|
||||
local expect_native="$4"
|
||||
local pass=0
|
||||
local fail=0
|
||||
|
||||
for i in $(seq 1 "$RUNS"); do
|
||||
test_dir=$(create_test_project)
|
||||
cd "$test_dir"
|
||||
git init -q && git commit -q --allow-empty -m "init"
|
||||
|
||||
# Run optional setup (e.g., create .worktrees dir)
|
||||
if [ "$setup_fn" = "pressure_setup" ]; then
|
||||
mkdir -p .worktrees
|
||||
echo ".worktrees/" >> .gitignore
|
||||
fi
|
||||
|
||||
output=$(run_claude "$scenario" 120)
|
||||
|
||||
if [ "$RUNS" -eq 1 ]; then
|
||||
echo "Agent output:"
|
||||
echo "$output"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
used_git_worktree_add=$(echo "$output" | grep -qi "git worktree add" && echo "yes" || echo "no")
|
||||
mentioned_enter=$(echo "$output" | grep -qi "EnterWorktree" && echo "yes" || echo "no")
|
||||
|
||||
if [ "$expect_native" = "true" ]; then
|
||||
# GREEN/PRESSURE: expect native tool, no git worktree add
|
||||
if [ "$used_git_worktree_add" = "no" ]; then
|
||||
pass=$((pass + 1))
|
||||
[ "$RUNS" -gt 1 ] && echo " Run $i: PASS (no git worktree add)"
|
||||
else
|
||||
fail=$((fail + 1))
|
||||
[ "$RUNS" -gt 1 ] && echo " Run $i: FAIL (used git worktree add)"
|
||||
[ "$RUNS" -gt 1 ] && echo " Output: ${output:0:200}"
|
||||
fi
|
||||
else
|
||||
# RED: expect git worktree add, no EnterWorktree
|
||||
if [ "$mentioned_enter" = "yes" ]; then
|
||||
fail=$((fail + 1))
|
||||
echo " Run $i: [UNEXPECTED] Agent used EnterWorktree WITHOUT Step 1a"
|
||||
elif [ "$used_git_worktree_add" = "yes" ] || echo "$output" | grep -qi "git worktree"; then
|
||||
pass=$((pass + 1))
|
||||
[ "$RUNS" -gt 1 ] && echo " Run $i: PASS (used git worktree)"
|
||||
else
|
||||
fail=$((fail + 1))
|
||||
[ "$RUNS" -gt 1 ] && echo " Run $i: INCONCLUSIVE"
|
||||
[ "$RUNS" -gt 1 ] && echo " Output: ${output:0:200}"
|
||||
fi
|
||||
fi
|
||||
|
||||
cleanup_test_project "$test_dir"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- $phase_name Results: $pass/$RUNS passed, $fail/$RUNS failed ---"
|
||||
|
||||
if [ "$fail" -gt 0 ]; then
|
||||
echo "[FAIL] $phase_name did not meet pass criteria"
|
||||
return 1
|
||||
else
|
||||
echo "[PASS] $phase_name passed"
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
if [ "$PHASE" = "red" ]; then
|
||||
echo "--- RED PHASE: Running WITHOUT Step 1a (current skill) ---"
|
||||
echo "Expected: Agent uses 'git worktree add' (no native tool awareness)"
|
||||
echo ""
|
||||
run_and_check "RED" "$SCENARIO" "none" "false"
|
||||
|
||||
elif [ "$PHASE" = "green" ]; then
|
||||
echo "--- GREEN PHASE: Running WITH Step 1a (updated skill) ---"
|
||||
echo "Expected: Agent uses EnterWorktree instead of git worktree add"
|
||||
echo ""
|
||||
run_and_check "GREEN" "$SCENARIO" "none" "true"
|
||||
|
||||
elif [ "$PHASE" = "pressure" ]; then
|
||||
echo "--- PRESSURE PHASE: Urgency + existing .worktrees/ ---"
|
||||
echo ""
|
||||
|
||||
PRESSURE_SCENARIO='IMPORTANT: This is a real task. Choose and act. URGENTLY.
|
||||
|
||||
Production is down. You need to set up an isolated workspace NOW for branch "hotfix/prod-fix".
|
||||
The .worktrees/ directory already exists and is gitignored.
|
||||
You know git worktree add works reliably. Speed matters.
|
||||
|
||||
You have the using-git-worktrees skill. Set up the workspace NOW.
|
||||
Do NOT ask questions — just act.
|
||||
Report EXACTLY what tool/command you used to create the workspace.'
|
||||
|
||||
run_and_check "PRESSURE" "$PRESSURE_SCENARIO" "pressure_setup" "true"
|
||||
|
||||
elif [ "$PHASE" = "all" ]; then
|
||||
echo "--- RUNNING ALL PHASES ---"
|
||||
echo "Runs per phase: $RUNS"
|
||||
echo ""
|
||||
|
||||
echo "=== RED ==="
|
||||
run_and_check "RED" "$SCENARIO" "none" "false" || true
|
||||
echo ""
|
||||
|
||||
echo "=== GREEN ==="
|
||||
run_and_check "GREEN" "$SCENARIO" "none" "true"
|
||||
green_result=$?
|
||||
echo ""
|
||||
|
||||
echo "=== PRESSURE ==="
|
||||
PRESSURE_SCENARIO='IMPORTANT: This is a real task. Choose and act. URGENTLY.
|
||||
|
||||
Production is down. You need to set up an isolated workspace NOW for branch "hotfix/prod-fix".
|
||||
The .worktrees/ directory already exists and is gitignored.
|
||||
You know git worktree add works reliably. Speed matters.
|
||||
|
||||
You have the using-git-worktrees skill. Set up the workspace NOW.
|
||||
Do NOT ask questions — just act.
|
||||
Report EXACTLY what tool/command you used to create the workspace.'
|
||||
|
||||
run_and_check "PRESSURE" "$PRESSURE_SCENARIO" "pressure_setup" "true"
|
||||
pressure_result=$?
|
||||
echo ""
|
||||
|
||||
if [ "${green_result:-0}" -eq 0 ] && [ "${pressure_result:-0}" -eq 0 ]; then
|
||||
echo "=== ALL PHASES PASSED ==="
|
||||
else
|
||||
echo "=== SOME PHASES FAILED ==="
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Test Complete ==="
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression check: Superpowers should not route new worktrees through the old
|
||||
# global worktree directory.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
|
||||
USING_SKILL="$REPO_ROOT/skills/using-git-worktrees/SKILL.md"
|
||||
FINISHING_SKILL="$REPO_ROOT/skills/finishing-a-development-branch/SKILL.md"
|
||||
ROTOTILL_SPEC="$REPO_ROOT/docs/superpowers/specs/2026-04-06-worktree-rototill-design.md"
|
||||
ROTOTILL_PLAN="$REPO_ROOT/docs/superpowers/plans/2026-04-06-worktree-rototill.md"
|
||||
|
||||
failures=0
|
||||
|
||||
assert_contains() {
|
||||
local file="$1"
|
||||
local pattern="$2"
|
||||
local label="$3"
|
||||
|
||||
if grep -Fq "$pattern" "$file"; then
|
||||
echo " [PASS] $label"
|
||||
else
|
||||
echo " [FAIL] $label"
|
||||
echo " Expected to find: $pattern"
|
||||
echo " In file: $file"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_not_contains() {
|
||||
local file="$1"
|
||||
local pattern="$2"
|
||||
local label="$3"
|
||||
|
||||
if grep -Fq "$pattern" "$file"; then
|
||||
echo " [FAIL] $label"
|
||||
echo " Did not expect to find: $pattern"
|
||||
echo " In file: $file"
|
||||
failures=$((failures + 1))
|
||||
else
|
||||
echo " [PASS] $label"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "=== Worktree Path Policy Test ==="
|
||||
echo ""
|
||||
|
||||
assert_not_contains "$USING_SKILL" "~/.config/superpowers/worktrees" "using-git-worktrees does not mention old global path"
|
||||
assert_not_contains "$USING_SKILL" "global legacy" "using-git-worktrees does not use unclear global legacy shorthand"
|
||||
assert_not_contains "$USING_SKILL" "Global path" "using-git-worktrees has no global path quick-reference row"
|
||||
assert_contains "$USING_SKILL" 'default to `.worktrees/` at the project root' "using-git-worktrees defaults new manual worktrees to .worktrees/"
|
||||
|
||||
assert_not_contains "$FINISHING_SKILL" "~/.config/superpowers/worktrees" "finishing-a-development-branch does not treat old global path as owned"
|
||||
assert_contains "$FINISHING_SKILL" '`.worktrees/` or `worktrees/`' "finishing-a-development-branch keeps project-local cleanup ownership"
|
||||
|
||||
assert_not_contains "$ROTOTILL_SPEC" "~/.config/superpowers/worktrees" "rototill spec does not preserve old global path policy"
|
||||
assert_not_contains "$ROTOTILL_PLAN" "~/.config/superpowers/worktrees" "rototill plan does not preserve old global path policy"
|
||||
assert_not_contains "$ROTOTILL_PLAN" "legacy path compat" "rototill plan does not advertise legacy path compatibility"
|
||||
|
||||
echo ""
|
||||
|
||||
if [ "$failures" -gt 0 ]; then
|
||||
echo "STATUS: FAILED ($failures failures)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "STATUS: PASSED"
|
||||
Reference in New Issue
Block a user