wehub-resource-sync
443 lines
20 KiB
Markdown
443 lines
20 KiB
Markdown
# PluginEval: Quality Evaluation Framework
|
||
|
||
PluginEval is a three-layer quality evaluation framework for Claude Code plugins and skills. It combines deterministic static analysis, LLM-based semantic judging, and Monte Carlo simulation to produce calibrated quality scores with confidence intervals.
|
||
|
||
## Overview
|
||
|
||
PluginEval answers the question: **"How good is this plugin or skill?"** It evaluates across 10 quality dimensions, detects anti-patterns, assigns letter grades, and awards quality badges (Bronze through Platinum).
|
||
|
||
### Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────┐
|
||
│ CLI / Commands │
|
||
│ score · certify · compare · init │
|
||
├─────────────────────────────────────────────────┤
|
||
│ Eval Engine │
|
||
│ Composite scoring, layer blending │
|
||
├────────────┬────────────────┬───────────────────┤
|
||
│ Layer 1 │ Layer 2 │ Layer 3 │
|
||
│ Static │ LLM Judge │ Monte Carlo │
|
||
│ Analysis │ (Semantic) │ (Statistical) │
|
||
│ <2s, free │ ~30s, 4 calls │ ~2min, 50 calls │
|
||
├────────────┴────────────────┴───────────────────┤
|
||
│ Parser Layer │
|
||
│ SKILL.md, agents/*.md, plugin.json │
|
||
├─────────────────────────────────────────────────┤
|
||
│ Statistical Methods │
|
||
│ Wilson CI · Bootstrap CI · Clopper-Pearson │
|
||
│ Cohen's κ · Coefficient of Variation │
|
||
├─────────────────────────────────────────────────┤
|
||
│ Corpus & Elo Ranking │
|
||
│ Gold standard index · Pairwise comparison │
|
||
└─────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Installation & Setup
|
||
|
||
PluginEval lives in `plugins/plugin-eval/` and uses [uv](https://docs.astral.sh/uv/) for dependency management.
|
||
|
||
```bash
|
||
cd plugins/plugin-eval
|
||
|
||
# Install core dependencies (static analysis only)
|
||
uv sync
|
||
|
||
# Install with LLM support (Layers 2 & 3)
|
||
uv sync --extra llm
|
||
|
||
# Install with direct API support
|
||
uv sync --extra api
|
||
|
||
# Install dev dependencies (tests, linting)
|
||
uv sync --extra dev
|
||
```
|
||
|
||
### Requirements
|
||
|
||
- Python ≥ 3.12
|
||
- Core: `pydantic`, `typer`, `rich`, `pyyaml`
|
||
- LLM layers: `claude-agent-sdk` (uses Claude Code Max plan by default)
|
||
- API alternative: `anthropic` SDK (requires `ANTHROPIC_API_KEY`)
|
||
|
||
## CLI Commands
|
||
|
||
### `score` — Evaluate a plugin or skill
|
||
|
||
```bash
|
||
# Quick evaluation (static only, instant)
|
||
uv run plugin-eval score path/to/skill --depth quick
|
||
|
||
# Standard evaluation (static + LLM judge)
|
||
uv run plugin-eval score path/to/skill --depth standard
|
||
|
||
# Deep evaluation (all three layers)
|
||
uv run plugin-eval score path/to/skill --depth deep
|
||
|
||
# Output formats
|
||
uv run plugin-eval score path/to/skill --output json
|
||
uv run plugin-eval score path/to/skill --output markdown
|
||
uv run plugin-eval score path/to/skill --output html
|
||
|
||
# CI gate: exit code 1 if below threshold
|
||
uv run plugin-eval score path/to/skill --threshold 70
|
||
```
|
||
|
||
**Options:**
|
||
|
||
| Option | Default | Description |
|
||
|--------|---------|-------------|
|
||
| `--depth` | `standard` | `quick`, `standard`, `deep`, `thorough` |
|
||
| `--output` | `markdown` | `json`, `markdown`, `html` |
|
||
| `--verbose` | `false` | Show detailed output |
|
||
| `--concurrency` | `4` | Max concurrent LLM calls (1–20) |
|
||
| `--threshold` | none | Minimum score; exit 1 if below |
|
||
|
||
### `certify` — Full certification with badge
|
||
|
||
Runs at `deep` depth (all three layers). Takes 15–20 minutes.
|
||
|
||
```bash
|
||
uv run plugin-eval certify path/to/skill --output markdown
|
||
```
|
||
|
||
### `compare` — Head-to-head comparison
|
||
|
||
Compare two skills side-by-side across all dimensions.
|
||
|
||
```bash
|
||
uv run plugin-eval compare path/to/skill-a path/to/skill-b
|
||
```
|
||
|
||
### `init` — Initialize corpus
|
||
|
||
Build a gold-standard corpus index from a plugins directory for Elo ranking.
|
||
|
||
```bash
|
||
uv run plugin-eval init plugins/ --corpus-dir ~/.plugineval/corpus
|
||
```
|
||
|
||
## Claude Code Integration
|
||
|
||
PluginEval is also a Claude Code plugin with agents and commands.
|
||
|
||
### Slash Commands
|
||
|
||
| Command | Description |
|
||
| ------------------ | -------------------------------------------------------- |
|
||
| `/eval <path>` | Evaluate a plugin or skill (orchestrates static + judge) |
|
||
| `/certify <path>` | Full certification pipeline with badge |
|
||
| `/compare <a> <b>` | Head-to-head skill comparison |
|
||
|
||
### Agents
|
||
|
||
| Agent | Model | Role |
|
||
| ------------------- | ------ | ---------------------------------------------------------------------- |
|
||
| `eval-orchestrator` | Opus | Coordinates evaluation: runs CLI, dispatches judge, computes composite |
|
||
| `eval-judge` | Sonnet | LLM judge: scores 4 semantic dimensions with anchored rubrics |
|
||
|
||
### Skill
|
||
|
||
The `evaluation-methodology` skill provides the full scoring methodology reference, including dimension definitions, rubric anchors, blend weights, and improvement guidance.
|
||
|
||
## The Three Evaluation Layers
|
||
|
||
### Layer 1: Static Analysis
|
||
|
||
**Speed:** < 2 seconds. **Cost:** Free (no LLM calls). **Deterministic.**
|
||
|
||
Runs seven structural sub-checks against the parsed SKILL.md:
|
||
|
||
| Sub-check | Weight | What it measures |
|
||
| ------------------------- | ------ | --------------------------------------------------------------------------------- |
|
||
| `frontmatter_quality` | 32% | Name, description length, trigger-phrase quality ("Use when…", "Use PROACTIVELY") |
|
||
| `orchestration_wiring` | 23% | Output/input documentation, code examples, orchestrator anti-pattern |
|
||
| `progressive_disclosure` | 14% | Line count vs. sweet spot (200–600 lines), references/ and assets/ directories |
|
||
| `structural_completeness` | 10% | Heading density, code blocks, examples section, troubleshooting section |
|
||
| `token_efficiency` | 9% | MUST/NEVER/ALWAYS density, duplicate-line detection |
|
||
| `ecosystem_coherence` | 6% | Cross-references to other skills/agents, "related"/"see also" mentions |
|
||
| `harness_portability` | 6% | Codex/Cursor/OpenCode/Gemini portability — body cap, tool refs, model aliases, name collisions |
|
||
|
||
Also detects anti-patterns (see below) and applies a multiplicative penalty.
|
||
|
||
### Layer 2: LLM Judge
|
||
|
||
**Speed:** ~30 seconds. **Cost:** 4 LLM calls (Haiku + Sonnet). **Requires `claude-agent-sdk`.**
|
||
|
||
Uses Claude as a semantic evaluator across 4 dimensions with anchored rubrics:
|
||
|
||
| Dimension | Model | Method |
|
||
| ----------------------- | ------ | ---------------------------------------------------------------------------- |
|
||
| `triggering_accuracy` | Haiku | Generates 10 synthetic prompts (5 should-trigger, 5 should-not), computes F1 |
|
||
| `orchestration_fitness` | Sonnet | Rates worker-vs-orchestrator role using 5-point anchored rubric |
|
||
| `output_quality` | Sonnet | Simulates 3 realistic tasks, evaluates expected output quality |
|
||
| `scope_calibration` | Sonnet | Rates scope appropriateness using 5-point anchored rubric |
|
||
|
||
All 4 assessments run concurrently with semaphore-based throttling.
|
||
|
||
### Layer 3: Monte Carlo Simulation
|
||
|
||
**Speed:** ~2 minutes (50 runs) to ~5 minutes (100 runs). **Cost:** 50–100 LLM calls. **Requires `claude-agent-sdk`.**
|
||
|
||
Generates 15 varied prompts via Haiku, then runs N simulations to compute statistical reliability:
|
||
|
||
| Metric | Measure | Statistical Method |
|
||
| ------------------ | --------------------------------------- | --------------------------------- |
|
||
| Activation rate | % of runs where skill activated | Wilson score CI |
|
||
| Output consistency | Mean quality + coefficient of variation | Bootstrap CI (1000 resamples) |
|
||
| Failure rate | % of runs that errored | Clopper-Pearson exact CI |
|
||
| Token efficiency | Median tokens, IQR, outlier detection | Normalized against 8000-token cap |
|
||
|
||
## Evaluation Depths
|
||
|
||
| Depth | Layers | Confidence Label | Time | Cost |
|
||
| ---------- | --------------------------------------- | ---------------- | ------ | -------------- |
|
||
| `quick` | Static only | Estimated | < 2s | Free |
|
||
| `standard` | Static + Judge | Assessed | ~30s | 4 LLM calls |
|
||
| `deep` | Static + Judge + Monte Carlo (50 runs) | Certified | ~3 min | ~54 LLM calls |
|
||
| `thorough` | Static + Judge + Monte Carlo (100 runs) | Certified+ | ~6 min | ~104 LLM calls |
|
||
|
||
## The 10 Quality Dimensions
|
||
|
||
Each dimension has a weight and receives scores from different layers, blended using per-dimension weights:
|
||
|
||
| Dimension | Weight | Static | Judge | Monte Carlo | What it measures |
|
||
| ------------------------- | ------ | ------ | ----- | ----------- | ------------------------------------------------ |
|
||
| `triggering_accuracy` | 25% | 0.15 | 0.25 | 0.60 | Does the description fire for the right prompts? |
|
||
| `orchestration_fitness` | 20% | 0.10 | 0.70 | 0.20 | Is it a composable worker, not an orchestrator? |
|
||
| `output_quality` | 15% | 0.00 | 0.40 | 0.60 | Would it produce correct, useful output? |
|
||
| `scope_calibration` | 12% | 0.30 | 0.55 | 0.15 | Is the scope well-sized for its domain? |
|
||
| `progressive_disclosure` | 10% | 0.80 | 0.20 | 0.00 | Does it use references/ for large content? |
|
||
| `token_efficiency` | 6% | 0.40 | 0.10 | 0.50 | Is it concise without repetition? |
|
||
| `robustness` | 5% | 0.00 | 0.20 | 0.80 | Does it handle varied inputs reliably? |
|
||
| `structural_completeness` | 3% | 0.90 | 0.10 | 0.00 | Does it have headings, code, examples? |
|
||
| `code_template_quality` | 2% | 0.30 | 0.70 | 0.00 | Are code examples production-ready? |
|
||
| `ecosystem_coherence` | 2% | 0.85 | 0.15 | 0.00 | Does it link to related skills/agents? |
|
||
|
||
### Composite Score Formula
|
||
|
||
```
|
||
Final = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty
|
||
```
|
||
|
||
Where `blended_score` for each dimension is a weighted combination of available layer scores, renormalized to the layers actually present.
|
||
|
||
## Quality Badges
|
||
|
||
| Badge | Score | Elo | Stars | Meaning |
|
||
| -------- | ----- | ------ | ----- | ------------------------ |
|
||
| Platinum | ≥ 90 | ≥ 1600 | ★★★★★ | Reference quality |
|
||
| Gold | ≥ 80 | ≥ 1500 | ★★★★ | Production ready |
|
||
| Silver | ≥ 70 | ≥ 1400 | ★★★ | Functional, needs polish |
|
||
| Bronze | ≥ 60 | ≥ 1300 | ★★ | Minimum viable |
|
||
|
||
Badges require both score AND Elo thresholds when Elo data is available.
|
||
|
||
## Letter Grades
|
||
|
||
Scores are also converted to letter grades:
|
||
|
||
| Grade | Score Range |
|
||
| ----- | ----------- |
|
||
| A+ | ≥ 97 |
|
||
| A | ≥ 93 |
|
||
| A- | ≥ 90 |
|
||
| B+ | ≥ 87 |
|
||
| B | ≥ 83 |
|
||
| B- | ≥ 80 |
|
||
| C+ | ≥ 77 |
|
||
| C | ≥ 73 |
|
||
| C- | ≥ 70 |
|
||
| D+ | ≥ 67 |
|
||
| D | ≥ 63 |
|
||
| D- | ≥ 60 |
|
||
| F | < 60 |
|
||
|
||
## Anti-Pattern Detection
|
||
|
||
The static analyzer detects these anti-patterns, each with a severity that contributes to a multiplicative penalty:
|
||
|
||
| Flag | Severity | Trigger |
|
||
| ------------------- | -------- | --------------------------------------------- |
|
||
| `OVER_CONSTRAINED` | 10% | > 15 MUST/ALWAYS/NEVER directives |
|
||
| `EMPTY_DESCRIPTION` | 10% | Description < 20 characters |
|
||
| `MISSING_TRIGGER` | 15% | No "Use when…" trigger phrase in description |
|
||
| `BLOATED_SKILL` | 10% | > 800 lines without a references/ directory |
|
||
| `ORPHAN_REFERENCE` | 5% | Dead link to a file in references/ |
|
||
| `DEAD_CROSS_REF` | 5% | Cross-reference to a non-existent skill/agent |
|
||
| `SKILL_OVER_CODEX_CAP` | 15% | Skill body > 8 KB without references/ (Codex hard-truncates) |
|
||
| `CLAUDE_TOOL_REFS` | 2–10% | Backticked CamelCase tool names (`` `Read` ``, `` `Bash` ``) |
|
||
| `CLAUDE_TOOL_PROSE` | 5% | Prose like "use the Read tool" (Codex prefers action verbs) |
|
||
| `AGENT_NAME_COLLISION` | 10% | Agent named `default`/`worker`/`explorer` (Codex built-ins) |
|
||
| `BARE_MODEL_ALIAS` | 3% | Bare `opus`/`sonnet`/`haiku` (use `inherit` for portability) |
|
||
|
||
Each `harness_portability` finding carries a `remediation` string surfaced via the
|
||
AntiPattern description, so the fix is in-context when the lint fires.
|
||
|
||
**Penalty formula:** `penalty = max(0.5, 1.0 − 0.05 × count)` — each anti-pattern reduces the score by 5%, flooring at 50%.
|
||
|
||
## Elo Ranking System
|
||
|
||
For relative quality comparison against a corpus of known skills:
|
||
|
||
- **Initial rating:** 1500
|
||
- **K-factor:** 32
|
||
- **Confidence intervals:** Bootstrap resampling (500 resamples)
|
||
- **Corpus management:** `init` command indexes all skills from a plugins directory
|
||
- **Reference selection:** Matches by category and similar line count
|
||
|
||
The Elo system uses the standard formula: `E(A) = 1 / (1 + 10^((Rb - Ra) / 400))`.
|
||
|
||
## Corpus Management
|
||
|
||
The corpus is a JSON index of all skills used for Elo comparisons:
|
||
|
||
```bash
|
||
# Build corpus from your plugins directory
|
||
uv run plugin-eval init plugins/ --corpus-dir ~/.plugineval/corpus
|
||
|
||
# The corpus stores:
|
||
# - Skill name, path, category, line count
|
||
# - Current Elo rating (updated after each comparison)
|
||
```
|
||
|
||
Reference skills are selected by matching category and approximate line count.
|
||
|
||
## Statistical Methods
|
||
|
||
PluginEval uses rigorous statistical methods throughout:
|
||
|
||
| Method | Used For | Details |
|
||
| ------------------------ | -------------------------- | ----------------------------------------- |
|
||
| Wilson score CI | Activation rate confidence | Handles small-sample binomial proportions |
|
||
| Bootstrap CI | Output quality confidence | 1000 resamples, percentile method |
|
||
| Clopper-Pearson | Failure rate confidence | Exact CI for small failure counts |
|
||
| Coefficient of variation | Output consistency | std/mean ratio; lower = more consistent |
|
||
| Cohen's kappa | Inter-rater agreement | For multi-judge scenarios |
|
||
|
||
All statistical functions are pure Python with no external dependencies (no scipy/numpy required).
|
||
|
||
## Parser
|
||
|
||
The parser extracts structured data from Claude Code plugin files:
|
||
|
||
- **Skills:** Parses SKILL.md frontmatter (name, description), counts headings, code blocks, languages, MUST/NEVER/ALWAYS directives, cross-references, and detects references/ and assets/ directories
|
||
- **Agents:** Parses agent .md frontmatter (name, description, model, tools), detects proactive triggers and skill references
|
||
- **Plugins:** Aggregates all skills and agents from a plugin directory
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
plugins/plugin-eval/
|
||
├── .claude-plugin/
|
||
│ └── plugin.json # Claude Code plugin manifest
|
||
├── agents/
|
||
│ ├── eval-orchestrator.md # Orchestrates evaluation (Opus)
|
||
│ └── eval-judge.md # LLM judge agent (Sonnet)
|
||
├── commands/
|
||
│ ├── eval.md # /eval slash command
|
||
│ ├── certify.md # /certify slash command
|
||
│ └── compare.md # /compare slash command
|
||
├── skills/
|
||
│ └── evaluation-methodology/
|
||
│ ├── SKILL.md # Full methodology reference
|
||
│ └── references/
|
||
│ └── rubrics.md # Detailed rubric anchors
|
||
├── src/plugin_eval/
|
||
│ ├── __init__.py
|
||
│ ├── cli.py # Typer CLI (score, certify, compare, init)
|
||
│ ├── engine.py # Eval engine (layer coordination, composite scoring)
|
||
│ ├── models.py # Pydantic models (Depth, Badge, EvalConfig, results)
|
||
│ ├── parser.py # Plugin/skill/agent parser
|
||
│ ├── reporter.py # JSON/Markdown/HTML output
|
||
│ ├── corpus.py # Gold standard corpus for Elo ranking
|
||
│ ├── elo.py # Elo rating calculator with bootstrap CI
|
||
│ ├── stats.py # Statistical methods (Wilson, bootstrap, Clopper-Pearson)
|
||
│ └── layers/
|
||
│ ├── __init__.py
|
||
│ ├── static.py # Layer 1: deterministic structural analysis
|
||
│ ├── judge.py # Layer 2: LLM semantic evaluation
|
||
│ └── monte_carlo.py # Layer 3: statistical reliability simulation
|
||
├── tests/ # Comprehensive test suite
|
||
│ ├── conftest.py
|
||
│ ├── test_cli.py
|
||
│ ├── test_engine.py
|
||
│ ├── test_static.py
|
||
│ ├── test_judge.py
|
||
│ ├── test_monte_carlo.py
|
||
│ ├── test_models.py
|
||
│ ├── test_parser.py
|
||
│ ├── test_reporter.py
|
||
│ ├── test_corpus.py
|
||
│ ├── test_elo.py
|
||
│ ├── test_stats.py
|
||
│ └── test_e2e.py # End-to-end tests against real plugins
|
||
├── pyproject.toml # uv/hatch project config
|
||
└── uv.lock
|
||
```
|
||
|
||
## Running Tests
|
||
|
||
```bash
|
||
cd plugins/plugin-eval
|
||
|
||
# Run all tests
|
||
uv run pytest
|
||
|
||
# Run with coverage
|
||
uv run pytest --cov=plugin_eval
|
||
|
||
# Run specific test file
|
||
uv run pytest tests/test_static.py
|
||
|
||
# Run e2e tests (requires real plugin corpus)
|
||
uv run pytest tests/test_e2e.py
|
||
```
|
||
|
||
## Example Output
|
||
|
||
### Markdown Report
|
||
|
||
```
|
||
# PluginEval Report
|
||
|
||
**Path:** `plugins/python-development/skills/async-python-patterns`
|
||
**Timestamp:** 2025-03-26T12:00:00+00:00
|
||
**Depth:** standard
|
||
|
||
## Overall Score
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Score | **78.3/100** |
|
||
| Confidence | Assessed |
|
||
| Badge | Silver |
|
||
|
||
## Layer Breakdown
|
||
|
||
| Layer | Score | Anti-Patterns |
|
||
|-------|-------|---------------|
|
||
| static | 0.742 | 0 |
|
||
| judge | 0.811 | 0 |
|
||
|
||
## Dimension Scores
|
||
|
||
| Dimension | Weight | Score | Grade |
|
||
|-----------|--------|-------|-------|
|
||
| Triggering Accuracy | 25% | 0.850 | B |
|
||
| Orchestration Fitness | 20% | 0.780 | C+ |
|
||
| Output Quality | 15% | 0.820 | B- |
|
||
| Scope Calibration | 12% | 0.750 | C |
|
||
| Progressive Disclosure | 10% | 0.600 | D- |
|
||
| Token Efficiency | 6% | 0.910 | A- |
|
||
| ...
|
||
```
|
||
|
||
## Tooling
|
||
|
||
- **Package manager:** [uv](https://docs.astral.sh/uv/)
|
||
- **Linter/formatter:** [ruff](https://docs.astral.sh/ruff/) (target Python 3.12, line length 100)
|
||
- **Type checker:** [ty](https://docs.astral.sh/ty/)
|
||
- **Test framework:** pytest with pytest-asyncio
|
||
- **Build system:** hatchling
|