feat: add iteration promotion bundles
This commit is contained in:
yaojingang
@@ -1,6 +1,6 @@
|
||||
PYTHON ?= python3
|
||||
|
||||
.PHONY: eval eval-suite route-scorecard route-confusion-check description-optimization judge-blind-eval description-optimization-check description-drift-history iteration-ledger results-panel regression-history context-reports failure-regression-check package-check package-failure-check snapshot-check validate lint governance-check resource-boundary-check quality-check test clean
|
||||
.PHONY: eval eval-suite route-scorecard route-confusion-check description-optimization judge-blind-eval description-optimization-check promotion-check description-drift-history iteration-ledger results-panel regression-history context-reports failure-regression-check package-check package-failure-check snapshot-check validate lint governance-check resource-boundary-check quality-check test clean
|
||||
|
||||
eval:
|
||||
$(PYTHON) scripts/trigger_eval.py --description-file evals/improved_description.txt --cases evals/trigger_cases.json --baseline-description-file evals/baseline_description.txt
|
||||
@@ -23,6 +23,9 @@ judge-blind-eval:
|
||||
description-optimization-check:
|
||||
$(PYTHON) tests/verify_description_optimization.py
|
||||
|
||||
promotion-check:
|
||||
$(PYTHON) tests/verify_promotion_checker.py
|
||||
|
||||
description-drift-history:
|
||||
$(PYTHON) scripts/render_description_drift_history.py
|
||||
|
||||
@@ -65,7 +68,7 @@ resource-boundary-check:
|
||||
quality-check:
|
||||
$(PYTHON) tests/verify_quality_checks.py
|
||||
|
||||
test: eval eval-suite route-scorecard route-confusion-check description-optimization description-optimization-check description-drift-history iteration-ledger regression-history context-reports failure-regression-check package-check package-failure-check snapshot-check validate lint governance-check resource-boundary-check quality-check
|
||||
test: eval eval-suite route-scorecard route-confusion-check description-optimization description-optimization-check promotion-check description-drift-history iteration-ledger regression-history context-reports failure-regression-check package-check package-failure-check snapshot-check validate lint governance-check resource-boundary-check quality-check
|
||||
|
||||
clean:
|
||||
rm -rf dist tests/tmp tests/tmp_snapshot
|
||||
|
||||
@@ -97,12 +97,14 @@ Full reports: [reports/eval_suite.json](reports/eval_suite.json) and [reports/fa
|
||||
- failure library regressions: anti-pattern families pass automated checks
|
||||
- governance and resource-boundary checks are part of the default test path
|
||||
- root governance maturity score: `90/100`; governed benchmark example: `95/100`
|
||||
- context budgets: root `989/1000`, complex benchmark `790/1000`, governed benchmark `760/1000`
|
||||
- quality density: root `131.4`, complex benchmark `164.6`, governed benchmark `171.1`
|
||||
- context budgets: root `886/1000`, complex benchmark `790/1000`, governed benchmark `760/1000`
|
||||
- quality density: root `146.7`, complex benchmark `164.6`, governed benchmark `171.1`
|
||||
- regression milestones are tracked in [reports/regression_history.md](reports/regression_history.md)
|
||||
- description drift history is tracked in [reports/description_drift_history.md](reports/description_drift_history.md)
|
||||
- route confusion is tracked in [reports/route_scorecard.md](reports/route_scorecard.md)
|
||||
- promotion evidence is summarized in [reports/iteration_ledger.md](reports/iteration_ledger.md)
|
||||
- promotion decisions are published in [reports/promotion_decisions.md](reports/promotion_decisions.md)
|
||||
- candidate lifecycle states are published in [reports/candidate_registry.md](reports/candidate_registry.md)
|
||||
- context budget summaries are tracked in [reports/context_budget.md](reports/context_budget.md)
|
||||
|
||||
## What It Does
|
||||
@@ -126,6 +128,8 @@ The repository now treats method as a first-class asset instead of scattered gui
|
||||
- [Skill Archetypes](references/skill-archetypes.md)
|
||||
- [Gate Selection](references/gate-selection.md)
|
||||
- [Non-Skill Decision Tree](references/non-skill-decision-tree.md)
|
||||
- [Regression Cause Taxonomy](references/regression-cause-taxonomy.md)
|
||||
- [Human Review Template](references/human-review-template.md)
|
||||
|
||||
## Why It Exists
|
||||
|
||||
@@ -177,6 +181,7 @@ Utility scripts that make the meta-skill operational:
|
||||
- `optimize_description.py`: generates candidate descriptions, scores them on dev, visible holdout, blind holdout, and adversarial holdout suites, then reports calibration and family health
|
||||
- `judge_blind_eval.py`: applies an independent rubric judge to blind-holdout prompts so blind acceptance is not backed only by the main threshold scorer
|
||||
- `run_description_optimization_suite.py`: runs description optimization across the root skill and governed examples, then writes reusable reports and optional drift snapshots with calibration and family summaries
|
||||
- `promotion_checker.py`: applies promotion policy to current description candidates, writes promotion decisions, builds candidate registries, and emits iteration bundles with review stubs
|
||||
- `render_description_drift_history.py`: turns description-optimization snapshots into a readable drift-history report
|
||||
- `build_confusion_matrix.py`: scores route confusion across tracked sibling skills and `no_route` cases, then writes a route scorecard and optional milestone snapshot
|
||||
- `render_iteration_ledger.py`: compresses regression milestones, description optimization drift, and route scorecards into one iteration-facing ledger
|
||||
@@ -212,6 +217,8 @@ Continuous integration entrypoint that runs the full local regression suite on p
|
||||
- Description drift history now records adversarial calibration gaps and family coverage, so routing changes can be judged on confidence and family stability rather than raw error counts alone.
|
||||
- Route confusion is now tracked explicitly across the root meta-skill, frontend review skill, governed incident skill, and `no_route` cases, so route theft is visible instead of implicit.
|
||||
- Promotion policy now requires visible holdout, blind holdout, adversarial holdout, and route confusion to stay clean before a description should be considered promotable.
|
||||
- Promotion checking now emits explicit decisions, candidate lifecycle states, iteration bundles, and human-review stubs rather than leaving promotion as a prose-only step.
|
||||
- Promotion decisions now distinguish “no candidate beat current” from “current still has residual route risk,” so iteration can be audited without forcing every issue into a false block.
|
||||
- Packaging validation now uses explicit contracts and YAML parsing, but it is still a lightweight local validation layer rather than a full platform integration suite.
|
||||
- `evals/failure-cases.md` captures known weak spots that should remain part of regression checks.
|
||||
- `failures/` captures reusable anti-pattern writeups and machine-runnable failure cases for routing, packaging, and authoring failures.
|
||||
@@ -219,6 +226,7 @@ Continuous integration entrypoint that runs the full local regression suite on p
|
||||
- Governance metadata and resource-boundary rules now have runnable checks instead of staying as prose only.
|
||||
- Governance checks now emit a maturity score so governed assets can be compared instead of only pass/fail checked.
|
||||
- Description optimization drift history is now versioned separately from the main trigger regression history so routing improvements are visible over time.
|
||||
- Iteration evidence now records why a candidate was kept, blocked, or promotable via a shared regression-cause taxonomy and bundle artifacts.
|
||||
- Declared maturity tiers are checked against recommended minimum governance scores, so `production`, `library`, and `governed` assets can be compared without forcing every strong example into the same label.
|
||||
- Context budgets are now tiered and explicit, so a governed skill can still choose a stricter `production`-sized initial-load budget.
|
||||
- Resource-boundary checks now detect decorative directories and compute a local quality-density signal instead of only checking raw token counts.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Human Review Template
|
||||
|
||||
Use this template when a description candidate is ready for human review after automated gates have run.
|
||||
|
||||
## Review Header
|
||||
|
||||
- target:
|
||||
- review date:
|
||||
- reviewer:
|
||||
- current description:
|
||||
- candidate description:
|
||||
- promotion decision:
|
||||
|
||||
## What Changed
|
||||
|
||||
- what was the intended improvement?
|
||||
- what route boundary is now clearer?
|
||||
- what family or holdout behavior improved?
|
||||
|
||||
## Automated Evidence
|
||||
|
||||
- visible holdout result:
|
||||
- blind holdout result:
|
||||
- judge-backed blind result:
|
||||
- adversarial result:
|
||||
- route confusion result:
|
||||
- calibration / risk band:
|
||||
|
||||
## Boundary Questions
|
||||
|
||||
- does the candidate trigger only when a reusable skill package is actually needed?
|
||||
- does it avoid one-off explanation, translation, or summary-only requests?
|
||||
- does it avoid stealing sibling routes?
|
||||
- does it stay short enough for the target maturity tier?
|
||||
|
||||
## Decision
|
||||
|
||||
- approve promotion
|
||||
- keep current
|
||||
- block candidate
|
||||
|
||||
## Notes
|
||||
|
||||
- rationale:
|
||||
- follow-up actions:
|
||||
- next review trigger:
|
||||
@@ -0,0 +1,80 @@
|
||||
# Regression Cause Taxonomy
|
||||
|
||||
This taxonomy explains how iteration regressions are classified when a description candidate is evaluated for promotion.
|
||||
|
||||
## Core Principle
|
||||
|
||||
A candidate should not be judged only by aggregate pass/fail counts. The iteration system should explain why a candidate was blocked, kept behind the current description, or promoted.
|
||||
|
||||
## Cause Tags
|
||||
|
||||
### `no_candidate_outperformed_current`
|
||||
|
||||
The selected winner is still the current description. No candidate earned promotion.
|
||||
|
||||
### `visible_holdout_regression`
|
||||
|
||||
The candidate regressed on the visible holdout suite by adding false positives or false negatives.
|
||||
|
||||
### `blind_holdout_regression`
|
||||
|
||||
The candidate regressed on blind holdout prompts. This blocks promotion because the failure is not only local to the tuning loop.
|
||||
|
||||
### `current_holdout_gap_present`
|
||||
|
||||
The current or selected winner still carries a visible holdout miss. Promotion may still stay on `keep_current`, but the iteration bundle should show the unresolved gap.
|
||||
|
||||
### `current_holdout_risk`
|
||||
|
||||
The visible holdout calibration still looks risky even when promotion is not blocked. This is an audit signal that the route boundary needs future work.
|
||||
|
||||
### `judge_blind_regression`
|
||||
|
||||
The rubric judge found worse blind-holdout behavior than the current or baseline description.
|
||||
|
||||
### `judge_blind_low_agreement`
|
||||
|
||||
The judge-backed blind evaluation did not produce enough agreement confidence to support promotion.
|
||||
|
||||
### `adversarial_regression`
|
||||
|
||||
The candidate performed worse on adversarial holdout prompts that simulate route collisions or disguised requests.
|
||||
|
||||
### `adversarial_overlap_risk`
|
||||
|
||||
The adversarial calibration layer reports an `overlap` risk band, meaning route boundaries are too weak for safe promotion.
|
||||
|
||||
### `adversarial_watch_risk`
|
||||
|
||||
The adversarial calibration layer reports a non-failing but cautionary risk band such as `watch` or `tight`.
|
||||
|
||||
### `family_instability`
|
||||
|
||||
At least one tracked family stops being clean under blind, judge-backed blind, or adversarial evaluation.
|
||||
|
||||
### `route_confusion`
|
||||
|
||||
The route confusion matrix shows route theft or misrouting between sibling skills.
|
||||
|
||||
### `route_ambiguity`
|
||||
|
||||
The route confusion matrix reports ambiguous cases near the configured margin-warning threshold.
|
||||
|
||||
### `longer_without_gain`
|
||||
|
||||
The candidate is materially longer than the current description without producing a better route outcome.
|
||||
|
||||
### `promotion_ready`
|
||||
|
||||
The candidate passed every promotion gate and is eligible for review and promotion.
|
||||
|
||||
## Usage
|
||||
|
||||
These cause tags should appear in:
|
||||
|
||||
- promotion decisions
|
||||
- iteration bundles
|
||||
- regression histories
|
||||
- human review summaries
|
||||
|
||||
They are intended to make iteration auditable rather than merely descriptive.
|
||||
@@ -0,0 +1,788 @@
|
||||
{
|
||||
"targets": [
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"winner_tokens": 65,
|
||||
"current_tokens": 65,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 0,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 0,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 0,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 0,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.672,
|
||||
"mean_non_trigger_score": 0.0,
|
||||
"mean_near_neighbor_score": 0.0,
|
||||
"min_positive_score": 0.468,
|
||||
"max_non_trigger_score": 0.0,
|
||||
"score_gap": 0.468,
|
||||
"threshold_margin": 0.138,
|
||||
"positive_threshold_buffer": 0.138,
|
||||
"negative_threshold_buffer": 0.33,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.399,
|
||||
"mean_non_trigger_score": 0.013,
|
||||
"mean_near_neighbor_score": 0.013,
|
||||
"min_positive_score": 0.392,
|
||||
"max_non_trigger_score": 0.026,
|
||||
"score_gap": 0.366,
|
||||
"threshold_margin": 0.062,
|
||||
"positive_threshold_buffer": 0.062,
|
||||
"negative_threshold_buffer": 0.304,
|
||||
"boundary_case_count": 2,
|
||||
"boundary_case_rate": 0.333,
|
||||
"risk_band": "watch"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.838,
|
||||
"mean_non_trigger_score": 0.0,
|
||||
"mean_near_neighbor_score": 0.0,
|
||||
"min_positive_score": 0.834,
|
||||
"max_non_trigger_score": 0.0,
|
||||
"score_gap": 0.834,
|
||||
"threshold_margin": 0.33,
|
||||
"positive_threshold_buffer": 0.504,
|
||||
"negative_threshold_buffer": 0.33,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 12,
|
||||
"clean_family_count": 12,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "workflow_to_skill",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_operationalize",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_operationalize",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_summary_translation_collision",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.607,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Create and improve agent skills.",
|
||||
"tokens": 8,
|
||||
"dev_errors": 2,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"tokens": 65,
|
||||
"dev_errors": 0,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Minimal",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 41,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Trigger when requests mention workflows, prompts, transcripts, or docs and the job is to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 72,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 6,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"visible_holdout_regression"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"winner_tokens": 50,
|
||||
"current_tokens": 50,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 0,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 0,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 0,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 0,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.681,
|
||||
"mean_non_trigger_score": 0.088,
|
||||
"mean_near_neighbor_score": 0.042,
|
||||
"min_positive_score": 0.49,
|
||||
"max_non_trigger_score": 0.211,
|
||||
"score_gap": 0.279,
|
||||
"threshold_margin": 0.01,
|
||||
"positive_threshold_buffer": 0.01,
|
||||
"negative_threshold_buffer": 0.269,
|
||||
"boundary_case_count": 1,
|
||||
"boundary_case_rate": 0.167,
|
||||
"risk_band": "tight"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.681,
|
||||
"mean_non_trigger_score": 0.177,
|
||||
"mean_near_neighbor_score": 0.226,
|
||||
"min_positive_score": 0.504,
|
||||
"max_non_trigger_score": 0.416,
|
||||
"score_gap": 0.088,
|
||||
"threshold_margin": 0.024,
|
||||
"positive_threshold_buffer": 0.024,
|
||||
"negative_threshold_buffer": 0.064,
|
||||
"boundary_case_count": 2,
|
||||
"boundary_case_rate": 0.333,
|
||||
"risk_band": "tight"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.34,
|
||||
"mean_positive_score": 0.71,
|
||||
"mean_non_trigger_score": 0.009,
|
||||
"mean_near_neighbor_score": 0.018,
|
||||
"min_positive_score": 0.544,
|
||||
"max_non_trigger_score": 0.035,
|
||||
"score_gap": 0.509,
|
||||
"threshold_margin": 0.204,
|
||||
"positive_threshold_buffer": 0.204,
|
||||
"negative_threshold_buffer": 0.305,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "security_and_a11y",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_premerge",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_premerge",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_quality_gate_review",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.8,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Review frontend code for accessibility, risky UI security issues, missing states, and common UX regressions. Use when asked to review React, UI, frontend components, forms, a11y, or pre-merge frontend quality.",
|
||||
"tokens": 52,
|
||||
"dev_errors": 4,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 62,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 64,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Trigger when requests mention React components, forms, frontend diffs, or UI code and the job is to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 84,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 90,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"winner_tokens": 37,
|
||||
"current_tokens": 37,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 1,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 1,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 2,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 1,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.367,
|
||||
"mean_non_trigger_score": 0.019,
|
||||
"mean_near_neighbor_score": 0.038,
|
||||
"min_positive_score": 0.08,
|
||||
"max_non_trigger_score": 0.058,
|
||||
"score_gap": 0.022,
|
||||
"threshold_margin": -0.4,
|
||||
"positive_threshold_buffer": -0.4,
|
||||
"negative_threshold_buffer": 0.422,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "overlap"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.899,
|
||||
"mean_non_trigger_score": 0.168,
|
||||
"mean_near_neighbor_score": 0.138,
|
||||
"min_positive_score": 0.798,
|
||||
"max_non_trigger_score": 0.386,
|
||||
"score_gap": 0.412,
|
||||
"threshold_margin": 0.094,
|
||||
"positive_threshold_buffer": 0.318,
|
||||
"negative_threshold_buffer": 0.094,
|
||||
"boundary_case_count": 1,
|
||||
"boundary_case_rate": 0.167,
|
||||
"risk_band": "healthy"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.34,
|
||||
"mean_positive_score": 0.827,
|
||||
"mean_non_trigger_score": 0.027,
|
||||
"mean_near_neighbor_score": 0.011,
|
||||
"min_positive_score": 0.654,
|
||||
"max_non_trigger_score": 0.056,
|
||||
"score_gap": 0.598,
|
||||
"threshold_margin": 0.284,
|
||||
"positive_threshold_buffer": 0.314,
|
||||
"negative_threshold_buffer": 0.284,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.657,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.657,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.69,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 5,
|
||||
"failing_families": [
|
||||
{
|
||||
"family": "packet_assembly",
|
||||
"errors": 1,
|
||||
"pass_rate": 0.0
|
||||
}
|
||||
],
|
||||
"weakest_family": {
|
||||
"family": "packet_assembly",
|
||||
"pass_rate": 0.0,
|
||||
"errors": 1
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_summary_only",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_summary_only",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_single_update_collision",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_gap_present",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.654,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, and stakeholder requirements. Use when asked to standardize incident review, severity assessment, incident communication, or incident packet assembly into one reusable operational skill. Do not use for generic debugging, one-off incident explanations, or brainstorming possible response ideas.",
|
||||
"tokens": 93,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 37,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 51,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Candidate Registry
|
||||
|
||||
| Target | Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors | Reason Tags |
|
||||
| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
|
||||
| `yao-meta-skill` | baseline | `Baseline` | reference | reference | 8 | 2 | 0 | - |
|
||||
| `yao-meta-skill` | current | `Current` | selected_by_dev | kept_current | 65 | 0 | 0 | - |
|
||||
| `yao-meta-skill` | candidate | `Minimal` | not_selected | blocked | 41 | 3 | 0 | weaker_dev_fit |
|
||||
| `yao-meta-skill` | candidate | `Balanced` | not_selected | blocked | 54 | 3 | 0 | weaker_dev_fit |
|
||||
| `yao-meta-skill` | candidate | `Artifact Aware` | not_selected | blocked | 72 | 3 | 0 | weaker_dev_fit, longer_without_gain |
|
||||
| `yao-meta-skill` | candidate | `Boundary` | not_selected | blocked | 78 | 3 | 0 | weaker_dev_fit, longer_without_gain |
|
||||
| `yao-meta-skill` | candidate | `Guardrail` | not_selected | blocked | 50 | 6 | 2 | weaker_dev_fit, visible_holdout_regression |
|
||||
| `team-frontend-review` | baseline | `Baseline` | reference | reference | 52 | 4 | 0 | - |
|
||||
| `team-frontend-review` | current | `Current` | selected_by_dev | kept_current | 50 | 3 | 0 | - |
|
||||
| `team-frontend-review` | candidate | `Guardrail` | not_selected | blocked | 62 | 3 | 0 | longer_without_gain |
|
||||
| `team-frontend-review` | candidate | `Balanced` | not_selected | blocked | 64 | 3 | 0 | longer_without_gain |
|
||||
| `team-frontend-review` | candidate | `Artifact Aware` | not_selected | blocked | 84 | 3 | 0 | longer_without_gain |
|
||||
| `team-frontend-review` | candidate | `Boundary` | not_selected | blocked | 90 | 3 | 0 | longer_without_gain |
|
||||
| `governed-incident-command` | baseline | `Baseline` | reference | reference | 93 | 1 | 2 | - |
|
||||
| `governed-incident-command` | current | `Current` | selected_by_dev | kept_current | 37 | 1 | 1 | - |
|
||||
| `governed-incident-command` | candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 2 | visible_holdout_regression, longer_without_gain |
|
||||
| `governed-incident-command` | candidate | `Balanced` | not_selected | blocked | 54 | 1 | 2 | visible_holdout_regression, longer_without_gain |
|
||||
| `governed-incident-command` | candidate | `Boundary` | not_selected | blocked | 78 | 1 | 1 | longer_without_gain |
|
||||
| `governed-incident-command` | candidate | `Artifact Aware` | not_selected | blocked | 78 | 1 | 1 | longer_without_gain |
|
||||
@@ -6,10 +6,10 @@
|
||||
"context_budget_tier": "production",
|
||||
"context_budget_limit": 1000,
|
||||
"skill_body_tokens": 795,
|
||||
"other_text_tokens": 225175,
|
||||
"other_text_tokens": 253469,
|
||||
"estimated_initial_load_tokens": 886,
|
||||
"estimated_total_text_tokens": 225970,
|
||||
"relevant_file_count": 119,
|
||||
"estimated_total_text_tokens": 254264,
|
||||
"relevant_file_count": 136,
|
||||
"unused_resource_dirs": [],
|
||||
"quality_signal_points": 130,
|
||||
"quality_density": 146.7
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"target": "governed-incident-command",
|
||||
"decision": "keep_current",
|
||||
"winner_label": "Current",
|
||||
"winner_changed": false,
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"cause_tags": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_gap_present",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"candidate_registry": [
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, and stakeholder requirements. Use when asked to standardize incident review, severity assessment, incident communication, or incident packet assembly into one reusable operational skill. Do not use for generic debugging, one-off incident explanations, or brainstorming possible response ideas.",
|
||||
"tokens": 93,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 37,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 51,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"review_template": {
|
||||
"target": "governed-incident-command",
|
||||
"current_description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"candidate_description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"focus": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_gap_present",
|
||||
"current_holdout_risk"
|
||||
]
|
||||
},
|
||||
"artifacts": {
|
||||
"skill": "examples/governed-incident-command/generated-skill/SKILL.md",
|
||||
"optimization_report": "examples/governed-incident-command/optimization/reports/description_optimization.json",
|
||||
"promotion_decisions": "reports/promotion_decisions.json",
|
||||
"candidate_registry": "reports/candidate_registry.json",
|
||||
"regression_cause_taxonomy": "references/regression-cause-taxonomy.md",
|
||||
"human_review_template": "references/human-review-template.md"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
# Iteration Bundle: governed-incident-command
|
||||
|
||||
- decision: `keep_current`
|
||||
- winner label: `Current`
|
||||
- winner changed: `False`
|
||||
- next action: Keep the current description and open a new candidate only when fresh route evidence appears.
|
||||
|
||||
## Cause Tags
|
||||
|
||||
- `no_candidate_outperformed_current`
|
||||
- `current_holdout_gap_present`
|
||||
- `current_holdout_risk`
|
||||
|
||||
## Gate Status
|
||||
|
||||
| Gate | Pass |
|
||||
| --- | --- |
|
||||
| `visible_holdout_non_regression` | True |
|
||||
| `blind_holdout_non_regression` | True |
|
||||
| `judge_blind_non_regression` | True |
|
||||
| `judge_blind_agreement` | True |
|
||||
| `adversarial_non_regression` | True |
|
||||
| `adversarial_risk_ok` | True |
|
||||
| `route_confusion_clean` | True |
|
||||
| `family_stability` | True |
|
||||
|
||||
## Candidate Registry
|
||||
|
||||
| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| baseline | `Baseline` | reference | reference | 93 | 1 | 2 |
|
||||
| current | `Current` | selected_by_dev | kept_current | 37 | 1 | 1 |
|
||||
| candidate | `Guardrail` | not_selected | blocked | 51 | 1 | 2 |
|
||||
| candidate | `Balanced` | not_selected | blocked | 54 | 1 | 2 |
|
||||
| candidate | `Boundary` | not_selected | blocked | 78 | 1 | 1 |
|
||||
| candidate | `Artifact Aware` | not_selected | blocked | 78 | 1 | 1 |
|
||||
|
||||
## Human Review Stub
|
||||
|
||||
- target: governed-incident-command
|
||||
- current description: Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.
|
||||
- candidate description: Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.
|
||||
- review focus: no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk
|
||||
|
||||
## Artifact Paths
|
||||
|
||||
- skill: `examples/governed-incident-command/generated-skill/SKILL.md`
|
||||
- optimization_report: `examples/governed-incident-command/optimization/reports/description_optimization.json`
|
||||
- promotion_decisions: `reports/promotion_decisions.json`
|
||||
- candidate_registry: `reports/candidate_registry.json`
|
||||
- regression_cause_taxonomy: `references/regression-cause-taxonomy.md`
|
||||
- human_review_template: `references/human-review-template.md`
|
||||
@@ -0,0 +1,8 @@
|
||||
# Human Review
|
||||
|
||||
- target: governed-incident-command
|
||||
- current description: Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.
|
||||
- candidate description: Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.
|
||||
- suggested focus: no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk
|
||||
|
||||
Use the shared template in `references/human-review-template.md` to complete the final decision.
|
||||
@@ -0,0 +1,119 @@
|
||||
{
|
||||
"target": "team-frontend-review",
|
||||
"decision": "keep_current",
|
||||
"winner_label": "Current",
|
||||
"winner_changed": false,
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"cause_tags": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"candidate_registry": [
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Review frontend code for accessibility, risky UI security issues, missing states, and common UX regressions. Use when asked to review React, UI, frontend components, forms, a11y, or pre-merge frontend quality.",
|
||||
"tokens": 52,
|
||||
"dev_errors": 4,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 62,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 64,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Trigger when requests mention React components, forms, frontend diffs, or UI code and the job is to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 84,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 90,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"review_template": {
|
||||
"target": "team-frontend-review",
|
||||
"current_description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"candidate_description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"focus": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_risk"
|
||||
]
|
||||
},
|
||||
"artifacts": {
|
||||
"skill": "examples/team-frontend-review/generated-skill/SKILL.md",
|
||||
"optimization_report": "examples/team-frontend-review/optimization/reports/description_optimization.json",
|
||||
"promotion_decisions": "reports/promotion_decisions.json",
|
||||
"candidate_registry": "reports/candidate_registry.json",
|
||||
"regression_cause_taxonomy": "references/regression-cause-taxonomy.md",
|
||||
"human_review_template": "references/human-review-template.md"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
# Iteration Bundle: team-frontend-review
|
||||
|
||||
- decision: `keep_current`
|
||||
- winner label: `Current`
|
||||
- winner changed: `False`
|
||||
- next action: Keep the current description and open a new candidate only when fresh route evidence appears.
|
||||
|
||||
## Cause Tags
|
||||
|
||||
- `no_candidate_outperformed_current`
|
||||
- `current_holdout_risk`
|
||||
|
||||
## Gate Status
|
||||
|
||||
| Gate | Pass |
|
||||
| --- | --- |
|
||||
| `visible_holdout_non_regression` | True |
|
||||
| `blind_holdout_non_regression` | True |
|
||||
| `judge_blind_non_regression` | True |
|
||||
| `judge_blind_agreement` | True |
|
||||
| `adversarial_non_regression` | True |
|
||||
| `adversarial_risk_ok` | True |
|
||||
| `route_confusion_clean` | True |
|
||||
| `family_stability` | True |
|
||||
|
||||
## Candidate Registry
|
||||
|
||||
| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| baseline | `Baseline` | reference | reference | 52 | 4 | 0 |
|
||||
| current | `Current` | selected_by_dev | kept_current | 50 | 3 | 0 |
|
||||
| candidate | `Guardrail` | not_selected | blocked | 62 | 3 | 0 |
|
||||
| candidate | `Balanced` | not_selected | blocked | 64 | 3 | 0 |
|
||||
| candidate | `Artifact Aware` | not_selected | blocked | 84 | 3 | 0 |
|
||||
| candidate | `Boundary` | not_selected | blocked | 90 | 3 | 0 |
|
||||
|
||||
## Human Review Stub
|
||||
|
||||
- target: team-frontend-review
|
||||
- current description: Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.
|
||||
- candidate description: Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.
|
||||
- review focus: no_candidate_outperformed_current, current_holdout_risk
|
||||
|
||||
## Artifact Paths
|
||||
|
||||
- skill: `examples/team-frontend-review/generated-skill/SKILL.md`
|
||||
- optimization_report: `examples/team-frontend-review/optimization/reports/description_optimization.json`
|
||||
- promotion_decisions: `reports/promotion_decisions.json`
|
||||
- candidate_registry: `reports/candidate_registry.json`
|
||||
- regression_cause_taxonomy: `references/regression-cause-taxonomy.md`
|
||||
- human_review_template: `references/human-review-template.md`
|
||||
@@ -0,0 +1,8 @@
|
||||
# Human Review
|
||||
|
||||
- target: team-frontend-review
|
||||
- current description: Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.
|
||||
- candidate description: Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.
|
||||
- suggested focus: no_candidate_outperformed_current, current_holdout_risk
|
||||
|
||||
Use the shared template in `references/human-review-template.md` to complete the final decision.
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"target": "yao-meta-skill",
|
||||
"decision": "keep_current",
|
||||
"winner_label": "Current",
|
||||
"winner_changed": false,
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"cause_tags": [
|
||||
"no_candidate_outperformed_current"
|
||||
],
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"candidate_registry": [
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Create and improve agent skills.",
|
||||
"tokens": 8,
|
||||
"dev_errors": 2,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"tokens": 65,
|
||||
"dev_errors": 0,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Minimal",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 41,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Trigger when requests mention workflows, prompts, transcripts, or docs and the job is to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 72,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 6,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"visible_holdout_regression"
|
||||
]
|
||||
}
|
||||
],
|
||||
"review_template": {
|
||||
"target": "yao-meta-skill",
|
||||
"current_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"candidate_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"focus": [
|
||||
"no_candidate_outperformed_current"
|
||||
]
|
||||
},
|
||||
"artifacts": {
|
||||
"skill": "SKILL.md",
|
||||
"optimization_report": "reports/description_optimization.json",
|
||||
"promotion_decisions": "reports/promotion_decisions.json",
|
||||
"candidate_registry": "reports/candidate_registry.json",
|
||||
"regression_cause_taxonomy": "references/regression-cause-taxonomy.md",
|
||||
"human_review_template": "references/human-review-template.md"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
# Iteration Bundle: yao-meta-skill
|
||||
|
||||
- decision: `keep_current`
|
||||
- winner label: `Current`
|
||||
- winner changed: `False`
|
||||
- next action: Keep the current description and open a new candidate only when fresh route evidence appears.
|
||||
|
||||
## Cause Tags
|
||||
|
||||
- `no_candidate_outperformed_current`
|
||||
|
||||
## Gate Status
|
||||
|
||||
| Gate | Pass |
|
||||
| --- | --- |
|
||||
| `visible_holdout_non_regression` | True |
|
||||
| `blind_holdout_non_regression` | True |
|
||||
| `judge_blind_non_regression` | True |
|
||||
| `judge_blind_agreement` | True |
|
||||
| `adversarial_non_regression` | True |
|
||||
| `adversarial_risk_ok` | True |
|
||||
| `route_confusion_clean` | True |
|
||||
| `family_stability` | True |
|
||||
|
||||
## Candidate Registry
|
||||
|
||||
| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| baseline | `Baseline` | reference | reference | 8 | 2 | 0 |
|
||||
| current | `Current` | selected_by_dev | kept_current | 65 | 0 | 0 |
|
||||
| candidate | `Minimal` | not_selected | blocked | 41 | 3 | 0 |
|
||||
| candidate | `Balanced` | not_selected | blocked | 54 | 3 | 0 |
|
||||
| candidate | `Artifact Aware` | not_selected | blocked | 72 | 3 | 0 |
|
||||
| candidate | `Boundary` | not_selected | blocked | 78 | 3 | 0 |
|
||||
| candidate | `Guardrail` | not_selected | blocked | 50 | 6 | 2 |
|
||||
|
||||
## Human Review Stub
|
||||
|
||||
- target: yao-meta-skill
|
||||
- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
|
||||
- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
|
||||
- review focus: no_candidate_outperformed_current
|
||||
|
||||
## Artifact Paths
|
||||
|
||||
- skill: `SKILL.md`
|
||||
- optimization_report: `reports/description_optimization.json`
|
||||
- promotion_decisions: `reports/promotion_decisions.json`
|
||||
- candidate_registry: `reports/candidate_registry.json`
|
||||
- regression_cause_taxonomy: `references/regression-cause-taxonomy.md`
|
||||
- human_review_template: `references/human-review-template.md`
|
||||
@@ -0,0 +1,8 @@
|
||||
# Human Review
|
||||
|
||||
- target: yao-meta-skill
|
||||
- current description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
|
||||
- candidate description: Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.
|
||||
- suggested focus: no_candidate_outperformed_current
|
||||
|
||||
Use the shared template in `references/human-review-template.md` to complete the final decision.
|
||||
@@ -27,6 +27,14 @@
|
||||
| 2026-04-01 | Judge-Backed Blind Eval | `team-frontend-review` | 0 | 0 | 1.0 | 0 | 0.509 | healthy | tokens stable; blind stable at 0; adversarial gate added with 0 errors; holdout stable at 0; adversarial calibration +0.509; risk n/a -> healthy |
|
||||
| 2026-04-01 | Judge-Backed Blind Eval | `governed-incident-command` | 0 | 0 | 1.0 | 0 | 0.598 | healthy | tokens stable; blind stable at 0; adversarial gate added with 0 errors; holdout stable at 1; adversarial calibration +0.598; risk n/a -> healthy |
|
||||
|
||||
## Current Promotion Decisions
|
||||
|
||||
| Target | Decision | Winner | Causes | Next Action |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `yao-meta-skill` | `keep_current` | `Current` | no_candidate_outperformed_current | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
| `team-frontend-review` | `keep_current` | `Current` | no_candidate_outperformed_current, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
| `governed-incident-command` | `keep_current` | `Current` | no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
|
||||
## Current Route Scorecard
|
||||
|
||||
- total cases: `13`
|
||||
|
||||
@@ -0,0 +1,794 @@
|
||||
{
|
||||
"ok": true,
|
||||
"summary": {
|
||||
"promote": 0,
|
||||
"keep_current": 3,
|
||||
"blocked": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"winner_tokens": 65,
|
||||
"current_tokens": 65,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 0,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 0,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 0,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 0,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.672,
|
||||
"mean_non_trigger_score": 0.0,
|
||||
"mean_near_neighbor_score": 0.0,
|
||||
"min_positive_score": 0.468,
|
||||
"max_non_trigger_score": 0.0,
|
||||
"score_gap": 0.468,
|
||||
"threshold_margin": 0.138,
|
||||
"positive_threshold_buffer": 0.138,
|
||||
"negative_threshold_buffer": 0.33,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.399,
|
||||
"mean_non_trigger_score": 0.013,
|
||||
"mean_near_neighbor_score": 0.013,
|
||||
"min_positive_score": 0.392,
|
||||
"max_non_trigger_score": 0.026,
|
||||
"score_gap": 0.366,
|
||||
"threshold_margin": 0.062,
|
||||
"positive_threshold_buffer": 0.062,
|
||||
"negative_threshold_buffer": 0.304,
|
||||
"boundary_case_count": 2,
|
||||
"boundary_case_rate": 0.333,
|
||||
"risk_band": "watch"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.33,
|
||||
"mean_positive_score": 0.838,
|
||||
"mean_non_trigger_score": 0.0,
|
||||
"mean_near_neighbor_score": 0.0,
|
||||
"min_positive_score": 0.834,
|
||||
"max_non_trigger_score": 0.0,
|
||||
"score_gap": 0.834,
|
||||
"threshold_margin": 0.33,
|
||||
"positive_threshold_buffer": 0.504,
|
||||
"negative_threshold_buffer": 0.33,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.66,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 12,
|
||||
"clean_family_count": 12,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "workflow_to_skill",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_operationalize",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_operationalize",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_summary_translation_collision",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.607,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Create and improve agent skills.",
|
||||
"tokens": 8,
|
||||
"dev_errors": 2,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, improve an existing skill, add evals, or package a skill for team reuse.",
|
||||
"tokens": 65,
|
||||
"dev_errors": 0,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Minimal",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 41,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Trigger when requests mention workflows, prompts, transcripts, or docs and the job is to create a skill, turn a repeated process into a reusable skill, or improve an existing skill.",
|
||||
"tokens": 72,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Use when asked to create a skill, turn a repeated process into a reusable skill, or improve an existing skill. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "yao-meta-skill",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Create, refactor, evaluate, and package agent skills from workflows, prompts, transcripts, docs, or notes. Do not use for summary-only requests, translation-only requests, or plain documentation tasks.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 6,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"weaker_dev_fit",
|
||||
"visible_holdout_regression"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"winner_tokens": 50,
|
||||
"current_tokens": 50,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 0,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 0,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 0,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 0,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.681,
|
||||
"mean_non_trigger_score": 0.088,
|
||||
"mean_near_neighbor_score": 0.042,
|
||||
"min_positive_score": 0.49,
|
||||
"max_non_trigger_score": 0.211,
|
||||
"score_gap": 0.279,
|
||||
"threshold_margin": 0.01,
|
||||
"positive_threshold_buffer": 0.01,
|
||||
"negative_threshold_buffer": 0.269,
|
||||
"boundary_case_count": 1,
|
||||
"boundary_case_rate": 0.167,
|
||||
"risk_band": "tight"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.681,
|
||||
"mean_non_trigger_score": 0.177,
|
||||
"mean_near_neighbor_score": 0.226,
|
||||
"min_positive_score": 0.504,
|
||||
"max_non_trigger_score": 0.416,
|
||||
"score_gap": 0.088,
|
||||
"threshold_margin": 0.024,
|
||||
"positive_threshold_buffer": 0.024,
|
||||
"negative_threshold_buffer": 0.064,
|
||||
"boundary_case_count": 2,
|
||||
"boundary_case_rate": 0.333,
|
||||
"risk_band": "tight"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.34,
|
||||
"mean_positive_score": 0.71,
|
||||
"mean_non_trigger_score": 0.009,
|
||||
"mean_near_neighbor_score": 0.018,
|
||||
"min_positive_score": 0.544,
|
||||
"max_non_trigger_score": 0.035,
|
||||
"score_gap": 0.509,
|
||||
"threshold_margin": 0.204,
|
||||
"positive_threshold_buffer": 0.204,
|
||||
"negative_threshold_buffer": 0.305,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.703,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "security_and_a11y",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_premerge",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_premerge",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_quality_gate_review",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.8,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Review frontend code for accessibility, risky UI security issues, missing states, and common UX regressions. Use when asked to review React, UI, frontend components, forms, a11y, or pre-merge frontend quality.",
|
||||
"tokens": 52,
|
||||
"dev_errors": 4,
|
||||
"holdout_errors": 0
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 50,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 62,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 64,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Trigger when requests mention React components, forms, frontend diffs, or UI code and the job is to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering.",
|
||||
"tokens": 84,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "team-frontend-review",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Review frontend code for accessibility, UI security, missing states, and UX regressions from React components, forms, frontend diffs, or UI code. Use when asked to review React components, run a pre-merge frontend review, or check a11y and unsafe rendering. Do not use for building a new page from scratch, backend-only review, or explaining frontend concepts.",
|
||||
"tokens": 90,
|
||||
"dev_errors": 3,
|
||||
"holdout_errors": 0,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"winner_label": "Current",
|
||||
"winner_description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"winner_tokens": 37,
|
||||
"current_tokens": 37,
|
||||
"winner_holdout_fp": 0,
|
||||
"winner_holdout_fn": 1,
|
||||
"current_holdout_fp": 0,
|
||||
"current_holdout_fn": 1,
|
||||
"baseline_holdout_fp": 0,
|
||||
"baseline_holdout_fn": 2,
|
||||
"winner_blind_holdout_fp": 0,
|
||||
"winner_blind_holdout_fn": 0,
|
||||
"current_blind_holdout_fp": 0,
|
||||
"current_blind_holdout_fn": 0,
|
||||
"baseline_blind_holdout_fp": 0,
|
||||
"baseline_blind_holdout_fn": 1,
|
||||
"winner_blind_holdout_total_errors": 0,
|
||||
"winner_judge_blind_holdout_fp": 0,
|
||||
"winner_judge_blind_holdout_fn": 0,
|
||||
"current_judge_blind_holdout_fp": 0,
|
||||
"current_judge_blind_holdout_fn": 0,
|
||||
"baseline_judge_blind_holdout_fp": 0,
|
||||
"baseline_judge_blind_holdout_fn": 0,
|
||||
"winner_judge_blind_holdout_total_errors": 0,
|
||||
"winner_adversarial_holdout_fp": 0,
|
||||
"winner_adversarial_holdout_fn": 0,
|
||||
"current_adversarial_holdout_fp": 0,
|
||||
"current_adversarial_holdout_fn": 0,
|
||||
"baseline_adversarial_holdout_fp": 0,
|
||||
"baseline_adversarial_holdout_fn": 0,
|
||||
"winner_adversarial_holdout_total_errors": 0,
|
||||
"calibration": {
|
||||
"holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.367,
|
||||
"mean_non_trigger_score": 0.019,
|
||||
"mean_near_neighbor_score": 0.038,
|
||||
"min_positive_score": 0.08,
|
||||
"max_non_trigger_score": 0.058,
|
||||
"score_gap": 0.022,
|
||||
"threshold_margin": -0.4,
|
||||
"positive_threshold_buffer": -0.4,
|
||||
"negative_threshold_buffer": 0.422,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "overlap"
|
||||
},
|
||||
"blind_holdout": {
|
||||
"threshold": 0.48,
|
||||
"mean_positive_score": 0.899,
|
||||
"mean_non_trigger_score": 0.168,
|
||||
"mean_near_neighbor_score": 0.138,
|
||||
"min_positive_score": 0.798,
|
||||
"max_non_trigger_score": 0.386,
|
||||
"score_gap": 0.412,
|
||||
"threshold_margin": 0.094,
|
||||
"positive_threshold_buffer": 0.318,
|
||||
"negative_threshold_buffer": 0.094,
|
||||
"boundary_case_count": 1,
|
||||
"boundary_case_rate": 0.167,
|
||||
"risk_band": "healthy"
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"threshold": 0.34,
|
||||
"mean_positive_score": 0.827,
|
||||
"mean_non_trigger_score": 0.027,
|
||||
"mean_near_neighbor_score": 0.011,
|
||||
"min_positive_score": 0.654,
|
||||
"max_non_trigger_score": 0.056,
|
||||
"score_gap": 0.598,
|
||||
"threshold_margin": 0.284,
|
||||
"positive_threshold_buffer": 0.314,
|
||||
"negative_threshold_buffer": 0.284,
|
||||
"boundary_case_count": 0,
|
||||
"boundary_case_rate": 0.0,
|
||||
"risk_band": "healthy"
|
||||
}
|
||||
},
|
||||
"judge_blind": {
|
||||
"winner": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.657,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"current": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.657,
|
||||
"rubric_version": "blind-v1"
|
||||
},
|
||||
"baseline": {
|
||||
"agreement_rate": 1.0,
|
||||
"mean_confidence": 0.69,
|
||||
"rubric_version": "blind-v1"
|
||||
}
|
||||
},
|
||||
"family_health": {
|
||||
"holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 5,
|
||||
"failing_families": [
|
||||
{
|
||||
"family": "packet_assembly",
|
||||
"errors": 1,
|
||||
"pass_rate": 0.0
|
||||
}
|
||||
],
|
||||
"weakest_family": {
|
||||
"family": "packet_assembly",
|
||||
"pass_rate": 0.0,
|
||||
"errors": 1
|
||||
}
|
||||
},
|
||||
"blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_summary_only",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"judge_blind_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "blind_summary_only",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
},
|
||||
"adversarial_holdout": {
|
||||
"family_count": 6,
|
||||
"clean_family_count": 6,
|
||||
"failing_families": [],
|
||||
"weakest_family": {
|
||||
"family": "adversarial_single_update_collision",
|
||||
"pass_rate": 1.0,
|
||||
"errors": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"drift_note": "blind, judge-backed blind, adversarial, and calibration gates active",
|
||||
"ok": true,
|
||||
"promotion": {
|
||||
"decision": "keep_current",
|
||||
"winner_changed": false,
|
||||
"all_gates_pass": true,
|
||||
"route_clean": true,
|
||||
"judge_agreement_ok": true,
|
||||
"causes": [
|
||||
"no_candidate_outperformed_current",
|
||||
"current_holdout_gap_present",
|
||||
"current_holdout_risk"
|
||||
],
|
||||
"next_action": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": true,
|
||||
"blind_holdout_non_regression": true,
|
||||
"judge_blind_non_regression": true,
|
||||
"judge_blind_agreement": true,
|
||||
"adversarial_non_regression": true,
|
||||
"adversarial_risk_ok": true,
|
||||
"route_confusion_clean": true,
|
||||
"family_stability": true
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"average_margin": 0.654,
|
||||
"misroute_count": 0,
|
||||
"ambiguous_case_count": 0
|
||||
}
|
||||
},
|
||||
"registry": [
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, and stakeholder requirements. Use when asked to standardize incident review, severity assessment, incident communication, or incident packet assembly into one reusable operational skill. Do not use for generic debugging, one-off incident explanations, or brainstorming possible response ideas.",
|
||||
"tokens": 93,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "current",
|
||||
"label": "Current",
|
||||
"ranking_state": "selected_by_dev",
|
||||
"promotion_state": "kept_current",
|
||||
"description": "Build governed incident command packets. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 37,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": []
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Guardrail",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 51,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Balanced",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 54,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 2,
|
||||
"reason_tags": [
|
||||
"visible_holdout_regression",
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Boundary",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Use when asked to standardize incident review, run severity assessment, or assemble incident communication. Do not use for generic debugging, one-off incident explanations, or drafting a single update.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "governed-incident-command",
|
||||
"role": "candidate",
|
||||
"label": "Artifact Aware",
|
||||
"ranking_state": "not_selected",
|
||||
"promotion_state": "blocked",
|
||||
"description": "Build governed incident command packets from alerts, incident notes, transcripts, or stakeholder requirements. Trigger when requests mention alerts, incident notes, transcripts, or stakeholder requirements and the job is to standardize incident review, run severity assessment, or assemble incident communication.",
|
||||
"tokens": 78,
|
||||
"dev_errors": 1,
|
||||
"holdout_errors": 1,
|
||||
"reason_tags": [
|
||||
"longer_without_gain"
|
||||
]
|
||||
}
|
||||
],
|
||||
"decision": "keep_current"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# Promotion Decisions
|
||||
|
||||
| Target | Decision | Winner | Route Clean | Judge Agreement | Adv Risk | Causes | Next Action |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| `yao-meta-skill` | `keep_current` | `Current` | True | 1.0 | healthy | no_candidate_outperformed_current | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
| `team-frontend-review` | `keep_current` | `Current` | True | 1.0 | healthy | no_candidate_outperformed_current, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
| `governed-incident-command` | `keep_current` | `Current` | True | 1.0 | healthy | no_candidate_outperformed_current, current_holdout_gap_present, current_holdout_risk | Keep the current description and open a new candidate only when fresh route evidence appears. |
|
||||
@@ -0,0 +1,417 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
OPTIMIZATION_REPORTS = {
|
||||
"yao-meta-skill": ROOT / "reports" / "description_optimization.json",
|
||||
"team-frontend-review": ROOT / "examples" / "team-frontend-review" / "optimization" / "reports" / "description_optimization.json",
|
||||
"governed-incident-command": ROOT
|
||||
/ "examples"
|
||||
/ "governed-incident-command"
|
||||
/ "optimization"
|
||||
/ "reports"
|
||||
/ "description_optimization.json",
|
||||
}
|
||||
|
||||
SKILL_PATHS = {
|
||||
"yao-meta-skill": ROOT / "SKILL.md",
|
||||
"team-frontend-review": ROOT / "examples" / "team-frontend-review" / "generated-skill" / "SKILL.md",
|
||||
"governed-incident-command": ROOT / "examples" / "governed-incident-command" / "generated-skill" / "SKILL.md",
|
||||
}
|
||||
|
||||
|
||||
def load_json(path: Path) -> dict:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def error_total(prefix: str, target: dict) -> int:
|
||||
return int(target.get(f"{prefix}_fp", 0)) + int(target.get(f"{prefix}_fn", 0))
|
||||
|
||||
|
||||
def family_clean(summary: dict | None) -> bool:
|
||||
if not summary:
|
||||
return False
|
||||
return summary.get("clean_family_count", 0) == summary.get("family_count", 0)
|
||||
|
||||
|
||||
def relative(path: Path) -> str:
|
||||
return str(path.relative_to(ROOT))
|
||||
|
||||
|
||||
def classify_target(target: dict, route_scorecard: dict) -> dict:
|
||||
route_summary = route_scorecard["summary"]
|
||||
route_stats = route_scorecard["route_stats"].get(target["name"], {})
|
||||
winner_changed = target["winner_label"] != "Current"
|
||||
visible_ok = error_total("winner_holdout", target) <= error_total("current_holdout", target) and error_total(
|
||||
"winner_holdout", target
|
||||
) <= error_total("baseline_holdout", target)
|
||||
blind_ok = error_total("winner_blind_holdout", target) <= error_total(
|
||||
"current_blind_holdout", target
|
||||
) and error_total("winner_blind_holdout", target) <= error_total("baseline_blind_holdout", target)
|
||||
judge_ok = error_total("winner_judge_blind_holdout", target) <= error_total(
|
||||
"current_judge_blind_holdout", target
|
||||
) and error_total("winner_judge_blind_holdout", target) <= error_total("baseline_judge_blind_holdout", target)
|
||||
judge_agreement = ((target.get("judge_blind") or {}).get("winner") or {}).get("agreement_rate")
|
||||
judge_agreement_ok = judge_agreement is not None and judge_agreement >= 0.95
|
||||
adversarial_ok = error_total("winner_adversarial_holdout", target) <= error_total(
|
||||
"current_adversarial_holdout", target
|
||||
) and error_total("winner_adversarial_holdout", target) <= error_total("baseline_adversarial_holdout", target)
|
||||
holdout_calibration = (target.get("calibration") or {}).get("holdout") or {}
|
||||
adversarial_calibration = (target.get("calibration") or {}).get("adversarial_holdout") or {}
|
||||
adversarial_risk = adversarial_calibration.get("risk_band")
|
||||
adversarial_risk_ok = adversarial_risk not in {"overlap", None}
|
||||
route_clean = (
|
||||
route_summary.get("misroute_count", 0) == 0
|
||||
and route_summary.get("ambiguous_case_count", 0) == 0
|
||||
and route_stats.get("precision") == 1.0
|
||||
and route_stats.get("recall") == 1.0
|
||||
)
|
||||
family_ok = all(
|
||||
family_clean((target.get("family_health") or {}).get(gate))
|
||||
for gate in ("blind_holdout", "judge_blind_holdout", "adversarial_holdout")
|
||||
)
|
||||
|
||||
causes = []
|
||||
if not winner_changed:
|
||||
causes.append("no_candidate_outperformed_current")
|
||||
if not visible_ok:
|
||||
causes.append("visible_holdout_regression")
|
||||
if not blind_ok:
|
||||
causes.append("blind_holdout_regression")
|
||||
if error_total("winner_holdout", target) > 0:
|
||||
causes.append("current_holdout_gap_present")
|
||||
if holdout_calibration.get("risk_band") in {"overlap", "watch", "tight"}:
|
||||
causes.append("current_holdout_risk")
|
||||
if not judge_ok:
|
||||
causes.append("judge_blind_regression")
|
||||
if not judge_agreement_ok:
|
||||
causes.append("judge_blind_low_agreement")
|
||||
if not adversarial_ok:
|
||||
causes.append("adversarial_regression")
|
||||
if adversarial_risk == "overlap":
|
||||
causes.append("adversarial_overlap_risk")
|
||||
elif adversarial_risk in {"watch", "tight"}:
|
||||
causes.append("adversarial_watch_risk")
|
||||
if not family_ok:
|
||||
causes.append("family_instability")
|
||||
if route_summary.get("misroute_count", 0):
|
||||
causes.append("route_confusion")
|
||||
if route_summary.get("ambiguous_case_count", 0):
|
||||
causes.append("route_ambiguity")
|
||||
if target["winner_tokens"] > target["current_tokens"] and not winner_changed:
|
||||
causes.append("longer_without_gain")
|
||||
|
||||
severe = (
|
||||
route_summary.get("misroute_count", 0) > 0
|
||||
or route_summary.get("ambiguous_case_count", 0) > 0
|
||||
or adversarial_risk == "overlap"
|
||||
or not judge_agreement_ok
|
||||
)
|
||||
|
||||
all_gates_pass = all(
|
||||
[visible_ok, blind_ok, judge_ok, judge_agreement_ok, adversarial_ok, adversarial_risk_ok, route_clean, family_ok]
|
||||
)
|
||||
|
||||
if winner_changed and all_gates_pass:
|
||||
decision = "promote"
|
||||
elif severe:
|
||||
decision = "blocked"
|
||||
else:
|
||||
decision = "keep_current"
|
||||
|
||||
if decision == "promote" and "promotion_ready" not in causes:
|
||||
causes.append("promotion_ready")
|
||||
|
||||
next_action = {
|
||||
"promote": "Review the promoted candidate, update the skill description, and snapshot the release.",
|
||||
"keep_current": "Keep the current description and open a new candidate only when fresh route evidence appears.",
|
||||
"blocked": "Do not promote. Investigate failing gates, route collisions, or calibration risk before generating new candidates.",
|
||||
}[decision]
|
||||
|
||||
return {
|
||||
"decision": decision,
|
||||
"winner_changed": winner_changed,
|
||||
"all_gates_pass": all_gates_pass,
|
||||
"route_clean": route_clean,
|
||||
"judge_agreement_ok": judge_agreement_ok,
|
||||
"causes": causes,
|
||||
"next_action": next_action,
|
||||
"gate_status": {
|
||||
"visible_holdout_non_regression": visible_ok,
|
||||
"blind_holdout_non_regression": blind_ok,
|
||||
"judge_blind_non_regression": judge_ok,
|
||||
"judge_blind_agreement": judge_agreement_ok,
|
||||
"adversarial_non_regression": adversarial_ok,
|
||||
"adversarial_risk_ok": adversarial_risk_ok,
|
||||
"route_confusion_clean": route_clean,
|
||||
"family_stability": family_ok,
|
||||
},
|
||||
"route_metrics": {
|
||||
"precision": route_stats.get("precision"),
|
||||
"recall": route_stats.get("recall"),
|
||||
"average_margin": route_stats.get("average_margin"),
|
||||
"misroute_count": route_summary.get("misroute_count"),
|
||||
"ambiguous_case_count": route_summary.get("ambiguous_case_count"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def build_candidate_entries(target_name: str, target_summary: dict, decision: str) -> list[dict]:
|
||||
report = load_json(OPTIMIZATION_REPORTS[target_name])
|
||||
winner_description = report["winner"]["description"]
|
||||
winner_dev_errors = report["winner"]["dev"]["false_positives"] + report["winner"]["dev"]["false_negatives"]
|
||||
entries = [
|
||||
{
|
||||
"name": target_name,
|
||||
"role": "baseline",
|
||||
"label": "Baseline",
|
||||
"ranking_state": "reference",
|
||||
"promotion_state": "reference",
|
||||
"description": report["baseline"]["description"] if report.get("baseline") else None,
|
||||
"tokens": report["baseline"]["estimated_tokens"] if report.get("baseline") else None,
|
||||
"dev_errors": (
|
||||
report["baseline"]["dev"]["false_positives"] + report["baseline"]["dev"]["false_negatives"]
|
||||
if report.get("baseline")
|
||||
else None
|
||||
),
|
||||
"holdout_errors": (
|
||||
report["baseline"]["holdout"]["false_positives"] + report["baseline"]["holdout"]["false_negatives"]
|
||||
if report.get("baseline") and report["baseline"].get("holdout")
|
||||
else None
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
for candidate in report["candidates"]:
|
||||
selected = candidate["description"] == winner_description
|
||||
role = "current" if candidate["id"] == "current" else "candidate"
|
||||
if selected and decision == "promote":
|
||||
promotion_state = "promoted"
|
||||
elif selected and role == "current":
|
||||
promotion_state = "kept_current"
|
||||
elif selected:
|
||||
promotion_state = "blocked"
|
||||
else:
|
||||
promotion_state = "blocked"
|
||||
|
||||
reason_tags = []
|
||||
candidate_dev_errors = candidate["dev"]["false_positives"] + candidate["dev"]["false_negatives"]
|
||||
if candidate_dev_errors > winner_dev_errors:
|
||||
reason_tags.append("weaker_dev_fit")
|
||||
holdout = candidate.get("holdout") or {}
|
||||
holdout_errors = holdout.get("false_positives", 0) + holdout.get("false_negatives", 0) if holdout else None
|
||||
if holdout_errors is not None and holdout_errors > error_total("current_holdout", target_summary):
|
||||
reason_tags.append("visible_holdout_regression")
|
||||
if candidate["estimated_tokens"] > target_summary["current_tokens"] and not selected:
|
||||
reason_tags.append("longer_without_gain")
|
||||
if not reason_tags and not selected:
|
||||
reason_tags.append("not_selected_by_dev_ranking")
|
||||
|
||||
entries.append(
|
||||
{
|
||||
"name": target_name,
|
||||
"role": role,
|
||||
"label": candidate["label"],
|
||||
"ranking_state": "selected_by_dev" if selected else "not_selected",
|
||||
"promotion_state": promotion_state,
|
||||
"description": candidate["description"],
|
||||
"tokens": candidate["estimated_tokens"],
|
||||
"dev_errors": candidate_dev_errors,
|
||||
"holdout_errors": holdout_errors,
|
||||
"reason_tags": reason_tags,
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def render_candidate_registry_md(targets: list[dict]) -> str:
|
||||
lines = [
|
||||
"# Candidate Registry",
|
||||
"",
|
||||
"| Target | Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors | Reason Tags |",
|
||||
"| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |",
|
||||
]
|
||||
for target in targets:
|
||||
for entry in target["registry"]:
|
||||
lines.append(
|
||||
f"| `{target['name']}` | {entry['role']} | `{entry['label']}` | {entry['ranking_state']} | {entry['promotion_state']} | {'-' if entry['tokens'] is None else entry['tokens']} | {'-' if entry['dev_errors'] is None else entry['dev_errors']} | {'-' if entry['holdout_errors'] is None else entry['holdout_errors']} | {', '.join(entry.get('reason_tags', [])) or '-'} |"
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def render_promotion_md(targets: list[dict]) -> str:
|
||||
lines = [
|
||||
"# Promotion Decisions",
|
||||
"",
|
||||
"| Target | Decision | Winner | Route Clean | Judge Agreement | Adv Risk | Causes | Next Action |",
|
||||
"| --- | --- | --- | --- | --- | --- | --- | --- |",
|
||||
]
|
||||
for target in targets:
|
||||
decision = target["decision"]
|
||||
judge_summary = ((target.get("judge_blind") or {}).get("winner") or {}).get("agreement_rate")
|
||||
adv_risk = ((target.get("calibration") or {}).get("adversarial_holdout") or {}).get("risk_band")
|
||||
lines.append(
|
||||
f"| `{target['name']}` | `{decision}` | `{target['winner_label']}` | {target['promotion']['route_clean']} | {judge_summary if judge_summary is not None else '-'} | {adv_risk or '-'} | {', '.join(target['promotion']['causes'])} | {target['promotion']['next_action']} |"
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def render_bundle_md(bundle: dict) -> str:
|
||||
review = bundle["review_template"]
|
||||
lines = [
|
||||
f"# Iteration Bundle: {bundle['target']}",
|
||||
"",
|
||||
f"- decision: `{bundle['decision']}`",
|
||||
f"- winner label: `{bundle['winner_label']}`",
|
||||
f"- winner changed: `{bundle['winner_changed']}`",
|
||||
f"- next action: {bundle['next_action']}",
|
||||
"",
|
||||
"## Cause Tags",
|
||||
"",
|
||||
]
|
||||
for cause in bundle["cause_tags"]:
|
||||
lines.append(f"- `{cause}`")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Gate Status",
|
||||
"",
|
||||
"| Gate | Pass |",
|
||||
"| --- | --- |",
|
||||
]
|
||||
)
|
||||
for gate, passed in bundle["gate_status"].items():
|
||||
lines.append(f"| `{gate}` | {passed} |")
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Candidate Registry",
|
||||
"",
|
||||
"| Role | Label | Ranking State | Promotion State | Tokens | Dev Errors | Holdout Errors |",
|
||||
"| --- | --- | --- | --- | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for entry in bundle["candidate_registry"]:
|
||||
lines.append(
|
||||
f"| {entry['role']} | `{entry['label']}` | {entry['ranking_state']} | {entry['promotion_state']} | {'-' if entry['tokens'] is None else entry['tokens']} | {'-' if entry['dev_errors'] is None else entry['dev_errors']} | {'-' if entry['holdout_errors'] is None else entry['holdout_errors']} |"
|
||||
)
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Human Review Stub",
|
||||
"",
|
||||
f"- target: {review['target']}",
|
||||
f"- current description: {review['current_description']}",
|
||||
f"- candidate description: {review['candidate_description']}",
|
||||
f"- review focus: {', '.join(review['focus'])}",
|
||||
"",
|
||||
"## Artifact Paths",
|
||||
"",
|
||||
]
|
||||
)
|
||||
for label, path in bundle["artifacts"].items():
|
||||
lines.append(f"- {label}: `{path}`")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Apply promotion policy and build iteration bundles.")
|
||||
parser.add_argument("--optimization-suite", default="reports/description_optimization_suite.json")
|
||||
parser.add_argument("--route-scorecard", default="reports/route_scorecard.json")
|
||||
parser.add_argument("--output-json", default="reports/promotion_decisions.json")
|
||||
parser.add_argument("--output-md", default="reports/promotion_decisions.md")
|
||||
parser.add_argument("--candidate-registry-json", default="reports/candidate_registry.json")
|
||||
parser.add_argument("--candidate-registry-md", default="reports/candidate_registry.md")
|
||||
parser.add_argument("--bundle-dir", default="reports/iteration_bundles")
|
||||
args = parser.parse_args()
|
||||
|
||||
suite = load_json(ROOT / args.optimization_suite)
|
||||
route_scorecard = load_json(ROOT / args.route_scorecard)
|
||||
bundle_dir = ROOT / args.bundle_dir
|
||||
bundle_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
targets = []
|
||||
decisions = []
|
||||
for target in suite["targets"]:
|
||||
promotion = classify_target(target, route_scorecard)
|
||||
registry = build_candidate_entries(target["name"], target, promotion["decision"])
|
||||
target["promotion"] = promotion
|
||||
target["registry"] = registry
|
||||
target["decision"] = promotion["decision"]
|
||||
targets.append(target)
|
||||
decisions.append(promotion["decision"])
|
||||
|
||||
bundle = {
|
||||
"target": target["name"],
|
||||
"decision": promotion["decision"],
|
||||
"winner_label": target["winner_label"],
|
||||
"winner_changed": promotion["winner_changed"],
|
||||
"next_action": promotion["next_action"],
|
||||
"cause_tags": promotion["causes"],
|
||||
"gate_status": promotion["gate_status"],
|
||||
"candidate_registry": registry,
|
||||
"review_template": {
|
||||
"target": target["name"],
|
||||
"current_description": target["winner_description"] if target["winner_label"] == "Current" else load_json(OPTIMIZATION_REPORTS[target["name"]])["current_description"],
|
||||
"candidate_description": target["winner_description"],
|
||||
"focus": promotion["causes"][:3] or ["promotion_ready"],
|
||||
},
|
||||
"artifacts": {
|
||||
"skill": relative(SKILL_PATHS[target["name"]]),
|
||||
"optimization_report": relative(OPTIMIZATION_REPORTS[target["name"]]),
|
||||
"promotion_decisions": relative(ROOT / args.output_json),
|
||||
"candidate_registry": relative(ROOT / args.candidate_registry_json),
|
||||
"regression_cause_taxonomy": "references/regression-cause-taxonomy.md",
|
||||
"human_review_template": "references/human-review-template.md",
|
||||
},
|
||||
}
|
||||
target_dir = bundle_dir / target["name"]
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
(target_dir / "bundle.json").write_text(json.dumps(bundle, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
(target_dir / "bundle.md").write_text(render_bundle_md(bundle), encoding="utf-8")
|
||||
(target_dir / "review.md").write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"# Human Review",
|
||||
"",
|
||||
f"- target: {bundle['review_template']['target']}",
|
||||
f"- current description: {bundle['review_template']['current_description']}",
|
||||
f"- candidate description: {bundle['review_template']['candidate_description']}",
|
||||
f"- suggested focus: {', '.join(bundle['review_template']['focus'])}",
|
||||
"",
|
||||
"Use the shared template in `references/human-review-template.md` to complete the final decision.",
|
||||
"",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
summary = {
|
||||
"ok": "blocked" not in decisions,
|
||||
"summary": {
|
||||
"promote": sum(1 for d in decisions if d == "promote"),
|
||||
"keep_current": sum(1 for d in decisions if d == "keep_current"),
|
||||
"blocked": sum(1 for d in decisions if d == "blocked"),
|
||||
},
|
||||
"targets": targets,
|
||||
}
|
||||
|
||||
output_json = ROOT / args.output_json
|
||||
output_md = ROOT / args.output_md
|
||||
candidate_json = ROOT / args.candidate_registry_json
|
||||
candidate_md = ROOT / args.candidate_registry_md
|
||||
output_json.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
output_md.write_text(render_promotion_md(targets), encoding="utf-8")
|
||||
candidate_json.write_text(json.dumps({"targets": targets}, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
candidate_md.write_text(render_candidate_registry_md(targets), encoding="utf-8")
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
if not summary["ok"]:
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -15,7 +15,9 @@ def load_snapshots(history_dir: Path) -> list[dict]:
|
||||
return [load_json(path) for path in sorted(history_dir.glob("*.json"))]
|
||||
|
||||
|
||||
def render_markdown(system_snapshots: list[dict], description_snapshots: list[dict], route_scorecard: dict) -> str:
|
||||
def render_markdown(
|
||||
system_snapshots: list[dict], description_snapshots: list[dict], route_scorecard: dict, promotion_decisions: dict
|
||||
) -> str:
|
||||
lines = [
|
||||
"# Iteration Ledger",
|
||||
"",
|
||||
@@ -53,6 +55,20 @@ def render_markdown(system_snapshots: list[dict], description_snapshots: list[di
|
||||
f"| {snapshot['date']} | {snapshot['label']} | `{target['name']}` | {'-' if blind_errors is None else blind_errors} | {'-' if judge_blind_errors is None else judge_blind_errors} | {judge_blind.get('agreement_rate', '-')} | {'-' if adversarial_errors is None else adversarial_errors} | {'-' if adversarial_calibration.get('score_gap') is None else adversarial_calibration.get('score_gap')} | {adversarial_calibration.get('risk_band', '-')} | {target.get('drift_note', '-')} |"
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Current Promotion Decisions",
|
||||
"",
|
||||
"| Target | Decision | Winner | Causes | Next Action |",
|
||||
"| --- | --- | --- | --- | --- |",
|
||||
]
|
||||
)
|
||||
for target in promotion_decisions.get("targets", []):
|
||||
lines.append(
|
||||
f"| `{target['name']}` | `{target['decision']}` | `{target['winner_label']}` | {', '.join(target['promotion']['causes'])} | {target['promotion']['next_action']} |"
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
@@ -79,17 +95,19 @@ def main() -> None:
|
||||
parser.add_argument("--history-dir", default="evals/history")
|
||||
parser.add_argument("--description-history-dir", default="evals/history/description_optimization")
|
||||
parser.add_argument("--route-scorecard", default="reports/route_scorecard.json")
|
||||
parser.add_argument("--promotion-decisions", default="reports/promotion_decisions.json")
|
||||
parser.add_argument("--output", default="reports/iteration_ledger.md")
|
||||
args = parser.parse_args()
|
||||
|
||||
system_snapshots = load_snapshots(ROOT / args.history_dir)
|
||||
description_snapshots = load_snapshots(ROOT / args.description_history_dir)
|
||||
route_scorecard = load_json(ROOT / args.route_scorecard)
|
||||
promotion_decisions = load_json(ROOT / args.promotion_decisions)
|
||||
|
||||
output = ROOT / args.output
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(
|
||||
render_markdown(system_snapshots, description_snapshots, route_scorecard),
|
||||
render_markdown(system_snapshots, description_snapshots, route_scorecard, promotion_decisions),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def main() -> None:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "scripts/promotion_checker.py"],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
payload = json.loads(proc.stdout)
|
||||
if proc.returncode != 0:
|
||||
raise SystemExit(proc.returncode)
|
||||
|
||||
summary = payload["summary"]
|
||||
assert summary["blocked"] == 0, payload
|
||||
assert summary["keep_current"] >= 1, payload
|
||||
|
||||
targets = payload["targets"]
|
||||
assert len(targets) >= 3, payload
|
||||
|
||||
candidate_registry = json.loads((ROOT / "reports" / "candidate_registry.json").read_text(encoding="utf-8"))
|
||||
assert len(candidate_registry["targets"]) == len(targets)
|
||||
|
||||
for target in targets:
|
||||
assert target["decision"] in {"promote", "keep_current", "blocked"}
|
||||
assert target["promotion"]["gate_status"]["route_confusion_clean"] is True
|
||||
assert target["promotion"]["gate_status"]["judge_blind_agreement"] is True
|
||||
assert target["promotion"]["causes"], target
|
||||
if target["winner_label"] == "Current":
|
||||
assert target["decision"] == "keep_current", target
|
||||
if target["name"] == "governed-incident-command":
|
||||
assert "current_holdout_gap_present" in target["promotion"]["causes"], target
|
||||
|
||||
bundle_dir = ROOT / "reports" / "iteration_bundles" / target["name"]
|
||||
assert (bundle_dir / "bundle.json").exists(), bundle_dir
|
||||
assert (bundle_dir / "bundle.md").exists(), bundle_dir
|
||||
assert (bundle_dir / "review.md").exists(), bundle_dir
|
||||
|
||||
registry_entries = target["registry"]
|
||||
roles = {entry["role"] for entry in registry_entries}
|
||||
assert "baseline" in roles and "current" in roles, target
|
||||
assert any(entry["role"] == "candidate" for entry in registry_entries), target
|
||||
|
||||
print(json.dumps({"ok": True, "summary": summary}, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user