tracer-cloud--opensre/docs/closed-loop-learning.mdx

---
title: "Closed-Loop Learning"
description: "Turn production investigation misses into regression evals"
---

# Closed-Loop Learning

OpenSRE captures accuracy feedback after every investigation. When you mark a result as **partial** or **inaccurate**, it is classified into a triage taxonomy and recorded as a *miss*. The `opensre misses` command surface lets you review trends, track recurrence, and convert top misses into reproducible benchmark scenarios — closing the loop from production usage back into the eval suite.

## Quick reference

| Command | What it does |
|---------|--------------|
| `opensre misses list` | Show recent misses with alert, taxonomy, rating, and root cause. |
| `opensre misses stats` | Taxonomy breakdown plus recurring `(alert, taxonomy)` pairs. |
| `opensre misses export --out PATH` | Write per-case `alert.json` files the benchmark runner can consume. |
| `opensre misses convert MISS_ID` | Convert a single miss into a scenario payload (stdout or `--out FILE`). |

## How a miss is captured

After every investigation the CLI shows the accuracy prompt. If you pick **partial** or **inaccurate** you'll be asked for a short note and a taxonomy bucket:

- **Retrieval gap** — the agent did not fetch the evidence it needed.
- **Reasoning gap** — it had the evidence but drew the wrong conclusion.
- **Tool failure** — a tool errored, timed out, or returned bad data.
- **Routing/prompt failure** — the wrong tools or plan were selected.
- **Unknown** — choose this only when none of the above clearly fit.

The miss is written to `~/.opensre/misses.jsonl` and an `investigation_miss_classified` event is emitted to PostHog with the run provenance, taxonomy, and (when available) `user_id` / `org_id`. The original feedback record in `~/.opensre/feedback.jsonl` is untouched.

## Reviewing trends

```bash
# Everything captured in the last week
opensre misses stats --since 7d

# Drill into just the retrieval gaps
opensre misses list --since 14d --taxonomy retrieval_gap

# Machine-readable output for dashboards or pipelines
opensre misses stats --since 30d --json
```

`stats` reports the **count per taxonomy** and the **recurring `(alert_name, taxonomy)` pairs** (seen more than once). Recurring pairs are the strongest signal that a regression scenario is overdue.

## Converting misses to regressions

`opensre misses export` writes one scenario per recurring `(alert, taxonomy)` pair, ordered by how often it has recurred. The output uses the same benchmark scenario `alert.json` shape, so the benchmark runner consumes it without any adapter changes:

```bash
opensre misses export \
  --since 7d --top 10 \
  --out tests/benchmarks/production_misses/
```

Each case directory contains an `alert.json` whose `commonAnnotations.scoring_points` dict (`expected_root_cause`, `expected_category`, `miss_notes`) carries the rubric for grading — the same location `opensre investigate --evaluate` already reads from, and the same one `strip_scoring_points_from_alert` removes before the agent sees the alert. The `_meta` block carries non-rubric provenance (`miss_id`, `original_run_id`, `taxonomy`). Commit the directory under `tests/benchmarks/` and the next benchmark run will include the new regressions.

## Weekly triage workflow

| Step | Owner | SLA |
|------|-------|-----|
| Run `opensre misses stats --since 7d` and review top recurring pairs | On-call engineer | Monday morning |
| Run `opensre misses export --since 7d --top 10 --out tests/benchmarks/production_misses/` | On-call engineer | Monday |
| Open a PR adding the new scenarios with a `benchmark` label | On-call engineer | Tuesday |
| Run the benchmark workflow against the PR branch | Reviewer | Wednesday |
| Track fix-rate week-over-week using PostHog `investigation_miss_classified` trends | Eng lead | Ongoing |

PostHog dashboards built on `investigation_miss_classified` (grouped by `taxonomy` and `alert_name`) provide the week-over-week trend view referenced by the SLAs.

## Privacy

Miss records live entirely on the engineer's machine in `~/.opensre/misses.jsonl`. To delete everything captured locally, remove the file.

The `investigation_miss_classified` PostHog event carries identifiers and structured metadata only:

- `miss_id`, `feedback_id`, `run_id`
- `taxonomy`, `rating`, `has_detail` (boolean — whether a note was provided, never the note itself)
- `alert_name`, `pipeline_name`, `root_cause_category`
- Optional `user_id`, `org_id` when running on a hosted/JWT path

The free-text note (`taxonomy_detail`) and the captured `root_cause` string are **never** sent to PostHog — they only exist in the local JSONL store, so removing `~/.opensre/misses.jsonl` removes them entirely.