chore: import upstream snapshot with attribution

docs/REPRODUCING.md

@@ -0,0 +1,429 @@
+# Reproducing the Benchmarks
+
+This document gives the exact commands to reproduce every benchmark number
+shown in the README and the `diagrams/`. Two people running the recipe below
+on different machines on different days should produce identical numbers,
+within float rounding.
+
+If you get different numbers, that's a bug — please file an issue.
+
+## Verifying the "saved tokens" number
+
+The CLI's `Token Savings` panel uses a `chars / 4` approximation labelled
+`estimated: true`, not a model-specific tokenizer. The approximation is
+designed to be both fast (no model load, no inference) and conservative.
+
+### How to verify against a real tokenizer
+
+```bash
+pip install tiktoken
+code-review-graph detect-changes --brief --verify
+```
+
+The panel grows a `Verified (tiktoken)` row showing the same calculation
+done with OpenAI's `cl100k_base` tokenizer (the GPT-4 family). If the
+estimate is significantly off, you'll see it immediately:
+
+```text
+┌───────────────────────── Token Savings ─────────────────────────┐
+│ Full context would be:     12,921 tokens                        │
+│ Graph context used:           762 tokens                        │
+│ Saved:                     12,159 tokens (~94%)                 │
+│ Verified (tiktoken):       10,835 tokens (~93%)  [11,611 → 776] │
+│ Breakdown: Functions 244 · Tests 191 · Risk 244 · Other 83      │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+### Calibration result (committed)
+
+A one-time calibration across 222 files / 2.2 MB of mixed source
+(Python, JS, TS, Go, Rust, RST, MD) pulled from the 6 test repos:
+
+| Repo | sample files | bytes | chars/4 estimate | tiktoken real | ratio est/real |
+|---|---:|---:|---:|---:|---:|
+| flask | 46 | 470,179 | 117,559 | 109,969 | 1.069 |
+| fastapi | 38 | 156,224 | 39,072 | 34,897 | 1.120 |
+| gin | 30 | 471,793 | 117,962 | 132,296 | 0.892 |
+| express | 23 | 296,805 | 74,207 | 83,575 | 0.888 |
+| httpx | 38 | 254,184 | 63,556 | 62,909 | 1.010 |
+| code-review-graph | 47 | 539,206 | 134,820 | 120,760 | 1.116 |
+| **OVERALL** | **222** | **2,188,391** | **547,176** | **544,406** | **1.005** |
+
+`chars / 4` is within **+0.5%** of real GPT-4 tokens in aggregate. Per-repo
+it swings between **-11%** (gin: lots of short Go identifiers) and **+12%**
+(fastapi: heavy docstrings and type hints), but the **ratio** stabilizes
+because both sides of the divide are equally biased.
+
+Reproduce the calibration with the snippet in this commit's
+`code_review_graph/context_savings.py:verify_with_tiktoken`, or
+inline-run the `--verify` flag on any commit.
+
+## What is and isn't deterministic
+
+| Reproducible | Reason |
+|---|---|
+| Tree-sitter parsing | Pure function of input bytes |
+| Node / edge counts | Deterministic upserts keyed by `qualified_name` |
+| FTS5 BM25 scores | Deterministic |
+| Embeddings via `all-MiniLM-L6-v2` on CPU | Model weights cache-pinned by SHA in HuggingFace cache |
+| Leiden community IDs | Seeded — `_LEIDEN_SEED=42` in `communities.py`, override with `CRG_LEIDEN_SEED` env var |
+| `naive_corpus_tokens` | Deterministic for a fixed git checkout |
+| `git clone` at a pinned SHA | Determines the source-of-truth byte stream |
+
+What used to make it **non**-reproducible (now fixed):
+
+- `commit: HEAD` in every `code_review_graph/eval/configs/*.yaml` — replaced with the pinned latest test-commit SHA per repo
+- `git clone --depth 50` silently fell back to wrong commits when the pinned SHAs were beyond the shallow window — now uses full clones with explicit `returncode` checks
+- Leiden ran with an unseeded RNG — now seeded
+- `nextjs.yaml` was a misnamed config evaluating this repo — renamed to `code-review-graph.yaml`
+- FTS5 was created but never populated by the eval framework's `full_build` call — `code_review_graph/eval/runner.py` now calls `postprocessing.run_post_processing` directly
+
+## Prerequisites
+
+- Python 3.10 or newer
+- `git` on PATH
+- Network access (~600 MB to clone the 6 upstream repos)
+- ~3 GB free disk
+- For the embedding step: roughly 700 MB extra for `torch` + `sentence-transformers`
+
+## Step 1 — Install with the right extras
+
+```bash
+git clone https://github.com/tirth8205/code-review-graph
+cd code-review-graph
+
+# eval extras: pyyaml + matplotlib (matplotlib only needed for `--report`)
+# embeddings extras: sentence-transformers + numpy
+uv sync --extra eval --extra embeddings     # or: pip install -e ".[eval,embeddings]"
+```
+
+## Step 2 — Run the formal eval
+
+This step clones 6 upstream repositories at pinned SHAs, builds a full graph
+for each (parser + cross-file resolvers + signatures + FTS5 + flows + Leiden
+communities), then runs the `token_efficiency`, `impact_accuracy`,
+`agent_baseline`, and `multi_hop_retrieval` benchmarks.
+
+```bash
+uv run code-review-graph eval \
+  --benchmark token_efficiency,impact_accuracy,agent_baseline,multi_hop_retrieval
+```
+
+Failure semantics (applies to every benchmark): a thrown tool call is **not**
+a measurement. The row is kept in the CSV with `status=error` for forensics,
+but excluded from every aggregate. (Two historical bugs made failures look
+like wins: a thrown `get_review_context` produced `graph_tokens=0` and a
+ratio of `naive/1`, and a thrown `analyze_changes` silently set
+`predicted = changed`, guaranteeing recall 1.0. Both are fixed; regression
+tests live in `tests/test_eval.py`.)
+
+Expected runtime on an M1/M2 Mac: roughly 8–15 minutes for the build phase,
+plus seconds per benchmark.
+
+Outputs:
+
+- `evaluate/test_repos/{express,fastapi,flask,gin,httpx,code-review-graph}/`
+- `evaluate/test_repos/<name>/.code-review-graph/graph.db`
+- `evaluate/results/<name>_<benchmark>_<date>.csv`
+
+## Step 3 — Generate embeddings (required for the standalone benchmark)
+
+The standalone token benchmark ships with 5 hardcoded natural-language
+questions. Without embeddings, hybrid search can't match them and the
+benchmark silently returns 0× reduction ratios (a loud warning will print).
+
+```bash
+for repo in express fastapi flask gin httpx code-review-graph; do
+  uv run code-review-graph embed --repo "evaluate/test_repos/$repo"
+done
+```
+
+Expected runtime: 2–5 minutes total. Vectors live inside the same `graph.db`.
+
+## Step 4 — Run the standalone token benchmark
+
+This benchmark compares **all source-file tokens** in the repo against
+**5 search hits + a few neighbor edges** for each of 5 sample questions. The
+ratio answers: *how many tokens does the graph let me skip on a typical
+question?*
+
+```bash
+uv run python <<'PY'
+import json
+from pathlib import Path
+from code_review_graph.graph import GraphStore
+from code_review_graph.token_benchmark import run_token_benchmark
+
+results = {}
+for repo in sorted(Path("evaluate/test_repos").iterdir()):
+    db = repo / ".code-review-graph" / "graph.db"
+    if not db.exists():
+        continue
+    store = GraphStore(str(db))
+    try:
+        results[repo.name] = run_token_benchmark(store, repo)
+    finally:
+        store.close()
+
+print(f"{'Repo':<22}{'naive_tokens':>16}{'avg_graph_tokens':>20}{'avg_ratio':>14}")
+print("-" * 72)
+for name, out in sorted(results.items(), key=lambda x: -x[1]["average_reduction_ratio"]):
+    pq = out["per_question"]
+    avg_graph = int(sum(r["graph_tokens"] for r in pq) / max(len(pq), 1))
+    print(f"{name:<22}{out['naive_corpus_tokens']:>16,}"
+          f"{avg_graph:>20,}{out['average_reduction_ratio']:>13.1f}×")
+
+Path("evaluate/standalone_token_benchmark.json").write_text(json.dumps(results, indent=2))
+PY
+```
+
+## Canonical numbers
+
+<!-- BEGIN canonical-stats -->
+Captured **2026-05-25** on macOS arm64, Python 3.11, sentence-transformers 5.5.1,
+`all-MiniLM-L6-v2`, `CRG_LEIDEN_SEED=42`. If your numbers differ by more than
+rounding, something in the chain has drifted — file an issue.
+
+### Standalone token benchmark (`code_review_graph/token_benchmark.py`)
+
+Each row is the average of 5 sample questions (`how does authentication work`,
+`what is the main entry point`, `how are database connections managed`,
+`what error handling patterns are used`, `how do tests verify core functionality`).
+
+| Repo | snapshot SHA | naive_corpus_tokens | avg graph_tokens | avg ratio |
+|---|---|---:|---:|---:|
+| fastapi | `0227991a` | 951,071 | 2,169 | **528.4×** |
+| code-review-graph | `84bde354` | 208,821 | 2,495 | **93.0×** |
+| gin | `5c00df8a` | 166,868 | 1,990 | **91.8×** |
+| flask | `a29f88ce` | 125,022 | 1,986 | **71.4×** |
+| express | `b4ab7d65` | 135,955 | 3,465 | **40.6×** |
+| httpx | `b55d4635` | 89,492 | 2,438 | **38.0×** |
+
+Range across 6 repos: **38× – 528×**. The numbers shifted down from a
+previous capture because (a) the test repos are now wiped/re-cloned from
+scratch — no leftover build artifacts or local caches inflate the naive
+baseline; and (b) the embedding text per node became richer in this same
+release (see `embeddings._node_to_text`), so the graph response itself is
+slightly bigger. Both are correctness improvements over the prior numbers.
+
+### Formal `token_efficiency` benchmark (`code_review_graph/eval/benchmarks/token_efficiency.py`)
+
+A different denominator: just the **changed-file content** for each commit,
+vs the full `get_review_context()` JSON. For small commits the response is
+larger than the input (it carries impact-radius edges + source snippets), so
+ratios here are intentionally < 1.0 — that is not a bug, it measures a
+different thing than the standalone benchmark.
+
+Raw per-commit CSVs in `evaluate/results/<repo>_token_efficiency_*.csv`.
+
+### Impact accuracy (`code_review_graph/eval/benchmarks/impact_accuracy.py`)
+
+13 commits across 6 repos. The benchmark emits two ground-truth modes side
+by side, distinguished by the `ground_truth_mode` CSV column:
+
+| Mode | Ground truth | What it tells you |
+|---|---|---|
+| `graph-derived (circular — upper bound)` | changed files + files with CALLS/IMPORTS_FROM edges into them — **derived from the same graph the predictor traverses** | An upper bound. Recall 1.0 here is partly true by construction, not independent evidence. |
+| `co-change (same commit, seed excluded)` | the *other* files the author actually touched in the same commit, given a single seed file | Independent-ish evidence from git history. Expect substantially lower recall. |
+
+The canonical numbers below were captured **in graph-derived mode only**
+(the co-change mode did not exist at capture time). Treat the recall row as
+a circular upper bound, not as "100% recall":
+
+| Metric (graph-derived mode — circular upper bound) | Value |
+|---|---|
+| Recall (mean across 13 commits) | **1.000** (upper bound on every commit) |
+| F1 (mean) | **0.714** |
+| F1 (median) | 0.667 |
+| F1 (min / max) | 0.455 / 1.000 |
+
+Canonical co-change numbers will be added after the next full capture — we
+do not quote them before measuring. Single-file commits are recorded with
+`status=skipped` in co-change mode (there is nothing independent to grade
+against).
+
+The blast-radius analysis over-predicts in some commits (precision ≈ 0.30 in the
+worst case, where 34 files are flagged for a 10-file change). That is
+intentional: a missed dependency is worse than an extra reviewed file.
+
+### Multi-hop retrieval (`code_review_graph/eval/benchmarks/multi_hop_retrieval.py`)
+
+11 hand-curated tasks across the 6 repos. Each task is a 2-step tool chain:
+
+1. `hybrid_search(nl_query, limit=10)` looks for a starting anchor node.
+2. `query_graph(<traversal_pattern>, target=<anchor>)` walks one hop along
+   `callers_of` / `callees_of` / `tests_for` / `imports_of` / etc.
+
+The task **scores 1.0** only if both the anchor is found in the top-K *and*
+the expected neighbor names are returned by the traversal. **Scores 0.0**
+otherwise (which collapses both "search missed the anchor" and "traversal
+returned the wrong set" — split those by inspecting `anchor_found` and
+`neighbor_recall` in the per-task CSV row).
+
+| Repo | Task | Anchor found | Rank | Neighbor recall | Score |
+|---|---|---|---:|---:|---:|
+| code-review-graph | crg-parse-file-callers | yes | 0 | 1.00 | **1.00** |
+| code-review-graph | crg-upsert-node-callers | yes | 4 | 1.00 | **1.00** |
+| express | express-create-application-callees | yes | 1 | 1.00 | **1.00** |
+| fastapi | fastapi-route-handler-callers | yes | 6 | 1.00 | **1.00** |
+| fastapi | fastapi-get-dependant-callers | no | — | 0.00 | **0.00** |
+| flask | flask-dispatch-callers | yes | 3 | 1.00 | **1.00** |
+| flask | flask-exception-callers | yes | 5 | 1.00 | **1.00** |
+| gin | gin-serve-http-callees | yes | 5 | 1.00 | **1.00** |
+| gin | gin-context-next-callers | yes | 0 | 1.00 | **1.00** |
+| httpx | httpx-client-request-callers | yes | 0 | 1.00 | **1.00** |
+| httpx | httpx-async-request-tests | yes | 7 | 1.00 | **1.00** |
+
+**Average score across 11 tasks: 0.909**. 10/11 tasks pass; the one remaining
+miss (`fastapi-get-dependant-callers`) targets a function spelled `get_dependant`
+("dependant" with an `a`) from a query phrased as "dependency declarations into
+a tree" — there is no lexical overlap and no extractable identifier in the
+query for the boosting heuristic to lock onto. Left as an honest miss; the
+fix would be either query rewriting or a richer embedding model.
+
+#### How the score went from 0.545 to 0.909 (the same-day fix)
+
+The v1 scaffold first scored **0.545** (6/11). Two changes brought it to
+**0.909** (10/11), both deterministic, both small, both committed in this
+same session:
+
+1. **`embeddings.py:_node_to_text`** — the embedded text per node used to be
+   just `"{name} {kind} in {parent}"`. It now also includes the dotted form
+   (`APIRoute.get_route_handler`), the identifier split into words
+   (`get route handler`), and the enclosing module directory (`routing`,
+   `fastapi`, `dependencies`). All re-embeddings are automatic — the text
+   hash changes, `EmbeddingStore.embed_nodes` re-embeds. See
+   `_split_identifier` for the casing/separator rules.
+
+2. **`search.py:extract_query_identifiers`** — natural-language queries
+   like "Who advances the gin middleware chain via Context.Next" now have
+   their dotted / snake_case / CamelCase identifier tokens extracted. Search
+   results whose `qualified_name` contains any extracted identifier get a
+   2.0× boost. This pushed `Context.Next` from rank 11 to rank 0.
+
+The remaining `fastapi-get-dependant-callers` failure cannot be fixed by
+either change because the query doesn't share any identifier or substring
+with the target — that's the boundary of the heuristic.
+
+This benchmark is a v1 scaffold (11 tasks). The intent is to track the
+**multi-hop tool chain** as the agent's actual usage pattern rather than just
+single-shot retrieval. Adding more tasks: append `multi_hop_tasks:` entries
+to any config under `code_review_graph/eval/configs/*.yaml` with the schema:
+
+```yaml
+multi_hop_tasks:
+  - id: my-task-id                # required, unique
+    nl_query: "natural language" # required, what an agent would ask
+    anchor_qualified_suffix:     # required, lowercased suffix of expected
+      "rel/path.py::owner.symbol" #   qualified_name (case-insensitive endswith)
+    traversal_pattern: callers_of # one of callers_of|callees_of|imports_of|
+                                  # importers_of|tests_for|inheritors_of|children_of
+    expected_neighbor_names:      # required, list of bare names that should
+      - "expected_one"            #   appear in the traversal result
+    k: 10                         # optional, top-K depth for the search step
+```
+
+### Build stats
+
+| Repo | Nodes | Edges | Flows | Communities | Embeddings | FTS idx rows |
+|---|---:|---:|---:|---:|---:|---:|
+| fastapi | 6,292 | 32,081 | 165 | 85 | 5,164 | 127 |
+| express | 1,912 | 18,877 | 4 | 7 | 1,771 | 47 |
+| gin | 1,589 | 17,237 | 114 | 41 | 1,491 | 29 |
+| code-review-graph | 1,418 | 8,877 | 104 | 11 | 1,326 | 38 |
+| flask | 1,415 | 8,259 | 78 | 13 | 1,329 | 35 |
+| httpx | 1,261 | 8,228 | 128 | 5 | 1,193 | 34 |
+
+Embeddings count is lower than node count because File nodes aren't
+embedded. FTS idx rows are far lower than node count because FTS5 stores
+inverted-index segments, not one row per indexed document.
+<!-- END canonical-stats -->
+
+## Agent baseline benchmark (`code_review_graph/eval/benchmarks/agent_baseline.py`)
+
+The whole-corpus baseline in the standalone token benchmark is an upper
+bound no real agent pays. This benchmark simulates what an agent actually
+does without the graph:
+
+1. Derive search terms from each question in the config's `agent_questions:`
+   list (identifier-shaped tokens via `search.extract_query_identifiers`,
+   plus plain keywords; falls back to the `search_queries` query strings
+   when absent).
+2. Pure-python grep over the corpus (no external `rg`/`grep` binary),
+   ranking source files by total case-insensitive match count
+   (deterministic; ties break on path).
+3. Read the top-3 files and token-count them (`chars/4`) as
+   `baseline_tokens`.
+4. Compare against the graph-query cost for the same question (5 hybrid
+   search hits + up to 5 neighbor edges per hit — the same accounting as the
+   standalone benchmark).
+
+Output: `evaluate/results/<repo>_agent_baseline_<date>.csv` with a
+`baseline_to_graph_ratio` per question. Rows where either side is zero are
+marked `status=no_graph_results` / `status=no_baseline_match` and excluded
+from aggregates (`agent_baseline.aggregate`). No canonical capture exists
+yet; numbers will be added to the canonical block above once captured —
+they are not quoted before being measured.
+
+## Weekly CI run (report-only)
+
+`.github/workflows/eval.yml` runs every Monday at 06:23 UTC (plus manual
+`workflow_dispatch`) against the two smallest pinned configs (`httpx`,
+`flask`) with the `token_efficiency`, `impact_accuracy`, and
+`agent_baseline` benchmarks. It uploads the CSVs as an artifact and writes
+a job-summary table. It is deliberately **report-only**: regressions do not
+fail the default branch yet.
+
+## Which benchmark measures what
+
+There are four different "token" benchmarks in the repo. They are all valid
+but measure different scenarios:
+
+| Benchmark | Naive baseline | Graph cost | Question answered |
+|---|---|---|---|
+| `code_review_graph/eval/benchmarks/token_efficiency.py` | sum of **changed-file content** for a specific commit | full `get_review_context()` JSON | "Is the graph cheaper than just reading the diffed files?" |
+| `code_review_graph/eval/benchmarks/agent_baseline.py` | **grep top-3 files** for the question's identifiers | 5 search hits + 5 neighbor edges per question | "Is the graph cheaper than a realistic grep-and-read agent?" |
+| `code_review_graph/eval/token_benchmark.py` | none — absolute per-workflow cost | sum of 5 MCP-tool responses | "How many tokens does a complete agent workflow cost?" |
+| `code_review_graph/token_benchmark.py` (standalone) | sum of **all source files** in repo | 5 search hits + 5 neighbor edges per question | "Is the graph cheaper than reading the whole repo?" |
+
+The `code_review_graph/eval/benchmarks/token_efficiency.py` numbers can be **less than 1.0×**
+for small commits (`get_review_context` carries impact-radius metadata and
+source snippets, which outweigh a tiny changed-file set). The standalone
+benchmark numbers are **always large** because the baseline is the entire
+repo — that is why the README leads with the median (~82×) and treats 528×
+as the max, and why `agent_baseline` exists as the realistic middle ground.
+Pick the one that matches the scenario you're talking about.
+
+## Generating diagrams
+
+The 9 diagrams in `diagrams/` are produced from `diagrams/generate_diagrams.py`.
+Excalidraw source files (`.excalidraw`) are gitignored (`*.excalidraw` line in
+`.gitignore`); only the rendered PNGs are tracked. Regenerate after a
+benchmark refresh:
+
+```bash
+uv run python diagrams/generate_diagrams.py
+# Open each .excalidraw at https://excalidraw.com to render/export
+```
+
+## Troubleshooting
+
+**`git clone failed`** — Network or upstream rate-limit. The fix is a clean
+retry; the eval doesn't auto-retry by design (loud failures > silent
+fallback).
+
+**`git checkout <sha> failed`** — Upstream rewrote history or removed the
+SHA. File an issue with the failing config so we can re-pin.
+
+**`No embeddings found in this graph`** warning during the standalone
+benchmark — you skipped Step 3. Run it.
+
+**Different community IDs between runs** — Make sure you're on the seeded
+`communities.py`. Check `grep _LEIDEN_SEED code_review_graph/communities.py`.
+You can override the seed via `CRG_LEIDEN_SEED=<int>` but all collaborators
+must agree on the same value.
+
+**Different `naive_corpus_tokens` than the canonical table** — Make sure
+`git rev-parse HEAD` inside each `evaluate/test_repos/<name>` matches the
+`commit:` field in the corresponding config file. If not, delete the clone
+and let Step 2 re-clone at the pinned SHA.