# codebase-memory-mcp — Evaluation Plan (159 Languages) > **Status:** Working plan document, **for peer review prior to execution**. This is a **plan, not a > result set** — it defines *how* the next evaluation is run and contains no scores. Execution happens > **downstream**, after this specification is reviewed; §15 requires a pilot run before the full sweep. > > **Single version.** There are no `v1/`, `v8/`, `v(x)/` result directories anymore — that scheme > is retired. Every run writes to one flat output tree (`eval-results/`) and overwrites the > previous run. History lives in git, not in versioned folders. --- ## 1. Purpose Measure how well `codebase-memory-mcp`'s structured knowledge-graph queries answer real developer questions compared to plain text exploration (Grep / Glob / Read), across **all 159 supported languages**, and — for the 9 LSP-hybrid languages — how well the deeper capabilities (**cross-repo intelligence** and **semantic / similarity edges**) actually perform. Two conditions answer the **same questions** on the **same repository** per language: | Condition | Tools allowed | Who runs it | |-----------|---------------|-------------| | **Graph** (the product) | Only MCP graph tools: `search_graph`, `trace_call_path`/`trace_path`, `query_graph`, `get_code_snippet`, `get_architecture`, `search_code`, `semantic_query` | Hybrid — see §4 | | **Explorer** (the baseline) | Only `Grep`, `Glob`, `Read` | Backgrounded `Explore` sub-agents | A third evaluator — an **LLM-as-a-Judge** (§9) — grades both answer sets blind against the actual source code. ### What changed from the old v8 plan | Old (v8) | New | |----------|-----| | 66 languages | **159 languages** (full `CBM_LANG_*` registry) | | 5 language *groups* share one 12-question set | **5 bespoke questions per language**, each its own subchapter (§12) | | Versioned result dirs `v(x)/` | **One** `eval-results/` tree, no versions | | MCP answered by a budget-capped sub-agent | **Hybrid**: main session orchestrates, graph-only sub-agents answer (§4) | | Manual grading | **LLM-as-a-Judge**, blinded (§9) | | No deep capability tests | **Deep-dive block** for the 9 LSP-hybrid languages: cross-repo + semantic/similar (§11) | --- ## 2. Scope: the 159 languages The supported set is the `CBMLanguage` enum in `internal/cbm/cbm.h` (`CBM_LANG_GO=0` … `CBM_LANG_CFML`, before `CBM_LANG_COUNT`). The canonical short name used throughout this plan is the lowercased enum suffix (`CBM_LANG_GO` → `go`, `CBM_LANG_CSHARP` → `csharp`, `CBM_LANG_COMMONLISP` → `commonlisp`). The complete repo assignment for all 159 is the master table in §8. ### The 9 LSP-hybrid languages (deep-dive cohort) These have dedicated hybrid LSP modules under `internal/cbm/lsp/` and therefore type-aware call/usage resolution, plus they are the only languages where cross-repo and semantic/similar edges are mature enough to deserve a deep-dive: `go` · `python` · `typescript` · `java` · `c` · `csharp` · `php` · `kotlin` · `rust` --- ## 3. Question design: 5 bespoke questions per language, mapped to 5 universal dimensions Each language gets **5 questions written specifically for its repository** (real symbol names, real files). To keep 159 bespoke languages comparable and aggregatable, every question is tagged with one of **five universal capability dimensions**. Question *N* of every language always targets dimension *DN*: | Dim | Name | What it tests | Primary graph tool | |-----|------|---------------|--------------------| | **D1** | Definition / API discovery | Find the public symbols, handlers, exported defs | `search_graph(label=…, min_degree=…)` | | **D2** | Relationship / call graph | Callers, callees, impl/inheritance, imports, references | `trace_call_path(direction="both")` | | **D3** | Targeted retrieval | Exact source of one named symbol | `get_code_snippet(qualified_name=…)` | | **D4** | Architecture / structure | Layering, module/dir map, entry/init points | `get_architecture(aspects=["all"])` | | **D5** | Cross-cutting / semantic | Domain-pattern search, similarity, config↔code, vocabulary-bridged search | `search_code` / `search_graph(semantic_query=…)` | This is the backbone that reconciles "fully bespoke" with "comparable": the *wording and targets* are bespoke; the *dimension* is fixed. Aggregation (§10) rolls up by dimension across all 159 languages, and by language group, without losing per-language specificity. **The dimensions themselves are not invented — they are anchored to the published software-comprehension literature and, for the major languages, cross-checked against external repo-level QA benchmarks (§3.1).** > For config / markup / schema languages (Group E) the dimensions are interpreted structurally: > D1 = top-level definitions, D2 = cross-file references/includes, D3 = retrieve the largest > definition, D4 = file/dir organization, D5 = duplication / naming-pattern / config↔code links. > > **[CR-7] D5 is NOT aggregated across groups.** Because D5's operational meaning differs by group > (vector semantic search for code languages vs. duplication/config↔code for Group E), a single > all-159 D5 rollup would be noise. D5 is reported **within each group only**. D1–D4 remain > cross-group comparable. Likewise, on languages where a dimension barely applies (e.g. a call graph > on `csv`/`gitignore`), the chapter says so and that dimension is marked **N/A** rather than forcing > an unnatural question — N/A questions are excluded from that language's mean, not scored 0. ### 3.1 External validity — questions grounded in the literature, not LLM-invented A peer reviewer raised the central validity threat for any benchmark whose questions are model-generated: **what claim do they have to being *typical* developer questions?** We answer this on two levels. **(a) D1–D5 are anchored to a citable taxonomy of real developer questions.** They map onto the canonical catalogue of questions programmers actually ask during software-evolution tasks — **Sillito, Murphy & De Volder, "Questions Programmers Ask During Software Evolution Tasks" (FSE 2006 / IEEE TSE 2008)**: 44 question types in 4 groups. This is supplemented by **LaToza & Myers, "Hard-to-answer Questions about Code" (2010)** and **Ko et al., "Information Needs in Collocated Software Development Teams" (ICSE 2007)**, which emphasise exactly the cross-file/relational questions the graph is built to answer. Mapping: | Sillito group | Representative Sillito questions | Our dimension | |---|---|---| | 1 — Finding initial focus points | "Which type/function represents X?", "Where is this defined?" | **D1** Definition/API discovery | | 2 — Building on focus points | "What calls this? What does it call? What implements this interface?" | **D2** Relationship/call graph | | (reading a focus point) | "What does this code actually do?" (the definition itself) | **D3** Targeted retrieval | | 3 — Understanding a subgraph | "How are these objects/layers related? How does control reach here?" | **D4** Architecture/structure | | 4 — Questions over groups of subgraphs | "Where are the cross-cutting concerns / similar code / config↔code links?" | **D5** Cross-cutting/semantic | So each question's *phrasing and target* are bespoke per language, but its *type* belongs to an externally-validated set — not a taxonomy we made up. This is the defensible external-validity story for reviewers. **(b) For the major languages we reuse / compare against published repo-level QA benchmarks** rather than relying solely on our own authored questions: | Benchmark | Scale / langs | Why relevant | Use here | |---|---|---|---| | **SWE-QA** (2025, [arXiv 2509.14635](https://arxiv.org/abs/2509.14635)) | 576 repo-level Q-A pairs from 11 repos; categories incl. intention understanding, **cross-file reasoning, multi-hop dependency** | Closest published set to *our* setting (repo-level, relational) | **Primary external set** — near drop-in for the major languages; report Graph vs Explorer against it | | CoReQA (2025, [arXiv 2501.03447](https://arxiv.org/abs/2501.03447)) | repo-level QA from issues/comments, 176 repos, 4 langs | Independent repo-level QA | Cross-check | | CodeRepoQA (2024, [arXiv 2412.14764](https://arxiv.org/abs/2412.14764)) | 585k entries from issue conversations, 5 langs | Large dev-knowledge QA | Reference / sampling | | RepoQA (2024, [arXiv 2406.06025](https://arxiv.org/abs/2406.06025)) | "needle function" long-context retrieval, 50 repos, 5 langs | Retrieval reference | D3-style retrieval baseline | | CodeQA / CS1QA / CoSQA | snippet-level | Criticised by newer work as too fine-grained | Cited as the *contrast* that motivates structural retrieval | **Adopted policy (the reviewer's proposal):** 1. **Anchor D1–D5 to Sillito** (the mapping above) for defensible external validity — stated in every chapter's framing, not just here. 2. **For the major languages** (the 9 LSP-hybrid + the most popular Group A–D languages), **reuse and compare against SWE-QA** where the repo overlaps or a close analogue exists, so part of the score rests on an independent, peer-published question set — not only our authored questions. 3. **Generate the multilingual remainder ourselves, but from independent ground truth (LSP symbol / reference data and git history) — never from the model.** Question *targets* come from LSP/git facts, keeping authoring independent of the system under test. This reinforces the §12 symmetric-authoring rule and [CR-1]. > This makes the benchmark's question provenance auditable: each question is (i) a Sillito-typed > developer question, and (ii) for major languages, corroborated by or drawn from SWE-QA; (iii) where > self-authored, seeded from LSP/git ground truth rather than the model under test. --- ## 4. Execution model (sequential, main-channel) **One language at a time, in the main channel, never concurrently.** The CBM/graph side of the evaluation — indexing and answering the 5 questions with graph tools — runs **entirely in the main channel**. The **only** subagent used during the run is the **Explorer** baseline (it just Grep/Glob/Reads — it never indexes — so it adds no indexing load). There are **no teams, no graph subagents, and never two indexes building at once.** > **Why sequential, not parallel.** Indexing is memory-heavy (a single large repo can use many GB of > RSS; two cold indexes at once risk OOM — see the indexer memory baseline). Running languages > concurrently would make the per-language index-time and resource numbers non-comparable and > could exhaust the box. Sequential is slower but **guarantees clean, comparable, resource-safe > measurements**. That trade-off is accepted deliberately. **Per-language loop (strictly ordered):** ``` for each language L in order (skip if manifest says done): 1. CLONE/locate L's repo (§6); record resolved commit SHA 2. COLD-INDEX L in the main channel ── TIME IT (clone+index) → key metric (§5) (only one index exists at this point; previous language's index already deleted in step 8) 3. RECORD graph stats: per-node-type + per-edge-type histogram, zeros kept (§7, §12) 4. GRAPH condition: the MAIN CHANNEL answers L's 5 questions using ONLY graph tools → write eval-results/L-graph.md (with BENCH markers + metrics) 5. EXPLORER condition: spawn ONE Explore subagent (Grep/Glob/Read only) on L's repo, answering the same 5 questions → it writes eval-results/L-explorer.md, then exits (this is the only subagent; it runs while/after step 4 — no indexing, no contention) 6. WRITE L's per-language report incl. index time + full histogram (§10.4) 7. JUDGE L (blind, §9) — see note below 8. DELETE L's index from ~/.cache/codebase-memory-mcp so the next language starts clean, then mark L done in manifest.json and move to L+1 ``` - **No graph subagent.** The graph answers are produced in the main channel. This is the literal "use the product the way a developer does" condition (the main session driving graph tools per the Code-Discovery priority order), and it keeps indexing single-threaded. - **Explorer subagent** (`subagent_type="Explore"`, Grep/Glob/Read only, no `team_name`, no `SendMessage`) writes its file as its last action and exits. One at a time, never overlapping the next language. - **Index lifecycle = cold every time.** Deleting L's index in step 8 means L+1 is always a cold first-run index — no cache reuse, honest first-run cost. Only one project DB exists at any moment. - **[CR-4] Atomic completion** for the Explorer file: write + `sync`, then a sentinel `eval-results/L.explorer.done`; the main channel waits on the sentinel, never a half-written file. - **[CR-8] Checkpoint / resume.** `eval-results/manifest.json` records per language `{done, index_ms, sha, files, started, finished}`. The loop **skips languages already `done`**, so a dropped/context-exhausted session resumes mid-sweep instead of restarting at language 1. > **Judging (§9) and the no-subagent rule.** The resource concern is about *concurrent indexing*; the > judge never indexes, so it poses no contention. To preserve blinding (the main channel knows which > answer it just authored), judging is a **separate downstream phase run after answers are collected > and the index is already deleted** — i.e. it never overlaps indexing. The judge reads the > anonymized A/B files (§9.1) with the mapping withheld. This is the one deliberate exception to > "main channel only"; if you would rather the main channel also judge (accepting non-blind grading), > say so and §9 changes accordingly. > **The cross-repo deep-dive (§11.1) is the one case two indexes coexist** — caller + callee for an > LSP pair. It is still done **sequentially in the main channel**, one LSP language at a time, and > both indexes are deleted before the next deep-dive. Two small service indexes is well within > resource limits; 159 concurrent is not. --- ## 5. Metrics captured per language, per condition > **Headline per-language metric: Index creation time.** For every language the report **always shows > the time to build the graph** = repo clone time + **cold index time** (ms/s), measured in the main > channel in step 2 of the §4 loop on a clean cache (previous index already deleted). This is a > first-class, always-displayed number — it is the cost a developer pays before they can ask a single > question, and it is the metric most sensitive to repo size and grammar/LSP performance. It is > recorded in `manifest.json` (`index_ms`), in each chapter's stats block, and in `SUMMARY.md`. | Metric | How | |--------|-----| | **Index time (clone + cold index)** | **KEY METRIC.** Wall-clock to clone + cold-index the repo in the main channel (§4 step 2), always shown per language | | **Quality** | LLM-judge score 0.0–1.0 per question (§9), averaged over the 5 (N/A dims excluded) | | **Tokens (narrow)** | Input+output tokens during the answering phase only, between `` (before Q1) and `` (after Q5) markers | | **Tokens (full-session)** | **[CR-3]** Total tokens the agent consumed end-to-end (incl. orientation: directory listings, initial probes, dead-end reads, formatting). This is the cost a real adopter pays. | | **Wall-clock** | Delta between the two markers | | **Tool calls** | Total tool invocations during answering (Graph: per MCP tool; Explorer: Grep/Glob/Read counts) | | **Zero-result rate** | Questions where the primary tool returned nothing | **[CR-3] Two token ratios are reported, clearly labeled:** `Token Ratio (answering)` and `Token Ratio (full-session)`. The full-session ratio is the headline honesty metric — the narrow one isolates per-question efficiency. The narrow metric excludes only agent spawn/teardown and judging; it does **not** hide the Explorer's orientation cost (that lives in the full-session metric). **[CR — asymmetry disclosure]** The two conditions are *not* perfectly symmetric: the Graph agent has a tool playbook (§12 hints) while the Explorer must orient itself. This is a real product advantage, not a measurement artifact — but the report states it plainly so readers don't mistake the comparison for "identical difficulty." **[CR — zero-result handling]** A zero-result answer is graded by the judge as **FAIL (0.0)** for that question (there is no claim to verify and the developer got nothing usable) — it is *not* dropped. N/A dimensions (§3) are different: those are excluded from the mean entirely. Derived per language: `Token Ratio = Explorer tokens / Graph tokens`, `Speed Ratio = Explorer time / Graph time`, `Quality Δ = Graph score − Explorer score`. --- ## 6. Phase 0 — Repository setup ```bash scripts/clone-bench-repos.sh /tmp/bench ``` Repos are cloned shallow (`--depth 1`). Shared repos use symlinks (§8 marks them). **Selection criteria** for the repo assigned to each language (§8): 1. **Real, popular OSS** — never local samples or private repos (≥1k stars where the ecosystem allows; for niche languages, the canonical project for that language). 2. **Substantial, idiomatic content in the target language** — the repo's primary or a large secondary language must be the one under test (verified by file count / LOC). 3. **Permissive enough to clone** (public; license noted is not required but availability is). 4. **Bounded size** — prefer repos that index in minutes, not hours; for giant projects (linux, llvm, nixpkgs, chromium) a documented **subdirectory** is used instead of the whole tree. 5. **Stable** — pinned by shallow clone at run time; the run records the resolved commit SHA. Rows flagged **⚠️** in §8 are lower-confidence picks (niche language, sourcehut-only mirror, or size/subset concerns) that should be validated before the first full run. --- ## 7. Phase 1 — Cold indexing (sequential, interleaved with eval) Indexing is **not** a bulk upfront pass. Per §4, each language is indexed **just-in-time**, evaluated, then its index is **deleted** before the next language — so **only one project DB ever exists** and every index is genuinely cold. The skeleton: ```bash # Start clean: no leftover DBs, fresh results tree rm -f ~/.cache/codebase-memory-mcp/*.db mkdir -p /tmp/eval-results for lang in $ALL_LANGS; do # ALL_LANGS = full 159-name list (§2 / §8) manifest_done "$lang" && continue # [CR-8] skip already-completed languages # --- step 2: cold index in the main channel, TIMED (key metric) --- t0=$(now_ms) scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/eval-results index_ms=$(( $(now_ms) - t0 )) # clone+index wall-clock → manifest + report (§5) # --- step 3: record per-type histograms (zeros back-filled) --- # node-types.json, edge-types.json (every label + all 32 edge types, zeros kept, §7 below) # --- steps 4-7: GRAPH answers (main channel) + EXPLORER answers (one Explore subagent) # + per-language report + (deferred, blind) judge --- # --- step 8: delete THIS language's index so the next is cold, then mark done --- rm -f ~/.cache/codebase-memory-mcp/*.db manifest_mark_done "$lang" "$index_ms" done ``` **Index time is captured and always reported** (`index_ms` = clone + cold index), per §5. **Index mode:** the 9 LSP-hybrid languages are indexed in **`full`** mode (all files **+ similarity/semantic edges**) so the semantic/similar deep-dive (§11) has data; `fast` mode skips similarity/semantic and must not be used for the deep-dive cohort. Cross-repo edges are built in a separate `cross-repo-intelligence` pass (§11) — the one place two indexes briefly coexist (§4). Per-language metrics recorded: file count, LOC, cold index time (ms), project name, resolved commit SHA, and — **not just totals** — a **per-type breakdown**: - `node-types.json` — a histogram of **node count by label** (`Function`, `Method`, `Class`, `Interface`, `Type`/`Enum`/`Struct`, `Field`, `Variable`, `Route`, `Module`, `Section`, `Macro`, `File`, `Folder`) + total. - `edge-types.json` — a histogram of **edge count by type, with one entry for every one of the 32 edge types, including those that came back `0`**. The canonical set is the indexer's own `ALL_EDGE_TYPES[]` (26 intra-repo types — `tests/test_lang_contract.c`): `CALLS`, `ASYNC_CALLS`, `HTTP_CALLS`, `GRPC_CALLS`, `GRAPHQL_CALLS`, `TRPC_CALLS`, `DEFINES`, `DEFINES_METHOD`, `IMPLEMENTS`, `INHERITS`, `OVERRIDE`, `DECORATES`, `IMPORTS`, `HANDLES`, `CONFIGURES`, `DEPENDS_ON`, `USAGE`, `DATA_FLOWS`, `SEMANTICALLY_RELATED`, `SIMILAR_TO`, `TESTS`, `TESTS_FILE`, `INFRA_MAPS`, `FILE_CHANGES_WITH`, `CONTAINS_FILE`, `CONTAINS_FOLDER` — **plus the 6 cross-repo types** from the cross-repo pass (`CROSS_HTTP_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_GRPC_CALLS`, `CROSS_GRAPHQL_CALLS`, `CROSS_TRPC_CALLS`, `CROSS_CHANNEL`) and a total. The writer **emits the full 32-type list and back-fills missing types with `0`** rather than recording only the types that appeared. These come straight from `query_graph` (`MATCH (n) RETURN labels(n), count(*)` and `MATCH ()-[r]->() RETURN type(r), count(*)`) / `get_graph_schema` / `cbm_store_count_edges_by_type`, then **reconciled against the canonical 32-edge / node-label lists so every type is present**. A `query_graph` aggregation only returns types that exist, so a present-but-zero row is impossible to get from the query alone — the metrics writer must add the zeros. **Zero-count types are the point**: `CALLS=0` is a `CALLS_MISSING` finding (§10.2), `IMPLEMENTS`/`INHERITS=0` means no OO-relation edges formed, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language is a semantic-index gap — each only visible because the row is kept at `0`. --- ## 8. Master repository assignment (all 159) Groups: **A** = Class-based OOP & Contracts · **B** = Systems & Low-level · **C** = Dynamic & Scripting · **D** = Functional & Formal · **E** = Config/Data/Markup/Schema/Build/Template/HDL/ Shader/Docs (sub-tag in *Notes*). **[LSP]** marks the 9 deep-dive languages. **⚠️** = validate before run. | # | Language | Grp | Repository | Notes | |---|----------|-----|------------|-------| | 1 | go | B | go-chi/chi | **[LSP]** standard chapter; deep-dive uses OTel pair (§11) | | 2 | python | C | httpie/cli | **[LSP]** | | 3 | javascript | C | expressjs/express | HTML symlinks here | | 4 | typescript | C | trpc/trpc | **[LSP]** | | 5 | tsx | C | shadcn-ui/ui | CSS symlinks here | | 6 | rust | B | meilisearch/meilisearch | **[LSP]** TOML symlinks here | | 7 | java | A | spring-projects/spring-petclinic | **[LSP]** SQL/XML symlink here | | 8 | cpp | B | nlohmann/json | CUDA shares concepts | | 9 | csharp | A | ardalis/CleanArchitecture | **[LSP]** | | 10 | php | A | laravel/framework | **[LSP]** (koel/koel alt) | | 11 | lua | C | awesomeWM/awesome | — | | 12 | scala | A | playframework/play-samples | — | | 13 | kotlin | A | JetBrains/Exposed | **[LSP]** | | 14 | ruby | A | sinatra/sinatra | — | | 15 | c | B | redis/redis | **[LSP]** Makefile symlinks here | | 16 | bash | C | bash-it/bash-it | — | | 17 | zig | B | tigerbeetle/tigerbeetle | — | | 18 | elixir | D | phoenixframework/phoenix | — | | 19 | haskell | D | jgm/pandoc | — | | 20 | ocaml | D | ocaml/dune | — | | 21 | objc | A | AFNetworking/AFNetworking | — | | 22 | swift | A | Alamofire/Alamofire | — | | 23 | dart | A | felangel/bloc | — | | 24 | perl | C | mojolicious/mojo | — | | 25 | groovy | A | spockframework/spock | — | | 26 | erlang | D | ninenines/cowboy | — | | 27 | r | C | tidyverse/dplyr | — | | 28 | html | E:markup | ← symlink javascript | benchmarked as Group E | | 29 | css | E:markup | ← symlink tsx | benchmarked as Group E | | 30 | scss | E:markup | twbs/bootstrap | — | | 31 | yaml | E:config | kubernetes/examples | k8s/kustomize symlink here | | 32 | toml | E:config | ← symlink rust | — | | 33 | hcl | E:iac | terraform-aws-modules/terraform-aws-eks | — | | 34 | sql | E:data | dbt-labs/jaffle_shop | ⚠️ small; alt symlink java | | 35 | dockerfile | E:build | docker-library/official-images | — | | 36 | clojure | D | clojure/clojure | — | | 37 | fsharp | D | giraffe-fsharp/Giraffe | — | | 38 | julia | D | SciML/DifferentialEquations.jl | — | | 39 | vimscript | C | SpaceVim/SpaceVim | — | | 40 | nix | C | nix-community/home-manager | (nixpkgs too large) | | 41 | commonlisp | D | lem-project/lem | — | | 42 | elm | D | elm/compiler | — | | 43 | fortran | B | fortran-lang/stdlib | (cp2k too large) | | 44 | cuda | B | NVIDIA/cuda-samples | — | | 45 | cobol | B | OCamlPro/gnucobol | ⚠️ test corpus | | 46 | verilog | E:hdl | YosysHQ/picorv32 | (yosys too large) | | 47 | emacslisp | C | magit/magit | — | | 48 | json | E:data | SchemaStore/schemastore | — | | 49 | xml | E:markup | ← symlink java | — | | 50 | markdown | E:docs | github/docs | — | | 51 | makefile | E:build | ← symlink c | — | | 52 | cmake | E:build | kitware/CMake | — | | 53 | protobuf | E:schema | googleapis/googleapis | — | | 54 | graphql | E:schema | graphql/graphql-js | (has .graphql) | | 55 | vue | E:markup | vuejs/core | — | | 56 | svelte | E:markup | sveltejs/svelte | — | | 57 | meson | E:build | mesonbuild/meson | — | | 58 | glsl | E:shader | repalash/Open-Shaders | — | | 59 | ini | E:config | ← symlink python | .cfg/.ini | | 60 | matlab | B | chebfun/chebfun | ⚠️ many .m | | 61 | lean | D | leanprover-community/mathlib4 | (subset) | | 62 | form | D | vermaseren/form | ⚠️ .frm examples | | 63 | magma | D | defeo/ss-isogeny-software | ⚠️ rare; validate | | 64 | wolfram | D | WolframResearch/WolframLanguageForJupyter | — | | 65 | solidity | A | OpenZeppelin/openzeppelin-contracts | — | | 66 | typst | E:docs | typst/packages | — | | 67 | gdscript | C | godotengine/godot-demo-projects | — | | 68 | gleam | D | gleam-lang/stdlib | — | | 69 | powershell | C | PowerShell/PowerShell | (modules subset) | | 70 | pascal | B | castle-engine/castle-engine | — | | 71 | dlang | B | dlang/phobos | — | | 72 | nim | B | nim-lang/Nim | — | | 73 | scheme | D | gambit/gambit | ⚠️ verify .scm coverage | | 74 | fennel | C | bakpakin/Fennel | — | | 75 | fish | C | fish-shell/fish-shell | (.fish functions) | | 76 | awk | C | e36freak/awk-libs | ⚠️ niche | | 77 | zsh | C | ohmyzsh/ohmyzsh | — | | 78 | tcl | C | tcltk/tcllib | — | | 79 | ada | B | AdaCore/Ada_Drivers_Library | — | | 80 | agda | D | agda/agda-stdlib | — | | 81 | racket | D | racket/racket | (subset) | | 82 | odin | B | odin-lang/Odin | (core lib) | | 83 | rescript | D | rescript-lang/rescript-core | — | | 84 | purescript | D | purescript-halogen/purescript-halogen | — | | 85 | nickel | D | tweag/nickel | (.ncl stdlib) | | 86 | crystal | A | crystal-lang/crystal | (stdlib) | | 87 | teal | C | teal-language/tl | — | | 88 | hare | B | harelang/hare | ⚠️ sourcehut mirror | | 89 | pony | A | ponylang/ponyc | (packages) | | 90 | luau | C | luau-lang/luau | ⚠️ tests/*.luau | | 91 | janet | C | janet-lang/spork | — | | 92 | sway | A | FuelLabs/sway-applications | — | | 93 | nasm | B | cirosantilli/x86-bare-metal-examples | — | | 94 | assembly | B | pret/pokered | (.asm disassembly) | | 95 | astro | E:markup | withastro/astro | (examples/docs) | | 96 | blade | E:template | monicahq/monica | (.blade.php) | | 97 | just | E:build | casey/just | (justfiles) | | 98 | gotemplate | E:template | prometheus-community/helm-charts | (.tpl) | | 99 | templ | E:template | a-h/templ | (examples) | | 100 | liquid | E:template | Shopify/dawn | (.liquid theme) | | 101 | jinja2 | E:template | sovereign/sovereign | (.j2) | | 102 | prisma | E:schema | prisma/prisma-examples | (schema.prisma) | | 103 | hyprlang | E:config | end-4/dots-hyprland | ⚠️ dotfiles | | 104 | dotenv | E:config | motdotla/dotenv | ⚠️ small fixtures | | 105 | diff | E:data | void-linux/void-packages | (.patch) | | 106 | wgsl | E:shader | gfx-rs/wgpu | (.wgsl examples) | | 107 | kdl | E:config | zellij-org/zellij | (.kdl config) | | 108 | json5 | E:config | json5/json5 | (tests) | | 109 | jsonnet | E:config | grafana/jsonnet-libs | — | | 110 | ron | E:config | ron-rs/ron | ⚠️ tests/examples | | 111 | thrift | E:schema | apache/thrift | (test/*.thrift) | | 112 | capnp | E:schema | capnproto/capnproto | (*.capnp) | | 113 | properties | E:config | ← symlink java | (.properties) | | 114 | sshconfig | E:config | mathiasbynens/dotfiles | ⚠️ few files | | 115 | bibtex | E:docs | JabRef/jabref | ⚠️ test .bib | | 116 | starlark | E:build | bazelbuild/rules_go | (.bzl) | | 117 | bicep | E:iac | Azure/bicep-registry-modules | (*.bicep) | | 118 | csv | E:data | vega/vega-datasets | ⚠️ data, not code | | 119 | requirements | E:config | huggingface/transformers | (requirements*.txt) | | 120 | hlsl | E:shader | microsoft/DirectX-Graphics-Samples | (.hlsl) | | 121 | vhdl | E:hdl | VUnit/vunit | ⚠️ examples | | 122 | systemverilog | E:hdl | lowRISC/ibex | (.sv) | | 123 | devicetree | E:config | u-boot/u-boot | (.dts subset) | | 124 | linkerscript | E:config | zephyrproject-rtos/zephyr | (.ld subset) | | 125 | gn | E:build | flutter/buildroot | ⚠️ (.gn/.gni) | | 126 | kconfig | E:build | buildroot/buildroot | (Config.in/Kconfig) | | 127 | bitbake | E:build | openembedded/meta-openembedded | (.bb) | | 128 | smali | E:data | JesusFreke/smali | ⚠️ tests/*.smali | | 129 | tablegen | E:data | llvm/llvm-project | ⚠️ subset (lib/Target/X86 *.td) | | 130 | ispc | B | ispc/ispc | (examples *.ispc) | | 131 | cairo | A | OpenZeppelin/cairo-contracts | — | | 132 | move | A | econia-labs/econia | (Move) | | 133 | squirrel | C | albertodemichelis/squirrel | (samples/*.nut) | | 134 | func | A | ton-blockchain/token-contract | ⚠️ FunC | | 135 | regex | E:data | (fixture corpus) | ⚠️ no natural repo — see §8.1 | | 136 | jsdoc | E:docs | lodash/lodash | (JSDoc-heavy JS) | | 137 | rst | E:docs | sphinx-doc/sphinx | (.rst) | | 138 | beancount | E:data | beancount/beancount | ⚠️ examples | | 139 | mermaid | E:docs | mermaid-js/mermaid | (demos/*.mmd) | | 140 | puppet | E:iac | puppetlabs/puppetlabs-apache | (.pp) | | 141 | po | E:docs | django/django | (locale/*.po) | | 142 | gitattributes | E:config | alexkaratarakis/gitattributes | (collection) | | 143 | gitignore | E:config | github/gitignore | (collection) | | 144 | slang | E:shader | shader-slang/slang | (tests/examples) | | 145 | llvm_ir | B | llvm/llvm-project | ⚠️ subset (.ll tests) | | 146 | smithy | E:schema | smithy-lang/smithy | (examples) | | 147 | wit | E:schema | bytecodealliance/wit-bindgen | (.wit) | | 148 | tlaplus | D | tlaplus/Examples | (.tla) | | 149 | pkl | E:config | apple/pkl-pantry | (.pkl) | | 150 | gomod | E:config | ← symlink go | (go.mod) | | 151 | apex | A | trailheadapps/ebikes-lwc | (Apex .cls) | | 152 | soql | E:data | ← symlink apex | (embedded SOQL) | | 153 | sosl | E:data | ← symlink apex | (embedded SOSL) | | 154 | kustomize | E:iac | kubernetes-sigs/kustomize | (examples) | | 155 | k8s | E:iac | ← symlink yaml | (apiVersion manifests) | | 156 | pine | C | pinecoders/pine-utils | ⚠️ niche | | 157 | qml | E:markup | lirios/lirios | ⚠️ (.qml) | | 158 | cfscript | A | ortus-solutions/coldbox-platform | (.cfc) | | 159 | cfml | E:template | ← symlink cfscript | (.cfm) | ### 8.1 Languages without a natural OSS repo `regex` indexes regular-expression grammar fragments and has no idiomatic standalone project. It is evaluated against a small curated **fixture corpus** (a directory of representative `.regex`/pattern files) committed under `tests/eval-fixtures/regex/`, treated as its "repo". Any other ⚠️ row that cannot be validated falls back to the same fixture-corpus approach and is noted as a coverage caveat in the final report (never silently dropped). --- ## 9. LLM-as-a-Judge Grading is done by an **LLM judge agent**, not by hand. This scales to 159 × 5 = 795 questions and removes human grader drift, at the cost of needing careful bias controls. ### 9.1 What the judge sees Per question, the judge agent receives: 1. The question text and its dimension (D1–D5). 2. **Two anonymized answers**, labeled `Answer A` and `Answer B` in **randomized order** — the judge is **not told which is Graph vs Explorer** (blind). The order is recorded so scores can be de-anonymized after grading. 3. **Ground-truth access**: the judge has read-only `Grep`/`Read`/`get_code_snippet` on the actual repo to *verify claims*, so it grades against reality, not plausibility. **[CR-9] Verification depth scales with the answer**: the judge checks **≥30% of the distinct symbols/paths the answer cites (minimum 5)**, not a flat 3–5 — so a long answer isn't rubber-stamped on a tiny sample. The **Completeness** sub-score must cite the *enumeration it compared against* (e.g. "answer listed 3 of the 12 handlers found in `handlers/`"), so Completeness is grounded in a count, not intuition. ### 9.2 Rubric — three sub-scores per answer, each 0.0–1.0 | Sub-score | Question | |-----------|----------| | **Correctness** | Do the named symbols / paths / line numbers actually exist and match? (verify a sample) | | **Completeness** | Does it cover the full scope the question asked, or only a fragment? | | **Specificity** | Concrete names/paths/lines vs vague prose? | Per-answer score = mean of the three. The judge must **cite the evidence it checked** for each sub-score (e.g. "verified `Cart.checkout` exists at `cart.go:88` ✓; claimed 12 handlers, found 11"). ### 9.3 Grade bands & per-question score | Grade | Score | Meaning | |-------|-------|---------| | PASS (P) | ≥ 0.80 | correct, complete, specific | | PARTIAL (/) | 0.40–0.79 | useful but incomplete or partly wrong | | FAIL (F) | < 0.40 | no useful or fundamentally wrong answer | ### 9.4 Bias controls - **Blind A/B labeling + randomized order** (above) so the judge can't favor "the graph one". - **Position-bias check**: a fraction of questions are judged twice with A/B **swapped**; if the preference flips, the question is flagged and re-judged. - **Verification-required**: a claim the judge cannot confirm in source caps Correctness at 0.5. - **[CR-2] Single disclosed judge model (decided).** One judge model, **named in `SUMMARY.md`**, runs all passes with a fixed prompt. Because the Graph/Explorer answers are written by a Claude-family agent, a same-family judge carries a documented **10–25% self-preference inflation**. Two things follow and are **mandatory**: 1. **Prefer a judge from a *different* family than the answer-writing agent** (e.g. answers written by Claude → judge with a non-Claude model) to minimize self-preference at no extra panel cost. 2. If the judge *is* same-family, the report must carry an explicit **self-preference caveat** and must not present the score as bias-free. (A cross-family panel was considered and deferred — see §16.3 fork B. It remains the stronger option if cross-provider access is available at execution time; the plan does not block on it.) - **3 independent passes**, per-question score = **median**; disagreement spread > 0.4 flags the question for manual spot-check. *Caveat recorded in the report: 3 passes of one model measure the model's consistency, not its bias — they tighten variance, they do not remove self-preference.* - **Fixed prompt** across all languages for comparability. - The judge never sees token/time/tool-call metrics — those are scored mechanically (§5), not by the LLM. ### 9.5 Judge output (`eval-results/-judged.json`) ```json { "language": "go", "repo": "...", "commit": "...", "questions": [ {"dim": "D1", "graph": {"correctness": 1.0, "completeness": 0.8, "specificity": 1.0, "score": 0.93, "grade": "P", "evidence": "verified 11/12 handlers exist ..."}, "explorer": {"correctness": 0.8, "completeness": 1.0, "specificity": 0.6, "score": 0.80, "grade": "P", "evidence": "..."}, "judge_spread": 0.10} ], "graph_score": 0.0, "explorer_score": 0.0 } ``` --- ## 10. Aggregation & final report One run → one report tree under `eval-results/` (no versions). Each `-judged.json` plus the mechanical metrics (§5) roll up into: ### 10.1 `eval-results/SUMMARY.md` (shareable, factual, no internal TODOs) - **Overview**: avg Graph vs Explorer quality, avg tokens, avg time, ratios. - **Quality by language** (159 rows): group, repo, nodes, edges, Graph score, Explorer score, tier. - **Graph composition / edge-type coverage matrix** (159 rows × every node label + every edge type): the per-type counts from §7, **with explicit `0`s kept**. This matrix is where systemic extraction gaps surface at a glance — e.g. a column of `CALLS = 0` across a family of grammars, or `IMPLEMENTS = 0` for languages that clearly have interfaces. A heat-map of zeros across languages is one of the most actionable artifacts for the dev team and feeds §10.2 directly. - **Token efficiency by language**: Graph vs Explorer tokens, % reduction, tool calls. - **Quality by dimension (D1–D5)**: avg Graph vs Explorer across all languages — shows *which kinds* of questions the graph wins/loses. - **Quality by group (A–E)**. - **Tier distribution** (§10.3). - **Deep-dive results** for the 9 LSP languages (§11 recall/precision). ### 10.2 `eval-results/IMPROVEMENTS.md` (internal) For every language where Graph < Explorer, or any FAIL: the failing dimension(s), a 5–15 line code sample read from the actual repo, and a root-cause tag from `{LABEL_MISMATCH, EXTRACTION_GAP, CALLS_MISSING, QUERY_STRATEGY, PARSE_ERROR}` with the lang-spec file (`internal/cbm/lang_specs.c`) and the tree-sitter node type that would fix it. Plus a priority-ordered recommendations list (severity × languages affected). ### 10.3 Language tier | Tier | Graph score | Meaning | |------|-------------|---------| | A | ≥ 0.67 | strong, competitive with Explorer | | B | 0.50–0.66 | good, identifiable gaps | | C | 0.42–0.49 | partial | | D | < 0.42 | weak; Explorer preferred | ### 10.4 Per-language report (`eval-results/.md`) The 5-question table (dim, MCP grade, Exp grade, calls, tokens), totals, ratios, tier, the **full graph-stats histogram** (`node-types.json` + `edge-types.json` from §7, rendered as the two-column table from the §12 template — every label and every edge type with its count, zeros included), and a Failure Analysis section for any FAIL. Deep-dive languages append their §11 block. --- ## 11. Deep-dive: the 9 LSP-hybrid languages Beyond the 5 standard questions, each of `go, python, typescript, java, c, csharp, php, kotlin, rust` gets a **deep-dive block** with two extra capability tests. These exist only for LSP-hybrid languages because only they have type-aware resolution and full-mode similarity/semantic edges. ### 11.1 Cross-repo intelligence **Capability under test:** `index_repository(mode="cross-repo-intelligence", target_projects=[…])` matches Routes/Channels across two indexed projects to create `CROSS_HTTP_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_CHANNEL` (and gRPC/GraphQL/tRPC) edges; `get_architecture` surfaces a `cross_repo_links` summary. **Setup:** each LSP language is given a **caller/callee repo pair** that genuinely calls across a service boundary. The richest realistic source is the **OpenTelemetry Demo** (`open-telemetry/opentelemetry-demo`), a polyglot system whose services call each other via gRPC/HTTP. Each pair = the service written in language *X* + the service it calls; the two service directories are indexed as **separate projects**, then the cross-repo pass is run. | Lang | Caller repo/service | Callee repo/service | Source | |------|---------------------|---------------------|--------| | go | otel-demo `checkout` (Go) | otel-demo `product-catalog` (Go) | OTel Demo (gRPC) | | python | otel-demo `recommendation` (Py) | otel-demo `product-catalog` (Go) | OTel Demo (gRPC) | | typescript | otel-demo `frontend` (TS/Next) | otel-demo `cart` / `checkout` | OTel Demo (gRPC/HTTP) | | java | otel-demo `ad` (Java) | otel-demo `feature-flag` / others | OTel Demo (gRPC) | | csharp | otel-demo `accounting`/`cart` (.NET) | otel-demo `checkout` (Go) | OTel Demo (gRPC) | | php | otel-demo `quote` (PHP) | otel-demo `shipping` (Rust) | OTel Demo (HTTP) | | kotlin | otel-demo `fraud-detection` (Kotlin) | otel-demo `checkout` (Go) | OTel Demo (Kafka/gRPC) | | rust | otel-demo `shipping` (Rust) | otel-demo `quote` (PHP) | OTel Demo (HTTP) | | c | ⚠️ **gap** — no C service in OTel Demo | candidate: `redis/redis` ↔ `redis/hiredis` | protocol, not HTTP routes — validate whether CROSS edges form; if not, mark as a documented capability gap | > **Honest caveat:** "one genuine cross-repo-call pair per language" is hard. OTel Demo cleanly > covers 8 of 9 (it natively exercises Go/Python/TS/Java/C#/PHP/Kotlin/Rust). **C has no service in > the demo** and Redis↔hiredis uses the RESP wire protocol (not HTTP routes), so cross-repo edges > almost certainly **will not** form — this is reported as a documented capability gap, not hidden. > **[CR-5] Gating validation (do this FIRST).** Before authoring any cross-repo question, run > `index_repository(mode="cross-repo-intelligence")` on **one** OTel pair (checkout→product-catalog) > and confirm `CROSS_HTTP_CALLS`/`CROSS_ASYNC_CALLS` edges actually form when two sub-directories of > the *same monorepo* are indexed as separate projects. This is the load-bearing assumption for the > entire deep-dive. **If edges do not form**, the fallback is to use genuinely separate repos > (independently published client + server, e.g. a service repo + its generated client SDK repo) or > to report cross-repo as "not evaluable from monorepo splits" — the plan does not proceed on an > unvalidated assumption. > **[CR — ground truth, Alt D]** The cross-repo **ground truth** (X1 actual-call set) is built from > the OTel Demo's own service-topology docs and integration tests where available, not solely from > the question author reading the caller's stubs — this avoids the "author writes both the answer key > and grades against it" circularity. **Cross-repo deep-dive questions (per pair):** - **X1 (recall):** "List every cross-service call from the caller into the callee — (caller symbol → callee route/handler)." Compare against a **ground-truth set** built by manually reading the caller's client stubs / request sites. Metric: **recall = found / actual**, **precision = correct / found**. - **X2 (architecture):** "From `get_architecture`, does `cross_repo_links` correctly summarize the caller→callee relationship (count, direction)?" Graded P/ /F by the judge against ground truth. ### 11.2 Semantic / similarity edges **Capability under test:** `full`/`moderate` index mode builds the two semantic edge types — **`SIMILAR_TO`** (near-duplicate / structural similarity, simhash-seeded) and **`SEMANTICALLY_RELATED`** (vector/embedding relation) — plus a semantic vector index; `search_graph(semantic_query=[…])` returns vocabulary-bridged cosine matches in `semantic_results`. The deep-dive checks both that the **edges exist** (their histogram counts are non-zero, §7) and that they are **correct/complete** (S1/S2 below). **Setup:** the language's standard single repo (§8), indexed in **`full`** mode. **Semantic/similarity deep-dive questions:** - **S1 (vocabulary bridging):** Pick a concept present under a *synonym* (e.g. the repo says "publish"/"emit"; query `semantic_query=["send","dispatch"]`). Did the right functions surface in `semantic_results` even though the literal token differs? Metric: **hit@k** against a ground-truth set of the real synonym-named functions. - **S2 (near-duplicate recall):** **[CR-6]** Ground truth is **not** a hand-picked 3–5 pairs (a sample that small has no statistical standing and hand-built clone sets are known to be heavily mislabelled). Instead: scope to **Type-1/2 near-exact duplicates** (where verification is tractable), build the candidate set **reproducibly from the indexer's own simhash output** over the full repo, and **independently confirm** each pair by normalized-token diff. Target a **≥20-pair** ground-truth set per language (fewer only if the repo genuinely has fewer — noted). Metric: **recall** over that confirmed set + **false-positive rate** on a sample of returned pairs. > "Did we catch *all* of them?" is the point: recall is reported **against a reproducible, > independently-confirmed ground-truth set** (simhash-seeded, token-diff-verified), so a high score > means real coverage — not cherry-picking, and not circular (the seed is not graded by the same > `SIMILAR_TO`/`SEMANTICALLY_RELATED` edges it tests; pairs are re-confirmed by token diff). ### 11.3 Deep-dive output & scoring Each deep-dive block writes `eval-results/-deepdive.json`: ```json {"language": "go", "cross_repo": {"pair": ["checkout","product-catalog"], "actual": 7, "found": 6, "recall": 0.86, "precision": 1.0, "X2_grade": "P"}, "semantic": {"S1_hit_at_5": 0.8, "S2_recall": 0.6, "S2_false_positive_rate": 0.1}} ``` Aggregated into a **Deep-Dive section** of `SUMMARY.md`: a 9-row table of cross-repo recall/precision and semantic hit@k / duplicate-recall, so regressions in these high-value features are visible run-over-run (git diff of `SUMMARY.md`). --- ## 12. Per-language chapter template Every one of the 159 languages gets its **own subchapter** in §14 with this exact structure. The 9 LSP languages additionally include the §11 deep-dive block. ```markdown ### . — [LSP?] **Repo:** (`/tmp/bench/`) **Symlink:** **Indexed in:** full | (fast for non-LSP) **Why this repo:** <1 line tying repo to §6 criteria> **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "" 2. **[D2 Relationship]** "<…>" 3. **[D3 Retrieval]** "<… name a real symbol in this repo>" 4. **[D4 Architecture]** "<…>" 5. **[D5 Cross-cutting/Semantic]** "<… domain pattern / config↔code / similarity for this repo>" **Expected graph tools (hint, not a script):** D1→search_graph(...), D2→trace_call_path(...), ... **Graph stats (filled at index time).** Report **every node label and every edge type on its own row, with its count — including a `0` where the index produced none.** Never group types and never omit a zero row: a `0` is critical information (it pinpoints exactly which construct the grammar/LSP failed to extract for this language). The canonical sets are the indexer's own `ALL_EDGE_TYPES[]` (26 intra-repo types, `tests/test_lang_contract.c`) **+** the 6 cross-repo types from the cross-repo pass = **32 edge types**, and the `get_graph_schema` node labels. _Node-type histogram:_ | Node label | Count | |---|---| | Function | _ | | Method | _ | | Class | _ | | Struct | _ | | Interface | _ | | Trait | _ | | Enum | _ | | Type | _ | | Field | _ | | Variable | _ | | Route | _ | | Module | _ | | Section | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | _Edge-type histogram (all 32 edge types listed every time, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Zero rows are mandatory, not optional. `CALLS = 0` flags broken call-extraction (a `CALLS_MISSING` > signal, §10.2); `IMPLEMENTS`/`INHERITS`/`OVERRIDE = 0` flags missing OO relations; > `SIMILAR_TO`/`SEMANTICALLY_RELATED = 0` on a full-mode LSP language flags a semantic-index gap. The > `CROSS_*` rows are `0` for non-LSP languages (no cross-repo pass) and carry real counts only for the > 9 LSP deep-dive pairs (§11.1). **Output of the analysis for this language:** - `eval-results/-graph.md` — 5 answers from the Graph condition (+ BENCH markers, metrics) - `eval-results/-explorer.md` — 5 answers from the Explorer condition - `eval-results/-judged.json` — LLM-judge scores (§9.5) - (LSP only) `eval-results/-deepdive.json` — §11.3 **How this language's result aggregates:** its 5 D-scores feed the by-dimension and by-group rollups (§10); its tier is computed from the mean Graph score; (LSP only) its deep-dive feeds the Deep-Dive section. ``` > **[CR-1] Symmetric authoring (the most important validity rule).** Questions must reference *real* > identifiers, but **how those identifiers are discovered determines fairness.** If every question > were authored by browsing the graph, the test would only ever ask about symbols the graph already > indexed successfully — invisibly hiding the graph's misses. So authoring is split by dimension: > > | Dim | Authored by | Rule | > |-----|-------------|------| > | **D1, D3** | **Grep-first author** | Find the target symbols by **text search only** (`grep`/`Read`), never via the graph. Guarantees the symbol is findable both ways, so the graph gets no head start. | > | **D2, D4** | Graph-first author | May use `trace_call_path`/`get_architecture` to pick a meaningful central symbol/layer. | > | **D5** | Either | D5 is **openly graph-favoring** (semantic/similarity/config↔code) and is labeled as such in the report — not presented as neutral. | > > A symbol used in a D1/D3 question that the graph later fails to return is a *finding*, not a bug in > the question — that's exactly the gap symmetric authoring is designed to expose. > > **Pinning.** During authoring, the repo's resolved commit SHA is recorded and baked into > `clone-bench-repos.sh`, so the run indexes the *same* HEAD the questions were written against. > > §14 contains two fully-worked exemplars now; the remaining 157 are generated against their cloned > repos (§15) following this authoring split. --- ## 13. Reproducibility ```bash # 1. Clone all 159 repos (shallow; skip existing) scripts/clone-bench-repos.sh /tmp/bench # 2. Cold index all 159 (LSP cohort in full mode) rm -f ~/.cache/codebase-memory-mcp/*.db mkdir -p /tmp/eval-results for lang in $ALL_LANGS; do scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/eval-results done # 3. Cross-repo pass for the 9 LSP pairs (index each service dir, then cross-repo-intelligence) # index_repository(mode="cross-repo-intelligence", target_projects=[callee]) — see §11.1 # 4. Graph phase — spawn graph-only sub-agents per language (writes -graph.md) # 5. Explorer phase— spawn Explore sub-agents per language (writes -explorer.md) # 6. Judge phase — spawn blind LLM-judge per language (writes -judged.json) # 7. Deep-dive — 9 LSP languages (writes -deepdive.json) # 8. Aggregate — SUMMARY.md + IMPROVEMENTS.md + per-language reports (no version dir) ``` `scripts/clone-bench-repos.sh` and `scripts/benchmark-index.sh` must be extended from 66 → 159 languages (and the symlink/subset rules in §8 added). That script change is part of executing this plan, tracked in §15. --- ## 14. Per-language chapters > Two annotated exemplars below establish the format (one LSP with deep-dive, one standard). The > **complete, QA-reviewed set of all 159 chapters is [Appendix A](#appendix-a--per-language-chapters-159)** > at the end of this document. Each chapter is a *draft specification* — its 5 questions are finalized > grep-first against the pinned commit at execution time per the §12 authoring split. ### 1. go — B (Systems) **[LSP]** **Repo:** go-chi/chi (`/tmp/bench/go`) **Symlink:** no **Indexed in:** full **Why this repo:** popular (~19k★), idiomatic Go, a router library whose public API, middleware chain, and tree-routing internals exercise D1–D5 cleanly. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "I'm auditing chi's public surface. List every exported function/method on the `Mux` and `Router` types, and the package-level constructors (`NewRouter`, `NewMux`)." 2. **[D2 Relationship]** "Trace `Mux.ServeHTTP` both ways: what does it call to route a request (down to `tree.FindRoute`), and who calls it?" 3. **[D3 Retrieval]** "Show me the full source of `(*node).findRoute` in `tree.go` — I need to understand the radix-tree walk before changing it." 4. **[D4 Architecture]** "Describe chi's layering: how do `Mux`, `Router`, `middleware`, and `tree` relate, and where is the request entry point?" 5. **[D5 Cross-cutting/Semantic]** "Find all middleware constructors (the `middleware` package's `func(http.Handler) http.Handler` wrappers) — where is cross-cutting behavior added?" **Expected graph tools:** D1→`search_graph(label="Method", qn_pattern=".*Mux.*", min_degree=1)`; D2→`trace_call_path(function="...Mux.ServeHTTP", direction="both")`; D3→`get_code_snippet(qualified_name=".../tree.(*node).findRoute")`; D4→`get_architecture(aspects=["all"])`; D5→`search_graph(label="Function", file_pattern="middleware/*.go")`. **Deep-dive (LSP):** - **Cross-repo:** pair = OTel-Demo `checkout` (Go) → `product-catalog` (Go), indexed as two projects. - *X1 recall*: enumerate gRPC calls from checkout into product-catalog (`GetProduct`, `ListProducts`, …); ground truth from checkout's client stubs. Report recall/precision. - *X2*: does `get_architecture.cross_repo_links` show checkout→product-catalog with correct count? - **Semantic/similarity** (chi, full index): - *S1*: `semantic_query=["dispatch","route"]` should surface `findRoute`/`routeHTTP` though the literal token differs — hit@5 vs ground truth. - *S2*: known near-duplicates (e.g. the per-method `Get/Post/Put/...` registration wrappers in `mux.go`) — do `SIMILAR_TO`/`SEMANTICALLY_RELATED` recover the family? recall + false-positive check. **Output:** `eval-results/go-graph.md`, `go-explorer.md`, `go-judged.json`, `go-deepdive.json`. **Aggregates into:** D1–D5 rollups, Group B, Go tier, and the Deep-Dive section. --- ### 19. haskell — D (Functional) **Repo:** jgm/pandoc (`/tmp/bench/haskell`) **Symlink:** no **Indexed in:** fast (non-LSP) **Why this repo:** large, popular (~33k★), heavily idiomatic Haskell with many readers/writers, typeclasses, and a clear module hierarchy — strong D1–D5 coverage. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the exported top-level functions of `Text.Pandoc.App` and the reader/writer entry points (`readMarkdown`, `writeHtml5`, …) that form pandoc's public API." 2. **[D2 Relationship]** "Trace `convertWithOpts` both ways: what reader/writer pipeline does it call, and who invokes it from the executable's `main`?" 3. **[D3 Retrieval]** "Show the source of the `Reader` data type / the `getReader` function in `Text.Pandoc.Readers` — I need its full definition." 4. **[D4 Architecture]** "Describe pandoc's module structure: how do `Readers`, `Writers`, `Definition` (the AST), and `App` relate from input to output?" 5. **[D5 Cross-cutting/Semantic]** "Find pandoc's typeclass instances and the AST-walking functions (`walk`, `query` over the `Pandoc`/`Block`/`Inline` types) — where is the document transformed?" **Expected graph tools:** D1→`search_graph(label="Function", file_pattern="src/Text/Pandoc/App*.hs")`; D2→`trace_call_path(function="convertWithOpts", direction="both")`; D3→`get_code_snippet(qualified_name="Text.Pandoc.Readers.getReader")`; D4→`get_architecture(aspects=["all"])`; D5→`search_code("instance ")` + `search_graph(name_pattern=".*walk.*|.*query.*")`. **Output:** `eval-results/haskell-graph.md`, `haskell-explorer.md`, `haskell-judged.json`. **Aggregates into:** D1–D5 rollups, Group D, Haskell tier. (No deep-dive — not an LSP language.) --- ## 15. Execution checklist (to run this plan) **Gate 0 — de-risk before bulk authoring (do these first):** - [ ] **[CR-5]** Validate cross-repo edges form on ONE OTel pair (checkout→product-catalog) indexed as two projects. If not, switch to genuinely-separate repos or report cross-repo as a gap. - [ ] **[Fork A]** Decide scope: **pilot** (9 LSP + ~10 representative others) vs full 159 now. - [ ] **[Fork B]** Decide judge: cross-family panel vs single disclosed model. **Build-out:** - [ ] Extend `scripts/clone-bench-repos.sh` to all 159 (symlinks + subset rules from §8); pin SHAs. - [ ] Extend `scripts/benchmark-index.sh` `ALL_LANGS` to 159; force `full` mode for the LSP cohort. - [ ] Add manifest-based **skip/resume** (§4 CR-8) and the `.done` sentinel protocol (§4 CR-4). - [ ] Validate every **⚠️** repo pick (availability, language content, size). - [ ] Build the `regex`/fixture-corpus directories (§8.1). - [ ] Generate per-language chapters (§12 template) against cloned repos using the **symmetric authoring split** (CR-1): D1/D3 grep-first, D2/D4 graph-first, D5 labeled graph-favoring. - [ ] **[Ext. validity, §3.1]** Map each chapter's 5 questions to their Sillito group; for the major languages, wire in **SWE-QA** items (and CoReQA/RepoQA where useful) as an independent question set and report Graph/Explorer against them; seed self-authored multilingual questions from LSP/git ground truth, not the model. - [ ] Confirm the C cross-repo pair forms CROSS edges or document the gap (§11.1). **Run:** - [ ] Run Phases B–E with checkpointing; produce `eval-results/SUMMARY.md` (name the judge model[s]), `IMPROVEMENTS.md`, per-language reports, and the Deep-Dive section. --- ## 16. Challenger review > Per project policy, this plan's methodology is reviewed by the `challenger` agent before it is > finalized. The full report is inserted here verbatim. Fixes adopted from it are marked > **[CR-n]** at their section; two strategic forks (§16.1) remain open decisions. ### 16.1 Challenger-driven changes adopted | # | Finding | Change | Where | |---|---------|--------|-------| | CR-1 | Questions authored from graph-discovered symbols bias D1/D3 toward graph-visible code | **Symmetric authoring**: D1 & D3 questions are authored *grep-first* (symbols found by text search, never via the graph); D2 & D4 graph-first; D5 is openly graph-favoring and labeled as such | §12 | | CR-2 | Single-family judge → 10–25% self-preference inflation | **Single disclosed judge model, preferring a non-Claude family**; bias caveated; cross-family panel kept as an execution-time upgrade (§16.3) | §9.4 | | CR-3 | Excluding Explorer orientation tokens flatters the Token Ratio | Report **both** a narrow (answering-phase) and a **full-session** token metric | §5 | | CR-4 | File-existence polling can read a half-written file | **Atomic completion**: agent writes output, fsyncs, then writes a `..done` sentinel; the orchestrator polls the sentinel only | §4 | | CR-5 | "OTel sub-dir = project" may not form CROSS edges | **Gate**: validate CROSS edges on one OTel pair *before* authoring the rest of the deep-dive | §11.1 | | CR-6 | 3–5 hand-picked duplicate pairs have no statistical standing | **S2 ground truth from the indexer's own simhash clusters** (reproducible), scoped to Type-1/2 near-exact dupes; minimum 20-pair set | §11.2 | | CR-7 | D5 means different things per group → cross-group rollup is noise | **D5 is reported within-group only**, never aggregated across all 159 | §3, §10 | | CR-8 | No checkpoint/resume; one dropout restarts from zero | **Completion manifest** (`eval-results/manifest.json`); every phase skips already-`.done` languages | §4, §13 | | CR-9 | Verification samples 3–5 of N claims → Completeness ungrounded | Judge verification sample **scales with claim count** (≥30% of cited symbols, min 5); Completeness must cite the enumeration it compared against | §9.1 | ### 16.2 Full challenger report (verbatim) > The report below is reproduced in full, unedited, as required by project policy. #### Challenge Review: EVALUATION_PLAN.md — codebase-memory-mcp 159-Language Benchmark ##### What Looks Good The three-pass median judge with disagreement-spread flagging (§9.4) is a sound baseline for LLM-as-a-judge consistency. Documenting the C cross-repo gap explicitly rather than papering over it (§11.1 caveat) is intellectually honest. Tying judging to ground-truth verification with capped Correctness at 0.5 for unverifiable claims (§9.4) is the right instinct, even if the implementation has holes (see below). The per-language tier system (§10.3) gives actionable output rather than a single aggregate number. ##### Assumptions to Verify **Assumption 1: the "bespoke questions written against real symbols" guarantee neutrality.** The plan states questions are written after Phase 0/1 once `search_graph`/`get_architecture` confirm actual identifiers (§12, authoring note). The person writing the questions therefore already used the graph to discover the symbols. This systematically biases D1, D2, and D3 questions toward symbols the graph indexed successfully. A function the graph missed — because of a parse error, label mismatch, or LSP gap — cannot appear in the question, because the author would never have discovered it that way. The Explorer is then asked to find things the graph already confirmed it can find. Consequence: quality gaps for underserved symbols are invisible in the data. **Assumption 2: "identical overhead cancels out" justifies excluding spawn/teardown tokens (§5).** This only holds if overhead is truly equal. The Explorer is an `Explore` sub-agent (open-ended, unlimited tool calls, no instructions about how to be efficient). The Graph agent is a graph-only agent with a narrow, well-specified tool set. The Explorer will consume more context just orienting itself — directory listings, initial Glob probes, dead-end reads. Excluding this overhead hides a real cost that a developer deciding "should I use the MCP?" would pay. The token ratio reported will favor Graph artificially, independently of quality. **Assumption 3: file-existence polling is reliable at scale (§4).** The plan detects agent completion by polling for `-graph.md`/`-explorer.md` existence. A file can be partially written when the polling check fires (write syscall in progress). There is no described fsync barrier, atomic rename, or completion-marker protocol (e.g., writing a `.done` sentinel after a final `fsync`). At 159 × 3 agents per batch, a race here corrupts the input to the judge phase silently. The plan notes the v8 lesson about `SendMessage` killing agents mid-write — this is the exact same failure mode, shifted from shutdown to detection. **Assumption 4: "one OTel Demo sub-directory = one project" validly exercises cross-repo intelligence.** This is the most critical structural assumption. The cross-repo feature (`CROSS_HTTP_CALLS`, `CROSS_ASYNC_CALLS`) is designed for separately maintained codebases with independent deploy cycles. Indexing two service directories from the same monorepo as "separate projects" is an artificial split. The monorepo may share proto definitions, a common `pb.go` directory, or Bazel build rules that the indexer would normally traverse. Whether the CROSS edges even form in this configuration has not been validated — the plan asks for it (§15) but doesn't answer it. If they don't form, 8 of 9 deep-dives measure nothing meaningful. **Assumption 5: the 3-pass judge is immune to self-preference bias.** The plan specifies "one judge model, fixed prompt" (§9.4) but does not name the judge model, and the two conditions' answers were presumably written by Claude sub-agents. If the judge is from the Claude family, self-preference bias of 10–25% has been empirically demonstrated across Llama/Claude/GPT pairs (Zheng et al., 2024; Panickssery et al., 2024). Three passes of the same biased model gives a precise median of a biased number, not an unbiased estimate. ##### Alternatives Worth Considering **Alternative A: Tiered evaluation — deep on 9 LSP + stratified sample of Group E (not all 159).** 795 hand-written questions across 159 languages is an enormous authoring investment. Most of the actionable signal lives in: (a) the 9 LSP languages where the graph has type-aware edges, (b) the ~20 popular languages where regressions would be noticed by users, and (c) a representative handful of Group E config/data languages. The other ~110 niche languages (fennel, squirrel, smali, beancount, pine, hyprlang…) will score similarly to each other and to a naive Grep baseline because the graph cannot index them structurally beyond token counting. A stratified sample of 30–40 languages would likely yield the same tier distribution with 1/4 the cost and would be rerunnable on demand. Trade-off: loses the "covers all 159" marketing claim. But a poorly-executed 159-language sweep is weaker evidence than a rigorous 40-language one. **Alternative B: Symmetric question authoring — half written from grep discovery, half from graph discovery.** To remove the symbol-discovery bias identified in Assumption 1, split the 5 questions: questions for D1 and D3 (concrete retrieval) are written by a Grep-first author who finds symbols by text search without touching the graph, and questions for D2 and D4 are written by a graph author. D5 is intentionally graph-favoring (semantic/similarity) — acknowledge that openly rather than presenting it as neutral. Trade-off: doubles authoring complexity per language but removes the most serious validity threat. **Alternative C: Use a heterogeneous judge panel (different model families).** Run three passes with three different judge models (e.g., GPT-4o, Claude, Gemini) rather than three passes with the same model. Per-question score is the cross-family median. Self-preference bias cancels rather than compounds. This is a well-established pattern in adversarial evaluation (MT-Bench, Chatbot Arena). Trade-off: 3× judge API cost, requires cross-provider API access. The cost is real but the bias problem is also real — a single-model judge's 3-pass median is not a substitute. **Alternative D: Use the OTel Demo's own integration-test fixtures as cross-repo ground truth.** The OTel Demo ships its own integration tests and service topology documentation, which manually enumerate which service calls which endpoint. This is a pre-existing, publicly defensible ground-truth source that avoids the "author reads the code and writes the ground truth" circularity problem in §11.1's X1 metric. Trade-off: ground truth scope is bounded by what the integration tests assert, which may miss some call paths. **Alternative E: Use BigCloneBench or a validated corpus for S2 near-duplicate ground truth.** The plan proposes "3–5 known near-duplicate / copy-pasted function pairs found by manual read or simhash dump" as the S2 ground truth. This is a sample of 3–5 pairs per language, constructed by the same team running the evaluation. The academic literature has demonstrated that hand-built near-duplicate ground truth is systematically biased and mislabelled even at scale (Krinke 2022, arXiv 2505.04311 — 93% mislabelling rate for weak Type-3/4 clones in BigCloneBench). A 3-pair sample has no statistical standing whatsoever. Better alternatives: use simhash-computed clusters on the full repo as pseudo-ground-truth (at least reproducible), or scope S2 to only exact/near-exact duplicates (Type-1/2) where manual verification is tractable. ##### Risk Register | Risk | Likelihood | Impact | Mitigation | |------|-----------|--------|------------| | Questions authored from graph discovery bias D1/D2/D3 toward graph-visible symbols | High | High | Symmetric authoring (Alt B) or explicit caveat in methodology | | Single-family judge produces self-preference inflation of 10–25% | High | High | Cross-family judge panel (Alt C) | | File-existence polling causes silent partial-write corruption at scale | Medium | High | Atomic sentinel file (`.done`) written only after the output file is flushed | | OTel monorepo sub-dirs do not form CROSS edges; 8/9 deep-dives are void | Medium | High | Validate a single pair before committing to this design; document fallback | | C cross-repo pair (redis/hiredis, RESP protocol) produces 0 CROSS edges | High | Medium | Already flagged — treat as documented gap; consider using a WASM/Wasm-C host if a genuine C HTTP service pair can be found | | 159-language sweep is not completable in one session without checkpointing | High | Medium | Add explicit checkpoint/resume logic to the script; describe failure-recovery in §13 | | ~30 flagged ⚠️ repos unavailable, too small, or wrong language on run day | Medium | Medium | Validate all ⚠️ rows before authoring questions; fallback fixture corpus per §8.1 | | Shallow clone at run time produces a different HEAD than during question authoring | Medium | Medium | Pin repos by commit SHA during authoring; bake SHA into `clone-bench-repos.sh` | | 3-pass median of same judge hides variance; passes are correlated not independent | Medium | Medium | Cross-family panel or acknowledge limitation explicitly in §9 | | Explorer spawn overhead excluded but material; Token Ratio misleads | Medium | Medium | Include full-session token cost as a second metric; label the narrow metric clearly | ##### Operational Concerns **Scale / execution feasibility.** 159 × (clone + cold index + graph agent + explorer agent + 3 judge passes) = at minimum 159 × 5 agent invocations = 795 sub-agent sessions, plus 159 × 3 = 477 judge passes. Cold indexing alone for 159 repos, some requiring "full" mode (semantic edges, vector index), will be measured in hours for languages like Haskell/pandoc, Python/httpie, or any repo with millions of lines. The plan gives no clock estimate, no session-continuation strategy if the main session times out, and no checkpoint format beyond "the file exists." One session dropout at language 87 means re-running from language 1 unless someone adds explicit resume logic. This is the single largest operational risk. **The Explorer "unlimited tool calls" vs Graph "fixed tool set" asymmetry.** The Graph agent knows exactly which tools to call: `search_graph` → `trace_call_path` → `get_code_snippet`. It has a clear playbook (the "expected graph tools" hint in §12). The Explorer has no playbook — it is a general `Explore` sub-agent that must orient itself, guess directory structure, and iteratively refine searches. These are not symmetric difficulty conditions. The Graph agent benefits from structured orientation that is invisible to the evaluation because the tool responses come pre-structured. This is not a flaw in the product — it is a real advantage. But the plan presents it as a neutral "same questions, same time window" comparison, which it is not. **The D5 dimension is structurally not neutral.** D5 is defined as "Cross-cutting / Semantic" with the primary tool listed as `search_code` / `search_graph(semantic_query=…)`. For Group E (config/markup/schema) languages, the plan reinterprets D5 as "duplication / naming-pattern / config↔code links." This re-interpretation is ad hoc and dimension-specific — the D5 graph tool advantage does not transfer to these languages in the same way. Aggregating D5 scores across Group A–E languages with such different operational definitions will produce a meaningless cross-group rollup. **No described failure threshold for an individual run.** If the Graph agent returns zero results on D2 (zero-result rate flagged in §5), does the run for that question still count? Is a zero-result a FAIL automatically, or does the judge still grade it? The rubric says Correctness is capped at 0.5 for unverified claims, but a zero-result has no claim to verify. This edge case likely produces a judge confusion artifact at scale. **The judge's ground-truth verification is token-expensive and shallow.** §9.1 says the judge verifies a "sample 3–5 cited symbols/files/lines per question." At 795 questions × 3 judge passes = 2,385 judge invocations, each doing live Grep/Read on the actual repo, this is non-trivial API cost and time. More importantly, sampling 3–5 symbols out of an answer that may cite 20–30 means 85%+ of the answer's claims go unverified. An answer that correctly identifies 3 of 12 handlers will pass the Correctness sample check at 1.0 (all 3 sampled claims correct) while a Completeness-aware judge would score it low. The plan splits Correctness and Completeness into separate sub-scores, which partially addresses this — but the verification step only feeds Correctness, leaving Completeness graded purely on the LLM's intuition about "full scope." ##### Research Findings - Self-preference bias (10–25% inflation) in same-family LLM judges: arXiv 2410.21819; "Quantifying and Mitigating Self-Preference"; FutureAGI bias mitigation survey - Frontier models exceed 50% error rates on advanced bias tests: Adaline LLM-as-a-Judge reliability - Position bias (judge flips preference when A/B swapped, even GPT-4): LLM-as-a-Judge biases (sebastiansigl.com) - Near-duplicate benchmark ground truth — BigCloneBench 93% mislabelling rate, "BigCloneBench Considered Harmful": arXiv 2505.04311; Krinke IWSC22 - Benchmark over-optimism from design choice multiplicity: arXiv 2106.02447 - Code search: grep vs. semantic divergence; only 20–30% solutions overlap: GrepRAG arXiv 2601.23254 - OTel Demo services and languages (confirmed): OpenTelemetry Demo Services - Multi-agent file-polling reliability risks, event-driven vs. polling: Multi-Agent Reliability Patterns - Redis/hiredis uses RESP (not HTTP routes), so CROSS edges cannot form: Redis RESP Protocol; Hiredis GitHub ##### Questions for the Developer 1. **Question authoring source of truth (§12 authoring note):** When you write "questions must cite real symbols, so they are filled in during Phase 0/1" — do you mean you will use the graph to discover those symbols, or will you independently verify them with Grep? If graph-first, you have the bias I described. What is your plan to ensure D1/D3 questions target symbols that Grep can also find? 2. **Judge model identity (§9.4):** What model will be the judge? If it is any Claude model, the same-family self-preference effect applies to every Claude-written Graph and Explorer answer. Have you considered a cross-family judge rotation, or at minimum disclosing the judge model in the report so readers can calibrate? 3. **CROSS edge formation in OTel sub-dirs (§11.1, §15):** Before writing 157 more language chapters, have you actually run `index_repository(mode="cross-repo-intelligence")` on two OTel service sub-dirs and confirmed that CROSS_HTTP_CALLS edges form? This is the load-bearing question for the entire deep-dive block. What is the fallback plan if they don't? 4. **Session continuity (§13):** What happens when the main session context window fills up or hits the usage limit at language 94? Is there a described checkpoint format — e.g., a manifest of completed languages that `clone-bench-repos.sh` can consult to skip already-done languages — or does the whole run restart from zero? 5. **D5 cross-group comparability (§3, §8):** You aggregate D5 scores across all 159 languages. But D5 for Go means `semantic_query=["dispatch","route"]` surfacing functions from a vector index. D5 for gitignore means "naming-pattern / config↔code links." These are different operations using different graph tools. Do you actually intend the cross-language D5 rollup in §10.1 to be meaningful, or is it cosmetic? 6. **S2 ground truth (§11.2):** "3–5 known near-duplicate function pairs" — how will you construct this set for each of the 9 LSP languages? Will you use the simhash output the indexer already produces, or is this a manual read? A 3-pair sample with no inter-rater agreement cannot support a recall claim. What is the minimum ground-truth size you consider credible? 7. **Token exclusion policy (§5):** If a developer is deciding whether to adopt codebase-memory-mcp, they pay the full session cost, including agent spawn, orientation, and formatting. Why should the reported "Token Ratio" exclude the Explorer's orientation cost? Would you consider reporting both the narrow metric and the full-session metric? 8. **The 159-vs-tiered question (over-engineering check):** The plan acknowledges ~30 ⚠️ repos need validation before questions can be written. For languages like `regex` (fixture corpus), `csv` (data, not code), `dotenv` (a handful of fixture files), `sshconfig` (few files), what meaningful D1–D5 question set can be written? Is the signal from these languages worth the cost of the infrastructure to collect it? ##### Recommendation The evaluation's most serious flaw is not operational — it is that the bespoke questions will almost certainly be written from graph-discovered symbols (because the authoring process says "do Phase 0/1 first"), making the graph condition a partial answer to its own test. Combined with a single-family judge that carries 10–25% self-preference bias, the result will be a well-executed benchmark that systematically overstates the graph's advantage. Fix these two issues — symmetric question authoring and a cross-family judge panel — before running the full 159-language sweep, or the SUMMARY.md will not be a credible external artifact. The 159-language scope is also premature given that ~30 repos need validation and session-continuity infrastructure does not yet exist; a pilot run on the 9 LSP languages plus 10 representative Group E languages would de-risk the methodology before committing to 795 hand-written questions. ### 16.3 Strategic forks — resolved - **Fork A — scope: RESOLVED → full 159, specified in this document; execution is downstream.** This document is a **specification to be peer-reviewed before any run**. It therefore enumerates all 159 languages with complete chapters (§14). The challenger's pilot recommendation is preserved as an **execution-time** safeguard: §15 Gate 0 requires a pilot (9 LSP + ~10 representative others) to validate the methodology *before* the downstream team commits to the full sweep. So: the *plan* covers 159; the *first run* should still be a pilot. - **Fork B — judge: RESOLVED → single disclosed model** (§9.4), preferring a non-Claude family to limit self-preference, with the bias caveat stated in the report. A cross-family panel remains a documented upgrade path if cross-provider access exists at run time. --- # Appendix A — Per-language chapters (159) > **⚠️ STATUS — these are DRAFT specifications, not the final question bank.** The chapters below > were **LLM-drafted from model knowledge of each repo** to establish the format, dimension mapping, > and deep-dive shape. Symbols that could not be confirmed carry a `[verify]` tag. They are **not yet > ground-truth-derived** and **must be regenerated at execution time** per §3.1 / §12: > > - **Question *types* are already grounded** in the Sillito et al. taxonomy (§3.1) — that part is final. > - **Question *targets* (the specific symbols/files) are NOT yet from ground truth.** At execution they > are replaced by: **SWE-QA** items for the major languages, and for the rest, targets **seeded from > independent LSP symbol/reference data + git history — never from the model**, then confirmed > grep-first against the pinned commit (symmetric-authoring split, [CR-1]). > > In short: each chapter shows *what kind* of question each dimension asks and *how* it is evaluated > (output, stats block with all 32 edge types, aggregation, LSP deep-dive). It does **not** yet contain > the final, ground-truth-sourced question set — do not read the drafted symbol names as validated. --- ### 1. go — B (Systems & Low-level) **[LSP]** **Repo:** go-chi/chi (`/tmp/bench/go`) **Symlink:** no **Indexed in:** full **Why this repo:** ~19k-star, dependency-free idiomatic Go HTTP router with a non-trivial radix-tree core and a real middleware ecosystem — substantial yet self-contained, matching the plan's "popular + idiomatic + buildable in isolation" repo-selection criteria for the Systems group. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the public API entry points for constructing and configuring a router: where are `NewRouter`, the `Router` interface, and the `Mux` type defined, and what is each one's file?" (all three are grep-findable exported identifiers — `NewRouter` and the `Router` interface in `chi.go`, the `Mux` type in `mux.go` — so plain text search can recover them too) 2. **[D2 Relationship]** "Trace the call graph around route registration: what does `(*Mux).handle` call on the way to inserting into the radix tree (expect a path to `(*node).InsertRoute` in `tree.go`), and who calls `(*Mux).handle` upstream (expect `Method`, `MethodFunc`, and the verb helpers `Get`/`Post`/...)?" 3. **[D3 Retrieval]** "Retrieve the full body of `(*node).findRoute` in `tree.go` — the radix-tree lookup that resolves a request path to a handler." (`findRoute` is a real, grep-findable method name) 4. **[D4 Architecture]** "Describe the package/file organization: the root `chi` package vs. the `middleware/` subpackage, and how the routing core (`mux.go`, `tree.go`, `chain.go`, `context.go`) is separated from the optional middleware stack." 5. **[D5 Cross-cutting/Semantic]** "*(graph-favoring)* Across the `middleware/` package, surface the handler-wrapping middleware constructors that share the `func(http.Handler) http.Handler` shape — e.g. `Logger`, `Recoverer`, `RealIP`, `Compress`, `Throttle` — using semantic similarity rather than a literal substring, to find the cross-cutting 'wrap an http.Handler' pattern even where the token 'middleware' never appears in the function name." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="NewRouter|Router|Mux", label="Function|Interface|Struct"); D2->trace_call_path(name="(*Mux).handle", direction=both); D3->get_code_snippet(qualified_name="github.com/go-chi/chi/v5.(*node).findRoute"); D4->get_architecture(project="go"); D5->search_code/semantic_query(["wrap","http.Handler","middleware","intercept"]). **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry Go checkout (`otel` Go services) -> `product-catalog` (Go), linked via **gRPC**. - **X1 (recall):** From the caller's generated client stubs (the `*Client` interfaces in the OTel demo's `genproto`/`pb` package, e.g. `ProductCatalogServiceClient.GetProduct`, `.ListProducts`, `.SearchProducts`) enumerate every cross-service call route caller->callee. Ground truth = the stub method set cross-checked against the OTel demo service-topology docs (checkout/frontend -> product-catalog edges). Report **recall = found/actual** and **precision** (found-correct / found-total). Target: recall >= 0.7 on the gRPC stub edges; precision penalizes spurious links to non-product-catalog services. - **X2:** Does `get_architecture.cross_repo_links` summarize the caller->callee link with a **count** and **direction** (caller=OTel checkout, callee=product-catalog)? Graded **P** (count+direction both correct) / **partial** (link present but count or direction wrong/missing) / **F** (no cross_repo_link emitted). - **Semantic/similarity** (go-chi/chi indexed in FULL mode): - **S1 (vocabulary bridging):** `semantic_query=["extract","path parameter","url variable"]` should surface chi's parameter-access API where the literal tokens differ — ground-truth set = `URLParam`, `URLParamFromCtx`, `RouteContext`, `(*Context).URLParam`. Metric **hit@5** against that synonym ground-truth set (a "wildcard/route variable" vocabulary that does not literally match the function names). - **S2 (near-duplicate recall):** Ground truth built reproducibly from the indexer's own simhash output over chi (>= 20 Type-1/Type-2 clone pairs, each token-diff confirmed). Prime candidates: the near-identical verb-helper methods (`Get`/`Post`/`Put`/`Delete`/`Patch`/`Head`/`Options`/`Connect`/`Trace` on `*Mux`, all thin wrappers over `handle`/`Method`) and the parallel `*Func` registration variants. Metric **recall** (simhash-seeded pairs recovered) + **false-positive rate** (flagged pairs whose token-diff exceeds the Type-2 threshold). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/go-graph.md`, `go-explorer.md`, `go-judged.json`, `go-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the go tier, and the Deep-Dive section. --- ### 2. python — C (Dynamic & Scripting) **[LSP]** **Repo:** httpie/cli (`/tmp/bench/python`) **Symlink:** no **Indexed in:** full **Why this repo:** ~36k-star, widely-used CLI HTTP client — idiomatic, substantial (multi-package: `cli/`, `output/`, `plugins/`, `manager/`) Python that exercises decorators, dynamic dispatch and plugin entry points, matching the plan's "popular + idiomatic + non-trivial size" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "In `httpie/client.py`, locate the definitions of `build_requests_session` and `collect_messages`, and report their fully-qualified names, file, and line range. (Both are top-level `def`s — grep-findable as `def build_requests_session` / `def collect_messages`.)" 2. **[D2 Relationship]** "Trace the call graph around `httpie.core.program` in both directions: which functions does `program` call (e.g. `collect_messages` in `client.py`), and which functions reach `program` (expected inbound: `main` -> `raw_main` -> `program`)?" 3. **[D3 Retrieval]** "Retrieve the full source of the `HTTPResponse` class in `httpie/models.py` (grep-findable as `class HTTPResponse`), including its `iter_body`, `iter_lines`, `metadata`, and `version` members." 4. **[D4 Architecture]** "Describe the package/module structure of the `httpie` package: the top-level modules (`core.py`, `client.py`, `models.py`, `sessions.py`, …) versus the sub-packages (`cli/`, `output/`, `plugins/`, `manager/`, `internal/`, `legacy/`), and how the entry layer (`__main__.py` -> `core.main`) sits above them." 5. **[D5 Cross-cutting/Semantic]** "*(graph-favoring)* Across `httpie/client.py`, surface the functions that assemble/normalize an outgoing request — including the request-building helpers whose names do NOT contain the token 'header', namely `make_request_kwargs`, `make_send_kwargs`, and `make_send_kwargs_mergeable_from_env` (each merges or normalizes headers/body into the request kwargs) — and group them with the explicitly-named header helpers (`make_default_headers`, `finalize_headers`, `transform_headers`, `apply_missing_repeated_headers`). Plain grep for 'header' recovers the four header-named helpers but UNDER-recalls the `make_*_kwargs` mergers that do header/body normalization without the literal token; semantic similarity should bridge that vocabulary gap." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="build_requests_session|collect_messages", label="Function"); D2->trace_call_path(qualified_name="httpie.core.program", direction="both"); D3->get_code_snippet(qualified_name="httpie.models.HTTPResponse"); D4->get_architecture(project="python"); D5->search_code/search_graph(semantic_query=["build http request","merge request headers and body"]). **Deep-dive (LSP):** - **Cross-repo:** pair = OTel **recommendation service (Python)** -> **product-catalog service (Go)** via gRPC (OpenTelemetry demo topology). * **X1 (recall):** enumerate every cross-service caller->callee route the graph reports for recommendation->product-catalog; ground truth = the gRPC client stub usage in the recommendation service. The recommendation server holds a `ProductCatalogServiceStub` and invokes exactly ONE method on it — `ListProducts` (called twice inside `get_product_list`); it does NOT call `GetProduct` (that edge belongs to other services such as frontend/checkout, not recommendation). So the single ground-truth edge is recommendation -> product-catalog via `ListProducts`. Cross-checked against the OTel demo service-topology docs. Metric: recall = found/actual, plus precision (spurious links / total reported). Target: recall >= 0.8, precision >= 0.9. * **X2:** does `get_architecture.cross_repo_links` summarize the recommendation->product-catalog link with correct **count** and **direction** (Py caller -> Go callee, not reversed)? Graded P (count+direction correct) / partial (link present but count or direction off) / F (link absent). - **Semantic/similarity** (httpie/cli, indexed FULL): * **S1 (vocabulary bridging):** `semantic_query=["serialize response to disk","save body to file"]` should surface the download/streaming writers (e.g. `Downloader` in `httpie/downloads.py`, write/stream helpers) although the literal tokens differ from "download". Metric: hit@5 against a synonym ground-truth set ({download, write, stream-to-file} <-> their real symbols). Target hit@5 >= 0.8. * **S2 (near-duplicate recall):** ground truth built reproducibly from the indexer's simhash output (>=20 Type-1/2 clone pairs, each confirmed by token-diff) — e.g. the parallel `iter_body`/`iter_lines` overrides across `HTTPResponse`/`HTTPRequest`/`HTTPMessage`. Metric: recall over the seeded pair set + false-positive rate (reported pairs not in ground truth). Target recall >= 0.7, FP rate <= 0.2. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/python-graph.md`, `python-explorer.md`, `python-judged.json`, `python-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the python tier, and the Deep-Dive section. --- ### 3. javascript — C (Dynamic & Scripting) **Repo:** expressjs/express @ `4.x` (`/tmp/bench/javascript`) **Symlink:** no **Indexed in:** fast **Why this repo:** The most-depended-upon Node.js web framework (tens of millions of weekly npm installs); its small, idiomatic `lib/` core (factory function, prototype-extended `app`/`req`/`res`, a layered router) is a compact yet representative sample of real-world dynamic JavaScript, matching the plan's "popular + substantial + idiomatic" repo-selection criteria. Branch pinned to `4.x` because the questions exercise the in-tree `lib/router/` subpackage; on the default branch (`master`, Express 5.x) the router was extracted to an external `router` package and `lib/router/` no longer exists, which would invalidate Q2/Q4 symbols. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `createApplication` factory function exported by `lib/express.js`, and list the public application methods defined on the `app` prototype in `lib/application.js` (e.g. `app.use`, `app.listen`, `app.set`, `app.handle`). All are grep-findable identifiers (`function createApplication`, `app.use = function`, ...)." 2. **[D2 Relationship]** "Map the request-dispatch call graph in both directions around the router prototype's `handle` method [verify: source symbol is `proto.handle` in `lib/router/index.js`, not literally `Router.prototype.handle`]: which application entry point reaches it (e.g. `app.handle`) and which layer/route methods it in turn invokes (e.g. `Layer.prototype.handle_request` [verify], `Route.prototype.dispatch` [verify]). These router symbols exist on the `4.x` branch only." 3. **[D3 Retrieval]** "Retrieve the full source of the `res.sendFile` method defined on the response prototype in `lib/response.js` (a single, well-known named symbol; grep-findable as `res.sendFile = function`)." 4. **[D4 Architecture]** "Describe the module/directory architecture of the `lib/` core: how `express.js`, `application.js`, `request.js`, `response.js`, and the `router/` subpackage (`index.js`, `route.js`, `layer.js`) [verify: `lib/router/` is 4.x-only] are organized and depend on one another." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find functions semantically related to HTTP content negotiation / response serialization across files — e.g. `res.json`, `res.jsonp`, `res.send` (in `lib/response.js`) and `req.accepts` (in `lib/request.js`) — that share behavior despite living in different prototypes, and surface the `lib/utils.js` helpers they reuse (e.g. content-type/etag helpers). Labeled graph-favoring: relies on semantic similarity + cross-file reuse, not a single grep token." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="createApplication|app\\.(use|listen|set|handle)"); D2->trace_call_path(name="Router.prototype.handle", direction="both") [verify exact indexed name; fall back to proto.handle / router.handle]; D3->get_code_snippet(qualified_name="res.sendFile"); D4->get_architecture(scope="lib/"); D5->search_code("HTTP content negotiation and response serialization helpers") + search_graph for cross-file reuse of lib/utils.js helpers. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/javascript-graph.md`, `javascript-explorer.md`, `javascript-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the javascript tier. --- ### 4. typescript — C (Dynamic & Scripting) **[LSP]** **Repo:** trpc/trpc (`/tmp/bench/typescript`) **Symlink:** no **Indexed in:** full **Why this repo:** ~37k-star, end-to-end-typesafe RPC monorepo (pnpm workspaces) whose heavy use of generics, builder patterns, and cross-package re-exports makes it an idiomatic, substantial TypeScript stress test — matching the plan's "popular + idiomatic + large enough to expose resolution gaps" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the public API entry points of the server core: locate the definition of the `initTRPC` builder, the `TRPCError` class, and the `callProcedure` function in `packages/server`. Report their qualified names, kind (function/class), and defining file." (all three are well-known, grep-findable tokens) 2. **[D2 Relationship]** "Trace the call graph around `callProcedure` (direction=both): what invokes it on the request path (e.g. `resolveResponse` / `resolveHTTPResponse` [verify]) and what it calls inward (input parsing, the resolver, `getErrorShape` [verify])? Report immediate callers and callees with depth 2." 3. **[D3 Retrieval]** "Retrieve the full source of `getErrorShape` [verify] from `packages/server` (the function that maps a thrown error to the wire-format error object). Return the exact body with its line range." (fallback symbol if absent: `getTRPCErrorFromUnknown` [verify]) 4. **[D4 Architecture]** "Describe the monorepo's package/dir organization: enumerate the `packages/*` workspaces (`server`, `client`, `react-query`, `next`, `react` [verify]) and the dependency direction between `@trpc/client` and `@trpc/server`. Which package defines the link abstraction (`httpBatchLink`, `httpLink`)?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find all functions that *construct or normalize a tRPC error from an unknown thrown value* even when they don't literally contain the substring 'getErrorShape' — i.e. semantic siblings of error-shaping (`getTRPCErrorFromUnknown` [verify], `getErrorFromUnknown` [verify], `TRPCError` construction sites). Rank by semantic similarity to the concept 'convert thrown value into typed RPC error'." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="initTRPC|TRPCError|callProcedure", label="Function|Class"); D2->trace_call_path(qualified_name=".*callProcedure", direction="both", depth=2); D3->get_code_snippet(qualified_name=".*getErrorShape"); D4->get_architecture(project="typescript"); D5->search_code(semantic_query=["convert thrown value into typed RPC error","normalize unknown error to TRPCError"]). **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry demo **frontend (TS/Next)** -> **cart/checkout** services via gRPC/HTTP. * **X1 (recall):** From the frontend's generated client stubs + OTel service-topology docs, enumerate the ground-truth caller->callee routes (e.g. `frontend -> CartService.GetCart`, `frontend -> CartService.AddItem`, `frontend -> CheckoutService.PlaceOrder` [verify]). Run cross-repo trace and report recall = found/actual and precision = correct/returned over that GT set. * **X2:** Inspect `get_architecture.cross_repo_links`: does it summarize the frontend->cart/checkout link with a correct edge **count** and **direction** (frontend as source)? Graded **P** (count+direction right) / **partial** (one wrong) / **F** (missing or reversed). - **Semantic/similarity (trpc/trpc, indexed in FULL mode):** * **S1 (vocabulary bridging):** `semantic_query=["middleware chain","interceptor pipeline","request hook"]` should surface the procedure-middleware machinery (`createMiddlewareFactory` [verify], `middlewares` pipeline in `procedureBuilder` [verify]) although the literal token may be `middleware`/`pipe`. Metric: **hit@5** against a synonym GT set {middleware factory, procedure pipe, resolver wrapper}. * **S2 (near-duplicate recall):** Build GT reproducibly from the indexer's simhash output (>=20 Type-1/2 clone pairs, token-diff confirmed) — expected hot spots: the repeated `httpLink`/`httpBatchLink` request-builder bodies and the per-package `index.ts` re-export barrels. Metric: **recall** = detected/GT-pairs and **false-positive rate** over flagged pairs. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/typescript-graph.md`, `typescript-explorer.md`, `typescript-judged.json`, `typescript-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the typescript tier, and the Deep-Dive section. --- ### 5. tsx — C (Dynamic & Scripting) **Repo:** shadcn-ui/ui (`/tmp/bench/tsx`) **Symlink:** no **Indexed in:** fast **Why this repo:** One of the most-starred React/tsx codebases on GitHub; a substantial, idiomatic pnpm monorepo (`apps/v4` Next.js app + a large `registry/` of real TSX components), matching the plan's criterion of popular, non-trivial, idiomatic repos per language. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `cn` class-name helper (in `apps/v4/lib/utils.ts` [verify]) and the `buttonVariants` export created with `cva` in the button component — return their declaring files and signatures. Both are plain grep-findable identifiers (`export function cn`, `buttonVariants = cva`)." 2. **[D2 Relationship]** "For the `Button` component (`apps/v4/registry/new-york-v4/ui/button.tsx` [verify]), show the full relationship view (direction=both): what it calls/imports (e.g. `cn`, `Slot`, `cva`) and which TSX files consume `Button`." 3. **[D3 Retrieval]** "Retrieve the complete source of `buttonVariants` — the `cva(...)` call defining the variant/size class maps — as a single focused snippet. (A single well-known named symbol, grep-findable.)" 4. **[D4 Architecture]** "Describe the monorepo's top-level architecture: the `apps/v4` Next.js app vs the `registry/` component tree vs the `lib/` helpers [verify], and how the directories are organized." 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Semantically find the components that follow the `cva`-based variant pattern (a `*Variants` export consumed via `cn(...)` + a `VariantProps` typed prop) across the registry, surfacing this cross-cutting idiom rather than a single literal match." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="cn|buttonVariants"); D2->trace_call_path(name="Button", direction="both"); D3->get_code_snippet(qualified_name=".*buttonVariants"); D4->get_architecture(project="tsx"); D5->search_code/semantic_query("cva variant pattern with VariantProps and cn"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/tsx-graph.md`, `tsx-explorer.md`, `tsx-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the tsx tier. --- ### 6. rust — B (Systems & Low-level) **[LSP]** **Repo:** meilisearch/meilisearch (`/tmp/bench/rust`) **Symlink:** no **Indexed in:** full **Why this repo:** A ~50k-star, production search engine that is large, idiomatic, and workspace-structured (multi-crate Cargo workspace with async Tokio actix-web HTTP, LMDB storage, and a custom query engine) — exercises the plan's "popular + substantial + idiomatic" repo-selection criteria for systems Rust. > Repo note: this tree ships TOML symlinks (Cargo manifest / config symlinks). The crate `.rs` sources themselves are not symlinked, so D1–D5 target real source files; the symlink caveat only affects manifest discovery and is flagged where relevant. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definition of the request-handler function `search_with_url_query` and the index-document handler `add_or_replace_documents` in the `meilisearch` crate's routes module — return their signatures and defining files." (Both are concrete `async fn` route handlers under `crates/meilisearch/src/routes/indexes/`; grep-findable by exact name.) 2. **[D2 Relationship]** "Show the full inbound+outbound call neighborhood of the milli search entry point: which HTTP handlers reach it and which lower-level functions (e.g. `execute_search` / ranking-rule evaluation) it calls. In milli, search is driven by the `Search` builder (`Search::new(...).execute()`), not a method on `Index` — resolve the actual entry symbol." [verify: confirm whether the entry is `milli::Search::execute` or `milli::search::new::execute_search`; do not assume a `milli::Index::search` method exists] 3. **[D3 Retrieval]** "Retrieve the complete source of the `SearchQuery` struct (the deserialized search-request parameters type defined in the `meilisearch` crate's search module, `crates/meilisearch/src/search/`)." (One named symbol; `SearchQuery` is a real, grep-findable struct.) 4. **[D4 Architecture]** "Produce the workspace-level architecture: the crate boundaries (`meilisearch`, `milli`, `index-scheduler`, `meilisearch-auth`, `meilisearch-types`, `dump`) and the dependency direction between the HTTP layer, the scheduler, and the storage/query engine." 5. **[D5 Cross-cutting/Semantic]** "**[graph-favoring]** Without using the literal token `ranking`, surface the functions that implement result ordering / relevancy scoring (e.g. ranking-rule application, sort, proximity/typo scoring) via semantic query." (Openly graph-favoring: relies on embedding/semantic match, not exact-token grep.) **Expected graph tools (hint, not a script):** D1->`search_graph(name_pattern="search_with_url_query|add_or_replace_documents", lang="rust")`; D2->`trace_call_path(qualified_name="", direction="both")`; D3->`get_code_snippet(qualified_name="...::SearchQuery")`; D4->`get_architecture(project="rust")`; D5->`search_code(semantic_query="result ordering relevancy scoring proximity")` / `search_graph(semantic_query=...)`. **Deep-dive (LSP):** - **Cross-repo:** **N/A as a true cross-*repo* HTTP edge — meilisearch is a single, standalone search engine; it ships no second service that issues HTTP calls into it, so there is no `caller-repo -> callee-repo` link to score.** Instead, the LSP cross-boundary capability is exercised **intra-workspace, cross-crate**, which is the realistic analogue for this monorepo: - **X1 (cross-crate recall):** enumerate every cross-crate call `caller-crate::fn -> callee-crate::fn` on the HTTP->engine path (e.g. `meilisearch::routes::indexes::search::* -> index_scheduler::* -> milli::*`), computing `recall = found/actual` and `precision`. Ground truth built from the `meilisearch` route handlers' resolved `use` paths plus the workspace `Cargo.toml` dependency graph (which crate may depend on which). Expected ≥1 directed cross-crate edge from the `meilisearch` HTTP layer into `index-scheduler` and from `index-scheduler` into `milli`. - **X2:** check whether `get_architecture.cross_repo_links` (or the cross-crate dependency summary) reports the `meilisearch -> index-scheduler -> milli` chain with edge **count** and **direction**; graded **P** (count+direction correct) / **partial** (link present, count or direction wrong) / **F** (absent). - **Semantic/similarity:** Repo = `meilisearch/meilisearch`, indexed **full**. - **S1 (vocabulary bridging):** `semantic_query=["fault tolerance", "retry on failure", "idempotent reprocessing"]` should surface the index-scheduler's task retry / batch re-run functions even though the literal tokens differ; metric **hit@5** vs a hand-built synonym ground-truth set (5–8 target functions in `index-scheduler`). - **S2 (near-duplicate recall):** ground truth built reproducibly from the indexer's own **simhash** output — collect ≥20 Type-1/Type-2 clone pairs (e.g. the many near-identical per-route `analytics` aggregator impls and serde `Deserr` boilerplate), each token-diff confirmed; metric **recall** (pairs recovered / 20) + **false-positive rate** (reported pairs that fail token-diff confirmation). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/rust-graph.md`, `rust-explorer.md`, `rust-judged.json`, `rust-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the rust tier, and the Deep-Dive section. --- ### 7. java — A (Class-based OOP & Contracts) **[LSP]** **Repo:** spring-projects/spring-petclinic (`/tmp/bench/java`) **Symlink:** no **Indexed in:** full **Why this repo:** The canonical Spring Boot reference app (~7k stars, used in official Spring docs and countless tutorials); small but idiomatic layered OOP (controllers, repositories, JPA entities, interfaces) that exercises Group A's class/interface/contract dimensions, matching the plan's "popular + idiomatic + substantial-enough" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the public methods (controller request-mapping handlers) declared on the class `OwnerController`, including `processCreationForm` and `showOwner`, with their declaring file and line range." (grep-findable: the literal token `OwnerController` and `processCreationForm` appear verbatim in `src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java`.) 2. **[D2 Relationship]** "Trace the full call graph (both directions) around `OwnerRepository.findById` — which controller methods invoke it (inbound) and what JPA/Spring Data plumbing it reaches (outbound)?" Expect the inbound callers inside `OwnerController` to be `showOwner` and the private helper `findOwner` (note: `processUpdateOwnerForm` calls `OwnerRepository.save`, not `findById`); `initUpdateOwnerForm` reaches `findById` only via `findOwner` [verify]. 3. **[D3 Retrieval]** "Retrieve the exact source of the method `Owner.addVisit` (the `Owner` aggregate's method that attaches a `Visit` to a `Pet` by id — signature `addVisit(Integer petId, Visit visit)`)." Single named symbol, grep-findable as `addVisit` in `owner/Owner.java`. 4. **[D4 Architecture]** "Describe the package/module layout of the application: the domain packages (`owner`, `vet`, `system`, `model`), how entity base classes (`BaseEntity`, `NamedEntity`, `Person`) are layered, and where the `*Controller` / `*Repository` pairs live." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find all repository interfaces that extend Spring Data's `JpaRepository`/`Repository` contract across packages (`OwnerRepository`, `VetRepository`, `PetTypeRepository`) without grepping for the literal `extends JpaRepository` — surface them by their structural role as persistence contracts." Labelled graph-favoring (structural contract/IMPLEMENTS reasoning, not a single literal token). **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*OwnerController.*", label="Method"); D2->trace_call_path(qualified_name="...OwnerRepository.findById", direction="both"); D3->get_code_snippet(qualified_name="...owner.Owner.addVisit"); D4->get_architecture(); D5->search_code/search_graph(semantic_query="persistence repository contract", relationship="IMPLEMENTS"). **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry Astronomy Shop **frontend** → **ad service (Java)**, via gRPC. (The Java ad service is a gRPC *server* — `AdServiceGrpc.AdServiceImplBase`, method `getAds` — and accesses feature flags via flagd/OpenFeature, not a gRPC feature-flag stub; so the graphable cross-service edge is the inbound call into the ad service, not an outbound one.) **X1 (recall):** enumerate every cross-service call caller->callee route landing on the ad service's gRPC entrypoint (`oteldemo.AdService/getAds`, served by `AdServiceImpl.getAds` [verify]) and compare against the OTel demo service-topology docs; report recall = found/actual and precision = correct/found. Ground truth = the published OTel topology edges incident to the ad service ∩ the ad service's exported gRPC method set. **X2:** check whether `get_architecture.cross_repo_links` summarizes the frontend->ad-service link (edge count, direction caller ⇒ Java-ad); graded P (count+direction correct) / partial (link present, count or direction wrong) / F (absent). - **Semantic/similarity:** Own repo spring-petclinic, FULL mode. **S1 (vocabulary bridging):** `semantic_query=["save owner", "persist customer", "store pet owner record"]` should surface `OwnerController.processCreationForm` + `OwnerRepository.save` though the literal token is "Owner"/"create", not "save customer"; metric hit@5 against a hand-built synonym ground-truth set (owner↔customer, save↔persist↔store, find↔lookup). **S2 (near-duplicate recall):** ground truth built reproducibly from the indexer's simhash output — collect ≥20 Type-1/2 clone pairs (e.g. the near-identical `findById`/`findAll` boilerplate and the parallel `*Controller` form-validation blocks), token-diff confirmed; metric = recall of those seeded pairs + false-positive rate (flagged-but-not-clone / total flagged). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/java-graph.md`, `java-explorer.md`, `java-judged.json`, `java-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the java tier, and the Deep-Dive section. --- ### 8. cpp — B (Systems & Low-level) **Repo:** nlohmann/json (`/tmp/bench/cpp`) **Symlink:** no **Indexed in:** fast **Why this repo:** One of the most-starred C++ libraries on GitHub (~40k+ stars), a single-header JSON library that is idiomatic, heavily template-metaprogrammed, and substantial — matching the plan's criterion of popular, real-world, structurally rich Systems/low-level C++. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the public API surface of the library's core type: find the definition of the `basic_json` class template and its primary type alias `json`, plus the member function `parse`. All three are grep-findable identifiers in `single_include/nlohmann/json.hpp`." 2. **[D2 Relationship]** "Map the relationships around the entry point `basic_json::parse` [verify: exact QN is namespace/version-sensitive — `basic_json` is a class template behind the `json` alias and lives in an inline ABI/version namespace, so the resolvable method path may be e.g. `nlohmann::basic_json::parse` or a versioned variant]: trace (direction=both) which functions it calls into (e.g. the `parser`/`sax` machinery) and which API surfaces or operators call it, to confirm the call graph links the public `parse` API to the internal parser implementation." 3. **[D3 Retrieval]** "Retrieve the full source of the SAX-style parser class `nlohmann::detail::parser` (or the streaming `json_sax` interface). Return the exact definition with its line boundaries — a single named symbol that also exists verbatim in the amalgamated header." 4. **[D4 Architecture]** "Describe the architecture/structure of the repo: the split between the developer-facing modular sources under `include/nlohmann/` (e.g. `detail/`, `thirdparty/`) and the generated `single_include/nlohmann/json.hpp` amalgamation, plus the `tests/`, `docs/`, and `tools/` (amalgamation) layout. Which directories hold the canonical source vs. the build artifact?" 5. **[D5 Cross-cutting/Semantic]** "(GRAPH-FAVORING) Semantically locate the error/exception-handling cross-cut: find the family of exception types and where they are thrown — e.g. `parse_error`, `type_error`, `out_of_range`, `invalid_iterator` — and the `JSON_THROW`/`JSON_TRY` macro usage that wires them in. This favors semantic/similarity search because the concept ('how does the library report errors') spans many files and macro-obscured throw sites rather than one grep token." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*basic_json.*|.*::parse$"); D2->trace_call_path(qualified_name="nlohmann::basic_json::parse" [verify], direction="both"); D3->get_code_snippet(qualified_name="nlohmann::detail::parser"); D4->get_architecture(...); D5->search_code/semantic_query("exception types thrown on parse/type errors; JSON_THROW macro"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/cpp-graph.md`, `cpp-explorer.md`, `cpp-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the cpp tier. --- ### 9. csharp — A (Class-based OOP & Contracts) **[LSP]** **Repo:** ardalis/CleanArchitecture (`/tmp/bench/csharp`) **Symlink:** no **Indexed in:** full **Why this repo:** High-star canonical .NET reference template (clean layering, DDD aggregates, CQRS via MediatR, FastEndpoints) — idiomatic and substantial C#, satisfying the plan's "popular + idiomatic + non-trivial size" repo-selection criteria for Group A. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the in-repo classes/interfaces that make up the sample aggregate's API surface — e.g. the `Contributor` entity class, the `EfRepository` implementation class, and the locally-declared contracts in `Core/Interfaces` (`IAggregateRoot`, `IDeleteContributorService`, `IEmailSender`) — and report each one's declaring file and kind (class vs interface). All cited names are plain-text greppable in `src/` (`grep -rn 'class Contributor' src/`, `grep -rn 'class EfRepository' src/`, `grep -rn 'interface IDeleteContributorService' src/`). NOTE: `IRepository`/`IReadRepository` are implemented by `EfRepository` but are *external* (Ardalis.Specification NuGet) and are deliberately excluded from the in-repo grep list." 2. **[D2 Relationship]** "Trace inbound and outbound calls of `EfRepository` (the generic `IRepository`/`IReadRepository` implementation): which use-case handlers / services depend on it (inbound, e.g. `DeleteContributorService` and the MediatR handlers under `UseCases/Contributors`), and which framework/specification members it reaches (outbound, e.g. the inherited `RepositoryBase` / Ardalis.Specification `ApplySpecification`). Direction = both." 3. **[D3 Retrieval]** "Return the exact source of the `Contributor` aggregate root (`Contributor` class in `Clean.Architecture.Core.ContributorAggregate`), including its (primary) constructor, the `UpdateName` method, and the domain-event registration inside `UpdateName`." (Single named symbol; greppable via `grep -rn 'public class Contributor' src/`, which matches the primary-constructor declaration `public class Contributor(ContributorName name) : ...`.) 4. **[D4 Architecture]** "Describe the layer structure of the solution: the `Clean.Architecture.Core` / `.UseCases` / `.Infrastructure` / `.Web` projects, the dependency direction between them (Web -> UseCases -> Core; Infrastructure -> Core), and where DI wiring lives (`Program.cs` plus the `Add*Configs` service-configuration extension methods [verify]; SharedKernel building blocks come from the external Ardalis.SharedKernel package, not an in-repo project [verify]). Confirm the dependency rule (inner layers know nothing of outer)." 5. **[D5 Cross-cutting/Semantic]** "GRAPH-FAVORING: find all handlers/services that *delete a contributor* even when the literal token differs (e.g. `DeleteContributorService` in Core/Services, the FastEndpoints `Delete` endpoint under `Web/Contributors`, and the `DeleteContributorCommand`/`DeleteContributorHandler` in UseCases), via semantic_query=[\"remove\",\"contributor\",\"command\"] rather than exact text — measure recall vs a grep-only baseline for the same intent." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="Contributor|EfRepository|IDeleteContributorService|IAggregateRoot", label="Class|Interface"); D2->trace_call_path(name="EfRepository", direction="both"); D3->get_code_snippet(qualified_name="...ContributorAggregate.Contributor"); D4->get_architecture(); D5->search_code / search_graph semantic_query=["remove","contributor","command"]. **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry demo `checkout` (Go) -> `cart` (.NET), via gRPC (the verified demo edge is `checkout -->|gRPC| cart`; the .NET `cart` service is the gRPC **callee/server**, not the caller). **X1 (recall):** enumerate every cross-service call caller->callee route that resolves into the .NET `cart` service — i.e. Go `checkout`'s generated client invoking `cart`'s `CartService` methods (`GetCart`/`AddItem`/`EmptyCart`) — using ground truth taken from the generated gRPC stubs (`*.cs` server stubs + Go client from `demo.proto`) cross-checked against the OTel demo service-topology docs; report recall = found / actual and precision = correct / found. NOTE: the .NET `accounting` service is an async **Kafka consumer** (`queue -->|TCP| accounting`), not a gRPC caller of checkout, so it is excluded from the gRPC X1 set and only appears (if at all) as an ASYNC_CALLS edge. **X2:** does `get_architecture.cross_repo_links` summarize the Go-checkout -> .NET-cart link (edge count and direction)? Graded P / partial / F (P = correct count + direction Go-checkout => .NET-cart; partial = link present but miscounted or undirected; F = absent). - **Semantic/similarity:** Indexed in FULL mode on ardalis/CleanArchitecture. **S1 (vocabulary bridging):** semantic_query=["remove","contributor"] (and ["fetch","by","id"]) must surface `DeleteContributorService` / the `Delete` endpoint (and the `GetById` endpoint / `ContributorByIdSpec` [verify]) though the source token is "Delete"/"Spec"; metric = hit@5 against a synonym ground-truth set (remove≈delete, fetch≈get, list≈enumerate). **S2 (near-duplicate recall):** ground truth built reproducibly from the indexer's simhash output (>=20 Type-1/2 clone pairs, each token-diff confirmed) — expected clusters are the near-identical FastEndpoints CRUD endpoints repeated across `Web/Contributors/*` (Create/Update/Delete/GetById request-response DTO + validator triples) and the parallel specification classes; metric = recall (seeded pairs recovered) + false-positive rate (flagged pairs that fail token-diff confirmation)." **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/csharp-graph.md`, `csharp-explorer.md`, `csharp-judged.json`, `csharp-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the csharp tier, and the Deep-Dive section. --- ### 10. php — A (Class-based OOP & Contracts) **[LSP]** **Repo:** laravel/framework (`/tmp/bench/php`) **Symlink:** no **Indexed in:** full **Why this repo:** The canonical, most-starred PHP framework — heavily interface-driven (`Illuminate\Contracts\*`), trait-laden, and DI-container-centric, exercising class/interface/trait/method extraction at real scale, matching the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `Illuminate\Container\Container` class and enumerate its public resolution API — specifically the `make`, `bind`, `singleton`, and `resolve` methods. Are the method signatures and declaring class reported correctly?" 2. **[D2 Relationship]** "For `Illuminate\Container\Container::make`, trace the call graph in both directions: what does `make` call internally (e.g. `resolve` / `build` [verify]) and which callers invoke `make` (e.g. the `App` facade / `Application` bootstrap path)?" 3. **[D3 Retrieval]** "Retrieve the full source of the single method `Illuminate\Database\Eloquent\Model::save` with exact line boundaries." 4. **[D4 Architecture]** "Describe the top-level package/namespace structure under `src/Illuminate` (Container, Database, Routing, Support, Contracts, ...) and how the `Contracts` namespace relates to the concrete implementations — i.e. the interface-vs-implementation layering." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find all classes that implement the `Illuminate\Contracts\Support\Arrayable` contract via its `toArray` method, and surface near-duplicate `toArray`/`jsonSerialize` implementations across `Collection`, `Model`, and resource classes. Semantic/IMPLEMENTS-edge query — not answerable by a single grep." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*Container.*", label="Class"/"Method"); D2->trace_call_path(qualified_name="Illuminate\\Container\\Container::make", direction="both"); D3->get_code_snippet(qualified_name="Illuminate\\Database\\Eloquent\\Model::save"); D4->get_architecture(); D5->search_code/semantic_query + search_graph(relationship="IMPLEMENTS"). **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry **shipping(Rust)** -> **quote(PHP)** over HTTP (OTel demo microservices topology). NOTE on direction: per the OTel demo docs the **shipping service (Rust) is the caller** and the **quote service (PHP) is the callee** — shipping issues an HTTP request to quote to compute shipping cost, hitting quote's `getquote` route. PHP is therefore the cross-repo *callee*; the edge to evaluate is the **inbound** cross-language link into this PHP repo. - **X1 (recall):** Enumerate every cross-service call edge caller(shipping/Rust)->callee(quote/PHP) with its route; report recall = found/actual and precision = correct/found. Ground truth = the shipping service's outbound HTTP client call to quote's `/getquote` route [verify route path] cross-checked against the OTel demo service-topology docs. Expected: small actual set (1–2 edges); recall/precision computed against that hand-built set. (This is a legitimate graph-vs-grep test: a grep inside the PHP repo alone cannot recover the Rust-side caller; only the linked cross-repo graph surfaces the inbound edge.) - **X2:** Does `get_architecture.cross_repo_links` summarize the shipping->quote link (edge count + direction)? Graded **P** (count and direction both correct, i.e. caller=shipping/Rust, callee=quote/PHP) / **partial** (link present but count or direction wrong — e.g. direction reported reversed) / **F** (link absent). - **Semantic/similarity:** Own repo laravel/framework, indexed in FULL mode. - **S1 (vocabulary bridging):** `semantic_query=["dependency injection resolve", "service container lookup", "instantiate bound abstract"]` should surface `Container::make` / `Container::resolve` / `Container::build` though the literal tokens differ. Metric = hit@5 against a synonym ground-truth set {`make`, `resolve`, `build`, `singleton`}. - **S2 (near-duplicate recall):** Ground truth built reproducibly from the indexer's simhash output (>=20 Type-1/2 pairs, token-diff confirmed) — e.g. the many parallel `Str`/`Arr` helper variants and the repetitive Eloquent relation `getResults`/`addConstraints` pairs. Metric = recall (simhash-flagged pairs the tool surfaces) + false-positive rate (flagged pairs that are not true Type-1/2 clones on manual token-diff). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/php-graph.md`, `php-explorer.md`, `php-judged.json`, `php-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the php tier, and the Deep-Dive section. --- ### 11. lua — C (Dynamic & Scripting) **Repo:** awesomeWM/awesome (`/tmp/bench/lua`) **Symlink:** no **Indexed in:** fast **Why this repo:** AwesomeWM is a top-tier (~6k-star) tiling window manager whose `lib/` is a large, idiomatic Lua standard library (awful/wibox/gears/naughty/menubar) — substantial real-world Lua with deep module nesting and call chains, matching the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `gears.timer` constructor and the `awful.spawn` function — both are grep-findable top-level API symbols. Does the tool surface their qualified names and source files?" 2. **[D2 Relationship]** "Starting from `awful.spawn` (in `lib/awful/spawn.lua`), trace the call graph in both directions: which awful modules call into spawn, and which lower-level functions (e.g. the C-exposed `awesome.spawn` global [verify — C-side binding, may not be a Lua graph node]) does it call?" 3. **[D3 Retrieval]** "Retrieve the full source of `gears.timer` (the timer object factory in `lib/gears/timer.lua`) as a single snippet, with correct start/end boundaries." 4. **[D4 Architecture]** "Describe the module/directory organization of the Lua library: how are the `awful`, `wibox`, `gears`, `naughty`, and `menubar` namespaces laid out under `lib/`, and how do submodules roll up into each namespace's `init.lua`?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find code semantically related to 'connecting and handling client/signal callbacks' across the library — e.g. `connect_signal` / `emit_signal` usage patterns — and surface duplication or shared signal-wiring conventions across `awful.client`, `awful.tag`, and `naughty`." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*spawn.*|.*timer.*"); D2->trace_call_path(name="spawn", direction="both"); D3->get_code_snippet(qualified_name="gears.timer"); D4->get_architecture(); D5->search_code/semantic_query("signal connect/emit handlers"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/lua-graph.md`, `lua-explorer.md`, `lua-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the lua tier. --- ### 12. scala — A (Class-based OOP & Contracts) **Repo:** playframework/play-samples (`/tmp/bench/scala`) **Symlink:** no **Indexed in:** fast **Why this repo:** Play's official sample monorepo is the most-referenced, idiomatic Scala teaching corpus (high GitHub visibility, maintained by the framework authors), and its REST-API example is a substantial trait/class/object-rich slice that exercises Group A's class-based OOP + contract (trait) concerns — matching the plan's "popular, idiomatic, substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `PostController` class and the `PostRepository` trait in `play-scala-rest-api-example` (package `v1.post`), and report each one's declared public methods (e.g. `index`, `process`, `show` on the controller; `create`, `list`, `get` on the trait)." (grep-findable: identifiers `PostController`, `PostRepository` appear verbatim in source.) 2. **[D2 Relationship]** "Show the inbound and outbound call relationships for `PostResourceHandler.create` — which controller action ultimately invokes it (note the call is made from `PostController.process` via the private helper `processJsonPost`, not directly in the action body), and which `PostRepository` method it calls in turn — so we can see the controller → handler → repository contract chain in both directions." 3. **[D3 Retrieval]** "Retrieve the full source of the `PostRepositoryImpl` class (the in-memory implementation of the `PostRepository` trait in `play-scala-rest-api-example`)." (grep-findable: identifier `PostRepositoryImpl` appears verbatim in source; D3 is a fair, symmetric retrieval target.) 4. **[D4 Architecture]** "Describe the package/module structure of the `play-scala-rest-api-example` app: how the `v1.post` package is organized (controller, router, handler, repository, action-builder) and how the `Module`/router wiring binds the trait `PostRepository` to its implementation." 5. **[D5 Cross-cutting/Semantic]** "Graph-favoring: across the Scala samples, find the components that play the same architectural role as `PostResourceHandler` — i.e. service/handler classes that mediate between an HTTP controller and a repository/DAO — even when they are named differently (e.g. *...Handler*, *...Service*, *...ResourceHandler*). Use semantic/similarity search rather than exact-name grep, then cross-link each to the controller and repository it sits between." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="PostController|PostRepository", label="Class|Trait"); D2->trace_call_path(qualified_name="v1.post.PostResourceHandler.create", direction="both"); D3->get_code_snippet(qualified_name="v1.post.PostRepositoryImpl"); D4->get_architecture(scope="play-scala-rest-api-example"); D5->search_code/semantic_query("service mediating controller and repository"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/scala-graph.md`, `scala-explorer.md`, `scala-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the scala tier. --- ### 13. kotlin — A (Class-based OOP & Contracts) **[LSP]** **Repo:** JetBrains/Exposed (`/tmp/bench/kotlin`) **Symlink:** no **Indexed in:** full **Why this repo:** JetBrains' own Kotlin SQL framework/ORM is a top-tier, high-star, idiomatic-Kotlin codebase that is heavily class/interface/contract-driven (Table DSL, Entity DAO, IColumnType hierarchy), matching Group A's OOP-and-contracts focus and the plan's "popular + substantial + idiomatic" repo-selection rule. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definition of the abstract base class `Table` and the `Column` class in the `org.jetbrains.exposed.sql` package, and list the public columnar factory methods declared on `Table` (e.g. `integer`, `varchar`, `text`, `bool`, `reference`). Both class names and these method names are plain grep-findable identifiers." 2. **[D2 Relationship]** "Show the bidirectional call graph (callers and callees) of the top-level `transaction(...)` function in `org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManager` [verify] — who invokes `transaction { }` and what it calls (e.g. `TransactionManager.currentOrNew`, `Transaction.commit`, `Transaction.rollback`)." 3. **[D3 Retrieval]** "Retrieve the exact source of the `SchemaUtils.create(vararg tables: Table)` method — one specific, grep-findable named symbol — with precise line boundaries." 4. **[D4 Architecture]** "Describe the module/package architecture of Exposed: how the core SQL DSL layer (`exposed-core`, `org.jetbrains.exposed.sql`) relates to the DAO/Entity layer (`exposed-dao`, `org.jetbrains.exposed.dao`) and the dialect/JDBC layers, and which directories hold each." 5. **[D5 Cross-cutting/Semantic]** "(GRAPH-FAVORING) Using semantic search, surface the code paths that perform 'persist a row to the database' even when the literal term is absent — i.e. the `insert`/`batchInsert`/`InsertStatement` machinery — and find structurally near-duplicate statement-builder classes (`InsertStatement` vs `UpdateStatement` vs `ReplaceStatement`)." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*Table.*|.*Column.*", label="Class"); D2->trace_call_path(qualified_name="...transaction", direction="both"); D3->get_code_snippet(qualified_name="org.jetbrains.exposed.sql.SchemaUtils.create"); D4->get_architecture(); D5->search_code(semantic_query="persist row to database") / search_graph(semantic_query=...). **Deep-dive (LSP):** - **Cross-repo:** Pair = OpenTelemetry demo `fraud-detection`(Kotlin, the caller) -> `checkout`(Go, the callee), linked via Kafka topic consumption / gRPC. **X1 (recall):** enumerate every cross-service call route caller->callee — fraud-detection consumes the `orders` Kafka topic produced by checkout (and any gRPC stubs it calls); report recall = found/actual and precision against ground truth derived from fraud-detection's Kafka consumer config + generated client stubs and the OTel demo service-topology docs. Expected actual set is small (1–2 routes: Kafka `orders` consume; possible gRPC), so a single miss swings recall hard — record exact numerator/denominator. **X2:** does `get_architecture.cross_repo_links` summarize the fraud-detection -> checkout link (edge count, direction)? Graded P/partial/F: P = correct count and direction, partial = link present but direction or count wrong, F = absent. - **Semantic/similarity:** (own repo JetBrains/Exposed, FULL index) **S1 (vocabulary bridging):** `semantic_query=["delete rows","remove records","purge entries"]` should surface `DeleteStatement` / `Table.deleteWhere` / `deleteAll` though the literal token differs; metric hit@5 against a synonym ground-truth set {`DeleteStatement`, `deleteWhere`, `deleteAll`, `deleteIgnoreWhere` [verify]}. **S2 (near-duplicate recall):** ground truth built reproducibly from the indexer's simhash output (>=20 Type-1/2 pairs, each token-diff confirmed) — expected clusters include the `*Statement` builders (Insert/Update/Replace/Delete), the per-dialect classes (`MysqlDialect`/`PostgreSQLDialect`/`SQLiteDialect` [verify]), and the `*ColumnType` family; metric = recall over the simhash-seeded pair set + false-positive rate (reported pairs whose token-diff exceeds the Type-2 threshold). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/kotlin-graph.md`, `kotlin-explorer.md`, `kotlin-judged.json`, `kotlin-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the kotlin tier, and the Deep-Dive section. --- ### 14. ruby — A (Class-based OOP & Contracts) **Repo:** sinatra/sinatra (`/tmp/bench/ruby`) **Symlink:** no **Indexed in:** fast **Why this repo:** Top-tier (~12k-star) idiomatic Ruby web framework whose `Sinatra::Base` DSL, mixins, and class-method routing make it a substantial, class-based OOP exemplar — matching the plan's "popular + idiomatic + substantial" repo-selection criteria for Group A. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `Sinatra::Base` class and its routing class methods `get` and `post` (public DSL) plus the underlying `route` class method (private) — return their declaring file and signatures." 2. **[D2 Relationship]** "Map the inbound and outbound call graph of `Sinatra::Base#dispatch!`: which methods invoke it on a request and which methods it calls (e.g. `process_route`, `route!`, `invoke`)." 3. **[D3 Retrieval]** "Retrieve the full body of `Sinatra::Base#process_route` exactly as written." 4. **[D4 Architecture]** "Describe how the `sinatra` gem is organized under `lib/sinatra/` — the core `base.rb` vs. the auto-running `main.rb`, the `Delegator`/`Helpers` mixins, and middleware files like `show_exceptions.rb`." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring, semantic) Find code semantically related to 'rendering a template / view layer' across the gem — e.g. `render`, the per-engine helpers (`erb`, `haml`, `markdown`), and `Templates` — even where the identifier names differ from the query terms." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="Sinatra::Base|^get$|^route$", label="Class|Method"); D2->trace_call_path(qualified_name="Sinatra::Base#dispatch!", direction="both"); D3->get_code_snippet(qualified_name="Sinatra::Base#process_route"); D4->get_architecture(scope="lib/sinatra"); D5->search_code(query="render template view engine")/semantic_query. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/ruby-graph.md`, `ruby-explorer.md`, `ruby-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the ruby tier. --- ### 15. c — B (Systems & Low-level) **[LSP]** **Repo:** redis/redis (`/tmp/bench/c`) **Symlink:** no **Indexed in:** full **Why this repo:** Among the most-starred C systems projects on GitHub (~65k stars), redis is large, idiomatic single-process C (event loop + command table + handcrafted data structures), exercising the plan's "popular, substantial, idiomatic" repo-selection criteria for the C tier. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List every definition whose name matches the hash-table API surface in `dict.c` — `dictAdd`, `dictFind`, `dictCreate`, `dictExpand`, `dictDelete` — and report the file and signature of each. (grep-findable: all are literal `dict*` identifiers.)" 2. **[D2 Relationship]** "Show the full caller/callee context of `processCommand` (server.c): which functions reach it (e.g. command-processing path from the networking layer) and what it dispatches to (e.g. `call`, `lookupCommand`, `addReplyError`)? Use direction=both." 3. **[D3 Retrieval]** "Retrieve the exact source of the single function `createStringObject` (object.c) with precise line boundaries — its full body, not the surrounding file." 4. **[D4 Architecture]** "Summarize the top-level module/file organization of redis under `src/`: how do the event loop (`ae.c`), command dispatch (`server.c`), object system (`object.c`), the data-type implementations (`t_string.c`, `t_hash.c`, `t_list.c`, …), and the string library (`sds.c`) group together? Report directory/file clustering, not symbol bodies." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the functions that *reply to a client over the wire* even when the literal token 'reply' is absent — e.g. functions that append protocol output to the client buffer. Seed a semantic_query=[\"respond\",\"client\",\"output buffer\"] and expect to surface `addReply*` family and `_addReplyToBuffer` [verify] though the query word 'respond' never appears in their names." **Expected graph tools (hint, not a script):** D1->search_graph(project="...redis", name_pattern="dict(Add|Find|Create|Expand|Delete)", label="Function"); D2->trace_call_path(name="processCommand", direction="both"); D3->get_code_snippet(qualified_name="...createStringObject"); D4->get_architecture(project="...redis"); D5->search_code / search_graph(semantic_query=["respond","client","output buffer"]). **Deep-dive (LSP):** - **Cross-repo:** pair = redis/redis <-> redis/hiredis. The link is the RESP wire protocol, **not** HTTP routes, so the linker's URL/route matching cannot form CROSS edges here — report as a **documented capability gap**, not a failure. - **X1 (recall):** Enumerate every cross-service call caller->callee route from hiredis's client stubs into a redis server command path (e.g. `redisCommand`/`redisAppendCommand` -> server-side `processCommand`/`*Command` handler). Ground truth = hiredis client stub set (`redisConnect`, `redisCommand`, `redisGetReply`, `redisReader` [verify]) cross-referenced with redis command-table handlers, plus any OTel service-topology doc for the pair. Because RESP is not HTTP, **expected found = 0 of N actual**; report recall = 0/N and precision = undefined/(n=0) and label the gap. Metric here measures honest gap-reporting, not edge formation. - **X2:** Does `get_architecture.cross_repo_links` summarize a redis<->hiredis caller->callee link (count, direction)? **Expected: F** (no RESP-aware cross edge) — graded P/partial/F; a "partial" is awarded only if the field at least lists both repos as indexed without inventing a spurious link. - **Semantic/similarity:** (redis/redis indexed in FULL mode) - **S1 (vocabulary bridging):** semantic_query=["fetch","key","read"] should surface `lookupKeyRead` / `lookupKeyReadWithFlags` [verify] and `getGenericCommand` [verify] though the literal token "fetch" is absent. Metric = hit@5 against a synonym ground-truth set {lookupKeyRead, lookupKeyReadWithFlags, lookupKeyWrite, getCommand, getGenericCommand}. - **S2 (near-duplicate recall):** Ground truth built reproducibly from the indexer's simhash output: take the top simhash-clustered pairs (>=20 Type-1/Type-2 clones) — strong candidates are the parallel `t_*` type handlers and the `addReply*` reply variants whose bodies differ only by token (e.g. `addReplyBulk` vs `addReplyBulkCBuffer` [verify], `lpushCommand` vs `rpushCommand` family) — and confirm each pair by token-diff before admitting it. Metric = recall (simhash-found / token-confirmed actual) + false-positive rate (admitted pairs that token-diff rejects). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/c-graph.md`, `c-explorer.md`, `c-judged.json`, `c-deepdive.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the c tier, and the Deep-Dive section. --- ### 16. bash — C (Dynamic & Scripting) **Repo:** bash-it/bash-it (`/tmp/bench/bash`) **Symlink:** no **Indexed in:** fast **Why this repo:** A widely-starred (>14k), idiomatic Bash framework whose functions, aliases, completions and themes span hundreds of sourced `.bash` files — substantial real-world shell with deep function-to-function sourcing, satisfying the plan's "popular + idiomatic + substantial" repo-selection criteria for Group C. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the top-level dispatcher function `bash-it` and the framework loader entrypoint `bash_it.sh`; list the public `_bash-it-*` helper functions (`_bash-it-aliases`, `_bash-it-completions`, `_bash-it-plugins`) that constitute the framework's enable/disable API." (grep-findable: these names appear literally as `function bash-it()` / `function _bash-it-aliases()` in `lib/helpers.bash`.) 2. **[D2 Relationship]** "Starting from the `bash-it` dispatcher, trace the call graph (both directions) through `_bash-it-main-help-*` [verify] / `_enable-thing` / `_disable-thing` to show how an `enable plugin ` invocation reaches the symlink-creation logic in `_enable-thing` that links a file from `plugins/available` into `plugins/enabled`." (structural framing allowed for D2.) 3. **[D3 Retrieval]** "Retrieve the full source of the single function `_bash-it-find-in-ancestor` (the ancestor-directory search helper in `lib/helpers.bash`)." (one real, grep-findable named symbol.) 4. **[D4 Architecture]** "Describe the directory/file organization of bash-it: the role of `bash_it.sh`, `lib/`, `plugins/available` vs `plugins/enabled`, `aliases/`, `completion/`, and `themes/`, and how the enabled-vs-available symlink convention structures the framework." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Surface naming-pattern and config↔code links: find clusters of similarly-named enable/disable helpers (`_enable-thing` / `_disable-thing` / `_enable-plugin` / `_disable-plugin` …) and the `_bash-it-reload` / `_bash-it-restart` reload functions, and identify duplicated theme-prompt scaffolding shared across `themes/*/` — a similarity/duplication query a plain grep cannot rank." (explicitly graph-favoring; included in Group C D5 only.) **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="^(bash-it|_bash-it-.*)$", label=Function); D2->trace_call_path(name="bash-it", direction="both"); D3->get_code_snippet(qualified_name="_bash-it-find-in-ancestor"); D4->get_architecture(); D5->search_code/semantic_query("theme prompt scaffolding duplication; enable/disable helper family"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/bash-graph.md`, `bash-explorer.md`, `bash-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the bash tier. --- ### 17. zig — B (Systems & Low-level) **Repo:** tigerbeetle/tigerbeetle (`/tmp/bench/zig`) **Symlink:** no **Indexed in:** fast **Why this repo:** TigerBeetle is the most-starred, most-idiomatic large Zig systems codebase (financial OLTP database) — heavy comptime generics, LSM storage, VSR consensus — making it a substantial, real-world stress test that matches the plan's "popular + idiomatic + non-trivial size" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the public API surface of the state machine: find the `StateMachineType` generic and its public methods `init`, `deinit`, `reset`, `open`, and `input_valid` in `src/state_machine.zig`. Do all five resolve with correct file/line, and is the comptime-generic `StateMachineType` recognized as a definition rather than missed?" (All targets are grep-findable plain identifiers.) 2. **[D2 Relationship]** "For `parse_addresses` (in `src/vsr.zig`), show the full call relationship in both directions: who calls it (e.g. config/CLI parsing paths) and what it calls. Does the graph reconstruct the inbound/outbound CALLS edges correctly?" 3. **[D3 Retrieval]** "Retrieve the exact source of the `exponential_backoff_with_jitter` function declared in `src/vsr.zig` — return only that function body with precise boundaries, no surrounding noise." (Single, well-known, grep-findable symbol.) 4. **[D4 Architecture]** "Describe the top-level module/folder architecture of `src/`: the major subsystems `vsr/`, `lsm/`, `io/`, `state_machine/`, `clients/`, `tigerbeetle/`, `stdx/`, and key entry file `src/tigerbeetle/main.zig`. Does the structural view capture the subsystem grouping and the main binary entry point?" 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Find the cross-cutting checksum/integrity machinery: surface declarations semantically related to data integrity — e.g. `checksum` and `ChecksumStream` in `src/vsr.zig`, the `ewah.zig` run-length codec, and superblock/manifest verification — that a plain grep for one keyword would not cohere into one set. Does semantic/similarity retrieval cluster the integrity-related symbols across files?" **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="StateMachineType|init|deinit|reset|open|input_valid", path~"src/state_machine.zig"); D2->trace_call_path(name="parse_addresses", direction="both"); D3->get_code_snippet(qualified_name="...exponential_backoff_with_jitter"); D4->get_architecture(scope="src"); D5->search_code/semantic_query("checksum integrity verification superblock manifest"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/zig-graph.md`, `zig-explorer.md`, `zig-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the zig tier. --- ### 18. elixir — D (Functional & Formal) **Repo:** phoenixframework/phoenix (`/tmp/bench/elixir`) **Symlink:** no **Indexed in:** fast **Why this repo:** Phoenix is the de-facto standard Elixir web framework (~21k+ GitHub stars), a large, idiomatic, macro-heavy codebase that exercises the plan's "popular + substantial + idiomatic" repo-selection criteria for the functional/formal group. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definitions of the public macros `plug` and `pipe_through/1` exposed by `Phoenix.Router`, plus the `Phoenix.Controller` module — where are they defined and what is their arity/signature?" (all are well-known, grep-findable identifiers: `defmacro plug(plug, opts \\ [])` (arity 2, callable as `plug/1`) and `defmacro pipe_through(pipes)` in `lib/phoenix/router.ex`, and `defmodule Phoenix.Controller` in `lib/phoenix/controller.ex`) 2. **[D2 Relationship]** "For `Phoenix.Controller.render/3`, show both directions: which functions does it call internally, and which call sites/macros invoke it across the controller and view-rendering pipeline?" 3. **[D3 Retrieval]** "Retrieve the full source of `Phoenix.Endpoint.__using__/1`, the macro that injects endpoint behaviour when a module does `use Phoenix.Endpoint`." (grep-findable: `defmacro __using__(opts)` in `lib/phoenix/endpoint.ex`) 4. **[D4 Architecture]** "Describe the top-level structure of the `lib/phoenix/` tree — the major subsystems (router, controller, channel, socket, endpoint, test) and how the directories/modules are organized." (Note: Phoenix.PubSub lives in the separate `phoenix_pubsub` package, NOT in this repo's `lib/phoenix/` — do not expect a pubsub subsystem here.) 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring / semantic) Across the codebase, find the functions/macros responsible for compiling and matching routes (e.g. route building, path matching, dispatch) even when their names don't share an obvious substring — i.e. semantically related routing-dispatch logic spread across `Phoenix.Router` and helper modules." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="plug|pipe_through|Phoenix\\.Controller"); D2->trace_call_path(qualified_name="Phoenix.Controller.render", direction="both"); D3->get_code_snippet(qualified_name="Phoenix.Endpoint.__using__"); D4->get_architecture(path="lib/phoenix"); D5->search_code/semantic_query("route compilation and path matching dispatch"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/elixir-graph.md`, `elixir-explorer.md`, `elixir-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the elixir tier. --- ### 19. haskell — D (Functional & Formal) **Repo:** jgm/pandoc (`/tmp/bench/haskell`) **Symlink:** no **Indexed in:** fast **Why this repo:** ~38k-star universal document converter — the canonical large, idiomatic Haskell codebase (readers/writers, typeclasses, monad transformers), exercising the Haskell extractor on real-world module/type/function structure rather than toy code. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Where is the top-level conversion entry point `convertWithOpts` defined, and where is the core `writeMarkdown` writer function declared? Both are exported, grep-findable names — does search_graph surface their defining modules (Text.Pandoc.App and Text.Pandoc.Writers.Markdown)?" 2. **[D2 Relationship]** "Trace the call relationships around `readMarkdown` [verify]: what does the Markdown reader call into (e.g. parser combinators / `Text.Pandoc.Parsing` helpers), and what callers invoke it? Use direction=both so both the dispatch in the reader registry and downstream callees appear." 3. **[D3 Retrieval]** "Retrieve the exact source of the `writeHtml5` [verify] function from Text.Pandoc.Writers.HTML — return just that function body, not the whole module." 4. **[D4 Architecture]** "Describe pandoc's module organization: how are `Text.Pandoc.Readers.*` and `Text.Pandoc.Writers.*` grouped relative to the shared core (`Text.Pandoc.Definition`, `Text.Pandoc.Options`, `Text.Pandoc.Parsing`)? Does the structure view recover the reader/writer/core layering?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Across all writer modules, find the functions that serialize a `Block` / `Inline` AST node to output — i.e. the family of `blockToX` / `inlineToX` helpers (`blockToMarkdown`, `inlineToHtml`, …) that share a naming pattern and structural role across Markdown/HTML/LaTeX writers. This duplication-by-convention pattern is what semantic/pattern search should cluster and plain grep on one name cannot." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="convertWithOpts|writeMarkdown"); D2->trace_call_path(name="readMarkdown", direction="both"); D3->get_code_snippet(qualified_name="Text.Pandoc.Writers.HTML.writeHtml5"); D4->get_architecture(scope="Text.Pandoc"); D5->search_code/semantic_query(query="serialize Block or Inline AST node to output", pattern=".*(blockTo|inlineTo).*"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/haskell-graph.md`, `haskell-explorer.md`, `haskell-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the haskell tier. --- ### 20. ocaml — D (Functional & Formal) **Repo:** ocaml/dune (`/tmp/bench/ocaml`) **Symlink:** no **Indexed in:** fast **Why this repo:** dune is the de-facto OCaml build system (~4k+ stars, used by virtually every modern OCaml project); it is large, idiomatic, multi-package OCaml (functors, modules, .ml/.mli pairs), matching the plan's criterion of a substantial, popular, idiomatic in-language codebase. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `Dune_project` module and its constructor/loader function `Dune_project.load` [verify]; list its public API as declared in the corresponding `.mli` interface." (grep-findable: the identifiers `Dune_project` and `load` appear verbatim in `src/dune_rules/dune_project.ml`/`.mli`.) 2. **[D2 Relationship]** "Starting from `Build_system.build_file` [verify] (or the nearest real entry in `src/dune_engine/build_system.ml`), trace both inbound callers and outbound callees to show how a requested target flows from the engine into rule evaluation." 3. **[D3 Retrieval]** "Retrieve the full source of the `Action_builder.t` type and the `Action_builder.bind`/`map` combinators [verify] in `src/dune_engine/action_builder.ml` — return only that symbol, not the whole file." (grep-findable identifier: `Action_builder`.) 4. **[D4 Architecture]** "Produce the high-level architecture of dune: the top-level package/library split (`src/dune_engine`, `src/dune_rules`, `src/dune_lang`, `bin/`, `vendor/`) and how the CLI entry (`bin/main.ml`) depends on the engine and rules layers." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find code semantically related to 'memoized incremental build computation' — e.g. the `Memo` module and its `Memo.create`/`Memo.exec` usage [verify] — and surface other call sites that participate in the same memoization/incrementality concern across `src/dune_engine`." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*Dune_project.*", label="Module/Function"); D2->trace_call_path(qualified_name="Build_system.build_file", direction="both"); D3->get_code_snippet(qualified_name="Action_builder.t"); D4->get_architecture(project="ocaml-dune"); D5->search_code("memoized incremental build") / search_graph(semantic_query="memoization incremental computation"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/ocaml-graph.md`, `ocaml-explorer.md`, `ocaml-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the ocaml tier. --- ### 21. objc — A (Class-based OOP & Contracts) **Repo:** AFNetworking/AFNetworking (`/tmp/bench/objc`) **Symlink:** no **Indexed in:** fast **Why this repo:** One of the most-starred Objective-C projects on GitHub (~33k stars) and the canonical idiomatic iOS/macOS networking library — substantial class hierarchies, formal `@protocol` contracts, and category usage make it a faithful exemplar of Group A's class-based OOP + contracts theme. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the `AFHTTPSessionManager` class and list its public initializers/factory methods (e.g. `manager`, `initWithBaseURL:`, `initWithBaseURL:sessionConfiguration:`). Identifier is plain-text grep-findable as `@interface AFHTTPSessionManager`." 2. **[D2 Relationship]** "Show the inbound and outbound relationships of `AFHTTPSessionManager` — confirm it inherits from `AFURLSessionManager`, and trace what its `GET:parameters:headers:progress:success:failure:` / `dataTaskWithHTTPMethod:...` methods call internally (e.g. into the request serializer and the superclass task-creation path). [verify] exact convenience-method signature varies by AFNetworking version (3.x adds `progress:`)." 3. **[D3 Retrieval]** "Retrieve the full source of the `AFSecurityPolicy` instance method `-evaluateServerTrust:forDomain:` (grep-findable token `evaluateServerTrust:forDomain:`)." 4. **[D4 Architecture]** "Describe the module/file organization of the `AFNetworking/` source directory and how the serialization layer (`AFURLRequestSerialization`/`AFURLResponseSerialization`) sits relative to the session-management layer (`AFURLSessionManager`/`AFHTTPSessionManager`) and the security/reachability helpers." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Find all classes that conform to the `AFURLRequestSerialization` protocol via its required method `requestBySerializingRequest:withParameters:error:` (e.g. `AFHTTPRequestSerializer`, `AFJSONRequestSerializer`, `AFPropertyListRequestSerializer`) and surface other request/response serializer types that follow the same contract/naming pattern. Labelled graph-favoring: this is a protocol-conformance + naming-pattern query that plain grep can only approximate." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="AFHTTPSessionManager", label="Class"); D2->trace_call_path(qualified_name="AFHTTPSessionManager", direction="both"); D3->get_code_snippet(qualified_name="AFSecurityPolicy.evaluateServerTrust:forDomain:"); D4->get_architecture(scope="AFNetworking/"); D5->search_code/semantic_query("classes conforming to AFURLRequestSerialization / requestBySerializingRequest:withParameters:error:"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/objc-graph.md`, `objc-explorer.md`, `objc-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the objc tier. --- ### 22. swift — A (Class-based OOP & Contracts) **Repo:** Alamofire/Alamofire (`/tmp/bench/swift`) **Symlink:** no **Indexed in:** fast **Why this repo:** ~41k-star, the de-facto idiomatic Swift HTTP networking library — substantial protocol-oriented + class-based design (Session/Request hierarchy, interceptor/serializer protocols), matching the plan's "popular, idiomatic, substantial" repo-selection criteria for Group A. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the public type `RequestInterceptor` and the two protocols it composes — `RequestAdapter` and `RequestRetrier`. Report each one's kind (protocol/typealias) and declaring file." 2. **[D2 Relationship]** "For `DataRequest`, show the call relationship both ways: which methods/initializers construct or return a `DataRequest` (inbound, e.g. via `Session.request`) and which serializer/response APIs it invokes (outbound, e.g. `response(...)` / `responseDecodable(...)`)." 3. **[D3 Retrieval]** "Retrieve the full source of the `AFError` enum definition (its cases and nested error types)." 4. **[D4 Architecture]** "Describe the module's structure: how the `Source/` tree is organized (Core, Features, Extensions, etc.) and where the `Session`, `Request` family, and `*Serializer` types live relative to one another." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the response-serialization concepts across the codebase — symbols semantically related to turning raw responses into typed values (e.g. `ResponseSerializer`, `DataResponseSerializerProtocol`, `DecodableResponseSerializer`, `responseDecodable`) — and group the conforming serializer implementations even when names differ. Label: semantic/similarity query, expected to favor the graph over plain grep." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="RequestInterceptor|RequestAdapter|RequestRetrier", label="Protocol"); D2->trace_call_path(qualified_name="...DataRequest", direction="both"); D3->get_code_snippet(qualified_name="Alamofire.AFError"); D4->get_architecture(project="swift"); D5->search_code/semantic_query("response serialization to typed value"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/swift-graph.md`, `swift-explorer.md`, `swift-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the swift tier. --- ### 23. dart — A (Class-based OOP & Contracts) **Repo:** felangel/bloc (`/tmp/bench/dart`) **Symlink:** no **Indexed in:** fast **Why this repo:** ~12k-star, canonical Dart state-management monorepo whose idiomatic class/mixin/abstract-interface hierarchy (Bloc, Cubit, BlocBase, observers) makes it a substantial, popular Group-A exemplar per the plan's popularity-plus-idiomaticity selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definition of the abstract class `Bloc` and its companion base `BlocBase` in `packages/bloc/lib/src/`, and list their public API surface (e.g. `add`, `on`, `emit`, `close`, `state`, `stream`, `isClosed`)." 2. **[D2 Relationship]** "Map the relationships of `Cubit`: what does it extend (`BlocBase`), which members it inherits vs. defines (it inherits `emit`/`onChange` from `BlocBase` and adds no overrides beyond its constructor — note that it is `Bloc`, not `Cubit`, that overrides `emit`), and who calls `BlocBase.emit` / `BlocBase.onChange` — show callers and callees in both directions." 3. **[D3 Retrieval]** "Retrieve the full source of the `BlocObserver` class (the lifecycle hooks `onCreate`, `onEvent`, `onChange`, `onTransition`, `onError`, `onClose`)." 4. **[D4 Architecture]** "Describe the architecture of the `bloc` monorepo: the `packages/` layout (bloc, flutter_bloc, hydrated_bloc, replay_bloc, bloc_test, bloc_concurrency) and, within `packages/bloc/lib/src/`, the class/mixin/abstract-interface hierarchy rooted at `BlocBase` (`Streamable`, `StateStreamable`, `Closable`, `Emittable`, `ErrorSink`)." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Locate the event-transformer concept across the codebase: the `EventTransformer`/`EventMapper` typedefs in `packages/bloc` and their concurrency implementations (`concurrent`, `sequential`, `droppable`, `restartable`) in `packages/bloc_concurrency` — surface the semantic cluster and the typedef<->implementation links, not just exact-name text matches." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="Bloc|BlocBase", label="Class"); D2->trace_call_path(qualified_name="...BlocBase.emit", direction="both"); D3->get_code_snippet(qualified_name="...BlocObserver"); D4->get_architecture(scope="packages/bloc"); D5->search_code/semantic_query("event transformer concurrency droppable restartable"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/dart-graph.md`, `dart-explorer.md`, `dart-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the dart tier. --- ### 24. perl — C (Dynamic & Scripting) **Repo:** mojolicious/mojo (`/tmp/bench/perl`) **Symlink:** no **Indexed in:** fast **Why this repo:** Mojolicious is the most-starred idiomatic Perl web framework (~2.7k stars), large and heavily OO via `Mojo::Base`/`has` accessors — a substantial, real-world Perl codebase that matches the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definition of the `Mojo::URL` package and its accessor/method `path` — where is the `package Mojo::URL;` declared and where is `sub path` (or its `has` accessor) defined?" 2. **[D2 Relationship]** "Map the relationships around `Mojolicious::Routes::Route::to`: what calls it (inbound) and what does it call (outbound), tracing the route-dispatch chain in both directions?" 3. **[D3 Retrieval]** "Retrieve the full source of the `dispatch` method in `Mojolicious::Routes` (i.e. `sub dispatch`) exactly as written, with its precise line boundaries." 4. **[D4 Architecture]** "Show the architecture of the distribution: how is `lib/Mojo/` (core toolkit) organized versus `lib/Mojolicious/` (the framework), and which directories hold the server, transaction, and DOM subsystems?" 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Semantically locate all code that builds or parses HTTP transactions/requests (e.g. `Mojo::Transaction`, `Mojo::Message::Request`, `Mojo::UserAgent` build paths) — surface conceptually-related request/response construction logic even when identifiers differ. (Graph-favoring: relies on semantic similarity, not a single grep token.)" **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="Mojo::URL|path"); D2->trace_call_path(qualified_name="Mojolicious::Routes::Route::to", direction="both"); D3->get_code_snippet(qualified_name="Mojolicious::Routes::dispatch"); D4->get_architecture(scope="lib/"); D5->search_code/semantic_query("build or parse HTTP request/response transaction"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/perl-graph.md`, `perl-explorer.md`, `perl-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the perl tier. --- ### 25. groovy — A (Class-based OOP & Contracts) **Repo:** spockframework/spock (`/tmp/bench/groovy`) **Symlink:** no **Indexed in:** fast **Why this repo:** Spock is the canonical, widely-adopted (~3.6k stars) Groovy BDD testing framework — idiomatic, substantial Groovy with a deep class hierarchy, traits, and AST transforms, satisfying the plan's "popular + idiomatic + non-trivial size" repo-selection criteria for Group A (Class-based OOP & Contracts). **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the public API surface a user extends to write specifications: locate the `Specification` base class (in `spock.lang`) and the `MockingApi` class it derives from in `spock-core`, and report the factory method declarations `Mock`, `Stub`, and `Spy` exposed for test authors." (All grep-findable identifiers: `class Specification`, `class MockingApi`, `public T Mock(`.) 2. **[D2 Relationship]** "Map the relationships around `Specification`: what is its inheritance chain (`Specification` extends `MockingApi`, which in turn derives from the internal `SpecInternals` base) and which callers/subclasses reference the `setupSpec`/`cleanupSpec` lifecycle methods? Note that `SpecificationContext` is a runtime collaborator referenced by the spec, not a superclass. Use direction=both to capture both ancestors and the fixture-method call sites." 3. **[D3 Retrieval]** "Retrieve the full source of the `IMockController` interface in `org.spockframework.mock` so a reviewer can read the exact mock-interaction contract it declares (`addInteraction`, `enterScope`, `leaveScope`, etc.)." (Grep-findable as `interface IMockController`.) 4. **[D4 Architecture]** "Describe the module/package architecture: how is the repo split across `spock-core`, `spock-spring`, `spock-junit4`, and `spock-bom`, and within `spock-core` how do the `org.spockframework.runtime`, `org.spockframework.mock`, and `spock.lang` packages divide responsibility (runtime engine vs mocking vs user-facing DSL)?" 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Semantically locate the AST-transformation machinery that rewrites spec source at compile time — find the `SpockTransform` global transform entry point and the spread of rewriting classes (`SpecRewriter`, `WhereBlockRewriter`, `DeepBlockRewriter`, `InteractionRewriter`, `SpecAnnotator`) in `org.spockframework.compiler` — using semantic_query/search_code, since this machinery is spread across the compiler package by behavior, not by a single shared name token." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*Specification.*|.*MockingApi.*", label="Class"); D2->trace_call_path(qualified_name="spock.lang.Specification", direction="both"); D3->get_code_snippet(qualified_name="org.spockframework.mock.IMockController"); D4->get_architecture(project="groovy"); D5->search_code/semantic_query("compile-time AST transformation rewriting specification blocks"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/groovy-graph.md`, `groovy-explorer.md`, `groovy-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group A only, Group A, the groovy tier. --- ### 26. erlang — D (Functional & Formal) **Repo:** ninenines/cowboy (`/tmp/bench/erlang`) **Symlink:** no **Indexed in:** fast **Why this repo:** Cowboy is the de-facto small, fast, modern HTTP server for Erlang/OTP (~7k GitHub stars, used across the BEAM ecosystem); its idiomatic OTP module/behaviour layout makes it a substantial, representative Erlang target per the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the public entry-point functions that start an HTTP/HTTPS listener — locate `cowboy:start_clear/3` and `cowboy:start_tls/3` in `src/cowboy.erl`. Are both arity-3 functions surfaced as definitions with their module qualifier?" 2. **[D2 Relationship]** "Starting from `cowboy_router:execute/2`, show the relationship neighborhood in both directions: what it calls (e.g. `cowboy_router:match/3` [verify]) and which callers reach it (the stream/handler pipeline). Does the call graph connect the router to `cowboy_handler:execute/2` [verify]?" 3. **[D3 Retrieval]** "Retrieve the full source of the single function `cowboy_router:compile/1` from `src/cowboy_router.erl` — the routes-compilation function — with exact clause boundaries." 4. **[D4 Architecture]** "Describe the top-level architecture: the `src/` module layout and how the protocol modules (`cowboy_http`, `cowboy_http2`, `cowboy_websocket`) relate to the transport/stream layer (`cowboy_clear`, `cowboy_tls`, `cowboy_stream`). Is the module-per-concern OTP structure visible?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the functions implementing HTTP response writing across protocol versions — semantically locate the `reply`/`stream_reply` paths in `cowboy_req` and how each protocol module fulfils the `cowboy_stream` behaviour callbacks. This favors semantic/behaviour-callback linking over a single literal grep term." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="cowboy:start_(clear|tls)"); D2->trace_path(function_name="cowboy_router:execute/2", direction="both"); D3->get_code_snippet(qualified_name="cowboy_router:compile/1"); D4->get_architecture (inspect src/ module clusters); D5->search_graph(semantic_query=["reply","stream_reply","cowboy_stream","behaviour"]) or search_code (HTTP reply / stream_reply across protocol versions; cowboy_stream behaviour callbacks). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/erlang-graph.md`, `erlang-explorer.md`, `erlang-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the erlang tier. --- ### 27. r — C (Dynamic & Scripting) **Repo:** tidyverse/dplyr (`/tmp/bench/r`) **Symlink:** no **Indexed in:** fast **Why this repo:** dplyr is the canonical, high-star tidyverse data-manipulation package and the most idiomatic large-scale R codebase (verb-based S3/generic dispatch under `R/`), matching the plan's "popular + substantial + idiomatic" repo-selection criteria for the r tier. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the definitions of the core dplyr verbs `mutate`, `filter`, `summarise`, `arrange`, and `select` in `R/` — are they captured as top-level function definitions (generics), and where is each defined?" 2. **[D2 Relationship]** "Show the call relationships for `summarise` (direction=both): which internal helpers it calls (e.g. `summarise_cols` [verify], `dplyr_col_modify`) and which exported functions reach it, distinguishing the generic from its `summarise.data.frame` method." 3. **[D3 Retrieval]** "Retrieve the full source of the `mutate.data.frame` method (the data-frame S3 method for `mutate`) exactly as defined in `R/mutate.R`." 4. **[D4 Architecture]** "Describe the structure of the `R/` source tree: how are the verb files (`mutate.R`, `filter.R`, `summarise.R`, `join.R`, `group-by.R`) organized, and how does the R layer relate to the `src/` C++ backend it wraps?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the family of functions implementing the tidy-select / column-selection semantics (e.g. `select`, `relocate`, `rename`, `pull` and shared `tidyselect`-style helpers) by semantic similarity rather than literal name match, and surface near-duplicate verb scaffolding across the `*.R` verb files." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="mutate|filter|summarise|arrange|select", label="Function"); D2->trace_call_path(qualified_name="summarise", direction="both"); D3->get_code_snippet(qualified_name="mutate.data.frame"); D4->get_architecture(path="R/"); D5->search_code(semantic_query="column selection / tidy-select verb helpers") / search_graph(semantic_query=...). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/r-graph.md`, `r-explorer.md`, `r-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the r tier. --- ### 28. html — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** expressjs/express (symlink javascript) (`/tmp/bench/html`) **Symlink:** yes **Indexed in:** fast **Why this repo:** Express is the most-depended-on Node web framework (60k+ stars, tens of millions of weekly installs). Its `examples/ejs/` tree ships real `.html` template files (`header.html`, `footer.html`, `users.html`) that are wired through `app.set('view engine', 'html')` + `app.engine('.html', ejs.__express)`, giving a genuine, idiomatic HTML/markup corpus — matching the plan's "popular + idiomatic + substantial" repo-selection criteria. (Note: express is HTML-light overall; the markup lives almost entirely under `examples/*/views/`, which the questions below stay scoped to.) **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level HTML document scaffolds declared in this repo — i.e. the files that open a full document with `` / `` / `` such as `examples/ejs/views/header.html`, plus the inline HTML the framework emits itself for `res.redirect` in `lib/response.js` (the `<!DOCTYPE html>…<body><p>…</p></body>` redirect body). Enumerate each with its file. (Grep-findable: these are literal `<html`/`<!DOCTYPE` tags in the source.)" 2. **[D2 Relationship]** "Trace the include/render chain for the EJS-as-HTML example views: which partials does `examples/ejs/views/users.html` pull in via the EJS `include(...)` directive (it includes `header.html` and `footer.html`), and which route handler renders it — i.e. the `res.render('users', …)` call in `examples/ejs/index.js`?" 3. **[D3 Retrieval]** "Retrieve the single largest complete HTML document definition in the repo — the full markup of `examples/ejs/views/header.html`, the only template that declares the whole `<!DOCTYPE html><html><head><title><%= title %>…` scaffold (the rest, e.g. `users.html` / `footer.html`, are fragments). Return it verbatim with its boundaries. (Grep-findable: locate it by grepping for ``.)" 4. **[D4 Architecture]** "Describe how HTML/template assets are organized across the repo: the per-example `views/` directory convention under `examples/` (e.g. `examples/ejs/views/`, `examples/auth/views/`), the separation of templates from route code in each example's `index.js`, and where framework-emitted markup lives (the inline redirect-body HTML in `lib/response.js`) versus user-facing example markup under `examples/*/views/`." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find duplicated / near-duplicate markup and the config<->markup linkage: which example view files share the same boilerplate document scaffold or fragment pattern across `examples/*/views/` (e.g. the `header`/`footer` split repeated in the `ejs` and `auth` examples [verify]), and which template files are bound to a configured view engine via the `app.set('view engine', …)` / `app.engine('.html', …)` calls in the corresponding example `index.js`? Label: graph-favoring — duplication clustering plus config-call-to-template-file linkage are not reachable by a single grep." **Expected graph tools (hint, not a script):** D1->search_code/grep for ``/`trace_call_path(direction=both, on the `res.render` handler in `examples/ejs/index.js`) plus the EJS `include` edges between view files; D3->get_code_snippet(qualified_name="examples/ejs/views/header.html"); D4->get_architecture(scope="examples/", views/ layout + lib/ framework markup); D5->search_code/semantic_query("shared document scaffold") + config-view-engine-to-template linkage. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/html-graph.md`, `html-explorer.md`, `html-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the html tier. --- ### 29. css — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** shadcn-ui/ui (symlink tsx) (`/tmp/bench/css`) **Symlink:** yes **Indexed in:** fast **Why this repo:** One of the most-starred React component ecosystems on GitHub; its design-token CSS (`globals.css` custom-property themes + `@layer` / `@tailwind` directives) is idiomatic, substantial, real-world CSS, satisfying the plan's "popular + idiomatic + non-trivial size" repo-selection criteria. > **Indexer note (fairness):** the CBM indexer's CSS spec extracts **no definition nodes** — no selectors, no custom-property declarations — only `@import` edges and the stylesheet module. There is **no `var()` define→use edge** and **no qualified name** for a selector block. Questions below are authored honestly around this: the structural dimensions are expected to be **grep-favoring or N/A**, not graph wins. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List every top-level selector block in `apps/www/styles/globals.css` [verify] — specifically the `:root` and `.dark` blocks that declare design tokens such as `--background`, `--foreground`, `--primary`, `--radius`. (Symmetric: these selectors are equally findable by `grep -nE ':root|\.dark'`. CSS yields no graph Definition nodes, so the graph is expected to do no better than — and likely worse than — plain grep here.)" 2. **[D2 Relationship]** "**N/A.** CSS has no call/reference graph in the indexer: `var(--border)` / `var(--ring)` usages are not modeled as define→use edges, and selector blocks are not nodes. There is no faithful relationship/trace question to ask; forcing one would test a capability the language extraction does not have. A `var()` cross-reference question would be a grep/text task, not a graph-relationship task." 3. **[D3 Retrieval]** "Retrieve the full body of the `:root` token declaration block in `apps/www/styles/globals.css` [verify], exactly as written, including every `--*` custom-property line. (Symmetric: the same block is retrievable by `grep -nA40 ':root' globals.css` or a plain file read. Note CSS exposes no qualified-name snippet target, so `get_code_snippet` has no symbol to resolve — both engines fall back to file/line retrieval.)" 4. **[D4 Architecture]** "Describe the stylesheet file/directory organization: where the global theme CSS lives (`apps/www/styles/` [verify]) versus per-registry / per-component theme CSS, and how the `@tailwind base/components/utilities` + `@layer base` structure partitions the single global stylesheet." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Surface the duplication and naming convention across the theme: the `:root` (light) and `.dark` blocks define a near-mirrored set of `--*` properties — identify the parallel/duplicated token names, and any config↔code link between these CSS custom properties and the Tailwind `theme.extend.colors` mapping that references `hsl(var(--...))`. Openly graph/semantic-favoring (mirrored-block similarity + token↔config linkage via semantic retrieval, not exact-string grep)." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=":root|\\.dark") and search_code (expected: little/no graph structure for CSS; grep parity is the honest baseline); D2->N/A (no relationship edges for CSS); D3->search_code / file retrieval for the `:root` block (no qualified-name snippet target); D4->get_architecture(scope="styles/"); D5->search_code(semantic_query="mirrored light/dark token blocks; hsl(var(--..)) Tailwind mapping"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/css-graph.md`, `css-explorer.md`, `css-judged.json`. **Aggregates into:** D1/D3/D4 cross-group rollups (D2 excluded as N/A), D5 within Group E only, Group E, the css tier. --- ### 30. scss — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** twbs/bootstrap (`/tmp/bench/scss`) **Symlink:** no **Indexed in:** fast **Why this repo:** ~170k-star, the canonical large SCSS codebase — deep mixin/function/variable layering and a fully `@import`/`@use`-driven module graph make it the most idiomatic, substantial SCSS target for the plan's "popular + representative" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List representative SCSS mixins and functions defined in Bootstrap's `scss/` tree — e.g. the `button-variant` mixin (in `scss/mixins/_buttons.scss`), the `make-col` mixin (in `scss/mixins/_grid.scss`), and the `color-contrast` function (in `scss/_functions.scss`) — and report the file each is declared in." (all grep-findable via the `@mixin`/`@function` keywords) 2. **[D2 Relationship]** "Trace the cross-file `@import`/`@use` graph rooted at `scss/bootstrap.scss`: which partials does it pull in (e.g. `_variables`, `_mixins`, `_buttons`), and which of those partials in turn reference the `_functions` and `_variables` partials? Show inbound + outbound include edges." 3. **[D3 Retrieval]** "Retrieve the full definition of the `button-variant` mixin from `scss/mixins/_buttons.scss`." (one real, grep-findable symbol) 4. **[D4 Architecture]** "Describe the `scss/` tree organization: the top-level partial layer (`_root`, `_reboot`, `_variables`, `_maps`, `_utilities`) vs. the `scss/mixins/`, `scss/helpers/`, `scss/forms/`, `scss/vendor/`, and `scss/utilities/` subdirectories, and how `bootstrap.scss` vs. `bootstrap-utilities.scss` / `bootstrap-grid.scss` entry points compose them." 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Find duplication and naming-pattern families across the partials: e.g. all `*-variant` mixins (`button-variant`, `button-outline-variant`, `alert-variant` [verify], `list-group-item-variant` [verify]) and the `$enable-*` boolean feature-flag variables in `_variables.scss`; then link each `$enable-*` flag to the partials whose output it gates. (Config<->code linkage + similarity — favors the graph/semantic index over plain grep.)" **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*(button-variant|make-col|color-contrast).*"); D2->trace_call_path(qualified_name=".../bootstrap.scss", direction="both", edge="IMPORTS"); D3->get_code_snippet(qualified_name="scss/mixins/_buttons.scss::button-variant"); D4->get_architecture(path="scss/"); D5->search_code/semantic_query("*-variant mixins and $enable-* feature flags"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/scss-graph.md`, `scss-explorer.md`, `scss-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the scss tier. --- ### 31. yaml — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** kubernetes/examples (`/tmp/bench/yaml`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical Kubernetes examples repo is high-popularity and is overwhelmingly idiomatic, substantial YAML (Deployments, Services, PVs, ConfigMaps across dozens of self-contained demos), matching the plan's criterion of a widely-used, language-representative corpus. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find all top-level Kubernetes resource definitions of `kind: Service` across the repo and list the manifest files that declare them (e.g. the `frontend` Service in the guestbook example). [verify]" 2. **[D2 Relationship]** "N/A — YAML is a config/data language: the knowledge graph models manifests as documents/keys, not as symbols with call or reference edges. The relationship a reviewer would want here (a Service's `spec.selector` linking to a workload's `template.labels`) is Kubernetes-semantic, not a generic code-graph edge, so there is no `CALLS`/`HANDLES`/`IMPLEMENTS` relationship for the graph to traverse. Forcing a `trace_call_path`-style question would be unnatural and would not exercise a real graph capability." 3. **[D3 Retrieval]** "Retrieve the full manifest definition for the `cassandra` StatefulSet (the largest workload definition in the cassandra example), including its `volumeClaimTemplates`. [verify]" 4. **[D4 Architecture]** "Describe the directory/file organization of the repo: how each example (guestbook, cassandra, mysql-wordpress-pv, storage/*) groups its YAML manifests, and which top-level folders contain the most manifest files." 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Across all examples, identify duplicated or near-duplicate manifest patterns and naming conventions — e.g. the recurring `app:`/`tier:` label scheme, repeated `PersistentVolumeClaim` blocks, and Services that share selector labels with multiple workloads — to surface config duplication and label<->workload links a plain text scan would miss. (Openly graph/semantic-favoring.)" **Expected graph tools (hint, not a script):** D1->search_graph(label/kind="Service", name_pattern=".*"); D2->N/A (no relationship edges between YAML manifest keys in a generic code graph; see D2 note); D3->get_code_snippet(qualified_name="...cassandra (StatefulSet)"); D4->get_architecture(project="yaml"); D5->search_code / search_graph(semantic_query="duplicated label selector / PVC patterns"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/yaml-graph.md`, `yaml-explorer.md`, `yaml-judged.json`. **Aggregates into:** D1-D4 cross-group rollups (D2 counted as N/A for yaml), D5 within Group E only, Group E, the yaml tier. --- ### 32. toml — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** meilisearch/meilisearch (symlink rust) (`/tmp/bench/toml`) **Symlink:** yes **Indexed in:** fast **Why this repo:** A ~50k-star, production Rust search engine whose Cargo workspace carries a large, idiomatic TOML surface (root workspace manifest, ~10 per-crate `Cargo.toml`, `rust-toolchain.toml`, build profiles) — matching the plan's "popular + substantial + idiomatic for the language" repo-selection criterion; the symlink reuses the already-cloned rust repo. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level TOML tables and key arrays declared in the workspace root `Cargo.toml` — specifically the `[workspace]` table, its `members` array, and the `[profile.release]` table — and report the file each is defined in." (All grep-findable: `^\[workspace\]`, `members =`, `^\[profile.release\]`.) 2. **[D2 Relationship]** "Trace the cross-file reference structure: which directories named in the root `Cargo.toml` `[workspace] members` array contain their own `Cargo.toml` manifest (e.g. `meilisearch`, `meilisearch-types`, `milli` [verify], `index-scheduler` [verify]), and which per-crate manifests reference the shared `[workspace.package]` via `workspace = true` inheritance?" 3. **[D3 Retrieval]** "Retrieve the full `[dependencies]` table from the `meilisearch` crate's `Cargo.toml` (the binary crate) — the single largest dependency block in the workspace — verbatim with all version and feature specifiers." 4. **[D4 Architecture]** "Describe the TOML file/directory organization of the Cargo workspace: the root workspace manifest at `Cargo.toml`, the per-crate manifests under each member directory, the toolchain pin in `rust-toolchain.toml` [verify], and how `[profile.*]` build tuning is centralized at the root." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Find duplication and config↔code links: which dependencies are declared in multiple crate `Cargo.toml` files (candidates for `[workspace.dependencies]` hoisting), and which crate directories named in the root `members` array map to a real Rust source tree (`src/lib.rs`/`src/main.rs`) versus a dangling/unbuilt entry?" **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*workspace.*|.*profile.*", project="toml"); D2->trace_call_path(direction="both") over manifest→member references (fallback: query_graph on CONTAINS_FILE / reference edges since TOML has no call graph); D3->get_code_snippet(qualified_name="meilisearch/Cargo.toml::dependencies" [verify]); D4->get_architecture(project="toml"); D5->search_code/semantic_query for repeated dependency keys + config-to-source mapping. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/toml-graph.md`, `toml-explorer.md`, `toml-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the toml tier. --- ### 33. hcl — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** terraform-aws-modules/terraform-aws-eks (`/tmp/bench/hcl`) **Symlink:** no **Indexed in:** fast **Why this repo:** The most-used community Terraform EKS module (thousands of GitHub stars, millions of registry pulls); large, idiomatic, multi-submodule HCL that exercises Group E's config/IaC discovery criteria far better than a toy `.tf` fixture. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level Terraform resource and module blocks declared in the root module — e.g. the `aws_eks_cluster.this` resource and the `aws_iam_role.this` / `aws_security_group.cluster` definitions — and identify which file each is declared in." (grep-findable: every block header is plain text, e.g. `resource "aws_eks_cluster" "this"`.) 2. **[D2 Relationship]** "Show the cross-file reference graph for the cluster-name input variable (`var.cluster_name` [verify] — renamed to `var.name` on current `master`; check the pinned ref's `variables.tf`): which resources, locals, outputs, and submodule `source = "./modules/..."` invocations consume it, both inbound and outbound?" 3. **[D3 Retrieval]** "Retrieve the full definition of the `resource \"aws_eks_cluster\" \"this\"` block in the root `main.tf` — the single largest core resource in the module." (real, grep-findable symbol.) 4. **[D4 Architecture]** "Describe the file/directory organization of this module: the root `*.tf` files (`main.tf`, `variables.tf`, `outputs.tf`, `versions.tf`) versus the submodule tree under `modules/` (e.g. `eks-managed-node-group`, `self-managed-node-group`, `fargate-profile`, `karpenter`), and how the root composes the submodules." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Find duplicated / near-duplicate IAM and security-group definitions across the root module and the node-group submodules (e.g. repeated `aws_iam_role` + `aws_iam_role_policy_attachment` patterns), and surface the config<->config naming convention (`this`, `_` resource names) that ties them together — the kind of similarity/duplication link grep cannot rank." **Expected graph tools (hint, not a script):** D1->search_graph(label="resource"|"module", name_pattern=".*aws_eks_cluster.*|.*aws_iam_role.*"); D2->trace_path(symbol="cluster_name" [verify: or "name" on master], direction=both); D3->get_code_snippet(qualified_name="aws_eks_cluster.this"); D4->get_architecture(scope="modules/"); D5->search_code/semantic_query("duplicate iam role security group node group"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/hcl-graph.md`, `hcl-explorer.md`, `hcl-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the hcl tier. --- ### 34. sql — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** dbt-labs/jaffle_shop (`/tmp/bench/sql`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical dbt demo project — the most widely cloned, idiomatic example of analytics-engineering SQL (staging→marts models wired by Jinja `ref()`), small enough to index fast yet structurally representative of how real SQL/dbt repos express cross-file dependencies. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level model definitions in this dbt project — confirm that the mart models `customers` (`models/customers.sql`) and `orders` (`models/orders.sql`) and the staging models `stg_customers`, `stg_orders`, `stg_payments` (`models/staging/`) are each surfaced as defined SQL units." (grep-findable: each is a `.sql` file whose basename is the model name.) 2. **[D2 Relationship]** "Show the cross-file reference graph for the `orders` model: which staging models does it pull from via `{{ ref(...) }}` (expect `stg_orders` and `stg_payments`), and — listing them or stating 'none' — which other models reference those same staging models?" 3. **[D3 Retrieval]** "Retrieve the full body of the single largest model definition, `orders` (`models/orders.sql`), including its payment-method pivot (`sum(case when payment_method = 'credit_card' then amount else 0 end) as credit_card_amount`, plus `coupon`/`bank_transfer`/`gift_card` branches) and the final `order_payments`→`final` CTE join." (grep-findable: `payment_method`, `credit_card_amount`.) 4. **[D4 Architecture]** "Describe the file/directory organization of the project: the `models/` root holding the two mart models, the `models/staging/` subtree holding the `stg_*` models, the seed CSVs under `data/` (e.g. `raw_customers.csv`), and the `dbt_project.yml` config at the root — and how the staging→marts layering is reflected in that layout." 5. **[D5 Cross-cutting/Semantic]** "(GRAPH-FAVORING) Surface the config↔code and duplication links a text search misses: connect `dbt_project.yml`'s `models:` materialization config and the `{{ ref('stg_*') }}` calls to the actual `stg_*.sql` files, and flag the repeated payment-method `sum(case when ...)` pivot pattern shared across the codebase. Label: semantic / config-to-code linkage — graph-favoring." [verify: exact `dbt_project.yml` model-config keys] **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*(customers|orders|stg_).*"); D2->trace_path(function_name="orders", mode="calls", direction="both"); D3->get_code_snippet(qualified_name="orders"); D4->get_architecture(project="sql"); D5->search_code(pattern="payment_method case when") / search_graph(semantic_query=["payment","method","pivot","ref","config"]). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/sql-graph.md`, `sql-explorer.md`, `sql-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the sql tier. --- ### 35. dockerfile — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** docker-library/official-images (`/tmp/bench/dockerfile`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical, highly-starred registry of Docker official images — substantial and idiomatic Dockerfile/build content (test-fixture Dockerfiles plus a build-orchestration toolchain), matching the plan's "popular + idiomatic + substantial" repo-selection criterion for Group E build languages. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level build instructions (the `FROM`, `RUN`, `ENV`, `ENTRYPOINT`/`CMD` directives) declared in the test fixture Dockerfile `test/tests/utc/Dockerfile`, and name the base image referenced by its `FROM` stanza." (Grep-findable: `FROM`, `RUN`, `ENV`, `CMD` are literal keyword tokens; the fixture path is a stable, greppable identifier. [verify] exact fixture name) 2. **[D2 Relationship]** "Show the cross-file reference chain from the `test/config.sh` test-registry definition to the individual test directories it includes — i.e. which test entries (`testAlias`/`globalTests` entries such as `utc`, `no-hard-coded-passwords`) are wired in, and which `Dockerfile` and `run.sh` each pulls in." (Structural include/reference framing across `config.sh` → `test/tests//`.) 3. **[D3 Retrieval]** "Retrieve the full contents of the largest single build-test definition: the orchestration script `test/run.sh` (the harness that builds each image and runs its per-test `run.sh`)." (One real, grep-findable named file.) 4. **[D4 Architecture]** "Describe the directory/file organization of the repository: the relationship between the manifest `library/` entries, the `bashbrew/` tooling, and the `test/` harness tree (`test/config.sh`, `test/tests/`, `test/tests//{Dockerfile,run.sh,expected-std-out.txt}`)." (Structural / get_architecture framing.) 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Across all fixture Dockerfiles under `test/tests/`, find duplicated/near-duplicate build patterns (e.g. repeated `FROM debian`/`FROM alpine` base selections and the recurring `run.sh`+`expected-std-out.txt` pairing convention), and surface the config<->code link between a `library/` manifest's `Directory:` field and the actual build context it names. Explicitly graph-favoring: relies on similarity/duplication detection and manifest-to-build-context linkage that plain text search cannot rank." **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern="FROM|RUN|ENV|CMD", project="dockerfile"); D2->trace_path(direction=both, from="test/config.sh"); D3->get_code_snippet(qualified_name="test/run.sh"); D4->get_architecture(project="dockerfile"); D5->search_code(semantic_query="duplicate base-image FROM patterns across test fixtures") / search_graph(semantic_query="manifest Directory field to build context"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/dockerfile-graph.md`, `dockerfile-explorer.md`, `dockerfile-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the dockerfile tier. --- ### 36. clojure — D (Functional & Formal) **Repo:** clojure/clojure (`/tmp/bench/clojure`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical Clojure language implementation — the most-starred, most idiomatic and substantial Clojure codebase in existence (a self-hosting Lisp whose `core.clj` defines the language in itself), satisfying the plan's "popular + idiomatic + substantial" repo-selection criteria for Group D. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the public definitions in `clojure.set` (`src/clj/clojure/set.clj`) — confirm the tool surfaces `union`, `intersection`, `difference`, `select`, `project`, `rename`, and `join`. Are `defn`/`defn-` forms recognized as defined symbols, and are private (`defn-`) defs distinguished from public ones?" 2. **[D2 Relationship]** "For `clojure.core/merge-with` (`src/clj/clojure/core.clj`), show callers and callees in both directions — e.g. that it sits over `reduce`/`assoc`/`get` and is itself used by higher-level seq/map helpers. Does the call graph resolve cross-form references inside a single namespace?" 3. **[D3 Retrieval]** "Retrieve the full source of the `clojure.core/lazy-seq` macro from `src/clj/clojure/core.clj`, with exact start/end lines including its docstring and the surrounding `defmacro` form." 4. **[D4 Architecture]** "Describe the top-level architecture: the split between the Clojure layer (`src/clj/clojure/*.clj`: `core.clj`, `set.clj`, `string.clj`, `walk.clj`, `zip.clj`, `pprint/`) and the Java runtime layer (`src/jvm/clojure/lang/*.java`: `RT.java`, `Compiler.java`, `PersistentVector.java`, `Var.java`, `Namespace.java`). How does the tool present the two-language bootstrap structure?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the persistent immutable collection implementations regardless of name — locate the family of `Persistent*`/`Transient*` classes in `src/jvm/clojure/lang/` (e.g. `PersistentVector`, `PersistentHashMap`, `PersistentTreeMap`, `TransientVector`) and the interfaces they share (`IPersistentCollection`, `IEditableCollection`). This rewards semantic/structural grouping over a single literal grep." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern in clojure.set, label=Definition); D2->trace_call_path(qualified_name="clojure.core/merge-with", direction="both"); D3->get_code_snippet(qualified_name="clojure.core/lazy-seq"); D4->get_architecture(scope=repo); D5->search_code/semantic_query("persistent immutable collection implementation") + search_graph(name_pattern=".*Persistent.*"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/clojure-graph.md`, `clojure-explorer.md`, `clojure-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the clojure tier. --- ### 37. fsharp — D (Functional & Formal) **Repo:** giraffe-fsharp/Giraffe (`/tmp/bench/fsharp`) **Symlink:** no **Indexed in:** fast **Why this repo:** Giraffe is the most-starred idiomatic F# web framework (functional, combinator-based `HttpHandler` pipeline), substantial and pure-F#, matching the plan's "popular + idiomatic + non-trivial size" repo-selection criteria for Group D. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the definition of the core `HttpHandler` type and the `HttpFunc` type alias, plus the public handler combinators `choose`, `route`, and `routef` — where are they declared and what are their signatures?" (all plain identifiers: `HttpHandler`/`HttpFunc`/`choose` in `Core.fs`, `route`/`routef` in `Routing.fs` — grep-findable too) 2. **[D2 Relationship]** "Trace the call relationships (both directions) around the `>=>` compose operator and `compose` in `Giraffe.Core`: which handlers/combinators invoke it, and what does it call internally to chain two `HttpHandler`s?" 3. **[D3 Retrieval]** "Retrieve the full source of the `routef` function in `Routing.fs` (the typed-format route handler)." (`routef` is a plain function name — grep-findable too) 4. **[D4 Architecture]** "Describe the module/file organization of the `src/Giraffe` project — how are concerns split across the source files, e.g. `Core` (handlers + response writers), `Routing`/`EndpointRouting`, `Negotiation`, `Streaming`, and the model-binding files (`ModelParser`, `ModelValidation`, `FormatExpressions`)?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find all response-writing combinators semantically similar to `text`/`json` — i.e. handlers that set a body and content-type and return an updated `HttpContext` — and surface the shared `HttpHandler` shape they all follow. Note these live alongside `text`/`json` in `Core.fs` plus the content-negotiating handlers in `Negotiation.fs`; a semantic query groups them by behaviour where plain text search (scanning one file for one keyword) would miss the cross-file set." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*HttpHandler.*|choose|routef?", label="Definition"); D2->trace_call_path(qualified_name="Giraffe.Core.compose", direction="both"); D3->get_code_snippet(qualified_name="Giraffe.Routing.routef"); D4->get_architecture(path="src/Giraffe"); D5->search_code/semantic_query("response writer handler sets body and content-type returning HttpContext"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/fsharp-graph.md`, `fsharp-explorer.md`, `fsharp-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the fsharp tier. --- ### 38. julia — D (Functional & Formal) **Repo:** SciML/DifferentialEquations.jl (`/tmp/bench/julia`) **Symlink:** no **Indexed in:** fast **Why this repo:** Flagship SciML solver suite (~3.1k stars, the canonical Julia ODE/SDE/DAE meta-package). NOTE: this repo is a *thin meta-package* — its entire `src/` is a single ~117-byte file (`src/DifferentialEquations.jl`) that does nothing but `@reexport` its dependencies. The actual solver code (`solve`, `__init`, algorithm structs) lives in *external* packages (SciMLBase, OrdinaryDiffEq, …), NOT in this checkout. Questions below are anchored on what is genuinely present in *this* repo (the module declaration + its reexport/dependency surface); symbols that live only in the reexported packages are explicitly labeled as such and carry `[verify]`. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Where is the public `DifferentialEquations` module declared, and what does the module body actually define? Expect the single declaration `module DifferentialEquations` in `src/DifferentialEquations.jl`, whose body is `using Reexport` plus a list of `@reexport using ` statements and nothing else." (Both the `module DifferentialEquations` token and the `@reexport using` lines are literal, grep-findable text in the one source file — plain `grep -n '@reexport' src/DifferentialEquations.jl` recovers the full API surface, so this is symmetric with text search.) 2. **[D2 Relationship]** "This package defines no intra-repo functions, so there are no `CALLS` edges to trace. Instead, enumerate the *import/reexport dependency relationship*: which packages does the `DifferentialEquations` module pull in via `@reexport using` / `using` (expect `Reexport`, `SciMLBase`, `OrdinaryDiffEq` on current `master`; older tags additionally list `DiffEqBase`, `StochasticDiffEq`, `DiffEqNoiseProcess`, `DelayDiffEq`, etc. [verify] — depends on the checked-out ref)? This is an IMPORTS-edge / module-dependency question, not a call-graph question." (Honest framing: D2's usual call-relationship target is genuinely absent here; the meaningful relationship in a pure reexport shell is the IMPORTS edge set, which both grep and the graph can recover.) 3. **[D3 Retrieval]** "Retrieve the exact source of the top-level module block in `src/DifferentialEquations.jl` — the entire `module DifferentialEquations ... end` with its `using Reexport` and `@reexport using` statements (the whole file is only ~8–15 lines depending on ref)." (The block is a real, grep-findable region in the single source file.) 4. **[D4 Architecture]** "Describe the structural organization: that `src/` holds exactly one tiny file, that `Project.toml` declares the real dependency set (`OrdinaryDiffEq`, `Reexport`, `SciMLBase` on current `master` [verify] — older tags list many more), and explain how DifferentialEquations.jl acts as a meta-package that aggregates and re-exposes external solver packages rather than implementing solvers itself. Contrast the near-empty `src/` against the breadth of the reexported namespace." 5. **[D5 Cross-cutting/Semantic]** "*(graph-favoring)* The user-facing API of this package (e.g. `solve`, `solve!`, `init`/`__init`, and algorithm-constructor structs such as `Tsit5`, `Rodas5` [verify]) is NOT defined in this repo — it is surfaced transitively through `@reexport`. Using semantic/cross-package reasoning rather than literal substring search over this checkout (where those tokens never appear), link each `@reexport using ` declaration to the kind of public symbols that package contributes to the merged `DifferentialEquations` namespace. This favors the graph's import/reexport modeling over grep, since grep on this repo finds none of the surfaced symbols." (Graph-favoring is legitimate here precisely because the symbols are absent from the local text — but note this also stresses whether the indexer crosses package boundaries at all; if it indexes only this checkout, BOTH tools will find nothing, which is itself a fair, informative result.) **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="DifferentialEquations", label="Module|Function"); D2->trace_call_path(qualified_name="...DifferentialEquations", direction="both") falling back to search_graph(relationship="IMPORTS") / get_architecture for the dependency edges; D3->get_code_snippet(qualified_name="DifferentialEquations"); D4->get_architecture(project="julia"); D5->search_code/semantic_query("reexported solver algorithm constructors / @reexport dependency surface"). NOTE: hints referencing `solve`/`__init` are intentionally absent because those symbols are not in this repo. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/julia-graph.md`, `julia-explorer.md`, `julia-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the julia tier. --- ### 39. vimscript — C (Dynamic & Scripting) **Repo:** SpaceVim/SpaceVim (`/tmp/bench/vimscript`) **Symlink:** no **Indexed in:** fast **Why this repo:** Top-starred (~20k) vimscript project; a large, idiomatic autoload-based codebase with deep `SpaceVim#...#fn` namespacing, exercising real-world vimscript structure per the plan's "popular + substantial + idiomatic" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Find the autoload entry-point functions `SpaceVim#begin` and `SpaceVim#end` defined in `autoload/SpaceVim.vim`. Does the graph surface them as distinct function definitions with their file and line, matching what `grep -n 'function! SpaceVim#'` would return?" 2. **[D2 Relationship]** "Starting from `SpaceVim#layers#load` (in `autoload/SpaceVim/layers.vim`), trace the call relationships in both directions: which functions invoke it (e.g. its own recursive call when loading a list of layers, plus any external callers from the config/bootstrap path [verify — callers live in `config/`-style files not confirmed here]) and which helpers it actually calls (e.g. `s:list_layers`, the dynamic `SpaceVim#layers#{layer}#set_variable` dispatch, and `SpaceVim#logger#info`/`#warn`)?" 3. **[D3 Retrieval]** "Retrieve the full source of the single function `SpaceVim#layers#isLoaded` from `autoload/SpaceVim/layers.vim` — exact body, signature, and line range, nothing else." 4. **[D4 Architecture]** "Describe the top-level architecture of the `autoload/SpaceVim/` tree: the role of the sibling modules (`layers.vim`, `plugins.vim`, `mapping.vim`, `logger.vim`, `api.vim`) and the `api/`, `layers/`, `plugins/`, `mapping/` subdirectories. How is functionality partitioned across autoload namespaces?" 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Across the autoload tree, find the functions responsible for *layer loading / enabling / reporting* (semantically related to `SpaceVim#layers#load`, `#disable`, `#isLoaded`, `#report`) even when their names don't share a single grep token. Surface naming-pattern duplication across the `s:list_layers` / `s:find_layers` private helpers and any config<->code links between `g:spacevim_*` options and the code that reads them." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="SpaceVim#(begin|end)"); D2->trace_call_path(name="SpaceVim#layers#load", direction="both"); D3->get_code_snippet(qualified_name="SpaceVim#layers#isLoaded"); D4->get_architecture(scope="autoload/SpaceVim"); D5->search_code/semantic_query("layer load enable report"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/vimscript-graph.md`, `vimscript-explorer.md`, `vimscript-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the vimscript tier. --- ### 40. nix — C (Dynamic & Scripting) **Repo:** nix-community/home-manager (`/tmp/bench/nix`) **Symlink:** no **Indexed in:** fast **Why this repo:** home-manager is the de-facto standard for declarative per-user Nix configuration (one of the most-starred Nix-language repos), with thousands of idiomatic `.nix` modules — substantial, real-world Nix that matches the plan's "popular + idiomatic + large" repo-selection criteria. **Language note (config/data language):** Nix is a declarative configuration language. "Functions" are attribute-set / lambda definitions; there is no conventional runtime call graph the indexer can resolve the way it does for an imperative language. Dimensions that depend on call relationships (D2) or semantic clustering (D5) are therefore handled honestly below — marked N/A with a reason where they do not naturally apply, rather than forced into unnatural questions. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the string-helper `storeFileName` in `modules/lib/strings.nix`. Confirm the graph resolves the same symbol that a plain `grep -rn 'storeFileName'` finds (grep-findable, plain-symbol)." 2. **[D2 Relationship]** "N/A for this config/data language. Nix module helpers are referenced by attribute-set composition and lazy interpolation, not a resolvable call graph; the indexer does not produce reliable CALLS edges between `.nix` lambdas, so a relationship/neighborhood question would not exercise a real graph capability. (If any CALLS edges are present for `dag.nix` helpers, they should be treated as best-effort, not scored.)" 3. **[D3 Retrieval]** "Retrieve the full definition of the DAG helper `entryAfter` from `modules/lib/dag.nix` — a single named symbol that also appears verbatim under `grep -rn 'entryAfter' modules/lib/dag.nix` (grep-findable, plain-symbol)." 4. **[D4 Architecture]** "Describe the structural organization of the `modules/` tree — how `modules/lib/`, `modules/programs/`, and `modules/services/` are arranged and where `modules/home-environment.nix` sits relative to them." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Surface the family of DAG entry helpers in `modules/lib/dag.nix` (`entryAfter`, `entryBefore`, `entryBetween`, `entryAnywhere` [verify], `entriesBetween` [verify]) as a related cluster, and identify whether any other `modules/lib/` file exposes a similarly-shaped helper family. This is a similarity/naming-pattern query where a structural/semantic index can group related symbols better than a single grep pattern. (If the index offers no semantic grouping over `.nix`, mark N/A with that reason — do not force it.)" **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="storeFileName"); D2->N/A (no reliable CALLS edges for Nix lambdas); D3->get_code_snippet(qualified_name="...dag.entryAfter"); D4->get_architecture(scope="modules/"); D5->search_code/semantic_query("DAG entry helper family") or N/A if no semantic grouping over .nix. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/nix-graph.md`, `nix-explorer.md`, `nix-judged.json`. **Aggregates into:** D1-D4 cross-group rollups (D2 recorded as N/A), D5 within Group C only, Group C, the nix tier. --- ### 41. commonlisp — D (Functional & Formal) **Repo:** lem-project/lem (`/tmp/bench/commonlisp`) **Symlink:** no **Indexed in:** fast **Why this repo:** Lem is one of the most-starred, actively-maintained Common Lisp projects (a full editor/IDE written in idiomatic CL with macros, CLOS, and a multi-package system), giving substantial real-world structure for every dimension — matching the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the command macro `define-command` and the buffer constructor `make-buffer` — both grep-findable top-level forms — and report their defining files and forms (`defmacro` vs `defun`)." 2. **[D2 Relationship]** "Starting from `current-buffer`, show the call relationships in both directions: what does it call, and which commands/functions (e.g. buffer-editing functions) call it? [verify the exact callers against the pinned commit]" 3. **[D3 Retrieval]** "Retrieve the full source of the `define-major-mode` macro [verify] — a single named symbol, grep-findable — so a reviewer can read its entire body without opening the file." 4. **[D4 Architecture]** "Describe Lem's top-level module organization: the `src/` core (buffer/point/window), `frontends/` (e.g. sdl2, ncurses), `extensions/`, and `modes/`, and how the `.asd` system definitions (e.g. `lem.asd`) tie packages together." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find the cluster of point/cursor-movement helpers (e.g. `move-point`, `character-offset`, `line-offset` [verify]) and surface duplicated movement-helper patterns across `src/` and `modes/` by their shared call relationships — plain grep finds the literal names but cannot group them by who-calls-what or rank them by structural importance the way the graph does." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="define-command|make-buffer"); D2->trace_call_path(name="current-buffer", direction="both"); D3->get_code_snippet(qualified_name="define-major-mode"); D4->get_architecture(); D5->search_code("point/cursor movement by line or character") then trace_call_path on the matched movement helpers to cluster by shared callers. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/commonlisp-graph.md`, `commonlisp-explorer.md`, `commonlisp-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the commonlisp tier. --- ### 42. elm — D (Functional & Formal) **Repo:** elm/core (`/tmp/bench/elm`) **Symlink:** no **Indexed in:** fast **Why this repo:** `elm/core` is the foundational standard-library package for the Elm language — the `.elm` modules (`Basics`, `List`, `Maybe`, `Result`, `String`, `Dict`, `Task`, …) that every Elm program implicitly depends on. It is ~85% Elm source (popular OSS, ~2.8k stars) and is the canonical place where the language's idiomatic `.elm` code actually lives, satisfying the plan's "popular + representative" criterion for the **elm** extractor. > **Repo-selection note (recorded for fairness):** The obvious candidate, `elm/compiler`, is ~95% **Haskell** (the compiler/builder/terminal sources in `compiler/src/`, `builder/src/`, `terminal/src/`); its `.elm` content is only test fixtures and `reactor/` UI assets and does NOT contain the standard library. Indexing `elm/compiler` under LANGUAGE=elm would measure almost nothing real, so this tier targets `elm/core`, where the Elm `.elm` sources genuinely live. All symbols cited below are public, in-repo `.elm` identifiers (grep-findable in `src/*.elm`) unless tagged `[verify]`. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "In the `Maybe` module (`src/Maybe.elm`), locate the public API — specifically `withDefault`, `map`, `andThen`, and the `Maybe` type itself (with its `Just`/`Nothing` constructors). Are all four discoverable by name?" *(All four are plain, grep-findable identifiers in `src/Maybe.elm`: the module header exposes `Maybe(..)`, `andThen`, `map`, `withDefault`. Grep finds these as easily as the graph; this question is intentionally symmetric.)* 2. **[D2 Relationship]** "Starting from `Maybe.andThen` in `src/Maybe.elm`, show the call relationship in both directions: which functions in `Maybe` reference `andThen`, and what does `andThen` itself reference (it pattern-matches on `Just`/`Nothing`). Then compare against `Result.andThen` in `src/Result.elm` to confirm the two are distinct symbols and not merged into one." 3. **[D3 Retrieval]** "Retrieve the complete source of `List.foldr` from `src/List.elm` — the named symbol, exact body, including the tail-recursive helper it delegates to (`foldrHelper`)." *(`foldr` and `foldrHelper` are canonical, grep-findable identifiers in `src/List.elm`; a grep for `foldr` returns the definition directly. Symmetric with Q1 — not a graph-only target.)* 4. **[D4 Architecture]** "Describe the module/directory organization of `src/`: how the core modules (`Basics`, `List`, `Maybe`, `Result`, `String`, `Dict`, `Set`, `Array`, `Task`, `Char`, `Bitwise`, `Tuple`) sit at the top of `src/`, and what lives under the `src/Elm/` and `src/Platform/` subdirectories (e.g. JSON, the platform/runtime glue)." *(Structural; graph framing acceptable, but a directory listing also answers it.)* 5. **[D5 Cross-cutting/Semantic]** "**(graph-favoring — semantic/duplication.)** Across the core modules, surface the family of functions that are semantic near-duplicates of `Maybe.map` — the `map`/`map2`…`map5` family replicated across `Maybe`, `Result`, `List`, and `Task`, plus the recurring `andThen` monadic-bind pattern — as one cross-module cluster. Plain text search can only approximate this via brittle per-name greps across files; the value is grouping the replicated *shape*, not matching one literal name." **Expected graph tools (hint, not a script):** D1->`search_graph(name_pattern="withDefault|andThen|^map$|^Maybe$", project="elm")`; D2->`trace_call_path(qualified_name="Maybe.andThen", direction="both")`; D3->`get_code_snippet(qualified_name="List.foldr")`; D4->`get_architecture(project="elm")`; D5->`search_code(semantic_query="map family / andThen monadic bind across Maybe Result List Task")` then `search_graph(name_pattern="^map[2-5]?$|^andThen$")`. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/elm-graph.md`, `elm-explorer.md`, `elm-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group D only, Group D, the elm tier. --- ### 43. fortran — B (Systems & Low-level) **Repo:** fortran-lang/stdlib (`/tmp/bench/fortran`) **Symlink:** no **Indexed in:** fast **Why this repo:** The community-standard Fortran Standard Library — the most-starred, most-active idiomatic modern Fortran codebase (modules, derived types, generic interfaces, `.fypp` templates), satisfying the plan's "popular + substantial + idiomatic" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the public sorting API in `stdlib_sorting`: where are the `sort`, `ord_sort`, and `sort_index` interfaces/subroutines defined, and what are their qualified names?" 2. **[D2 Relationship]** "Take `ord_sort` in `stdlib_sorting` and show its call relationships in both directions — which internal helper procedures (e.g. the merge/insertion-sort workers it dispatches to) it calls, and which public entry points reach it." 3. **[D3 Retrieval]** "Retrieve the full source of the `string_type` derived type defined in `stdlib_string_type`." 4. **[D4 Architecture]** "Describe the top-level architecture of stdlib: how `src/` is organized into topical subdirectories (`sorting/`, `strings/`, `stats/`, `linalg/`, …), each holding its `stdlib_*` module sources as `.fypp` templates (preprocessed to `.f90`), and how the build (`CMakeLists.txt` per subdir) and `test/` directories relate to those module sources." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Starting from `mean` in `stdlib_stats`, map the family of statistical reduction/aggregation procedures it is conceptually related to (`var`, `moment`, `corr`, `cov`, `median`) — surface how they are grouped/connected structurally (shared module, common `.fypp` generation, cross-references) rather than by any single shared name token. (Honest note: each target is itself a plainly-named public symbol that grep can find individually; the graph-favoring claim is about recovering the *relationship/grouping* across them in one pass, not about hidden symbols.)" **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*(sort|ord_sort|sort_index).*", label="Function|Subroutine|Interface"); D2->trace_call_path(qualified_name="...ord_sort", direction="both"); D3->get_code_snippet(qualified_name="...string_type"); D4->get_architecture(project="fortran"); D5->search_graph(name_pattern=".*(mean|var|moment|corr|cov|median).*", project="fortran") + trace_call_path to expose the shared-module/cross-reference grouping. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/fortran-graph.md`, `fortran-explorer.md`, `fortran-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the fortran tier. --- ### 44. cuda — B (Systems & Low-level) **Repo:** NVIDIA/cuda-samples (`/tmp/bench/cuda`) **Symlink:** no **Indexed in:** fast **Why this repo:** NVIDIA's official, widely-starred reference corpus of idiomatic CUDA C/C++ kernels and host code — substantial, multi-category, and the canonical exemplar of host/device split that a code graph must handle for the language. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the definition of the error-checking helper `checkCudaErrors` and the device-selection helper `findCudaDevice` in `Common/helper_cuda.h`; what are their signatures and where are they declared?" (both are grep-findable macro/template names used across nearly every sample) 2. **[D2 Relationship]** "Starting from the `vectorAdd` kernel in `Samples/0_Introduction/vectorAdd/vectorAdd.cu`, trace the call graph in both directions: which host function (`main`) launches it, and which device/runtime symbols does the launch path touch (e.g. `cudaMalloc`, `cudaMemcpy`, `checkCudaErrors`)?" 3. **[D3 Retrieval]** "Retrieve the full body of the `MatrixMulCUDA` templated kernel from `Samples/0_Introduction/matrixMul/matrixMul.cu`." (single named symbol, grep-findable) 4. **[D4 Architecture]** "Describe the top-level structure of the repo: the numbered sample-category directories (`0_Introduction`, `1_Utilities`, `2_Concepts_and_Techniques`, `3_CUDA_Features`, `4_CUDA_Libraries`, `5_Domain_Specific`, `6_Performance` [verify]) and the shared `Common/` helper headers, and how individual samples depend on `Common/`." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Across the corpus, find the samples that semantically implement a parallel reduction pattern — e.g. `reduce`, `reduction`, `threadFenceReduction` [verify] — and surface the shared idiom (shared-memory accumulation + `__syncthreads()`), plus the config<->code link between each sample's `Makefile`/`*_vs*.vcxproj` and its `.cu` source. Labeled graph-favoring: grep alone cannot cluster by reduction semantics." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="checkCudaErrors|findCudaDevice"); D2->trace_call_path(name="vectorAdd", direction="both"); D3->get_code_snippet(qualified_name="MatrixMulCUDA"); D4->get_architecture(); D5->search_code/semantic_query("parallel reduction shared memory __syncthreads"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/cuda-graph.md`, `cuda-explorer.md`, `cuda-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the cuda tier. --- ### 45. cobol — B (Systems & Low-level) **Repo:** OCamlPro/gnucobol (`/tmp/bench/cobol`) **Symlink:** no **Indexed in:** fast **Why this repo:** GnuCOBOL is the canonical open-source COBOL toolchain; its shipped test corpus is large, idiomatic COBOL with real PROGRAM-IDs, COPY/CALL usage and division structure — substantial and grep-findable, matching the plan's "popular + idiomatic + substantial" repo-selection criteria for the Systems group. (Note: the bulk of committed, grep-findable COBOL lives inline in the GNU Autotest `.at` files under `tests/testsuite.src/`; the NIST COBOL85 `.cob` sources under `tests/cobol85/` are downloaded at test time rather than committed.) **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the COBOL program units (the `PROGRAM-ID` definitions) embedded in the GnuCOBOL test corpus — primarily the inline programs inside `tests/testsuite.src/*.at` — and report where the program `prog` [verify] and any sample with a `PROGRAM-ID. main` [verify] are declared. Both must be reachable by a plain grep for `PROGRAM-ID`." 2. **[D2 Relationship]** "For a test program that issues a static `CALL \"subprog\"` to a called subprogram (e.g. a caller/callee pair in the `run_*` autotest cases such as `run_misc.at` / `run_file.at` [verify]), show the call relationship in both directions: which program(s) CALL the subprogram, and what that subprogram itself CALLs." 3. **[D3 Retrieval]** "Retrieve the full source of the COBOL paragraph/section named `MAIN-LOGIC` [verify] (or, if absent, the program's `PROCEDURE DIVISION` body for the program-id you confirm in Q1) exactly as written, including its `DISPLAY`/`MOVE` statements. The target name is a literal identifier, so it must also be locatable by a plain grep." 4. **[D4 Architecture]** "Describe the structural organization of the COBOL test corpus: the `tests/` directory layout — the GNU Autotest harness files in `tests/testsuite.src/*.at` (e.g. `run_misc.at`, `run_file.at`, `syn_*.at` [verify]) versus the NIST COBOL85 conformance suite under `tests/cobol85/` (downloaded `newcob.val` rather than committed `.cob` files) [verify] — and how copybooks (`COPY` members) relate to the programs that include them." 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Find COBOL programs that are semantically near-duplicates — e.g. the family of arithmetic/`COMPUTE` conformance tests or the `DISPLAY`-only smoke programs — that share near-identical PROCEDURE DIVISION shape but differ only in literals/data items. This is openly graph-favoring (similarity / naming-pattern clustering) and is not expected to be reproducible by a single grep." **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern="(?i)PROGRAM-ID|MAIN|prog"); D2->trace_call_path(name="subprog", direction="both"); D3->get_code_snippet(qualified_name="...MAIN-LOGIC"); D4->get_architecture(scope="tests/"); D5->search_code(semantic_query="COBOL program with COMPUTE/DISPLAY conformance body")/search_graph(semantic). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/cobol-graph.md`, `cobol-explorer.md`, `cobol-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group B only, Group B, the cobol tier. --- ### 46. verilog — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** YosysHQ/picorv32 (`/tmp/bench/verilog`) **Symlink:** no **Indexed in:** fast **Why this repo:** Canonical, widely-cited (~3k stars) size-optimized RISC-V CPU core written in idiomatic, substantial single-file Verilog with multiple module variants and PCPI co-processors — a representative HDL target per the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level `module` definitions declared in this repo (e.g. `picorv32`, `picorv32_axi`, `picorv32_wb`, `picorv32_pcpi_mul`, `picorv32_pcpi_div`) and report the file and line where each is declared." (All are grep-findable via `module ` in `picorv32.v`.) 2. **[D2 Relationship]** "Map the cross-module structure around the core `picorv32`: which wrapper modules embed it (`picorv32_axi`, `picorv32_wb`) and which sub-modules it pulls in (the PCPI units `picorv32_pcpi_mul` / `picorv32_pcpi_fast_mul` / `picorv32_pcpi_div`, plus the `picorv32_regs` register file). Note: picorv32 is plain Verilog-2005 with no package `import`s, and the indexer's Verilog call-extraction does NOT model `module_instantiation` as a graph edge [verify], so the instantiation links must be recovered from the grep-findable instantiation sites (`picorv32 (...)`, `picorv32_pcpi_* (...)`) rather than from a graph reference edge — report which side (graph vs grep) actually surfaces each link." 3. **[D3 Retrieval]** "Retrieve the full source of the largest module definition in this repo — the core `picorv32` module — with its exact start/end boundaries (`module picorv32` … `endmodule`)." (grep-findable boundary tokens.) 4. **[D4 Architecture]** "Describe the file/module organization of the repo: how the synthesizable RTL (`picorv32.v`), the bus-wrapper variants (AXI/Wishbone), the co-processor (PCPI) modules, and the testbench/firmware directories are arranged relative to one another." 5. **[D5 Cross-cutting/Semantic]** "[GRAPH-FAVORING] Identify naming-pattern / duplication structure across the bus adapters: which modules share the `picorv32_axi*` vs `picorv32_wb*` prefix families, and which interface-signal groups (e.g. the `mem_axi_*` AXI handshake ports) recur across `picorv32_axi` and `picorv32_axi_adapter`? Surface near-duplicate port/parameter blocks that plain grep would not cluster semantically. Note: in fast (non-LSP) mode `SIMILAR_TO`/`SEMANTICALLY_RELATED` edges are typically absent [verify], so the clustering leans on Module-node naming families plus `search_code`, not on a dedicated similarity edge." **Expected graph tools (hint, not a script):** D1->search_graph(label="Module"|definition, name_pattern="picorv32.*"); D2->search_graph(name_pattern="picorv32.*") for the Module-node set + trace_call_path(qualified_name="picorv32", direction="both") best-effort (Verilog instantiation is NOT a modeled edge [verify], so cross-check against grep instantiation sites); D3->get_code_snippet(qualified_name="picorv32"); D4->get_architecture(); D5->search_code/semantic_query(prefix-family + recurring AXI port clustering; semantic edges sparse in fast mode [verify]). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/verilog-graph.md`, `verilog-explorer.md`, `verilog-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the verilog tier. --- ### 47. emacslisp — C (Dynamic & Scripting) **Repo:** magit/magit (`/tmp/bench/emacslisp`) **Symlink:** no **Indexed in:** fast **Why this repo:** Magit is the most-starred and de-facto reference Emacs Lisp project (a full Git porcelain), large and idiomatic enough to exercise the indexer's `defun`/`defmacro`/`defcustom` handling — matching the plan's "popular, substantial, idiomatic" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level interactive command definitions whose names match `magit-stage.*` (e.g. `magit-stage`, `magit-stage-file`, `magit-stage-modified`) and report which file each is defined in." — all grep-findable as `(defun magit-stage` etc. in `lisp/magit-apply.el`. 2. **[D2 Relationship]** "For `magit-refresh`, show both directions of its call graph: which commands invoke it (inbound, e.g. `magit-status`/post-command refresh hooks) and which helpers it calls (outbound, e.g. `magit-refresh-buffer`, `magit-run-hook-with-benchmark`)." 3. **[D3 Retrieval]** "Retrieve the full source of the macro `magit-insert-section` exactly as defined in `lisp/magit-section.el`." — single named symbol, grep-findable as `(defmacro magit-insert-section`. 4. **[D4 Architecture]** "Describe the module/file organization of the `lisp/` directory: how `magit.el` ties together the feature files (`magit-status.el`, `magit-diff.el`, `magit-log.el`, `magit-commit.el`, `magit-process.el`, `magit-git.el`) and where the section-rendering layer (`magit-section.el`) sits relative to them." 5. **[D5 Cross-cutting/Semantic]** "[graph-favoring] Find the cluster of functions that shell out to Git — i.e. semantically 'run a git subprocess' — such as `magit-call-git`, `magit-run-git`, `magit-run-git-async`, `magit-start-git`, `magit-git-string`, `magit-git-insert`; report naming-pattern duplication and which are thin wrappers over which." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="magit-stage.*", label="Function"); D2->trace_call_path(name="magit-refresh", direction="both"); D3->get_code_snippet(qualified_name="magit-insert-section"); D4->get_architecture(scope="lisp/"); D5->search_code/semantic_query("functions that run a git subprocess"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/emacslisp-graph.md`, `emacslisp-explorer.md`, `emacslisp-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group C only, Group C, the emacslisp tier. --- ### 48. json — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** SchemaStore/schemastore (`/tmp/bench/json`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical, high-traffic registry of JSON Schemas (thousands of `.json` schema files under `src/schemas/json/` plus the `src/api/json/catalog.json` index); it is the most idiomatic and substantial pure-JSON/JSON-Schema corpus on GitHub, matching the plan's "popular + idiomatic + substantial" repo-selection criteria for Group E. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "Locate the two top-level registry-defining documents: the `schemas` array in `src/api/json/catalog.json` (the index) and the catalog's own meta-schema `src/schemas/json/schema-catalog.json`. Enumerate the top-level keys these files declare (e.g. `schemas`, `version`, `url`, `fileMatch`, `name`, `description`) and report which file declares each." (Grep-findable: `"fileMatch"`, `"schemas"`, and the filename `catalog.json` appear verbatim; this dimension must be answerable by plain grep, not graph-only. Caveat: whether the graph emits `Definition` nodes for JSON object keys is itself dubious for a pure-data language [verify].) 2. **[D2 Relationship]** "N/A — JSON is a config/data language. The relationship a reviewer would actually want here is JSON-Schema-semantic: a `$ref`/`$schema` string pointer linking one schema document to another, or a `catalog.json` entry's `url`/`fileMatch` mapping to a schema file. These are string-valued pointers inside data, not generic code-graph symbols — the knowledge graph models these `.json` files as documents/keys, not as nodes joined by `CALLS`/`HANDLES`/`IMPLEMENTS`/reference edges, so there is no relationship edge for `trace_path` to traverse. Forcing a `trace_call_path`-style question would be unnatural and would not exercise a real graph capability; this linkage is instead surfaced as the explicitly graph/semantic-favoring D5 below." 3. **[D3 Retrieval]** "Retrieve a single large definition block verbatim: the full `src/schemas/json/github-workflow.json` schema (one of the largest individual schema documents in the repo), or the largest single entry-set block of the `schemas` array inside `src/api/json/catalog.json`." (Grep-findable filename; this dimension must be answerable by plain grep on the filename, not graph-only.) 4. **[D4 Architecture]** "Describe the directory/file organization: how `src/schemas/json/` (the schema documents), `src/test//` (per-schema fixtures), and `src/api/json/catalog.json` (the index) are organized and how they relate." 5. **[D5 Cross-cutting/Semantic — graph-favoring]** "Detect duplication and naming-pattern links — find schema files in `src/schemas/json/` that share near-identical structure (e.g. the many `tsconfig*.json` / `*.config.json` variants), and link each `catalog.json` entry's `url`/`fileMatch` to the actual schema file it points at (index<->document linkage). Plain text search cannot surface structural similarity or resolve the index-to-file mapping. (Openly graph/semantic-favoring.)" **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*catalog.*|.*schemas.*", project="json") [verify the graph emits Definition nodes for JSON keys; fall back to search_code on the literal keys]; D2->N/A (JSON has no call/reference graph between data keys in a generic code graph; see D2 note); D3->get_code_snippet(qualified_name="src/schemas/json/github-workflow.json" [verify JSON files are snippet-addressable]); D4->get_architecture(project="json"); D5->search_code / search_graph(semantic_query="near-duplicate schema structure + catalog url<->file links"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/json-graph.md`, `json-explorer.md`, `json-judged.json`. **Aggregates into:** D1-D4 cross-group rollups (D2 counted as N/A for json), D5 within Group E only, Group E, the json tier. --- ### 49. xml — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** spring-projects/spring-petclinic (symlink java) (`/tmp/bench/xml`) **Symlink:** yes **Indexed in:** fast **Why this repo:** Spring PetClinic is the canonical, high-popularity reference Spring Boot app; its Maven `pom.xml`, cache and CI XML are idiomatic, hand-maintained build/config markup — substantial enough to exercise structural XML queries, matching the plan's "popular + idiomatic in-language" repo criterion. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "In the root `pom.xml`, list the top-level `` definitions under `` — e.g. the `spring-boot-maven-plugin` and the `spring-javaformat-maven-plugin` [verify] entries. (Symmetric: grep-findable too — artifactIds appear as literal `` text under ``.)" 2. **[D2 Relationship]** "N/A for build/config XML. A root `pom.xml` has no call/reference graph in the codebase-memory sense — `` inheritance and `${...}` property placeholders are Maven's own resolution mechanics, not CALLS/IMPORTS edges the graph models. (Best-effort, non-scoring sub-probe: which `` POM does this inherit from — `spring-boot-starter-parent` — answerable by either tool from literal text; documented here only so the dimension is acknowledged, not forced into a fake relationship query.)" 3. **[D3 Retrieval]** "Retrieve the full `` block (the largest single definition) from the root `pom.xml`. (Symmetric: grep-findable too — the `` open/close tags are literal anchors; verbatim retrieval is the test, not structure.)" 4. **[D4 Architecture]** "Describe the XML/config file organization of the repo: where build (`pom.xml`), CI workflow (`.github/workflows/*.yml` [verify]), and resource config (`src/main/resources/**`) markup live relative to the Java source tree." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring) Within the XML/config surface, which Maven property keys defined in `pom.xml` (e.g. ``, `` [verify]) are reused via `${...}` in the same or sibling config files, and are any version literals duplicated across `pom.xml` and CI workflow files [verify]? (Note: config↔Java-runtime semantic links are out of scope for XML — there is no graph edge from a Maven `` to Java usage — so this stays inside the markup/config tree where a cross-cutting query is meaningful.)" **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*plugin.*"); D2->N/A (no call/reference graph for build XML — acknowledge, do not invoke trace_call_path); D3->get_code_snippet(qualified_name="pom.xml:"); D4->get_architecture(scope="config/markup tree"); D5->search_code("\\$\\{.*version\\}")/query_graph for duplicated version literals across config files. **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/xml-graph.md`, `xml-explorer.md`, `xml-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the xml tier. --- ### 50. markdown — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** github/docs (`/tmp/bench/markdown`) **Symlink:** no **Indexed in:** fast **Why this repo:** One of GitHub's most-starred docs repos (~16k stars), with thousands of substantial, idiomatic Markdown files carrying rich YAML frontmatter and Liquid includes — exactly the "popular + idiomatic + substantial" profile the plan's repo-selection criteria demand for a markup/docs tier. > **Graph model note (markdown):** The extractor models a Markdown file as a single `document` module node and maps each heading (`atx_heading`/`setext_heading`) to a "class"-labeled Definition node. It does **not** extract YAML frontmatter keys, `children:` lists, or `{% data reusables.* %}` Liquid includes, and markdown has **no CALLS / IMPORTS edges**. Questions and expected-tool hints below are written to reflect that real model — frontmatter/include facts are grep-findable text, not graph nodes. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the section headings (the `#`/`##` ATX headings the indexer captures as Definition nodes) in `content/get-started/start-your-journey/hello-world.md` [verify], e.g. the top-level `# Hello World` title and its sub-section headings." — symmetric: grep-findable via the literal heading lines `# `/`## ` in the file, and graph-findable as heading Definition nodes. 2. **[D2 Relationship]** "N/A for markdown — the code graph extracts no CALLS/IMPORTS edges and does not model `children:` frontmatter or `{% data reusables.* %}` Liquid includes, so there is no symbol-to-symbol relationship to traverse. (Reason: markdown lang-spec defines only a `document` module type and heading class types; no call/import/field types.) The cross-file include graph is answerable only by grep/text traversal, not by the graph, so it is intentionally not posed as a graph question." 3. **[D3 Retrieval]** "Retrieve the contents of `content/get-started/index.md` — the document node for that landing page — and report its leading section (title heading + intro paragraph)." — symmetric: grep-findable by opening the named file, and graph-findable via the `document`/heading node for that path. 4. **[D4 Architecture]** "Describe the directory organization of the `content/` tree: enumerate the top-level product folders (`actions`, `authentication`, `code-security`, `get-started`, `pull-requests`, `repositories`, `rest`, …) and explain the `index.md`-per-folder convention plus the parallel `data/reusables/` tree." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Identify duplicated or near-duplicate prose across `data/reusables/**.md` and `content/**.md` — e.g. reusable snippets whose text is also inlined verbatim in pages instead of referenced via `{% data reusables.* %}`. This is openly graph/semantic-favoring (near-duplicate detection over heading/document nodes + text), is best-effort for markdown (the graph stores headings + document text, not prose-level dedup), and is scored within Group E only." **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", project="markdown", name_pattern=".*Hello World.*") [verify exact heading text]; D2->N/A (no CALLS/IMPORTS edges for markdown; use grep for frontmatter `children:`/Liquid includes); D3->get_code_snippet(qualified_name="") [verify QN format]; D4->get_architecture(scope="content/"); D5->search_code(query="reusables inlined duplicate prose") / search_graph(semantic_query="near-duplicate documentation snippets"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/markdown-graph.md`, `markdown-explorer.md`, `markdown-judged.json`. **Aggregates into:** D1-D4 cross-group rollups (D2 records N/A), D5 within Group E only, Group E, the markdown tier. --- ### 51. makefile — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** redis/redis (symlink c) (`/tmp/bench/makefile`) **Symlink:** yes **Indexed in:** fast **Why this repo:** Redis is a top-tier (60k+ star) C project whose hand-written, multi-level GNU Make build (`Makefile` → `src/Makefile` → `deps/Makefile`) is large and idiomatic, exercising real-world makefile structure rather than a generated stub — matching the plan's "popular + substantial + idiomatic" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level variable and target definitions declared in `src/Makefile` — e.g. the `STD`, `OPT`, `FINAL_CFLAGS`, `FINAL_LDFLAGS` variable assignments [verify] and the `$(REDIS_SERVER_NAME)` / `redis-server` build targets [verify]. Are these grep-findable identifiers surfaced as graph definition nodes?" 2. **[D2 Relationship]** "How does the root `Makefile` delegate to the sub-build? Identify the cross-file reference where the top-level targets (e.g. `all:`, `install:`) hand off into `src/Makefile` — in redis this is a `cd src && $(MAKE) $@` shell recursion [verify] rather than a `-C` flag, so confirm the exact mechanism. Report whether the graph models this as a relationship edge (e.g. INCLUDES/IMPORTS or a delegation edge) at all, and if not, note that recursive-make handoff may be N/A for a code graph (it is shell recursion, not a code call)." 3. **[D3 Retrieval]** "Retrieve the full recipe block for the `install:` target as defined in `src/Makefile` [verify] (its body and prerequisites), returning only that one definition rather than the whole file." 4. **[D4 Architecture]** "Describe the file/directory organization of the build system: the layering of root `Makefile`, `src/Makefile`, `deps/Makefile`, and the generated `src/Makefile.dep` [verify] — how the three Make units compose and where each lives in the tree." 5. **[D5 Cross-cutting/Semantic]** "(Graph-favoring.) Find naming-pattern duplication and config↔code links across the makefiles: where the same dependency-toolchain pattern (`$(MAKE) ... -C ../deps` / `-C deps`) recurs [verify], and where build variables like `$(REDIS_SERVER_NAME)` [verify] map to the produced binary names — clusters that text search cannot group by similarity." **Expected graph tools (hint, not a script):** D1->search_graph(label="Definition", name_pattern=".*(STD|OPT|FINAL_CFLAGS|REDIS_SERVER_NAME).*"); D2->search_graph(label="Definition", name_pattern=".*(all|install)$") + query_graph for any cross-file/INCLUDES edge into src/Makefile (expect possible N/A — recursive make is shell recursion, not a graph call edge); D3->get_code_snippet(qualified_name="src/Makefile::install"); D4->get_architecture(scope="build"); D5->search_code/query_graph("recursive $(MAKE) -C dependency build pattern"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/makefile-graph.md`, `makefile-explorer.md`, `makefile-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the makefile tier. --- ### 52. cmake — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** kitware/CMake (`/tmp/bench/cmake`) **Symlink:** no **Indexed in:** fast **Why this repo:** The canonical, most-starred CMake-language corpus on GitHub — its `Modules/*.cmake` tree is large, idiomatic, and authored by the language's own maintainers, making it the reference body of `function()`/`macro()`/`include()` build code the plan's repo-selection criteria call for. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level CMake-language definitions provided by `Modules/FindPkgConfig.cmake` — specifically the `function(pkg_check_modules ...)` and `macro(pkg_search_module ...)` public entry points — and report their defining file and line. (grep-findable: the literal tokens `pkg_check_modules` and `pkg_search_module`.)" 2. **[D2 Relationship]** "Show the cross-file include/reference graph for `function(FetchContent_MakeAvailable ...)` in `Modules/FetchContent.cmake`: what helper functions it invokes (e.g. `FetchContent_GetProperties`, `FetchContent_Populate`) and which modules `include()` or call it, traversing both directions." 3. **[D3 Retrieval]** "Retrieve the full body of the `function(ExternalProject_Add ...)` definition in `Modules/ExternalProject.cmake` — its largest single public definition — exactly as written, with no surrounding module boilerplate. (grep-findable: the literal token `ExternalProject_Add` locates the definition site; the value tested is exact-boundary body retrieval, not symbol discovery.)" 4. **[D4 Architecture]** "Describe the organization of the CMake-language module tree: how `Modules/` is partitioned (top-level helpers vs. `Modules/FindXXX.cmake` find-modules vs. `Modules/Platform/`), and where `CMakeLists.txt` build entry points sit relative to `Source/`." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Across all `Modules/*.cmake`, find duplicated/near-duplicate argument-parsing blocks that re-implement `cmake_parse_arguments(...)` handling, and flag find-modules that follow the same `find_path` + `find_library` + `find_package_handle_standard_args` naming pattern — surfacing config<->convention links plain grep cannot cluster." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern=".*pkg_(check|search)_module.*", label="Function"); D2->trace_call_path(name="FetchContent_MakeAvailable", direction="both"); D3->get_code_snippet(qualified_name="ExternalProject_Add"); D4->get_architecture(scope="Modules"); D5->search_code/semantic_query("cmake_parse_arguments duplicate argument parsing find_package_handle_standard_args pattern"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/cmake-graph.md`, `cmake-explorer.md`, `cmake-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the cmake tier. --- ### 53. protobuf — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** googleapis/googleapis (`/tmp/bench/protobuf`) **Symlink:** no **Indexed in:** fast **Why this repo:** Canonical, high-star (~8.6k) Google monorepo of idiomatic, substantial `.proto` API definitions — the de-facto reference corpus for protobuf, matching the plan's "popular + idiomatic + substantial" repo-selection criteria. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level `message` and `service` definitions declared in `google/longrunning/operations.proto` — at minimum the `Operations` service and the `Operation` message — and report the `.proto` file each is declared in." (grep-symmetric: `message Operation`/`service Operations` are plain-text findable.) 2. **[D2 Relationship]** "Starting from `google/longrunning/operations.proto`, map its cross-file `import` references (e.g. `google/protobuf/any.proto`, `google/rpc/status.proto`, `google/api/annotations.proto`) and show which imported types (`google.protobuf.Any`, `google.rpc.Status`) are actually referenced by fields inside the `Operation` message." 3. **[D3 Retrieval]** "Retrieve the full definition of the `Operation` message from `google/longrunning/operations.proto`, including all its fields (`name`, `metadata`, `done`, and the `result` oneof)." (grep-symmetric: `message Operation {` is plain-text findable.) 4. **[D4 Architecture]** "Describe the directory/package organization under `google/` — how API surfaces are grouped into namespaces such as `google/rpc/`, `google/type/`, `google/api/`, and `google/longrunning/` — and how `package` declarations mirror that directory layout." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Find `.proto` definitions semantically similar to a generic error/status carrier — surface duplication and convergent naming across `google.rpc.Status` (`google/rpc/status.proto`), the error-detail messages in `google/rpc/error_details.proto`, and the canonical codes in `google/rpc/code.proto` — and flag the cross-file naming-pattern / type-reference links a plain grep over a single file would miss." **Expected graph tools (hint, not a script):** D1->search_graph(label="message|service", name_pattern=".*Operation.*"); D2->query_graph IMPORTS edges from `google/longrunning/operations.proto` plus type-reference resolution into the `Operation` message fields (protobuf has IMPORTS + DEFINES edges, not CALLS); D3->get_code_snippet(qualified_name="google.longrunning.Operation"); D4->get_architecture(scope="google/"); D5->search_code/semantic_query("error status detail carrier message"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/protobuf-graph.md`, `protobuf-explorer.md`, `protobuf-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the protobuf tier. --- ### 54. graphql — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** graphql/graphql-js (`/tmp/bench/graphql`) **Symlink:** no **Indexed in:** fast **Why this repo:** graphql-js is the reference JavaScript implementation of the GraphQL spec (20k+ stars, the canonical SDL/schema toolkit), giving a large, idiomatic body of schema-definition and type-system code — squarely matching the plan's "popular + substantial + idiomatic" repo-selection criteria for Group E (schema). **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level type-system definition classes exported by the schema layer — e.g. `GraphQLObjectType`, `GraphQLInterfaceType`, `GraphQLScalarType`, `GraphQLEnumType`, and `GraphQLSchema` — and confirm where each is declared in `src/type/definition.ts` / `src/type/schema.ts`." 2. **[D2 Relationship]** "Starting from the `buildSchema` entry point, show the cross-file reference chain into `buildASTSchema` and the `extendSchema`/`extendSchemaImpl` machinery — what depends on it and what it pulls in (direction=both)." 3. **[D3 Retrieval]** "Retrieve the full definition of the `coerceInputValue` function in `src/utilities/coerceInputValue.ts`." [verify] 4. **[D4 Architecture]** "Describe the directory/module organization of the `src/` tree — how the `type/`, `language/`, `execution/`, `validation/`, and `utilities/` subdirectories partition the schema, AST/parser, and execution concerns." 5. **[D5 Cross-cutting/Semantic]** "(graph-favoring) Across the validation layer, surface the recurring naming/structure pattern of the `src/validation/rules/*.ts` rule modules (each exporting a `*Rule` validator) and link these rule definitions to where they are aggregated in `specifiedRules` — a duplication/naming-pattern + definition<->aggregation query that plain grep cannot rank structurally." **Expected graph tools (hint, not a script):** D1->search_graph(name_pattern="GraphQL.*Type|GraphQLSchema", label="Class"); D2->trace_call_path(name="buildSchema", direction="both"); D3->get_code_snippet(qualified_name="coerceInputValue"); D4->get_architecture(); D5->search_code/semantic_query("validation rule *Rule modules aggregated in specifiedRules"). **Graph stats (filled at index time — every node label and all 32 edge types on their own row, zeros kept):** _Index time (clone + cold index): ___ s — **KEY METRIC** (§5)_ _Node-type histogram:_ | Node label | Count | | Node label | Count | |---|---|---|---|---| | Function | _ | | Type | _ | | Method | _ | | Field | _ | | Class | _ | | Variable | _ | | Struct | _ | | Route | _ | | Interface | _ | | Module | _ | | Trait | _ | | Section | _ | | Enum | _ | | Macro | _ | | File | _ | | Folder | _ | | **Total nodes** | _ | | | | _Edge-type histogram (all 32 edge types, zeros included):_ | Edge type | Count | | Edge type | Count | |---|---|---|---|---| | CALLS | _ | | DEPENDS_ON | _ | | ASYNC_CALLS | _ | | USAGE | _ | | HTTP_CALLS | _ | | DATA_FLOWS | _ | | GRPC_CALLS | _ | | SEMANTICALLY_RELATED | _ | | GRAPHQL_CALLS | _ | | SIMILAR_TO | _ | | TRPC_CALLS | _ | | TESTS | _ | | DEFINES | _ | | TESTS_FILE | _ | | DEFINES_METHOD | _ | | INFRA_MAPS | _ | | IMPLEMENTS | _ | | FILE_CHANGES_WITH | _ | | INHERITS | _ | | CONTAINS_FILE | _ | | OVERRIDE | _ | | CONTAINS_FOLDER | _ | | DECORATES | _ | | CROSS_HTTP_CALLS *(LSP pass)* | _ | | IMPORTS | _ | | CROSS_ASYNC_CALLS *(LSP pass)* | _ | | HANDLES | _ | | CROSS_GRPC_CALLS *(LSP pass)* | _ | | CONFIGURES | _ | | CROSS_GRAPHQL_CALLS *(LSP pass)* | _ | | | | | CROSS_TRPC_CALLS *(LSP pass)* | _ | | | | | CROSS_CHANNEL *(LSP pass)* | _ | | **Total edges** | _ | | | | > Every row is reported even at `0` — a zero is critical signal: `CALLS=0` = broken call-extraction (`CALLS_MISSING`), `IMPLEMENTS`/`INHERITS`/`OVERRIDE=0` = no OO-relation edges, `SIMILAR_TO`/`SEMANTICALLY_RELATED=0` on a full-mode LSP language = semantic-index gap. `CROSS_*` rows are `0` for non-LSP languages and carry real counts only for the 9 LSP cross-repo pairs. **Output of the analysis:** `eval-results/graphql-graph.md`, `graphql-explorer.md`, `graphql-judged.json`. **Aggregates into:** D1-D4 cross-group rollups, D5 within Group E only, Group E, the graphql tier. --- ### 55. vue — E (Config/Data/Markup/Schema/Build/Template/HDL/Shader/Docs) **Repo:** vuejs/core (`/tmp/bench/vue`) **Symlink:** no **Indexed in:** fast **Why this repo:** vuejs/core is the canonical, highest-popularity Vue codebase (the framework's own monorepo), giving the broadest authentic sample of `.vue` SFC markup plus the template-AST definitions that drive it — exactly the substantial, idiomatic Group-E artifact the plan's "popular + representative" repo-selection criterion calls for. **The 5 questions** (bespoke; dimension in brackets): 1. **[D1 Definition/API]** "List the top-level template-AST node-type definitions in the compiler — e.g. the `NodeTypes` enum and node interfaces such as `ElementNode`, `InterpolationNode`, and `DirectiveNode` declared in `packages/compiler-core/src/ast.ts`. Are these top-level definitions enumerated?" (grep-findable: `enum NodeTypes`, `interface ElementNode`.) 2. **[D2 Relationship]** "Starting from the SFC descriptor produced by `parse` in `packages/compiler-sfc/src/parse.ts`, which template/script/style blocks reference (include) which other definitions across files — e.g. how the `