chore: import upstream snapshot with attribution
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Retrieval baselines bench
|
||||
|
||||
Reproducible per-baseline NDCG@10 + latency table across six
|
||||
candidate retrieval systems. The harness ships **six** adapters
|
||||
behind a single uniform Go-side surface; per-adapter `Available()`
|
||||
checks let `--smoke` verify wiring without paying for the heavy
|
||||
ones.
|
||||
|
||||
Supported adapters:
|
||||
|
||||
| Adapter | Type | Install |
|
||||
|----------------|---------------|--------------------------------------------------------------------|
|
||||
| ripgrep | Go-native | `brew install ripgrep` (or distro equivalent) |
|
||||
| probe | Go-native | `cargo install probe-ai` |
|
||||
| colgrep | Python | `pip install colgrep` (requires CUDA-capable GPU) |
|
||||
| grepai | Python | `pip install grepai-cli` |
|
||||
| coderankembed | Python+model | `pip install sentence-transformers transformers torch` (model ~440MB on first run) |
|
||||
| semble | Python | `pip install semble` |
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
# Smoke check: report which adapters are available right now
|
||||
go run ./bench/baselines -smoke
|
||||
|
||||
# Full table against the gortex repo itself, all adapters
|
||||
go run ./bench/baselines
|
||||
|
||||
# Just the locally-installable ones (ripgrep + probe)
|
||||
go run ./bench/baselines -against ripgrep,probe
|
||||
|
||||
# JSON output for downstream tooling
|
||||
go run ./bench/baselines -format json -json bench/results/baselines.json
|
||||
```
|
||||
|
||||
Flags:
|
||||
|
||||
- `-repo PATH` — corpus to query (default `.`)
|
||||
- `-queries PATH` — JSON query set (default `queries.json` here)
|
||||
- `-groundtruth PATH` — JSON per-query expected file paths (default
|
||||
`groundtruth.json` here)
|
||||
- `-against LIST` — comma-separated adapter names; unknown names
|
||||
error out so a typo doesn't silently drop a baseline
|
||||
- `-top-k N` — top-K per query (default 10, matches NDCG@10)
|
||||
- `-out PATH` — primary output (default stdout)
|
||||
- `-json PATH` — companion JSON metrics output
|
||||
- `-format markdown|json` — primary format
|
||||
- `-smoke` — skip runs; only probe `Available()` for each adapter
|
||||
- `-timeout DURATION` — per-adapter wall-clock cap (default 5m)
|
||||
|
||||
## How NDCG@10 is computed
|
||||
|
||||
For each query the harness computes:
|
||||
|
||||
```
|
||||
DCG@10 = Σ (gain_i / log2(i + 2)) for i in [0, min(K, |hits|))
|
||||
IDCG@10 = Σ (1 / log2(i + 2)) for i in [0, min(K, |expected|))
|
||||
NDCG@10 = DCG@10 / IDCG@10
|
||||
```
|
||||
|
||||
with `gain_i = 1` when `hits[i]` is in the ground-truth set,
|
||||
`0` otherwise. The reported number is the **mean** across queries
|
||||
with a non-empty ground-truth entry. Queries the adapter failed
|
||||
on (Error set) are skipped from the mean — they're shown in the
|
||||
per-query JSON for debugging but don't penalize the headline.
|
||||
|
||||
## Adding an adapter
|
||||
|
||||
1. Add a struct in `adapters.go` that satisfies the `adapter`
|
||||
interface (`Name`, `Available`, `Run`).
|
||||
2. Register it in `allAdapters()` — that single list drives the CLI,
|
||||
the smoke output, and `adapterByName`.
|
||||
3. For Python baselines: embed `pythonBaselineAdapter` and call
|
||||
`pythonModuleAvailable` / `runPythonModule` (or
|
||||
`runPythonScript` if you need a custom wrapper). Document the
|
||||
install in the table above.
|
||||
@@ -0,0 +1,333 @@
|
||||
// Package main defines the baseline adapter framework: a single Go
|
||||
// surface that uniformly invokes external retrieval baselines
|
||||
// (ripgrep, probe, ColGREP, grepai, CodeRankEmbed Hybrid, semble)
|
||||
// against a shared query set, then reports per-baseline NDCG@10 +
|
||||
// latency in a comparable table.
|
||||
//
|
||||
// Per-adapter contract:
|
||||
//
|
||||
// - Name() canonical baseline name, used in CLI / table cols
|
||||
// - Available() cheap probe: does this baseline run on this box?
|
||||
// - Run(queries) per-query result list; honest empty-on-failure
|
||||
//
|
||||
// Go-native baselines (ripgrep, probe) shell to their respective
|
||||
// binaries. Python-heavy ones (ColGREP, grepai, CodeRankEmbed) shell
|
||||
// to `python3 -m <module>` and require an explicit per-package install
|
||||
// that the harness documents in bench/baselines/README.md but does
|
||||
// NOT attempt automatically. This keeps `gortex eval baselines
|
||||
// --smoke` cheap on a fresh CI runner — it reports each baseline's
|
||||
// availability without trying to install anything.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// adapter is the interface every baseline implementation satisfies.
|
||||
type adapter interface {
|
||||
Name() string
|
||||
Available() (bool, string)
|
||||
Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult
|
||||
}
|
||||
|
||||
// queryResult is the per-query outcome from one adapter.
|
||||
type queryResult struct {
|
||||
Query string `json:"query"`
|
||||
Hits []string `json:"hits"`
|
||||
Latency time.Duration `json:"-"`
|
||||
LatencyMs float64 `json:"latency_ms"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// allAdapters returns the canonical adapter set, in stable order.
|
||||
// Adding a new baseline = adding one line here + one adapter
|
||||
// implementation. The smoke check uses this list directly so the
|
||||
// CLI never drifts from the registry.
|
||||
func allAdapters() []adapter {
|
||||
return []adapter{
|
||||
&ripgrepAdapter{},
|
||||
&probeAdapter{},
|
||||
&colgrepAdapter{},
|
||||
&grepaiAdapter{},
|
||||
&coderankAdapter{},
|
||||
&sembleAdapter{},
|
||||
}
|
||||
}
|
||||
|
||||
// adapterByName looks up one adapter by canonical name; nil when no
|
||||
// match (the caller should check before invoking).
|
||||
func adapterByName(name string) adapter {
|
||||
for _, a := range allAdapters() {
|
||||
if strings.EqualFold(a.Name(), name) {
|
||||
return a
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- ripgrep --------------------------------------------------------
|
||||
|
||||
type ripgrepAdapter struct{}
|
||||
|
||||
func (*ripgrepAdapter) Name() string { return "ripgrep" }
|
||||
|
||||
func (*ripgrepAdapter) Available() (bool, string) {
|
||||
if _, err := exec.LookPath("rg"); err != nil {
|
||||
return false, "rg not on PATH"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
func (*ripgrepAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
out := make([]queryResult, 0, len(queries))
|
||||
for _, q := range queries {
|
||||
start := time.Now()
|
||||
cmd := exec.CommandContext(ctx, "rg", "--files-with-matches", q, repoRoot)
|
||||
var buf bytes.Buffer
|
||||
cmd.Stdout = &buf
|
||||
cmd.Stderr = io.Discard
|
||||
err := cmd.Run()
|
||||
latency := time.Since(start)
|
||||
r := queryResult{Query: q, Latency: latency, LatencyMs: msFrom(latency)}
|
||||
if err != nil {
|
||||
if ee, ok := err.(*exec.ExitError); ok && ee.ExitCode() == 1 {
|
||||
// rg exit code 1 = no matches; not an error.
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
r.Error = err.Error()
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
hits := parseLines(buf.String(), repoRoot, topK)
|
||||
r.Hits = hits
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// --- probe ----------------------------------------------------------
|
||||
|
||||
type probeAdapter struct{}
|
||||
|
||||
func (*probeAdapter) Name() string { return "probe" }
|
||||
|
||||
func (*probeAdapter) Available() (bool, string) {
|
||||
if _, err := exec.LookPath("probe"); err != nil {
|
||||
return false, "probe binary not on PATH (install: cargo install probe-ai)"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
func (*probeAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
out := make([]queryResult, 0, len(queries))
|
||||
for _, q := range queries {
|
||||
start := time.Now()
|
||||
cmd := exec.CommandContext(ctx, "probe", "search", "--paths-only", q, repoRoot)
|
||||
var buf bytes.Buffer
|
||||
cmd.Stdout = &buf
|
||||
cmd.Stderr = io.Discard
|
||||
err := cmd.Run()
|
||||
latency := time.Since(start)
|
||||
r := queryResult{Query: q, Latency: latency, LatencyMs: msFrom(latency)}
|
||||
if err != nil {
|
||||
r.Error = err.Error()
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
r.Hits = parseLines(buf.String(), repoRoot, topK)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// --- ColGREP --------------------------------------------------------
|
||||
|
||||
type colgrepAdapter struct{ pythonBaselineAdapter }
|
||||
|
||||
func (*colgrepAdapter) Name() string { return "colgrep" }
|
||||
|
||||
func (a *colgrepAdapter) Available() (bool, string) {
|
||||
return a.pythonModuleAvailable("colgrep",
|
||||
"pip install colgrep (requires CUDA-capable GPU for the bundled ONNX model)")
|
||||
}
|
||||
|
||||
func (a *colgrepAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return a.runPythonModule(ctx, "colgrep", queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// --- grepai ---------------------------------------------------------
|
||||
|
||||
type grepaiAdapter struct{ pythonBaselineAdapter }
|
||||
|
||||
func (*grepaiAdapter) Name() string { return "grepai" }
|
||||
|
||||
func (a *grepaiAdapter) Available() (bool, string) {
|
||||
return a.pythonModuleAvailable("grepai_cli",
|
||||
"pip install grepai-cli")
|
||||
}
|
||||
|
||||
func (a *grepaiAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return a.runPythonModule(ctx, "grepai_cli", queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// --- CodeRankEmbed Hybrid -------------------------------------------
|
||||
|
||||
type coderankAdapter struct{ pythonBaselineAdapter }
|
||||
|
||||
func (*coderankAdapter) Name() string { return "coderankembed" }
|
||||
|
||||
func (a *coderankAdapter) Available() (bool, string) {
|
||||
// CodeRankEmbed ships as a model on Hugging Face; the bench
|
||||
// invoker is a small Python script we ship in
|
||||
// bench/baselines/python/coderankembed_runner.py. The script
|
||||
// requires `pip install sentence-transformers transformers
|
||||
// torch` and is documented in bench/baselines/README.md.
|
||||
return a.pythonModuleAvailable("sentence_transformers",
|
||||
"pip install sentence-transformers transformers torch (multi-GB model download on first run)")
|
||||
}
|
||||
|
||||
func (a *coderankAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return a.runPythonScript(ctx, "bench/baselines/python/coderankembed_runner.py", queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// --- semble ---------------------------------------------------------
|
||||
|
||||
type sembleAdapter struct{ pythonBaselineAdapter }
|
||||
|
||||
func (*sembleAdapter) Name() string { return "semble" }
|
||||
|
||||
func (a *sembleAdapter) Available() (bool, string) {
|
||||
return a.pythonModuleAvailable("semble",
|
||||
"pip install semble")
|
||||
}
|
||||
|
||||
func (a *sembleAdapter) Run(ctx context.Context, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return a.runPythonModule(ctx, "semble.cli", queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// --- shared Python-adapter plumbing ---------------------------------
|
||||
|
||||
// pythonBaselineAdapter embeds the common shell-out logic that all
|
||||
// Python-based baselines reuse. Per-baseline behaviour comes from
|
||||
// the module name + (optional) wrapper script path.
|
||||
type pythonBaselineAdapter struct{}
|
||||
|
||||
// pythonModuleAvailable runs `python3 -c "import <mod>"`. Returns
|
||||
// true when the import succeeds; otherwise returns false with the
|
||||
// install hint so the smoke output is actionable.
|
||||
func (pythonBaselineAdapter) pythonModuleAvailable(module, installHint string) (bool, string) {
|
||||
pythons := []string{"python3", "python"}
|
||||
var lastErr string
|
||||
for _, py := range pythons {
|
||||
if _, err := exec.LookPath(py); err != nil {
|
||||
continue
|
||||
}
|
||||
cmd := exec.Command(py, "-c", "import "+module)
|
||||
cmd.Stderr = io.Discard
|
||||
if err := cmd.Run(); err == nil {
|
||||
return true, ""
|
||||
} else {
|
||||
lastErr = err.Error()
|
||||
}
|
||||
}
|
||||
if lastErr == "" {
|
||||
return false, "python3 not on PATH"
|
||||
}
|
||||
return false, fmt.Sprintf("python module %q not importable (%s)", module, installHint)
|
||||
}
|
||||
|
||||
// runPythonModule invokes the module's CLI via `python3 -m <module>`,
|
||||
// forwarding the queries on stdin one per line. Mirrors how most
|
||||
// Python retrieval baselines we wrap accept input.
|
||||
func (pythonBaselineAdapter) runPythonModule(ctx context.Context, module string, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return pythonBaselineRun(ctx, []string{"-m", module}, queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// runPythonScript invokes a wrapper script with the same I/O
|
||||
// convention. The script lives under bench/baselines/python/ and is
|
||||
// shipped with the harness so each adapter has a known-good entry
|
||||
// point.
|
||||
func (pythonBaselineAdapter) runPythonScript(ctx context.Context, scriptPath string, queries []string, repoRoot string, topK int) []queryResult {
|
||||
return pythonBaselineRun(ctx, []string{scriptPath}, queries, repoRoot, topK)
|
||||
}
|
||||
|
||||
// pythonBaselineRun is the shared dispatcher: pipes queries to the
|
||||
// Python process on stdin and parses one-hit-per-line JSON from
|
||||
// stdout. Each baseline's wrapper is responsible for emitting the
|
||||
// canonical JSON shape: {"query": "...", "hits": ["path1", ...]}.
|
||||
func pythonBaselineRun(ctx context.Context, args []string, queries []string, repoRoot string, topK int) []queryResult {
|
||||
out := make([]queryResult, 0, len(queries))
|
||||
for _, q := range queries {
|
||||
start := time.Now()
|
||||
// We invoke the script once per query so a per-query
|
||||
// failure doesn't drag the whole run. Higher overhead than
|
||||
// streaming but fundamentally honest.
|
||||
baseArgs := append([]string{}, args...)
|
||||
baseArgs = append(baseArgs, "--repo", repoRoot, "--top-k", fmt.Sprintf("%d", topK), "--query", q)
|
||||
cmd := exec.CommandContext(ctx, "python3", baseArgs...)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
err := cmd.Run()
|
||||
latency := time.Since(start)
|
||||
r := queryResult{Query: q, Latency: latency, LatencyMs: msFrom(latency)}
|
||||
if err != nil {
|
||||
r.Error = fmt.Sprintf("%v: %s", err, strings.TrimSpace(stderr.String()))
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
r.Hits = parseLines(stdout.String(), repoRoot, topK)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// --- helpers --------------------------------------------------------
|
||||
|
||||
// parseLines splits the adapter's stdout into one hit per line,
|
||||
// strips the repo-root prefix so all adapters return repo-relative
|
||||
// paths, dedups while preserving order, and caps at topK.
|
||||
func parseLines(s, repoRoot string, topK int) []string {
|
||||
seen := map[string]bool{}
|
||||
out := []string{}
|
||||
for line := range strings.SplitSeq(s, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
rel := strings.TrimPrefix(line, repoRoot+"/")
|
||||
if seen[rel] {
|
||||
continue
|
||||
}
|
||||
seen[rel] = true
|
||||
out = append(out, rel)
|
||||
if topK > 0 && len(out) >= topK {
|
||||
break
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func msFrom(d time.Duration) float64 {
|
||||
return float64(d.Microseconds()) / 1000.0
|
||||
}
|
||||
|
||||
// adapterNames returns the names in canonical order for table
|
||||
// rendering / smoke output.
|
||||
func adapterNames() []string {
|
||||
all := allAdapters()
|
||||
names := make([]string, len(all))
|
||||
for i, a := range all {
|
||||
names[i] = a.Name()
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"comment": "Per-query expected file paths against the gortex repo. NDCG@10 = mean over queries of DCG_at_10 / IDCG_at_10 where gain=1 for hits in this list and 0 otherwise. Mirrors the token-efficiency ground truth so the two benches are directly comparable.",
|
||||
"queries": {
|
||||
"AddObservation": [
|
||||
"internal/savings/store.go"
|
||||
],
|
||||
"IsSymbolQuery": [
|
||||
"internal/search/rerank/query_kind.go"
|
||||
],
|
||||
"FileCoherenceSignal": [
|
||||
"internal/search/rerank/signals_file_coherence.go"
|
||||
],
|
||||
"alphaFuse": [
|
||||
"internal/search/hybrid.go"
|
||||
],
|
||||
"savings dashboard rendering": [
|
||||
"cmd/gortex/savings.go",
|
||||
"internal/savings/events.go"
|
||||
],
|
||||
"rerank pipeline default signals": [
|
||||
"internal/search/rerank/pipeline.go"
|
||||
],
|
||||
"Indexer Index method": [
|
||||
"internal/indexer/indexer.go"
|
||||
],
|
||||
"graph build edges": [
|
||||
"internal/graph/graph.go"
|
||||
],
|
||||
"MCP server start": [
|
||||
"internal/mcp/server.go",
|
||||
"cmd/gortex/mcp.go"
|
||||
],
|
||||
"token counting tiktoken": [
|
||||
"internal/tokens/tokens.go"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
// Command baselines runs the per-baseline retrieval comparison:
|
||||
// dispatch the same query set through each registered adapter,
|
||||
// compare the per-query hit lists against a shared ground truth,
|
||||
// and emit a per-adapter NDCG@10 + latency table.
|
||||
//
|
||||
// Smoke mode (`--smoke`) skips actual runs and reports which adapters
|
||||
// are available on the local box — useful from CI to verify wiring
|
||||
// without paying for the heavy Python baselines.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
repo := flag.String("repo", ".", "indexed corpus path")
|
||||
queriesPath := flag.String("queries", "bench/baselines/queries.json", "JSON query set")
|
||||
truthPath := flag.String("groundtruth", "bench/baselines/groundtruth.json", "JSON per-query expected file paths")
|
||||
against := flag.String("against", "ripgrep,probe,colgrep,grepai,coderankembed,semble", "comma-separated adapter names to run (default: all)")
|
||||
topK := flag.Int("top-k", 10, "top-K candidates per query (NDCG@10 uses K=10)")
|
||||
out := flag.String("out", "", "output table path (default stdout)")
|
||||
jsonOut := flag.String("json", "", "optional JSON metrics output")
|
||||
format := flag.String("format", "markdown", "markdown | json")
|
||||
smoke := flag.Bool("smoke", false, "skip runs; only probe availability of each requested adapter")
|
||||
timeout := flag.Duration("timeout", 5*time.Minute, "per-adapter wall-clock cap")
|
||||
flag.Parse()
|
||||
|
||||
requested, err := resolveAdapters(*against)
|
||||
if err != nil {
|
||||
die("%v", err)
|
||||
}
|
||||
|
||||
if *smoke {
|
||||
smokeReport(requested, *format, *out)
|
||||
return
|
||||
}
|
||||
|
||||
absRepo, err := filepath.Abs(*repo)
|
||||
if err != nil {
|
||||
die("repo path: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(absRepo); err != nil {
|
||||
die("repo path: %v", err)
|
||||
}
|
||||
|
||||
queries, err := loadQueries(*queriesPath)
|
||||
if err != nil {
|
||||
die("queries: %v", err)
|
||||
}
|
||||
truth, err := loadGroundTruth(*truthPath)
|
||||
if err != nil {
|
||||
die("groundtruth: %v", err)
|
||||
}
|
||||
|
||||
rows := make([]adapterRow, 0, len(requested))
|
||||
for _, a := range requested {
|
||||
fmt.Fprintf(os.Stderr, "[baselines] %s ... ", a.Name())
|
||||
avail, why := a.Available()
|
||||
if !avail {
|
||||
fmt.Fprintf(os.Stderr, "skipped (%s)\n", why)
|
||||
rows = append(rows, adapterRow{
|
||||
Adapter: a.Name(),
|
||||
Skipped: why,
|
||||
})
|
||||
continue
|
||||
}
|
||||
t0 := time.Now()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), *timeout)
|
||||
results := a.Run(ctx, queries, absRepo, *topK)
|
||||
cancel()
|
||||
elapsed := time.Since(t0)
|
||||
|
||||
row := adapterRow{
|
||||
Adapter: a.Name(),
|
||||
PerQuery: results,
|
||||
TotalDuration: elapsed.String(),
|
||||
}
|
||||
row.MedianLatencyMs = medianLatency(results)
|
||||
row.NDCGAt10 = ndcgAt10(results, truth, *topK)
|
||||
fmt.Fprintf(os.Stderr, "NDCG@10 %.3f · median %.1fms · total %.1fs\n",
|
||||
row.NDCGAt10, row.MedianLatencyMs, elapsed.Seconds())
|
||||
rows = append(rows, row)
|
||||
}
|
||||
|
||||
var primary []byte
|
||||
switch strings.ToLower(*format) {
|
||||
case "markdown", "md":
|
||||
primary = []byte(renderMarkdown(rows))
|
||||
case "json":
|
||||
primary = mustMarshalJSON(rows)
|
||||
default:
|
||||
die("unknown --format %q", *format)
|
||||
}
|
||||
if err := writeOutput(*out, primary); err != nil {
|
||||
die("write output: %v", err)
|
||||
}
|
||||
if *jsonOut != "" {
|
||||
if err := writeOutput(*jsonOut, mustMarshalJSON(rows)); err != nil {
|
||||
die("write json: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// adapterRow is the per-baseline outcome with the columns the
|
||||
// published table cares about.
|
||||
type adapterRow struct {
|
||||
Adapter string `json:"adapter"`
|
||||
NDCGAt10 float64 `json:"ndcg_at_10"`
|
||||
MedianLatencyMs float64 `json:"median_latency_ms"`
|
||||
TotalDuration string `json:"total_duration"`
|
||||
PerQuery []queryResult `json:"per_query,omitempty"`
|
||||
Skipped string `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// resolveAdapters expands the --against flag into the requested
|
||||
// adapter set, preserving the caller's order. Unknown names error
|
||||
// out so a typo doesn't silently drop a baseline.
|
||||
func resolveAdapters(spec string) ([]adapter, error) {
|
||||
if strings.TrimSpace(spec) == "" {
|
||||
return allAdapters(), nil
|
||||
}
|
||||
var out []adapter
|
||||
for tok := range strings.SplitSeq(spec, ",") {
|
||||
name := strings.TrimSpace(tok)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
a := adapterByName(name)
|
||||
if a == nil {
|
||||
return nil, fmt.Errorf("unknown adapter %q (known: %s)",
|
||||
name, strings.Join(adapterNames(), ", "))
|
||||
}
|
||||
out = append(out, a)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return allAdapters(), nil
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// smokeReport prints one row per requested adapter showing
|
||||
// available / not-available with the install hint. Output respects
|
||||
// --format; default markdown.
|
||||
func smokeReport(adapters []adapter, format, out string) {
|
||||
if strings.ToLower(format) == "json" {
|
||||
rows := make([]map[string]any, 0, len(adapters))
|
||||
for _, a := range adapters {
|
||||
avail, why := a.Available()
|
||||
row := map[string]any{
|
||||
"adapter": a.Name(),
|
||||
"available": avail,
|
||||
}
|
||||
if why != "" {
|
||||
row["why"] = why
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
raw, _ := json.MarshalIndent(rows, "", " ")
|
||||
_ = writeOutput(out, append(raw, '\n'))
|
||||
return
|
||||
}
|
||||
var b strings.Builder
|
||||
fmt.Fprintln(&b, "# Baseline-adapter smoke check")
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, "| adapter | available | reason |")
|
||||
fmt.Fprintln(&b, "|---------|:---------:|--------|")
|
||||
for _, a := range adapters {
|
||||
avail, why := a.Available()
|
||||
mark := "✗"
|
||||
if avail {
|
||||
mark = "✓"
|
||||
}
|
||||
if why == "" {
|
||||
why = "—"
|
||||
}
|
||||
fmt.Fprintf(&b, "| %s | %s | %s |\n", a.Name(), mark, why)
|
||||
}
|
||||
_ = writeOutput(out, []byte(b.String()))
|
||||
}
|
||||
|
||||
// ndcgAt10 computes NDCG@10 across the result set. Per query: gain
|
||||
// is 1 for ground-truth hits, 0 otherwise; DCG = sum gain_i /
|
||||
// log2(i+2); IDCG = DCG of the perfect ordering. Mean across
|
||||
// queries; 0 when the result set is empty.
|
||||
func ndcgAt10(results []queryResult, truth map[string][]string, k int) float64 {
|
||||
if len(results) == 0 {
|
||||
return 0
|
||||
}
|
||||
var sum float64
|
||||
counted := 0
|
||||
for _, r := range results {
|
||||
if r.Error != "" {
|
||||
continue
|
||||
}
|
||||
expected := truth[r.Query]
|
||||
if len(expected) == 0 {
|
||||
continue
|
||||
}
|
||||
expSet := map[string]bool{}
|
||||
for _, e := range expected {
|
||||
expSet[e] = true
|
||||
}
|
||||
// DCG@k
|
||||
var dcg float64
|
||||
hits := r.Hits
|
||||
if len(hits) > k {
|
||||
hits = hits[:k]
|
||||
}
|
||||
for i, h := range hits {
|
||||
if expSet[h] {
|
||||
dcg += 1.0 / math.Log2(float64(i+2))
|
||||
}
|
||||
}
|
||||
// IDCG@k: best case is min(len(expected), k) hits at the top.
|
||||
idealHits := min(len(expected), k)
|
||||
var idcg float64
|
||||
for i := range idealHits {
|
||||
idcg += 1.0 / math.Log2(float64(i+2))
|
||||
}
|
||||
if idcg == 0 {
|
||||
continue
|
||||
}
|
||||
sum += dcg / idcg
|
||||
counted++
|
||||
}
|
||||
if counted == 0 {
|
||||
return 0
|
||||
}
|
||||
return sum / float64(counted)
|
||||
}
|
||||
|
||||
func medianLatency(results []queryResult) float64 {
|
||||
vs := make([]float64, 0, len(results))
|
||||
for _, r := range results {
|
||||
if r.Error == "" {
|
||||
vs = append(vs, r.LatencyMs)
|
||||
}
|
||||
}
|
||||
if len(vs) == 0 {
|
||||
return 0
|
||||
}
|
||||
sort.Float64s(vs)
|
||||
return vs[len(vs)/2]
|
||||
}
|
||||
|
||||
// --- rendering ------------------------------------------------------
|
||||
|
||||
func renderMarkdown(rows []adapterRow) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintln(&b, "# Retrieval-baselines NDCG@10 + speed")
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, "_NDCG@10 against the ground-truth set + median per-query latency for each baseline. Adapters reporting `skipped` weren't available on this box; see the install hints in `bench/baselines/README.md`._")
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, "| adapter | NDCG@10 | median latency | total | status |")
|
||||
fmt.Fprintln(&b, "|---------|--------:|---------------:|------:|--------|")
|
||||
for _, r := range rows {
|
||||
if r.Skipped != "" {
|
||||
fmt.Fprintf(&b, "| %s | — | — | — | skipped: %s |\n", r.Adapter, r.Skipped)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "| %s | %.3f | %s | %s | ✓ |\n",
|
||||
r.Adapter, r.NDCGAt10, fmtMs(r.MedianLatencyMs), r.TotalDuration)
|
||||
}
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, summaryLine(rows))
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func summaryLine(rows []adapterRow) string {
|
||||
ran := 0
|
||||
bestName := ""
|
||||
var bestScore float64
|
||||
for _, r := range rows {
|
||||
if r.Skipped != "" {
|
||||
continue
|
||||
}
|
||||
ran++
|
||||
if r.NDCGAt10 > bestScore {
|
||||
bestScore = r.NDCGAt10
|
||||
bestName = r.Adapter
|
||||
}
|
||||
}
|
||||
if ran == 0 {
|
||||
return "_no adapters available — install at least one baseline (see bench/baselines/README.md)_"
|
||||
}
|
||||
return fmt.Sprintf("**Summary:** %d/%d adapter(s) ran. Best NDCG@10: %s @ %.3f.",
|
||||
ran, len(rows), bestName, bestScore)
|
||||
}
|
||||
|
||||
func fmtMs(v float64) string {
|
||||
if v == 0 {
|
||||
return "—"
|
||||
}
|
||||
if v < 1 {
|
||||
return fmt.Sprintf("%.2fms", v)
|
||||
}
|
||||
if v < 1000 {
|
||||
return fmt.Sprintf("%.1fms", v)
|
||||
}
|
||||
return fmt.Sprintf("%.2fs", v/1000.0)
|
||||
}
|
||||
|
||||
// --- helpers --------------------------------------------------------
|
||||
|
||||
func die(format string, args ...any) {
|
||||
fmt.Fprintf(os.Stderr, "baselines: "+format+"\n", args...)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
func writeOutput(path string, body []byte) error {
|
||||
if path == "" {
|
||||
_, err := os.Stdout.Write(body)
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, body, 0o644)
|
||||
}
|
||||
|
||||
func mustMarshalJSON(rows []adapterRow) []byte {
|
||||
b, err := json.MarshalIndent(rows, "", " ")
|
||||
if err != nil {
|
||||
die("marshal json: %v", err)
|
||||
}
|
||||
return append(b, '\n')
|
||||
}
|
||||
|
||||
func loadQueries(path string) ([]string, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc struct {
|
||||
Queries []string `json:"queries"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
return doc.Queries, nil
|
||||
}
|
||||
|
||||
func loadGroundTruth(path string) (map[string][]string, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc struct {
|
||||
Queries map[string][]string `json:"queries"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return nil, fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
if doc.Queries == nil {
|
||||
doc.Queries = map[string][]string{}
|
||||
}
|
||||
return doc.Queries, nil
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAllAdapters_RegisteredInStableOrder(t *testing.T) {
|
||||
got := allAdapters()
|
||||
want := []string{"ripgrep", "probe", "colgrep", "grepai", "coderankembed", "semble"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("adapter count = %d, want %d", len(got), len(want))
|
||||
}
|
||||
for i, a := range got {
|
||||
if a.Name() != want[i] {
|
||||
t.Errorf("adapter[%d] = %q, want %q", i, a.Name(), want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdapterByName(t *testing.T) {
|
||||
if a := adapterByName("ripgrep"); a == nil || a.Name() != "ripgrep" {
|
||||
t.Errorf("adapterByName(ripgrep) = %v", a)
|
||||
}
|
||||
if a := adapterByName("RIPGREP"); a == nil { // case-insensitive
|
||||
t.Error("adapterByName should be case-insensitive")
|
||||
}
|
||||
if a := adapterByName("nonexistent"); a != nil {
|
||||
t.Errorf("adapterByName(nonexistent) should be nil, got %v", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveAdapters_EmptyMeansAll(t *testing.T) {
|
||||
got, err := resolveAdapters("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != len(allAdapters()) {
|
||||
t.Errorf("empty spec = %d adapters, want %d", len(got), len(allAdapters()))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveAdapters_Subset(t *testing.T) {
|
||||
got, err := resolveAdapters("ripgrep,probe")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 || got[0].Name() != "ripgrep" || got[1].Name() != "probe" {
|
||||
t.Errorf("subset = %v, want [ripgrep, probe] in order", names(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveAdapters_UnknownErrors(t *testing.T) {
|
||||
if _, err := resolveAdapters("ripgrep,bogus"); err == nil {
|
||||
t.Error("expected error for unknown adapter")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDCGAt10_PerfectRanking(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{Query: "q", Hits: []string{"a", "b", "c"}},
|
||||
}
|
||||
truth := map[string][]string{"q": {"a", "b"}}
|
||||
got := ndcgAt10(results, truth, 10)
|
||||
// IDCG = 1/log2(2) + 1/log2(3) = 1 + 0.6309 = 1.6309
|
||||
// DCG = 1/log2(2) + 1/log2(3) + 0 = same = 1.6309
|
||||
// NDCG = 1.0
|
||||
if got < 0.99 || got > 1.01 {
|
||||
t.Errorf("perfect ranking NDCG@10 = %.4f, want 1.0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDCGAt10_WrongOrderPenalized(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{Query: "q", Hits: []string{"miss1", "miss2", "a", "b"}},
|
||||
}
|
||||
truth := map[string][]string{"q": {"a", "b"}}
|
||||
got := ndcgAt10(results, truth, 10)
|
||||
// DCG = 0 + 0 + 1/log2(4) + 1/log2(5) ≈ 0.5 + 0.431 = 0.931
|
||||
// IDCG = 1/log2(2) + 1/log2(3) ≈ 1.0 + 0.631 = 1.631
|
||||
// NDCG = 0.931/1.631 ≈ 0.571
|
||||
if got > 0.7 {
|
||||
t.Errorf("wrong-order NDCG@10 = %.4f, expected <0.7", got)
|
||||
}
|
||||
if got < 0.4 {
|
||||
t.Errorf("wrong-order NDCG@10 = %.4f, expected >0.4", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDCGAt10_AllMissesScoreZero(t *testing.T) {
|
||||
results := []queryResult{{Query: "q", Hits: []string{"x", "y"}}}
|
||||
truth := map[string][]string{"q": {"a", "b"}}
|
||||
if got := ndcgAt10(results, truth, 10); got != 0 {
|
||||
t.Errorf("all-miss NDCG@10 = %.4f, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDCGAt10_EmptyTruthSkipped(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{Query: "q1", Hits: []string{"a"}},
|
||||
{Query: "q2", Hits: []string{"a"}},
|
||||
}
|
||||
truth := map[string][]string{"q2": {"a"}}
|
||||
// q1 has no truth → skipped; q2 has 1 hit at position 0 → NDCG=1.
|
||||
got := ndcgAt10(results, truth, 10)
|
||||
if math.Abs(got-1.0) > 0.01 {
|
||||
t.Errorf("mixed-truth NDCG@10 = %.4f, want ~1.0 (q1 should be skipped)", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDCGAt10_ErroredQuerySkipped(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{Query: "q1", Error: "boom"},
|
||||
{Query: "q2", Hits: []string{"a"}},
|
||||
}
|
||||
truth := map[string][]string{"q1": {"a"}, "q2": {"a"}}
|
||||
// q1 errored → skipped; only q2 counts → NDCG=1.
|
||||
got := ndcgAt10(results, truth, 10)
|
||||
if math.Abs(got-1.0) > 0.01 {
|
||||
t.Errorf("errored-skipped NDCG@10 = %.4f, want ~1.0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMedianLatency(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{LatencyMs: 30},
|
||||
{LatencyMs: 10},
|
||||
{LatencyMs: 20},
|
||||
}
|
||||
if got := medianLatency(results); got != 20 {
|
||||
t.Errorf("median = %.2f, want 20", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMedianLatency_ErrorsExcluded(t *testing.T) {
|
||||
results := []queryResult{
|
||||
{LatencyMs: 30, Error: "boom"},
|
||||
{LatencyMs: 10},
|
||||
{LatencyMs: 20},
|
||||
}
|
||||
if got := medianLatency(results); got != 20 {
|
||||
t.Errorf("median (errors excluded) = %.2f, want 20", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLines_DedupAndCap(t *testing.T) {
|
||||
body := `/repo/a.go
|
||||
/repo/b.go
|
||||
/repo/a.go
|
||||
/repo/c.go
|
||||
/repo/d.go
|
||||
`
|
||||
got := parseLines(body, "/repo", 3)
|
||||
want := []string{"a.go", "b.go", "c.go"}
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("got[%d] = %q, want %q", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLines_TopKZeroMeansUnlimited(t *testing.T) {
|
||||
body := "a\nb\nc\nd\ne\nf\ng\n"
|
||||
got := parseLines(body, "", 0)
|
||||
if len(got) != 7 {
|
||||
t.Errorf("got %d lines, want 7 (topK=0 unlimited)", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderMarkdown_HasHeaderAndRows(t *testing.T) {
|
||||
rows := []adapterRow{
|
||||
{Adapter: "ripgrep", NDCGAt10: 0.45, MedianLatencyMs: 12.3, TotalDuration: "1.2s"},
|
||||
{Adapter: "colgrep", Skipped: "python module not importable"},
|
||||
}
|
||||
md := renderMarkdown(rows)
|
||||
for _, want := range []string{
|
||||
"# Retrieval-baselines NDCG@10",
|
||||
"| ripgrep |",
|
||||
"0.450",
|
||||
"| colgrep |",
|
||||
"skipped:",
|
||||
"**Summary:**",
|
||||
} {
|
||||
if !strings.Contains(md, want) {
|
||||
t.Errorf("markdown missing %q\n----\n%s", want, md)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSmokeReport_Markdown(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
out := filepath.Join(dir, "smoke.md")
|
||||
smokeReport([]adapter{&ripgrepAdapter{}, &colgrepAdapter{}}, "markdown", out)
|
||||
raw, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body := string(raw)
|
||||
for _, want := range []string{
|
||||
"# Baseline-adapter smoke check",
|
||||
"| ripgrep |",
|
||||
"| colgrep |",
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("smoke markdown missing %q\n----\n%s", want, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSmokeReport_JSON(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
out := filepath.Join(dir, "smoke.json")
|
||||
smokeReport([]adapter{&ripgrepAdapter{}}, "json", out)
|
||||
raw, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var got []map[string]any
|
||||
if err := json.Unmarshal(raw, &got); err != nil {
|
||||
t.Fatalf("smoke JSON unparseable: %v\n%s", err, raw)
|
||||
}
|
||||
if len(got) != 1 || got[0]["adapter"] != "ripgrep" {
|
||||
t.Errorf("smoke JSON shape wrong: %v", got)
|
||||
}
|
||||
if _, ok := got[0]["available"]; !ok {
|
||||
t.Errorf("smoke JSON missing 'available' field: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRipgrepAdapter_NoMatchesReturnsEmpty(t *testing.T) {
|
||||
a := &ripgrepAdapter{}
|
||||
avail, _ := a.Available()
|
||||
if !avail {
|
||||
t.Skip("rg not installed on this box")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "f.go"), []byte("package x\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
out := a.Run(ctx, []string{"definitely_not_in_the_corpus_XYZ"}, dir, 10)
|
||||
if len(out) != 1 {
|
||||
t.Fatalf("got %d results, want 1", len(out))
|
||||
}
|
||||
if out[0].Error != "" {
|
||||
t.Errorf("no-match should not error, got %q", out[0].Error)
|
||||
}
|
||||
if len(out[0].Hits) != 0 {
|
||||
t.Errorf("no-match should yield empty hits, got %v", out[0].Hits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadQueries(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "q.json")
|
||||
if err := os.WriteFile(path, []byte(`{"queries":["a","b","c"]}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := loadQueries(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Errorf("got %d, want 3", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadGroundTruth(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "gt.json")
|
||||
body := `{"queries":{"q":["a.go"]}}`
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := loadGroundTruth(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got["q"]) != 1 || got["q"][0] != "a.go" {
|
||||
t.Errorf("got %v, want q:[a.go]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func names(adapters []adapter) []string {
|
||||
out := make([]string, len(adapters))
|
||||
for i, a := range adapters {
|
||||
out[i] = a.Name()
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CodeRankEmbed Hybrid baseline runner.
|
||||
|
||||
Wrapper that lets the Go-side bench/baselines harness invoke
|
||||
CodeRankEmbed Hybrid without per-baseline Go code growing model-
|
||||
download logic. Usage:
|
||||
|
||||
python3 bench/baselines/python/coderankembed_runner.py \\
|
||||
--repo PATH --query "validateToken" --top-k 10
|
||||
|
||||
Emits one repo-relative path per line on stdout. Errors go to
|
||||
stderr; non-zero exit when the model isn't available.
|
||||
|
||||
Install: `pip install sentence-transformers transformers torch`.
|
||||
First run downloads the CodeRankEmbed model (~440 MB).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--repo", required=True, help="indexed corpus path")
|
||||
ap.add_argument("--query", required=True, help="single query string")
|
||||
ap.add_argument("--top-k", type=int, default=10)
|
||||
args = ap.parse_args()
|
||||
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
except ImportError as e:
|
||||
print(
|
||||
f"coderankembed_runner: missing dependency ({e}). "
|
||||
"pip install sentence-transformers transformers torch",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
model = SentenceTransformer("nomic-ai/CodeRankEmbed", trust_remote_code=True)
|
||||
|
||||
# Index every file under repo (cheap for sub-million LoC; the
|
||||
# ground-truth fixture is the gortex repo itself).
|
||||
repo = Path(args.repo).resolve()
|
||||
paths: list[Path] = []
|
||||
texts: list[str] = []
|
||||
for p in repo.rglob("*"):
|
||||
if not p.is_file():
|
||||
continue
|
||||
if any(seg.startswith(".") for seg in p.relative_to(repo).parts):
|
||||
continue
|
||||
if p.suffix.lower() not in {
|
||||
".go", ".py", ".ts", ".tsx", ".js", ".jsx", ".rs",
|
||||
".java", ".kt", ".swift", ".rb", ".cs", ".cpp", ".c",
|
||||
".h", ".hpp", ".md", ".yaml", ".yml", ".json",
|
||||
}:
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="ignore")
|
||||
except OSError:
|
||||
continue
|
||||
if not text.strip():
|
||||
continue
|
||||
paths.append(p)
|
||||
texts.append(text[:8000]) # truncate to keep the embed cheap
|
||||
|
||||
if not texts:
|
||||
print(
|
||||
"coderankembed_runner: no indexable files under repo",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
embeds = model.encode(texts, show_progress_bar=False, convert_to_numpy=True)
|
||||
qe = model.encode([args.query], show_progress_bar=False, convert_to_numpy=True)[0]
|
||||
|
||||
# Cosine similarity → rank.
|
||||
import numpy as np
|
||||
|
||||
sims = embeds @ qe / (
|
||||
(np.linalg.norm(embeds, axis=1) * np.linalg.norm(qe)) + 1e-12
|
||||
)
|
||||
order = np.argsort(-sims)[: args.top_k]
|
||||
for idx in order:
|
||||
rel = paths[idx].relative_to(repo)
|
||||
print(rel.as_posix())
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"comment": "Query set for the retrieval-baselines NDCG@10 bench. Same shape as the token-efficiency set so the two benches stay comparable: an adapter that scores well here should also produce low tokens-per-response there.",
|
||||
"queries": [
|
||||
"AddObservation",
|
||||
"IsSymbolQuery",
|
||||
"FileCoherenceSignal",
|
||||
"alphaFuse",
|
||||
"savings dashboard rendering",
|
||||
"rerank pipeline default signals",
|
||||
"Indexer Index method",
|
||||
"graph build edges",
|
||||
"MCP server start",
|
||||
"token counting tiktoken"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
# Daemon-mode MCP-tool latency
|
||||
|
||||
Per-tool p50 / p95 / p99 latency for the production MCP dispatch
|
||||
path. Builds an in-process MCP server against a target corpus,
|
||||
fires N `Handler.CallToolStrict` invocations per tool, aggregates
|
||||
latencies into a published table.
|
||||
|
||||
## What it measures
|
||||
|
||||
- **Handler-end-to-end latency** for each MCP tool: JSON arg parse
|
||||
→ tool dispatch → handler logic → response encode. Same code
|
||||
path the production stdio / HTTP / daemon-socket front-ends use.
|
||||
- **Per-tool spread**: cheap tools (`graph_stats`, `get_callers`)
|
||||
separate from heavy ones (`smart_context`, `get_repo_outline`)
|
||||
so the published table shows realistic operating envelope.
|
||||
|
||||
## What it does NOT measure
|
||||
|
||||
- **Stdio framing** (gortex mcp's pipe overhead)
|
||||
- **Daemon socket dispatch** (gortex daemon's UNIX socket / HTTP
|
||||
ingress overhead)
|
||||
- **Network RTT** (if reaching the daemon remotely)
|
||||
|
||||
Each adds a roughly constant ~0.1-1 ms per call on a warm pipe;
|
||||
the handler latency below dominates user-perceived response time.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
# Default: index `.` and fire 200 iters per tool
|
||||
go run ./bench/daemon-latency
|
||||
|
||||
# Higher iter count for tighter percentiles
|
||||
go run ./bench/daemon-latency -iter 500
|
||||
|
||||
# Specific subset of tools (useful for tuning one signal)
|
||||
go run ./bench/daemon-latency -tools graph_stats,search_symbols
|
||||
|
||||
# CSV / JSON outputs for downstream tooling
|
||||
go run ./bench/daemon-latency -csv bench/results/dl.csv -json bench/results/dl.json
|
||||
```
|
||||
|
||||
Flags:
|
||||
|
||||
- `-repo PATH` — corpus to index (default `.`)
|
||||
- `-iter N` — iterations per tool (default 200; warm-up of N/10
|
||||
is added on top)
|
||||
- `-tools LIST` — comma-separated subset
|
||||
- `-out PATH` — primary output (default stdout)
|
||||
- `-csv PATH` / `-json PATH` — companion outputs
|
||||
- `-format markdown|csv|json` — primary format
|
||||
|
||||
Or via the CLI surface:
|
||||
|
||||
```sh
|
||||
gortex bench daemon-latency --out-dir bench/results
|
||||
```
|
||||
|
||||
## Tools benchmarked
|
||||
|
||||
| tool | shape |
|
||||
|------|-------|
|
||||
| `graph_stats` | no-arg snapshot; cheap |
|
||||
| `search_symbols` | 1 query arg; rotated through 10 fixtures so a per-query cache doesn't trivially hit |
|
||||
| `get_symbol_source` | 1 id arg; pinned to a sampled function from the indexed graph |
|
||||
| `get_callers` | 1 id arg + limit |
|
||||
| `find_usages` | 1 id arg |
|
||||
| `get_file_summary` | 1 path arg; pinned to a sampled file |
|
||||
| `smart_context` | 1 task arg; expensive, fewer iters per cycle |
|
||||
| `get_repo_outline` | no-arg; walks whole graph |
|
||||
|
||||
Sampled targets are picked once at start so each tool sees the
|
||||
same target across iterations — the per-call latency reflects
|
||||
handler arithmetic, not target lookup.
|
||||
|
||||
## Methodology
|
||||
|
||||
- Warm-up of `iter/10` (min 5) per tool primes any lazy
|
||||
initialisation in the handler / graph before the measured loop
|
||||
starts.
|
||||
- Per-iteration latency captured via `time.Since(start)` with
|
||||
μs precision.
|
||||
- Percentiles computed via the nearest-rank method:
|
||||
`idx = (pct × n) / 100`. For N=200 → p95=sorted[190].
|
||||
- Errors are counted in `error_rate` but their latencies are
|
||||
still measured (an error path that takes 3× the happy-path
|
||||
time is itself a signal).
|
||||
|
||||
## Honest caveats
|
||||
|
||||
- Numbers are operator-machine-specific. Absolute values vary 2-5×
|
||||
across hardware classes; the **relative spread** between tools
|
||||
(cheap vs heavy) is what publishes reproducibly.
|
||||
- Cold-cache effects show up most in `search_symbols` (BM25
|
||||
re-ranks under load) and `smart_context` (assembles fresh
|
||||
context each call). Warm-up reduces but doesn't eliminate them.
|
||||
- Smoke run on the gortex repo (71k nodes, Apple M3 Max):
|
||||
- `graph_stats` p50 4.2ms · p95 5.5ms
|
||||
- `search_symbols` p50 1.2ms · p95 22.4ms
|
||||
- `get_symbol_source` p50 0.19ms · p95 0.9ms
|
||||
- `get_callers` / `find_usages` p50 < 0.02ms (graph lookup)
|
||||
- `smart_context` p50 1.5ms · p95 24ms
|
||||
- `get_repo_outline` p50 60ms · p95 217ms
|
||||
|
||||
Median p95 across tools: 5.5 ms. Median p99: 5.9 ms.
|
||||
@@ -0,0 +1,447 @@
|
||||
// Command daemon-latency measures per-tool MCP dispatch latency.
|
||||
// Builds an in-process MCP server against a target corpus, fires N
|
||||
// `CallTool` invocations per tool, reports p50 / p95 / p99 per tool
|
||||
// and a top-line summary.
|
||||
//
|
||||
// What it measures: tool-handler latency end-to-end through the
|
||||
// real MCP dispatch path (`Handler.CallTool` invoked via the same
|
||||
// `server.MCPServer` the production stdio / HTTP / daemon
|
||||
// front-ends use). What it does NOT measure: stdio framing,
|
||||
// daemon socket dispatch, JSON-RPC envelope overhead. Those add a
|
||||
// small constant per call (typically <1 ms on a warm pipe); the
|
||||
// handler latency dominates the user-perceived response time.
|
||||
//
|
||||
// The bench therefore reflects "daemon-mode handler cost", which
|
||||
// is the load-bearing number for the daemon-latency publication.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"github.com/zzet/gortex/internal/config"
|
||||
"github.com/zzet/gortex/internal/graph"
|
||||
"github.com/zzet/gortex/internal/indexer"
|
||||
gortexmcp "github.com/zzet/gortex/internal/mcp"
|
||||
"github.com/zzet/gortex/internal/parser"
|
||||
"github.com/zzet/gortex/internal/parser/languages"
|
||||
"github.com/zzet/gortex/internal/query"
|
||||
internalserver "github.com/zzet/gortex/internal/server"
|
||||
)
|
||||
|
||||
// toolCall is one synthetic MCP request the bench fires per
|
||||
// iteration. ArgsFn lets the bench vary the args across iterations
|
||||
// (e.g. different query strings) so the dispatch path isn't
|
||||
// trivially memoised by an upstream cache.
|
||||
type toolCall struct {
|
||||
Tool string
|
||||
ArgsFn func(iter int) map[string]any
|
||||
WarmupN int
|
||||
IterN int
|
||||
// SkipIfMissing lets a tool opt out when its substrate isn't
|
||||
// in the indexed graph (e.g. nothing to call get_callers on).
|
||||
SkipIfMissing func(g *graph.Graph) bool
|
||||
}
|
||||
|
||||
// result captures the per-tool aggregate the bench publishes.
|
||||
type result struct {
|
||||
Tool string `json:"tool"`
|
||||
Iters int `json:"iters"`
|
||||
P50Ms float64 `json:"p50_ms"`
|
||||
P95Ms float64 `json:"p95_ms"`
|
||||
P99Ms float64 `json:"p99_ms"`
|
||||
MeanMs float64 `json:"mean_ms"`
|
||||
MaxMs float64 `json:"max_ms"`
|
||||
ErrorRate float64 `json:"error_rate"`
|
||||
Skipped string `json:"skipped,omitempty"`
|
||||
Started time.Time `json:"-"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
repo := flag.String("repo", ".", "corpus to index for the bench")
|
||||
iter := flag.Int("iter", 200, "iterations per tool (warm-up of iter/10 is added on top)")
|
||||
out := flag.String("out", "", "primary output path (default stdout)")
|
||||
jsonOut := flag.String("json", "", "companion JSON metrics output")
|
||||
csvOut := flag.String("csv", "", "companion CSV output")
|
||||
format := flag.String("format", "markdown", "markdown | json | csv")
|
||||
tools := flag.String("tools", "", "comma-separated subset (default: all known tools)")
|
||||
flag.Parse()
|
||||
|
||||
absRepo, err := filepath.Abs(*repo)
|
||||
if err != nil {
|
||||
die("repo path: %v", err)
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "[daemon-latency] indexing %s...\n", absRepo)
|
||||
g, srv := buildInProcessServer(absRepo)
|
||||
fmt.Fprintf(os.Stderr, "[daemon-latency] indexed %d nodes\n", len(g.AllNodes()))
|
||||
|
||||
handler := internalserver.NewHandler(srv.MCPServer(), g, "bench", zap.NewNop())
|
||||
|
||||
// Build the call set against the freshly indexed graph so each
|
||||
// synthetic request has at least some structural validity (a
|
||||
// real symbol id, an extant file path).
|
||||
calls := defaultCalls(g, *iter)
|
||||
if *tools != "" {
|
||||
calls = filterCalls(calls, strings.Split(*tools, ","))
|
||||
}
|
||||
|
||||
rows := make([]result, 0, len(calls))
|
||||
for _, c := range calls {
|
||||
if c.SkipIfMissing != nil && c.SkipIfMissing(g) {
|
||||
rows = append(rows, result{Tool: c.Tool, Iters: 0, Skipped: "no eligible substrate in indexed graph"})
|
||||
fmt.Fprintf(os.Stderr, "[daemon-latency] %-22s skipped (no substrate)\n", c.Tool)
|
||||
continue
|
||||
}
|
||||
row := runOne(handler, c)
|
||||
rows = append(rows, row)
|
||||
fmt.Fprintf(os.Stderr, "[daemon-latency] %-22s p50=%6.2fms p95=%6.2fms p99=%6.2fms iters=%d errs=%.0f%%\n",
|
||||
c.Tool, row.P50Ms, row.P95Ms, row.P99Ms, row.Iters, row.ErrorRate*100)
|
||||
}
|
||||
|
||||
var primary []byte
|
||||
switch strings.ToLower(*format) {
|
||||
case "markdown", "md":
|
||||
primary = []byte(renderMarkdown(rows, absRepo, g))
|
||||
case "csv":
|
||||
primary = []byte(renderCSV(rows))
|
||||
case "json":
|
||||
primary = mustMarshalJSON(rows)
|
||||
default:
|
||||
die("unknown --format %q", *format)
|
||||
}
|
||||
if err := writeOutput(*out, primary); err != nil {
|
||||
die("write output: %v", err)
|
||||
}
|
||||
if *csvOut != "" {
|
||||
if err := writeOutput(*csvOut, []byte(renderCSV(rows))); err != nil {
|
||||
die("write csv: %v", err)
|
||||
}
|
||||
}
|
||||
if *jsonOut != "" {
|
||||
if err := writeOutput(*jsonOut, mustMarshalJSON(rows)); err != nil {
|
||||
die("write json: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- in-process server ---------------------------------------------
|
||||
|
||||
// buildInProcessServer wires the same Server the production stdio /
|
||||
// daemon front-ends use, against a fresh in-process graph of repoRoot.
|
||||
// Identical wiring to `cmd/gortex/eval_recall.go`'s indexed-server
|
||||
// path so the bench reflects production handler arithmetic.
|
||||
func buildInProcessServer(repoRoot string) (*graph.Graph, *gortexmcp.Server) {
|
||||
g := graph.New()
|
||||
reg := parser.NewRegistry()
|
||||
languages.RegisterAll(reg)
|
||||
cfg := config.Config{}
|
||||
idx := indexer.New(g, reg, cfg.Index, zap.NewNop())
|
||||
if _, err := idx.Index(repoRoot); err != nil {
|
||||
die("index %s: %v", repoRoot, err)
|
||||
}
|
||||
eng := query.NewEngine(g)
|
||||
eng.SetSearch(idx.Search())
|
||||
srv := gortexmcp.NewServer(eng, g, idx, nil, zap.NewNop(), cfg.Guards.Rules)
|
||||
srv.RunAnalysis()
|
||||
return g, srv
|
||||
}
|
||||
|
||||
// --- call set -------------------------------------------------------
|
||||
|
||||
// defaultCalls returns the canonical bench surface. We focus on
|
||||
// tools agents actually call in production (the headline savings
|
||||
// drivers) — covering both cheap (graph_stats) and expensive
|
||||
// (smart_context) shapes so the published table shows the spread.
|
||||
func defaultCalls(g *graph.Graph, iter int) []toolCall {
|
||||
if iter <= 0 {
|
||||
iter = 200
|
||||
}
|
||||
warmup := max(iter/10, 5)
|
||||
|
||||
// Pick representative symbol IDs / file paths from the indexed
|
||||
// graph so the synthetic requests have real targets.
|
||||
var sampleFnID, sampleFilePath string
|
||||
for _, n := range g.AllNodes() {
|
||||
if n == nil {
|
||||
continue
|
||||
}
|
||||
if sampleFnID == "" && (n.Kind == graph.KindFunction || n.Kind == graph.KindMethod) {
|
||||
sampleFnID = n.ID
|
||||
}
|
||||
if sampleFilePath == "" && n.Kind == graph.KindFile && n.FilePath != "" {
|
||||
sampleFilePath = n.FilePath
|
||||
}
|
||||
if sampleFnID != "" && sampleFilePath != "" {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
queries := []string{
|
||||
"validateToken", "Indexer", "search", "newServer",
|
||||
"handler", "config", "graph", "rerank", "query", "savings",
|
||||
}
|
||||
|
||||
return []toolCall{
|
||||
{
|
||||
Tool: "graph_stats",
|
||||
ArgsFn: func(_ int) map[string]any { return map[string]any{} },
|
||||
WarmupN: warmup, IterN: iter,
|
||||
},
|
||||
{
|
||||
Tool: "search_symbols",
|
||||
ArgsFn: func(i int) map[string]any {
|
||||
return map[string]any{
|
||||
"query": queries[i%len(queries)],
|
||||
"limit": float64(20),
|
||||
}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
},
|
||||
{
|
||||
Tool: "get_symbol_source",
|
||||
ArgsFn: func(_ int) map[string]any {
|
||||
return map[string]any{"id": sampleFnID}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
SkipIfMissing: func(g *graph.Graph) bool { return sampleFnID == "" },
|
||||
},
|
||||
{
|
||||
Tool: "get_callers",
|
||||
ArgsFn: func(_ int) map[string]any {
|
||||
return map[string]any{"id": sampleFnID, "limit": float64(50)}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
SkipIfMissing: func(g *graph.Graph) bool { return sampleFnID == "" },
|
||||
},
|
||||
{
|
||||
Tool: "find_usages",
|
||||
ArgsFn: func(_ int) map[string]any {
|
||||
return map[string]any{"id": sampleFnID}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
SkipIfMissing: func(g *graph.Graph) bool { return sampleFnID == "" },
|
||||
},
|
||||
{
|
||||
Tool: "get_file_summary",
|
||||
ArgsFn: func(_ int) map[string]any {
|
||||
return map[string]any{"path": sampleFilePath}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
SkipIfMissing: func(g *graph.Graph) bool { return sampleFilePath == "" },
|
||||
},
|
||||
{
|
||||
Tool: "smart_context",
|
||||
ArgsFn: func(i int) map[string]any {
|
||||
return map[string]any{"task": "find " + queries[i%len(queries)]}
|
||||
},
|
||||
// smart_context is heavy — fewer iterations so the
|
||||
// whole bench stays reasonable. Still produces a
|
||||
// credible p50/p95 with 30-50 samples.
|
||||
WarmupN: 3, IterN: iter / 5,
|
||||
},
|
||||
{
|
||||
Tool: "get_repo_outline",
|
||||
ArgsFn: func(_ int) map[string]any {
|
||||
return map[string]any{}
|
||||
},
|
||||
WarmupN: warmup, IterN: iter,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func filterCalls(calls []toolCall, names []string) []toolCall {
|
||||
want := map[string]bool{}
|
||||
for _, n := range names {
|
||||
want[strings.TrimSpace(n)] = true
|
||||
}
|
||||
out := make([]toolCall, 0, len(calls))
|
||||
for _, c := range calls {
|
||||
if want[c.Tool] {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// --- run loop -------------------------------------------------------
|
||||
|
||||
func runOne(handler *internalserver.Handler, c toolCall) result {
|
||||
ctx := context.Background()
|
||||
// Warm-up: prime any lazy initialisation in the handler /
|
||||
// graph so the measured iterations are steady-state.
|
||||
for i := range c.WarmupN {
|
||||
_, _ = handler.CallToolStrict(ctx, c.Tool, c.ArgsFn(i))
|
||||
}
|
||||
|
||||
latencies := make([]time.Duration, 0, c.IterN)
|
||||
errors := 0
|
||||
for i := range c.IterN {
|
||||
t := time.Now()
|
||||
_, err := handler.CallToolStrict(ctx, c.Tool, c.ArgsFn(i))
|
||||
latencies = append(latencies, time.Since(t))
|
||||
if err != nil {
|
||||
errors++
|
||||
}
|
||||
}
|
||||
r := result{
|
||||
Tool: c.Tool,
|
||||
Iters: c.IterN,
|
||||
}
|
||||
if len(latencies) > 0 {
|
||||
r.P50Ms = pctMs(latencies, 50)
|
||||
r.P95Ms = pctMs(latencies, 95)
|
||||
r.P99Ms = pctMs(latencies, 99)
|
||||
r.MaxMs = pctMs(latencies, 100)
|
||||
r.MeanMs = meanMs(latencies)
|
||||
r.ErrorRate = float64(errors) / float64(len(latencies))
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func pctMs(xs []time.Duration, pct int) float64 {
|
||||
if len(xs) == 0 {
|
||||
return 0
|
||||
}
|
||||
sorted := make([]time.Duration, len(xs))
|
||||
copy(sorted, xs)
|
||||
slices.Sort(sorted)
|
||||
idx := (pct * len(sorted)) / 100
|
||||
if idx >= len(sorted) {
|
||||
idx = len(sorted) - 1
|
||||
}
|
||||
return float64(sorted[idx].Microseconds()) / 1000.0
|
||||
}
|
||||
|
||||
func meanMs(xs []time.Duration) float64 {
|
||||
if len(xs) == 0 {
|
||||
return 0
|
||||
}
|
||||
var sum time.Duration
|
||||
for _, x := range xs {
|
||||
sum += x
|
||||
}
|
||||
avg := sum / time.Duration(len(xs))
|
||||
return float64(avg.Microseconds()) / 1000.0
|
||||
}
|
||||
|
||||
// --- rendering ------------------------------------------------------
|
||||
|
||||
func renderMarkdown(rows []result, repoRoot string, g *graph.Graph) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintln(&b, "# Daemon-mode MCP-tool latency")
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintf(&b, "_Corpus: `%s` (%d nodes). In-process handler dispatch — measures `Handler.CallToolStrict` end-to-end. Daemon socket overhead adds typically <1 ms on a warm pipe; the handler latency below dominates user-perceived response time._\n",
|
||||
repoRoot, len(g.AllNodes()))
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, "| tool | iters | p50 | p95 | p99 | mean | max | errors |")
|
||||
fmt.Fprintln(&b, "|------|------:|----:|----:|----:|-----:|----:|-------:|")
|
||||
for _, r := range rows {
|
||||
if r.Skipped != "" {
|
||||
fmt.Fprintf(&b, "| %s | — | — | — | — | — | — | skipped: %s |\n", r.Tool, r.Skipped)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "| %s | %d | %s | %s | %s | %s | %s | %.0f%% |\n",
|
||||
r.Tool, r.Iters,
|
||||
fmtMs(r.P50Ms), fmtMs(r.P95Ms), fmtMs(r.P99Ms),
|
||||
fmtMs(r.MeanMs), fmtMs(r.MaxMs),
|
||||
r.ErrorRate*100,
|
||||
)
|
||||
}
|
||||
fmt.Fprintln(&b)
|
||||
fmt.Fprintln(&b, summary(rows))
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func renderCSV(rows []result) string {
|
||||
var b strings.Builder
|
||||
w := csv.NewWriter(&b)
|
||||
_ = w.Write([]string{"tool", "iters", "p50_ms", "p95_ms", "p99_ms", "mean_ms", "max_ms", "error_rate", "skipped"})
|
||||
for _, r := range rows {
|
||||
_ = w.Write([]string{
|
||||
r.Tool,
|
||||
fmt.Sprintf("%d", r.Iters),
|
||||
fmt.Sprintf("%.3f", r.P50Ms),
|
||||
fmt.Sprintf("%.3f", r.P95Ms),
|
||||
fmt.Sprintf("%.3f", r.P99Ms),
|
||||
fmt.Sprintf("%.3f", r.MeanMs),
|
||||
fmt.Sprintf("%.3f", r.MaxMs),
|
||||
fmt.Sprintf("%.4f", r.ErrorRate),
|
||||
r.Skipped,
|
||||
})
|
||||
}
|
||||
w.Flush()
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func mustMarshalJSON(rows []result) []byte {
|
||||
b, err := json.MarshalIndent(rows, "", " ")
|
||||
if err != nil {
|
||||
die("marshal json: %v", err)
|
||||
}
|
||||
return append(b, '\n')
|
||||
}
|
||||
|
||||
func summary(rows []result) string {
|
||||
ran := 0
|
||||
var p95s, p99s []float64
|
||||
for _, r := range rows {
|
||||
if r.Skipped != "" {
|
||||
continue
|
||||
}
|
||||
ran++
|
||||
p95s = append(p95s, r.P95Ms)
|
||||
p99s = append(p99s, r.P99Ms)
|
||||
}
|
||||
if ran == 0 {
|
||||
return "_no tools ran (all skipped)_"
|
||||
}
|
||||
sort.Float64s(p95s)
|
||||
sort.Float64s(p99s)
|
||||
medianP95 := p95s[len(p95s)/2]
|
||||
medianP99 := p99s[len(p99s)/2]
|
||||
return fmt.Sprintf("**Summary:** %d/%d tools ran. Median p95 across tools: %s. Median p99: %s.",
|
||||
ran, len(rows), fmtMs(medianP95), fmtMs(medianP99))
|
||||
}
|
||||
|
||||
func fmtMs(v float64) string {
|
||||
switch {
|
||||
case v == 0:
|
||||
return "—"
|
||||
case v < 1.0:
|
||||
return fmt.Sprintf("%.2fms", v)
|
||||
case v < 1000:
|
||||
return fmt.Sprintf("%.1fms", v)
|
||||
default:
|
||||
return fmt.Sprintf("%.2fs", v/1000.0)
|
||||
}
|
||||
}
|
||||
|
||||
// --- helpers --------------------------------------------------------
|
||||
|
||||
func die(format string, args ...any) {
|
||||
fmt.Fprintf(os.Stderr, "daemon-latency: "+format+"\n", args...)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
func writeOutput(path string, body []byte) error {
|
||||
if path == "" {
|
||||
_, err := os.Stdout.Write(body)
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, body, 0o644)
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/zzet/gortex/internal/graph"
|
||||
)
|
||||
|
||||
func TestPctMs_NearestRank(t *testing.T) {
|
||||
// Build [1, 2, …, 100] ms.
|
||||
xs := make([]time.Duration, 100)
|
||||
for i := range xs {
|
||||
xs[i] = time.Duration(i+1) * time.Millisecond
|
||||
}
|
||||
cases := map[int]float64{
|
||||
50: 51.0, // idx = 50*100/100 = 50 → sorted[50] = 51ms
|
||||
95: 96.0, // idx = 95 → 96ms
|
||||
99: 100.0, // idx = 99 → 100ms
|
||||
}
|
||||
for p, want := range cases {
|
||||
got := pctMs(xs, p)
|
||||
if got != want {
|
||||
t.Errorf("pctMs(%d) = %.2f, want %.2f", p, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPctMs_EmptyReturnsZero(t *testing.T) {
|
||||
if got := pctMs(nil, 50); got != 0 {
|
||||
t.Errorf("pctMs(nil) = %.2f, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPctMs_SingleSampleAllPctReturnIt(t *testing.T) {
|
||||
xs := []time.Duration{5 * time.Millisecond}
|
||||
for _, p := range []int{50, 95, 99} {
|
||||
if got := pctMs(xs, p); got != 5.0 {
|
||||
t.Errorf("pctMs(single, %d) = %.2f, want 5.0", p, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMeanMs(t *testing.T) {
|
||||
xs := []time.Duration{
|
||||
1 * time.Millisecond,
|
||||
2 * time.Millisecond,
|
||||
3 * time.Millisecond,
|
||||
4 * time.Millisecond,
|
||||
}
|
||||
if got := meanMs(xs); got != 2.5 {
|
||||
t.Errorf("meanMs = %.2f, want 2.5", got)
|
||||
}
|
||||
if got := meanMs(nil); got != 0 {
|
||||
t.Errorf("meanMs(nil) = %.2f, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFmtMs_Buckets(t *testing.T) {
|
||||
cases := map[float64]string{
|
||||
0: "—",
|
||||
0.25: "0.25ms",
|
||||
3.7: "3.7ms",
|
||||
1500: "1.50s",
|
||||
60_000: "60.00s",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := fmtMs(in); got != want {
|
||||
t.Errorf("fmtMs(%v) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterCalls_KeepsRequestedSubset(t *testing.T) {
|
||||
g := graph.New()
|
||||
all := defaultCalls(g, 10)
|
||||
got := filterCalls(all, []string{"graph_stats", "search_symbols"})
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("got %d, want 2", len(got))
|
||||
}
|
||||
gotNames := map[string]bool{}
|
||||
for _, c := range got {
|
||||
gotNames[c.Tool] = true
|
||||
}
|
||||
if !gotNames["graph_stats"] || !gotNames["search_symbols"] {
|
||||
t.Errorf("got %v, want graph_stats + search_symbols", gotNames)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultCalls_PopulatesSubstrate(t *testing.T) {
|
||||
g := graph.New()
|
||||
g.AddNode(&graph.Node{ID: "f.go::Sym", Name: "Sym", Kind: graph.KindFunction, FilePath: "f.go"})
|
||||
g.AddNode(&graph.Node{ID: "file:f.go", Kind: graph.KindFile, FilePath: "f.go"})
|
||||
|
||||
calls := defaultCalls(g, 50)
|
||||
if len(calls) == 0 {
|
||||
t.Fatal("defaultCalls returned 0 entries")
|
||||
}
|
||||
// At least the headline tools must be present.
|
||||
want := map[string]bool{
|
||||
"graph_stats": true, "search_symbols": true,
|
||||
"get_symbol_source": true, "smart_context": true,
|
||||
}
|
||||
got := map[string]bool{}
|
||||
for _, c := range calls {
|
||||
got[c.Tool] = true
|
||||
}
|
||||
for w := range want {
|
||||
if !got[w] {
|
||||
t.Errorf("default call set missing %q", w)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultCalls_SkipsTargetlessSubstrate(t *testing.T) {
|
||||
g := graph.New()
|
||||
// No function / method nodes → get_symbol_source / get_callers /
|
||||
// find_usages should report SkipIfMissing=true.
|
||||
calls := defaultCalls(g, 10)
|
||||
for _, c := range calls {
|
||||
if c.Tool == "get_symbol_source" || c.Tool == "get_callers" || c.Tool == "find_usages" {
|
||||
if c.SkipIfMissing == nil || !c.SkipIfMissing(g) {
|
||||
t.Errorf("%s should skip when no callable symbols available", c.Tool)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderMarkdown_HasHeaderAndRows(t *testing.T) {
|
||||
rows := []result{
|
||||
{Tool: "graph_stats", Iters: 200, P50Ms: 1.2, P95Ms: 4.5, P99Ms: 8.9, MeanMs: 2.0, MaxMs: 12.5},
|
||||
{Tool: "smart_context", Iters: 40, P50Ms: 50.0, P95Ms: 200.0, P99Ms: 300.0, MeanMs: 80.0, MaxMs: 350.0},
|
||||
{Tool: "skipped_one", Skipped: "no substrate"},
|
||||
}
|
||||
g := graph.New()
|
||||
md := renderMarkdown(rows, "/tmp/repo", g)
|
||||
for _, want := range []string{
|
||||
"# Daemon-mode MCP-tool latency",
|
||||
"| graph_stats |",
|
||||
"| smart_context |",
|
||||
"| skipped_one |",
|
||||
"skipped:",
|
||||
"**Summary:**",
|
||||
"2/3 tools",
|
||||
} {
|
||||
if !strings.Contains(md, want) {
|
||||
t.Errorf("markdown missing %q\n----\n%s", want, md)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderCSV_HasHeaderAndRows(t *testing.T) {
|
||||
rows := []result{
|
||||
{Tool: "graph_stats", Iters: 200, P50Ms: 1.2, P95Ms: 4.5, P99Ms: 8.9},
|
||||
}
|
||||
out := renderCSV(rows)
|
||||
if !strings.HasPrefix(out, "tool,iters,p50_ms,p95_ms,p99_ms,") {
|
||||
t.Errorf("CSV header missing or wrong:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "graph_stats,200,") {
|
||||
t.Errorf("CSV body missing graph_stats row:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMustMarshalJSON_RoundTrip(t *testing.T) {
|
||||
rows := []result{
|
||||
{Tool: "graph_stats", Iters: 100, P95Ms: 5.5, P99Ms: 8.2},
|
||||
}
|
||||
raw := mustMarshalJSON(rows)
|
||||
var got []result
|
||||
if err := json.Unmarshal(raw, &got); err != nil {
|
||||
t.Fatalf("round-trip: %v", err)
|
||||
}
|
||||
if len(got) != 1 || got[0].Tool != "graph_stats" || got[0].P95Ms != 5.5 {
|
||||
t.Errorf("round-trip lost data: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummary_AllSkippedYieldsExplicitMessage(t *testing.T) {
|
||||
rows := []result{{Tool: "x", Skipped: "y"}, {Tool: "z", Skipped: "w"}}
|
||||
got := summary(rows)
|
||||
if !strings.Contains(got, "no tools ran") {
|
||||
t.Errorf("summary should explicitly say no tools ran, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummary_MedianP95AndP99(t *testing.T) {
|
||||
rows := []result{
|
||||
{Tool: "a", P95Ms: 1, P99Ms: 2},
|
||||
{Tool: "b", P95Ms: 5, P99Ms: 10},
|
||||
{Tool: "c", P95Ms: 10, P99Ms: 20},
|
||||
}
|
||||
got := summary(rows)
|
||||
// Median p95 of [1, 5, 10] = 5
|
||||
if !strings.Contains(got, "Median p95 across tools: 5.0ms") {
|
||||
t.Errorf("summary missing median p95=5.0ms; got:\n%s", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "gortex-di-angular-fixture",
|
||||
"private": true,
|
||||
"description": "Minimal Angular fixture for Gortex DI recall evaluation. Not installed — static corpus.",
|
||||
"dependencies": {
|
||||
"@angular/core": "^17.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
# Angular DI-recall fixture
|
||||
#
|
||||
# Modern Angular has two DI shapes:
|
||||
# 1. Constructor injection: `constructor(private svc: Svc) {}` — same
|
||||
# shape as NestJS. Type-aware resolution already handles this.
|
||||
# 2. inject() function: `private svc = inject(Svc)` — field
|
||||
# initializer calls `inject(Target)` to resolve the dependency.
|
||||
# The call site is a function call to `inject`, not to Target.
|
||||
# Without a special pass, the dependency is invisible — same
|
||||
# shape as FastAPI's Depends(target) in that respect.
|
||||
#
|
||||
# Angular's @Injectable({ providedIn: 'root' }) is metadata only —
|
||||
# doesn't affect graph shape, just marks the class as injectable. No
|
||||
# extraction needed for the recall queries in this fixture.
|
||||
name: gortex-angular-di
|
||||
cases:
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 1 — exact: named symbols
|
||||
# ------------------------------------------------------------------
|
||||
- id: exact-UsersService
|
||||
tier: exact
|
||||
query: UsersService
|
||||
expected: [src/app/users/users.service.ts::UsersService]
|
||||
- id: exact-AuthService
|
||||
tier: exact
|
||||
query: AuthService
|
||||
expected: [src/app/auth/auth.service.ts::AuthService]
|
||||
- id: exact-UsersListComponent
|
||||
tier: exact
|
||||
query: UsersListComponent
|
||||
expected: [src/app/users/users.controller.ts::UsersListComponent]
|
||||
- id: exact-findOne
|
||||
tier: exact
|
||||
query: findOne
|
||||
expected: [src/app/users/users.service.ts::UsersService.findOne]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 2 — concept
|
||||
# ------------------------------------------------------------------
|
||||
- id: concept-list-all-users
|
||||
tier: concept
|
||||
query: list every user
|
||||
expected: [src/app/users/users.service.ts::UsersService.findAll]
|
||||
- id: concept-validate-credentials
|
||||
tier: concept
|
||||
query: validate user credentials
|
||||
expected: [src/app/auth/auth.service.ts::AuthService.validate]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 3 — di: classic constructor injection with explicit types
|
||||
# ------------------------------------------------------------------
|
||||
# UsersListComponent has `constructor(private auth: AuthService)`.
|
||||
# `this.auth.validate(...)` in validateAndList resolves via the
|
||||
# parameter-property typing pass — Angular uses the same TS shape
|
||||
# as NestJS here.
|
||||
- id: di-call_chain-validateAndList
|
||||
tier: di
|
||||
query: "call_chain:src/app/users/users.controller.ts::UsersListComponent.validateAndList"
|
||||
expected:
|
||||
- src/app/auth/auth.service.ts::AuthService.validate
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 4 — di_gap: inject() function-style injection
|
||||
# ------------------------------------------------------------------
|
||||
# AuthService uses `private users = inject(UsersService)`. The
|
||||
# `users.findOne(userId)` call site can only resolve to UsersService.findOne
|
||||
# if the graph knows `this.users` has type UsersService. Without
|
||||
# inject()-aware extraction, the field's type is unknown and the
|
||||
# call chain stops here.
|
||||
- id: di_gap-call_chain-AuthService-validate
|
||||
tier: di_gap
|
||||
query: "call_chain:src/app/auth/auth.service.ts::AuthService.validate"
|
||||
expected:
|
||||
- src/app/users/users.service.ts::UsersService.findOne
|
||||
|
||||
# UsersListComponent.displayList calls `this.users.findAll()` where
|
||||
# `users = inject(UsersService)` — same gap.
|
||||
- id: di_gap-call_chain-displayList
|
||||
tier: di_gap
|
||||
query: "call_chain:src/app/users/users.controller.ts::UsersListComponent.displayList"
|
||||
expected:
|
||||
- src/app/users/users.service.ts::UsersService.findAll
|
||||
|
||||
# Direct callers query: who uses UsersService? With inject()
|
||||
# extraction, the answer includes classes that inject() it, not
|
||||
# just constructor-injected consumers.
|
||||
- id: di_gap-callers-UsersService-findAll
|
||||
tier: di_gap
|
||||
query: "callers:src/app/users/users.service.ts::UsersService.findAll"
|
||||
expected:
|
||||
- src/app/users/users.controller.ts::UsersListComponent.displayList
|
||||
|
||||
- id: di_gap-callers-UsersService-findOne
|
||||
tier: di_gap
|
||||
query: "callers:src/app/users/users.service.ts::UsersService.findOne"
|
||||
expected:
|
||||
- src/app/auth/auth.service.ts::AuthService.validate
|
||||
|
||||
# Ambiguous method names: SessionService also has a findOne() method.
|
||||
# SessionService.currentUser calls `this.users.findOne(...)` where
|
||||
# `users = inject(UsersService)`. A name-only fallback would pick
|
||||
# either SessionService.findOne or UsersService.findOne arbitrarily.
|
||||
# This case tests whether inject()-aware type resolution routes the
|
||||
# call correctly to UsersService.findOne.
|
||||
- id: di_gap-call_chain-SessionService-currentUser
|
||||
tier: di_gap
|
||||
query: "call_chain:src/app/auth/session.service.ts::SessionService.currentUser"
|
||||
expected:
|
||||
- src/app/users/users.service.ts::UsersService.findOne
|
||||
@@ -0,0 +1,18 @@
|
||||
import { Injectable, inject } from '@angular/core';
|
||||
import { UsersService } from '../users/users.service';
|
||||
|
||||
// Modern Angular: `inject()` function-style DI. No constructor required
|
||||
// — the service is resolved via Angular's injector context at field-
|
||||
// initialization time. This is the shape ordinary static analysis
|
||||
// misses entirely, because the dependency edge lives inside an
|
||||
// argument of a generic-looking function call (`inject(X)`) rather
|
||||
// than in a typed constructor param.
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class AuthService {
|
||||
private readonly users = inject(UsersService);
|
||||
|
||||
validate(userId: string, token: string): boolean {
|
||||
const u = this.users.findOne(userId);
|
||||
return u !== undefined && token === `tok_${u.id}`;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { Injectable, inject } from '@angular/core';
|
||||
import { UsersService } from '../users/users.service';
|
||||
|
||||
// Second service with a findOne() method — creates a name-collision
|
||||
// with UsersService.findOne. A resolver that picks up the class field
|
||||
// `this.users = inject(UsersService)` as UsersService-typed will
|
||||
// correctly route `this.users.findOne(...)` to UsersService. A
|
||||
// name-only fallback would arbitrarily pick one of the two findOne
|
||||
// implementations.
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class SessionService {
|
||||
private readonly users = inject(UsersService);
|
||||
|
||||
// Same method name as UsersService — deliberate collision so the
|
||||
// call `this.users.findOne(...)` forces type-aware resolution.
|
||||
findOne(sessionId: string): { userId: string } | undefined {
|
||||
return { userId: sessionId };
|
||||
}
|
||||
|
||||
currentUser(sessionId: string): string | undefined {
|
||||
const s = this.findOne(sessionId);
|
||||
if (!s) return undefined;
|
||||
return this.users.findOne(s.userId)?.email;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
export interface User {
|
||||
id: string;
|
||||
email: string;
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { Component, inject } from '@angular/core';
|
||||
import { UsersService } from './users.service';
|
||||
import { AuthService } from '../auth/auth.service';
|
||||
|
||||
// A plain-ish component-like class that mixes both DI styles:
|
||||
// - `inject(UsersService)` — function form
|
||||
// - constructor parameter-property — classic Angular (pre-14)
|
||||
@Component({
|
||||
selector: 'users-list',
|
||||
template: '',
|
||||
})
|
||||
export class UsersListComponent {
|
||||
private readonly users = inject(UsersService);
|
||||
|
||||
constructor(private readonly auth: AuthService) {}
|
||||
|
||||
displayList(): string[] {
|
||||
return this.users.findAll().map((u) => u.email);
|
||||
}
|
||||
|
||||
validateAndList(userId: string, token: string): string[] {
|
||||
this.auth.validate(userId, token);
|
||||
return this.displayList();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { Injectable } from '@angular/core';
|
||||
import { User } from './user.model';
|
||||
|
||||
// Modern Angular `providedIn: 'root'` — makes the service available
|
||||
// app-wide without a NgModule providers array. The service is a plain
|
||||
// TypeScript class with a few methods agents would want to trace from
|
||||
// their call sites.
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class UsersService {
|
||||
private readonly store = new Map<string, User>();
|
||||
|
||||
findOne(id: string): User | undefined {
|
||||
return this.store.get(id);
|
||||
}
|
||||
|
||||
findAll(): User[] {
|
||||
return Array.from(this.store.values());
|
||||
}
|
||||
|
||||
create(email: string): User {
|
||||
const id = `usr_${this.store.size + 1}`;
|
||||
const u: User = { id, email };
|
||||
this.store.set(id, u);
|
||||
return u;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "esnext",
|
||||
"experimentalDecorators": true,
|
||||
"emitDecoratorMetadata": true,
|
||||
"strict": true,
|
||||
"baseUrl": "./src"
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends
|
||||
|
||||
from app.auth.service import AuthService
|
||||
|
||||
|
||||
# Function-style dependency that wraps a class-based dependency —
|
||||
# standard FastAPI pattern for "require this to be valid before the
|
||||
# handler runs". The returned AuthService instance is what the
|
||||
# handler ends up with when it declares `auth: AuthService = Depends(require_auth)`.
|
||||
def require_auth(auth: AuthService = Depends(AuthService)) -> AuthService:
|
||||
return auth
|
||||
|
||||
|
||||
# PEP 593 / modern FastAPI shorthand — same resolution as the default-
|
||||
# value form but typed via Annotated. Both shapes must resolve to
|
||||
# AuthService.validate for any call-chain query from a handler using
|
||||
# `auth: CurrentAuth` to reach the underlying implementation.
|
||||
CurrentAuth = Annotated[AuthService, Depends(AuthService)]
|
||||
@@ -0,0 +1,19 @@
|
||||
from fastapi import Depends, HTTPException
|
||||
|
||||
from app.users.service import UserService
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Class-based dependency that ITSELF depends on another service.
|
||||
FastAPI resolves the nested Depends chain automatically; statically
|
||||
the dependency is visible via the __init__ parameter's Depends()
|
||||
default value."""
|
||||
|
||||
def __init__(self, users: UserService = Depends(UserService)) -> None:
|
||||
self._users = users
|
||||
|
||||
def validate(self, user_id: str, token: str) -> bool:
|
||||
u = self._users.find_one(user_id)
|
||||
if u is None or token != f"tok_{u.id}":
|
||||
raise HTTPException(status_code=401)
|
||||
return True
|
||||
@@ -0,0 +1,14 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Settings:
|
||||
database_url: str = "postgres://localhost/test"
|
||||
feature_flags: dict[str, bool] = None
|
||||
|
||||
|
||||
def get_settings() -> Settings:
|
||||
"""Factory-style dependency. FastAPI will call this once per request
|
||||
(unless cached with @lru_cache) and inject the result into any
|
||||
handler that declares `settings: Settings = Depends(get_settings)`."""
|
||||
return Settings(feature_flags={"beta": True})
|
||||
@@ -0,0 +1,7 @@
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.users.router import router as users_router
|
||||
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(users_router)
|
||||
@@ -0,0 +1,8 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class User:
|
||||
id: str
|
||||
email: str
|
||||
name: str
|
||||
@@ -0,0 +1,51 @@
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
|
||||
from app.auth.deps import CurrentAuth, require_auth
|
||||
from app.auth.service import AuthService
|
||||
from app.config.settings import Settings, get_settings
|
||||
from app.users.models import User
|
||||
from app.users.service import UserService
|
||||
|
||||
router = APIRouter(prefix="/users")
|
||||
|
||||
|
||||
# Default-value form: `users: UserService = Depends(UserService)`.
|
||||
# Most common shape in real FastAPI code.
|
||||
@router.get("/")
|
||||
def list_users(users: UserService = Depends(UserService)) -> list[User]:
|
||||
return users.find_all()
|
||||
|
||||
|
||||
# Nested / chained Depends: the handler depends on require_auth, which
|
||||
# itself depends on AuthService, which depends on UserService. Whole
|
||||
# chain is discoverable statically — each Depends() call exposes its
|
||||
# target function/class as a default-value expression.
|
||||
@router.get("/{user_id}")
|
||||
def get_user(
|
||||
user_id: str,
|
||||
users: UserService = Depends(UserService),
|
||||
auth: AuthService = Depends(require_auth),
|
||||
) -> User:
|
||||
u = users.find_one(user_id)
|
||||
if u is None:
|
||||
raise ValueError(f"no user {user_id}")
|
||||
# Touch the auth-resolved service so the call chain from get_user
|
||||
# reaches AuthService.validate and by extension UserService.find_one.
|
||||
_ = auth.validate(user_id, f"tok_{u.id}")
|
||||
return u
|
||||
|
||||
|
||||
# Annotated-form Depends (PEP 593): `auth: Annotated[AuthService, Depends(...)]`.
|
||||
# Semantically equivalent to the default-value form; a modern FastAPI
|
||||
# codebase uses it everywhere.
|
||||
@router.post("/")
|
||||
def create_user(
|
||||
email: str,
|
||||
name: str,
|
||||
users: Annotated[UserService, Depends(UserService)],
|
||||
settings: Annotated[Settings, Depends(get_settings)],
|
||||
) -> User:
|
||||
_ = settings.database_url
|
||||
return users.create(email, name)
|
||||
@@ -0,0 +1,24 @@
|
||||
from typing import Optional
|
||||
|
||||
from app.users.models import User
|
||||
|
||||
|
||||
class UserService:
|
||||
"""Plain service class — registered as a FastAPI dependency via
|
||||
class-based `Depends(UserService)`. Its methods are called from
|
||||
route handlers that receive the instance through a parameter."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._users: dict[str, User] = {}
|
||||
|
||||
def find_one(self, user_id: str) -> Optional[User]:
|
||||
return self._users.get(user_id)
|
||||
|
||||
def find_all(self) -> list[User]:
|
||||
return list(self._users.values())
|
||||
|
||||
def create(self, email: str, name: str) -> User:
|
||||
uid = f"usr_{len(self._users) + 1}"
|
||||
u = User(id=uid, email=email, name=name)
|
||||
self._users[uid] = u
|
||||
return u
|
||||
@@ -0,0 +1,8 @@
|
||||
[project]
|
||||
name = "gortex-di-fastapi-fixture"
|
||||
version = "0.0.0"
|
||||
description = "Minimal FastAPI fixture for Gortex DI recall evaluation. Not installed — static corpus for the indexer to parse."
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"fastapi>=0.100",
|
||||
]
|
||||
@@ -0,0 +1,124 @@
|
||||
# FastAPI DI-recall fixture
|
||||
#
|
||||
# FastAPI's dependency injection is less "magic" than NestJS — every
|
||||
# dependency is the default value of a parameter, either via
|
||||
# `= Depends(target)` or `Annotated[T, Depends(target)]`. Target can be
|
||||
# a callable (function) or a class. Static analysis CAN see the target
|
||||
# because it's a value expression at the call site.
|
||||
#
|
||||
# Tiers (same as the nestjs fixture):
|
||||
# exact — symbol-name text queries
|
||||
# concept — natural-language paraphrases
|
||||
# di — shapes that a type-aware Python resolver would already
|
||||
# resolve (explicit class parameter types)
|
||||
# di_gap — shapes that require Depends-aware extraction to traverse
|
||||
# (nested Depends, function-wrapped Depends, Annotated)
|
||||
#
|
||||
# Expected IDs use the repo-stripped path form the Gortex TypeScript
|
||||
# and Python extractors produce, e.g.
|
||||
# app/users/router.py::get_user
|
||||
# app/users/service.py::UserService.find_one
|
||||
name: gortex-fastapi-di
|
||||
cases:
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 1 — exact: can the Python extractor find named symbols?
|
||||
# ------------------------------------------------------------------
|
||||
- id: exact-UserService
|
||||
tier: exact
|
||||
query: UserService
|
||||
expected: [app/users/service.py::UserService]
|
||||
- id: exact-AuthService
|
||||
tier: exact
|
||||
query: AuthService
|
||||
expected: [app/auth/service.py::AuthService]
|
||||
- id: exact-get_settings
|
||||
tier: exact
|
||||
query: get_settings
|
||||
expected: [app/config/settings.py::get_settings]
|
||||
- id: exact-require_auth
|
||||
tier: exact
|
||||
query: require_auth
|
||||
expected: [app/auth/deps.py::require_auth]
|
||||
- id: exact-find_one
|
||||
tier: exact
|
||||
query: find_one
|
||||
expected: [app/users/service.py::UserService.find_one]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 2 — concept: paraphrase queries
|
||||
# ------------------------------------------------------------------
|
||||
- id: concept-look-up-user
|
||||
tier: concept
|
||||
query: look up a user by their id
|
||||
expected: [app/users/service.py::UserService.find_one]
|
||||
- id: concept-validate-token
|
||||
tier: concept
|
||||
query: validate an auth token
|
||||
expected: [app/auth/service.py::AuthService.validate]
|
||||
- id: concept-application-settings
|
||||
tier: concept
|
||||
query: application settings
|
||||
expected: [app/config/settings.py::Settings]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 3 — di: class-based Depends with explicit types
|
||||
# ------------------------------------------------------------------
|
||||
# AuthService's constructor has `users: UserService = Depends(UserService)`.
|
||||
# The declared type `UserService` is explicit Python, so a type-aware
|
||||
# resolver can link `self._users.find_one(...)` to UserService.find_one.
|
||||
- id: di-call_chain-AuthService-validate
|
||||
tier: di
|
||||
query: "call_chain:app/auth/service.py::AuthService.validate"
|
||||
expected: [app/users/service.py::UserService.find_one]
|
||||
|
||||
# Handler with class Depends — `users: UserService = Depends(UserService)`.
|
||||
# Same class-type annotation shape.
|
||||
- id: di-call_chain-list_users
|
||||
tier: di
|
||||
query: "call_chain:app/users/router.py::list_users"
|
||||
expected: [app/users/service.py::UserService.find_all]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 4 — di_gap: shapes requiring Depends-aware extraction
|
||||
# ------------------------------------------------------------------
|
||||
# Function-wrapped Depends: the handler gets AuthService via a
|
||||
# require_auth dependency that itself returns whatever its own
|
||||
# Depends yields. The call chain from get_user should reach both
|
||||
# require_auth and the nested AuthService.validate.
|
||||
- id: di_gap-call_chain-get_user
|
||||
tier: di_gap
|
||||
query: "call_chain:app/users/router.py::get_user"
|
||||
expected:
|
||||
# Body calls (explicit invocation on the injected instance).
|
||||
- app/users/service.py::UserService.find_one
|
||||
- app/auth/service.py::AuthService.validate
|
||||
# Depends chain — the handler effectively calls each target at
|
||||
# request time via FastAPI's resolver.
|
||||
- app/auth/deps.py::require_auth
|
||||
|
||||
# require_auth → AuthService → UserService (two hops of Depends).
|
||||
- id: di_gap-callers-AuthService-validate
|
||||
tier: di_gap
|
||||
query: "callers:app/auth/service.py::AuthService.validate"
|
||||
expected: [app/users/router.py::get_user]
|
||||
|
||||
# Annotated[T, Depends(target)] — modern FastAPI shorthand. The
|
||||
# handler declares `users: Annotated[UserService, Depends(UserService)]`
|
||||
# and calls users.create(...). The Depends target is a class; its
|
||||
# type is resolvable from the annotation; the method call should
|
||||
# link to UserService.create.
|
||||
- id: di_gap-call_chain-create_user
|
||||
tier: di_gap
|
||||
query: "call_chain:app/users/router.py::create_user"
|
||||
expected:
|
||||
- app/users/service.py::UserService.create
|
||||
- app/config/settings.py::get_settings
|
||||
|
||||
# Factory-function Depends: `settings: Annotated[Settings, Depends(get_settings)]`.
|
||||
# get_settings returns Settings; the handler then touches settings.database_url.
|
||||
# With Depends-aware extraction, callers of get_settings should include the handler.
|
||||
- id: di_gap-callers-get_settings
|
||||
tier: di_gap
|
||||
query: "callers:app/config/settings.py::get_settings"
|
||||
expected: [app/users/router.py::create_user]
|
||||
@@ -0,0 +1,42 @@
|
||||
<?php
|
||||
|
||||
namespace App\Http\Controllers;
|
||||
|
||||
use App\Http\Middleware\AdminMiddleware;
|
||||
use App\Http\Middleware\AuthenticateMiddleware;
|
||||
use App\Repositories\UserRepository;
|
||||
use App\Services\Clock;
|
||||
|
||||
class UserController
|
||||
{
|
||||
// Constructor-injected typed dependencies. Laravel's container
|
||||
// autowires from the type hints; UserRepository resolves through
|
||||
// the binding in AppServiceProvider::register() to
|
||||
// EloquentUserRepository.
|
||||
public function __construct(
|
||||
private UserRepository $users,
|
||||
private Clock $clock,
|
||||
) {
|
||||
// Controller middleware: registered imperatively in the
|
||||
// constructor. 'auth' applies to every action; 'admin' is
|
||||
// filtered with ->only(['destroy']), same shape as Rails
|
||||
// before_action + only:.
|
||||
$this->middleware(AuthenticateMiddleware::class);
|
||||
$this->middleware(AdminMiddleware::class)->only(['destroy']);
|
||||
}
|
||||
|
||||
public function index(): array
|
||||
{
|
||||
return $this->users->all();
|
||||
}
|
||||
|
||||
public function show(string $id): ?array
|
||||
{
|
||||
return $this->users->find($id);
|
||||
}
|
||||
|
||||
public function destroy(string $id): string
|
||||
{
|
||||
return 'deleted at ' . $this->clock->now();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
<?php
|
||||
|
||||
namespace App\Http\Middleware;
|
||||
|
||||
use Closure;
|
||||
|
||||
class AdminMiddleware
|
||||
{
|
||||
public function handle($request, Closure $next)
|
||||
{
|
||||
if (empty($request->user['admin'])) {
|
||||
return response('forbidden', 403);
|
||||
}
|
||||
return $next($request);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
<?php
|
||||
|
||||
namespace App\Http\Middleware;
|
||||
|
||||
use Closure;
|
||||
|
||||
// Laravel middleware. handle() is what the framework calls before the
|
||||
// controller action runs. There's no explicit call site in the
|
||||
// controller or route definition — the middleware is referenced by
|
||||
// its short alias ("auth") or class name in a call like
|
||||
// $this->middleware('auth').
|
||||
class AuthenticateMiddleware
|
||||
{
|
||||
public function handle($request, Closure $next)
|
||||
{
|
||||
if (empty($request->user)) {
|
||||
return response('unauthorized', 401);
|
||||
}
|
||||
return $next($request);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
<?php
|
||||
|
||||
namespace App\Providers;
|
||||
|
||||
use App\Repositories\EloquentUserRepository;
|
||||
use App\Repositories\UserRepository;
|
||||
use App\Services\Clock;
|
||||
use Illuminate\Support\ServiceProvider;
|
||||
|
||||
class AppServiceProvider extends ServiceProvider
|
||||
{
|
||||
// register() is Laravel's container-binding entry point. Every
|
||||
// bind/singleton/instance call here declares that when someone
|
||||
// asks the container for the first argument, give them an
|
||||
// instance produced from the second. These bindings are the DI
|
||||
// gap that matters — consumers typed against the interface won't
|
||||
// reach the implementation without this info.
|
||||
public function register(): void
|
||||
{
|
||||
// bind: a fresh EloquentUserRepository per resolve.
|
||||
$this->app->bind(UserRepository::class, EloquentUserRepository::class);
|
||||
|
||||
// singleton: one Clock shared across the app lifetime.
|
||||
$this->app->singleton(Clock::class, function ($app) {
|
||||
return new Clock('UTC');
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
<?php
|
||||
|
||||
namespace App\Repositories;
|
||||
|
||||
// Concrete UserRepository. Service provider binds this to the
|
||||
// interface — without that binding, the Laravel container has no way
|
||||
// to pick an implementation from the interface alone. Same shape as
|
||||
// NestJS useClass / Spring @Bean return-type binding.
|
||||
class EloquentUserRepository implements UserRepository
|
||||
{
|
||||
public function find(string $id): ?array
|
||||
{
|
||||
return ['id' => $id];
|
||||
}
|
||||
|
||||
public function all(): array
|
||||
{
|
||||
return [];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
<?php
|
||||
|
||||
namespace App\Repositories;
|
||||
|
||||
interface UserRepository
|
||||
{
|
||||
public function find(string $id): ?array;
|
||||
public function all(): array;
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
<?php
|
||||
|
||||
namespace App\Services;
|
||||
|
||||
class Clock
|
||||
{
|
||||
public function __construct(private string $timezone = 'UTC') {}
|
||||
|
||||
public function now(): string
|
||||
{
|
||||
return 'now-in-' . $this->timezone;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "gortex/di-laravel-fixture",
|
||||
"type": "project",
|
||||
"description": "Minimal Laravel fixture for Gortex DI recall — not installed.",
|
||||
"require": {
|
||||
"laravel/framework": "^11.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
# Laravel DI-recall fixture
|
||||
#
|
||||
# Two DI gaps matter in real-world Laravel code:
|
||||
# 1. Controller middleware (`$this->middleware('auth')` in the ctor,
|
||||
# optionally filtered with ->only([...]) / ->except([...])).
|
||||
# Same decorator-dispatch shape as NestJS @UseGuards / Rails
|
||||
# before_action / Phoenix plug. Framework invokes middleware's
|
||||
# handle() before the action; no explicit call site.
|
||||
# 2. Service provider bindings (`$this->app->bind(Interface::class,
|
||||
# Impl::class)` and ->singleton(...)). Container-registered
|
||||
# interface-to-implementation or token-to-factory mappings —
|
||||
# same shape as NestJS useClass / Spring @Bean.
|
||||
name: gortex-laravel-di
|
||||
cases:
|
||||
|
||||
# exact
|
||||
- id: exact-UserController
|
||||
tier: exact
|
||||
query: UserController
|
||||
expected: [app/Http/Controllers/UserController.php::UserController]
|
||||
- id: exact-UserRepository
|
||||
tier: exact
|
||||
query: UserRepository
|
||||
expected: [app/Repositories/UserRepository.php::UserRepository]
|
||||
- id: exact-EloquentUserRepository
|
||||
tier: exact
|
||||
query: EloquentUserRepository
|
||||
expected: [app/Repositories/EloquentUserRepository.php::EloquentUserRepository]
|
||||
- id: exact-AuthenticateMiddleware
|
||||
tier: exact
|
||||
query: AuthenticateMiddleware
|
||||
expected: [app/Http/Middleware/AuthenticateMiddleware.php::AuthenticateMiddleware]
|
||||
|
||||
# di — typed constructor injection. Should work via PHP type system
|
||||
# if the extractor captures constructor param types.
|
||||
- id: di-call_chain-index
|
||||
tier: di
|
||||
query: "call_chain:app/Http/Controllers/UserController.php::UserController.index"
|
||||
expected:
|
||||
- app/Repositories/UserRepository.php::UserRepository.all
|
||||
|
||||
# di_gap: middleware dispatch
|
||||
- id: di_gap-callers-AuthenticateMiddleware-handle
|
||||
tier: di_gap
|
||||
query: "callers:app/Http/Middleware/AuthenticateMiddleware.php::AuthenticateMiddleware.handle"
|
||||
expected:
|
||||
- app/Http/Controllers/UserController.php::UserController.index
|
||||
- app/Http/Controllers/UserController.php::UserController.show
|
||||
- app/Http/Controllers/UserController.php::UserController.destroy
|
||||
|
||||
# Admin middleware is only: ['destroy'] — just one action binds it.
|
||||
- id: di_gap-callers-AdminMiddleware-handle
|
||||
tier: di_gap
|
||||
query: "callers:app/Http/Middleware/AdminMiddleware.php::AdminMiddleware.handle"
|
||||
expected:
|
||||
- app/Http/Controllers/UserController.php::UserController.destroy
|
||||
|
||||
- id: di_gap-call_chain-UserController-show
|
||||
tier: di_gap
|
||||
query: "call_chain:app/Http/Controllers/UserController.php::UserController.show"
|
||||
expected:
|
||||
- app/Http/Middleware/AuthenticateMiddleware.php::AuthenticateMiddleware.handle
|
||||
|
||||
# di_gap: service provider bind(Interface, Impl)
|
||||
# $this->app->bind(UserRepository::class, EloquentUserRepository::class)
|
||||
# binds the interface to the impl. usages:UserRepository should
|
||||
# include AppServiceProvider as the binding site.
|
||||
- id: di_gap-usages-UserRepository
|
||||
tier: di_gap
|
||||
query: "usages:app/Repositories/UserRepository.php::UserRepository"
|
||||
expected:
|
||||
- app/Providers/AppServiceProvider.php::AppServiceProvider
|
||||
|
||||
# singleton binding — Clock is bound by a factory closure.
|
||||
- id: di_gap-usages-Clock
|
||||
tier: di_gap
|
||||
query: "usages:app/Services/Clock.php::Clock"
|
||||
expected:
|
||||
- app/Providers/AppServiceProvider.php::AppServiceProvider
|
||||
@@ -0,0 +1,8 @@
|
||||
<?php
|
||||
|
||||
use App\Http\Controllers\UserController;
|
||||
use Illuminate\Support\Facades\Route;
|
||||
|
||||
Route::get('/users', [UserController::class, 'index']);
|
||||
Route::get('/users/{id}', [UserController::class, 'show']);
|
||||
Route::delete('/users/{id}', [UserController::class, 'destroy']);
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "gortex-di-nestjs-fixture",
|
||||
"private": true,
|
||||
"description": "Minimal NestJS fixture for Gortex DI recall evaluation. Not installed — purely a static corpus for the indexer to parse.",
|
||||
"dependencies": {
|
||||
"@nestjs/common": "^10.0.0",
|
||||
"@nestjs/core": "^10.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
# NestJS DI-recall fixture
|
||||
#
|
||||
# Measures what Gortex's retrieval stack can find on a minimal NestJS
|
||||
# app *without* DI-aware extraction — i.e. what's reachable from plain
|
||||
# BM25 + engine fallback + raw call graph. Option 1 (extend `contracts`
|
||||
# to DI) should move the "di" tier numbers up without regressing the
|
||||
# other two.
|
||||
#
|
||||
# Tiers
|
||||
# -----
|
||||
# exact — symbol-name / signature queries. Text rankers should ace
|
||||
# these; failure means the TypeScript extractor missed something.
|
||||
# concept — natural-language paraphrases. Baseline expectation similar
|
||||
# to the Go concept tier.
|
||||
# di — relational queries that only return the right answer when a
|
||||
# DI edge exists in the graph. Queries use the GraphRanker's
|
||||
# "callers:<id>" / "call_chain:<id>" format. Expected IDs are
|
||||
# the controllers / services that inject the subject via a
|
||||
# constructor parameter. Today these return 0% because no
|
||||
# such edge is emitted.
|
||||
#
|
||||
# Expected ID format: `<relative-ts-path>::<ClassName>.<method>` —
|
||||
# matches what the in-tree TypeScript extractor produces.
|
||||
name: gortex-nestjs-di
|
||||
cases:
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 1 — exact: can the TS extractor find each symbol by name?
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
- id: exact-UsersService
|
||||
tier: exact
|
||||
query: UsersService
|
||||
expected: [src/users/users.service.ts::UsersService]
|
||||
|
||||
- id: exact-UsersController
|
||||
tier: exact
|
||||
query: UsersController
|
||||
expected: [src/users/users.controller.ts::UsersController]
|
||||
|
||||
- id: exact-AuthService
|
||||
tier: exact
|
||||
query: AuthService
|
||||
expected: [src/auth/auth.service.ts::AuthService]
|
||||
|
||||
- id: exact-AuthGuard
|
||||
tier: exact
|
||||
query: AuthGuard
|
||||
expected: [src/auth/auth.guard.ts::AuthGuard]
|
||||
|
||||
- id: exact-findOne
|
||||
tier: exact
|
||||
query: findOne
|
||||
expected: [src/users/users.service.ts::UsersService.findOne]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 2 — concept: paraphrased / intent-based text queries
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
- id: concept-look-up-user-by-id
|
||||
tier: concept
|
||||
query: look up a user by their id
|
||||
expected: [src/users/users.service.ts::UsersService.findOne]
|
||||
|
||||
- id: concept-http-guard
|
||||
tier: concept
|
||||
query: guard that validates an incoming request
|
||||
expected: [src/auth/auth.guard.ts::AuthGuard.canActivate]
|
||||
|
||||
- id: concept-issue-auth-token
|
||||
tier: concept
|
||||
query: issue an authentication token
|
||||
expected: [src/auth/auth.service.ts::AuthService.issueToken]
|
||||
|
||||
- id: concept-create-new-user
|
||||
tier: concept
|
||||
query: create a new user record
|
||||
expected: [src/users/users.service.ts::UsersService.create]
|
||||
|
||||
- id: concept-list-all-users
|
||||
tier: concept
|
||||
query: list every user
|
||||
expected: [src/users/users.service.ts::UsersService.findAll]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 3 — di: requires a DI edge to answer correctly
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# Who injects UsersService? Answer: UsersController, AuthService.
|
||||
# Without DI edges, get_callers on the service's methods sees nothing
|
||||
# past the class's own internal references.
|
||||
- id: di-callers-UsersService-findOne
|
||||
tier: di
|
||||
query: "callers:src/users/users.service.ts::UsersService.findOne"
|
||||
expected:
|
||||
- src/users/users.controller.ts::UsersController.getUser
|
||||
- src/auth/auth.service.ts::AuthService.validate
|
||||
- src/auth/auth.service.ts::AuthService.issueToken
|
||||
|
||||
- id: di-callers-UsersService-findAll
|
||||
tier: di
|
||||
query: "callers:src/users/users.service.ts::UsersService.findAll"
|
||||
expected:
|
||||
- src/users/users.controller.ts::UsersController.list
|
||||
|
||||
- id: di-callers-UsersService-create
|
||||
tier: di
|
||||
query: "callers:src/users/users.service.ts::UsersService.create"
|
||||
expected:
|
||||
- src/users/users.controller.ts::UsersController.create
|
||||
|
||||
- id: di-callers-AuthService-validate
|
||||
tier: di
|
||||
query: "callers:src/auth/auth.service.ts::AuthService.validate"
|
||||
expected:
|
||||
- src/auth/auth.guard.ts::AuthGuard.canActivate
|
||||
|
||||
# Call chain FROM a controller method should reach the service method
|
||||
# it injects. Tests forward traversal through DI boundaries.
|
||||
- id: di-call_chain-UsersController-getUser
|
||||
tier: di
|
||||
query: "call_chain:src/users/users.controller.ts::UsersController.getUser"
|
||||
expected:
|
||||
- src/users/users.service.ts::UsersService.findOne
|
||||
|
||||
- id: di-call_chain-AuthGuard-canActivate
|
||||
tier: di
|
||||
query: "call_chain:src/auth/auth.guard.ts::AuthGuard.canActivate"
|
||||
expected:
|
||||
- src/auth/auth.service.ts::AuthService.validate
|
||||
- src/users/users.service.ts::UsersService.findOne
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 4 — di_gap: shapes Tier-2 type resolution cannot handle.
|
||||
# These queries should be near-0% without module-aware extraction;
|
||||
# they're the cases that would justify shipping option 1.
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# Abstract-class injection: NotificationsController injects Notifier
|
||||
# (abstract), module binds it to EmailNotifier. Finding the caller of
|
||||
# EmailNotifier.notify requires knowing that binding.
|
||||
- id: di_gap-callers-EmailNotifier-notify
|
||||
tier: di_gap
|
||||
query: "callers:src/notifications/email-notifier.service.ts::EmailNotifier.notify"
|
||||
expected:
|
||||
- src/notifications/notifications.controller.ts::NotificationsController.send
|
||||
|
||||
# Call-chain forward from the controller into the concrete method
|
||||
# behind the abstract binding. Same gap, traversing the other way.
|
||||
- id: di_gap-call_chain-NotificationsController-send
|
||||
tier: di_gap
|
||||
query: "call_chain:src/notifications/notifications.controller.ts::NotificationsController.send"
|
||||
expected:
|
||||
# Both are legitimate downstream calls from the handler: the guard
|
||||
# fires before the body via @UseGuards dispatch, the body's
|
||||
# `this.notifier.notify()` resolves through the useClass binding.
|
||||
- src/notifications/email-notifier.service.ts::EmailNotifier.notify
|
||||
- src/auth/auth.guard.ts::AuthGuard.canActivate
|
||||
|
||||
# @Inject(DATABASE_URL) token consumer. There's no type linking
|
||||
# ConfigService's dbUrl param to the `{ provide: DATABASE_URL,
|
||||
# useValue: ... }` entry — only the provider registry knows.
|
||||
- id: di_gap-usages-DATABASE_URL
|
||||
tier: di_gap
|
||||
query: "usages:src/config/config.tokens.ts::DATABASE_URL"
|
||||
expected:
|
||||
- src/config/config.service.ts::ConfigService
|
||||
- src/config/config.module.ts::ConfigModule
|
||||
|
||||
- id: di_gap-usages-FEATURE_FLAGS
|
||||
tier: di_gap
|
||||
query: "usages:src/config/config.tokens.ts::FEATURE_FLAGS"
|
||||
expected:
|
||||
- src/config/config.service.ts::ConfigService
|
||||
- src/config/config.module.ts::ConfigModule
|
||||
|
||||
# --- @UseGuards decorator-dispatch ---
|
||||
# NotificationsController.send is decorated with @UseGuards(AuthGuard).
|
||||
# The framework invokes AuthGuard.canActivate before the handler runs,
|
||||
# but there is no explicit call site — the binding lives in the
|
||||
# decorator metadata. get_callers on canActivate should include the
|
||||
# guarded handler.
|
||||
- id: di_gap-callers-AuthGuard-canActivate
|
||||
tier: di_gap
|
||||
query: "callers:src/auth/auth.guard.ts::AuthGuard.canActivate"
|
||||
expected:
|
||||
- src/notifications/notifications.controller.ts::NotificationsController.send
|
||||
|
||||
# --- Factory provider with inject: [Dep] ---
|
||||
# BillingService injects DB_CONNECTION, which BillingModule binds to a
|
||||
# DatabaseConnection produced by a factory that takes ConfigService.
|
||||
# Two signals to check:
|
||||
# 1. Token-level: usages of DB_CONNECTION.
|
||||
# 2. Call chain: BillingService.charge should reach
|
||||
# DatabaseConnection.query via the factory's produced binding.
|
||||
- id: di_gap-usages-DB_CONNECTION
|
||||
tier: di_gap
|
||||
query: "usages:src/billing/billing.tokens.ts::DB_CONNECTION"
|
||||
expected:
|
||||
- src/billing/billing.service.ts::BillingService
|
||||
- src/billing/billing.module.ts::BillingModule
|
||||
|
||||
- id: di_gap-call_chain-BillingService-charge
|
||||
tier: di_gap
|
||||
query: "call_chain:src/billing/billing.service.ts::BillingService.charge"
|
||||
expected:
|
||||
- src/billing/database.connection.ts::DatabaseConnection.query
|
||||
|
||||
# --- forwardRef circular injection ---
|
||||
# FeatureAService and FeatureBService inject each other via
|
||||
# `forwardRef(() => …)`. Static type resolution can't evaluate the
|
||||
# arrow, so without DI extraction the receiver types of `this.b` and
|
||||
# `this.a` are unknown and call chains across the cycle break.
|
||||
- id: di_gap-call_chain-FeatureAService-doA
|
||||
tier: di_gap
|
||||
query: "call_chain:src/feature/feature-a.service.ts::FeatureAService.doA"
|
||||
expected:
|
||||
- src/feature/feature-b.service.ts::FeatureBService.doB
|
||||
|
||||
- id: di_gap-call_chain-FeatureBService-callA
|
||||
tier: di_gap
|
||||
query: "call_chain:src/feature/feature-b.service.ts::FeatureBService.callA"
|
||||
expected:
|
||||
- src/feature/feature-a.service.ts::FeatureAService.doA
|
||||
|
||||
# --- Property (field) injection ---
|
||||
# NestJS permits @Inject on class fields instead of constructor params.
|
||||
# AuditService uses both the implicit (`@Inject() field!: T`) and
|
||||
# explicit-token (`@Inject(TOKEN) field: ...`) shapes. Without field-
|
||||
# level decorator extraction the call chain stops at AuditService —
|
||||
# the call to `this.users.findOne` has no receiver_type on its edge,
|
||||
# and find_usages on DATABASE_URL misses AuditService entirely.
|
||||
- id: di_gap-call_chain-AuditService-recordLogin
|
||||
tier: di_gap
|
||||
query: "call_chain:src/audit/audit.service.ts::AuditService.recordLogin"
|
||||
expected:
|
||||
- src/users/users.service.ts::UsersService.findOne
|
||||
|
||||
- id: di_gap-usages-DATABASE_URL-includes-audit
|
||||
tier: di_gap
|
||||
query: "usages:src/config/config.tokens.ts::DATABASE_URL"
|
||||
expected:
|
||||
- src/audit/audit.service.ts::AuditService
|
||||
|
||||
# --- Dynamic modules (forRoot / forFeature) ---
|
||||
# CacheModule.forRoot(ttl) returns a DynamicModule whose providers
|
||||
# array binds CACHE_TTL_SECONDS to the runtime ttl value. The binding
|
||||
# lives inside a static method body, not a @Module decorator — so
|
||||
# without dynamic-module extraction find_usages on the token surfaces
|
||||
# only the consumer, never the provider module.
|
||||
- id: di_gap-usages-CACHE_TTL_SECONDS
|
||||
tier: di_gap
|
||||
query: "usages:src/cache/cache.tokens.ts::CACHE_TTL_SECONDS"
|
||||
expected:
|
||||
- src/cache/cache.service.ts::CacheService
|
||||
- src/cache/cache.module.ts::CacheModule
|
||||
@@ -0,0 +1,23 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { UsersModule } from './users/users.module';
|
||||
import { AuthModule } from './auth/auth.module';
|
||||
import { NotificationsModule } from './notifications/notifications.module';
|
||||
import { ConfigModule } from './config/config.module';
|
||||
import { BillingModule } from './billing/billing.module';
|
||||
import { FeatureModule } from './feature/feature.module';
|
||||
import { AuditModule } from './audit/audit.module';
|
||||
import { CacheModule } from './cache/cache.module';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
UsersModule,
|
||||
AuthModule,
|
||||
NotificationsModule,
|
||||
ConfigModule,
|
||||
BillingModule,
|
||||
FeatureModule,
|
||||
AuditModule,
|
||||
CacheModule.forRoot(60),
|
||||
],
|
||||
})
|
||||
export class AppModule {}
|
||||
@@ -0,0 +1,11 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { UsersModule } from '../users/users.module';
|
||||
import { ConfigModule } from '../config/config.module';
|
||||
import { AuditService } from './audit.service';
|
||||
|
||||
@Module({
|
||||
imports: [UsersModule, ConfigModule],
|
||||
providers: [AuditService],
|
||||
exports: [AuditService],
|
||||
})
|
||||
export class AuditModule {}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { Inject, Injectable } from '@nestjs/common';
|
||||
import { UsersService } from '../users/users.service';
|
||||
import { DATABASE_URL } from '../config/config.tokens';
|
||||
|
||||
// Property injection: NestJS supports @Inject on class fields for cases
|
||||
// where the class can't declare a constructor (or the author prefers it
|
||||
// over parameter properties). Two shapes exercised:
|
||||
// - `@Inject() field!: T` — implicit token, class type drives binding
|
||||
// - `@Inject(TOKEN) field: ...` — explicit token, same as constructor form
|
||||
@Injectable()
|
||||
export class AuditService {
|
||||
@Inject()
|
||||
private readonly users!: UsersService;
|
||||
|
||||
@Inject(DATABASE_URL)
|
||||
private readonly dbUrl!: string;
|
||||
|
||||
async recordLogin(userId: string): Promise<string> {
|
||||
const user = await this.users.findOne(userId);
|
||||
return `audit(${this.dbUrl}): login by ${user.id}`;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import {
|
||||
CanActivate,
|
||||
ExecutionContext,
|
||||
Injectable,
|
||||
UnauthorizedException,
|
||||
} from '@nestjs/common';
|
||||
import { AuthService } from './auth.service';
|
||||
|
||||
@Injectable()
|
||||
export class AuthGuard implements CanActivate {
|
||||
constructor(private readonly authService: AuthService) {}
|
||||
|
||||
async canActivate(context: ExecutionContext): Promise<boolean> {
|
||||
const req = context.switchToHttp().getRequest();
|
||||
const userId = req.headers['x-user-id'];
|
||||
const token = req.headers['x-token'];
|
||||
if (!userId || !token) {
|
||||
throw new UnauthorizedException('missing credentials');
|
||||
}
|
||||
return this.authService.validate(userId, token);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { UsersModule } from '../users/users.module';
|
||||
import { AuthService } from './auth.service';
|
||||
import { AuthGuard } from './auth.guard';
|
||||
|
||||
@Module({
|
||||
imports: [UsersModule],
|
||||
providers: [AuthService, AuthGuard],
|
||||
exports: [AuthService, AuthGuard],
|
||||
})
|
||||
export class AuthModule {}
|
||||
@@ -0,0 +1,20 @@
|
||||
import { Injectable, UnauthorizedException } from '@nestjs/common';
|
||||
import { UsersService } from '../users/users.service';
|
||||
|
||||
@Injectable()
|
||||
export class AuthService {
|
||||
constructor(private readonly usersService: UsersService) {}
|
||||
|
||||
async validate(userId: string, token: string): Promise<boolean> {
|
||||
const user = await this.usersService.findOne(userId);
|
||||
if (!user || token !== `tok_${user.id}`) {
|
||||
throw new UnauthorizedException('invalid token');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
async issueToken(userId: string): Promise<string> {
|
||||
const user = await this.usersService.findOne(userId);
|
||||
return `tok_${user.id}`;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { ConfigModule } from '../config/config.module';
|
||||
import { ConfigService } from '../config/config.service';
|
||||
import { DB_CONNECTION } from './billing.tokens';
|
||||
import { DatabaseConnection } from './database.connection';
|
||||
import { BillingService } from './billing.service';
|
||||
|
||||
// Factory provider: the binding DB_CONNECTION → DatabaseConnection is
|
||||
// produced by calling `dbFactory(cfg)` at module bootstrap. `inject:
|
||||
// [ConfigService]` is how Nest passes the factory its dependencies.
|
||||
// There is no `new DatabaseConnection(...)` call anywhere else in the
|
||||
// codebase; any graph analysis that wants to traverse "consumers of
|
||||
// DB_CONNECTION → DatabaseConnection methods" must read this factory.
|
||||
const dbFactory = (cfg: ConfigService) =>
|
||||
new DatabaseConnection(cfg.getDatabaseUrl());
|
||||
|
||||
@Module({
|
||||
imports: [ConfigModule],
|
||||
providers: [
|
||||
{
|
||||
provide: DB_CONNECTION,
|
||||
useFactory: dbFactory,
|
||||
inject: [ConfigService],
|
||||
},
|
||||
BillingService,
|
||||
],
|
||||
exports: [BillingService],
|
||||
})
|
||||
export class BillingModule {}
|
||||
@@ -0,0 +1,27 @@
|
||||
import { Inject, Injectable } from '@nestjs/common';
|
||||
import { DatabaseConnection } from './database.connection';
|
||||
import { DB_CONNECTION } from './billing.tokens';
|
||||
|
||||
// Consumer of the factory-provided DatabaseConnection. The `db` param's
|
||||
// declared type IS DatabaseConnection — so with parameter-property typing
|
||||
// the receiver type resolution should work for `this.db.query(...)`. But
|
||||
// the *binding* from the token DB_CONNECTION to DatabaseConnection lives
|
||||
// only inside the factory — a graph walk from any consumer to "who
|
||||
// produces DB_CONNECTION" has no edge to follow without DI extraction.
|
||||
@Injectable()
|
||||
export class BillingService {
|
||||
constructor(
|
||||
@Inject(DB_CONNECTION) private readonly db: DatabaseConnection,
|
||||
) {}
|
||||
|
||||
async charge(userId: string, cents: number): Promise<void> {
|
||||
await this.db.query(`INSERT INTO charges (user_id, amount) VALUES ('${userId}', ${cents})`);
|
||||
}
|
||||
|
||||
async totals(userId: string): Promise<number> {
|
||||
const rows = await this.db.query<{ total: number }>(
|
||||
`SELECT SUM(amount) AS total FROM charges WHERE user_id = '${userId}'`,
|
||||
);
|
||||
return rows[0]?.total ?? 0;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
// Separate file for the injection token so the module import graph
|
||||
// reflects real NestJS practice — consumers import the token without
|
||||
// pulling in the module and its factory provider.
|
||||
export const DB_CONNECTION = 'DB_CONNECTION';
|
||||
@@ -0,0 +1,17 @@
|
||||
// Concrete class wired up through a factory provider. The only place
|
||||
// the binding `"DB_CONNECTION" → DatabaseConnection` appears is inside
|
||||
// a `useFactory` in billing.module.ts — no `new DatabaseConnection(...)`
|
||||
// call survives anywhere else, so Tier-2 type resolution can't reach
|
||||
// this class from consumers that inject `@Inject('DB_CONNECTION')`.
|
||||
export class DatabaseConnection {
|
||||
constructor(public readonly url: string) {}
|
||||
|
||||
async query<T>(sql: string): Promise<T[]> {
|
||||
console.log(`[${this.url}] ${sql}`);
|
||||
return [];
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
// Close the pool in real code.
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { DynamicModule, Module } from '@nestjs/common';
|
||||
import { CACHE_TTL_SECONDS } from './cache.tokens';
|
||||
import { CacheService } from './cache.service';
|
||||
|
||||
// Dynamic module: NestJS's forRoot / forFeature pattern returns a
|
||||
// runtime-computed module config. The providers array here is
|
||||
// structurally identical to a @Module providers array — same
|
||||
// { provide: X, useValue: ... } shape — but lives inside a static
|
||||
// method body instead of a decorator.
|
||||
@Module({})
|
||||
export class CacheModule {
|
||||
static forRoot(ttlSeconds: number): DynamicModule {
|
||||
return {
|
||||
module: CacheModule,
|
||||
providers: [
|
||||
{ provide: CACHE_TTL_SECONDS, useValue: ttlSeconds },
|
||||
CacheService,
|
||||
],
|
||||
exports: [CacheService],
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { Inject, Injectable } from '@nestjs/common';
|
||||
import { CACHE_TTL_SECONDS } from './cache.tokens';
|
||||
|
||||
// Consumer of a token provided only through CacheModule.forRoot().
|
||||
// Without dynamic-module extraction the graph sees the @Inject here
|
||||
// but no provider, so find_usages(CACHE_TTL_SECONDS) returns only
|
||||
// this consumer and leaves orphan-detection incomplete.
|
||||
@Injectable()
|
||||
export class CacheService {
|
||||
constructor(@Inject(CACHE_TTL_SECONDS) private readonly ttl: number) {}
|
||||
|
||||
ttlSeconds(): number {
|
||||
return this.ttl;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
export const CACHE_TTL_SECONDS = 'CACHE_TTL_SECONDS';
|
||||
@@ -0,0 +1,19 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { ConfigService } from './config.service';
|
||||
|
||||
// Second-hop consumer: ConfigConsumer calls ConfigService, which itself
|
||||
// received its values via @Inject(TOKEN). The test is whether the graph
|
||||
// can reach *from* ConfigConsumer *to* the methods on ConfigService —
|
||||
// standard typed injection, should work — and whether the TOKEN-keyed
|
||||
// providers show up in any cross-reference (they won't, without DI
|
||||
// extraction).
|
||||
@Injectable()
|
||||
export class ConfigConsumer {
|
||||
constructor(private readonly config: ConfigService) {}
|
||||
|
||||
describe(): string {
|
||||
return `db=${this.config.getDatabaseUrl()} flags=${JSON.stringify(
|
||||
this.config.isEnabled('beta'),
|
||||
)}`;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { DATABASE_URL, FEATURE_FLAGS } from './config.tokens';
|
||||
import { ConfigService } from './config.service';
|
||||
import { ConfigConsumer } from './config.consumer';
|
||||
|
||||
@Module({
|
||||
providers: [
|
||||
{ provide: DATABASE_URL, useValue: 'postgres://localhost/test' },
|
||||
{ provide: FEATURE_FLAGS, useValue: { beta: true } },
|
||||
ConfigService,
|
||||
ConfigConsumer,
|
||||
],
|
||||
exports: [ConfigService, ConfigConsumer],
|
||||
})
|
||||
export class ConfigModule {}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { Inject, Injectable } from '@nestjs/common';
|
||||
import { DATABASE_URL, FEATURE_FLAGS } from './config.tokens';
|
||||
|
||||
@Injectable()
|
||||
export class ConfigService {
|
||||
// Both constructor params receive their values via string-keyed
|
||||
// @Inject — no type info survives into the graph for the binder to
|
||||
// traverse.
|
||||
constructor(
|
||||
@Inject(DATABASE_URL) private readonly dbUrl: string,
|
||||
@Inject(FEATURE_FLAGS) private readonly flags: Record<string, boolean>,
|
||||
) {}
|
||||
|
||||
getDatabaseUrl(): string {
|
||||
return this.dbUrl;
|
||||
}
|
||||
|
||||
isEnabled(flag: string): boolean {
|
||||
return Boolean(this.flags[flag]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
// String-keyed injection tokens. These are the NestJS idiom for
|
||||
// providing values that have no class identity — environment URLs,
|
||||
// configuration primitives, feature flags. A consumer asks for them
|
||||
// via `@Inject(DATABASE_URL)`; the Module provides them via
|
||||
// `{ provide: DATABASE_URL, useValue: '...' }`. No type-aware
|
||||
// resolution can cross this boundary.
|
||||
export const DATABASE_URL = 'DATABASE_URL';
|
||||
export const FEATURE_FLAGS = 'FEATURE_FLAGS';
|
||||
@@ -0,0 +1,20 @@
|
||||
import { forwardRef, Inject, Injectable } from '@nestjs/common';
|
||||
import { FeatureBService } from './feature-b.service';
|
||||
|
||||
// Two services that reference each other. NestJS requires forwardRef()
|
||||
// to break the import cycle; the lazy arrow function `() => FeatureBService`
|
||||
// is evaluated after module init. Static extraction can't call the arrow
|
||||
// at parse time, so the `b` parameter's declared type is effectively
|
||||
// unknown (or `any`) at extraction. Type-aware resolution can't link
|
||||
// `this.b.doB()` to FeatureBService.doB without following the forwardRef.
|
||||
@Injectable()
|
||||
export class FeatureAService {
|
||||
constructor(
|
||||
@Inject(forwardRef(() => FeatureBService))
|
||||
private readonly b: FeatureBService,
|
||||
) {}
|
||||
|
||||
doA(): string {
|
||||
return this.b.doB() + ' from A';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { forwardRef, Inject, Injectable } from '@nestjs/common';
|
||||
import { FeatureAService } from './feature-a.service';
|
||||
|
||||
@Injectable()
|
||||
export class FeatureBService {
|
||||
constructor(
|
||||
@Inject(forwardRef(() => FeatureAService))
|
||||
private readonly a: FeatureAService,
|
||||
) {}
|
||||
|
||||
doB(): string {
|
||||
return 'B';
|
||||
}
|
||||
|
||||
callA(): string {
|
||||
return this.a.doA();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { FeatureAService } from './feature-a.service';
|
||||
import { FeatureBService } from './feature-b.service';
|
||||
|
||||
@Module({
|
||||
providers: [FeatureAService, FeatureBService],
|
||||
exports: [FeatureAService, FeatureBService],
|
||||
})
|
||||
export class FeatureModule {}
|
||||
@@ -0,0 +1,11 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { Notifier } from './notifier.interface';
|
||||
|
||||
@Injectable()
|
||||
export class EmailNotifier extends Notifier {
|
||||
async notify(userId: string, message: string): Promise<void> {
|
||||
// In real code this would call an SMTP client; here the logic is
|
||||
// deliberately trivial — the fixture only needs the method to exist.
|
||||
console.log(`email to ${userId}: ${message}`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import { Body, Controller, Post, UseGuards } from '@nestjs/common';
|
||||
import { Notifier } from './notifier.interface';
|
||||
import { AuthGuard } from '../auth/auth.guard';
|
||||
|
||||
@Controller('notifications')
|
||||
export class NotificationsController {
|
||||
// Injected by the abstract base class — the runtime instance will
|
||||
// be EmailNotifier (see notifications.module.ts), but the static
|
||||
// type here is `Notifier` so a type-aware resolver has no way to
|
||||
// pick the right concrete method.
|
||||
constructor(private readonly notifier: Notifier) {}
|
||||
|
||||
// @UseGuards binds the guard's canActivate() to this route. There is
|
||||
// no explicit call site to AuthGuard.canActivate anywhere in source —
|
||||
// the framework invokes it at request time. Without decorator-dispatch
|
||||
// extraction, get_callers on canActivate is empty for this endpoint.
|
||||
@Post('send')
|
||||
@UseGuards(AuthGuard)
|
||||
async send(@Body('userId') userId: string, @Body('message') message: string): Promise<void> {
|
||||
await this.notifier.notify(userId, message);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { Notifier } from './notifier.interface';
|
||||
import { EmailNotifier } from './email-notifier.service';
|
||||
import { SmsNotifier } from './sms-notifier.service';
|
||||
import { NotificationsController } from './notifications.controller';
|
||||
import { AuthModule } from '../auth/auth.module';
|
||||
|
||||
// The `provide: Notifier, useClass: EmailNotifier` binding is the only
|
||||
// place a static reader can learn that NotificationsController's
|
||||
// `notifier` parameter resolves to EmailNotifier specifically.
|
||||
@Module({
|
||||
imports: [AuthModule],
|
||||
controllers: [NotificationsController],
|
||||
providers: [
|
||||
{ provide: Notifier, useClass: EmailNotifier },
|
||||
SmsNotifier,
|
||||
],
|
||||
exports: [Notifier, SmsNotifier],
|
||||
})
|
||||
export class NotificationsModule {}
|
||||
@@ -0,0 +1,8 @@
|
||||
// Abstract base class used as the injection token — NestJS supports
|
||||
// `constructor(private n: Notifier)` where Notifier is an abstract class
|
||||
// and the Module provides `{ provide: Notifier, useClass: EmailNotifier }`.
|
||||
// Type-aware resolution can't cross this indirection without knowing the
|
||||
// provider binding; exactly the gap option 1's DI extractor targets.
|
||||
export abstract class Notifier {
|
||||
abstract notify(userId: string, message: string): Promise<void>;
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { Notifier } from './notifier.interface';
|
||||
|
||||
// Second implementation of the same abstract class. Makes the
|
||||
// abstract-injection case ambiguous — module-level binding is what
|
||||
// actually picks between EmailNotifier and SmsNotifier. This is the
|
||||
// shape Tier-2 type-aware resolution cannot handle.
|
||||
@Injectable()
|
||||
export class SmsNotifier extends Notifier {
|
||||
async notify(userId: string, message: string): Promise<void> {
|
||||
console.log(`sms to ${userId}: ${message}`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
export class User {
|
||||
constructor(
|
||||
public readonly id: string,
|
||||
public readonly email: string,
|
||||
public readonly name: string,
|
||||
) {}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { Body, Controller, Get, Param, Post } from '@nestjs/common';
|
||||
import { UsersService } from './users.service';
|
||||
import { User } from './user.entity';
|
||||
|
||||
@Controller('users')
|
||||
export class UsersController {
|
||||
constructor(private readonly usersService: UsersService) {}
|
||||
|
||||
@Get()
|
||||
async list(): Promise<User[]> {
|
||||
return this.usersService.findAll();
|
||||
}
|
||||
|
||||
@Get(':id')
|
||||
async getUser(@Param('id') id: string): Promise<User> {
|
||||
return this.usersService.findOne(id);
|
||||
}
|
||||
|
||||
@Post()
|
||||
async create(
|
||||
@Body('email') email: string,
|
||||
@Body('name') name: string,
|
||||
): Promise<User> {
|
||||
return this.usersService.create(email, name);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import { Module } from '@nestjs/common';
|
||||
import { UsersController } from './users.controller';
|
||||
import { UsersService } from './users.service';
|
||||
|
||||
@Module({
|
||||
controllers: [UsersController],
|
||||
providers: [UsersService],
|
||||
exports: [UsersService],
|
||||
})
|
||||
export class UsersModule {}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { Injectable, NotFoundException } from '@nestjs/common';
|
||||
import { User } from './user.entity';
|
||||
|
||||
@Injectable()
|
||||
export class UsersService {
|
||||
private readonly users: Map<string, User> = new Map();
|
||||
|
||||
async findOne(id: string): Promise<User> {
|
||||
const u = this.users.get(id);
|
||||
if (!u) {
|
||||
throw new NotFoundException(`user ${id} not found`);
|
||||
}
|
||||
return u;
|
||||
}
|
||||
|
||||
async findAll(): Promise<User[]> {
|
||||
return Array.from(this.users.values());
|
||||
}
|
||||
|
||||
async create(email: string, name: string): Promise<User> {
|
||||
const id = `usr_${this.users.size + 1}`;
|
||||
const user = new User(id, email, name);
|
||||
this.users.set(id, user);
|
||||
return user;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2021",
|
||||
"module": "commonjs",
|
||||
"experimentalDecorators": true,
|
||||
"emitDecoratorMetadata": true,
|
||||
"strict": true,
|
||||
"baseUrl": "./src"
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
defmodule MyAppWeb.UserController do
|
||||
use Phoenix.Controller, namespace: MyAppWeb
|
||||
|
||||
# Controller-level plug: runs before every action in this module.
|
||||
# Phoenix resolves the binding at request time via the plug chain;
|
||||
# there's no explicit call site between the action and the plug
|
||||
# function.
|
||||
plug :authenticate
|
||||
|
||||
# Guarded form with `when action in [...]`: same DI shape as Rails's
|
||||
# `before_action :name, only: [...]`. The plug only fires for the
|
||||
# listed actions.
|
||||
plug :load_user when action in [:show, :update, :delete]
|
||||
|
||||
# Regular actions. Bodies deliberately do NOT call authenticate or
|
||||
# load_user directly — the only path from these actions to those
|
||||
# plugs is the decorator-dispatch edge we want to extract.
|
||||
def index(conn, _params) do
|
||||
render(conn, :index)
|
||||
end
|
||||
|
||||
def show(conn, _params) do
|
||||
render(conn, :show, user: conn.assigns[:user])
|
||||
end
|
||||
|
||||
def update(conn, params) do
|
||||
user = conn.assigns[:user]
|
||||
_ = Map.merge(user, params)
|
||||
render(conn, :show, user: user)
|
||||
end
|
||||
|
||||
def delete(conn, _params) do
|
||||
send_resp(conn, 204, "")
|
||||
end
|
||||
|
||||
# Plug functions themselves. Their bodies have real call edges (to
|
||||
# Plug.Conn helpers etc.) which the existing extractor handles;
|
||||
# only the binding from actions to these functions is DI-gap
|
||||
# territory.
|
||||
def authenticate(conn, _opts) do
|
||||
case get_session(conn, :user_id) do
|
||||
nil -> halt_unauthorized(conn)
|
||||
_ -> conn
|
||||
end
|
||||
end
|
||||
|
||||
def load_user(conn, _opts) do
|
||||
user = %{id: conn.params["id"]}
|
||||
assign(conn, :user, user)
|
||||
end
|
||||
|
||||
defp halt_unauthorized(conn) do
|
||||
send_resp(conn, 401, "unauthorized")
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,7 @@
|
||||
defmodule MyAppWeb.Router do
|
||||
use Phoenix.Router
|
||||
|
||||
scope "/", MyAppWeb do
|
||||
resources "/users", UserController
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,13 @@
|
||||
defmodule MyApp.MixProject do
|
||||
# Not built — static corpus for the Gortex indexer.
|
||||
use Mix.Project
|
||||
|
||||
def project do
|
||||
[
|
||||
app: :my_app,
|
||||
version: "0.0.0",
|
||||
elixir: "~> 1.14",
|
||||
deps: [{:phoenix, "~> 1.7"}]
|
||||
]
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,58 @@
|
||||
# Phoenix DI-recall fixture
|
||||
#
|
||||
# Phoenix's controller-level DI is `plug :function_name` — binds a
|
||||
# middleware function to run before each action. Same shape as Rails
|
||||
# before_action / NestJS @UseGuards: no explicit call site, framework
|
||||
# dispatches based on module metadata.
|
||||
#
|
||||
# Only controller-scoped plugs are covered here. Router-pipeline
|
||||
# plugs (`pipeline :browser do plug X end`) cross module boundaries
|
||||
# and are noted as future work.
|
||||
name: gortex-phoenix-di
|
||||
cases:
|
||||
|
||||
# exact
|
||||
- id: exact-UserController
|
||||
tier: exact
|
||||
query: UserController
|
||||
expected: [lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController]
|
||||
- id: exact-authenticate
|
||||
tier: exact
|
||||
query: authenticate
|
||||
expected: [lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.authenticate]
|
||||
- id: exact-load_user
|
||||
tier: exact
|
||||
query: load_user
|
||||
expected: [lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.load_user]
|
||||
|
||||
# di_gap: controller-scoped plug dispatch
|
||||
# `plug :authenticate` runs for every action. callers:authenticate
|
||||
# should return every action in the controller.
|
||||
- id: di_gap-callers-authenticate
|
||||
tier: di_gap
|
||||
query: "callers:lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.authenticate"
|
||||
expected:
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.index
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.show
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.update
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.delete
|
||||
|
||||
# `plug :load_user when action in [:show, :update, :delete]` —
|
||||
# only fires for those three actions. index/index-like should NOT
|
||||
# be linked.
|
||||
- id: di_gap-callers-load_user
|
||||
tier: di_gap
|
||||
query: "callers:lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.load_user"
|
||||
expected:
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.show
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.update
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.delete
|
||||
|
||||
# Forward direction: call_chain from a guarded action should reach
|
||||
# the plug functions it depends on.
|
||||
- id: di_gap-call_chain-show
|
||||
tier: di_gap
|
||||
query: "call_chain:lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.show"
|
||||
expected:
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.authenticate
|
||||
- lib/my_app_web/controllers/user_controller.ex::MyAppWeb.UserController.load_user
|
||||
@@ -0,0 +1,3 @@
|
||||
source 'https://rubygems.org'
|
||||
# Not installed — purely a static corpus for the Gortex indexer.
|
||||
gem 'rails', '~> 7.1'
|
||||
@@ -0,0 +1,13 @@
|
||||
class ApplicationController < ActionController::Base
|
||||
# Module-level before_action — runs for every action in every
|
||||
# subclass. Rails resolves this at request-time via the callback
|
||||
# chain; statically the binding between :require_login and the
|
||||
# actions it guards has no explicit call site.
|
||||
before_action :require_login
|
||||
|
||||
private
|
||||
|
||||
def require_login
|
||||
redirect_to '/login' unless session[:user_id]
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,20 @@
|
||||
class SessionsController < ApplicationController
|
||||
# skip_before_action removes the inherited :require_login for the
|
||||
# listed actions — login/create obviously can't require a pre-
|
||||
# existing login. Same decorator-dispatch shape, opposite sign.
|
||||
skip_before_action :require_login, only: [:new, :create]
|
||||
|
||||
def new
|
||||
end
|
||||
|
||||
def create
|
||||
user = User.authenticate(params[:email], params[:password])
|
||||
session[:user_id] = user.id
|
||||
redirect_to '/'
|
||||
end
|
||||
|
||||
def destroy
|
||||
session.delete(:user_id)
|
||||
redirect_to '/login'
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,44 @@
|
||||
class UsersController < ApplicationController
|
||||
# Targeted before_action: only fires for the listed actions. The
|
||||
# comma-separated symbol list + `only:` / `except:` filters are the
|
||||
# common Rails idiom for attaching methods to a subset of actions.
|
||||
before_action :set_user, only: [:show, :update, :destroy]
|
||||
before_action :authorize_admin, only: :destroy
|
||||
after_action :log_request
|
||||
|
||||
def index
|
||||
@users = User.all
|
||||
end
|
||||
|
||||
def show
|
||||
render json: @user
|
||||
end
|
||||
|
||||
def update
|
||||
@user.update(user_params)
|
||||
render json: @user
|
||||
end
|
||||
|
||||
def destroy
|
||||
@user.destroy
|
||||
head :no_content
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def set_user
|
||||
@user = User.find(params[:id])
|
||||
end
|
||||
|
||||
def authorize_admin
|
||||
head :forbidden unless current_user&.admin?
|
||||
end
|
||||
|
||||
def log_request
|
||||
Rails.logger.info("request: #{request.path}")
|
||||
end
|
||||
|
||||
def user_params
|
||||
params.require(:user).permit(:email, :name)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,18 @@
|
||||
class User < ApplicationRecord
|
||||
has_many :posts
|
||||
|
||||
validates :email, presence: true, uniqueness: true
|
||||
|
||||
def admin?
|
||||
role == 'admin'
|
||||
end
|
||||
|
||||
def self.authenticate(email, password)
|
||||
user = find_by(email: email)
|
||||
user if user&.valid_password?(password)
|
||||
end
|
||||
|
||||
def valid_password?(_pw)
|
||||
true
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,6 @@
|
||||
Rails.application.routes.draw do
|
||||
resources :users
|
||||
get '/login', to: 'sessions#new'
|
||||
post '/login', to: 'sessions#create'
|
||||
delete '/logout', to: 'sessions#destroy'
|
||||
end
|
||||
@@ -0,0 +1,90 @@
|
||||
# Rails DI-recall fixture
|
||||
#
|
||||
# Rails "DI" is mostly callback dispatch — before_action / after_action
|
||||
# / skip_before_action in controllers attach methods to run before or
|
||||
# after actions, without any explicit call site. This is the same shape
|
||||
# as NestJS @UseGuards or Phoenix plug. Routes (`get '/users', to: ...`)
|
||||
# are already handled by Gortex's HTTPExtractor so they're not the
|
||||
# focus here.
|
||||
#
|
||||
# Tiering:
|
||||
# exact — named symbols
|
||||
# concept — paraphrases
|
||||
# di — plain method call resolution (no callback extraction
|
||||
# needed)
|
||||
# di_gap — callback dispatch that's invisible without
|
||||
# before_action-aware extraction
|
||||
name: gortex-rails-di
|
||||
cases:
|
||||
|
||||
# exact
|
||||
- id: exact-UsersController
|
||||
tier: exact
|
||||
query: UsersController
|
||||
expected: [app/controllers/users_controller.rb::UsersController]
|
||||
- id: exact-ApplicationController
|
||||
tier: exact
|
||||
query: ApplicationController
|
||||
expected: [app/controllers/application_controller.rb::ApplicationController]
|
||||
- id: exact-require_login
|
||||
tier: exact
|
||||
query: require_login
|
||||
expected: [app/controllers/application_controller.rb::ApplicationController.require_login]
|
||||
- id: exact-authenticate
|
||||
tier: exact
|
||||
query: authenticate
|
||||
expected: [app/models/user.rb::User.authenticate]
|
||||
|
||||
# concept
|
||||
- id: concept-destroy-session
|
||||
tier: concept
|
||||
query: log a user out
|
||||
expected: [app/controllers/sessions_controller.rb::SessionsController.destroy]
|
||||
|
||||
# di — real method body calls (User.find, session[:user_id], etc.)
|
||||
- id: di-call_chain-create
|
||||
tier: di
|
||||
query: "call_chain:app/controllers/sessions_controller.rb::SessionsController.create"
|
||||
expected: [app/models/user.rb::User.authenticate]
|
||||
|
||||
# di_gap — callback dispatch
|
||||
# NOTE: inherited callbacks (ApplicationController.before_action
|
||||
# :require_login → UsersController actions) aren't tracked. Would
|
||||
# require cross-file class-hierarchy walking. See the Rails feature
|
||||
# notes for future work.
|
||||
|
||||
# before_action :set_user, only: [:show, :update, :destroy] — the
|
||||
# filter list matters: only those three actions should call set_user.
|
||||
- id: di_gap-callers-set_user
|
||||
tier: di_gap
|
||||
query: "callers:app/controllers/users_controller.rb::UsersController.set_user"
|
||||
expected:
|
||||
- app/controllers/users_controller.rb::UsersController.show
|
||||
- app/controllers/users_controller.rb::UsersController.update
|
||||
- app/controllers/users_controller.rb::UsersController.destroy
|
||||
|
||||
# after_action :log_request — runs AFTER every action.
|
||||
- id: di_gap-callers-log_request
|
||||
tier: di_gap
|
||||
query: "callers:app/controllers/users_controller.rb::UsersController.log_request"
|
||||
expected:
|
||||
- app/controllers/users_controller.rb::UsersController.index
|
||||
- app/controllers/users_controller.rb::UsersController.show
|
||||
|
||||
# authorize_admin is before_action with only: :destroy (single symbol
|
||||
# form, not an array). Verifies the extractor handles both shapes.
|
||||
- id: di_gap-callers-authorize_admin
|
||||
tier: di_gap
|
||||
query: "callers:app/controllers/users_controller.rb::UsersController.authorize_admin"
|
||||
expected:
|
||||
- app/controllers/users_controller.rb::UsersController.destroy
|
||||
|
||||
# Forward direction: call_chain from a guarded action should include
|
||||
# the before_action callbacks. Tests the same edges from the other
|
||||
# side of the BFS.
|
||||
- id: di_gap-call_chain-UsersController-show
|
||||
tier: di_gap
|
||||
query: "call_chain:app/controllers/users_controller.rb::UsersController.show"
|
||||
expected:
|
||||
- app/controllers/users_controller.rb::UsersController.set_user
|
||||
- app/controllers/application_controller.rb::ApplicationController.require_login
|
||||
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Not built — purely a static corpus for the Gortex indexer. -->
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<groupId>com.example</groupId>
|
||||
<artifactId>gortex-di-spring-fixture</artifactId>
|
||||
<version>0.0.0</version>
|
||||
<packaging>jar</packaging>
|
||||
</project>
|
||||
@@ -0,0 +1,114 @@
|
||||
# Spring Boot DI-recall fixture
|
||||
#
|
||||
# Spring DI shapes in scope:
|
||||
# - @Autowired field injection
|
||||
# - @Autowired constructor injection (equivalent to @Inject)
|
||||
# - @Qualifier for multi-implementation disambiguation
|
||||
# - @Bean factory methods in a @Configuration class
|
||||
#
|
||||
# Tiering matches the NestJS/FastAPI fixtures:
|
||||
# exact — symbol-name queries
|
||||
# concept — paraphrase queries
|
||||
# di — shapes Java's explicit-type system already handles
|
||||
# (constructor param typed as the concrete class/interface
|
||||
# — the resolver links method calls on `this.field` to
|
||||
# the declared type)
|
||||
# di_gap — shapes a type-aware resolver can't handle alone
|
||||
# (interface injection with multiple impls, @Bean factory
|
||||
# return values, @Qualifier disambiguation)
|
||||
name: gortex-spring-di
|
||||
cases:
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 1 — exact
|
||||
# ------------------------------------------------------------------
|
||||
- id: exact-UserService
|
||||
tier: exact
|
||||
query: UserService
|
||||
expected: [src/main/java/com/example/users/UserService.java::UserService]
|
||||
- id: exact-OrderService
|
||||
tier: exact
|
||||
query: OrderService
|
||||
expected: [src/main/java/com/example/orders/OrderService.java::OrderService]
|
||||
- id: exact-JpaUserRepository
|
||||
tier: exact
|
||||
query: JpaUserRepository
|
||||
expected: [src/main/java/com/example/users/JpaUserRepository.java::JpaUserRepository]
|
||||
- id: exact-systemClock
|
||||
tier: exact
|
||||
query: systemClock
|
||||
expected: [src/main/java/com/example/config/Clocks.java::Clocks.systemClock]
|
||||
- id: exact-lookup
|
||||
tier: exact
|
||||
query: lookup
|
||||
expected: [src/main/java/com/example/users/UserService.java::UserService.lookup]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 2 — concept
|
||||
# ------------------------------------------------------------------
|
||||
- id: concept-find-user-cache-fallback
|
||||
tier: concept
|
||||
query: look up a user with cache fallback
|
||||
expected: [src/main/java/com/example/users/UserService.java::UserService.lookup]
|
||||
- id: concept-system-clock
|
||||
tier: concept
|
||||
query: provide the system clock
|
||||
expected: [src/main/java/com/example/config/Clocks.java::Clocks.systemClock]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 3 — di: Java's explicit type system handles these
|
||||
# ------------------------------------------------------------------
|
||||
# UserService.lookup calls this.cache.findById(...) and this.primary.findById(...).
|
||||
# `cache` is a constructor-injected InMemoryUserRepository (concrete class),
|
||||
# `primary` is a field-injected UserRepository (interface). Expected: the
|
||||
# concrete findById — Java's type-aware resolver gets the concrete call directly.
|
||||
- id: di-call_chain-UserService-lookup
|
||||
tier: di
|
||||
query: "call_chain:src/main/java/com/example/users/UserService.java::UserService.lookup"
|
||||
expected:
|
||||
- src/main/java/com/example/users/InMemoryUserRepository.java::InMemoryUserRepository.findById
|
||||
|
||||
- id: di-call_chain-OrderService-describe
|
||||
tier: di
|
||||
query: "call_chain:src/main/java/com/example/orders/OrderService.java::OrderService.describe"
|
||||
expected:
|
||||
- src/main/java/com/example/users/UserService.java::UserService.lookup
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tier 4 — di_gap: Spring-specific resolution beyond Java types
|
||||
# ------------------------------------------------------------------
|
||||
# Interface injection + multiple implementations: UserService.primary is
|
||||
# typed as UserRepository. Two @Repository beans implement it
|
||||
# (JpaUserRepository, InMemoryUserRepository). Without a @Primary or
|
||||
# @Qualifier annotation Spring itself fails at runtime — the static
|
||||
# resolver picks one arbitrarily. Either concrete implementation is
|
||||
# a legitimate answer here; a future @Primary-aware extractor would
|
||||
# narrow this to a specific impl.
|
||||
- id: di_gap-call_chain-UserService-lookup-interface
|
||||
tier: di_gap
|
||||
query: "call_chain:src/main/java/com/example/users/UserService.java::UserService.lookup"
|
||||
expected:
|
||||
- src/main/java/com/example/users/JpaUserRepository.java::JpaUserRepository.findById
|
||||
- src/main/java/com/example/users/InMemoryUserRepository.java::InMemoryUserRepository.findById
|
||||
|
||||
# @Bean factory method: OrderService receives a Clock via constructor,
|
||||
# but Clock isn't a @Service or @Component — it's provided by
|
||||
# Clocks.systemClock() which returns Clock. Without @Bean-aware
|
||||
# extraction, callers(systemClock) is empty and users of Clock can't
|
||||
# trace back to its factory.
|
||||
- id: di_gap-callers-systemClock
|
||||
tier: di_gap
|
||||
query: "callers:src/main/java/com/example/config/Clocks.java::Clocks.systemClock"
|
||||
expected:
|
||||
- src/main/java/com/example/orders/OrderService.java::OrderService
|
||||
|
||||
# @Qualifier disambiguation: UserService constructor param is
|
||||
# @Qualifier("inMemoryUsers") InMemoryUserRepository. The type is
|
||||
# explicit (concrete class), so Java's resolver already handles it.
|
||||
# This case should pass even WITHOUT Spring-aware extraction — tests
|
||||
# that the qualified injection doesn't break the baseline.
|
||||
- id: di-call_chain-UserService-cache-path
|
||||
tier: di
|
||||
query: "call_chain:src/main/java/com/example/users/UserService.java::UserService.lookup"
|
||||
expected:
|
||||
- src/main/java/com/example/users/InMemoryUserRepository.java::InMemoryUserRepository.findById
|
||||
@@ -0,0 +1,17 @@
|
||||
package com.example.config;
|
||||
|
||||
import java.time.Clock;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
// @Configuration + @Bean: factory-method style provider. Consumers
|
||||
// that @Autowire a Clock field receive whatever this method returns.
|
||||
// Statically the binding between the return type (Clock) and the
|
||||
// producing method (systemClock) is visible only inside this class.
|
||||
@Configuration
|
||||
public class Clocks {
|
||||
@Bean
|
||||
public Clock systemClock() {
|
||||
return Clock.systemUTC();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package com.example.orders;
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import com.example.users.User;
|
||||
import com.example.users.UserService;
|
||||
|
||||
// Multi-field injection by constructor. The Clock dependency is
|
||||
// satisfied by the @Bean method in com.example.config.Clocks; the
|
||||
// UserService dependency is the typical @Service-annotated class.
|
||||
@Service
|
||||
public class OrderService {
|
||||
private final UserService users;
|
||||
private final Clock clock;
|
||||
|
||||
@Autowired
|
||||
public OrderService(UserService users, Clock clock) {
|
||||
this.users = users;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
public String describe(String userId) {
|
||||
User u = users.lookup(userId).orElseThrow();
|
||||
Instant now = clock.instant();
|
||||
return "order for " + u.getEmail() + " at " + now;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package com.example.users;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
// Second implementation. Without @Primary on one of the two, a plain
|
||||
// @Autowired UserRepository field is ambiguous — Spring requires a
|
||||
// @Qualifier to pick between them.
|
||||
@Repository("inMemoryUsers")
|
||||
public class InMemoryUserRepository implements UserRepository {
|
||||
private final Map<String, User> users = new HashMap<>();
|
||||
|
||||
public Optional<User> findById(String id) {
|
||||
return Optional.ofNullable(users.get(id));
|
||||
}
|
||||
|
||||
public User save(User u) {
|
||||
users.put(u.getId(), u);
|
||||
return u;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package com.example.users;
|
||||
|
||||
import java.util.Optional;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
// First implementation of UserRepository. The @Primary elsewhere
|
||||
// determines which Spring picks when a consumer @Autowires the
|
||||
// interface without a qualifier.
|
||||
@Repository
|
||||
public class JpaUserRepository implements UserRepository {
|
||||
public Optional<User> findById(String id) {
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
public User save(User u) {
|
||||
return u;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package com.example.users;
|
||||
|
||||
public class User {
|
||||
private final String id;
|
||||
private final String email;
|
||||
|
||||
public User(String id, String email) {
|
||||
this.id = id;
|
||||
this.email = email;
|
||||
}
|
||||
|
||||
public String getId() { return id; }
|
||||
public String getEmail() { return email; }
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
package com.example.users;
|
||||
|
||||
import java.util.Optional;
|
||||
|
||||
public interface UserRepository {
|
||||
Optional<User> findById(String id);
|
||||
User save(User u);
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package com.example.users;
|
||||
|
||||
import java.util.Optional;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
// Mixed injection styles: field injection for the primary repository
|
||||
// and constructor injection for the qualified in-memory one. Both are
|
||||
// common in production Spring code.
|
||||
@Service
|
||||
public class UserService {
|
||||
@Autowired
|
||||
private UserRepository primary;
|
||||
|
||||
private final InMemoryUserRepository cache;
|
||||
|
||||
@Autowired
|
||||
public UserService(@Qualifier("inMemoryUsers") InMemoryUserRepository cache) {
|
||||
this.cache = cache;
|
||||
}
|
||||
|
||||
public Optional<User> lookup(String id) {
|
||||
Optional<User> hit = cache.findById(id);
|
||||
if (hit.isPresent()) {
|
||||
return hit;
|
||||
}
|
||||
return primary.findById(id);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"name": "gortex/di-symfony-fixture",
|
||||
"type": "project",
|
||||
"description": "Minimal Symfony fixture for Gortex DI recall — not installed.",
|
||||
"require": {
|
||||
"symfony/framework-bundle": "^6.3",
|
||||
"symfony/event-dispatcher": "^6.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
# Symfony attribute-based DI fixture
|
||||
#
|
||||
# Symfony's PHP-native DI shape is PHP 8 attributes: #[AsEventListener]
|
||||
# binds a method (or class) to an event, #[Route] maps a method to an
|
||||
# HTTP route (already handled by the HTTPExtractor), #[Autowire]
|
||||
# injects a specific service. This fixture focuses on event-listener
|
||||
# attributes — the clearest DI-as-attribute case.
|
||||
name: gortex-symfony-di
|
||||
cases:
|
||||
|
||||
# exact
|
||||
- id: exact-UserCreated
|
||||
tier: exact
|
||||
query: UserCreated
|
||||
expected: [src/Event/UserCreated.php::UserCreated]
|
||||
- id: exact-WelcomeEmailListener
|
||||
tier: exact
|
||||
query: WelcomeEmailListener
|
||||
expected: [src/EventListener/WelcomeEmailListener.php::WelcomeEmailListener]
|
||||
- id: exact-AuditListener
|
||||
tier: exact
|
||||
query: AuditListener
|
||||
expected: [src/EventListener/AuditListener.php::AuditListener]
|
||||
|
||||
# di_gap: #[AsEventListener] binds listener to event. Without
|
||||
# attribute extraction, find_usages(UserCreated) misses the
|
||||
# listener classes entirely — only the `new UserCreated(...)`
|
||||
# construction site in the controller shows up.
|
||||
- id: di_gap-usages-UserCreated
|
||||
tier: di_gap
|
||||
query: "usages:src/Event/UserCreated.php::UserCreated"
|
||||
expected:
|
||||
- src/EventListener/WelcomeEmailListener.php::WelcomeEmailListener.onCreated
|
||||
- src/EventListener/AuditListener.php::AuditListener
|
||||
|
||||
- id: di_gap-usages-UserDeleted
|
||||
tier: di_gap
|
||||
query: "usages:src/Event/UserDeleted.php::UserDeleted"
|
||||
expected:
|
||||
- src/EventListener/WelcomeEmailListener.php::WelcomeEmailListener.onDeleted
|
||||
@@ -0,0 +1,27 @@
|
||||
<?php
|
||||
|
||||
namespace App\Controller;
|
||||
|
||||
use App\Event\UserCreated;
|
||||
use Symfony\Component\EventDispatcher\EventDispatcherInterface;
|
||||
use Symfony\Component\HttpFoundation\Response;
|
||||
use Symfony\Component\Routing\Attribute\Route;
|
||||
|
||||
// Standard Symfony controller. Constructor-injected dispatcher +
|
||||
// route attributes. The event dispatch in register() will fire every
|
||||
// listener bound to UserCreated — the listener bindings live in the
|
||||
// #[AsEventListener] attributes in the EventListener directory.
|
||||
class UserController
|
||||
{
|
||||
public function __construct(
|
||||
private EventDispatcherInterface $events,
|
||||
) {}
|
||||
|
||||
#[Route('/users', methods: ['POST'])]
|
||||
public function register(): Response
|
||||
{
|
||||
$event = new UserCreated('usr_1');
|
||||
$this->events->dispatch($event);
|
||||
return new Response('ok');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
<?php
|
||||
|
||||
namespace App\Event;
|
||||
|
||||
// Domain event fired when a user is registered. Listeners are bound
|
||||
// via #[AsEventListener(event: UserCreated::class)] on their methods;
|
||||
// Symfony's dispatcher routes the event instance to each listener.
|
||||
class UserCreated
|
||||
{
|
||||
public function __construct(public readonly string $userId) {}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
<?php
|
||||
|
||||
namespace App\Event;
|
||||
|
||||
class UserDeleted
|
||||
{
|
||||
public function __construct(public readonly string $userId) {}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
<?php
|
||||
|
||||
namespace App\EventListener;
|
||||
|
||||
use App\Event\UserCreated;
|
||||
use Symfony\Component\EventDispatcher\Attribute\AsEventListener;
|
||||
|
||||
// Class-level attribute shorthand. Symfony scans the class for a
|
||||
// method whose signature matches the event type; here that's
|
||||
// __invoke taking UserCreated. This is a shorter spelling of the
|
||||
// method-level form above.
|
||||
#[AsEventListener(event: UserCreated::class)]
|
||||
class AuditListener
|
||||
{
|
||||
public function __invoke(UserCreated $event): void
|
||||
{
|
||||
$_ = $event->userId;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
<?php
|
||||
|
||||
namespace App\EventListener;
|
||||
|
||||
use App\Event\UserCreated;
|
||||
use App\Event\UserDeleted;
|
||||
use Symfony\Component\EventDispatcher\Attribute\AsEventListener;
|
||||
|
||||
// Method-level attributes. Each listener method declares exactly
|
||||
// which event it handles. Symfony dispatches the right method when
|
||||
// the event is fired — no explicit call site from anywhere, the
|
||||
// attribute IS the binding.
|
||||
class WelcomeEmailListener
|
||||
{
|
||||
#[AsEventListener(event: UserCreated::class, priority: 10)]
|
||||
public function onCreated(UserCreated $event): void
|
||||
{
|
||||
// Normally sends an email; body is a stub for this fixture.
|
||||
$_ = $event->userId;
|
||||
}
|
||||
|
||||
#[AsEventListener(event: UserDeleted::class)]
|
||||
public function onDeleted(UserDeleted $event): void
|
||||
{
|
||||
$_ = $event->userId;
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user