zzet--gortex/docs/semantic-search.md

Semantic search

Default-on. A baked GloVe-50d table (~3.8 MB embedded in the binary, top 20k tokens) gives every install hybrid BM25 + vector search out of the box — no flag, no model download, no native dependency. Reciprocal Rank Fusion blends the two channels, and the BM25↔vector balance is scored continuously from the query's shape (identifier density, separators, stopwords) rather than bucketed into a discrete class — so a half-identifier query lands between the symbol and natural-language blends instead of jumping a whole tier. After ranking, an optional pure-cosine refinement pass re-scores the top results with the exact embedding distance the rank-based fusion discards.

Configuration

Switch or tune providers in .gortex.yaml:

embedding:
  enabled: true                # default — pass `false` to disable
  provider: static             # static | local | api  (default: static)
  variant: ""                  # optional named local model (e.g. a Hugot variant); empty = the provider default
  api_url: http://localhost:11434
  api_model: nomic-embed-text
  chunk_threshold_lines: 60    # symbols longer than this get split
  chunk_window_lines: 40       # AST-aware window size
  api_concurrency: 4           # bounded worker pool for hosted providers

Selecting a different embedding model (variant, or GORTEX_EMBEDDINGS_VARIANT) that changes vector dimensionality re-embeds the graph on next index; the persisted index guards against a dimension mismatch.

Where the config lives — and what wins

The embedding: block can sit in more than one place; precedence, highest first:

1. --embeddings / --embeddings-url / --embeddings-model flags and the GORTEX_EMBEDDINGS* env vars — one-shot overrides. A URL forces the api provider; GORTEX_EMBEDDINGS=0/1 toggles the vector channel; GORTEX_EMBEDDINGS_VARIANT pins a local model.
2. Repo-local .gortex.yaml — the per-project embedding: block (loaded by viper, so GORTEX_EMBEDDING_* env keys also merge here).
3. Global ~/.gortex/config.yaml — a user-level embedding: block layered under the repo-local one: every field the repo leaves unset inherits the global value (the tri-state enabled: too), so one block can serve every repo. An unrecognised top-level key in this file is ignored with a startup warning (contains keys gortex does not recognize) rather than silently dropped — the usual cause of an embedding: block that "does nothing" is placing it under the wrong key here.

Provider	Quality	Offline	Native deps	Notes
static (default)	Good for identifier-shaped queries	Yes	None	Baked GloVe-50d table, CPU-only, zero setup
local (Hugot MiniLM-L6-v2)	Better for NL queries	After first run	None	Auto-downloads ~90 MB to ~/.gortex/models/
api (Ollama / OpenAI)	Best	No	None	Bounded concurrent worker pool — tune via api_concurrency

AST sub-chunking

Symbols longer than chunk_threshold_lines are split into AST-aware windows (block statements, case clauses, field groups) before embedding; each window is vectorised independently and de-duplicated back to the parent symbol at query time, so a large function lands as one hit grounded in the specific chunk that matched — chunk IDs never leak into results.

Input truncation

Every embedding input is capped at the model's positional budget — max_position_embeddings from the model's config.json minus the two special-token slots (510 for MiniLM/BERT; larger for wide-context variants) — before it reaches inference. A transformer cannot attend past that window, so trimming the tail is lossless by construction, and it is load-bearing: the pure-Go tokenizer path does not enforce the limit itself, so a single over-budget input would otherwise reach inference at full length and abort the entire vector-index build with a tensor shape mismatch — dropping the daemon to text-only search. AST chunking already splits long symbols into sub-budget windows, so truncation only ever trims a pathological single chunk. If the model directory lacks a readable config.json the budget falls back to 510; if the tokenizer itself can't load, truncation degrades to a rune clamp rather than disabling the backend.

Persistent index

The vector index and the chunk → symbol map are persisted in the daemon snapshot; restarts re-warm in milliseconds without re-embedding the graph. Daemon snapshot schema is forward-compatible — older snapshots load with an empty vector layer and rebuild incrementally.

Vocabulary bridging without an LLM

A curated equivalence table (auth ↔ authentication ↔ login, delete ↔ remove ↔ destroy, …) plus per-repo auto-concept mining from symbol-name token co-occurrence expands queries deterministically — runs alongside (and dedup against) any LLM expansion. Toggle via search.equivalence_classes. A weighted concept-relatedness layer sits on top of the flat classes (e.g. auth pulls in token / session at lower priority) without merging the distinct concepts. When an LLM expander is configured, vocab_anchored: true constrains its invented terms to tokens that actually occur in the repo's symbol vocabulary.

HITS reranking

A hubs-and-authorities pass over the reference/call graph contributes a hits signal to the rerank pipeline — heavily-referenced symbols outrank shallow utility nodes, and the hub penalty (authority / (1 + hub)) demotes called-by-everything infra so it doesn't drown the result page.

Edge-provenance attenuation

Centrality (HITS + PageRank) and a dedicated rerank signal weight call/reference edges by how they were resolved: the abundant LSP-dispatch / framework-wiring tier — and the weak name-only tier — are attenuated relative to the structurally-unambiguous tier. Dense LSP enrichment otherwise inflates the apparent centrality of utility and framework code over genuine domain authorities. The weighting is a no-op on graphs with no resolution provenance recorded, so it never changes ranking where the data is absent.

Other rerank refinements

• Generated-file demotion — a generated file (*.pb.go, mock_*.go, *_pb2.py, …) is ranked below a real same-named hand-written implementation, but only when one exists.
• Source over test — when a query surfaces both an implementation and its test, the implementation is lifted above the test (only when both co-occur, so it never shifts the rest of the page).

Sparse sub-word tokenization (opt-in)

An optional tokenizer stage emits sub-word n-grams whose split points come from a per-repo boundary table learned from symbol names at index time, trading exact-identifier precision for recall on typo/fragment queries. Off by default (it is reindex-required and precision-sensitive); enable with GORTEX_SPARSE_NGRAM=1. Applies to the BM25 backend.

Keyword-soup defense

Boolean / OR-soup queries (A OR B OR 'no access' OR …) — and operator-free keyword lists (parse decode unmarshal token jwt cache) and comma-enumerations — defeat embedding retrieval. The query classifier detects all three, skips wasted LLM expansion, and splits the soup into terms fused via the existing BM25 expansion path; a query_advice nudge rides on the response. Genuine natural-language questions stay classified as concept. Tune via search.keyword_soup_rewrite: split | nudge | off.

Prose corpus

Markdown headings + section bodies become first-class searchable nodes (KindDoc) — search_symbols corpus: "docs" returns ranked README / ADR / design-doc sections; corpus: "all" mixes them with code hits. A docs query runs its own retrieval channel (a parallel doc-biased fetch, not merely a post-filter over the code fetch) and applies a prose weight profile that suppresses code-structural rerank signals (API/type-signature, definition-bias) which are meaningless for prose. Section node IDs are derived from the heading path, so incremental reindex of a touched markdown file produces stable IDs.

Per-keyword TaskMemory

The combo store now keys symbol associations both on the whole query and per keyword, so a new task with similar keywords inherits learned ranking from prior searches even when the exact phrasing differs. Exact-query matches still dominate; per-keyword evidence is the lower-confidence generalisation.

Build-tag backends

Opt-in faster local backends via build tags:

go build -tags embeddings_onnx ./cmd/gortex/          # needs: brew install onnxruntime
go build -tags "embeddings_gomlx XLA" ./cmd/gortex/   # needs libtokenizers.a on the linker path — use `make build-gomlx` (see below)

The embeddings_onnx backend (GTE-small) never auto-downloads: place model.onnx and vocab.txt in ~/.gortex/models/gte-small/ yourself and install the ONNX Runtime native library (brew install onnxruntime, or the distro equivalent). Without both, the backend reports "ONNX model not found" and the local chain falls through to the pure-Go Hugot backend.

The GoMLX/XLA backend requires both tags — embeddings_gomlx alone links a disabled XLA stub and always falls through to the pure-Go backend; the XLA tag is what compiles the real XLA session. It also statically links the rust tokenizer, so the build needs libtokenizers.a on the linker path (a prebuilt archive from daulet/tokenizers releases, at /usr/lib or /usr/local/lib); make build-gomlx downloads it for you. At runtime the XLA/PJRT plugin auto-downloads (~100 MB). XLA/PJRT runtime viability is platform-dependent and still experimental — if the plugin fails to load, the local chain degrades to the pure-Go Hugot backend and the startup log names the failed backend (see Troubleshooting). The default pure-Go backend needs no tags and no native libraries, and is the reliable path.

Build tag	Backend	Model	Extra dependency	Status
(none)	Hugot pure-Go	MiniLM-L6-v2 (auto-download)	none	default — reliable path
embeddings_gomlx XLA	Hugot + XLA/GoMLX	MiniLM-L6-v2 (auto-download)	libtokenizers.a (build) + PJRT plugin (runtime download)	experimental — XLA/PJRT runtime is platform-dependent
embeddings_onnx	ONNX Runtime	GTE-small (manual placement)	libonnxruntime + hand-placed model	manual setup — never auto-downloads

The legacy --embeddings / --embeddings-url / --embeddings-model CLI flags and the GORTEX_EMBEDDINGS* env vars still take precedence over the config block — useful for one-shot overrides without editing .gortex.yaml.

Troubleshooting

Semantic search degrading to text-only (BM25 / FTS5) is always logged — match the daemon log line to the cause:

• embeddings enabled ... provider: local (hugot/fp32), dim: 384 — working as intended: the transformer backend is active at its true width.
• embeddings enabled ... provider: local → static fallback, dim: 50 — a local config could not construct any transformer backend and fell back to static GloVe. The preceding embedding backend unavailable — degraded to static fallback warnings name each backend and why (an uncached model with downloads disabled, a missing embeddings_onnx model, …). Fix the named backend or accept static; the width and provider name now tell the truth rather than echoing the configured name.
• vector index aborted on chunk failure — an embedding call failed (API timeout / auth, or an over-long input on a build without truncation) and the whole vector index was dropped to avoid a half-embedded, mis-scoring index. Text search stays live. gortex eval embedders reports the concrete cause as vector build failed: … instead of a bare "no vector data".
• vector index built ... dropped: N (N > 0) — N malformed vectors (nil / wrong width) were skipped; the sample_ids warning names the first few. A non-zero dropped from a healthy provider is worth investigating.
• vector index disabled — embedding text count exceeds threshold — the corpus is larger than embedding.max_symbols; raise it if you have the memory headroom.
• ~/.gortex/config.yaml contains keys gortex does not recognize — a top-level key (commonly an embedding: block nested one level too deep, or a typo) is being ignored; move it to a recognised key.

search_symbols assist: modes

• auto (default) — skips LLM for identifier queries, expands NL queries
• on — forces expansion + rerank
• off — pure BM25
• deep — adds a body-grounded verification pass; +1.5–4 s; quality is highly model-dependent — unreliable on 3B local models, fine on 7B+ or hosted

See llm.md for provider configuration.