wehub-resource-sync
26382a7ac6
CI / Clippy (push) Failing after 15m13s
CI / Test (ubuntu-latest) (push) Failing after 16m1s
CI / Test (macos-latest) (push) Has been cancelled
CI / Test (windows-latest) (push) Has been cancelled
CI / Build (no embeddings / no ORT) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Cookbook (Node) (push) Has been cancelled
CI / Pi Extension (Node) (push) Has been cancelled
CI / Rust SDK (lean-ctx-client) (push) Has been cancelled
CI / Embed SDK (lean-ctx-sdk) (push) Has been cancelled
CI / Python SDK (leanctx) (push) Has been cancelled
CI / Hermes Plugin (Python) (push) Has been cancelled
CI / SDK Conformance Matrix (push) Has been cancelled
CI / Coverage (push) Has been cancelled
CI / cargo-deny (push) Has been cancelled
CI / Adversarial Safety (push) Has been cancelled
CI / Benchmarks (push) Has been cancelled
CI / Output-Quality Gate (eval A/B) (push) Has been cancelled
CI / Documentation (push) Has been cancelled
CI / CI Green (push) Has been cancelled
JetBrains Plugin / Actionlint (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (rust) (push) Has been cancelled
JetBrains Plugin / Validation (push) Has been cancelled
JetBrains Plugin / Build (push) Has been cancelled
JetBrains Plugin / Test (push) Has been cancelled
Security Check / Security Scan (push) Has been cancelled
36 lines
3.8 KiB
Plaintext
36 lines
3.8 KiB
Plaintext
# lean-ctx curated retrieval eval suite (#686)
|
|
#
|
|
# Hand-authored, hand-verified labels for THIS repository. Each line is one
|
|
# `EvalQuery`: a natural-language, agent-style question (deliberately NOT just a
|
|
# symbol name, so a hit cannot come from trivial verbatim token overlap) mapped
|
|
# to the canonical source file that answers it. Used by:
|
|
#
|
|
# lean-ctx benchmark eval-ab --suite rust/eval/search-suite.ndjson
|
|
#
|
|
# to compare the dense default against the lean (BM25) lower bound. No generated
|
|
# or mock data — every target was opened and confirmed.
|
|
{"query":"combine two ranked search result lists with reciprocal rank fusion","expected_files":["core/hybrid_search.rs"],"category":"search"}
|
|
{"query":"build the BM25 inverted index from source files and score documents","expected_files":["core/bm25_index/mod.rs"],"category":"search"}
|
|
{"query":"split a source file into symbol chunks for indexing","expected_files":["core/bm25_index/chunking.rs"],"category":"search"}
|
|
{"query":"rerank candidate search results before returning them","expected_files":["core/search_reranking.rs"],"category":"search"}
|
|
{"query":"sparse lexical expansion retrieval with learned term weights","expected_files":["core/splade_retrieval.rs"],"category":"search"}
|
|
{"query":"approximate nearest neighbor graph for vector similarity","expected_files":["core/hnsw.rs"],"category":"search"}
|
|
{"query":"spreading activation over the project graph to find related files","expected_files":["core/spreading_activation.rs"],"category":"search"}
|
|
{"query":"retrieval quality harness measuring recall and mean reciprocal rank","expected_files":["core/eval_harness.rs"],"category":"eval"}
|
|
{"query":"resolve the project root from the editor workspace folder paths","expected_files":["server/roots.rs"],"category":"server"}
|
|
{"query":"compress a file read to reduce the number of tokens","expected_files":["core/compressor.rs"],"category":"compression"}
|
|
{"query":"estimate request cost from the per-token model pricing table","expected_files":["core/gain/model_pricing.rs"],"category":"pricing"}
|
|
{"query":"record realized token savings into the ledger","expected_files":["core/savings_ledger/mod.rs"],"category":"savings"}
|
|
{"query":"sign an eval artifact and verify its signature offline","expected_files":["core/eval_ab/artifact.rs"],"category":"eval"}
|
|
{"query":"run an A/B output-quality comparison with a live model","expected_files":["core/eval_ab/mod.rs"],"category":"eval"}
|
|
{"query":"forward an incoming proxy request to the upstream provider","expected_files":["proxy/forward.rs"],"category":"proxy"}
|
|
{"query":"load and persist the configuration from a toml file","expected_files":["core/config/mod.rs"],"category":"config"}
|
|
{"query":"attribute token cost between cooperating agents","expected_files":["core/a2a/cost_attribution.rs"],"category":"a2a"}
|
|
{"query":"prove billed savings of a long-lived cache-aware session versus a cold one","expected_files":["core/scorecard/dual_arm.rs"],"category":"savings"}
|
|
{"query":"iterative bounded code exploration returning file and line citations","expected_files":["tools/ctx_explore.rs"],"category":"exploration"}
|
|
{"query":"where are the MCP tools registered into the server registry","expected_files":["server/registry.rs"],"category":"exploration"}
|
|
{"query":"which built-in tool profile decides minimal versus standard versus power","expected_files":["core/tool_profiles.rs"],"category":"exploration"}
|
|
{"query":"pre-warm the heavy search and graph indices a tool needs before dispatch","expected_files":["core/index_orchestrator.rs"],"category":"exploration"}
|
|
{"query":"pick a minimal non-redundant set of spans that covers the query under a token budget","expected_files":["core/context_packing.rs"],"category":"exploration"}
|
|
{"query":"parse a natural language task into keyword and file path hints","expected_files":["core/task_relevance.rs"],"category":"exploration"}
|