wehub-resource-sync
26382a7ac6
CI / Clippy (push) Failing after 15m13s
CI / Test (ubuntu-latest) (push) Failing after 16m1s
CI / Test (macos-latest) (push) Has been cancelled
CI / Test (windows-latest) (push) Has been cancelled
CI / Build (no embeddings / no ORT) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Cookbook (Node) (push) Has been cancelled
CI / Pi Extension (Node) (push) Has been cancelled
CI / Rust SDK (lean-ctx-client) (push) Has been cancelled
CI / Embed SDK (lean-ctx-sdk) (push) Has been cancelled
CI / Python SDK (leanctx) (push) Has been cancelled
CI / Hermes Plugin (Python) (push) Has been cancelled
CI / SDK Conformance Matrix (push) Has been cancelled
CI / Coverage (push) Has been cancelled
CI / cargo-deny (push) Has been cancelled
CI / Adversarial Safety (push) Has been cancelled
CI / Benchmarks (push) Has been cancelled
CI / Output-Quality Gate (eval A/B) (push) Has been cancelled
CI / Documentation (push) Has been cancelled
CI / CI Green (push) Has been cancelled
JetBrains Plugin / Actionlint (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (rust) (push) Has been cancelled
JetBrains Plugin / Validation (push) Has been cancelled
JetBrains Plugin / Build (push) Has been cancelled
JetBrains Plugin / Test (push) Has been cancelled
Security Check / Security Scan (push) Has been cancelled
15 lines
1.5 KiB
Plaintext
15 lines
1.5 KiB
Plaintext
# lean-ctx footprint ablation suite (#959).
|
|
#
|
|
# Unlike the accuracy suite, the "context" under test is lean-ctx's OWN injected
|
|
# footprint (rules block, tool schemas, wakeup briefing), not a corpus. Tasks are
|
|
# chosen to be footprint-SENSITIVE so removing an element can move the pass-rate:
|
|
# - route-* probe whether the advertised tool schemas enable correct tool choice,
|
|
# - rule-* probes whether the rules block changes behaviour,
|
|
# - control-* is footprint-INSENSITIVE (pure reasoning) and must stay NonInferior,
|
|
# proving the harness does not spuriously credit the footprint.
|
|
{"id":"route-callers","domain":"qa","prompt":"A user asks for every caller of a function. Which single lean-ctx tool should you call first? Answer with just the tool name.","workspace":"footprint/ws","answers":["ctx_callgraph"]}
|
|
{"id":"route-symbol","domain":"qa","prompt":"Which lean-ctx tool finds the exact definition of a named symbol? Answer with just the tool name.","workspace":"footprint/ws","answers":["ctx_symbol"]}
|
|
{"id":"route-read-large","domain":"qa","prompt":"To read a large source file with the fewest tokens, which lean-ctx tool do you use? Answer with just the tool name.","workspace":"footprint/ws","answers":["ctx_read"]}
|
|
{"id":"rule-before-edit","domain":"qa","prompt":"Per the lean-ctx rules, what must you do to a file before editing it?","workspace":"footprint/ws","answers":["read"]}
|
|
{"id":"control-arithmetic","domain":"qa","prompt":"What is 17 plus 25? Answer with the number only.","workspace":"footprint/ws","answers":["42"]}
|