wehub-resource-sync
26382a7ac6
CI / Clippy (push) Failing after 15m13s
CI / Test (ubuntu-latest) (push) Failing after 16m1s
CI / Test (macos-latest) (push) Has been cancelled
CI / Test (windows-latest) (push) Has been cancelled
CI / Build (no embeddings / no ORT) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Cookbook (Node) (push) Has been cancelled
CI / Pi Extension (Node) (push) Has been cancelled
CI / Rust SDK (lean-ctx-client) (push) Has been cancelled
CI / Embed SDK (lean-ctx-sdk) (push) Has been cancelled
CI / Python SDK (leanctx) (push) Has been cancelled
CI / Hermes Plugin (Python) (push) Has been cancelled
CI / SDK Conformance Matrix (push) Has been cancelled
CI / Coverage (push) Has been cancelled
CI / cargo-deny (push) Has been cancelled
CI / Adversarial Safety (push) Has been cancelled
CI / Benchmarks (push) Has been cancelled
CI / Output-Quality Gate (eval A/B) (push) Has been cancelled
CI / Documentation (push) Has been cancelled
CI / CI Green (push) Has been cancelled
JetBrains Plugin / Actionlint (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (rust) (push) Has been cancelled
JetBrains Plugin / Validation (push) Has been cancelled
JetBrains Plugin / Build (push) Has been cancelled
JetBrains Plugin / Test (push) Has been cancelled
Security Check / Security Scan (push) Has been cancelled
1.8 KiB
1.8 KiB
Tokenizer-aware Translation Driver v1 (TokenizerTranslationDriverV1)
GitLab: #2310
LeanCTX nutzt “Translation” als letzten Optimierungsschritt für synthetische Context-Formate (z.B. TDD Signatures), damit Output-Tokens modell-/tokenizer-spezifisch reduziert werden können.
Ziele
- Deterministisch: gleicher Input + gleicher
model_key+ gleiche Profile Policy ⇒ gleiches Translation Ruleset. - Safe defaults: Default verändert bestehende CRP/TDD Formate nicht (opt-in).
- Tokenizer-aware: Rulesets können Unicode → ASCII tauschen, wenn Unicode im Ziel-Tokenizer teurer ist.
- Verifier-safe: Translation darf keine File Paths oder Identifiers “kaputtoptimieren”.
- Bounded: Translation läuft auf Tool-Outputs, aber wird für JSON-Ausgaben übersprungen (machine-readable bleibt exakt).
Aktivierung (Policy)
Translation wird über Profile gesteuert:
profile.translation.enabled = true|falseprofile.translation.ruleset = "legacy" | "ascii" | "auto"
Default: enabled=false, ruleset="legacy".
Deterministische Ruleset Selection
ruleset="auto" verwendet den Model Key aus:
LEAN_CTX_MODEL(oderLCTX_MODEL)
Heuristik (v1):
- OpenAI/GPT-family (
modelenthältgpt) →ascii - sonst →
legacy
Überspringen von JSON Outputs
Tool-Outputs, die als JSON parsebar sind ({...} oder [...]), werden nicht verändert.
Messung / Bench
ctx_benchmark zeigt token_cost (o200k_base) vor/nach Ruleset-Translation für TDD/Signature Outputs.
Relevanter Code
- Driver:
rust/src/core/tokenizer_translation_driver.rs - Token cost oracle:
rust/src/core/tokens.rs - Empirische safe rules:
rust/src/core/neural/token_optimizer.rs - Verification safety:
rust/src/core/output_verification.rs