chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,644 @@
|
||||
---
|
||||
title: "OmniRoute Auto-Combo Engine"
|
||||
version: 3.8.40
|
||||
lastUpdated: 2026-06-28
|
||||
---
|
||||
|
||||
# OmniRoute Auto-Combo Engine
|
||||
|
||||
> **For Users**: Looking for a quick start? See the [Auto-Combo User Guide](../getting-started/AUTO-COMBO-GUIDE.md) for simple explanations and examples.
|
||||
|
||||
> Self-managing model chains with adaptive scoring + zero-config auto-routing
|
||||
|
||||
## Zero-Config Auto-Routing (`auto/` prefix)
|
||||
|
||||
> **NEW:** No combo creation required. Use `auto/` prefix directly in any client.
|
||||
|
||||
### Quick Examples
|
||||
|
||||
| Model ID | Variant | Behavior |
|
||||
| -------------- | ------- | ------------------------------------------------------------------------ |
|
||||
| `auto` | default | All connected providers, LKGP strategy, balanced weights |
|
||||
| `auto/coding` | coding | Quality-first weights, suitable for code generation |
|
||||
| `auto/fast` | fast | Low-latency weighted selection |
|
||||
| `auto/cheap` | cheap | Cost-optimized routing (lowest cost first) |
|
||||
| `auto/offline` | offline | Favors providers with highest quota availability |
|
||||
| `auto/smart` | smart | Quality-first + higher exploration rate (10%) for better model discovery |
|
||||
| `auto/lkgp` | lkgp | Explicit LKGP (same as default `auto`) |
|
||||
|
||||
### Category × Tier Composition (`auto/<category>:<tier>`)
|
||||
|
||||
OpenRouter-style suffixes separate **what kind of route** (category) from **how to optimize it** (tier), so you can compose them freely (#4235 Phase B, `open-sse/services/autoCombo/suffixComposition.ts`):
|
||||
|
||||
- **Categories** (filter the candidate pool by capability): `coding` · `reasoning` · `vision` · `chat` · `multimodal`. `vision`/`multimodal` keep vision-capable models; `reasoning` keeps reasoning/thinking models.
|
||||
- **Tiers** (pick the scoring weights / pool filter): `fast` (ship-fast) · `cheap` (alias `floor`, cost-saver) · `reliable` (circuit-breaker health + latency stability) · `free` / `pro` (filter the pool by model tier via `classifyTier` — free-tier vs. premium).
|
||||
|
||||
| Example | Resolves to |
|
||||
| ---------------------- | ------------------------------------------------------- |
|
||||
| `auto/coding:fast` | coding pool, low-latency weights |
|
||||
| `auto/coding:cheap` | coding pool, cost-optimized (alias `auto/coding:floor`) |
|
||||
| `auto/reasoning:pro` | reasoning/thinking models only, premium tier |
|
||||
| `auto/vision` | vision-capable models (no tier → balanced weights) |
|
||||
| `auto/multimodal:free` | multimodal-capable models, free tier only |
|
||||
|
||||
Any valid `auto/<category>[:<tier>]` resolves on demand; a curated subset is advertised in `/v1/models` and the dashboard (`AUTO_SUFFIX_VARIANTS` in `open-sse/services/autoCombo/builtinCatalog.ts`). Filtering is **fail-open** — if a constraint matches no connected models, the full pool is used so routing never breaks. The core scorer (`combo.ts`) is unchanged; the category/tier filter is applied in `buildAutoCandidates`.
|
||||
|
||||
> **Live model intelligence:** auto-routing fitness is informed by live **Arena ELO** rankings + **models.dev** tier data when the `ARENA_ELO_SYNC_ENABLED` flag is on (falls back to the static fitness map otherwise).
|
||||
|
||||
**How to use:**
|
||||
|
||||
```bash
|
||||
# Any IDE or CLI tool that supports OpenAI format
|
||||
Base URL: http://localhost:20128/v1
|
||||
API Key: <your-endpoint-key>
|
||||
|
||||
# In your code/config, set model to:
|
||||
model: "auto" # balanced default
|
||||
model: "auto/coding" # best for coding tasks
|
||||
model: "auto/fast" # fastest available
|
||||
model: "auto/cheap" # cheapest per token
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
|
||||
1. OmniRoute detects `auto/` prefix in `src/sse/handlers/chat.ts`
|
||||
2. Queries all **active provider connections** from the database
|
||||
3. Filters to those with valid credentials (API key or OAuth token)
|
||||
4. Determines the model per connection (`connection.defaultModel` or provider's first model)
|
||||
5. Builds a **virtual combo** in-memory (not stored in DB)
|
||||
6. Routes using the selected variant's weight profile + LKGP strategy
|
||||
|
||||
**Key properties:**
|
||||
|
||||
- ✅ **Always-on:** No toggle, no combo creation, no configuration needed
|
||||
- ✅ **Dynamic:** Reflects current connected providers automatically
|
||||
- ✅ **Session stickiness:** LKGP ensures last successful provider is prioritized
|
||||
- ✅ **Multi-account aware:** Each provider connection becomes a separate candidate
|
||||
- ✅ **No DB writes:** Virtual combo exists only for the request, zero persistence overhead
|
||||
|
||||
**Behind the scenes:**
|
||||
|
||||
```txt
|
||||
Request: { model: "auto/coding" }
|
||||
↓
|
||||
src/sse/handlers/chat.ts detects prefix
|
||||
↓
|
||||
createVirtualAutoCombo('coding') → candidatePool from active connections
|
||||
↓
|
||||
handleComboChat (same engine as persisted combos)
|
||||
↓
|
||||
Auto-scoring selects best provider/model per request
|
||||
```
|
||||
|
||||
**Implementation files:**
|
||||
|
||||
| File | Purpose |
|
||||
| --------------------------------------------------------- | ----------------------------------------- |
|
||||
| `open-sse/services/autoCombo/autoPrefix.ts` | Prefix parser (`parseAutoPrefix`) |
|
||||
| `open-sse/services/autoCombo/virtualFactory.ts` | Creates virtual `AutoComboConfig` objects |
|
||||
| `open-sse/services/autoCombo/providerRegistryAccessor.ts` | Test hook for mocking provider registry |
|
||||
| `src/sse/handlers/chat.ts` | Integration: auto prefix short-circuit |
|
||||
| `src/shared/constants/providers.ts` | `SYSTEM_PROVIDERS.auto` system entry |
|
||||
|
||||
## How It Works (Persisted Auto-Combos)
|
||||
|
||||
The Auto-Combo Engine dynamically selects the best provider/model for each request using a **12-factor scoring function** (defined in `open-sse/services/autoCombo/scoring.ts` → `DEFAULT_WEIGHTS`). All weights sum to **1.0**.
|
||||
|
||||
|
||||
|
||||
> Source: [diagrams/auto-combo-12factor.mmd](../diagrams/auto-combo-12factor.mmd) (regenerate via `npm run docs:render-diagrams`).
|
||||
|
||||
| Factor | Default Weight | Description |
|
||||
| :-------------------- | :------------- | :------------------------------------------------------------------------------------------------- |
|
||||
| `health` | 0.20 | Health score from circuit breaker (CLOSED=1.0, HALF_OPEN=0.5, OPEN=0.0) |
|
||||
| `quota` | 0.15 | Remaining quota / rate-limit headroom [0..1] |
|
||||
| `costInv` | 0.15 | Inverse **blended** cost (60% input + 40% output token price, normalized) — cheaper = higher score |
|
||||
| `latencyInv` | 0.12 | Inverse p95 latency normalized to pool — faster = higher score |
|
||||
| `taskFit` | 0.08 | Task-type fitness (coding, review, planning, analysis, debugging, docs) |
|
||||
| `stability` | 0.05 | Variance-based stability (low latency stdDev / error rate) |
|
||||
| `tierPriority` | 0.05 | Account-tier priority — Ultra=1.0, Pro=0.67, Standard=0.33, Free=0.0 |
|
||||
| `tierAffinity` | 0.05 | Affinity between the candidate's tier and the manifest-recommended tier |
|
||||
| `specificityMatch` | 0.05 | Match between request specificity (manifest hint) and model tier |
|
||||
| `contextAffinity` | 0.05 | Affinity between the request's context-window need and the model's context window |
|
||||
| `connectionDensity` | 0.05 | Spreads load across connections of the same provider (anti-concentration) |
|
||||
| `resetWindowAffinity` | 0.00 | Bias toward connections whose quota reset window is favorable (disabled by default) |
|
||||
|
||||
**Sum:** `0.20 + 0.15 + 0.15 + 0.12 + 0.08 + 0.05 + 0.05 + 0.05 + 0.05 + 0.05 + 0.05 + 0.00 = 1.0` (validated by `validateWeights()`).
|
||||
|
||||
## Mode Packs
|
||||
|
||||
Four pre-defined weight profiles in `open-sse/services/autoCombo/modePacks.ts`. Each pack overrides the default weights to bias selection toward a specific goal. Below are the **full weight tables per pack** (each row sums to 1.0).
|
||||
|
||||
| Factor | ship-fast | cost-saver | quality-first | offline-friendly |
|
||||
| :----------- | :-------- | :--------- | :------------ | :--------------- |
|
||||
| quota | 0.14 | 0.14 | 0.10 | **0.37** |
|
||||
| health | 0.28 | 0.19 | 0.18 | 0.28 |
|
||||
| costInv | 0.05 | **0.37** | 0.05 | 0.10 |
|
||||
| latencyInv | **0.32** | 0.05 | 0.05 | 0.05 |
|
||||
| taskFit | 0.10 | 0.10 | **0.37** | 0.00 |
|
||||
| stability | 0.00 | 0.05 | 0.15 | 0.10 |
|
||||
| tierPriority | 0.05 | 0.05 | 0.05 | 0.05 |
|
||||
|
||||
Notes:
|
||||
|
||||
- `tierAffinity` and `specificityMatch` are not set in mode packs — `calculateScore()` treats them as `?? 0` when absent.
|
||||
- Each pack's emphasis at a glance:
|
||||
- **ship-fast** → latencyInv 0.32 + health 0.28 (low-latency, healthy connections)
|
||||
- **cost-saver** → costInv 0.37 (cheapest tokens win)
|
||||
- **quality-first** → taskFit 0.37 + stability 0.15 (best model for the task, consistent)
|
||||
- **offline-friendly** → quota 0.37 + health 0.28 (max headroom regardless of speed/cost)
|
||||
|
||||
### Per-Request Controls (headers) — #6023 / #6024 / #6025
|
||||
|
||||
An `auto` combo can be steered **per request** via two headers, without mutating the
|
||||
combo's stored config. These apply only to the `auto` strategy and only for the request
|
||||
that carries them; the combo's saved `modePack`/`budgetCap` are used when the header is
|
||||
absent.
|
||||
|
||||
| Header | Accepts | Effect |
|
||||
| :------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `X-OmniRoute-Mode` | a preset alias (`fast`, `balanced`, `quality`, `cheap`, `reliable`, `offline`) or a raw pack name (`ship-fast`, `cost-saver`, `quality-first`, `offline-friendly`, `reliability-first`) | Overrides the scoring weights for this request. `balanced`/`default` force the default weights (no pack). Unknown values are ignored (config preserved). |
|
||||
| `X-OmniRoute-Budget` | a positive number (max USD per request) | Hard cost ceiling: candidates whose estimated cost exceeds it are filtered before selection, falling back to the cheapest healthy candidate if all exceed. Non-positive/garbage values are ignored. |
|
||||
|
||||
```bash
|
||||
# Force the fastest profile and cap this request at $0.05
|
||||
curl -sS http://localhost:20128/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-OmniRoute-Mode: fast" \
|
||||
-H "X-OmniRoute-Budget: 0.05" \
|
||||
-d '{"model":"auto","messages":[{"role":"user","content":"hi"}]}'
|
||||
```
|
||||
|
||||
Resolution is a pure function (`open-sse/services/autoCombo/requestControls.ts`); the
|
||||
resolved values feed the engine's existing `config.modePack` / `config.budgetCap` inputs.
|
||||
|
||||
## All Routing Strategies
|
||||
|
||||
OmniRoute's combo engine supports **18 routing strategies** (declared in `src/shared/constants/routingStrategies.ts` → `ROUTING_STRATEGY_VALUES`). The Auto Combo engine itself is exposed under the `auto` strategy; the others are available for persisted combos.
|
||||
|
||||
| Strategy | Description |
|
||||
| :------------------ | :--------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `priority` | First-target ordered list with explicit priority |
|
||||
| `weighted` | Weighted random by per-target weight |
|
||||
| `round-robin` | Cycle through targets in order |
|
||||
| `context-relay` | Hand off context across targets (long conversations) |
|
||||
| `fill-first` | Fill each target's quota before moving to next |
|
||||
| `p2c` | Power-of-2-choices random load balancing |
|
||||
| `random` | Uniform random selection |
|
||||
| `least-used` | Pick target with lowest current load |
|
||||
| `cost-optimized` | Minimize $ per request given catalog pricing |
|
||||
| `reset-aware` ⭐ | Prioritize by quota reset time — short reset windows ranked higher |
|
||||
| `reset-window` | Prefer targets whose quota window resets soonest |
|
||||
| `headroom` | Pick the target with the most remaining quota headroom |
|
||||
| `strict-random` | Random without deduplication of repeats |
|
||||
| `auto` | Use Auto Combo scoring (9-factor) — **recommended** |
|
||||
| `lkgp` | Last-Known-Good Path (sticky route to last successful target) |
|
||||
| `context-optimized` | Pick target with best fit for current context size |
|
||||
| `fusion` 🧬 | Fan out to a panel of models in parallel, then synthesize one answer via a judge (see below) |
|
||||
| `pipeline` | Run targets sequentially, threading each step's output into the next step's input; only the final answer is returned (#6396) |
|
||||
|
||||
⭐ = New in v3.8.0 · 🧬 = New in v3.8.36
|
||||
|
||||
## Fusion Strategy
|
||||
|
||||
`fusion` is the one strategy that does **not** pick a single target. It fans the prompt
|
||||
out to **every panel model in parallel**, then a configurable **judge model** synthesizes
|
||||
a single final answer from all panel responses. Ported from upstream `decolua/9router`
|
||||
(OpenRouter's Fusion design); implementation in `open-sse/services/fusion.ts`.
|
||||
|
||||
How it works:
|
||||
|
||||
1. **Fan-out** — the prompt is sent to every panel model at once, forced non-streaming
|
||||
with tools stripped (the judge needs complete prose to synthesize).
|
||||
2. **Quorum-grace collection** — as soon as `minPanel` answers arrive, a short grace
|
||||
timer starts for the stragglers, then fusion proceeds with whatever was collected.
|
||||
This caps the slowest model's penalty on wall time, bounded by a hard timeout.
|
||||
3. **Judge synthesis** — panel answers are anonymized (`Source 1`, `Source 2`, … — so
|
||||
the judge weighs substance, not model brand) and handed to the judge, which analyzes
|
||||
consensus / contradictions / partial coverage / unique insights / blind spots, then
|
||||
writes **one** authoritative answer. The judge call keeps the client's original
|
||||
`stream` flag + tools, so streaming and downstream tool use still work.
|
||||
4. **Graceful degradation** — 0 panel answers → `503`; exactly 1 survivor → that answer
|
||||
is returned directly (nothing to fuse); a single-model panel answers directly.
|
||||
|
||||
### Configuration
|
||||
|
||||
Configured on the combo's `config` blob (no schema migration — it reuses the existing
|
||||
`combos` table):
|
||||
|
||||
| Field | Type | Default | Purpose |
|
||||
| :--------------------------------------- | :------- | :---------------- | :-------------------------------------------------------------------------------------- |
|
||||
| `config.judgeModel` | `string` | first panel model | Model that synthesizes the final answer |
|
||||
| `config.fusionTuning.minPanel` | `number` | `2` | Successful answers required before the grace timer starts (clamped to `[2, panelSize]`) |
|
||||
| `config.fusionTuning.stragglerGraceMs` | `number` | `8000` | How long to wait for laggards once quorum is reached |
|
||||
| `config.fusionTuning.panelHardTimeoutMs` | `number` | `90000` | Absolute cap so one hung model can't stall the request |
|
||||
|
||||
Defaults live in `FUSION_DEFAULTS` (`open-sse/services/fusion.ts`).
|
||||
|
||||
### Example
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:20128/api/combos \
|
||||
-H "Authorization: Bearer <key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "fusion-panel",
|
||||
"strategy": "fusion",
|
||||
"targets": [
|
||||
{ "model": "cc/claude-opus-4-7" },
|
||||
{ "model": "cx/gpt-5.5" },
|
||||
{ "model": "glm/glm-5.1" }
|
||||
],
|
||||
"config": {
|
||||
"judgeModel": "cc/claude-opus-4-7",
|
||||
"fusionTuning": { "minPanel": 2, "stragglerGraceMs": 8000, "panelHardTimeoutMs": 90000 }
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Then call it like any combo: `{"model":"fusion-panel","messages":[...]}`.
|
||||
|
||||
## Virtual Auto-Combo Factory
|
||||
|
||||
The Auto Combo engine doesn't require pre-defined combos. Instead, `open-sse/services/autoCombo/virtualFactory.ts` builds candidates on-the-fly:
|
||||
|
||||
1. Pulls `getProviderConnections({ isActive: true })` (all enabled connections)
|
||||
2. Filters to those with valid credentials (API key or non-expired OAuth token via `hasUsableOAuthToken()`)
|
||||
3. Cross-references with `getProviderRegistry()` for model availability + pricing
|
||||
4. For each tuple `(provider, model, connection)`, builds a `VirtualAutoComboCandidate`
|
||||
5. Picks `connection.defaultModel` (or the registry's first model) as the dispatch target
|
||||
6. Scores each candidate using the 9-factor `scorePool()` and the variant's weight pack
|
||||
7. Returns the resulting in-memory `AutoComboConfig` for `handleComboChat()` — never persisted to DB
|
||||
|
||||
This means **adding a new provider with `auto/*` enabled automatically expands the candidate pool** — no manual combo editing needed. The virtual combo is rebuilt per request, so newly-added or newly-healthy connections are picked up immediately.
|
||||
|
||||
## Self-Healing
|
||||
|
||||
- **Temporary exclusion**: Score < 0.2 → excluded for 5 min (progressive backoff, max 30 min)
|
||||
- **Circuit breaker awareness**: OPEN → auto-excluded; HALF_OPEN → probe requests
|
||||
- **Incident mode**: >50% OPEN → disable exploration, maximize stability
|
||||
- **Cooldown recovery**: After exclusion, first request is a "probe" with reduced timeout
|
||||
|
||||
## Bandit Exploration
|
||||
|
||||
5% of requests (configurable) are routed to random providers for exploration. Disabled in incident mode.
|
||||
|
||||
## API
|
||||
|
||||
There is **no dedicated `POST /api/combos/auto` endpoint** — Auto-Combo is consumed in two ways:
|
||||
|
||||
1. **Zero-config (recommended):** Send any chat completion request with `model: "auto"` or `model: "auto/<variant>"`. The virtual factory builds the combo per request — no persistence, no API calls needed.
|
||||
|
||||
2. **Persisted combo with `strategy: "auto"`:** Create a regular combo via `POST /api/combos` and set `strategy: "auto"` plus `config.auto.weights` / `config.auto.candidatePool`. The same scoring engine is used; the combo is stored in `combos` and reusable by ID.
|
||||
|
||||
For discovery, `GET /api/combos/auto` lists every variant with its resolved candidate pool plus `context_length` / `max_output_tokens` — the MAX across the candidate pool's windows. Clients (e.g. the opencode plugin) must advertise these values instead of `0`: a zero context disables opencode's auto-compaction entirely, letting sessions grow until the gateway's history purge destroys context. MAX is safe to advertise because the auto-combo context pre-filter routes oversized requests to large-window candidates.
|
||||
|
||||
```bash
|
||||
# Zero-config usage (no combo creation)
|
||||
curl -X POST http://localhost:20128/v1/chat/completions \
|
||||
-H "Authorization: Bearer <key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"auto/coding","messages":[{"role":"user","content":"Hello"}]}'
|
||||
|
||||
# Persisted auto combo via the regular combos endpoint
|
||||
curl -X POST http://localhost:20128/api/combos \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"id":"my-auto","name":"Auto Coder","strategy":"auto","config":{"auto":{"candidatePool":["anthropic","google","openai"],"weights":{"quota":0.15,"health":0.3,"costInv":0.05,"latencyInv":0.35,"taskFit":0.1,"stability":0,"tierPriority":0.05}}}}'
|
||||
```
|
||||
|
||||
### Auto router strategies
|
||||
|
||||
Persisted `strategy: "auto"` combos can set `config.routerStrategy` (or legacy
|
||||
`config.auto.routerStrategy`) to one of:
|
||||
|
||||
- `rules` — default weighted scoring
|
||||
- `cost` / `eco` — cheapest healthy provider
|
||||
- `latency` / `fast` — lowest p95 latency with reliability penalty
|
||||
- `sla-aware` / `sla` — prefer candidates that satisfy p95 latency, error-rate, and optional
|
||||
cost SLOs
|
||||
- `lkgp` — last known good provider first
|
||||
|
||||
### Router strategies in detail
|
||||
|
||||
The auto-combo engine exposes 5 pluggable **RouterStrategy** implementations that
|
||||
you can swap via `config.routerStrategy` (or the legacy `config.auto.routerStrategy`).
|
||||
Each strategy picks one provider from the candidate pool, given a `RoutingContext`
|
||||
(task type, tool/vision hints, token estimate, optional SLA policy, optional
|
||||
last-known-good provider).
|
||||
|
||||
#### 1. `rules` (default) — 6-factor weighted scoring
|
||||
|
||||
Wraps the existing scoring engine. Filters out `OPEN` circuit-breaker
|
||||
candidates, then runs `scorePool()` with the current task type and `getTaskFitness()`,
|
||||
picking the top-scoring provider.
|
||||
|
||||
```ts
|
||||
class RulesStrategyImpl implements RouterStrategy {
|
||||
readonly name = "rules";
|
||||
readonly description =
|
||||
"6-factor weighted scoring: quota, health, cost, latency, taskFit, stability";
|
||||
|
||||
select(pool, context) {
|
||||
const eligible = pool.filter((c) => c.circuitBreakerState !== "OPEN");
|
||||
const ranked = scorePool(
|
||||
eligible.length > 0 ? eligible : pool,
|
||||
context.taskType,
|
||||
undefined,
|
||||
getTaskFitness
|
||||
);
|
||||
return { provider: ranked[0].provider /* ... */ };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Default. Use when you want a balanced trade-off across all signals.
|
||||
|
||||
**Alias**: `rules` (no alias)
|
||||
|
||||
---
|
||||
|
||||
#### 2. `cost` / `eco` — cheapest healthy provider
|
||||
|
||||
Sorts the candidate pool by `costPer1MTokens` (ascending) and picks the cheapest.
|
||||
Filters out `OPEN` candidates first.
|
||||
|
||||
```ts
|
||||
class CostStrategyImpl implements RouterStrategy {
|
||||
readonly name = "cost";
|
||||
readonly description = "Always selects cheapest available provider";
|
||||
|
||||
select(pool, context) {
|
||||
const healthy = pool.filter((c) => c.circuitBreakerState !== "OPEN");
|
||||
const sorted = [...healthy].sort((a, b) => a.costPer1MTokens - b.costPer1MTokens);
|
||||
return { provider: sorted[0].provider /* ... */ };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Cost-sensitive workloads, batch processing, or background jobs.
|
||||
|
||||
**Aliases**: `cost`, `eco`
|
||||
|
||||
---
|
||||
|
||||
#### 3. `latency` / `fast` — lowest p95 latency with reliability penalty
|
||||
|
||||
Sorts by `p95LatencyMs + (errorRate * 1000)`. The error-rate penalty ensures
|
||||
unreliable providers are ranked lower even if their nominal latency is low.
|
||||
|
||||
```ts
|
||||
class LatencyStrategyImpl implements RouterStrategy {
|
||||
readonly name = "latency";
|
||||
readonly description = "Prioritizes lowest p95 latency with reliability weighting";
|
||||
|
||||
select(pool, context) {
|
||||
const healthy = pool.filter((c) => c.circuitBreakerState !== "OPEN");
|
||||
const sorted = [...healthy].sort(
|
||||
(a, b) => a.p95LatencyMs + a.errorRate * 1000 - (b.p95LatencyMs + b.errorRate * 1000)
|
||||
);
|
||||
return { provider: sorted[0].provider /* ... */ };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Latency-sensitive workloads like real-time chat, autocomplete, or
|
||||
interactive coding assistants.
|
||||
|
||||
**Aliases**: `latency`, `fast`
|
||||
|
||||
---
|
||||
|
||||
#### 4. `sla-aware` / `sla` — latency/error/cost SLO compliance
|
||||
|
||||
Scores each candidate by how well it satisfies the configured SLO policy:
|
||||
|
||||
| Factor | Weight | Formula |
|
||||
| --------------- | ------ | ------------------------------------------------- |
|
||||
| Latency score | 35% | `threshold / max(value, ε)` |
|
||||
| Error score | 35% | `threshold / max(value, ε)` |
|
||||
| Health score | 15% | `1.0` (CLOSED) / `0.5` (HALF_OPEN) / `0.0` (OPEN) |
|
||||
| Cost score | 10% | `threshold / max(value, ε)` or inverse normalized |
|
||||
| Stability score | 5% | inverse normalized latency stddev |
|
||||
|
||||
When `hardConstraints: true`, candidates are sorted primarily by **violation score**
|
||||
(how far they exceed any SLO), then by composite score. Otherwise it's just
|
||||
the composite score.
|
||||
|
||||
```ts
|
||||
class SLAStrategyImpl implements RouterStrategy {
|
||||
readonly name = "sla-aware";
|
||||
readonly description =
|
||||
"Selects the provider most likely to satisfy latency, error-rate, and cost SLOs";
|
||||
|
||||
select(pool, context) {
|
||||
// ... scores each candidate against policy: { targetP95Ms, maxErrorRate, maxCostPer1MTokens, hardConstraints }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**SLA fields** (set on the combo config):
|
||||
|
||||
```json
|
||||
{
|
||||
"strategy": "auto",
|
||||
"config": {
|
||||
"routerStrategy": "sla-aware",
|
||||
"slaTargetP95Ms": 1500,
|
||||
"slaMaxErrorRate": 0.05,
|
||||
"slaMaxCostPer1MTokens": 5,
|
||||
"slaHardConstraints": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Production workloads with strict latency, error-rate, or cost budgets.
|
||||
|
||||
**Aliases**: `sla-aware`, `sla`
|
||||
|
||||
---
|
||||
|
||||
#### 5. `lkgp` — last known good provider first
|
||||
|
||||
Tries the **last known good provider** (if set) first, then falls back to the
|
||||
`rules` strategy. Useful for session stickiness — the same provider handles
|
||||
follow-up requests in a conversation.
|
||||
|
||||
```ts
|
||||
class LKGPStrategyImpl implements RouterStrategy {
|
||||
readonly name = "lkgp";
|
||||
readonly description = "Tries last known good provider first, then falls back to rules";
|
||||
|
||||
select(pool, context) {
|
||||
if (context.lkgpEnabled === false) {
|
||||
return getStrategy("rules").select(pool, context);
|
||||
}
|
||||
|
||||
if (context.lastKnownGoodProvider) {
|
||||
const candidates = pool.filter(
|
||||
(c) => c.provider === context.lastKnownGoodProvider && c.circuitBreakerState !== "OPEN"
|
||||
);
|
||||
if (candidates.length > 0) {
|
||||
return { provider: candidates[0].provider /* ... */ };
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback to rules strategy
|
||||
return getStrategy("rules").select(pool, context);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Multi-turn conversations where you want the same provider to handle
|
||||
follow-up requests (e.g., for caching, context continuity, or pricing consistency).
|
||||
|
||||
**Alias**: `lkgp` (no alias)
|
||||
|
||||
---
|
||||
|
||||
### Custom router strategies
|
||||
|
||||
You can register your own `RouterStrategy` implementation via the public API:
|
||||
|
||||
```ts
|
||||
import {
|
||||
registerStrategy,
|
||||
type RouterStrategy,
|
||||
} from "@omniroute/open-sse/services/autoCombo/routerStrategy";
|
||||
|
||||
class MyCustomStrategy implements RouterStrategy {
|
||||
readonly name = "my-custom";
|
||||
readonly description = "My custom routing strategy";
|
||||
|
||||
select(pool, context) {
|
||||
// Your routing logic here
|
||||
return {
|
||||
provider: pool[0].provider,
|
||||
model: pool[0].model,
|
||||
strategy: this.name,
|
||||
reason: "MyCustomStrategy: ...",
|
||||
candidatesConsidered: pool.length,
|
||||
finalScore: 1.0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
registerStrategy("my-custom", new MyCustomStrategy());
|
||||
```
|
||||
|
||||
Then use it:
|
||||
|
||||
```json
|
||||
{
|
||||
"strategy": "auto",
|
||||
"config": {
|
||||
"routerStrategy": "my-custom"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Router strategy selection guide
|
||||
|
||||
| Use case | Strategy | Reason |
|
||||
| ----------------- | ----------- | ------------------------------------ |
|
||||
| Balanced workload | `rules` | Default — considers all factors |
|
||||
| Minimize cost | `cost` | Always picks cheapest |
|
||||
| Minimize latency | `latency` | Picks fastest reliable provider |
|
||||
| Strict SLOs | `sla-aware` | Filters by p95/error/cost thresholds |
|
||||
| Multi-turn chat | `lkgp` | Session stickiness |
|
||||
|
||||
SLA-aware fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"strategy": "auto",
|
||||
"config": {
|
||||
"routerStrategy": "sla-aware",
|
||||
"slaTargetP95Ms": 1500,
|
||||
"slaMaxErrorRate": 0.05,
|
||||
"slaMaxCostPer1MTokens": 5,
|
||||
"slaHardConstraints": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Task Fitness
|
||||
|
||||
30+ models scored across 6 task types (`coding`, `review`, `planning`, `analysis`, `debugging`, `documentation`). Supports wildcard patterns (e.g., `*-coder` → high coding score).
|
||||
|
||||
## Auto Variants Recap
|
||||
|
||||
Including the bare `auto` (default) plus the 6 `AutoVariant` values declared in `autoPrefix.ts`, there are **7 invokable model IDs**:
|
||||
|
||||
`auto`, `auto/coding`, `auto/fast`, `auto/cheap`, `auto/offline`, `auto/smart`, `auto/lkgp`
|
||||
|
||||
(`AutoVariant` itself enumerates 6 values; the 7th option is "no variant" — bare `auto` — handled by `parseAutoPrefix()` as `variant: undefined`.)
|
||||
|
||||
## How tiers fit Auto-Combo
|
||||
|
||||
The 12-factor scoring function (`open-sse/services/autoCombo/scoring.ts`) treats tier
|
||||
membership as two signals: `tierPriority` (0.05) and `tierAffinity` (0.05). See the
|
||||
canonical [scoring factor table](#how-it-works-persisted-auto-combos) above for the full
|
||||
`DEFAULT_WEIGHTS` set — the per-pack overrides (ship-fast/cost-saver/quality-first/
|
||||
offline-friendly) are listed in the "Weight profiles per pack" table.
|
||||
|
||||
Tier alone does **not** force Tier 1 first — if Tier 1 latency is bad or
|
||||
cost-vs-quality is suboptimal, Tier 2 wins. To force tier ordering, use combo
|
||||
strategy `priority` and arrange providers by tier.
|
||||
|
||||
To strongly favor Tier 1 (subscription), increase `tierPriority` weight:
|
||||
|
||||
```json
|
||||
{
|
||||
"strategy": "auto",
|
||||
"config": { "auto": { "weights": { "tierPriority": 0.3, "costInv": 0.05 } } }
|
||||
}
|
||||
```
|
||||
|
||||
See `docs/marketing/TIERS.md` for tier definitions and provider classification.
|
||||
|
||||
## Testing & Coverage
|
||||
|
||||
### Deterministic routing-decision matrix (`npm run test:combo:matrix`)
|
||||
|
||||
`tests/integration/combo-matrix/*.test.ts` proves the routing **decision** of all 17
|
||||
public strategies end-to-end through the real combo pipeline with a mocked upstream.
|
||||
Coverage includes:
|
||||
|
||||
- All 18 `ROUTING_STRATEGY_VALUES` strategies (ordered, weighted, cost, context, fusion, …).
|
||||
- `quota-share` (internal) end-to-end: DRR fairness + saturation deprioritization via the
|
||||
real `selectQuotaShareTarget` seam (`registerQuotaFetcher` / `setLKGP` /
|
||||
`__setHeadroomSaturationFetcherForTests`).
|
||||
- `context-relay` universal-handoff coverage across every target count.
|
||||
|
||||
This suite runs in CI (`test:integration` job) with `--test-concurrency=1` and
|
||||
`--test-force-exit` so it is deterministic and does not require live credentials.
|
||||
|
||||
### Gated live smoke (NOT in CI — real providers)
|
||||
|
||||
| Command | What it does |
|
||||
| :------------------------------------- | :----------------------------------------------------------------------------- |
|
||||
| `npm run test:combo:live` | In-process real routing with `RUN_COMBO_LIVE=1`; snapshots a live OmniRoute DB |
|
||||
| `npm run test:combo:live:vps` | HTTP calls against a live OmniRoute server (set `COMBO_LIVE_BASE_URL`) |
|
||||
| `npm run test:combo:live:vps:failover` | Same, with deliberate failover scenarios |
|
||||
|
||||
These smoke tests exercise the real wire path (combo → provider → completion). They are
|
||||
intentionally excluded from CI because they require live credentials and VPS access.
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
| :-------------------------------------------------------- | :------------------------------------------------------------------------- |
|
||||
| `open-sse/services/autoCombo/scoring.ts` | 9-factor scoring function, `DEFAULT_WEIGHTS`, pool norm |
|
||||
| `open-sse/services/autoCombo/taskFitness.ts` | Model × task fitness lookup |
|
||||
| `open-sse/services/autoCombo/engine.ts` | Selection logic, bandit, budget cap |
|
||||
| `open-sse/services/autoCombo/selfHealing.ts` | Exclusion, probes, incident mode |
|
||||
| `open-sse/services/autoCombo/modePacks.ts` | 4 weight profiles (ship-fast, cost-saver, quality-first, offline-friendly) |
|
||||
| `open-sse/services/autoCombo/autoPrefix.ts` | `auto/` prefix parser + 6 variants |
|
||||
| `open-sse/services/autoCombo/virtualFactory.ts` | Builds in-memory `AutoComboConfig` from live connections |
|
||||
| `open-sse/services/autoCombo/providerRegistryAccessor.ts` | Test hook for mocking provider registry |
|
||||
| `src/shared/constants/routingStrategies.ts` | `ROUTING_STRATEGY_VALUES` (18 strategies) |
|
||||
| `src/sse/handlers/chat.ts` | Integration: auto-prefix short-circuit |
|
||||
@@ -0,0 +1,370 @@
|
||||
---
|
||||
title: "Quota Sharing Engine"
|
||||
version: 3.8.40
|
||||
lastUpdated: 2026-06-28
|
||||
---
|
||||
|
||||
# Quota Sharing Engine
|
||||
|
||||
> **Doc reference**: `docs/routing/QUOTA_SHARE.md`
|
||||
> Part of Group B (plans 16 + 22).
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Quota Sharing Engine distributes a provider's time-based quota (e.g. Codex
|
||||
5-hour window, Kimi 1500 req/h) fairly across multiple API keys that share the
|
||||
same connection.
|
||||
|
||||
**Problem it solves:** OmniRoute proxies many API keys against the same upstream
|
||||
provider account. Without sharing logic, a burst from key A can exhaust the
|
||||
provider quota for the hour, leaving keys B and C blocked until the window resets.
|
||||
The engine prevents this by:
|
||||
|
||||
1. Tracking each key's rolling consumption per dimension (%, requests, tokens, $).
|
||||
2. Applying a work-conserving fair-share algorithm: a key may borrow from idle
|
||||
shares while the global pool is not saturated.
|
||||
3. Enforcing the result in the hot path (`chatCore.ts`) before the request
|
||||
reaches the upstream executor.
|
||||
|
||||
---
|
||||
|
||||
## Algorithm: Fair-Share Work-Conserving
|
||||
|
||||
Implemented in `src/lib/quota/fairShare.ts`.
|
||||
|
||||
### Modes
|
||||
|
||||
| Condition | Mode | Behaviour |
|
||||
| ------------------------------------------ | ------------ | ------------------------------------------------------ |
|
||||
| `globalUsedPercent < saturationThreshold` | **Generous** | Key may borrow up to global limit minus consumed-total |
|
||||
| `globalUsedPercent >= saturationThreshold` | **Strict** | Enforce individual fair share strictly |
|
||||
|
||||
Default `saturationThreshold = 0.5` (env `QUOTA_SATURATION_THRESHOLD`).
|
||||
|
||||
### Per-dimension decision
|
||||
|
||||
For each active dimension in the pool, the engine computes:
|
||||
|
||||
```
|
||||
fairShareAllowed = poolLimit × (allocationWeight / 100)
|
||||
consumed = current rolling value for this key (from QuotaStore.peek)
|
||||
remaining = fairShareAllowed - consumed
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
- **`policy = hard`**: if `consumed > fairShareAllowed` and mode is strict → **block**.
|
||||
- **`policy = soft`**: if `consumed > fairShareAllowed` and mode is strict → **penalize** (deprioritize in combo; never hard-block).
|
||||
- **`policy = burst`**: allow while global headroom exists regardless of fair share.
|
||||
|
||||
### Cap absoluto
|
||||
|
||||
`capValue` + `capUnit` on an allocation is a hard ceiling independent of mode or
|
||||
policy. Any dimension where `consumed >= capValue` always **blocks** the request.
|
||||
|
||||
### Multi-dimension check
|
||||
|
||||
A request is blocked if **any** dimension in the pool would block it. Dimensions
|
||||
are independent — a 5h% exhaustion does not affect the weekly% dimension.
|
||||
|
||||
### Borrowing
|
||||
|
||||
In generous mode, a key whose allocation is under-consumed can use surplus from
|
||||
other keys' unallocated shares. The formula is:
|
||||
|
||||
```
|
||||
maxAllowed = globalLimit - consumedByOtherKeys
|
||||
```
|
||||
|
||||
where `consumedByOtherKeys = consumedTotal - consumedByThisKey`. The teto global
|
||||
(pool `limit` for that dimension) is always the hard ceiling.
|
||||
|
||||
---
|
||||
|
||||
## Sliding Window Counter
|
||||
|
||||
Implemented in `src/lib/quota/sqliteQuotaStore.ts` and `redisQuotaStore.ts`.
|
||||
|
||||
Two buckets per `(apiKeyId, dimensionKey)`:
|
||||
|
||||
- `curr`: current bucket (`floor(nowMs / windowMs)`)
|
||||
- `prev`: previous bucket (`curr - 1`)
|
||||
|
||||
Effective rolling value:
|
||||
|
||||
```
|
||||
effectiveBucketIndex = floor(nowMs / windowMs)
|
||||
bucketStartMs = effectiveBucketIndex × windowMs
|
||||
elapsed = nowMs - bucketStartMs
|
||||
weight = 1 - elapsed / windowMs
|
||||
|
||||
effective = prev × weight + curr
|
||||
```
|
||||
|
||||
**Precision**: ~99% accurate. The error is at most 1% of the window size at the
|
||||
boundary between buckets (inherent to the 2-bucket approximation).
|
||||
|
||||
### Concurrency
|
||||
|
||||
SQLite driver: in-memory mutex per `(apiKeyId | dimensionKey)` key prevents the
|
||||
read-modify-write race. Pattern mirrors `src/sse/services/auth.ts` anti-thundering-herd.
|
||||
|
||||
Redis driver: Lua EVAL script for atomic increment — runs as a single Redis command.
|
||||
|
||||
---
|
||||
|
||||
## Drivers
|
||||
|
||||
### SQLite (default, 0-install)
|
||||
|
||||
- Table: `quota_consumption` (see migration `073_quota_pools.sql` / `074_quota_consumption.sql`).
|
||||
- Best for single-instance deployments.
|
||||
- All persistence is in the existing OmniRoute SQLite DB (`DATA_DIR/storage.sqlite`).
|
||||
|
||||
### Redis (optional, multi-instance)
|
||||
|
||||
- Requires `ioredis` npm package.
|
||||
- Counters stored in Redis; metadata (pools/allocations) still in SQLite.
|
||||
- Best for multi-replica deployments where counters must be shared.
|
||||
|
||||
### Switching drivers
|
||||
|
||||
Via settings UI (`/dashboard/settings` → Quota Store), or via env vars:
|
||||
|
||||
```bash
|
||||
QUOTA_STORE_DRIVER=redis
|
||||
QUOTA_STORE_REDIS_URL=redis://localhost:6379
|
||||
```
|
||||
|
||||
DB setting has precedence over env. If `driver=redis` but URL is absent or
|
||||
`ioredis` is not installed, the factory falls back to SQLite and logs a warning.
|
||||
|
||||
Driver selection order:
|
||||
|
||||
1. DB setting `quotaStore.driver`
|
||||
2. Env `QUOTA_STORE_DRIVER`
|
||||
3. Default: `sqlite`
|
||||
|
||||
---
|
||||
|
||||
## Multi-Dimension
|
||||
|
||||
A pool can have multiple dimensions. Each dimension is independent:
|
||||
|
||||
```ts
|
||||
QuotaDimension {
|
||||
unit: "percent" | "requests" | "tokens" | "usd",
|
||||
window: "5h" | "hourly" | "daily" | "weekly" | "monthly",
|
||||
limit: number, // global pool ceiling for this dimension
|
||||
}
|
||||
```
|
||||
|
||||
**Example: Codex plan** (5h% + weekly%):
|
||||
|
||||
```json
|
||||
[
|
||||
{ "unit": "percent", "window": "5h", "limit": 100 },
|
||||
{ "unit": "percent", "window": "weekly", "limit": 100 }
|
||||
]
|
||||
```
|
||||
|
||||
A request must satisfy all dimensions to be allowed.
|
||||
|
||||
---
|
||||
|
||||
## Plan Resolver
|
||||
|
||||
Implemented in `src/lib/quota/planResolver.ts`.
|
||||
|
||||
Precedence (highest to lowest):
|
||||
|
||||
1. **Manual DB override** — `provider_plans` table, per `connectionId`.
|
||||
2. **Known catalog** — `src/lib/quota/planRegistry.ts` (data-only).
|
||||
3. **Empty plan** — no dimensions, manual configuration required.
|
||||
|
||||
### Known catalog
|
||||
|
||||
| Provider | Dimensions |
|
||||
| --------------------- | ------------------------------------------------------------- |
|
||||
| `codex` | `percent/5h/100`, `percent/weekly/100` |
|
||||
| `glm` | `tokens/5h` (limit=0, unknown), `tokens/weekly` |
|
||||
| `minimax` | `tokens/5h`, `tokens/weekly` |
|
||||
| `bailian` | `percent/5h/100`, `percent/weekly/100`, `percent/monthly/100` |
|
||||
| `kimi` | `requests/hourly/1500` |
|
||||
| `alibaba` | `requests/monthly/90000` |
|
||||
| `openai`, `anthropic` | No default — manual configuration required |
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Integration
|
||||
|
||||
### PRE hook (`open-sse/handlers/chatCore.ts`)
|
||||
|
||||
Runs before the upstream executor, after auth and policy checks:
|
||||
|
||||
```
|
||||
resolveComboTargets / handleSingleModel
|
||||
→ enforceQuotaShare(apiKeyId, connectionId, provider, estimatedCost)
|
||||
→ getQuotaStore().peek() per dimension
|
||||
→ fairShare.decideFairShare()
|
||||
→ if block → return 429 (buildErrorBody, Hard Rule #12)
|
||||
→ if allow + deprioritize → set quotaSoftPenalty=true on candidate
|
||||
→ executor.execute()
|
||||
```
|
||||
|
||||
**Fail-open**: if `enforceQuotaShare` throws, the request is allowed through
|
||||
with a `pino.warn` log. This prevents a quota-engine bug from blocking all
|
||||
traffic.
|
||||
|
||||
### POST hook (record consumption)
|
||||
|
||||
After a successful response:
|
||||
|
||||
```
|
||||
executor returns success
|
||||
→ spendRecorder.recordConsumption(apiKeyId, connectionId, provider, actualCost)
|
||||
→ getQuotaStore().consume() per dimension
|
||||
→ fail-open: errors logged as pino.warn, never propagated to client
|
||||
```
|
||||
|
||||
**Drift note**: if `consume` fails post-response, the rolling counter under-counts.
|
||||
The saturation signal from the provider (e.g. `anthropic-ratelimit-unified-5h-utilization`)
|
||||
corrects the global estimate on the next request.
|
||||
|
||||
### Combo soft penalty (`open-sse/services/combo.ts`)
|
||||
|
||||
When `decision.deprioritize === true`:
|
||||
|
||||
```ts
|
||||
if (candidate.quotaSoftPenalty) {
|
||||
score *= QUOTA_SOFT_DEPRIORITIZE_FACTOR; // default 0.7
|
||||
}
|
||||
```
|
||||
|
||||
The penalty is applied after all other scoring factors. It lowers the auto-combo
|
||||
probability of selecting a saturated key without hard-blocking it.
|
||||
|
||||
---
|
||||
|
||||
## UI Walkthrough
|
||||
|
||||
### `/dashboard/costs/quota-share` — Main pools page
|
||||
|
||||
Components (all in `src/app/(dashboard)/dashboard/costs/quota-share/`):
|
||||
|
||||
| Component | Purpose |
|
||||
| ---------------------- | ----------------------------------------------------------------- |
|
||||
| `QuotaConceptCard` | Introductory card explaining quota sharing to new users |
|
||||
| `CreatePoolModal` | Create a new quota pool (connection + name + initial allocations) |
|
||||
| `PoolCard` | Per-pool summary: name, connection, allocation count |
|
||||
| `DimensionBar` | Per-dimension stacked bar: each key's share + global usage |
|
||||
| `AllocationTable` | Table with consumed, fair share, deficit/surplus, borrowing flag |
|
||||
| `BurnRateChart` | EMA burn-rate line chart (lazy Recharts via `dynamic()`) |
|
||||
| `EditAllocationsModal` | Edit allocation weights, caps, and policies for a pool |
|
||||
|
||||
The page hooks:
|
||||
|
||||
- `usePools` — fetches `GET /api/quota/pools` every 30s.
|
||||
- `usePoolUsage` — fetches `GET /api/quota/pools/[id]/usage` on demand.
|
||||
- `useLocalStoragePoolMigration` — runs once on mount to migrate legacy LS data.
|
||||
|
||||
### `/dashboard/costs/quota-share/plans` — Provider plan config
|
||||
|
||||
- `ProviderPlanConfigClient.tsx`: dropdown to select a provider, view resolved
|
||||
plan (auto from catalog or manual override), and edit dimensions.
|
||||
- Changes write to `PUT /api/quota/plans/[connectionId]`.
|
||||
- Deletion reverts to catalog or empty plan.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| ---------------------------------- | --------- | ------------------------------------------------------ |
|
||||
| `QUOTA_STORE_DRIVER` | `sqlite` | Driver to use: `sqlite` or `redis` |
|
||||
| `QUOTA_STORE_REDIS_URL` | _(empty)_ | Redis URL, e.g. `redis://localhost:6379` |
|
||||
| `QUOTA_SATURATION_THRESHOLD` | `0.5` | 0..1; `>= threshold` activates strict mode |
|
||||
| `QUOTA_SOFT_DEPRIORITIZE_FACTOR` | `0.7` | 0..1; multiplier for soft-policy combo score |
|
||||
| `QUOTA_CONSUMPTION_RETENTION_DAYS` | `14` | Days before GC removes old `quota_consumption` buckets |
|
||||
|
||||
DB settings (`quotaStore.*`) override env vars.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Redis configured but not connecting
|
||||
|
||||
Check that `ioredis` is installed (`npm ls ioredis`) and `QUOTA_STORE_REDIS_URL`
|
||||
is reachable. On connection failure the factory falls back to SQLite (logged at
|
||||
`warn`).
|
||||
|
||||
### `peek` returns stale / fail-open
|
||||
|
||||
If `peek` throws, `enforceQuotaShare` treats the result as "allow" (fail-open).
|
||||
Check `pino` logs for `quota:enforce` and `quota:factory` entries to identify
|
||||
the root cause.
|
||||
|
||||
### Consumption counter drift
|
||||
|
||||
If the actual provider usage differs from the counters, it is expected — the
|
||||
2-bucket sliding window has ~1% error at window boundaries, and `consume` is
|
||||
fire-and-forget post-response. The saturation signal (`saturationSignals.ts`)
|
||||
reads the real provider utilization with a 30s TTL and adjusts `globalUsedPercent`
|
||||
accordingly.
|
||||
|
||||
### Pool shows "no data" for burn rate
|
||||
|
||||
`computeBurnRate` requires at least 2 historical samples. New pools without prior
|
||||
`consume` calls will show `tokensPerSecond: 0` and `timeToExhaustionMs: null`.
|
||||
|
||||
---
|
||||
|
||||
## Migration from localStorage
|
||||
|
||||
When `/dashboard/costs/quota-share` first loads, the hook `useLocalStoragePoolMigration`
|
||||
checks:
|
||||
|
||||
1. `localStorage.getItem("omniroute:quota-share:pools")` is non-empty.
|
||||
2. `GET /api/quota/pools` returns `[]` (DB is empty).
|
||||
|
||||
If both are true, it posts each legacy pool to `POST /api/quota/pools` in batch,
|
||||
then removes the localStorage key. The migration is idempotent: condition 2 prevents
|
||||
re-migration.
|
||||
|
||||
---
|
||||
|
||||
## Internal Strategy Classification
|
||||
|
||||
`quota-share` is an **internal-only** routing strategy (`INTERNAL_ROUTING_STRATEGY_VALUES` in
|
||||
`src/shared/constants/routingStrategies.ts`). It is used exclusively by system-minted
|
||||
`qtSd/` pool combos and is deliberately excluded from `ROUTING_STRATEGY_VALUES` so it never
|
||||
appears as a user-selectable option in the UI or API.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage
|
||||
|
||||
Two layers of automated coverage ship with the quota-share engine:
|
||||
|
||||
| Suite | Command | What it covers |
|
||||
| :----------------- | :--------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Unit (29 tests) | `node --import tsx/esm --test tests/unit/quota-share-strategy.test.ts` | DRR scheduler, saturation gating, concurrency caps, fairShare math, backlog queueing |
|
||||
| Integration matrix | `npm run test:combo:matrix` | End-to-end routing decision through the real combo pipeline; DRR fairness + saturation deprioritization via live seams (`registerQuotaFetcher`, `setLKGP`, `__setHeadroomSaturationFetcherForTests`) |
|
||||
|
||||
The integration matrix runs in CI alongside the other 17 public strategies. The unit suite
|
||||
can be run standalone.
|
||||
|
||||
---
|
||||
|
||||
## DB Schema Summary
|
||||
|
||||
Three tables added by migrations `073–075`:
|
||||
|
||||
- `quota_pools` + `quota_allocations` — pool definitions and per-key allocations.
|
||||
- `quota_consumption` — rolling 2-bucket counters per `(apiKeyId, dimensionKey)`.
|
||||
- `provider_plans` — manual provider plan overrides (dimensions JSON per connectionId).
|
||||
|
||||
All tables added via idempotent `CREATE TABLE IF NOT EXISTS` migrations.
|
||||
@@ -0,0 +1,167 @@
|
||||
---
|
||||
title: "Reasoning Replay Cache"
|
||||
version: 3.8.40
|
||||
lastUpdated: 2026-06-28
|
||||
---
|
||||
|
||||
# Reasoning Replay Cache
|
||||
|
||||
> **Source of truth:** `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`
|
||||
> **Last updated:** 2026-06-28 — v3.8.40
|
||||
|
||||
OmniRoute captures assistant `reasoning_content` produced by thinking-mode models and replays it transparently on multi-turn requests when the upstream provider requires it. This eliminates the HTTP 400 errors that strict providers raise when a client's conversation history is missing the prior turn's reasoning.
|
||||
|
||||
## Why This Exists
|
||||
|
||||
Several thinking-mode providers reject a follow-up turn unless the **previous assistant message includes the original `reasoning_content`**. The upstream returns 400 with messages like:
|
||||
|
||||
```
|
||||
Param Incorrect: The reasoning_content in the thinking mode must be passed back to the API.
|
||||
```
|
||||
|
||||
But typical clients (Cursor, Cline, Roo Code, OpenAI SDK) strip `reasoning_content` from the history they replay. OmniRoute restores it from a server-side cache so the request the upstream sees is consistent. Issue #1628 introduced the hybrid memory/SQLite persistence so the cache survives process restarts.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Turn N (assistant generates):
|
||||
→ response contains reasoning_content + tool_calls
|
||||
→ cacheReasoningFromAssistantMessage() writes (memory + DB), keyed by every tool_call.id
|
||||
→ forward response to client (which may or may not retain reasoning)
|
||||
|
||||
Turn N+1 (client sends follow-up):
|
||||
→ translator detects: requiresReasoningReplay(provider, model) === true
|
||||
→ for each assistant message with tool_calls and no reasoning_content:
|
||||
lookupReasoning(toolCalls[0].id) → memory → DB
|
||||
hit → msg.reasoning_content = cached; recordReplay()
|
||||
miss → msg.reasoning_content = "" (legacy fallback for older DeepSeek)
|
||||
→ upstream sees consistent history → no 400
|
||||
```
|
||||
|
||||
Capture happens in `open-sse/handlers/chatCore.ts` (two sites, around lines 4093 and 4380). Replay happens in `open-sse/translator/index.ts` after schema coercion but before dispatch.
|
||||
|
||||
## Storage — Hybrid Memory + SQLite
|
||||
|
||||
The hot path uses an in-memory `Map` (LRU-by-creation) backed by a SQLite table for crash recovery and dashboard visibility.
|
||||
|
||||
| Layer | Implementation | Purpose |
|
||||
| ------ | ---------------------------------------------- | -------------------------------------- |
|
||||
| Memory | `Map` in `open-sse/services/reasoningCache.ts` | Fast lookups, evicts oldest at 2000 |
|
||||
| DB | `reasoning_cache` table (`src/lib/db/`) | Persists across restarts, drives stats |
|
||||
|
||||
Writes go to both. Reads consult memory first, then fall back to DB (DB hits are promoted back into memory). DB failures are non-fatal — the in-memory cache continues to serve the hot path.
|
||||
|
||||
**Defaults:**
|
||||
|
||||
- TTL: `2h` (`TTL_MS = 2 * 60 * 60 * 1000`)
|
||||
- Max memory entries: `2000` (`MAX_MEMORY_ENTRIES`)
|
||||
- Eviction: oldest `createdAt` first
|
||||
|
||||
## Database Schema
|
||||
|
||||
Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS reasoning_cache (
|
||||
tool_call_id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
reasoning TEXT NOT NULL,
|
||||
char_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
expires_at INTEGER NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
Indexes: `expires_at`, `provider`, `model`, `created_at`. `expires_at` is stored as Unix epoch seconds; the SELECT layer normalizes legacy text values via `EXPIRES_AT_EPOCH_SQL`.
|
||||
|
||||
## Provider / Model Detection
|
||||
|
||||
Replay is enabled when `requiresReasoningReplay(provider, model)` returns `true`. The function checks two lists in `open-sse/services/reasoningCache.ts`.
|
||||
|
||||
**Provider IDs (exact match, case-insensitive):**
|
||||
|
||||
- `deepseek`
|
||||
- `opencode-go`
|
||||
- `siliconflow`
|
||||
- `nebius`
|
||||
- `deepinfra`
|
||||
- `sambanova`
|
||||
- `fireworks`
|
||||
- `together`
|
||||
- `xiaomi-mimo`
|
||||
|
||||
**Model regex patterns (case-insensitive):**
|
||||
|
||||
- `/deepseek-r1/i`
|
||||
- `/deepseek-reasoner/i`
|
||||
- `/deepseek-chat/i`
|
||||
- `/deepseek[-/]?v4[-.]flash/i` and `/deepseek[-/]?v4[-.]pro/i` (V4 Flash / Pro, optional `-free` suffix)
|
||||
- `/(deepseek|zen\/deepseek)-v4/i`
|
||||
- `/kimi-k2/i`
|
||||
- `/qwq/i`
|
||||
- `/qwen.*think/i`
|
||||
- `/glm.*think/i`
|
||||
- `/^mimo[-.]?v\d/i`
|
||||
|
||||
Adding a new strict provider/model means appending to one of these lists and writing a unit test asserting replay injection. The PR description should cite the exact upstream 400 string that motivated the change.
|
||||
|
||||
## REST API
|
||||
|
||||
The cache exposes two endpoints under `src/app/api/cache/reasoning/route.ts`. Both require management authentication (`isAuthenticated` from `@/shared/utils/apiAuth`).
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
| ------ | --------------------------------------------------------- | -------------------------------------------------------- |
|
||||
| GET | `/api/cache/reasoning` | Stats + paginated entries |
|
||||
| GET | `/api/cache/reasoning?provider=deepseek&model=...&limit=` | Filtered listing (`limit` clamped to `[1, 200]`) |
|
||||
| DELETE | `/api/cache/reasoning` | Clear everything (memory + DB) and reset hit/miss counts |
|
||||
| DELETE | `/api/cache/reasoning?provider=deepseek` | Clear only entries for one provider |
|
||||
| DELETE | `/api/cache/reasoning?toolCallId=call_abc` | Delete a single entry |
|
||||
|
||||
**GET response shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"stats": {
|
||||
"memoryEntries": 12,
|
||||
"dbEntries": 47,
|
||||
"totalEntries": 47,
|
||||
"totalChars": 138291,
|
||||
"hits": 84,
|
||||
"misses": 6,
|
||||
"replays": 81,
|
||||
"replayRate": "90.0%",
|
||||
"byProvider": { "deepseek": { "entries": 32, "chars": 98412 } },
|
||||
"byModel": { "deepseek-reasoner": { "entries": 32, "chars": 98412 } },
|
||||
"oldestEntry": "2026-05-13T10:00:00.000Z",
|
||||
"newestEntry": "2026-05-13T11:42:11.000Z"
|
||||
},
|
||||
"entries": [
|
||||
{
|
||||
"toolCallId": "call_abc",
|
||||
"provider": "deepseek",
|
||||
"model": "deepseek-reasoner",
|
||||
"reasoning": "...",
|
||||
"charCount": 3128,
|
||||
"createdAt": "...",
|
||||
"expiresAt": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- **Cleanup:** `cleanupReasoningCache()` purges expired memory entries and runs `DELETE FROM reasoning_cache WHERE expires_at <= unixepoch('now')`. Health-check workers call this periodically.
|
||||
- **Crash recovery:** After a restart, memory is empty but the DB still holds unexpired entries. The first lookup for a given `tool_call_id` is a DB hit; subsequent lookups are memory hits.
|
||||
- **No reasoning, no cache:** `cacheReasoningFromAssistantMessage` returns `0` when the assistant message has no `reasoning_content` / `reasoning` field, so non-thinking responses cost nothing.
|
||||
- **Non-strict providers:** When `requiresReasoningReplay` is `false` and the target format is OpenAI, the translator **strips** any `reasoning_content` field from outgoing messages — OpenAI Chat Completions does not accept it.
|
||||
|
||||
## See Also
|
||||
|
||||
- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) — circuit breakers, cooldowns, model lockouts
|
||||
- [TROUBLESHOOTING.md](../guides/TROUBLESHOOTING.md) — diagnosing upstream 400s
|
||||
- Source: `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`, `open-sse/translator/index.ts`
|
||||
- Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
||||
- API route: `src/app/api/cache/reasoning/route.ts`
|
||||
- Original issue: #1628
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"title": "Routing",
|
||||
"pages": ["AUTO-COMBO", "QUOTA_SHARE", "REASONING_REPLAY"]
|
||||
}
|
||||
Reference in New Issue
Block a user