opensquilla--opensquilla/docs/features/LLM-ensemble-design.md

# LLM Ensemble Design: Static Lineups & Dynamic Router Selection

`llm_ensemble` runs a **B5 fusion** turn: several *proposer* models each draft
an answer, and one *aggregator* model fuses those drafts into the final
response. This document describes how the **set of models** is chosen for a
turn. It does not cover the ensemble runtime mechanics (streaming, timeouts,
quorum, fallback) — only model selection.

## Why an ensemble instead of a single model

Any single model has a fixed set of blind spots: the failure modes of its
training data, its decoding randomness, and its particular biases. Asking that
one model again doesn't remove them — it re-rolls the *same* distribution. An
ensemble attacks the problem from a different angle: draft an answer with
several *different* models, then have an aggregator reconcile them. The wins
over a single-model turn:

- **Error cancellation / higher accuracy.** Independent models rarely make the
  *same* mistake on the same input. When drafts disagree, the aggregator can
  cross-check them and keep the answer the majority supports; when they agree,
  that agreement is real signal that the answer is solid. Idiosyncratic
  one-off errors get outvoted instead of shipped.
- **Coverage through diversity.** Different vendors/families/architectures have
  genuinely different strengths — one is better at code, another at long-form
  reasoning, another at careful instruction-following. A lineup that spans them
  covers more of the input space than any single model's strong suit. This is
  exactly why selection *rewards diversity* (distinct vendor/family/
  architecture) rather than picking the top-N by raw quality.
- **Robustness & availability.** A single model is a single point of failure —
  one timeout, rate-limit, or degraded response fails the whole turn. With a
  quorum of proposers the turn still succeeds as long as *enough* drafts come
  back, and the aggregator simply fuses what arrived.
- **Reduced variance.** Fusing several drafts smooths out per-call sampling
  noise, so repeated runs of the same prompt are more stable and less
  sensitive to an unlucky roll of the dice on any one model.
- **A critic pass, not just a vote.** The aggregator is a model in its own
  right: it can spot a draft that's confidently wrong, prefer the
  better-reasoned answer over the more verbose one, and synthesize the best
  parts of several drafts — a step a single-model turn never gets.

The cost is real — an N-proposer turn spends roughly N+1 model calls and its
latency is bounded by the slowest proposer plus the aggregator. The selection
strategies below exist to spend that budget well: match ensemble size and
composition to how hard the turn actually is, rather than always paying for the
largest lineup. Lower-difficulty turns get a small, cheap lineup; harder turns
get more proposers and stronger critics.

## Selection strategies

There are three selection strategies, dispatched by
`llm_ensemble.selection_mode` in
`build_ensemble_provider_from_config`
(`src/opensquilla/provider/ensemble.py`):

| `selection_mode` | Family | Status |
|------------------|--------|--------|
| `static_openrouter_b5` | Static lineup | Default for fresh configs |
| `static_tokenrhythm_b5` | Static lineup | Supported |
| `custom_b5` | Static lineup (user-authored) | Supported |
| `router_dynamic` | Dynamic selection | Legacy |

The first two families are **static**: the lineup is fixed ahead of the turn,
either from a packaged preset or from an explicit user-authored list. The last
is **dynamic**: the lineup is scored and assembled per turn from the router's
own tier decision.

Fresh configs default to `static_openrouter_b5`. The Web UI offers only the
static families (preset + custom); `router_dynamic` is no longer offered there
and stored configs surface a one-click migration to `custom_b5`. Direct
TOML/RPC configuration keeps working for every mode.

---

# Part 1 — Static Lineups (current design)

A static lineup is fixed before the turn runs: a set of proposer models plus one
aggregator model, all known ahead of time. No per-turn scoring happens. Two
variants share this shape:

- **Presets** — `static_openrouter_b5` / `static_tokenrhythm_b5`: packaged,
  hard-coded lineups on a single provider.
- **Custom** — `custom_b5`: an explicit user-authored lineup with
  role-labelled candidates and a single aggregator.

Both variants belong to the same **fixed-lineup defaults family**
(`is_static_b5` in the builder) and therefore inherit the same runtime defaults
(quorum, timeouts, no shuffle, quorum grace) — see
[Shared fixed-lineup defaults](#shared-fixed-lineup-defaults).

## 1.1 Static presets

Source: `_build_static_b5_members`, `STATIC_B5_PROFILES`
(`src/opensquilla/provider/ensemble.py`).

Each preset is a `StaticB5Profile` — four fixed proposers plus one aggregator,
all bound to a single provider:

| Profile | Provider | Proposers | Aggregator |
|---------|----------|-----------|------------|
| `static_openrouter_b5` | `openrouter` | `deepseek/deepseek-v4-pro`, `z-ai/glm-5.2`, `moonshotai/kimi-k2.7-code`, `qwen/qwen3.7-max` | `z-ai/glm-5.2` |
| `static_tokenrhythm_b5` | `tokenrhythm` | `deepseek-v4-pro`, `glm-5.2`, `kimi-k2.7-code`, `qwen3.7-max` | `glm-5.2` |

The TokenRhythm profile is a mirror of the OpenRouter one: same aggregation
shape and defaults, the same four models, only the provider and the model-id
naming differ (OpenRouter-style `vendor/model` slugs vs. TokenRhythm's bare
names).

`_build_static_b5_members` simply materializes the profile: each proposer model
becomes an `EnsembleMemberConfig` labeled `proposer_1..N`, the aggregator model
becomes one labeled `aggregator`, and the selection plan records the profile
name, proposer/aggregator models, and proposer count. There is nothing to score
— the lineup is the profile.

### Credential gate

`static_b5_credential_available` decides whether the ensemble may run: it
resolves an API key for every member (all four proposers + aggregator) using the
same key-resolution order as the runtime (see
[Member provider resolution](#member-provider-resolution)). A user whose active
provider differs but whose environment carries the profile provider's env key
(e.g. `OPENROUTER_API_KEY`, `TOKENRHYTHM_API_KEY`) is treated as opted in. If any
member cannot resolve a key, the ensemble is skipped rather than posting a turn
upstream with an empty bearer token.

## 1.2 Custom lineup (`custom_b5`)

Source: `_build_custom_b5_members`, `_custom_b5_candidates`
(`src/opensquilla/provider/ensemble.py`); schema `LlmEnsembleCandidateConfig`
(`src/opensquilla/gateway/config.py`).

`custom_b5` lets an operator author the lineup explicitly via
`llm_ensemble.candidates`. Each candidate row carries:

- **`provider`** / **`model`** — required, non-empty; provider is lower-cased.
- **`role`** — advisory label, one of `""` (unassigned), `primary`, `contrast`,
  `fast_check`, `critic`, or the structural `aggregator`. Unknown values coerce
  to `""` instead of failing, so a hand-edited config never blocks boot.
- **`enabled`** — disabled rows are kept for read compatibility but never
  counted or run.

Lineup assembly (`_build_custom_b5_members`):

1. Every enabled row whose role is **not** `aggregator` runs as a proposer,
   labeled by its role (or `proposer_N` when unassigned).
2. The single row with role `aggregator` fuses the drafts. Proposer rows dedupe
   on `(provider, model)`; the aggregator row may legitimately reuse a
   proposer's model (a model both drafts and fuses).
3. **Fallback:** if no `aggregator` row exists, the aggregator falls back to the
   currently routed model — the same model the user would have gotten without
   the ensemble — so a proposer-only config still runs instead of erroring at
   turn time. The plan records `aggregator.source` as `candidate_role` or
   `inherited_model` accordingly.

### Lineup bounds & validation

Enforced by `LlmEnsembleConfig._validate_custom_b5_lineup`
(`src/opensquilla/gateway/config.py`), checked **only** when
`selection_mode == "custom_b5"` (presets carry fixed lineups; `router_dynamic`
selects per turn):

- At most **one** enabled candidate may carry role `aggregator`.
- Enabled proposer count must stay within
  `[CUSTOM_B5_MIN_PROPOSERS=2, CUSTOM_B5_MAX_PROPOSERS=6]`.
- Total per-turn calls are capped at `CUSTOM_B5_MAX_TOTAL_CALLS=8`
  (proposers + aggregator).

### Readiness gate

`custom_b5_lineup_ready` returns `(ready, reason)` before wrapping the turn. It
fails closed with a machine-readable reason when the lineup is not runnable:
`no_proposers`, `unknown_provider:<p>`, or `missing_credential:<p>` (a member
whose provider requires a key but resolves none). This mirrors the static-preset
gate — a member with an empty bearer token would post the conversation upstream
unauthenticated, so the wrap is skipped.

## 1.3 Shared fixed-lineup defaults

Both static families set `is_static_b5 = True` in
`build_ensemble_provider_from_config`, which swaps the legacy per-turn defaults
for the fixed-lineup family defaults. The swap is **only** applied when the
configured value still equals the legacy default (`_static_default_if_legacy`),
so any operator override is preserved:

| Parameter | Legacy (`router_dynamic`) | Fixed-lineup default |
|-----------|---------------------------|----------------------|
| `min_successful_proposers` | 1 | 3 (presets) / `N-1` (custom, "all but one") |
| `proposer_timeout_seconds` | 3600 | 300 |
| `aggregator_timeout_seconds` | 3600 | 480 |
| `shuffle_candidates` | `True` | `False` |
| `quorum_grace_seconds` | 0 | 30 |

`min_successful_proposers` is additionally clamped down to the actual proposer
count. Both the configured and effective values (min-success, timeouts, shuffle)
are recorded in the selection plan for debugging.

## 1.4 Member provider resolution

Every member (static or custom) resolves its concrete `ProviderConfig` through
`_member_provider_config`, which layers member intent over the inherited/routed
provider config:

- **API key** — a member-level `api_key_env` env var if set; else the inherited
  key when the member's provider matches the active provider; else the provider
  registry's env key (e.g. `OPENROUTER_API_KEY`).
- **`base_url`** — member override, else the inherited base URL (same provider)
  or the provider spec's default base URL.
- **`proxy` / `org_id` / `provider_routing`** — inherited only when the member
  shares the active provider; otherwise reset.

This is what lets a static/custom lineup run against a provider the user isn't
actively routing to, as long as that provider's credential is present in the
environment.

## 1.5 Configuration surface

```toml
[llm_ensemble]
enabled = true
selection_mode = "static_openrouter_b5"   # or static_tokenrhythm_b5 / custom_b5
```

Custom lineup:

```toml
[llm_ensemble]
enabled = true
selection_mode = "custom_b5"

[[llm_ensemble.candidates]]
provider = "openrouter"
model = "deepseek/deepseek-v4-pro"
role = "primary"

[[llm_ensemble.candidates]]
provider = "openrouter"
model = "z-ai/glm-5.2"
role = "contrast"

[[llm_ensemble.candidates]]
provider = "openrouter"
model = "z-ai/glm-5.2"
role = "aggregator"
```

Static presets expose no lineup tuning — the models are fixed in code. Custom
lineups are tuned entirely through the `candidates` list (subject to the bounds
above). Both share the fixed-lineup runtime defaults, which an operator may
still override explicitly (`min_successful_proposers`,
`proposer_timeout_seconds`, `aggregator_timeout_seconds`, `shuffle_candidates`).

---

# Part 2 — `router_dynamic` Selection (legacy)

> **Status: legacy.** `router_dynamic` remains fully supported for existing
> configs but is no longer offered in the Web UI. Stored `router_dynamic`
> configs surface a one-click migration to `custom_b5`. Direct TOML/RPC
> configuration keeps working as described below.

`router_dynamic` is the dynamic model-selection strategy: instead of a fixed
lineup, it picks proposers and the aggregator **per turn**, driven by
SquillaRouter's tier decision for that turn. Enable it with
`llm_ensemble.selection_mode = "router_dynamic"`.

Source: `src/opensquilla/provider/ensemble.py`
(`_candidate_pool`, `_score_dynamic_candidate`, `_select_dynamic_candidate`,
`_build_router_dynamic_members`).

## 2.1 Why dynamic selection

A fixed proposer/aggregator list can't adapt to the model actually chosen for a
turn, and forces operators to hand-tune which models pair well together at each
router tier. `router_dynamic` instead:

- reuses the model SquillaRouter already picked for the turn as the **anchor**
  proposer, so the ensemble never contradicts the router's own tier decision;
- fills the remaining proposer slots and the aggregator slot by scoring a pool
  of candidate models against a per-tier "slot template";
- penalizes re-selecting a model that's already in the ensemble, so proposers
  stay diverse instead of collapsing onto a few high-quality models.

## 2.2 Inputs

`_build_router_dynamic_members` takes three things:

1. **`inherited_provider_config`** — the provider/model SquillaRouter already
   resolved for this turn (becomes the anchor).
2. **`turn_metadata`** — carries `routed_tier` (`c0`–`c3`), `routing_confidence`
   (0.0–1.0), and `routing_extra` (`final_tier`/`base_tier` fallbacks used if
   `routed_tier` is missing). Defaults to tier `c1` if nothing usable is found.
3. **`config`** — `llm_ensemble.model_options` and `squilla_router.tiers`,
   used to build the candidate pool.

## 2.3 Candidate pool

`_candidate_pool` assembles a deduplicated list of `(provider, model)`
candidates, in this order:

1. **Router anchor** — the inherited provider/model (`source="router_anchor"`).
   This is always `pool[0]` and always becomes the first proposer.
2. **`llm_ensemble.model_options`** — the operator-configured candidate list
   (`source="model_options"`). If a model string contains `/` it's assumed to
   be an OpenRouter-style id and routed via `openrouter`; otherwise it inherits
   the anchor's provider.
3. **`squilla_router.tiers[*].model`** — every model configured for a
   SquillaRouter tier (`source="router_tier:<tier>"`), so tier-specific models
   the operator has wired into the router are eligible even if not listed in
   `model_options`.

Each candidate is annotated with priors from `_DYNAMIC_MODEL_CATALOG` — a
built-in table of ~14 known models with `tier`, `quality` (0–1), `cost_latency`
(0–1, higher = cheaper/faster), `family`, `vendor`, and `architecture`. Models
not in the catalog fall back to tier-average priors (`_tier_quality_prior`,
`_tier_cost_latency_prior`) derived from the model string or tier hint.

## 2.4 Slot templates

Each router tier maps to an ordered list of proposer "slots"
(`_DYNAMIC_TIER_SLOTS`):

| Tier | Slots |
|------|-------|
| `c0` | `anchor`, `cheap_contrast` |
| `c1` | `anchor`, `balanced_contrast` |
| `c2` | `anchor`, `adjacent_tier_check`, `orthogonal_family` |
| `c3` | `anchor`, `strong_critic`, `orthogonal_family`, `fast_sanity` |

Lower tiers (cheap/simple turns) get a small, cost-biased ensemble; higher
tiers (hard turns) get more proposers with slots biased toward quality and
contrast. The `anchor` slot is always filled by the router's own model and is
never scored — it's taken as-is.

Each tier also maps to an aggregator slot (`_DYNAMIC_AGGREGATOR_SLOT`):
`c0→aggregator_fast`, `c1→aggregator_balanced`, `c2`/`c3→aggregator_strong`.

## 2.5 Scoring a candidate for a slot

For every non-anchor slot, every pool candidate is scored and the best one is
selected (`_select_dynamic_candidate` → `_score_dynamic_candidate`):

```
score = weights.quality   * quality_prior
      + weights.affinity  * router_affinity_score
      + weights.diversity * diversity_score
      + weights.cost      * cost_latency_prior
      + weights.role      * role_match_score(slot)
      - duplicate_penalty
```

Each slot has its own weight vector (`_DYNAMIC_SLOT_WEIGHTS`), e.g.
`cheap_contrast` weights `cost` and `role` heavily and `affinity` lightly,
while `strong_critic` weights `quality` and `role` heavily and `cost` almost
not at all.

### Score components

- **`router_affinity_score`** — how close the candidate's tier prior is to the
  turn's `routed_tier`, scaled by `routing_confidence`. Low router confidence
  relaxes tier matching instead of forcing a brittle lock, since a
  low-confidence route is itself uncertain about the right tier.
- **`diversity_score`** — rewards a candidate whose family/vendor/provider/
  tier/architecture aren't already represented among the proposers picked so
  far in this turn (checked incrementally, slot by slot).
- **`role_match_score`** — slot-specific logic (see below), combining tier
  targeting, contrast against the anchor, quality, or cost depending on what
  that slot is supposed to contribute.
- **`duplicate_penalty`** — `_DYNAMIC_SELECTED_PENALTY[slot] * times_already_selected`.
  Selecting the same `(provider, model)` again is allowed but costs
  increasingly more as the same model keeps winning slots.

### Role match by slot

`_role_match_score` differs by slot — this is where each slot's intent is
actually encoded:

- **`cheap_contrast`** — favors tier `c0`/`c1`, contrast with the anchor, and
  cost/latency. A cheap "second opinion."
- **`balanced_contrast`** — favors tier `c1`/`c2`, contrast, and quality.
- **`adjacent_tier_check`** — favors a tier one step above/below the routed
  tier (`adjacent_distance == 1`), plus quality. Checks whether a
  slightly-different-strength model agrees.
- **`orthogonal_family`** — favors contrast and diversity above all — a
  model from a different vendor/family/architecture than the anchor.
- **`strong_critic`** — favors tier `c3` and quality heavily — the strongest
  available model as a critic, used only at higher tiers.
- **`fast_sanity`** — favors tier `c0`/`c1` and cost/latency — a fast,
  cheap sanity check, used only at `c3`.
- **`aggregator_fast` / `aggregator_balanced` / `aggregator_strong`** — each
  balances tier targeting and quality differently; `aggregator_strong`
  weights quality highest and cost lowest, since the aggregator's output is
  the final response.

### Tie-breaking

Candidates are sorted by `(score, quality_prior, cost_latency_prior,
-pool_index)` descending, so ties fall back to higher quality, then higher
cost/latency score, then earlier pool position (closer to the anchor/operator-
configured list) wins.

## 2.6 Selection order

`_build_router_dynamic_members` runs slots in the tier's template order:

1. `anchor` — taken directly, no scoring.
2. Remaining proposer slots, in order — each selection is added to `selected`
   and `selected_counts` before the next slot is scored, so later slots see
   updated diversity/duplicate state.
3. The aggregator slot, scored last, against the same accumulated `selected`
   state as the proposers (so it also gets a duplicate penalty if it repeats
   a proposer's model).

## 2.7 Output

The function returns `(profile_name, proposers, aggregator, selection_plan)`:

- `profile_name` — `"router_dynamic/<tier>"`, e.g. `"router_dynamic/c2"`.
- `proposers` — one `EnsembleMemberConfig` per slot, labeled by slot name
  (`anchor`, `cheap_contrast`, ...).
- `aggregator` — one `EnsembleMemberConfig`, labeled `aggregator`.
- `selection_plan` — a full trace for observability, including the resolved
  tier/confidence, the anchor, the slot template, per-slot score breakdowns
  (`_score_trace`, including the top-3 scored candidates per slot for
  debugging near-misses), the aggregator's score breakdown, the full
  candidate pool, and `duplicate_policy: "selected_penalty"`.

`build_ensemble_provider_from_config` (the public entrypoint) additionally
clamps `min_successful_proposers` down to `len(proposers)` if the configured
value exceeds how many proposer slots the tier's template actually produced
— e.g. configuring `min_successful_proposers=4` at tier `c0` (2 slots) yields
an effective minimum of 2. Both the configured and effective values are
recorded in `selection_plan` for debugging.

## 2.8 Configuration surface

```toml
[llm_ensemble]
enabled = true
selection_mode = "router_dynamic"
```

What operators can tune:

- `llm_ensemble.model_options` — extends the candidate pool beyond the router
  anchor and configured router tiers.
- `llm_ensemble.min_successful_proposers` — desired minimum successful
  proposers (clamped per-turn as described above).
- `squilla_router.tiers[*].model` — indirectly expands the candidate pool and
  determines which model becomes the anchor for a given tier.

There is no operator control over slot templates, weights, or the model
catalog priors — those are fixed in code. Unlike the static families,
`router_dynamic` keeps the legacy runtime defaults (timeouts 3600s,
`shuffle_candidates=True`, `min_successful_proposers=1`).