wehub-resource-sync
168 lines
7.4 KiB
Markdown
168 lines
7.4 KiB
Markdown
---
|
|
title: "Reasoning Replay Cache"
|
|
version: 3.8.40
|
|
lastUpdated: 2026-06-28
|
|
---
|
|
|
|
# Reasoning Replay Cache
|
|
|
|
> **Source of truth:** `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`
|
|
> **Last updated:** 2026-06-28 — v3.8.40
|
|
|
|
OmniRoute captures assistant `reasoning_content` produced by thinking-mode models and replays it transparently on multi-turn requests when the upstream provider requires it. This eliminates the HTTP 400 errors that strict providers raise when a client's conversation history is missing the prior turn's reasoning.
|
|
|
|
## Why This Exists
|
|
|
|
Several thinking-mode providers reject a follow-up turn unless the **previous assistant message includes the original `reasoning_content`**. The upstream returns 400 with messages like:
|
|
|
|
```
|
|
Param Incorrect: The reasoning_content in the thinking mode must be passed back to the API.
|
|
```
|
|
|
|
But typical clients (Cursor, Cline, Roo Code, OpenAI SDK) strip `reasoning_content` from the history they replay. OmniRoute restores it from a server-side cache so the request the upstream sees is consistent. Issue #1628 introduced the hybrid memory/SQLite persistence so the cache survives process restarts.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Turn N (assistant generates):
|
|
→ response contains reasoning_content + tool_calls
|
|
→ cacheReasoningFromAssistantMessage() writes (memory + DB), keyed by every tool_call.id
|
|
→ forward response to client (which may or may not retain reasoning)
|
|
|
|
Turn N+1 (client sends follow-up):
|
|
→ translator detects: requiresReasoningReplay(provider, model) === true
|
|
→ for each assistant message with tool_calls and no reasoning_content:
|
|
lookupReasoning(toolCalls[0].id) → memory → DB
|
|
hit → msg.reasoning_content = cached; recordReplay()
|
|
miss → msg.reasoning_content = "" (legacy fallback for older DeepSeek)
|
|
→ upstream sees consistent history → no 400
|
|
```
|
|
|
|
Capture happens in `open-sse/handlers/chatCore.ts` (two sites, around lines 4093 and 4380). Replay happens in `open-sse/translator/index.ts` after schema coercion but before dispatch.
|
|
|
|
## Storage — Hybrid Memory + SQLite
|
|
|
|
The hot path uses an in-memory `Map` (LRU-by-creation) backed by a SQLite table for crash recovery and dashboard visibility.
|
|
|
|
| Layer | Implementation | Purpose |
|
|
| ------ | ---------------------------------------------- | -------------------------------------- |
|
|
| Memory | `Map` in `open-sse/services/reasoningCache.ts` | Fast lookups, evicts oldest at 2000 |
|
|
| DB | `reasoning_cache` table (`src/lib/db/`) | Persists across restarts, drives stats |
|
|
|
|
Writes go to both. Reads consult memory first, then fall back to DB (DB hits are promoted back into memory). DB failures are non-fatal — the in-memory cache continues to serve the hot path.
|
|
|
|
**Defaults:**
|
|
|
|
- TTL: `2h` (`TTL_MS = 2 * 60 * 60 * 1000`)
|
|
- Max memory entries: `2000` (`MAX_MEMORY_ENTRIES`)
|
|
- Eviction: oldest `createdAt` first
|
|
|
|
## Database Schema
|
|
|
|
Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS reasoning_cache (
|
|
tool_call_id TEXT PRIMARY KEY,
|
|
provider TEXT NOT NULL,
|
|
model TEXT NOT NULL,
|
|
reasoning TEXT NOT NULL,
|
|
char_count INTEGER NOT NULL DEFAULT 0,
|
|
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
expires_at INTEGER NOT NULL
|
|
);
|
|
```
|
|
|
|
Indexes: `expires_at`, `provider`, `model`, `created_at`. `expires_at` is stored as Unix epoch seconds; the SELECT layer normalizes legacy text values via `EXPIRES_AT_EPOCH_SQL`.
|
|
|
|
## Provider / Model Detection
|
|
|
|
Replay is enabled when `requiresReasoningReplay(provider, model)` returns `true`. The function checks two lists in `open-sse/services/reasoningCache.ts`.
|
|
|
|
**Provider IDs (exact match, case-insensitive):**
|
|
|
|
- `deepseek`
|
|
- `opencode-go`
|
|
- `siliconflow`
|
|
- `nebius`
|
|
- `deepinfra`
|
|
- `sambanova`
|
|
- `fireworks`
|
|
- `together`
|
|
- `xiaomi-mimo`
|
|
|
|
**Model regex patterns (case-insensitive):**
|
|
|
|
- `/deepseek-r1/i`
|
|
- `/deepseek-reasoner/i`
|
|
- `/deepseek-chat/i`
|
|
- `/deepseek[-/]?v4[-.]flash/i` and `/deepseek[-/]?v4[-.]pro/i` (V4 Flash / Pro, optional `-free` suffix)
|
|
- `/(deepseek|zen\/deepseek)-v4/i`
|
|
- `/kimi-k2/i`
|
|
- `/qwq/i`
|
|
- `/qwen.*think/i`
|
|
- `/glm.*think/i`
|
|
- `/^mimo[-.]?v\d/i`
|
|
|
|
Adding a new strict provider/model means appending to one of these lists and writing a unit test asserting replay injection. The PR description should cite the exact upstream 400 string that motivated the change.
|
|
|
|
## REST API
|
|
|
|
The cache exposes two endpoints under `src/app/api/cache/reasoning/route.ts`. Both require management authentication (`isAuthenticated` from `@/shared/utils/apiAuth`).
|
|
|
|
| Method | Endpoint | Description |
|
|
| ------ | --------------------------------------------------------- | -------------------------------------------------------- |
|
|
| GET | `/api/cache/reasoning` | Stats + paginated entries |
|
|
| GET | `/api/cache/reasoning?provider=deepseek&model=...&limit=` | Filtered listing (`limit` clamped to `[1, 200]`) |
|
|
| DELETE | `/api/cache/reasoning` | Clear everything (memory + DB) and reset hit/miss counts |
|
|
| DELETE | `/api/cache/reasoning?provider=deepseek` | Clear only entries for one provider |
|
|
| DELETE | `/api/cache/reasoning?toolCallId=call_abc` | Delete a single entry |
|
|
|
|
**GET response shape:**
|
|
|
|
```json
|
|
{
|
|
"stats": {
|
|
"memoryEntries": 12,
|
|
"dbEntries": 47,
|
|
"totalEntries": 47,
|
|
"totalChars": 138291,
|
|
"hits": 84,
|
|
"misses": 6,
|
|
"replays": 81,
|
|
"replayRate": "90.0%",
|
|
"byProvider": { "deepseek": { "entries": 32, "chars": 98412 } },
|
|
"byModel": { "deepseek-reasoner": { "entries": 32, "chars": 98412 } },
|
|
"oldestEntry": "2026-05-13T10:00:00.000Z",
|
|
"newestEntry": "2026-05-13T11:42:11.000Z"
|
|
},
|
|
"entries": [
|
|
{
|
|
"toolCallId": "call_abc",
|
|
"provider": "deepseek",
|
|
"model": "deepseek-reasoner",
|
|
"reasoning": "...",
|
|
"charCount": 3128,
|
|
"createdAt": "...",
|
|
"expiresAt": "..."
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Operational Notes
|
|
|
|
- **Cleanup:** `cleanupReasoningCache()` purges expired memory entries and runs `DELETE FROM reasoning_cache WHERE expires_at <= unixepoch('now')`. Health-check workers call this periodically.
|
|
- **Crash recovery:** After a restart, memory is empty but the DB still holds unexpired entries. The first lookup for a given `tool_call_id` is a DB hit; subsequent lookups are memory hits.
|
|
- **No reasoning, no cache:** `cacheReasoningFromAssistantMessage` returns `0` when the assistant message has no `reasoning_content` / `reasoning` field, so non-thinking responses cost nothing.
|
|
- **Non-strict providers:** When `requiresReasoningReplay` is `false` and the target format is OpenAI, the translator **strips** any `reasoning_content` field from outgoing messages — OpenAI Chat Completions does not accept it.
|
|
|
|
## See Also
|
|
|
|
- [RESILIENCE_GUIDE.md](../architecture/RESILIENCE_GUIDE.md) — circuit breakers, cooldowns, model lockouts
|
|
- [TROUBLESHOOTING.md](../guides/TROUBLESHOOTING.md) — diagnosing upstream 400s
|
|
- Source: `src/lib/db/reasoningCache.ts`, `open-sse/services/reasoningCache.ts`, `open-sse/translator/index.ts`
|
|
- Migration: `src/lib/db/migrations/033_create_reasoning_cache.sql`
|
|
- API route: `src/app/api/cache/reasoning/route.ts`
|
|
- Original issue: #1628
|