wehub-resource-sync
227 lines
11 KiB
Plaintext
227 lines
11 KiB
Plaintext
---
|
|
title: "Recovery boot"
|
|
sidebarTitle: "Recovery boot"
|
|
description: "Recover from cancel-mid-stream, crashes, and OOM kills with full conversational context. The smart default Just Works; the onRecoveryBoot hook is the override path for advanced policies."
|
|
---
|
|
|
|
When a `chat.agent` run dies in the middle of streaming a response — the user cancels, the worker OOMs, or an unhandled exception kills the process — the durable streams hold what was in flight. The next run boots as a continuation, reads both stream tails, and reconstructs a chain that preserves the partial response so any follow-up (`keep going`, `actually do X instead`, a new question) has full context.
|
|
|
|
The behavior is automatic. The `onRecoveryBoot` hook is opt-in for policies that need something different.
|
|
|
|
## The scenario
|
|
|
|
```ts
|
|
// Turn 1 is mid-essay when the user clicks Cancel.
|
|
window.__chat.send("Write me a long essay about espresso");
|
|
// ... assistant has written 3000 characters ...
|
|
window.__chat.stop(); // OR: server-side cancel_run
|
|
|
|
// User decides what they want next.
|
|
window.__chat.send("keep going"); // OR: "what's 7+8?", or anything
|
|
```
|
|
|
|
The cancelled run never wrote `onTurnComplete`. The snapshot is stale or absent. `session.out` has a half-written assistant message. `session.in` has the original user message (the run consumed it but never marked the turn complete) plus the new follow-up.
|
|
|
|
A naive continuation would either re-run the cancelled essay (the user already chose to stop) or drop everything (no context for the follow-up). Recovery boot handles this without either failure mode.
|
|
|
|
## The smart default
|
|
|
|
On a continuation boot, the runtime reads:
|
|
|
|
- **Snapshot** — settled turns persisted by the last successful `onTurnComplete`.
|
|
- **`session.out` tail past the snapshot cursor** — closed assistant turns plus, optionally, a `partialAssistant` (the trailing message whose stream never received a `finish` chunk). `cleanupAbortedParts` has already stripped streaming-in-progress fragments.
|
|
- **`session.in` tail past the last `turn-complete` cursor** — user messages the dead run hadn't acknowledged.
|
|
|
|
If both `partialAssistant` and `inFlightUsers` are non-empty, the runtime splices `[firstInFlightUser, partialAssistant]` onto the chain. The remaining in-flight users dispatch as fresh turns. The model sees:
|
|
|
|
```
|
|
[ ...settledMessages, // chain through the last completed turn
|
|
firstInFlightUser, // the question the dead run was answering
|
|
partialAssistant, // the dead run's incomplete response
|
|
followUpUser ] // the new turn the customer just sent
|
|
```
|
|
|
|
Modern instruction-following models prioritize the latest user message. The follow-up determines the response:
|
|
|
|
| Follow-up | Model behavior |
|
|
|---|---|
|
|
| "keep going" / "continue" / "more" | Continues the partial essay from where it stopped. |
|
|
| "actually, what's 7+8?" | Answers the new question. Prior context doesn't derail it. |
|
|
| "scrap that, do something else" | Abandons the partial work and follows the new direction. |
|
|
|
|
No customer code needed for any of these.
|
|
|
|
## When to register `onRecoveryBoot`
|
|
|
|
The hook fires when recovery state is non-empty (either `partialAssistant` is defined or there's at least one in-flight user). Register it when you need a policy different from "preserve context":
|
|
|
|
- **Drop the partial entirely.** Your UX means "cancel discards the work — start fresh from the follow-up."
|
|
- **Synthesize tool results.** The partial has tool calls in `input-available` state (HITL was mid-call when the run died). Return a chain that has fabricated `output-available` results so the model can continue.
|
|
- **Emit a recovery banner.** Write a `data-chat-recovery` UIMessage chunk via `ctx.writer` so the frontend can render "Recovering interrupted response..." before the model speaks.
|
|
- **Persist recovered state.** Use `beforeBoot` to flush the partial to your own database before the next turn starts.
|
|
|
|
```ts
|
|
import { chat } from "@trigger.dev/sdk/ai";
|
|
|
|
export const myChat = chat.agent({
|
|
id: "my-chat",
|
|
onRecoveryBoot: async ({ partialAssistant, inFlightUsers, writer, cause, previousRunId }) => {
|
|
writer.write({
|
|
type: "data-chat-recovery",
|
|
data: { cause, previousRunId, partialPresent: partialAssistant !== undefined },
|
|
transient: true,
|
|
});
|
|
// Return nothing → fall through to smart default.
|
|
},
|
|
run: async ({ messages, signal }) =>
|
|
streamText({ model, messages, abortSignal: signal }),
|
|
});
|
|
```
|
|
|
|
## Hook reference
|
|
|
|
### Fires when
|
|
|
|
The hook fires once on a continuation boot, AFTER both stream tails have been read, AND only when there's a partial assistant — the mid-stream-died signal:
|
|
|
|
```ts
|
|
const shouldFire = partialAssistant !== undefined;
|
|
```
|
|
|
|
In-flight users alone don't fire the hook. Graceful exits like `chat.requestUpgrade()` and `chat.endRun()` may leave an unacknowledged user on `session.in` (the message that triggered the upgrade, the next message after endRun), but no partial — that's a normal continuation, not recovery. The next message just dispatches as turn 1 on the new run via the normal session.in pump.
|
|
|
|
Skipped scenarios (where the hook does NOT fire):
|
|
|
|
- A clean continuation after `chat.endRun()` with no buffered follow-up.
|
|
- A fresh chat (no continuation, attempt 1).
|
|
- An OOM retry that booted onto a complete snapshot (no partial on the tail).
|
|
- `chat.requestUpgrade()` graceful exit — predecessor ended cleanly before processing, no partial.
|
|
- An agent with [`hydrateMessages`](/ai-chat/lifecycle-hooks#hydratemessages) registered. Customers using `hydrateMessages` own persistence — recovery decisions live in their own DB query.
|
|
|
|
### Event shape
|
|
|
|
```ts
|
|
type RecoveryBootEvent<TUIM extends UIMessage = UIMessage> = {
|
|
ctx: TaskRunContext;
|
|
chatId: string;
|
|
runId: string;
|
|
previousRunId: string;
|
|
cause: "cancelled" | "crashed" | "unknown";
|
|
settledMessages: TUIM[];
|
|
inFlightUsers: TUIM[];
|
|
partialAssistant: TUIM | undefined;
|
|
pendingToolCalls: Array<{
|
|
toolCallId: string;
|
|
toolName: string;
|
|
input: unknown;
|
|
partIndex: number;
|
|
}>;
|
|
writer: ChatWriter;
|
|
};
|
|
```
|
|
|
|
<Note>
|
|
`cause` is currently always `"unknown"` — the run engine doesn't yet plumb the
|
|
real reason into the continuation payload. The enum is forward-looking; don't
|
|
branch behavior on it for now.
|
|
</Note>
|
|
|
|
### Return shape
|
|
|
|
Every field is optional. Returning `undefined` (or nothing) accepts the smart default for every field.
|
|
|
|
```ts
|
|
type RecoveryBootResult<TUIM extends UIMessage = UIMessage> = {
|
|
chain?: TUIM[];
|
|
recoveredTurns?: TUIM[];
|
|
beforeBoot?: () => Promise<void>;
|
|
};
|
|
```
|
|
|
|
- **`chain`** — replaces the seed chain. Defaults to `[...settledMessages, firstInFlightUser, partialAssistant]` when both partial and in-flight users exist, otherwise `settledMessages` alone.
|
|
- **`recoveredTurns`** — user messages to dispatch as fresh turns after the chain is restored. Defaults to `inFlightUsers.slice(1)` when the smart default consumed the first user, otherwise `inFlightUsers`.
|
|
- **`beforeBoot`** — runs after the writer flushes and before the first recovered turn fires. Use for blocking persistence (write the partial to your DB so a later turn can reference it). Errors bubble — wrap your own try/catch if you want to soft-fail.
|
|
|
|
## Examples
|
|
|
|
### Drop the partial — strict "cancel means discard"
|
|
|
|
The customer's UX treats cancel as "throw the work away":
|
|
|
|
```ts
|
|
onRecoveryBoot: async ({ inFlightUsers, partialAssistant }) => {
|
|
if (!partialAssistant) return; // No partial → nothing to drop
|
|
return {
|
|
chain: undefined, // Use settledMessages, don't splice partial
|
|
recoveredTurns: inFlightUsers.slice(1) // Still skip the first user (the dead run was answering it)
|
|
};
|
|
}
|
|
```
|
|
|
|
### Synthesize tool results for a mid-call interruption
|
|
|
|
The dead run was processing a tool call when it died. The partial has tool parts in `input-available` state with no `output-available`. Synthesize a result so the model can keep going:
|
|
|
|
```ts
|
|
onRecoveryBoot: async ({ partialAssistant, pendingToolCalls, settledMessages, inFlightUsers }) => {
|
|
if (pendingToolCalls.length === 0) return;
|
|
|
|
// Rebuild the partial with synthetic outputs for any input-available tool call.
|
|
const repaired = {
|
|
...partialAssistant!,
|
|
parts: partialAssistant!.parts!.map((part, i) => {
|
|
const pending = pendingToolCalls.find(p => p.partIndex === i);
|
|
if (!pending) return part;
|
|
return {
|
|
...part,
|
|
state: "output-available" as const,
|
|
output: { interrupted: true, reason: "previous run was cancelled" },
|
|
};
|
|
}),
|
|
};
|
|
|
|
return {
|
|
chain: [...settledMessages, inFlightUsers[0]!, repaired],
|
|
recoveredTurns: inFlightUsers.slice(1),
|
|
};
|
|
}
|
|
```
|
|
|
|
### Persist the partial before the next turn fires
|
|
|
|
```ts
|
|
onRecoveryBoot: async ({ chatId, partialAssistant }) => {
|
|
return {
|
|
beforeBoot: async () => {
|
|
if (partialAssistant) {
|
|
await db.partial.create({
|
|
data: { chatId, partialJson: JSON.stringify(partialAssistant) },
|
|
});
|
|
}
|
|
},
|
|
};
|
|
}
|
|
```
|
|
|
|
## Interaction with other features
|
|
|
|
### `hydrateMessages`
|
|
|
|
If your agent registers [`hydrateMessages`](/ai-chat/lifecycle-hooks#hydratemessages), the runtime skips snapshot read, `session.out` replay, `session.in` replay, AND `onRecoveryBoot`. Your DB is the source of truth — recovery decisions live in your own query. To detect a cancel-recovery scenario yourself, persist a `runState: "in-progress"` flag in `onTurnStart` and check for it in `hydrateMessages`.
|
|
|
|
### `chat.requestUpgrade()`
|
|
|
|
[`chat.requestUpgrade()`](/ai-chat/patterns/version-upgrades) is a graceful exit — the old run doesn't crash, it returns cleanly. The new continuation run boots with a clean `session.out` tail (`partialAssistant` is undefined) and the upgrade-trigger message on `session.in` (one in-flight user). The smart default doesn't splice (it requires both partial AND in-flight users), so the chain is just `settledMessages` and the in-flight user dispatches as a fresh turn. `onRecoveryBoot` still fires (there's an in-flight user) — use it to emit an "upgraded" signal to the UI if you want.
|
|
|
|
### Hooks throwing
|
|
|
|
If the body of `onRecoveryBoot` throws (or rejects), the runtime logs a warning and falls back to the smart default — the run does not fail. Wrap your own try/catch if you want stricter handling.
|
|
|
|
`beforeBoot` is the exception: it's the contract you opted into for blocking persistence, so errors thrown there **bubble** and fail the run rather than dispatch recovered turns against half-persisted state. Wrap it yourself if you want to soft-fail.
|
|
|
|
## See also
|
|
|
|
- [OOM resilience](/ai-chat/patterns/oom-resilience) — `oomMachine` opt-in for automatic memory-driven recovery; uses the same recovery boot path.
|
|
- [Persistence and replay](/ai-chat/patterns/persistence-and-replay) — the snapshot + dual-tail replay model that recovery boot sits on top of.
|
|
- [Lifecycle hooks](/ai-chat/lifecycle-hooks) — where `onRecoveryBoot` sits in the broader hook taxonomy.
|