chore: import upstream snapshot with attribution
Deploy (to testing) and Test Playground Preview Worker / Deploy Playground Preview Worker (testing) (push) Has been skipped
Deploy Workers Shared Staging / Deploy Workers Shared Staging (push) Failing after 0s
Prerelease / build (push) Has been skipped
Handle Changesets / Handle Changesets (push) Has been cancelled
Semgrep OSS scan / semgrep-oss (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:30:11 +08:00
commit 70bf21e064
5213 changed files with 731895 additions and 0 deletions
+296
View File
@@ -0,0 +1,296 @@
# @cloudflare/workflows-shared
## 0.12.0
### Minor Changes
- [#14465](https://github.com/cloudflare/workers-sdk/pull/14465) [`2fedb1f`](https://github.com/cloudflare/workers-sdk/commit/2fedb1fc811efb3f7544c569e57383cabd4f14f8) Thanks [@vaishnav-mk](https://github.com/vaishnav-mk)! - Add rollback support when terminating Workflow instances
`WorkflowInstance.terminate({ rollback: true })` now runs registered rollback handlers before marking a local Workflow instance as terminated. Wrangler also supports this via `wrangler workflows instances terminate --rollback`, including local mode.
The rollback option is only sent for terminate operations and is rejected by the Local Explorer API for pause, resume, and restart actions.
## 0.11.2
### Patch Changes
- [#14318](https://github.com/cloudflare/workers-sdk/pull/14318) [`f32e9c1`](https://github.com/cloudflare/workers-sdk/commit/f32e9c1fdbf8ab7c0e68afa613ec61e367be04cb) Thanks [@vaishnav-mk](https://github.com/vaishnav-mk)! - Add step context to Workflows rollback handlers
Rollback handlers now receive the original step context under `ctx`, making `ctx.step.name`, `ctx.step.count`, `ctx.attempt`, and the resolved step `config` available during rollback. The legacy `stepName` field remains available and is equivalent to `${ctx.step.name}-${ctx.step.count}`.
`rollbackConfig` is now limited to retry and timeout settings, matching the behavior supported by rollback handlers.
- [#14314](https://github.com/cloudflare/workers-sdk/pull/14314) [`5c3bb11`](https://github.com/cloudflare/workers-sdk/commit/5c3bb118a99da70c5c1efb07df37f685e7044ba6) Thanks [@harryzcy](https://github.com/harryzcy)! - Bump esbuild to 0.28.1
This update includes several bug fixes from esbuild versions 0.27.3 through 0.28.1. See the [esbuild changelog](https://github.com/evanw/esbuild/blob/v0.28.1/CHANGELOG.md) for details.
## 0.11.1
### Patch Changes
- [#14134](https://github.com/cloudflare/workers-sdk/pull/14134) [`262dfc2`](https://github.com/cloudflare/workers-sdk/commit/262dfc2b32531165f94ba87c70ce75fcb1490b61) Thanks [@matingathani](https://github.com/matingathani)! - Fix `wrangler dev` Workflows crashing with `SQLITE_TOOBIG` when a step returns a large `Uint8Array`
`JSON.stringify` encodes each byte of a `Uint8Array` as a separate numeric key
(`{"0":1,"1":2,...}`), producing a string ~10× larger than the array's byte
length. A 200 KB `Uint8Array` became a ~2 MB JSON string that exceeded SQLite's
blob limit, crashing the Workflow step. The same bytes returned as an
`ArrayBuffer` succeeded because `JSON.stringify(ArrayBuffer)``{}`.
The step log metadata (used by the local Workflows explorer) now stores a
human-readable description for `TypedArray` and `ArrayBuffer` outputs
(`[Uint8Array(200000 bytes)]`) instead of attempting to JSON-serialise the raw
bytes. The actual step value is unaffected — it is preserved in Durable Object
key-value storage via structured clone for replay by subsequent steps.
## 0.11.0
### Minor Changes
- [#13983](https://github.com/cloudflare/workers-sdk/pull/13983) [`9803586`](https://github.com/cloudflare/workers-sdk/commit/98035862e1e303ca1f380d8d2694ad3d4659e3be) Thanks [@vaishnav-mk](https://github.com/vaishnav-mk)! - Add rollback support for local Workflows development
Workflow steps can now register a compensation callback with trailing rollback options: `step.do(name, fn, { rollback })` and `step.do(name, config, fn, { rollback, rollbackConfig })`. When the workflow fails, the local engine runs every registered rollback in reverse step-start order (LIFO), giving steps the opportunity to undo their side effects.
Each rollback executes through an internal rollback-scoped `Context.do`, so it inherits the existing retry / timeout / attempt-tracking machinery. `rollbackConfig` lets users override the per-rollback config.
Note: the public rollback option type lands with workerd's `workflows_step_rollback` compat flag. Until that ships, the trailing rollback options only flow through when called through the StepPromise wrapper from a worker that has the flag enabled.
## 0.10.0
### Minor Changes
- [#13565](https://github.com/cloudflare/workers-sdk/pull/13565) [`b04eedf`](https://github.com/cloudflare/workers-sdk/commit/b04eedfcdc713d04cbb4f1722ebe056c9dc4cb6e) Thanks [@vaishnav-mk](https://github.com/vaishnav-mk)! - Add restart from step support for local Workflows development
Workflow instances can now be restarted from a specific step in local development. When restarting from a step, all earlier steps preserve their cached results and replay instantly, while the target step and everything after it re-execute.
The `WorkflowInstance.restart()` method now accepts an optional `{ from: { name, count?, type? } }` parameter to specify which step to restart from.
## 0.9.1
### Patch Changes
- [#13560](https://github.com/cloudflare/workers-sdk/pull/13560) [`7567ef7`](https://github.com/cloudflare/workers-sdk/commit/7567ef703f1bf157ef29e6d19dd0dd9f1ff8771f) Thanks [@vaishnav-mk](https://github.com/vaishnav-mk)! - Preserve NonRetryableError message and name when the `workflows_preserve_non_retryable_error_message` compatibility flag is enabled, instead of replacing it with a generic error message.
## 0.9.0
### Minor Changes
- [#13351](https://github.com/cloudflare/workers-sdk/pull/13351) [`3788983`](https://github.com/cloudflare/workers-sdk/commit/3788983378793781829798bb10ae710fb452f746) Thanks [@pombosilva](https://github.com/pombosilva)! - Add `step` and `config` properties to the workflow step context
The callback passed to `step.do()` now receives `ctx.step` (with `name` and `count`) and `ctx.config` (the fully resolved step configuration with defaults merged in), in addition to the existing `ctx.attempt`.
## 0.8.0
### Minor Changes
- [#13145](https://github.com/cloudflare/workers-sdk/pull/13145) [`5b60405`](https://github.com/cloudflare/workers-sdk/commit/5b60405954a2866a67920f5a7a48748861f58a60) Thanks [@Caio-Nogueira](https://github.com/Caio-Nogueira)! - Add support for ReadableStream on workflow steps. This allows users to overcome the 1MB limit per step output.
`ReadableStream<Uint8Array>` is already serializable on the workers platform. This feature makes it native to workflows as well by persisting each chunk and replaying it if needed
## 0.7.2
### Patch Changes
- [#13086](https://github.com/cloudflare/workers-sdk/pull/13086) [`d4c6158`](https://github.com/cloudflare/workers-sdk/commit/d4c61587094a2a2ceee35acfb3619c95e0a993fe) Thanks [@pombosilva](https://github.com/pombosilva)! - Add Workflows support to the local explorer UI.
The local explorer (`/cdn-cgi/explorer/`) now includes a full Workflows dashboard for viewing and managing workflow instances during local development.
UI features:
- Workflow instance list with status badges, creation time, action buttons, and pagination
- Status summary bar with instance counts per status
- Status filter dropdown and search
- Instance detail page with step history, params/output cards, error display, and expandable step details
- Create instance dialog with optional ID and JSON params
## 0.7.1
### Patch Changes
- [#12985](https://github.com/cloudflare/workers-sdk/pull/12985) [`17a57e6`](https://github.com/cloudflare/workers-sdk/commit/17a57e641798dca0b298bd91dcdcef6be30d38b6) Thanks [@pombosilva](https://github.com/pombosilva)! - Fix `waitForEvent` delivering events to stale waiters after timeout.
When a `step.waitForEvent()` call timed out, its resolver was not removed from the workflow's internal waiters map. This meant the next `step.waitForEvent()` for the same event type would have its incoming event consumed by the dead resolver instead of the live one, causing the workflow to hang indefinitely.
## 0.7.0
### Minor Changes
- [#12881](https://github.com/cloudflare/workers-sdk/pull/12881) [`8729f3d`](https://github.com/cloudflare/workers-sdk/commit/8729f3d0954c5325a0a28da6fa87129411819787) Thanks [@pombosilva](https://github.com/pombosilva)! - Workflow instances now support pause, resume, restart, and terminate in local dev.
```js
const instance = await env.MY_WORKFLOW.create({
id: "my-instance",
});
await instance.pause(); // pauses after the current step completes
await instance.resume(); // resumes from where it left off
await instance.restart(); // restarts the workflow from the beginning
await instance.terminate(); // terminates the workflow immediately
```
## 0.6.0
### Minor Changes
- [#11970](https://github.com/cloudflare/workers-sdk/pull/11970) [`f235827`](https://github.com/cloudflare/workers-sdk/commit/f235827c10c36996c89f50af76aac48326f42b0c) Thanks [@pombosilva](https://github.com/pombosilva)! - Adds step context with attempt count to step.do() callbacks.
Workflow step callbacks now receive a context object containing the current attempt number (1-indexed).
This allows developers to access which retry attempt is currently executing.
Example:
```ts
await step.do("my-step", async (ctx) => {
// ctx.attempt is 1 on first try, 2 on first retry, etc.
console.log(`Attempt ${ctx.attempt}`);
});
```
## 0.5.0
### Minor Changes
- [#12622](https://github.com/cloudflare/workers-sdk/pull/12622) [`bf9cb3d`](https://github.com/cloudflare/workers-sdk/commit/bf9cb3d32d4710dbefd7d3c412aefe1558ecd57e) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Add configurable step limits for Workflows
You can now set a maximum number of steps for a Workflow instance via the `limits.steps` configuration in your Wrangler config. When a Workflow instance exceeds this limit, it will fail with an error indicating the limit was reached.
```jsonc
// wrangler.jsonc
{
"workflows": [
{
"binding": "MY_WORKFLOW",
"name": "my-workflow",
"class_name": "MyWorkflow",
"limits": {
"steps": 5000
}
}
]
}
```
The `steps` value must be an integer between 1 and 25,000. If not specified, the default limit of 10,000 steps is used. Step limits are also enforced in local development via `wrangler dev`.
## 0.4.0
### Minor Changes
- [#11648](https://github.com/cloudflare/workers-sdk/pull/11648) [`eac5cf7`](https://github.com/cloudflare/workers-sdk/commit/eac5cf74db6d1b0865f5dc3a744ff28e695d53ca) Thanks [@pombosilva](https://github.com/pombosilva)! - Add Workflows test handlers in vitest-pool-workers to get the Workflow instance output and error:
- `getOutput()`: Returns the output of the successfully completed Workflow instance.
- `getError()`: Returns the error information of the errored Workflow instance.
Example:
```ts
// First wait for the workflow instance to complete:
await expect(
instance.waitForStatus({ status: "complete" })
).resolves.not.toThrow();
// Then, get its output
const output = await instance.getOutput();
// Or for errored workflow instances, get their error:
await expect(
instance.waitForStatus({ status: "errored" })
).resolves.not.toThrow();
const error = await instance.getError();
```
## 0.3.9
### Patch Changes
- [#11448](https://github.com/cloudflare/workers-sdk/pull/11448) [`2b4813b`](https://github.com/cloudflare/workers-sdk/commit/2b4813b18076817bb739491246313c32b403651f) Thanks [@edmundhung](https://github.com/edmundhung)! - Builds package with esbuild `v0.27.0`
## 0.3.8
### Patch Changes
- [#10919](https://github.com/cloudflare/workers-sdk/pull/10919) [`ca6c010`](https://github.com/cloudflare/workers-sdk/commit/ca6c01017ccc39671e8724a6b9a5aa37a5e07e57) Thanks [@Caio-Nogueira](https://github.com/Caio-Nogueira)! - migrate workflow to use a wrapped binding
## 0.3.7
### Patch Changes
- [#10813](https://github.com/cloudflare/workers-sdk/pull/10813) [`545afe5`](https://github.com/cloudflare/workers-sdk/commit/545afe504ab6c3c44373fc47d58a2641aadb0d2d) Thanks [@pombosilva](https://github.com/pombosilva)! - Workflows are now created if the Request gets redirected after creation
## 0.3.6
### Patch Changes
- [#10785](https://github.com/cloudflare/workers-sdk/pull/10785) [`d09cab3`](https://github.com/cloudflare/workers-sdk/commit/d09cab3b86149a67c471401daa64ff631cfb4e49) Thanks [@pombosilva](https://github.com/pombosilva)! - Workflows names and instance IDs are now properly validated with production limits.
## 0.3.5
### Patch Changes
- [#10219](https://github.com/cloudflare/workers-sdk/pull/10219) [`28494f4`](https://github.com/cloudflare/workers-sdk/commit/28494f413bba3c509c56762b9260edd0ffef4f28) Thanks [@dario-piotrowicz](https://github.com/dario-piotrowicz)! - fix `NonRetryableError` thrown with an empty error message not stopping workflow retries locally
## 0.3.4
### Patch Changes
- [#9367](https://github.com/cloudflare/workers-sdk/pull/9367) [`2e12e6e`](https://github.com/cloudflare/workers-sdk/commit/2e12e6e6f37d35805fcf6b0f8916cb4136543837) Thanks [@Caio-Nogueira](https://github.com/Caio-Nogueira)! - Fix instance hydration after abort. Some use cases were causing instances to not be properly rehydrated after server shutdown, which would cause the instance to be lost.
## 0.3.3
### Patch Changes
- [#9033](https://github.com/cloudflare/workers-sdk/pull/9033) [`2c50115`](https://github.com/cloudflare/workers-sdk/commit/2c501151d3d1a563681cdb300a298b83862b60e2) Thanks [@dario-piotrowicz](https://github.com/dario-piotrowicz)! - chore: convert wrangler.toml files into wrangler.jsonc ones
## 0.3.2
### Patch Changes
- [#8775](https://github.com/cloudflare/workers-sdk/pull/8775) [`ec7e621`](https://github.com/cloudflare/workers-sdk/commit/ec7e6212199272f9811a30a84922823c82d7d650) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Add `waitForEvent` and `sendEvent` support for local dev
## 0.3.1
### Patch Changes
- [#8606](https://github.com/cloudflare/workers-sdk/pull/8606) [`18edbc2`](https://github.com/cloudflare/workers-sdk/commit/18edbc2a6b02e6b7cd0a027e5b90ff043da6ee79) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Implement local dev for createBatch
## 0.3.0
### Minor Changes
- [#7334](https://github.com/cloudflare/workers-sdk/pull/7334) [`869ec7b`](https://github.com/cloudflare/workers-sdk/commit/869ec7b916487ec43b958a27bdfea13588c5685f) Thanks [@penalosa](https://github.com/penalosa)! - Packages in Workers SDK now support the versions of Node that Node itself supports (Current, Active, Maintenance). Currently, that includes Node v18, v20, and v22.
## 0.2.3
### Patch Changes
- [#8123](https://github.com/cloudflare/workers-sdk/pull/8123) [`7f565c5`](https://github.com/cloudflare/workers-sdk/commit/7f565c5c8844cd8c42137ed653e0665fa54950d1) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Make NonRetryableError work and cache errored steps
## 0.2.2
### Patch Changes
- [#7575](https://github.com/cloudflare/workers-sdk/pull/7575) [`7216835`](https://github.com/cloudflare/workers-sdk/commit/7216835bf7489804905751c6b52e75a8945e7974) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Make `Instance.status()` return type the same as production
## 0.2.1
### Patch Changes
- [#7520](https://github.com/cloudflare/workers-sdk/pull/7520) [`805ad2b`](https://github.com/cloudflare/workers-sdk/commit/805ad2b3959210b0d838daf789f580f329e1d7dd) Thanks [@bruxodasilva](https://github.com/bruxodasilva)! - Fixed a bug in local development where fetching a Workflow instance by ID would return a Workflow status, even if that instance did not exist. This only impacted the `get()` method on the Worker bindings.
## 0.2.0
### Minor Changes
- [#7286](https://github.com/cloudflare/workers-sdk/pull/7286) [`563439b`](https://github.com/cloudflare/workers-sdk/commit/563439bd02c450921b28d721d36be5a70897690d) Thanks [@LuisDuarte1](https://github.com/LuisDuarte1)! - Add proper engine persistance in .wrangler and fix multiple workflows in miniflare
## 0.1.2
### Patch Changes
- [#7225](https://github.com/cloudflare/workers-sdk/pull/7225) [`bb17205`](https://github.com/cloudflare/workers-sdk/commit/bb17205f1cc357cabc857ab5cad61b6a4f3b8b93) Thanks [@bruxodasilva](https://github.com/bruxodasilva)! - - Fix workflows binding to create a workflow without arguments
- Fix workflows instance.id not working the same way in wrangler local dev as it does in production
## 0.1.1
### Patch Changes
- [#7045](https://github.com/cloudflare/workers-sdk/pull/7045) [`5ef6231`](https://github.com/cloudflare/workers-sdk/commit/5ef6231a5cefbaaef123e6e8ee899fb81fc69e3e) Thanks [@RamIdeas](https://github.com/RamIdeas)! - Add preliminary support for Workflows in wrangler dev
+3
View File
@@ -0,0 +1,3 @@
# `@cloudflare/workflows-shared`
This package powers the local development experience for [Cloudflare Workflows](https://developers.cloudflare.com/workflows/) in workers-sdk and Wrangler.
+59
View File
@@ -0,0 +1,59 @@
{
"name": "@cloudflare/workflows-shared",
"version": "0.12.0",
"private": true,
"description": "Package that is used at Cloudflare to power some internal features of Cloudflare Workflows.",
"keywords": [
"cloudflare",
"cloudflare workflows",
"workflows"
],
"homepage": "https://github.com/cloudflare/workers-sdk/tree/main/packages/workflows-shared#readme",
"bugs": {
"url": "https://github.com/cloudflare/workers-sdk/issues"
},
"license": "MIT OR Apache-2.0",
"author": "wrangler@cloudflare.com",
"repository": {
"type": "git",
"url": "https://github.com/cloudflare/workers-sdk.git",
"directory": "packages/workflows-shared"
},
"files": [
"dist"
],
"types": "./dist",
"scripts": {
"build": "esbuild ./src/local-binding-worker.ts --format=esm --bundle --outfile=dist/local-binding-worker.mjs --sourcemap=external --external:cloudflare:*",
"check:type": "tsc",
"clean": "node -r esbuild-register ../../tools/clean/clean.ts dist",
"deploy": "echo 'no deploy'",
"dev": "pnpm run bundle:local-binding --watch",
"test:ci": "vitest run",
"test:watch": "vitest"
},
"dependencies": {
"heap-js": "^2.5.0",
"itty-time": "^2.0.2",
"mime": "^3.0.0",
"zod": "^3.22.3"
},
"devDependencies": {
"@cloudflare/vitest-pool-workers": "catalog:default",
"@cloudflare/workers-tsconfig": "workspace:*",
"@cloudflare/workers-types": "catalog:default",
"@types/mime": "^3.0.4",
"esbuild": "catalog:default",
"typescript": "catalog:default",
"vitest": "catalog:default"
},
"engines": {
"node": ">=22.0.0"
},
"volta": {
"extends": "../../package.json"
},
"workers-sdk": {
"prerelease": true
}
}
+464
View File
@@ -0,0 +1,464 @@
import { RpcTarget, WorkerEntrypoint } from "cloudflare:workers";
import { InstanceEvent, instanceStatusName } from "./instance";
import {
isUserTriggeredPause,
isUserTriggeredRestart,
isUserTriggeredTerminate,
WorkflowError,
} from "./lib/errors";
import { isValidWorkflowInstanceId } from "./lib/validators";
import type {
DatabaseInstance,
DatabaseVersion,
DatabaseWorkflow,
Engine,
EngineLogs,
} from "./engine";
import type { InstanceStatus as EngineInstanceStatus } from "./instance";
import type {
WorkflowInstanceModifier,
WorkflowIntrospectionOperation,
WorkflowIntrospectionStreamResult,
} from "./types";
type Env = {
ENGINE: DurableObjectNamespace<Engine>;
BINDING_NAME: string;
WORKFLOW_NAME: string;
};
type WorkflowIntrospectionSession = {
id: string;
operations: WorkflowIntrospectionOperation[];
instanceIds: string[];
};
// workerd may construct a fresh WorkflowBinding object for each RPC call. Store
// sessions at module scope so start/modify/get/dispose calls, and later
// WorkflowBinding.create() calls, all see the same active Workflow session.
const workflowIntrospectionSessions = new Map<
string,
WorkflowIntrospectionSession
>();
function getWorkflowIntrospectionSession(
workflowName: string,
sessionId: string
): WorkflowIntrospectionSession {
const session = workflowIntrospectionSessions.get(workflowName);
if (session?.id !== sessionId) {
throw new Error(
`Workflow ${JSON.stringify(workflowName)} does not have an active introspection session for this introspector.`
);
}
return session;
}
function isWorkflowIntrospectionStreamResult(
value: unknown
): value is WorkflowIntrospectionStreamResult {
return (
value !== null &&
typeof value === "object" &&
"__workflowIntrospectionStreamResult" in value &&
"chunks" in value &&
value.__workflowIntrospectionStreamResult === true &&
Array.isArray(value.chunks)
);
}
function createWorkflowIntrospectionReadableStream(
result: WorkflowIntrospectionStreamResult
): ReadableStream<Uint8Array> {
return new ReadableStream<Uint8Array>({
start(controller) {
for (const chunk of result.chunks) {
controller.enqueue(chunk.slice());
}
controller.close();
},
});
}
async function applyWorkflowIntrospectionOperation(
modifier: WorkflowInstanceModifier,
operation: WorkflowIntrospectionOperation
) {
switch (operation.type) {
case "disableSleeps":
await modifier.disableSleeps(operation.steps);
break;
case "disableRetryDelays":
await modifier.disableRetryDelays(operation.steps);
break;
case "mockStepResult":
await modifier.mockStepResult(
operation.step,
isWorkflowIntrospectionStreamResult(operation.stepResult)
? createWorkflowIntrospectionReadableStream(operation.stepResult)
: operation.stepResult
);
break;
case "mockStepError": {
const error = new Error(operation.error.message);
error.name = operation.error.name;
await modifier.mockStepError(operation.step, error, operation.times);
break;
}
case "forceStepTimeout":
await modifier.forceStepTimeout(operation.step, operation.times);
break;
case "mockEvent":
await modifier.mockEvent(operation.event);
break;
case "forceEventTimeout":
await modifier.forceEventTimeout(operation.step);
break;
}
}
// TODO(vaish): import from @cloudflare/workers-types once restart options are published
export interface RestartFromStep {
name: string;
count?: number;
type?: "do" | "sleep" | "waitForEvent";
}
export interface WorkflowInstanceTerminateOptions {
/**
* If true, run registered rollback handlers before terminating the instance.
*/
rollback?: boolean;
}
export interface WorkflowInstanceRestartOptions {
from?: RestartFromStep;
}
// this.env.WORKFLOW is WorkflowBinding
export class WorkflowBinding extends WorkerEntrypoint<Env> {
constructor(ctx: ExecutionContext, env: Env) {
super(ctx, env);
}
public async create({
id = crypto.randomUUID(),
params = {},
}: WorkflowInstanceCreateOptions = {}): Promise<{
id: string;
}> {
if (!isValidWorkflowInstanceId(id)) {
throw new WorkflowError("Workflow instance has invalid id");
}
const stubId = this.env.ENGINE.idFromName(id);
const stub = this.env.ENGINE.get(stubId);
const introspectionSession = workflowIntrospectionSessions.get(
this.env.WORKFLOW_NAME
);
if (introspectionSession !== undefined) {
const modifier = stub.getInstanceModifier();
introspectionSession.instanceIds.push(id);
for (const operation of introspectionSession.operations) {
await applyWorkflowIntrospectionOperation(modifier, operation);
}
}
const now = new Date().toISOString();
const initPromise = stub
.init(
0, // accountId: number,
{} as DatabaseWorkflow, // workflow: DatabaseWorkflow,
{} as DatabaseVersion, // version: DatabaseVersion,
{
id,
created_on: now,
modified_on: now,
workflow_id: "",
version_id: "",
status: 0, // InstanceStatus.Queued
started_on: now,
ended_on: null,
} satisfies DatabaseInstance,
{
timestamp: new Date(),
payload: params as Readonly<typeof params>,
instanceId: id,
workflowName: this.env.WORKFLOW_NAME,
}
)
.then((val) => {
if (val !== undefined) {
val[Symbol.dispose]();
}
})
.catch(() => {
// Suppress all rejections: create() should queue and
// return immediately
});
this.ctx.waitUntil(initPromise);
return {
id,
};
}
public async get(id: string): Promise<WorkflowInstance> {
const stubId = this.env.ENGINE.idFromName(id);
const stub = this.env.ENGINE.get(stubId);
// Pass a getter function so WorkflowHandle can get a fresh stub after abort
const getStub = () => this.env.ENGINE.get(this.env.ENGINE.idFromName(id));
const handle = new WorkflowHandle(id, stub, getStub);
try {
await handle.status();
} catch {
throw new Error("instance.not_found");
}
return handle;
}
public async createBatch(
batch: WorkflowInstanceCreateOptions<unknown>[]
): Promise<{ id: string }[]> {
if (batch.length === 0) {
throw new Error(
"WorkflowError: batchCreate should have at least 1 instance"
);
}
return await Promise.all(
batch.map(async (val) => {
const res = await this.create(val);
return res;
})
);
}
public async unsafeGetBindingName(): Promise<string> {
// async because of rpc
return this.env.BINDING_NAME;
}
public async unsafeStartIntrospection(): Promise<string> {
if (workflowIntrospectionSessions.has(this.env.WORKFLOW_NAME)) {
throw new Error(
`Workflow ${JSON.stringify(this.env.WORKFLOW_NAME)} already has an active introspection session for binding ${JSON.stringify(this.env.BINDING_NAME)}.`
);
}
const sessionId = crypto.randomUUID();
workflowIntrospectionSessions.set(this.env.WORKFLOW_NAME, {
id: sessionId,
operations: [],
instanceIds: [],
});
return sessionId;
}
public async unsafeStopIntrospection(sessionId: string): Promise<void> {
const session = workflowIntrospectionSessions.get(this.env.WORKFLOW_NAME);
if (session?.id === sessionId) {
workflowIntrospectionSessions.delete(this.env.WORKFLOW_NAME);
}
}
public async unsafeSetIntrospectionOperations(
sessionId: string,
operations: WorkflowIntrospectionOperation[]
): Promise<void> {
const session = getWorkflowIntrospectionSession(
this.env.WORKFLOW_NAME,
sessionId
);
session.operations = operations;
}
public async unsafeGetIntrospectionInstances(
sessionId: string
): Promise<string[]> {
return getWorkflowIntrospectionSession(this.env.WORKFLOW_NAME, sessionId)
.instanceIds;
}
public async unsafeGetInstanceModifier(instanceId: string): Promise<unknown> {
// async because of rpc
const stubId = this.env.ENGINE.idFromName(instanceId);
const stub = this.env.ENGINE.get(stubId);
const instanceModifier = stub.getInstanceModifier();
return instanceModifier;
}
public async unsafeWaitForStepResult(
instanceId: string,
name: string,
index?: number
): Promise<unknown> {
const stubId = this.env.ENGINE.idFromName(instanceId);
const stub = this.env.ENGINE.get(stubId);
return await stub.waitForStepResult(name, index);
}
public async unsafeAbort(instanceId: string, reason?: string): Promise<void> {
const stubId = this.env.ENGINE.idFromName(instanceId);
const stub = this.env.ENGINE.get(stubId);
try {
await stub.unsafeAbort(reason);
} catch {
// do nothing because we want to dispose this instance
}
}
public async unsafeWaitForStatus(
instanceId: string,
status: string
): Promise<void> {
const stubId = this.env.ENGINE.idFromName(instanceId);
const stub = this.env.ENGINE.get(stubId);
return await stub.waitForStatus(status);
}
public async unsafeGetOutputOrError(
instanceId: string,
isOutput: boolean
): Promise<unknown> {
const stubId = this.env.ENGINE.idFromName(instanceId);
const stub = this.env.ENGINE.get(stubId);
return await stub.getOutputOrError(isOutput);
}
}
export class WorkflowHandle extends RpcTarget implements WorkflowInstance {
private stub: DurableObjectStub<Engine>;
constructor(
public id: string,
stub: DurableObjectStub<Engine>,
private getStub: () => DurableObjectStub<Engine>
) {
super();
this.stub = stub;
}
public async pause(): Promise<void> {
try {
await this.stub.changeInstanceStatus("pause");
} catch (e) {
// pause causes instance abortion
if (!isUserTriggeredPause(e)) {
throw e;
}
}
}
public async resume(): Promise<void> {
await this.stub.changeInstanceStatus("resume");
}
public async terminate(
options?: WorkflowInstanceTerminateOptions
): Promise<void> {
try {
await this.stub.changeInstanceStatus("terminate", undefined, options);
} catch (e) {
// terminate causes instance abortion
if (!isUserTriggeredTerminate(e)) {
throw e;
}
}
}
public async restart(
options?: WorkflowInstanceRestartOptions
): Promise<void> {
try {
await this.stub.changeInstanceStatus("restart", options?.from);
} catch (e) {
// restart causes instance abortion
if (!isUserTriggeredRestart(e)) {
throw e;
}
}
// trigger restart flow after abortion
this.stub = this.getStub();
await this.stub.attemptRestart();
}
public async status(): Promise<
InstanceStatus & { __LOCAL_DEV_STEP_OUTPUTS: unknown[] }
> {
// Both getStatus() and readLogs() must use the same fresh stub.
// After pause/restart/terminate aborts the DO, the stub goes stale
const fetchStatusAndLogs = async () => {
const status = await this.stub.getStatus();
// NOTE(lduarte): for some reason, sync functions over RPC are typed as never instead of Promise<EngineLogs>
const logs = await (this.stub.readLogs() as unknown as Promise<
EngineLogs & Disposable
>);
return { status, logs };
};
let result: {
status: EngineInstanceStatus;
logs: EngineLogs & Disposable;
};
try {
result = await fetchStatusAndLogs();
} catch {
this.stub = this.getStub();
result = await fetchStatusAndLogs();
}
// Dispose the RPC handle when the method scope exits
using logs = result.logs;
const filteredLogs = logs.logs.filter(
(log) =>
log.event === InstanceEvent.STEP_SUCCESS ||
log.event === InstanceEvent.WAIT_COMPLETE
);
const stepOutputs = filteredLogs.map((log) =>
log.event === InstanceEvent.STEP_SUCCESS
? log.metadata.result
: log.metadata.payload
);
const workflowOutput =
logs.logs.find((log) => log.event === InstanceEvent.WORKFLOW_SUCCESS)
?.metadata.result ?? null;
const workflowError = logs.logs.find(
(log) => log.event === InstanceEvent.WORKFLOW_FAILURE
)?.metadata.error;
return {
status: instanceStatusName(result.status),
__LOCAL_DEV_STEP_OUTPUTS: stepOutputs,
output: workflowOutput,
error: workflowError,
};
}
public async sendEvent(args: {
payload: unknown;
type: string;
}): Promise<void> {
await this.stub.receiveEvent({
payload: args.payload,
type: args.type,
timestamp: new Date(),
});
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
export * from "./binding";
export * from "./context";
export * from "./engine";
export * from "./instance";
export * from "./modifier";
+224
View File
@@ -0,0 +1,224 @@
import type { ResolvedStepConfig } from "./context";
import type {
DatabaseInstance,
DatabaseVersion,
DatabaseWorkflow,
} from "./engine";
import type { WorkflowEvent } from "cloudflare:workers";
export type Instance = {
id: string;
created_on: string;
modified_on: string;
workflow_id: string;
version_id: string;
status: InstanceStatus;
started_on: string | null;
ended_on: string | null;
};
export const INSTANCE_METADATA = `INSTANCE_METADATA`;
export type InstanceMetadata = {
accountId: number;
workflow: DatabaseWorkflow;
version: DatabaseVersion;
instance: DatabaseInstance;
event: WorkflowEvent<unknown>;
};
export enum InstanceStatus {
Queued = 0, // Queued and waiting to start
Running = 1,
Paused = 2, // TODO (WOR-73): Implement pause
Errored = 3, // Stopped due to a user or system Error
Terminated = 4, // Stopped explicitly by user
Complete = 5, // Successful completion
WaitingForPause = 6,
Waiting = 7,
}
export function instanceStatusName(status: InstanceStatus) {
switch (status) {
case InstanceStatus.Queued:
return "queued";
case InstanceStatus.Running:
return "running";
case InstanceStatus.Paused:
return "paused";
case InstanceStatus.Errored:
return "errored";
case InstanceStatus.Terminated:
return "terminated";
case InstanceStatus.Complete:
return "complete";
case InstanceStatus.WaitingForPause:
return "waitingForPause";
case InstanceStatus.Waiting:
return "waiting";
default:
return "unknown";
}
}
export const instanceStatusNames = [
"queued",
"running",
"paused",
"errored",
"terminated",
"complete",
"waitingForPause",
"waiting",
"unknown",
] as const;
export function toInstanceStatus(status: string): InstanceStatus {
switch (status) {
case "queued":
return InstanceStatus.Queued;
case "running":
return InstanceStatus.Running;
case "paused":
return InstanceStatus.Paused;
case "errored":
return InstanceStatus.Errored;
case "terminated":
return InstanceStatus.Terminated;
case "complete":
return InstanceStatus.Complete;
case "waitingForPause":
return InstanceStatus.WaitingForPause;
case "waiting":
return InstanceStatus.Waiting;
case "unknown":
throw new Error("unknown cannot be parsed into a InstanceStatus");
default:
throw new Error(
`${status} was not handled because it's not a valid InstanceStatus`
);
}
}
export const enum InstanceEvent {
WORKFLOW_QUEUED = 0,
WORKFLOW_START = 1,
WORKFLOW_SUCCESS = 2,
WORKFLOW_FAILURE = 3,
WORKFLOW_TERMINATED = 4,
STEP_START = 5,
STEP_SUCCESS = 6,
STEP_FAILURE = 7,
SLEEP_START = 8,
SLEEP_COMPLETE = 9,
ATTEMPT_START = 10,
ATTEMPT_SUCCESS = 11,
ATTEMPT_FAILURE = 12,
// It's here just to make it sequential and to not have gaps in the event types.
__INTERNAL_PROD = 13,
WAIT_START = 14,
WAIT_COMPLETE = 15,
WAIT_TIMED_OUT = 16,
ROLLBACK_START = 17,
ROLLBACK_STEP_START = 18,
ROLLBACK_ATTEMPT_START = 19,
ROLLBACK_ATTEMPT_SUCCESS = 20,
ROLLBACK_ATTEMPT_FAILURE = 21,
ROLLBACK_STEP_SUCCESS = 22,
ROLLBACK_STEP_FAILURE = 23,
ROLLBACK_COMPLETE = 24,
ROLLBACK_FAILED = 25,
}
export const enum InstanceTrigger {
API = 0,
BINDING = 1,
EVENT = 2,
CRON = 3,
}
export function instanceTriggerName(trigger: InstanceTrigger) {
switch (trigger) {
case InstanceTrigger.API:
return "api";
case InstanceTrigger.BINDING:
return "binding";
case InstanceTrigger.EVENT:
return "event";
case InstanceTrigger.CRON:
return "cron";
default:
return "unknown";
}
}
export type RawInstanceLog = {
id: number;
timestamp: string;
event: InstanceEvent;
groupKey: string | null;
target: string | null;
metadata: string;
};
export type InstanceAttempt = {
start: string;
end: string | null;
success: boolean | null;
error: { name: string; message: string } | null;
};
export type InstanceStepLog = {
name: string;
start: string;
end: string | null;
attempts: InstanceAttempt[];
config: ResolvedStepConfig;
output: unknown;
success: boolean | null;
type: "step";
};
export type InstanceSleepLog = {
name: string;
start: string;
end: string;
finished: boolean;
type: "sleep";
};
export type InstanceTerminateLog = {
type: "termination";
trigger: {
source: string;
};
};
export type InstanceLogsResponse = {
params: Record<string, unknown>;
trigger: {
source: ReturnType<typeof instanceTriggerName>;
};
versionId: string;
queued: string;
start: string | null;
end: string | null;
steps: (InstanceStepLog | InstanceSleepLog | InstanceTerminateLog)[];
success: boolean | null;
error: { name: string; message: string } | null;
output: Rpc.Serializable<unknown>;
};
export type WakerPriorityEntry = {
hash: string;
type: WakerPriorityType;
targetTimestamp: number;
};
export type WakerPriorityType = "sleep" | "retry" | "timeout";
@@ -0,0 +1,292 @@
import type {
ModifierCallback,
WorkflowBinding,
WorkflowInstanceIntrospector,
WorkflowInstanceModifier,
WorkflowIntrospectionOperation,
WorkflowIntrospectionStreamResult,
WorkflowIntrospector,
WorkflowStepSelector,
} from "./types";
function normalizeStreamMockChunk(value: unknown): Uint8Array {
if (value instanceof Uint8Array) {
return value;
}
if (value instanceof ArrayBuffer) {
return new Uint8Array(value);
}
if (ArrayBuffer.isView(value) && !(value instanceof DataView)) {
return new Uint8Array(value.buffer, value.byteOffset, value.byteLength);
}
throw new TypeError(
"Workflow mockStepResult() ReadableStream chunks must be ArrayBuffer or TypedArray values."
);
}
async function readStreamMockChunks(
stream: ReadableStream<unknown>
): Promise<Uint8Array[]> {
if (stream.locked) {
throw new TypeError(
"Workflow mockStepResult() received a locked or unreadable ReadableStream."
);
}
const chunks: Uint8Array[] = [];
const reader = stream.getReader();
let fullyRead = false;
try {
while (true) {
const result = await reader.read();
if (result.done) {
fullyRead = true;
return chunks;
}
const chunk = normalizeStreamMockChunk(result.value);
if (chunk.byteLength === 0) {
continue;
}
chunks.push(chunk.slice());
}
} finally {
if (!fullyRead) {
await reader
.cancel("stream mock consumption stopped before completion")
.catch(() => {});
}
try {
reader.releaseLock();
} catch {
/** Reader may still be processing cancel(). */
}
}
}
export class WorkflowInstanceModificationRecorder implements WorkflowInstanceModifier {
constructor(private readonly operations: WorkflowIntrospectionOperation[]) {}
async disableSleeps(steps?: WorkflowStepSelector[]): Promise<void> {
this.operations.push({ type: "disableSleeps", steps });
}
async disableRetryDelays(steps?: WorkflowStepSelector[]): Promise<void> {
this.operations.push({ type: "disableRetryDelays", steps });
}
async mockStepResult(
step: WorkflowStepSelector,
stepResult: unknown
): Promise<void> {
if (stepResult instanceof ReadableStream) {
const streamResult: WorkflowIntrospectionStreamResult = {
__workflowIntrospectionStreamResult: true,
chunks: await readStreamMockChunks(stepResult),
};
this.operations.push({
type: "mockStepResult",
step,
stepResult: streamResult,
});
return;
}
this.operations.push({ type: "mockStepResult", step, stepResult });
}
async mockStepError(
step: WorkflowStepSelector,
error: Error,
times?: number
): Promise<void> {
this.operations.push({
type: "mockStepError",
step,
error: { name: error.name, message: error.message },
times,
});
}
async forceStepTimeout(
step: WorkflowStepSelector,
times?: number
): Promise<void> {
this.operations.push({ type: "forceStepTimeout", step, times });
}
async mockEvent(event: { type: string; payload: unknown }): Promise<void> {
this.operations.push({ type: "mockEvent", event });
}
async forceEventTimeout(step: WorkflowStepSelector): Promise<void> {
this.operations.push({ type: "forceEventTimeout", step });
}
}
export class WorkflowIntrospectorHandle implements WorkflowIntrospector {
#disposed = false;
#sessionId: string | undefined;
#instanceIntrospectors = new Map<string, WorkflowInstanceIntrospector>();
#operations: WorkflowIntrospectionOperation[] = [];
constructor(private readonly workflow: WorkflowBinding) {}
async start(): Promise<void> {
this.#sessionId = await this.workflow.unsafeStartIntrospection();
}
private getSessionId(): string {
if (this.#sessionId === undefined) {
throw new Error("Workflow introspection has not started.");
}
return this.#sessionId;
}
async modifyAll(fn: ModifierCallback): Promise<void> {
const sessionId = this.getSessionId();
await fn(new WorkflowInstanceModificationRecorder(this.#operations));
await this.workflow.unsafeSetIntrospectionOperations(
sessionId,
this.#operations
);
}
async get(): Promise<WorkflowInstanceIntrospector[]> {
const sessionId = this.getSessionId();
await this.syncInstanceIntrospectors(sessionId);
return Array.from(this.#instanceIntrospectors.values());
}
private async syncInstanceIntrospectors(sessionId: string): Promise<void> {
const instanceIds =
await this.workflow.unsafeGetIntrospectionInstances(sessionId);
for (const instanceId of instanceIds) {
if (!this.#instanceIntrospectors.has(instanceId)) {
this.#instanceIntrospectors.set(
instanceId,
new WorkflowInstanceIntrospectorHandle(this.workflow, instanceId)
);
}
}
}
private async stopIntrospectionSession(sessionId: string): Promise<void> {
try {
await this.syncInstanceIntrospectors(sessionId);
} finally {
await this.workflow.unsafeStopIntrospection(sessionId);
}
}
private async disposeInstanceIntrospectors(): Promise<void> {
try {
await Promise.all(
Array.from(this.#instanceIntrospectors.values(), (introspector) =>
introspector.dispose()
)
);
} finally {
this.#instanceIntrospectors.clear();
}
}
/** Keep this bound; explicit resource management may call the disposer unbound. */
dispose = async (): Promise<void> => {
if (this.#disposed) {
return;
}
this.#disposed = true;
const sessionId = this.#sessionId;
try {
if (sessionId !== undefined) {
await this.stopIntrospectionSession(sessionId);
}
} finally {
await this.disposeInstanceIntrospectors();
}
};
async [Symbol.asyncDispose](): Promise<void> {
await this.dispose();
}
}
export class WorkflowInstanceIntrospectorHandle implements WorkflowInstanceIntrospector {
#instanceModifier: WorkflowInstanceModifier | undefined;
#instanceModifierPromise: Promise<WorkflowInstanceModifier> | undefined;
constructor(
private readonly workflow: WorkflowBinding,
private readonly instanceId: string
) {
this.#instanceModifierPromise = workflow
.unsafeGetInstanceModifier(instanceId)
.then((modifier) => {
this.#instanceModifier = modifier as WorkflowInstanceModifier;
this.#instanceModifierPromise = undefined;
return this.#instanceModifier;
});
// To avoid an unhandled rejection when the handle is used without modify()
void this.#instanceModifierPromise.catch(() => {});
}
async modify(fn: ModifierCallback): Promise<WorkflowInstanceIntrospector> {
if (this.#instanceModifierPromise !== undefined) {
this.#instanceModifier = await this.#instanceModifierPromise;
}
if (this.#instanceModifier === undefined) {
throw new Error(
"could not apply modifications due to internal error. Retrying the test may resolve the issue."
);
}
await fn(this.#instanceModifier);
return this;
}
async waitForStepResult(step: WorkflowStepSelector): Promise<unknown> {
return await this.workflow.unsafeWaitForStepResult(
this.instanceId,
step.name,
step.index
);
}
async waitForStatus(status: string): Promise<void> {
if (status === "queued") {
return;
}
await this.workflow.unsafeWaitForStatus(this.instanceId, status);
}
async getOutput(): Promise<unknown> {
return await this.workflow.unsafeGetOutputOrError(this.instanceId, true);
}
async getError(): Promise<{ name: string; message: string }> {
return (await this.workflow.unsafeGetOutputOrError(
this.instanceId,
false
)) as { name: string; message: string };
}
/** Keep this bound; explicit resource management may call the disposer unbound. */
dispose = async (): Promise<void> => {
await this.workflow.unsafeAbort(this.instanceId, "Instance dispose");
};
async [Symbol.asyncDispose](): Promise<void> {
await this.dispose();
}
}
@@ -0,0 +1,10 @@
export async function computeHash(value: string) {
const msgUint8 = new TextEncoder().encode(value); // encode as (utf-8) Uint8Array
const hashBuffer = await crypto.subtle.digest("SHA-1", msgUint8); // hash the message
const hashArray = Array.from(new Uint8Array(hashBuffer)); // convert buffer to byte array
const hashHex = hashArray
.map((b) => b.toString(16).padStart(2, "0"))
.join(""); // convert bytes to hex string
return hashHex;
}
@@ -0,0 +1,79 @@
import { ms } from "itty-time";
import { DelayFunctionError } from "./retries";
import type {
WorkflowDelayDuration,
WorkflowDelayFunction,
WorkflowDynamicDelayContext,
} from "cloudflare:workers";
export const DEFAULT_RETRY_DELAY_MS = 1000;
export const DELAY_FUNCTION_TIMEOUT_MS = ms("5 seconds");
type DelayLogger = {
warn(...msgs: unknown[]): void;
debug(...msgs: unknown[]): void;
};
type Waiter = (ms: number, opts?: { signal?: AbortSignal }) => Promise<void>;
export async function invokeDelayFunction(
delay: WorkflowDelayFunction,
input: WorkflowDynamicDelayContext,
options: {
timeoutMs: number;
wait: Waiter;
signal?: AbortSignal;
logger?: DelayLogger;
}
): Promise<WorkflowDelayDuration> {
const { wait, signal, logger } = options;
const logFields = { step: input.ctx.step.name, attempt: input.ctx.attempt };
const settled = new AbortController();
const timeoutSignal = signal
? AbortSignal.any([signal, settled.signal])
: settled.signal;
type Raced =
| { timedOut: true }
| { timedOut: false; value: WorkflowDelayDuration };
try {
const result = await Promise.race<Raced>([
Promise.resolve(delay(input)).then((value) => ({
timedOut: false,
value,
})),
wait(options.timeoutMs, { signal: timeoutSignal })
.then((): Raced => ({ timedOut: true }))
.catch((): Raced => ({ timedOut: true })),
]);
if (result.timedOut) {
if (signal?.aborted) {
logger?.debug(
"delay function aborted by engine shutdown; using default delay",
logFields
);
return DEFAULT_RETRY_DELAY_MS;
}
logger?.warn(
`delay function did not resolve within ${options.timeoutMs}ms`,
logFields
);
throw new DelayFunctionError(
`did not return within ${options.timeoutMs / 1000} seconds`
);
}
return result.value;
} catch (e) {
if (e instanceof DelayFunctionError) {
throw e;
}
const message = e instanceof Error ? e.message : String(e);
logger?.warn("delay function threw", { ...logFields, error: message });
throw new DelayFunctionError(`threw an error: ${message}`);
} finally {
settled.abort();
}
}
+127
View File
@@ -0,0 +1,127 @@
export class WorkflowTimeoutError extends Error {
name = "WorkflowTimeoutError";
}
export class WorkflowInternalError extends Error {
name = "WorkflowInternalError";
}
export class WorkflowFatalError extends Error {
name = "WorkflowFatalError";
toJSON() {
return {
name: this.name,
message: this.message,
};
}
}
export class NonRetryableDelayError extends WorkflowFatalError {
name = "NonRetryableDelayError";
}
export class PreservedNonRetryableError extends WorkflowFatalError {
name = "NonRetryableError";
constructor(err: Error) {
// When the error crosses an RPC boundary, the name gets
// prepended to the message (e.g. "NonRetryableError: msg",
// or just "NonRetryableError" if the original message was empty).
// Parse it back out so we surface the original message.
const message =
err.name === "NonRetryableError"
? err.message
: err.message.replace(/^NonRetryableError:?\s*/, "");
super(message);
}
}
export class WorkflowError extends Error {
name = "WorkflowError";
}
export class InvalidStepReadableStreamError extends Error {
name = "InvalidStepReadableStreamError";
}
export class OversizedStreamChunkError extends Error {
name = "OversizedStreamChunkError";
}
export class UnsupportedStreamChunkError extends Error {
name = "UnsupportedStreamChunkError";
}
export class StreamOutputStorageLimitError extends Error {
name = "StreamOutputStorageLimitError";
}
export function createWorkflowError(
message: string,
errorCode: string
): WorkflowError {
return new WorkflowError(`(${errorCode}) ${message}`);
}
const ABORT_PREFIX = "Aborting engine:" as const;
export const ABORT_REASONS = {
USER_PAUSE: `${ABORT_PREFIX} User called pause`,
USER_RESTART: `${ABORT_PREFIX} User called restart`,
USER_TERMINATE: `${ABORT_PREFIX} User called terminate`,
NON_RETRYABLE_ERROR: `${ABORT_PREFIX} A step threw a NonRetryableError`,
NOT_SERIALISABLE: `${ABORT_PREFIX} Value is not serialisable`,
STORAGE_LIMIT_EXCEEDED: `${ABORT_PREFIX} Storage limit exceeded`,
GRACE_PERIOD_COMPLETE: `${ABORT_PREFIX} Grace period complete`,
} as const;
const ABORT_REASON_SET: ReadonlySet<string> = new Set(
Object.values(ABORT_REASONS)
);
function getErrorMessage(e: unknown): string | undefined {
if (e instanceof Error) {
return e.message;
}
if (typeof e === "object" && e !== null) {
const msg = (e as { message?: string }).message;
if (typeof msg === "string") {
return msg;
}
}
return undefined;
}
export function isAbortError(e: unknown): boolean {
const msg = getErrorMessage(e);
return msg !== undefined && ABORT_REASON_SET.has(msg);
}
export function isUserTriggeredPause(e: unknown): boolean {
return getErrorMessage(e) === ABORT_REASONS.USER_PAUSE;
}
export function isUserTriggeredRestart(e: unknown): boolean {
return getErrorMessage(e) === ABORT_REASONS.USER_RESTART;
}
export function isUserTriggeredTerminate(e: unknown): boolean {
return getErrorMessage(e) === ABORT_REASONS.USER_TERMINATE;
}
function getCompatFlag(name: string): boolean {
// eslint-disable-next-line @typescript-eslint/no-explicit-any -- safe globalThis access for environments where cloudflare global may not exist
return (globalThis as any).Cloudflare?.compatibilityFlags?.[name] ?? false;
}
export function shouldPreserveNonRetryableError(): boolean {
return getCompatFlag("workflows_preserve_non_retryable_error_message");
}
export function stepNotFoundError(name: string): WorkflowError {
return createWorkflowError(
`Step "${name}" not found in execution history`,
"instance.cannot_restart"
);
}
@@ -0,0 +1,190 @@
import { ms } from "itty-time";
import type { Engine } from "../engine";
import type { WorkflowSleepDuration } from "cloudflare:workers";
export const ENGINE_TIMEOUT = ms("5 minutes" satisfies WorkflowSleepDuration);
let latestGracePeriodTimestamp: number | undefined = undefined;
export type WaitingPromiseType = "pause";
export type GracePeriodCallback = (engine: Engine, timeoutMs: number) => void;
export class GracePeriodSemaphore {
#counter: number = 0;
readonly callback: GracePeriodCallback;
readonly timeoutMs: number;
#waitingPromises: {
rejectCallback: () => void;
resolveCallback: (value: unknown) => void;
type: WaitingPromiseType;
}[] = [];
#canInitiateSteps = true;
#waitingSteps: {
rejectCallback: () => void;
resolveCallback: (value: unknown) => void;
}[] = [];
constructor(callback: GracePeriodCallback, timeoutMs: number) {
this.callback = callback;
this.timeoutMs = timeoutMs;
}
// acquire takes engine to be the same as release
async acquire(_engine: Engine) {
if (!this.#canInitiateSteps) {
await new Promise((resolve, reject) => {
this.#waitingSteps.push({
resolveCallback: resolve,
rejectCallback: reject,
});
});
}
// when the counter goes from 0 to 1 - we can safely reject the previous grace period
if (this.#counter == 0) {
latestGracePeriodTimestamp = undefined;
}
this.#counter += 1;
}
async release(engine: Engine) {
this.#counter = Math.max(this.#counter - 1, 0);
if (this.#counter == 0) {
// Trigger timeout promise, no need to await here,
// this can be triggered slightly after it's not time sensitive
this.callback(engine, this.timeoutMs);
// Resolve any promises waiting for all steps to finish (e.g. pause)
for (const promise of this.#waitingPromises) {
promise.resolveCallback(undefined);
}
this.#waitingPromises = [];
}
}
async waitUntilNothingIsRunning(
type: WaitingPromiseType,
callback: () => Promise<void>
): Promise<void> {
this.#canInitiateSteps = false;
if (this.#counter > 0) {
try {
await new Promise((resolve, reject) => {
this.#waitingPromises.push({
resolveCallback: resolve,
rejectCallback: reject,
type,
});
});
} catch {
// If the promise gets rejected (e.g. resume cancels the pause),
// allow steps to run again and unblock any that were waiting
for (const promise of this.#waitingSteps) {
promise.resolveCallback(undefined);
}
this.#waitingSteps = [];
this.#canInitiateSteps = true;
return;
}
}
await callback();
// Allow steps to run again and unblock any that were waiting
for (const promise of this.#waitingSteps) {
promise.resolveCallback(undefined);
}
this.#waitingSteps = [];
this.#canInitiateSteps = true;
}
cancelWaitingPromisesByType(type: WaitingPromiseType) {
const sameTypePromises = this.#waitingPromises.filter(
(val) => val.type === type
);
if (sameTypePromises.length === 0) {
return;
}
for (const promise of sameTypePromises) {
promise.rejectCallback();
}
this.#waitingPromises = this.#waitingPromises.filter(
(val) => val.type !== type
);
this.#canInitiateSteps = true;
// Unblock any steps that were waiting to acquire while the pause was pending
for (const promise of this.#waitingSteps) {
promise.resolveCallback(undefined);
}
this.#waitingSteps = [];
}
dispose() {
// Reject all waiting step promises so they stop blocking
for (const promise of this.#waitingSteps) {
promise.rejectCallback();
}
this.#waitingSteps = [];
// Reject all waiting promises
for (const promise of this.#waitingPromises) {
promise.rejectCallback();
}
this.#waitingPromises = [];
this.#canInitiateSteps = false;
}
isRunningStep() {
return this.#counter > 0;
}
}
export const startGracePeriod: GracePeriodCallback = async (
engine: Engine,
timeoutMs: number
) => {
const gracePeriodHandler = async () => {
const thisTimestamp = new Date().valueOf();
// TODO: Should the grace period be 5 mins every time or 5 mins across lifetimes?
// At the moment, it looks like this will reset to 5 mins if a metal crashes
// There might possibly waste memory waiting for `timeoutMs` every time
// We should eventually strongly persist and respect this value across lifetimes
// We are starting a new grace period
// 1. There should not be one already set
// 2. Or if there is, it should be in the past
if (
!(
latestGracePeriodTimestamp === undefined ||
latestGracePeriodTimestamp < thisTimestamp
)
) {
throw new Error(
"Can't start grace period since there is already an active one started on " +
latestGracePeriodTimestamp
);
}
latestGracePeriodTimestamp = thisTimestamp;
await scheduler.wait(timeoutMs);
if (
thisTimestamp !== latestGracePeriodTimestamp ||
engine.timeoutHandler.isRunningStep()
) {
return;
}
// priorityQueue is set before the user code runs which implies that a grace period cannot start
// before init finishes where it is set
// Ensure next alarm is set before we abort
await engine.priorityQueue?.handleNextAlarm();
// await engine.abort(ABORT_REASONS.GRACE_PERIOD_COMPLETE);
};
void gracePeriodHandler().catch(() => {
// Swallow — the engine is shutting down (abort kills the context,
// which rejects the scheduler.wait inside gracePeriodHandler)
});
};
@@ -0,0 +1,171 @@
import { InstanceEvent } from "../instance";
import { MODIFIER_KEYS } from "../modifier";
import type { RestartFromStep } from "../binding";
import type { RawInstanceLog } from "../instance";
const EVENT_MAP_PREFIX = "EVENT_MAP";
const RESTART_FROM_STEP_KEY = "RESTART_FROM_STEP";
const KV_STEP_SUFFIXES = [
"-value",
"-error",
"-config",
"-metadata",
"-log-written",
"-pending",
"-value-stream-meta",
] as const;
export function resolveGroupKeysToWipe(
sql: DurableObjectStorage["sql"],
param: RestartFromStep
): Set<string> | null {
const stepTypeToEvent: Record<string, InstanceEvent> = {
do: InstanceEvent.STEP_START,
sleep: InstanceEvent.SLEEP_START,
waitForEvent: InstanceEvent.WAIT_START,
};
const targetEvent = stepTypeToEvent[param.type ?? "do"];
const targetCount = param.count ?? 1;
const cursor = sql.exec<RawInstanceLog>(
"SELECT event, target, groupKey FROM states WHERE event IN (?, ?, ?) ORDER BY id",
InstanceEvent.STEP_START,
InstanceEvent.SLEEP_START,
InstanceEvent.WAIT_START
);
let nameOccurrence = 0;
let found = false;
const groupKeys = new Set<string>();
for (const row of cursor) {
if (row.groupKey === null) {
continue;
}
const groupKey = String(row.groupKey);
if (found) {
groupKeys.add(groupKey);
continue;
}
if (row.target === null) {
continue;
}
const rawStepName = String(row.target).replace(/-\d+$/, "");
if (rawStepName !== param.name) {
continue;
}
if (row.event !== targetEvent) {
continue;
}
nameOccurrence++;
if (nameOccurrence === targetCount) {
found = true;
groupKeys.add(groupKey);
}
}
return found ? groupKeys : null;
}
function getMockedEventMapKeys(allKeys: Map<string, unknown>): Set<string> {
const mockEventTypes = new Set<string>();
for (const key of allKeys.keys()) {
if (key.startsWith(MODIFIER_KEYS.MOCK_EVENT)) {
mockEventTypes.add(key.slice(MODIFIER_KEYS.MOCK_EVENT.length));
}
}
if (mockEventTypes.size === 0) {
return new Set();
}
const preserved = new Set<string>();
for (const key of allKeys.keys()) {
if (key.startsWith(`${EVENT_MAP_PREFIX}\n`)) {
const eventType = key.split("\n")[1];
if (eventType !== undefined && mockEventTypes.has(eventType)) {
preserved.add(key);
}
}
}
return preserved;
}
async function deleteGroupKeyBatch(
storage: DurableObjectStorage,
batch: string[]
): Promise<void> {
const kvKeys: string[] = [];
for (const groupKey of batch) {
for (const suffix of KV_STEP_SUFFIXES) {
kvKeys.push(`${groupKey}${suffix}`);
}
storage.sql.exec("DELETE FROM states WHERE groupKey = ?", groupKey);
storage.sql.exec(
"DELETE FROM streaming_step_chunks WHERE attempt != 0 AND cache_key = ?",
groupKey
);
}
await storage.delete(kvKeys);
}
export async function wipeRestartState(
storage: DurableObjectStorage,
engineStatusKey: string,
pauseDatetimeKey: string,
groupKeysToWipe: Set<string> | null
): Promise<void> {
if (groupKeysToWipe) {
await deleteGroupKeyBatch(storage, Array.from(groupKeysToWipe));
storage.sql.exec(
"DELETE FROM states WHERE groupKey IS NULL AND event NOT IN (?, ?)",
InstanceEvent.WORKFLOW_START,
InstanceEvent.WORKFLOW_QUEUED
);
} else {
const cursor = storage.sql.exec<{ groupKey: string }>(
"SELECT DISTINCT groupKey FROM states WHERE groupKey IS NOT NULL"
);
const allGroupKeys = [...cursor].map((r) => r.groupKey);
if (allGroupKeys.length > 0) {
await deleteGroupKeyBatch(storage, allGroupKeys);
}
storage.sql.exec("DELETE FROM states");
}
const keysToDelete: string[] = [engineStatusKey, pauseDatetimeKey];
const allKeys = await storage.list();
const preservedEventMapKeys = getMockedEventMapKeys(allKeys);
for (const key of allKeys.keys()) {
if (
key.startsWith(`${EVENT_MAP_PREFIX}\n`) &&
!preservedEventMapKeys.has(key)
) {
keysToDelete.push(key);
}
}
await storage.delete(keysToDelete);
storage.sql.exec("DELETE FROM priority_queue");
}
export async function readAndClearRestartFromStep(
storage: DurableObjectStorage
): Promise<RestartFromStep | undefined> {
const value = await storage.get<RestartFromStep>(RESTART_FROM_STEP_KEY);
await storage.delete(RESTART_FROM_STEP_KEY);
return value;
}
export async function storeRestartFromStep(
storage: DurableObjectStorage,
from: RestartFromStep
): Promise<void> {
await storage.put(RESTART_FROM_STEP_KEY, from);
}
@@ -0,0 +1,46 @@
import { ms } from "itty-time";
import type { ResolvedStepConfig, StepState } from "../context";
import type { WorkflowSleepDuration } from "cloudflare:workers";
export class DelayFunctionError extends Error {
constructor(reason: string) {
super(reason);
this.name = "DelayFunctionError";
}
}
export function calcRetryDuration(
config: ResolvedStepConfig,
stepState: StepState,
delayValue: unknown
): number {
const { attemptedCount: attemptCount } = stepState;
const { retries } = config;
let base: number;
try {
base = ms(delayValue as WorkflowSleepDuration);
} catch {
throw new DelayFunctionError(
'returned an invalid delay value (expected a number of ms or a duration string like "30 seconds")'
);
}
if (!Number.isFinite(base) || base < 0) {
throw new DelayFunctionError(
'returned an invalid delay value (expected a number of ms or a duration string like "30 seconds")'
);
}
switch (retries.backoff) {
case "exponential": {
return base * Math.pow(2, attemptCount - 1);
}
case "linear": {
return base * attemptCount;
}
case "constant":
default: {
return base;
}
}
}
@@ -0,0 +1,211 @@
import { InstanceEvent } from "../instance";
import { WorkflowFatalError } from "./errors";
import { isValidStepConfig } from "./validators";
import type { WorkflowStepContext } from "../context";
import type { Engine } from "../engine";
import type { WorkflowStepConfig } from "cloudflare:workers";
type UserErrorField = {
isUserError?: boolean;
};
// `:` can't appear in user-step cacheKeys (sha1-hex + `-` only).
export const ROLLBACK_CACHE_KEY_PREFIX = "rollback:";
export type RollbackContext = {
ctx: WorkflowStepContext;
error: Error;
output: unknown;
/** @deprecated Use `${ctx.step.name}-${ctx.step.count}` instead. */
stepName: string;
};
// dup() to outlive the originating step.do call; Symbol.dispose locally
// (calling `.dispose()` would RPC to a non-existent remote method).
export type RollbackFn = ((ctx: RollbackContext) => Promise<void>) & {
dup?: () => RollbackFn;
[Symbol.dispose]?: () => void;
};
export type WorkflowStepRollbackConfig = Pick<
WorkflowStepConfig,
"retries" | "timeout"
>;
export type WorkflowStepRollbackOptions = {
rollback: RollbackFn;
rollbackConfig?: WorkflowStepRollbackConfig;
};
export type RollbackRegistryEntry = {
fn: RollbackFn;
stepContext: WorkflowStepContext;
output?: unknown;
config?: WorkflowStepConfig;
};
export type RollbackRegistration = RollbackRegistryEntry & {
cacheKey: string;
};
export function parseRollbackOptions(
stepName: string,
options: unknown
): WorkflowStepRollbackOptions | undefined {
if (options === undefined) {
return undefined;
}
if (
typeof options !== "object" ||
options === null ||
Array.isArray(options)
) {
const error = new WorkflowFatalError(
`Rollback options for "${stepName}" must be an object`
) as Error & UserErrorField;
error.isUserError = true;
throw error;
}
const rollbackOptions = options as Partial<WorkflowStepRollbackOptions>;
if (typeof rollbackOptions.rollback !== "function") {
const error = new WorkflowFatalError(
`Rollback for "${stepName}" must be a function`
) as Error & UserErrorField;
error.isUserError = true;
throw error;
}
if (
rollbackOptions.rollbackConfig !== undefined &&
!isValidStepConfig(rollbackOptions.rollbackConfig)
) {
const error = new WorkflowFatalError(
`Rollback config for "${stepName}" is in a invalid format. See https://developers.cloudflare.com/workflows/build/sleeping-and-retrying/`
) as Error & UserErrorField;
error.isUserError = true;
throw error;
}
return rollbackOptions as WorkflowStepRollbackOptions;
}
export function dupRollbackStub(fn: RollbackFn): RollbackFn {
return fn.dup ? fn.dup() : fn;
}
export function disposeRollbackStub(fn: RollbackFn): void {
try {
fn[Symbol.dispose]?.();
} catch (err) {
console.warn("Failed to dispose rollback stub", err);
}
}
export function registerRollbackFn(
registry: Map<string, RollbackRegistryEntry>,
registration: RollbackRegistration
): void {
const { cacheKey, fn, stepContext, output, config } = registration;
const existing = registry.get(cacheKey);
if (existing) {
// Existing entry already owns the duped rollback stub. Duplicate registrations
// refresh context/output only; this helper has not duped the incoming fn.
registry.set(cacheKey, {
...existing,
stepContext,
...("output" in registration && { output }),
});
return;
}
registry.set(cacheKey, {
fn: dupRollbackStub(fn),
stepContext,
...("output" in registration && { output }),
...(config !== undefined && { config }),
});
}
export function clearRollbackRegistry(
registry: Map<string, RollbackRegistryEntry>
): void {
for (const entry of registry.values()) {
disposeRollbackStub(entry.fn);
}
registry.clear();
}
// LIFO; halts on first failure. Goes through Context.do so each rollback
// inherits retries/timeouts/attempt-logging.
export async function executeRollbacks(
engine: Engine,
triggerError: Error
): Promise<{ ranAny: boolean; allSucceeded: boolean }> {
const eligibleSteps = engine.readEligibleRollbackStepsDesc();
if (eligibleSteps.length === 0) {
clearRollbackRegistry(engine.rollbackRegistry);
return { ranAny: false, allSucceeded: true };
}
engine.writeLog(InstanceEvent.ROLLBACK_START, null, null, {
triggerError: { name: triggerError.name, message: triggerError.message },
totalSteps: eligibleSteps.length,
});
let allSucceeded = true;
let completed = 0;
try {
for (const step of eligibleSteps) {
const entry = engine.rollbackRegistry.get(step.cacheKey);
if (entry === undefined) {
engine.writeLog(
InstanceEvent.ROLLBACK_STEP_FAILURE,
step.cacheKey,
step.target,
{
error: {
name: "RollbackMissing",
message: "Rollback function not available in registry",
},
}
);
allSucceeded = false;
break;
}
const ctx = engine.createRollbackContext({ cacheKey: step.cacheKey });
try {
await ctx.do(step.target, entry.config ?? {}, async () => {
await entry.fn({
ctx: structuredClone(entry.stepContext),
error: triggerError,
output: entry.output,
stepName: step.target,
});
});
completed++;
} catch {
// Context.do already wrote ROLLBACK_STEP_FAILURE; halt the chain.
allSucceeded = false;
break;
} finally {
disposeRollbackStub(entry.fn);
engine.rollbackRegistry.delete(step.cacheKey);
}
}
} finally {
clearRollbackRegistry(engine.rollbackRegistry);
}
engine.writeLog(
allSucceeded
? InstanceEvent.ROLLBACK_COMPLETE
: InstanceEvent.ROLLBACK_FAILED,
null,
null,
{ totalSteps: eligibleSteps.length, completedSteps: completed }
);
return { ranAny: completed > 0, allSucceeded };
}
@@ -0,0 +1,698 @@
import {
InvalidStepReadableStreamError,
OversizedStreamChunkError,
StreamOutputStorageLimitError,
UnsupportedStreamChunkError,
WorkflowTimeoutError,
} from "./errors";
// ── Constants ───────────────────────────────────────────────────────────────
export const DEFAULT_STREAM_OUTPUT_CHUNK_SIZE = 256 * 1024;
export const STREAM_OUTPUT_META_SUFFIX = "-value-stream-meta";
export const MAX_STREAM_OUTPUT_INPUT_CHUNK_BYTES = 16 * 1024 * 1024;
export const STREAMING_STEP_CHUNKS_TABLE = "streaming_step_chunks";
export const MAX_OUTPUT_SHOWN_IN_LOGS = 1024;
const DO_STORAGE_LIMIT = 1024 * 1024 * 1024 + 100 * 1024 * 1024;
const STREAM_OUTPUT_STORAGE_WRITE_HEADROOM_BYTES = 16 * 1024;
// ── Types ───────────────────────────────────────────────────────────────────
export enum StreamOutputState {
Pending = "pending",
Committing = "committing",
Complete = "complete",
}
export type StreamOutputMeta = {
version: 1;
state: StreamOutputState;
attempt: number;
startedAt: number;
chunkCount: number;
totalBytes: number;
committedAt: number | null;
};
export type StoredStreamOutputPreview =
| { type: "text"; output: string }
| { type: "binary" };
export class InvalidStoredStreamOutputError extends Error {
name = "InvalidStoredStreamOutputError";
}
// ── Helpers ─────────────────────────────────────────────────────────────────
export function getStreamOutputMetaKey(cacheKey: string): string {
return `${cacheKey}${STREAM_OUTPUT_META_SUFFIX}`;
}
export function isReadableStreamLike(
value: unknown
): value is ReadableStream<unknown> {
return value instanceof ReadableStream;
}
function createInvalidStepReadableStreamError(): InvalidStepReadableStreamError {
return new InvalidStepReadableStreamError(
"Step returned a ReadableStream that is already locked or otherwise unreadable. Return a fresh, unlocked ReadableStream from step.do()."
);
}
function createOversizedStreamChunkError(): OversizedStreamChunkError {
return new OversizedStreamChunkError(
`Step returned a ReadableStream chunk larger than the maximum allowed size of ${MAX_STREAM_OUTPUT_INPUT_CHUNK_BYTES} bytes. ` +
"Return smaller chunks from step.do()."
);
}
/**
* Normalize any incoming chunk to Uint8Array.
* Accepts ArrayBuffer, TypedArrays (except DataView), and Uint8Array directly.
* Rejects strings, objects, and other non-binary types.
*/
function normalizeChunkToUint8Array(value: unknown): Uint8Array {
if (value instanceof Uint8Array) {
return value;
}
if (value instanceof ArrayBuffer) {
return new Uint8Array(value);
}
if (ArrayBuffer.isView(value) && !(value instanceof DataView)) {
return new Uint8Array(value.buffer, value.byteOffset, value.byteLength);
}
throw new UnsupportedStreamChunkError(
"Step returned a ReadableStream with unsupported chunk type. " +
"Only ArrayBuffer and TypedArray chunks are supported."
);
}
// ── Buffer helpers ──────────────────────────────────────────────────────────
function takeBufferedBytes(
bufferedChunks: Uint8Array[],
byteLength: number
): Uint8Array {
const output = new Uint8Array(byteLength);
let offset = 0;
while (offset < byteLength) {
const chunk = bufferedChunks[0];
const remaining = byteLength - offset;
if (chunk.byteLength <= remaining) {
output.set(chunk, offset);
offset += chunk.byteLength;
bufferedChunks.shift();
continue;
}
output.set(chunk.subarray(0, remaining), offset);
bufferedChunks[0] = chunk.subarray(remaining);
offset += remaining;
}
return output;
}
// ── Stream iteration ────────────────────────────────────────────────────────
async function* iterateStreamChunks(
stream: ReadableStream<unknown>,
signal?: AbortSignal
): AsyncGenerator<Uint8Array> {
if (stream.locked) {
throw createInvalidStepReadableStreamError();
}
if (signal?.aborted) {
throw (
signal.reason ??
new DOMException("The operation was aborted.", "AbortError")
);
}
let reader: ReadableStreamDefaultReader<unknown>;
try {
reader = stream.getReader();
} catch (error) {
if (error instanceof TypeError) {
throw createInvalidStepReadableStreamError();
}
throw error;
}
const onAbort = () => {
void reader
.cancel(
signal?.reason ??
new DOMException("The operation was aborted.", "AbortError")
)
.catch(() => {});
};
signal?.addEventListener("abort", onAbort, { once: true });
let fullyRead = false;
try {
while (true) {
let readResult: ReadableStreamReadResult<unknown>;
try {
readResult = await reader.read();
} catch (readError) {
// When the abort signal has already fired, the read error is an
// expected cancellation (e.g. step timeout or engine shutdown).
// Re-throw the original abort reason so callers can distinguish
// timeouts from genuine stream failures.
if (signal?.aborted) {
throw (
signal.reason ??
new DOMException("The operation was aborted.", "AbortError")
);
}
// Any other read failure (broken pipe, upstream connection
// drop, encoding mismatch, etc.) means the ReadableStream the
// step returned is unusable.
throw new InvalidStepReadableStreamError(
"Failed to read from step ReadableStream output. " +
(readError instanceof Error ? readError.message : String(readError))
);
}
if (signal?.aborted) {
throw (
signal.reason ??
new DOMException("The operation was aborted.", "AbortError")
);
}
if (readResult.done) {
fullyRead = true;
return;
}
yield normalizeChunkToUint8Array(readResult.value);
}
} finally {
signal?.removeEventListener("abort", onAbort);
if (!fullyRead) {
await reader
.cancel(
new Error("stream output consumption stopped before completion")
)
.catch(() => {});
}
try {
reader.releaseLock();
} catch {
// Reader may still be processing cancel()
}
}
}
// ── SQL helpers ─────────────────────────────────────────────────────────────
function deleteAttemptChunks(
storage: DurableObjectStorage,
cacheKey: string,
attempt: number
): void {
// eslint-disable-next-line workers-sdk/no-unsafe-command-execution -- DO SQL exec, not child_process
storage.sql.exec(
`DELETE FROM ${STREAMING_STEP_CHUNKS_TABLE} WHERE cache_key = ? AND attempt = ?`,
cacheKey,
attempt
);
}
async function deleteMetaForAttempt(
storage: DurableObjectStorage,
cacheKey: string,
attempt: number
): Promise<void> {
const metaKey = getStreamOutputMetaKey(cacheKey);
const maybeMeta = await storage.get<StreamOutputMeta>(metaKey);
if (maybeMeta === undefined) {
return;
}
if (maybeMeta.attempt !== attempt) {
return;
}
await storage.delete(metaKey);
}
// ── Integrity validation ────────────────────────────────────────────────────
type StreamOutputChunkSummary = {
chunkCount: number;
minChunkIndex: number | null;
maxChunkIndex: number | null;
totalBytes: number;
};
function getStreamOutputChunkSummary(
storage: DurableObjectStorage,
cacheKey: string,
attempt: number
): StreamOutputChunkSummary {
const row = storage.sql
.exec<StreamOutputChunkSummary>(
[
`SELECT`,
` COUNT(*) AS chunkCount,`,
` MIN(chunk_index) AS minChunkIndex,`,
` MAX(chunk_index) AS maxChunkIndex,`,
` CAST(COALESCE(SUM(LENGTH(chunk)), 0) AS INTEGER) AS totalBytes`,
`FROM ${STREAMING_STEP_CHUNKS_TABLE}`,
`WHERE cache_key = ? AND attempt = ?`,
].join("\n"),
cacheKey,
attempt
)
.one();
if (row === null) {
throw new Error("Expected stream chunk summary query to return a row");
}
return row;
}
export function getInvalidStoredStreamOutputError(
storage: DurableObjectStorage,
cacheKey: string,
meta: StreamOutputMeta
): InvalidStoredStreamOutputError | undefined {
const summary = getStreamOutputChunkSummary(storage, cacheKey, meta.attempt);
if (meta.chunkCount === 0) {
if (
summary.chunkCount === 0 &&
summary.totalBytes === 0 &&
summary.minChunkIndex === null &&
summary.maxChunkIndex === null
) {
return undefined;
}
} else if (
summary.chunkCount === meta.chunkCount &&
summary.minChunkIndex === 0 &&
summary.maxChunkIndex === meta.chunkCount - 1 &&
summary.totalBytes === meta.totalBytes
) {
return undefined;
}
return new InvalidStoredStreamOutputError(
`Stored streamed step output is corrupt or incomplete for cache key ${cacheKey}. ` +
`Expected ${meta.chunkCount} chunks / ${meta.totalBytes} bytes, found ` +
`${summary.chunkCount} chunks / ${summary.totalBytes} bytes with chunk index range ` +
`${summary.minChunkIndex ?? "null"}..${summary.maxChunkIndex ?? "null"}.`
);
}
// ── Preview ─────────────────────────────────────────────────────────────────
function readStreamOutputPreviewBytes(options: {
storage: DurableObjectStorage;
cacheKey: string;
attempt: number;
maxBytes: number;
}): Uint8Array {
const { storage, cacheKey, attempt, maxBytes } = options;
// eslint-disable-next-line workers-sdk/no-unsafe-command-execution -- DO SQL exec, not child_process
const cursor = storage.sql.exec<{
chunk_index: number;
chunk: ArrayBuffer;
}>(
`SELECT chunk_index, chunk FROM ${STREAMING_STEP_CHUNKS_TABLE} WHERE cache_key = ? AND attempt = ? ORDER BY chunk_index`,
cacheKey,
attempt
);
const previewChunks: Uint8Array[] = [];
let expectedChunkIndex = 0;
let totalBytes = 0;
while (totalBytes < maxBytes) {
const row = cursor.next();
if (row.done) {
break;
}
if (row.value.chunk_index !== expectedChunkIndex) {
throw new InvalidStoredStreamOutputError(
`Missing chunk ${expectedChunkIndex} for streamed step output`
);
}
if (!(row.value.chunk instanceof ArrayBuffer)) {
throw new InvalidStoredStreamOutputError(
"Invalid chunk type returned from streaming_step_chunks table"
);
}
const chunkBytes = new Uint8Array(row.value.chunk);
const remainingBytes = maxBytes - totalBytes;
const previewChunk =
chunkBytes.byteLength > remainingBytes
? chunkBytes.subarray(0, remainingBytes)
: chunkBytes;
previewChunks.push(previewChunk);
totalBytes += previewChunk.byteLength;
expectedChunkIndex++;
}
return takeBufferedBytes(previewChunks, totalBytes);
}
export function getStoredStreamOutputPreview(options: {
storage: DurableObjectStorage;
cacheKey: string;
meta: StreamOutputMeta;
maxChars: number;
}): StoredStreamOutputPreview {
const { storage, cacheKey, meta, maxChars } = options;
if (meta.state !== StreamOutputState.Complete) {
throw new Error(
"Cannot preview streamed step output before it is complete"
);
}
// UTF-8 uses at most 4 bytes per code point
const maxPreviewBytes = maxChars * 4;
const previewBytes = readStreamOutputPreviewBytes({
storage,
cacheKey,
attempt: meta.attempt,
maxBytes: maxPreviewBytes,
});
const previewTruncatedByBytes = meta.totalBytes > previewBytes.byteLength;
try {
const decoded = new TextDecoder("utf-8", {
fatal: true,
ignoreBOM: false,
}).decode(previewBytes, { stream: previewTruncatedByBytes });
const previewOutput = decoded.substring(0, maxChars);
if (decoded.length > maxChars || previewTruncatedByBytes) {
return { type: "text", output: previewOutput + "[truncated output]" };
}
return { type: "text", output: previewOutput };
} catch {
return { type: "binary" };
}
}
// ── Cleanup ─────────────────────────────────────────────────────────────────
export async function cleanupPendingStreamOutput(
storage: DurableObjectStorage,
cacheKey: string
): Promise<void> {
const metaKey = getStreamOutputMetaKey(cacheKey);
const maybeMeta = await storage.get<StreamOutputMeta>(metaKey);
if (maybeMeta === undefined) {
return;
}
if (maybeMeta.state === StreamOutputState.Complete) {
return;
}
await rollbackStreamOutput(storage, cacheKey, maybeMeta.attempt);
}
export async function rollbackStreamOutput(
storage: DurableObjectStorage,
cacheKey: string,
attempt: number
): Promise<void> {
deleteAttemptChunks(storage, cacheKey, attempt);
await deleteMetaForAttempt(storage, cacheKey, attempt);
}
// ── Write ───────────────────────────────────────────────────────────────────
async function doWriteStreamOutput(options: {
storage: DurableObjectStorage;
cacheKey: string;
attempt: number;
stream: ReadableStream<unknown>;
chunkSizeBytes?: number;
signal?: AbortSignal;
skipMetaWrite?: boolean;
}): Promise<StreamOutputMeta> {
const { storage, cacheKey, attempt, stream, signal, skipMetaWrite } = options;
const chunkSizeBytes =
options.chunkSizeBytes ?? DEFAULT_STREAM_OUTPUT_CHUNK_SIZE;
const metaKey = getStreamOutputMetaKey(cacheKey);
const maybeInvalidState = (additionalBytes = 0): unknown => {
if (signal?.aborted) {
return (
signal.reason ??
new DOMException("The operation was aborted.", "AbortError")
);
}
const currentStorageBytes = storage.sql.databaseSize;
if (
currentStorageBytes +
additionalBytes +
STREAM_OUTPUT_STORAGE_WRITE_HEADROOM_BYTES >
DO_STORAGE_LIMIT
) {
return new StreamOutputStorageLimitError(
"The instance has exceeded the 1GiB storage limit"
);
}
};
const initialInvalidState = maybeInvalidState();
if (initialInvalidState !== undefined) {
throw initialInvalidState;
}
const startedAt = Date.now();
if (!skipMetaWrite) {
await storage.put(metaKey, {
version: 1,
state: StreamOutputState.Pending,
attempt,
startedAt,
chunkCount: 0,
totalBytes: 0,
committedAt: null,
} satisfies StreamOutputMeta);
}
let chunkCount = 0;
let totalBytes = 0;
const bufferedChunks: Uint8Array[] = [];
let bufferedBytes = 0;
let outputCommitted = false;
const flushChunk = async (bytes: Uint8Array) => {
const invalidState = maybeInvalidState(bytes.byteLength);
if (invalidState !== undefined) {
throw invalidState;
}
// eslint-disable-next-line workers-sdk/no-unsafe-command-execution -- DO SQL exec, not child_process
storage.sql.exec(
`INSERT INTO ${STREAMING_STEP_CHUNKS_TABLE} (cache_key, attempt, chunk_index, chunk) VALUES (?, ?, ?, ?)`,
cacheKey,
attempt,
chunkCount,
bytes
);
totalBytes += bytes.byteLength;
chunkCount++;
};
try {
for await (const bytes of iterateStreamChunks(stream, signal)) {
if (bytes.byteLength === 0) {
continue;
}
if (bytes.byteLength > MAX_STREAM_OUTPUT_INPUT_CHUNK_BYTES) {
throw createOversizedStreamChunkError();
}
bufferedChunks.push(bytes);
bufferedBytes += bytes.byteLength;
// NOTE: we want chunks with fixed length,
// that's why we buffer them in-memory here
while (bufferedBytes >= chunkSizeBytes) {
const chunk = takeBufferedBytes(bufferedChunks, chunkSizeBytes);
bufferedBytes -= chunk.byteLength;
await flushChunk(chunk);
}
}
// Last chunk (remainder)
if (bufferedBytes > 0) {
await flushChunk(takeBufferedBytes(bufferedChunks, bufferedBytes));
bufferedBytes = 0;
}
const meta = {
version: 1,
state: StreamOutputState.Complete,
attempt,
startedAt,
chunkCount,
totalBytes,
committedAt: Date.now(),
} satisfies StreamOutputMeta;
const invalidState = maybeInvalidState();
if (invalidState !== undefined) {
throw invalidState;
}
if (!skipMetaWrite) {
// Transition to Committing to signal it's safe to await.
// Mock stream setup still needs the SQL chunks and returned meta, but
// must not publish the normal stream cache key because Context.do()
// treats it as a completed prior run and skips step execution logging.
await storage.put(metaKey, {
version: 1,
state: StreamOutputState.Committing,
attempt,
startedAt,
chunkCount,
totalBytes,
committedAt: null,
} satisfies StreamOutputMeta);
await storage.put(metaKey, meta);
}
outputCommitted = true;
return meta;
} catch (error) {
if (!outputCommitted) {
await rollbackStreamOutput(storage, cacheKey, attempt);
}
throw error;
}
}
export async function writeStreamOutput(options: {
storage: DurableObjectStorage;
cacheKey: string;
attempt: number;
stream: ReadableStream<unknown>;
chunkSizeBytes?: number;
signal?: AbortSignal;
timeoutTask?: Promise<never>;
skipMetaWrite?: boolean;
}): Promise<StreamOutputMeta> {
const { storage, cacheKey, attempt, timeoutTask, ...writeOptions } = options;
const writeTask = doWriteStreamOutput({
storage,
cacheKey,
attempt,
...writeOptions,
});
if (timeoutTask === undefined) {
return writeTask;
}
try {
return await Promise.race([writeTask, timeoutTask]);
} catch (error) {
if (error instanceof WorkflowTimeoutError) {
if (options.skipMetaWrite) {
void writeTask.catch(() => {});
throw error;
}
const maybeMeta = await storage.get<StreamOutputMeta>(
getStreamOutputMetaKey(cacheKey)
);
if (
maybeMeta?.attempt === attempt &&
(maybeMeta.state === StreamOutputState.Committing ||
maybeMeta.state === StreamOutputState.Complete)
) {
// Safe to await -- not in the middle of writing chunks
return await writeTask;
}
void writeTask.catch(() => {});
throw error;
}
throw error;
}
}
// ── Replay ──────────────────────────────────────────────────────────────────
export function createReplayReadableStream(options: {
storage: DurableObjectStorage;
cacheKey: string;
meta: StreamOutputMeta;
}): ReadableStream<Uint8Array> {
const { storage, cacheKey, meta } = options;
if (meta.state !== StreamOutputState.Complete) {
throw new Error("Cannot replay streamed step output before it is complete");
}
// eslint-disable-next-line workers-sdk/no-unsafe-command-execution -- DO SQL exec, not child_process
const chunkCursor = storage.sql.exec<{
chunk_index: number;
chunk: ArrayBuffer;
}>(
`SELECT chunk_index, chunk FROM ${STREAMING_STEP_CHUNKS_TABLE} WHERE cache_key = ? AND attempt = ? ORDER BY chunk_index`,
cacheKey,
meta.attempt
);
let index = 0;
return new ReadableStream<Uint8Array>({
pull(controller) {
if (index >= meta.chunkCount) {
controller.close();
return;
}
const row = chunkCursor.next();
if (row.done) {
controller.error(
new Error(`Missing chunk ${index} for streamed step output`)
);
return;
}
if (row.value.chunk_index !== index) {
controller.error(
new Error(`Missing chunk ${index} for streamed step output`)
);
return;
}
if (!(row.value.chunk instanceof ArrayBuffer)) {
controller.error(
new Error(
"Invalid chunk type returned from streaming_step_chunks table"
)
);
return;
}
controller.enqueue(new Uint8Array(row.value.chunk));
index++;
},
});
}
@@ -0,0 +1,296 @@
import Heap from "heap-js";
import type {
InstanceMetadata,
WakerPriorityEntry,
WakerPriorityType,
} from "../instance";
const wakerPriorityEntryComparator = (
a: WakerPriorityEntry,
b: WakerPriorityEntry
) => {
return a.targetTimestamp - b.targetTimestamp;
};
const enum SQLiteBoolean {
FALSE = 0,
TRUE = 1,
}
const enum EntryType {
RETRY = 0,
SLEEP = 1,
TIMEOUT = 2,
}
type PriorityQueueDBEntry = {
id: number;
created_on: string;
target_timestamp: number;
action: SQLiteBoolean;
entryType: number;
hash: string;
};
export class TimePriorityQueue {
#heap: Heap<WakerPriorityEntry> = new Heap(wakerPriorityEntryComparator);
#ctx: DurableObjectState;
constructor(ctx: DurableObjectState, _instanceMetadata: InstanceMetadata) {
this.#ctx = ctx;
this.#heap.init(this.getEntries());
}
popPastEntries(): WakerPriorityEntry[] | undefined {
// early return if there is nothing in the queue
if (this.#heap.length === 0) {
return;
}
const res: WakerPriorityEntry[] = [];
const currentTimestamp = new Date().valueOf();
// heap-js does not have a ordered iterator that doesn't consume the input so we
// peek the first one, and pop if it's old until it's empty or in the future
while (true) {
const element = this.#heap.peek();
if (element === undefined) {
break;
}
if (element.targetTimestamp > currentTimestamp) {
break;
}
// at this point, targetTimestamp is older so we can pop this node because it's no
// longer relevant
res.push(element);
this.#heap.pop();
}
this.#ctx.storage.transactionSync(() => {
for (const entry of res) {
this.removeEntryDB(entry);
}
});
return res;
}
/**
* `add` is ran using a transaction so it's race condition free, if it's ran atomically
* @param entry
*/
async add(entry: WakerPriorityEntry) {
await this.#ctx.storage.transaction(async () => {
// TODO: Handle this
// const waker = this.#env.WAKERS.idFromName(
// this.#instanceMetadata.instance.id
// );
// const wakerStub = this.#env.WAKERS.get(waker);
// We can optimise this by only calling it if time is sooner than any other
// await wakerStub.wake(
// new Date(entry.targetTimestamp),
// this.#instanceMetadata
// );
this.#heap.add(entry);
this.addEntryDB(entry);
});
}
/**
* `remove` is ran using a transaction so it's race condition free, if it's ran atomically
* @param entry
*/
remove(entry: Omit<WakerPriorityEntry, "targetTimestamp">) {
this.#ctx.storage.transactionSync(() => {
this.removeFirst((e) => {
if (e.hash === entry.hash && e.type === entry.type) {
return true;
}
return false;
});
});
}
offsetAll(offset: number) {
// Clear the entire PQ table and re-insert only the offset entries.
// We can't use the append-only add/remove pattern here because the
// UNIQUE (action, entryType, hash) constraint would conflict with
// the original action=1 rows still in the table.
this.#ctx.storage.transactionSync(() => {
const entries = this.#heap.toArray();
// Wipe the table — removes all historical add/remove rows
this.#ctx.storage.sql.exec("DELETE FROM priority_queue");
const newEntries = entries.map((value) => ({
...value,
targetTimestamp: value.targetTimestamp + offset,
}));
for (const entry of newEntries) {
this.addEntryDB(entry);
}
// re-init in-memory heap
this.#heap = new Heap(wakerPriorityEntryComparator);
this.#heap.init(newEntries);
});
}
popTypeAll(entryType: WakerPriorityType) {
this.#ctx.storage.transactionSync(() => {
this.filter((e) => e.type !== entryType);
});
}
// Idempotent, perhaps name should suggest so
async handleNextAlarm() {
const nextWakeCall = this.#heap.peek();
if (nextWakeCall === undefined) {
return;
}
// TODO: Handle this
// const waker = this.#env.WAKERS.idFromName(
// this.#instanceMetadata.instance.id
// );
// const wakerStub = this.#env.WAKERS.get(waker);
// await wakerStub.wake(
// new Date(nextWakeCall.targetTimestamp),
// this.#instanceMetadata
// );
}
getFirst(
callbackFn: (a: WakerPriorityEntry) => boolean
): WakerPriorityEntry | undefined {
// clone it so that people cant just modify the entry on the PQ
return structuredClone(this.#heap.toArray().find(callbackFn));
}
private removeFirst(callbackFn: (a: WakerPriorityEntry) => boolean) {
const elements = this.#heap.toArray();
const index = elements.findIndex(callbackFn);
if (index === -1) {
return;
}
const removedEntry = elements.splice(index, 1)[0];
this.removeEntryDB(removedEntry);
this.#heap = new Heap(wakerPriorityEntryComparator);
this.#heap.init(elements);
}
private filter(callbackFn: (a: WakerPriorityEntry) => boolean) {
const filteredElements = this.#heap.toArray().filter(callbackFn);
const removedElements = this.#heap.toArray().filter((a) => !callbackFn(a));
this.#ctx.storage.transactionSync(() => {
for (const entry of removedElements) {
this.removeEntryDB(entry);
}
});
this.#heap = new Heap(wakerPriorityEntryComparator);
this.#heap.init(filteredElements);
}
length() {
return this.#heap.length;
}
private getEntries() {
const entries = [
...this.#ctx.storage.sql.exec("SELECT * FROM priority_queue ORDER BY id"),
] as PriorityQueueDBEntry[];
const activeEntries: WakerPriorityEntry[] = [];
entries.forEach((val) => {
const entryType = toWakerPriorityType(val.entryType);
// 0 - removed
if (val.action == 0) {
const index = activeEntries.findIndex(
(activeVal) =>
val.hash == activeVal.hash && entryType == activeVal.type
);
// if it's found remove it from the active list
if (index !== -1) {
activeEntries.splice(index, 1);
}
} else {
// 1 - added
const index = activeEntries.findIndex(
(activeVal) =>
val.hash == activeVal.hash && entryType == activeVal.type
);
// if it's found remove it from the active list
if (index === -1) {
activeEntries.push({
hash: val.hash,
targetTimestamp: val.target_timestamp,
type: entryType,
});
}
}
});
return activeEntries;
}
private removeEntryDB(entry: WakerPriorityEntry) {
this.#ctx.storage.sql.exec(
`
INSERT INTO priority_queue (target_timestamp, action, entryType, hash)
VALUES (?, ?, ? ,?)
`,
entry.targetTimestamp,
SQLiteBoolean.FALSE,
fromWakerPriorityType(entry.type),
entry.hash
);
}
checkIfExistedInPast(entry: Omit<WakerPriorityEntry, "targetTimestamp">) {
return (
this.#ctx.storage.sql
.exec(
"SELECT * FROM priority_queue WHERE entryType = ? AND hash = ? AND action = ?",
fromWakerPriorityType(entry.type),
entry.hash,
0
)
.toArray().length >= 1
);
}
private addEntryDB(entry: WakerPriorityEntry) {
this.#ctx.storage.sql.exec(
`
INSERT INTO priority_queue (target_timestamp, action, entryType, hash)
VALUES (?, ?, ? ,?)
`,
entry.targetTimestamp,
SQLiteBoolean.TRUE,
fromWakerPriorityType(entry.type),
entry.hash
);
}
}
const toWakerPriorityType = (entryType: EntryType): WakerPriorityType => {
switch (entryType) {
case EntryType.RETRY:
return "retry";
case EntryType.SLEEP:
return "sleep";
case EntryType.TIMEOUT:
return "timeout";
}
};
const fromWakerPriorityType = (entryType: WakerPriorityType): EntryType => {
switch (entryType) {
case "retry":
return EntryType.RETRY;
case "sleep":
return EntryType.SLEEP;
case "timeout":
return EntryType.TIMEOUT;
default:
throw new Error(`WakerPriorityType "${entryType}" has not been handled`);
}
};
@@ -0,0 +1,87 @@
import { ms } from "itty-time";
import { z } from "zod";
export const MAX_WORKFLOW_NAME_LENGTH = 64;
export const MAX_WORKFLOW_INSTANCE_ID_LENGTH = 100;
export const MAX_STEP_NAME_LENGTH = 256;
export const ALLOWED_STRING_ID_PATTERN = "^[a-zA-Z0-9_][a-zA-Z0-9-_]*$";
const ALLOWED_WORKFLOW_INSTANCE_ID_REGEX = new RegExp(
ALLOWED_STRING_ID_PATTERN
);
const ALLOWED_WORKFLOW_NAME_REGEX = ALLOWED_WORKFLOW_INSTANCE_ID_REGEX;
// eslint-disable-next-line no-control-regex -- intentional use of control character range to detect invalid characters in workflow names
const CONTROL_CHAR_REGEX = new RegExp("[\x00-\x1F]");
export function isValidWorkflowName(name: string): boolean {
if (typeof name !== "string") {
return false;
}
if (name.length > MAX_WORKFLOW_NAME_LENGTH) {
return false;
}
return ALLOWED_WORKFLOW_NAME_REGEX.test(name);
}
export function isValidWorkflowInstanceId(id: string): boolean {
if (typeof id !== "string") {
return false;
}
if (id.length > MAX_WORKFLOW_INSTANCE_ID_LENGTH) {
return false;
}
return ALLOWED_WORKFLOW_INSTANCE_ID_REGEX.test(id);
}
export function isValidStepName(name: string): boolean {
if (name.length > MAX_STEP_NAME_LENGTH) {
return false;
}
return !CONTROL_CHAR_REGEX.test(name);
}
const STEP_CONFIG_SCHEMA = z
.object({
retries: z
.object({
delay: z.number().gte(0).or(z.string()).or(z.function()),
limit: z.number().gte(0),
backoff: z.enum(["constant", "linear", "exponential"]).optional(),
})
.strict()
.optional(),
timeout: z.number().gte(0).or(z.string()).optional(),
})
.strict();
export function isValidStepConfig(stepConfig: unknown): boolean {
const config = STEP_CONFIG_SCHEMA.safeParse(stepConfig);
if (!config.success) {
return false;
}
if (
config.data.retries !== undefined &&
typeof config.data.retries.delay !== "function" &&
Number.isNaN(ms(config.data.retries.delay))
) {
return false;
}
if (config.data.timeout !== undefined) {
const timeout = config.data.timeout;
if (timeout == 0 || Number.isNaN(ms(config.data.timeout))) {
return false;
}
}
return true;
}
@@ -0,0 +1,2 @@
export { Engine } from "./engine";
export { WorkflowBinding } from "./binding";
+251
View File
@@ -0,0 +1,251 @@
import { RpcTarget } from "cloudflare:workers";
import { computeHash } from "./lib/cache";
import { isReadableStreamLike, writeStreamOutput } from "./lib/streams";
import type { Event } from "./context";
import type { Engine } from "./engine";
export type StepSelector = {
name: string;
index?: number;
};
type UserEvent = {
type: string;
payload: unknown;
};
// KV key prefixes/values used by the modifier/mock system
export const MODIFIER_KEYS = {
REPLACE_RESULT: "replace-result-",
MOCK_STEP_ERROR: "mock-step-error-",
MOCK_EVENT: "mock-event-",
FORCE_STEP_TIMEOUT: "force-step-timeout-",
FORCE_EVENT_TIMEOUT: "force-event-timeout-",
FAILURE_INDEX: "failure-index-",
DISABLE_SLEEP: "disable-sleep-",
DISABLE_ALL_SLEEPS: "disableAllSleeps",
DISABLE_RETRY_DELAY: "disable-retry-delay-",
DISABLE_ALL_RETRY_DELAYS: "disableAllRetryDelays",
} as const;
export function isModifierKey(key: string): boolean {
return Object.values(MODIFIER_KEYS).some((v) => key.startsWith(v));
}
export class WorkflowInstanceModifier extends RpcTarget {
#engine: Engine;
#state: DurableObjectState;
constructor(engine: Engine, state: DurableObjectState) {
super();
this.#engine = engine;
this.#state = state;
}
async #getWaitForEventCacheKey(step: StepSelector): Promise<string> {
let count = 1;
if (step.index) {
count = step.index;
}
const name = `${step.name}-${count}`;
const hash = await computeHash(name);
const cacheKey = `${hash}-${count}`;
const waitForEventKey = `${cacheKey}-value`;
return waitForEventKey;
}
async #getBaseCacheKey(step: StepSelector): Promise<string> {
const hash = await computeHash(step.name);
let count = 1;
if (step.index) {
count = step.index;
}
return `${hash}-${count}`;
}
async #getStepCacheKey(step: StepSelector): Promise<string> {
const baseCacheKey = await this.#getBaseCacheKey(step);
return `${baseCacheKey}-value`;
}
#getAndIncrementCounter = async (valueKey: string, by: number) => {
const counterKey = `${MODIFIER_KEYS.FAILURE_INDEX}${valueKey}`;
const next = (await this.#state.storage.get<number>(counterKey)) ?? 1;
await this.#state.storage.put(counterKey, next + by);
return next;
};
async #getSleepStepDisableKey(step: StepSelector): Promise<string> {
let count = 1;
if (step.index) {
count = step.index;
}
const sleepNameCountHash = await computeHash(step.name + count);
return `${MODIFIER_KEYS.DISABLE_SLEEP}${sleepNameCountHash}`;
}
async disableSleeps(steps?: StepSelector[]): Promise<void> {
if (!steps) {
await this.#state.storage.put(MODIFIER_KEYS.DISABLE_ALL_SLEEPS, true);
} else {
for (const step of steps) {
const sleepDisableKey = await this.#getSleepStepDisableKey(step);
await this.#state.storage.put(sleepDisableKey, true);
}
}
}
async disableRetryDelays(steps?: StepSelector[]): Promise<void> {
if (!steps) {
await this.#state.storage.put(
MODIFIER_KEYS.DISABLE_ALL_RETRY_DELAYS,
true
);
} else {
for (const step of steps) {
const valueKey = await this.#getStepCacheKey(step);
await this.#state.storage.put(
`${MODIFIER_KEYS.DISABLE_RETRY_DELAY}${valueKey}`,
true
);
}
}
}
// step.do() flow: It first checks if a result or error is already in the cache and, if so, returns it immediately.
// If nothing is in the cache, it checks for remaining attempts and runs the user's code against the defined timeout.
// Since `step.do()` performs this initial cache check, directly changing the `valueKey` would cause it to
// assume the value was pre-cached, preventing it from writing any logs about the step's execution state.
// Storing the value under a separate key is crucial because it ensures all execution logs for the step are
// generated, rather than the step being skipped due to a premature cache hit.
async mockStepResult(step: StepSelector, stepResult: unknown): Promise<void> {
const valueKey = await this.#getStepCacheKey(step);
if (
await this.#state.storage.get(
`${MODIFIER_KEYS.REPLACE_RESULT}${valueKey}`
)
) {
throw new Error(
`[WorkflowIntrospector] Trying to mock step '${step.name}' multiple times!`
);
}
if (isReadableStreamLike(stepResult)) {
// ReadableStream is not structured-cloneable, so we consume it eagerly
// and store the chunks via the streaming infrastructure. We use attempt 0
// to distinguish mock chunks from real execution attempts (which start at 1).
const baseCacheKey = await this.#getBaseCacheKey(step);
const streamMeta = await writeStreamOutput({
storage: this.#state.storage,
cacheKey: baseCacheKey,
attempt: 0,
stream: stepResult,
skipMetaWrite: true,
});
await this.#state.storage.put(
`${MODIFIER_KEYS.REPLACE_RESULT}${valueKey}`,
{
__mockStreamOutput: true,
cacheKey: baseCacheKey,
meta: streamMeta,
}
);
} else {
await this.#state.storage.put(
`${MODIFIER_KEYS.REPLACE_RESULT}${valueKey}`,
stepResult
);
}
}
// Same logic of `mockStepResult` but stores an error instead of a value.
async mockStepError(
step: StepSelector,
error: Error,
times?: number
): Promise<void> {
const valueKey = await this.#getStepCacheKey(step);
const serializableError = {
name: error.name,
message: error.message,
};
if (
await this.#state.storage.get(
`${MODIFIER_KEYS.REPLACE_RESULT}${valueKey}`
)
) {
throw new Error(
`[WorkflowIntrospector] Trying to mock error on step '${step.name}' after mocking its result!`
);
}
if (times) {
const start = await this.#getAndIncrementCounter(valueKey, times);
const mockErrorsPuts = Array.from({ length: times }, (_, i) => {
const attempt = start + i;
const mockErrorKey = `${MODIFIER_KEYS.MOCK_STEP_ERROR}${valueKey}-${attempt}`;
return this.#state.storage.put(mockErrorKey, serializableError);
});
await Promise.all(mockErrorsPuts);
} else {
const mockErrorKey = `${MODIFIER_KEYS.MOCK_STEP_ERROR}${valueKey}`;
await this.#state.storage.put(mockErrorKey, serializableError);
}
}
async forceStepTimeout(step: StepSelector, times?: number) {
const valueKey = await this.#getStepCacheKey(step);
if (
await this.#state.storage.get(
`${MODIFIER_KEYS.REPLACE_RESULT}${valueKey}`
)
) {
throw new Error(
`[WorkflowIntrospector] Trying to force timeout on step '${step.name}' after mocking its result!`
);
}
if (times) {
const start = await this.#getAndIncrementCounter(valueKey, times);
const forceTimeouts = Array.from({ length: times }, (_, i) => {
const attempt = start + i;
const forceStepTimeoutKey = `${MODIFIER_KEYS.FORCE_STEP_TIMEOUT}${valueKey}-${attempt}`;
return this.#state.storage.put(forceStepTimeoutKey, true);
});
await Promise.all(forceTimeouts);
} else {
const forceStepTimeoutKey = `${MODIFIER_KEYS.FORCE_STEP_TIMEOUT}${valueKey}`;
await this.#state.storage.put(forceStepTimeoutKey, true);
}
}
async mockEvent(event: UserEvent): Promise<void> {
const myEvent: Event = {
timestamp: new Date(),
payload: event.payload,
type: event.type,
};
await this.#state.storage.put(
`${MODIFIER_KEYS.MOCK_EVENT}${event.type}`,
true
);
await this.#engine.receiveEvent(myEvent);
}
async forceEventTimeout(step: StepSelector): Promise<void> {
const waitForEventKey = await this.#getWaitForEventCacheKey(step);
await this.#state.storage.put(
`${MODIFIER_KEYS.FORCE_EVENT_TIMEOUT}${waitForEventKey}`,
true
);
}
}
+89
View File
@@ -0,0 +1,89 @@
export type WorkflowStepSelector = {
name: string;
index?: number;
};
export type WorkflowInstanceModifier = {
disableSleeps(steps?: WorkflowStepSelector[]): Promise<void>;
disableRetryDelays(steps?: WorkflowStepSelector[]): Promise<void>;
mockStepResult(
step: WorkflowStepSelector,
stepResult: unknown
): Promise<void>;
mockStepError(
step: WorkflowStepSelector,
error: Error,
times?: number
): Promise<void>;
forceStepTimeout(step: WorkflowStepSelector, times?: number): Promise<void>;
mockEvent(event: { type: string; payload: unknown }): Promise<void>;
forceEventTimeout(step: WorkflowStepSelector): Promise<void>;
};
export type WorkflowIntrospectionStreamResult = {
__workflowIntrospectionStreamResult: true;
chunks: Uint8Array[];
};
export type WorkflowIntrospectionOperation =
| { type: "disableSleeps"; steps?: WorkflowStepSelector[] }
| { type: "disableRetryDelays"; steps?: WorkflowStepSelector[] }
| {
type: "mockStepResult";
step: WorkflowStepSelector;
stepResult: unknown;
}
| {
type: "mockStepError";
step: WorkflowStepSelector;
error: { name: string; message: string };
times?: number;
}
| { type: "forceStepTimeout"; step: WorkflowStepSelector; times?: number }
| { type: "mockEvent"; event: { type: string; payload: unknown } }
| { type: "forceEventTimeout"; step: WorkflowStepSelector };
export type WorkflowBinding = {
unsafeGetInstanceModifier(
instanceId: string
): Promise<WorkflowInstanceModifier>;
unsafeWaitForStepResult(
instanceId: string,
name: string,
index?: number
): Promise<unknown>;
unsafeWaitForStatus(instanceId: string, status: string): Promise<void>;
unsafeGetOutputOrError(
instanceId: string,
isOutput: boolean
): Promise<unknown>;
unsafeAbort(instanceId: string, reason?: string): Promise<void>;
unsafeStartIntrospection(): Promise<string>;
unsafeSetIntrospectionOperations(
sessionId: string,
operations: WorkflowIntrospectionOperation[]
): Promise<void>;
unsafeStopIntrospection(sessionId: string): Promise<void>;
unsafeGetIntrospectionInstances(sessionId: string): Promise<string[]>;
};
export type ModifierCallback = (
modifier: WorkflowInstanceModifier
) => Promise<void>;
export interface WorkflowInstanceIntrospector {
modify(fn: ModifierCallback): Promise<WorkflowInstanceIntrospector>;
waitForStepResult(step: WorkflowStepSelector): Promise<unknown>;
waitForStatus(status: string): Promise<void>;
getOutput(): Promise<unknown>;
getError(): Promise<{ name: string; message: string }>;
dispose(): Promise<void>;
[Symbol.asyncDispose](): Promise<void>;
}
export interface WorkflowIntrospector {
modifyAll(fn: ModifierCallback): Promise<void>;
get(): Promise<WorkflowInstanceIntrospector[]>;
dispose(): Promise<void>;
[Symbol.asyncDispose](): Promise<void>;
}
@@ -0,0 +1,695 @@
import { createExecutionContext, runInDurableObject } from "cloudflare:test";
import { env } from "cloudflare:workers";
import { describe, it, vi } from "vitest";
import { InstanceEvent, InstanceStatus } from "../src";
import { WorkflowBinding } from "../src/binding";
import { setTestWorkflowCallback } from "./test-entry";
import type { WorkflowHandle } from "../src/binding";
import type { Engine, EngineLogs } from "../src/engine";
import type { WorkflowEvent } from "cloudflare:workers";
let instanceCounter = 0;
function uniqueId(prefix = "instance"): string {
return `${prefix}-${++instanceCounter}`;
}
function createBinding(): WorkflowBinding {
const ctx = createExecutionContext();
return new WorkflowBinding(ctx, {
ENGINE: env.ENGINE,
BINDING_NAME: "TEST_WORKFLOW",
WORKFLOW_NAME: "test-workflow",
});
}
async function waitUntilLogEvent(
engineStub: DurableObjectStub<Engine>,
event: InstanceEvent,
timeout = 5000
): Promise<void> {
await vi.waitUntil(
async () => {
const logs = (await engineStub.readLogs()) as EngineLogs;
const hasEvent = logs.logs.some((log) => log.event === event);
return hasEvent;
},
{ timeout }
);
}
describe("WorkflowBinding", () => {
describe("create()", () => {
it("should create an instance with provided id and params", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (event) => {
return (event as WorkflowEvent<{ key: string }>).payload;
});
const params = { key: "test-value" };
const result = await binding.create({ id, params });
expect(result.id).toBe(id);
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_SUCCESS);
const instance = await binding.get(id);
const status = await instance.status();
expect(status.output).toEqual(params);
});
it("should pass the workflow name in the event", async ({ expect }) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (event) => {
return (event as WorkflowEvent<unknown>).workflowName;
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_SUCCESS);
const instance = await binding.get(id);
const status = await instance.status();
expect(status.output).toBe("test-workflow");
});
it("should auto-generate id when not provided", async ({ expect }) => {
const binding = createBinding();
setTestWorkflowCallback(async () => "done");
const result = await binding.create();
expect(result.id).toBeDefined();
expect(result.id.length).toBeGreaterThan(0);
// Wait for the workflow to complete before the test ends so
// the fire-and-forget init() RPC settles before teardown.
const instance = await binding.get(result.id);
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
});
it("should throw WorkflowError for invalid instance id", async ({
expect,
}) => {
const binding = createBinding();
await expect(binding.create({ id: "#invalid!" })).rejects.toThrow(
"Workflow instance has invalid id"
);
});
});
describe("get()", () => {
it("should return a WorkflowHandle for an existing instance", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async () => "done");
await binding.create({ id });
const instance = await binding.get(id);
expect(instance).toMatchObject({
id,
status: expect.any(Function),
pause: expect.any(Function),
resume: expect.any(Function),
terminate: expect.any(Function),
restart: expect.any(Function),
});
// Wait for the workflow to complete before the test ends so
// the fire-and-forget init() RPC settles before teardown.
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
});
});
describe("createBatch()", () => {
it("should create multiple instances in a batch", async ({ expect }) => {
const binding = createBinding();
const ids = ["batch-1", "batch-2", "batch-3"];
setTestWorkflowCallback(async () => "done");
const results = await binding.createBatch(ids.map((id) => ({ id })));
expect(results).toHaveLength(3);
expect(results.map((r) => r.id)).toEqual(ids);
for (const id of ids) {
const instance = await binding.get(id);
expect(instance.id).toBe(id);
}
// Wait for all batch workflows to complete before the test ends
// so the fire-and-forget init() RPCs settle before teardown.
for (const id of ids) {
const instance = await binding.get(id);
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
}
});
it("should throw error for empty batch", async ({ expect }) => {
const binding = createBinding();
await expect(binding.createBatch([])).rejects.toThrow(
"WorkflowError: batchCreate should have at least 1 instance"
);
});
});
});
describe("WorkflowBinding", () => {
it("should not call dispose when sending an event to an instance", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
const receivedEvent = await step.waitForEvent("wait-for-test-event", {
type: "test-event",
timeout: "10 seconds",
});
return receivedEvent;
});
const createdInstance = await binding.create({ id });
expect(createdInstance.id).toBe(id);
const instance = await binding.get(id);
expect(instance.id).toBe(id);
const disposeSpy = vi.fn();
await runInDurableObject<Engine, void>(engineStub, (engine) => {
const originalReceiveEvent = engine.receiveEvent.bind(engine);
engine.receiveEvent = (event) => {
const result = originalReceiveEvent(event);
return Object.assign(result, {
[Symbol.dispose]: disposeSpy,
});
};
});
using _ = (await instance.sendEvent({
type: "test-event",
payload: { test: "data" },
})) as unknown as Disposable;
await vi.waitUntil(
async () => {
const status = await instance.status();
return status.status === "complete";
},
{ timeout: 5000 }
);
expect(disposeSpy).not.toHaveBeenCalled();
});
});
describe("WorkflowHandle", () => {
describe("status()", () => {
it("should return running status for a workflow waiting for an event", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.waitForEvent("wait-for-event", {
type: "some-event",
timeout: "2 seconds",
});
return "completed";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WAIT_START);
const instance = await binding.get(id);
const status = await instance.status();
expect(status.status).toBe("running");
expect(status.output).toBeNull();
expect(status.error).toBeUndefined();
// Terminate the waiting workflow so the init() RPC settles
// before teardown (the 2-second waitForEvent timeout would
// otherwise leave the workflow running past test end).
await instance.terminate();
});
it("should return complete status and output for a successful workflow", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
const expectedOutput = { result: "success", value: 42 };
setTestWorkflowCallback(async () => expectedOutput);
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_SUCCESS);
const instance = await binding.get(id);
const status = await instance.status();
expect(status.status).toBe("complete");
expect(status.output).toEqual(expectedOutput);
expect(status.error).toBeUndefined();
});
it("should return errored status and error for a failed workflow", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async () => {
throw new Error("Workflow failed intentionally");
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_FAILURE);
const instance = await binding.get(id);
const status = await instance.status();
expect(status.status).toBe("errored");
expect(status.error).toBeDefined();
expect(status.error?.message).toBe("Workflow failed intentionally");
expect(status.output).toBeNull();
});
it("should return step outputs in __LOCAL_DEV_STEP_OUTPUTS", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
const step1Result = await step.do(
"step-1",
async () => "result-from-step-1"
);
const step2Result = await step.do("step-2", async () => ({
data: "result-from-step-2",
}));
const step3Result = await step.do("step-3", async () => 123);
return { step1Result, step2Result, step3Result };
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_SUCCESS);
const instance = (await binding.get(id)) as WorkflowHandle;
const status = await instance.status();
expect(status.status).toBe("complete");
expect(status.__LOCAL_DEV_STEP_OUTPUTS).toHaveLength(3);
expect(status.__LOCAL_DEV_STEP_OUTPUTS[0]).toBe("result-from-step-1");
expect(status.__LOCAL_DEV_STEP_OUTPUTS[1]).toEqual({
data: "result-from-step-2",
});
expect(status.__LOCAL_DEV_STEP_OUTPUTS[2]).toBe(123);
});
it("should return terminated status for a terminated instance", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.waitForEvent("wait-for-event", {
type: "some-event",
timeout: "1 second",
});
return "completed";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WAIT_START);
const instance = await binding.get(id);
await instance.terminate();
const newInstance = await binding.get(id);
const status = await newInstance.status();
expect(status.status).toBe("terminated");
});
});
describe("sendEvent()", () => {
it("should deliver event payload to a waiting workflow", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
const receivedEvent = await step.waitForEvent("wait-for-event", {
type: "my-event-type",
timeout: "2 seconds",
});
return receivedEvent;
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WAIT_START);
const instance = await binding.get(id);
const eventPayload = { message: "hello", count: 42 };
await instance.sendEvent({
type: "my-event-type",
payload: eventPayload,
});
await vi.waitUntil(
async () => {
const status = await instance.status();
return status.status === "complete";
},
{ timeout: 5000 }
);
const status = await instance.status();
expect(status.output).toMatchObject({
payload: eventPayload,
type: "my-event-type",
});
});
it("should handle multiple sequential events", async ({ expect }) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
const event1 = await step.waitForEvent("wait-1", {
type: "event-type-1",
timeout: "10 seconds",
});
const event2 = await step.waitForEvent("wait-2", {
type: "event-type-2",
timeout: "10 seconds",
});
return { first: event1.payload, second: event2.payload };
});
await binding.create({ id });
const instance = await binding.get(id);
await waitUntilLogEvent(engineStub, InstanceEvent.WAIT_START);
await instance.sendEvent({
type: "event-type-1",
payload: { value: "first" },
});
// Wait for the second waitForEvent
await vi.waitUntil(
async () => {
const logs = (await engineStub.readLogs()) as EngineLogs;
const waitStarts = logs.logs.filter(
(log) => log.event === InstanceEvent.WAIT_START
);
return waitStarts.length === 2;
},
{ timeout: 5000 }
);
await instance.sendEvent({
type: "event-type-2",
payload: { value: "second" },
});
await vi.waitUntil(
async () => {
const status = await instance.status();
return status.status === "complete";
},
{ timeout: 5000 }
);
const status = await instance.status();
expect(status.output).toEqual({
first: { value: "first" },
second: { value: "second" },
});
});
});
describe("terminate()", () => {
it("should terminate a running workflow instance", async ({ expect }) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.waitForEvent("wait-for-event", {
type: "some-event",
timeout: "1 second",
});
await step.do("should not be called", async () => {
return "should not be called";
});
return "should never complete";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WAIT_START);
const instance = await binding.get(id);
await instance.terminate();
// Get a new stub since the engine was aborted
const newEngineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
const status = await runInDurableObject(newEngineStub, (engine) => {
return engine.getStatus();
});
expect(status).toBe(InstanceStatus.Terminated);
const logs = (await newEngineStub.readLogs()) as EngineLogs;
const hasTerminatedEvent = logs.logs.some(
(log) => log.event === InstanceEvent.WORKFLOW_TERMINATED
);
expect(hasTerminatedEvent).toBe(true);
// assert that step.do never started
const hasStepStart = logs.logs.some(
(log) => log.event === InstanceEvent.STEP_START
);
expect(hasStepStart).toBe(false);
});
});
describe("restart()", () => {
it("should restart a workflow instance", async ({ expect }) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.sleep("sleep", 250);
return "complete";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.WORKFLOW_SUCCESS);
let instance = await binding.get(id);
let status = await instance.status();
expect(status.status).toBe("complete");
// restart() aborts the old DO, gets a fresh stub, and calls attemptRestart()
// The service binding (USER_WORKFLOW) survives the abort, so no re-setup needed
await instance.restart();
const statusAfterRestart = await instance.status();
expect(statusAfterRestart.status).toBe("running");
// Wait for the restarted workflow to complete via status polling
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
// Verify second run completed
instance = await binding.get(id);
status = await instance.status();
expect(status.status).toBe("complete");
});
});
describe("pause()", () => {
it("should pause a running workflow", async ({ expect }) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.do("long-step", async () => {
await scheduler.wait(500);
return "result-1";
});
// step-2 should never run because pause takes effect after long-step
await step.do("step-2", async () => "result-2");
return "done";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.STEP_START);
const instance = await binding.get(id);
// Pause while long-step is in flight
await instance.pause();
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "paused";
},
{ timeout: 5000 }
);
const finalStatus = await instance.status();
expect(finalStatus.status).toBe("paused");
});
});
describe("resume()", () => {
it("should resume a paused workflow and complete it", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.do("long-step", async () => {
await scheduler.wait(500);
return "result-1";
});
await step.do("step-2", async () => "result-2");
return "all-done";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.STEP_START);
const instance = await binding.get(id);
// Pause while long-step is in flight
await instance.pause();
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "paused";
},
{ timeout: 5000 }
);
await instance.resume();
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
const finalStatus = await instance.status();
expect(finalStatus.status).toBe("complete");
expect(finalStatus.output).toBe("all-done");
});
it("should cancel a pending pause when resume is called before step finishes", async ({
expect,
}) => {
const id = uniqueId();
const binding = createBinding();
const engineStub = env.ENGINE.get(env.ENGINE.idFromName(id));
setTestWorkflowCallback(async (_event, step) => {
await step.do("long-step", async () => {
await scheduler.wait(1000);
return "long-result";
});
await step.do("step-after", async () => "final-result");
return "completed";
});
await binding.create({ id });
await waitUntilLogEvent(engineStub, InstanceEvent.STEP_START);
const instance = await binding.get(id);
// Pause while long-step is in flight — sets WaitingForPause
await instance.pause();
const statusAfterPause = await instance.status();
expect(statusAfterPause.status).toBe("waitingForPause");
// resume before the step finishes — this should cancel the pending pause
await instance.resume();
// status should go back to Running
const statusAfterResume = await instance.status();
expect(statusAfterResume.status).toBe("running");
await vi.waitUntil(
async () => {
const s = await instance.status();
return s.status === "complete";
},
{ timeout: 5000 }
);
const finalStatus = await instance.status();
expect(finalStatus.status).toBe("complete");
expect(finalStatus.output).toBe("completed");
});
});
});
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+15
View File
@@ -0,0 +1,15 @@
/* eslint-disable */
declare namespace Cloudflare {
interface Env {
ENGINE: DurableObjectNamespace<import("../src/index").Engine>;
USER_WORKFLOW: import("cloudflare:workers").WorkflowEntrypoint;
}
}
declare module "workerd:unsafe" {
const unsafe: {
abortAllDurableObjects(): Promise<void>;
};
export default unsafe;
}
@@ -0,0 +1,164 @@
import { describe, it, vi } from "vitest";
import {
DEFAULT_RETRY_DELAY_MS,
invokeDelayFunction,
} from "../../src/lib/delay";
import { DelayFunctionError } from "../../src/lib/retries";
import type { WorkflowDynamicDelayContext } from "cloudflare:workers";
const input: WorkflowDynamicDelayContext = {
ctx: {
step: { name: "step", count: 1 },
attempt: 1,
config: { retries: { limit: 5, backoff: "constant" }, timeout: 0 },
},
error: new Error("boom"),
};
const makeLogger = () => ({ warn: vi.fn(), debug: vi.fn() });
const noTimeout =
(): ((ms: number, opts?: { signal?: AbortSignal }) => Promise<void>) => () =>
new Promise<void>(() => {});
describe("invokeDelayFunction", () => {
it("returns the value a synchronous delay function produces", async ({
expect,
}) => {
const logger = makeLogger();
const result = await invokeDelayFunction(() => 1234, input, {
timeoutMs: 5000,
wait: noTimeout(),
logger,
});
expect(result).toBe(1234);
expect(logger.warn).not.toHaveBeenCalled();
expect(logger.debug).not.toHaveBeenCalled();
});
it("awaits an asynchronous delay function", async ({ expect }) => {
const logger = makeLogger();
const result = await invokeDelayFunction(async () => 1234, input, {
timeoutMs: 5000,
wait: noTimeout(),
logger,
});
expect(result).toBe(1234);
expect(logger.warn).not.toHaveBeenCalled();
expect(logger.debug).not.toHaveBeenCalled();
});
it("throws when the delay function never resolves (timeout)", async ({
expect,
}) => {
const logger = makeLogger();
const promise = invokeDelayFunction(
() => new Promise<number>(() => {}),
input,
{
timeoutMs: 5000,
wait: () => Promise.resolve(),
logger,
}
);
await expect(promise).rejects.toThrow(
new DelayFunctionError("did not return within 5 seconds")
);
expect(logger.warn).toHaveBeenCalledTimes(1);
expect(logger.warn).toHaveBeenCalledWith(
expect.stringContaining("did not resolve"),
{ step: "step", attempt: 1 }
);
expect(logger.debug).not.toHaveBeenCalled();
});
it("throws when the delay function throws synchronously", async ({
expect,
}) => {
const logger = makeLogger();
const promise = invokeDelayFunction(
() => {
throw new Error("sync boom");
},
input,
{
timeoutMs: 5000,
wait: noTimeout(),
logger,
}
);
await expect(promise).rejects.toThrow(
new DelayFunctionError("threw an error: sync boom")
);
expect(logger.warn).toHaveBeenCalledWith(
expect.stringContaining("threw"),
expect.objectContaining({ error: "sync boom" })
);
});
it("throws when the delay function rejects", async ({ expect }) => {
const logger = makeLogger();
const promise = invokeDelayFunction(
async () => {
throw new Error("async boom");
},
input,
{
timeoutMs: 5000,
wait: noTimeout(),
logger,
}
);
await expect(promise).rejects.toThrow(
new DelayFunctionError("threw an error: async boom")
);
expect(logger.warn).toHaveBeenCalledWith(
expect.stringContaining("threw"),
expect.objectContaining({ error: "async boom" })
);
});
it("logs a debug (not a warn) when the provided signal aborts before the function resolves", async ({
expect,
}) => {
const logger = makeLogger();
const abort = new AbortController();
const promise = invokeDelayFunction(
() => new Promise<number>(() => {}),
input,
{
timeoutMs: 5000,
wait: (_ms, opts) =>
new Promise<void>((resolve) => {
opts?.signal?.addEventListener("abort", () => resolve(), {
once: true,
});
}),
signal: abort.signal,
logger,
}
);
abort.abort();
await expect(promise).resolves.toBe(DEFAULT_RETRY_DELAY_MS);
expect(logger.debug).toHaveBeenCalledTimes(1);
expect(logger.debug).toHaveBeenCalledWith(
expect.stringContaining("engine shutdown"),
{ step: "step", attempt: 1 }
);
expect(logger.warn).not.toHaveBeenCalled();
});
it("throws without a logger", async ({ expect }) => {
const promise = invokeDelayFunction(
() => {
throw new Error("sync boom");
},
input,
{
timeoutMs: 5000,
wait: noTimeout(),
}
);
await expect(promise).rejects.toThrow(DelayFunctionError);
});
});
@@ -0,0 +1,51 @@
import { WorkerEntrypoint } from "cloudflare:workers";
import type { WorkflowEvent, WorkflowStep } from "cloudflare:workers";
// Test entry point — re-exports everything from src/index.ts and adds
// a TestWorkflow class that can be bound as USER_WORKFLOW via serviceBindings.
// This allows the workflow entrypoint to survive DO aborts (unlike the old
// setWorkflowEntrypoint pattern which manually mutated instance.env).
//
// NOTE: We extend WorkerEntrypoint (not WorkflowEntrypoint) because workerd
// only recognises WorkerEntrypoint subclasses for service-binding RPC.
// WorkflowEntrypoint is a higher-level abstraction used by the Workflows
// platform; for our test harness the engine just needs a target with a
// callable run() method.
export * from "../src/index";
type WorkflowCallback = (
event: unknown,
step: WorkflowStep
) => Promise<unknown>;
let __testWorkflowCallback: WorkflowCallback | undefined;
/**
* Set the workflow callback that TestWorkflow.run() will delegate to.
* Call this before creating or restarting a workflow instance in tests.
*/
export function setTestWorkflowCallback(
cb: WorkflowCallback | undefined
): void {
__testWorkflowCallback = cb;
}
/**
* A WorkerEntrypoint subclass for tests that delegates run() to a
* module-level callback. Configured as the USER_WORKFLOW service binding
* in vitest.config.ts so it survives DO aborts (unlike manual env injection).
*/
export class TestWorkflow extends WorkerEntrypoint {
async run(
event: Readonly<WorkflowEvent<unknown>>,
step: WorkflowStep
): Promise<unknown> {
if (!__testWorkflowCallback) {
throw new Error(
"TestWorkflow callback not set — call setTestWorkflowCallback() before running the workflow"
);
}
return await __testWorkflowCallback(event, step);
}
}
@@ -0,0 +1,11 @@
{
"extends": "@cloudflare/workers-tsconfig/tsconfig.json",
"compilerOptions": {
"moduleResolution": "bundler",
"types": [
"@cloudflare/workers-types",
"@cloudflare/vitest-pool-workers/types"
]
},
"include": ["./**/*.ts"]
}
+74
View File
@@ -0,0 +1,74 @@
import { env } from "cloudflare:test";
import { setTestWorkflowCallback } from "./test-entry";
import type {
DatabaseInstance,
DatabaseVersion,
DatabaseWorkflow,
Engine,
} from "../src/engine";
import type { WorkflowStep } from "cloudflare:workers";
// Track fire-and-forget init() RPC promises so they can be settled
// in afterAll hooks before vitest tears down miniflare
const pendingInits: Promise<unknown>[] = [];
/**
* Await all tracked init promises (with a timeout safety net).
* Call this in afterAll() for every test file that uses runWorkflow().
*/
export async function settlePendingWorkflows(): Promise<void> {
await Promise.allSettled(pendingInits);
pendingInits.length = 0;
}
export async function runWorkflow(
instanceId: string,
callback: (event: unknown, step: WorkflowStep) => Promise<unknown>
): Promise<DurableObjectStub<Engine>> {
const engineId = env.ENGINE.idFromName(instanceId);
const engineStub = env.ENGINE.get(engineId);
setTestWorkflowCallback(callback);
// Fire-and-forget: suppress all rejections to prevent unhandled
// rejections inside workerd (which crash the HTTP server on Windows).
// Abort errors (pause, terminate, restart) are expected; workflow
// errors (NonRetryableError, step failures) are also expected for
// tests that intentionally trigger them. Tests validate correctness
// through separate log/status assertions, not through init()'s result.
const initPromise = engineStub
.init(
12346,
{} as DatabaseWorkflow,
{} as DatabaseVersion,
{ id: instanceId } as DatabaseInstance,
{ payload: {}, timestamp: new Date(), instanceId, workflowName: "" }
)
.catch(() => {});
pendingInits.push(initPromise);
return engineStub;
}
export async function runWorkflowAndAwait(
instanceId: string,
callback: (event: unknown, step: WorkflowStep) => Promise<unknown>
): Promise<DurableObjectStub<Engine>> {
const engineId = env.ENGINE.idFromName(instanceId);
const engineStub = env.ENGINE.get(engineId);
setTestWorkflowCallback(callback);
await engineStub
.init(
12346,
{} as DatabaseWorkflow,
{} as DatabaseVersion,
{ id: instanceId } as DatabaseInstance,
{ payload: {}, timestamp: new Date(), instanceId, workflowName: "" }
)
.catch(() => {});
return engineStub;
}
@@ -0,0 +1,123 @@
import { describe, it } from "vitest";
import {
isValidStepConfig,
isValidStepName,
isValidWorkflowInstanceId,
isValidWorkflowName,
MAX_STEP_NAME_LENGTH,
MAX_WORKFLOW_INSTANCE_ID_LENGTH,
MAX_WORKFLOW_NAME_LENGTH,
} from "../src/lib/validators";
describe("Workflow name validation", () => {
it.for([
"",
NaN,
undefined,
" ",
"\n\nhello",
"w".repeat(MAX_WORKFLOW_NAME_LENGTH + 1),
"#1231231!!!!",
"-badName",
])("should reject invalid names", (value, { expect }) => {
expect(isValidWorkflowName(value as string)).toBe(false);
});
it.for([
"abc",
"NAME_123-hello",
"a-valid-string",
"w".repeat(MAX_WORKFLOW_NAME_LENGTH),
])("should accept valid names", (value, { expect }) => {
expect(isValidWorkflowName(value as string)).toBe(true);
});
});
describe("Workflow instance ID validation", () => {
it.for([
"",
NaN,
undefined,
" ",
"\n\nhello",
"w".repeat(MAX_WORKFLOW_INSTANCE_ID_LENGTH + 1),
"#1231231!!!!",
])("should reject invalid IDs", (value, { expect }) => {
expect(isValidWorkflowInstanceId(value as string)).toBe(false);
});
it.for([
"abc",
"NAME_123-hello",
"a-valid-string",
"w".repeat(MAX_WORKFLOW_INSTANCE_ID_LENGTH),
])("should accept valid IDs", (value, { expect }) => {
expect(isValidWorkflowInstanceId(value as string)).toBe(true);
});
});
describe("Workflow instance step name validation", () => {
it.for(["\x00", "w".repeat(MAX_STEP_NAME_LENGTH + 1)])(
"should reject invalid names",
(value, { expect }) => {
expect(isValidStepName(value as string)).toBe(false);
}
);
it.for([
"abc",
"NAME_123-hello",
"a-valid-string",
"w".repeat(MAX_STEP_NAME_LENGTH),
"valid step name",
])("should accept valid names", (value, { expect }) => {
expect(isValidStepName(value as string)).toBe(true);
});
});
describe("Workflow step config validation", () => {
it.for([
{ timeout: "5 years", retries: { limit: 1 } },
{ timeout: "5 years", retries: { delay: 50 } },
{ timeout: "5 years", retries: { backoff: "exponential" } },
{
timeout: "5 years",
retries: {
delay: "10 minutes",
limit: 5,
"i-like-trains": "yes".repeat(100),
},
},
{
retries: { limit: 3, delay: "i like trains", backoff: "constant" },
timeout: "10 minutes",
},
{
retries: { limit: 3, delay: 10, backoff: "constant" },
timeout: 0,
},
])("should reject invalid step configs", (value, { expect }) => {
expect(isValidStepConfig(value)).toBe(false);
});
it("should accept a dynamic delay function", ({ expect }) => {
const config = {
retries: { limit: 3, delay: () => "10 seconds", backoff: "constant" },
timeout: "10 minutes",
};
expect(isValidStepConfig(config)).toBe(true);
});
it.for([
{
retries: { limit: 0, delay: 100000, backoff: "exponential" },
timeout: "15 minutes",
},
{
retries: { limit: 5, delay: 0, backoff: "constant" },
timeout: "2 minutes",
},
])("should accept valid step configs", (value, { expect }) => {
expect(isValidStepConfig(value)).toBe(true);
});
});
+18
View File
@@ -0,0 +1,18 @@
{
"extends": "@cloudflare/workers-tsconfig/tsconfig.json",
"compilerOptions": {
"target": "esnext",
"lib": ["es2022"],
"module": "esnext",
"moduleResolution": "bundler",
"types": ["@cloudflare/workers-types/experimental"],
"noEmit": true,
"isolatedModules": true,
"allowSyntheticDefaultImports": true,
"forceConsistentCasingInFileNames": true,
"strict": true,
"skipLibCheck": true
},
"include": ["**/*.ts", "vitest.config.mts"],
"exclude": ["node_modules", "dist", "**/tests", "**/*.test.ts"]
}
+12
View File
@@ -0,0 +1,12 @@
{
"$schema": "http://turbo.build/schema.json",
"extends": ["//"],
"tasks": {
"build": {
"outputs": ["dist/**"]
},
"check:type": {
"dependsOn": ["build", "@cloudflare/workers-utils#build"]
}
}
}
@@ -0,0 +1,19 @@
import { cloudflareTest } from "@cloudflare/vitest-pool-workers";
import { defineConfig, mergeConfig } from "vitest/config";
import configShared from "../../vitest.shared";
export default mergeConfig(
configShared,
defineConfig({
plugins: [
cloudflareTest({
miniflare: {
compatibilityFlags: ["service_binding_extra_handlers"],
},
wrangler: {
configPath: "./wrangler.jsonc",
},
}),
],
})
);
+17
View File
@@ -0,0 +1,17 @@
{
"name": "workflows-shared-test",
"main": "tests/test-entry.ts",
"compatibility_date": "2025-02-04",
"compatibility_flags": ["enable_ctx_exports"],
"durable_objects": {
"bindings": [{ "name": "ENGINE", "class_name": "Engine" }],
},
"migrations": [{ "tag": "v1", "new_sqlite_classes": ["Engine"] }],
"services": [
{
"binding": "USER_WORKFLOW",
"service": "workflows-shared-test",
"entrypoint": "TestWorkflow",
},
],
}