chore: import upstream snapshot with attribution

internal-packages/testcontainers/README.md

@@ -0,0 +1,78 @@
+# Test containers
+
+Vitest utilities for writing tests against real Postgres, Prisma, Redis and ClickHouse - we don't mock
+(see the root `CLAUDE.md`), we boot containers. Also exposes a duration-weighted shard sequencer for
+splitting slow suites across CI shards.
+
+## Choosing a fixture
+
+Most tests share one set of containers per vitest worker (booted once, reset between tests) - this is
+much faster than a container per test. Reach for an isolated variant only when a test needs it.
+
+| Fixture                          | Postgres       | Redis    | ClickHouse | Use for                                 |
+| -------------------------------- | -------------- | -------- | ---------- | --------------------------------------- |
+| `redisTest`                      | -              | shared   | -          | redis-only tests                        |
+| `postgresTest`                   | shared (clone) | -        | -          | db-only tests                           |
+| `containerTest`                  | shared (clone) | shared   | shared     | the default - needs all three           |
+| `isolatedRedisTest`              | -              | per-test | -          | background redis work (see below)       |
+| `containerTestWithIsolatedRedis` | shared (clone) | per-test | shared     | background redis work + db/clickhouse   |
+| `replicationContainerTest`       | per-test       | per-test | shared     | Postgres→ClickHouse logical replication |
+
+"shared (clone)" = one Postgres per worker with a template database; each test gets a fast `CREATE
+DATABASE ... TEMPLATE` clone, so schema isn't re-pushed per test.
+
+### The background-work gotcha
+
+If a test spawns work that **outlives the test body** - a `RunEngine`, a `redis-worker` Worker, a
+`BatchQueue` - and that work isn't fully drained before the test ends, you **must** use an isolated
+redis fixture (`isolatedRedisTest` / `containerTestWithIsolatedRedis`).
+
+On the shared fixture, the leaked background loop keeps polling the one worker-scoped redis after the
+test's clients close, bleeding into the next test. The symptom is an intermittent `"Connection is
+closed"` error or a test that hangs until its timeout. `FLUSHALL` between tests does **not** fix this -
+it clears data, not live connections/loops, so per-test key prefixes won't help either. A plain
+db/redis test with no lingering background work is fine on the shared fixtures.
+
+## Sharding (`./sequencer`)
+
+CI splits the slow suites with `vitest --shard=i/N`. `DurationShardingSequencer` replaces vitest's
+default file-count split with a duration-weighted one: it reads `test-timings.json` at the repo root
+(`{ "<repo-relative path>": <ms> }`) and greedily bin-packs files so each shard does roughly equal
+_work_, not an equal _number of files_. The packing is deterministic, so every shard computes the same
+bins and runs each file exactly once.
+
+Configs opt in via:
+
+```ts
+import { DurationShardingSequencer } from "@internal/testcontainers/sequencer";
+// in defineConfig:
+test: {
+  sequence: {
+    sequencer: DurationShardingSequencer,
+  },
+}
+```
+
+### Adding tests - nothing to do
+
+New test files are discovered by vitest's glob and sharded automatically. A file with no entry in
+`test-timings.json` is given the **median** duration as a fallback, so it's still placed on exactly one
+shard - correctness never depends on the timings being present or current.
+
+What the timings affect is **balance**. A new heavy test estimated at the median can be under-weighted
+and land on an already-full shard, making that shard slower. There's headroom between the current
+makespan and the CI budget to absorb this, so it tolerates drift - but if a shard creeps toward the
+budget, refresh the timings.
+
+### Refreshing `test-timings.json`
+
+Measure each shard with the JSON reporter and write per-file `endTime - startTime` (ms), keyed by
+repo-relative path, back into `test-timings.json`. Set `GITHUB_ACTIONS=true` so suites that
+`skipIf(CI)` are excluded, matching what actually runs on CI:
+
+```bash
+GITHUB_ACTIONS=true pnpm exec vitest run --reporter=json --outputFile=/tmp/run.json
+```
+
+Stale entries for deleted/renamed files are harmless (they're simply ignored). This is a periodic
+chore, not a per-PR one.

internal-packages/testcontainers/TESTING.md

@@ -0,0 +1,60 @@
+# Fast local testing loop
+
+These tests use real Docker containers (Postgres, ClickHouse, Redis, Electric, MinIO) via testcontainers - never mocks. This guide is the fast inner loop for working on them.
+
+## Prerequisites
+
+- **Docker daemon running.** That's it - testcontainers boots its own containers. You do **not** need `pnpm run docker` (that compose stack is for running the app, and is separate).
+
+## The loop
+
+```bash
+# 1. Build upstream deps once (turbo-caches them; only re-runs when a dep changes)
+pnpm run build --filter @internal/run-engine
+
+# 2. Iterate by running vitest DIRECTLY in the package - not via `turbo run test`
+cd internal-packages/run-engine
+pnpm exec vitest run src/engine/tests/ttl.test.ts        # one file
+pnpm exec vitest src/engine/tests/ttl.test.ts            # watch mode, tightest loop
+pnpm exec vitest run src/engine/tests/ --reporter=verbose # per-test timings
+```
+
+> **Why run vitest directly, not `turbo run test`?** The `test` turbo task is cacheable
+> (`outputs: []`). A second `turbo run test` with no input change replays the cached
+> result in ~0ms instead of executing - useless when you're measuring timing. Run vitest
+> directly (or `turbo run test --force`) so tests actually run.
+
+## Measuring container boot/teardown vs test time
+
+Container lifecycle (boot + migrate + teardown) dominates these suites. To see the split:
+
+```bash
+# JSON timing lines are gated on TESTCONTAINERS_TIMING locally (always on in CI),
+# and need --disableConsoleIntercept so vitest doesn't swallow them.
+TESTCONTAINERS_TIMING=1 pnpm exec vitest run <file> --disableConsoleIntercept
+```
+
+## Approximating the 2-core CI runner locally (flake repro)
+
+To reproduce CI-like CPU pressure on a beefy local machine - useful when a test only flakes under
+the 2-core CI runner:
+
+```bash
+# cap each testcontainer's CPU/mem (TESTCONTAINERS_CPU = cores, TESTCONTAINERS_MEMORY_GB = GB),
+# and pin the test runner to 2 cores. Off unless the env vars are set.
+TESTCONTAINERS_CPU=2 TESTCONTAINERS_MEMORY_GB=2 taskset -c 0,1 pnpm exec vitest run <file>
+```
+
+Note: in practice the scoped tests here are latency/IO/sleep-bound, not CPU-bound, so this changes
+timings little - the original CI slowness was per-test container _boots_, which worker-scoping removed.
+Keep it for the cases that genuinely starve on CPU (e.g. timing races against a worker poll).
+
+## Timing harness
+
+Or use the harness, which aggregates the split for you:
+
+```bash
+node internal-packages/testcontainers/scripts/measure-test-timing.mjs \
+  src/client/client.test.ts --cwd internal-packages/clickhouse --runs 3
+# -> run 1/3  passed=true  wall=10.58s  teardown=0.67s ...
+```

internal-packages/testcontainers/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@internal/testcontainers",
+  "private": true,
+  "version": "0.0.1",
+  "main": "./src/index.ts",
+  "types": "./src/index.ts",
+  "exports": {
+    ".": "./src/index.ts",
+    "./webapp": "./src/webapp.ts",
+    "./sequencer": {
+      "types": "./src/sequencer.d.cts",
+      "default": "./src/sequencer.cjs"
+    }
+  },
+  "dependencies": {
+    "@clickhouse/client": "^1.11.1",
+    "@opentelemetry/api": "^1.9.1",
+    "@trigger.dev/database": "workspace:*",
+    "ioredis": "~5.6.0"
+  },
+  "devDependencies": {
+    "@internal/run-ops-database": "workspace:*",
+    "@testcontainers/postgresql": "^11.14.0",
+    "@testcontainers/redis": "^11.14.0",
+    "@trigger.dev/core": "workspace:*",
+    "std-env": "^3.9.0",
+    "testcontainers": "^11.14.0",
+    "tinyexec": "^0.3.0"
+  },
+  "scripts": {
+    "typecheck": "tsc --noEmit"
+  }
+}

internal-packages/testcontainers/scripts/measure-test-timing.mjs

@@ -0,0 +1,76 @@
+#!/usr/bin/env node
+// Measure testcontainers boot/teardown vs test time for a single test file.
+//
+// Usage (from any package dir, or pass --cwd):
+//   node <path>/measure-test-timing.mjs <testFile> [--cwd <packageDir>] [--runs N]
+//
+// Relies on the TESTCONTAINERS_TIMING log gate in src/logs.ts and runs vitest with
+// --disableConsoleIntercept so the JSON timing lines reach stdout.
+
+import { spawn } from "node:child_process";
+
+const args = process.argv.slice(2);
+const testFile = args.find((a) => !a.startsWith("--"));
+const cwd = valueOf("--cwd") ?? process.cwd();
+const runs = Number(valueOf("--runs") ?? "1");
+
+function valueOf(flag) {
+  const i = args.indexOf(flag);
+  return i >= 0 ? args[i + 1] : undefined;
+}
+
+if (!testFile) {
+  console.error("usage: measure-test-timing.mjs <testFile> [--cwd dir] [--runs N]");
+  process.exit(1);
+}
+
+function runOnce() {
+  return new Promise((resolve) => {
+    const child = spawn("pnpm", ["exec", "vitest", "run", testFile, "--disableConsoleIntercept"], {
+      cwd,
+      env: { ...process.env, TESTCONTAINERS_TIMING: "1" },
+    });
+
+    let out = "";
+    const collect = (buf) => (out += buf.toString());
+    child.stdout.on("data", collect);
+    child.stderr.on("data", collect);
+
+    child.on("close", () => {
+      const cleanups = [];
+      let duration = null;
+      for (const line of out.split("\n")) {
+        const trimmed = line.trim();
+        if (trimmed.startsWith("{")) {
+          try {
+            const ev = JSON.parse(trimmed);
+            if (ev.type === "cleanup") cleanups.push(ev);
+          } catch {}
+        }
+        const m = trimmed.match(/Duration\s+([\d.]+)s/);
+        if (m) duration = Number(m[1]);
+      }
+      resolve({ cleanups, duration, passed: /Tests\s+\d+ passed/.test(out) });
+    });
+  });
+}
+
+for (let i = 0; i < runs; i++) {
+  const { cleanups, duration, passed } = await runOnce();
+  const byResource = {};
+  for (const c of cleanups) {
+    const key = c.resource.split(":")[0];
+    byResource[key] ??= { totalMs: 0, count: 0 };
+    byResource[key].totalMs += c.durationMs ?? 0;
+    byResource[key].count += 1;
+  }
+  const teardownMs = Object.values(byResource).reduce((a, r) => a + r.totalMs, 0);
+  console.log(
+    `\nrun ${i + 1}/${runs}  passed=${passed}  wall=${duration}s  teardown=${(
+      teardownMs / 1000
+    ).toFixed(2)}s`
+  );
+  for (const [res, r] of Object.entries(byResource)) {
+    console.log(`  teardown ${res}: ${(r.totalMs / 1000).toFixed(2)}s over ${r.count}`);
+  }
+}

internal-packages/testcontainers/src/clickhouse.ts

@@ -0,0 +1,201 @@
+import type { ClickHouseClient } from "@clickhouse/client";
+import { readdir, readFile } from "node:fs/promises";
+import { resolve } from "node:path";
+import type { StartedTestContainer } from "testcontainers";
+import { AbstractStartedContainer, GenericContainer, Wait } from "testcontainers";
+
+const CLICKHOUSE_PORT = 9000;
+const CLICKHOUSE_HTTP_PORT = 8123;
+
+export class ClickHouseContainer extends GenericContainer {
+  private username = "test";
+  private password = "test";
+  private database = "test";
+
+  constructor(
+    image = "clickhouse/clickhouse-server:26.2.19.43-alpine@sha256:c6ad6a7eb2fb5999df3adfb8b69a0c7222c68fa9b8f6b04a088564ebbc959251"
+  ) {
+    super(image);
+    this.withExposedPorts(CLICKHOUSE_PORT, CLICKHOUSE_HTTP_PORT);
+    this.withWaitStrategy(
+      Wait.forHttp("/", CLICKHOUSE_HTTP_PORT).forResponsePredicate(
+        (response) => response === "Ok.\n"
+      )
+    );
+    this.withStartupTimeout(120_000);
+
+    // Setting this high ulimits value proactively prevents the "Too many open files" error,
+    //  especially under potentially heavy load during testing.
+    this.withUlimits({
+      nofile: {
+        hard: 262144,
+        soft: 262144,
+      },
+    });
+  }
+
+  public withDatabase(database: string): this {
+    this.database = database;
+    return this;
+  }
+
+  public withUsername(username: string): this {
+    this.username = username;
+    return this;
+  }
+
+  public withPassword(password: string): this {
+    this.password = password;
+    return this;
+  }
+
+  public override async start(): Promise<StartedClickHouseContainer> {
+    this.withEnvironment({
+      CLICKHOUSE_USER: this.username,
+      CLICKHOUSE_PASSWORD: this.password,
+      CLICKHOUSE_DB: this.database,
+    });
+
+    return new StartedClickHouseContainer(
+      await super.start(),
+      this.database,
+      this.username,
+      this.password
+    );
+  }
+}
+
+export class StartedClickHouseContainer extends AbstractStartedContainer {
+  constructor(
+    startedTestContainer: StartedTestContainer,
+    private readonly database: string,
+    private readonly username: string,
+    private readonly password: string
+  ) {
+    super(startedTestContainer);
+  }
+
+  public getPort(): number {
+    return super.getMappedPort(CLICKHOUSE_PORT);
+  }
+
+  public getHttpPort(): number {
+    return super.getMappedPort(CLICKHOUSE_HTTP_PORT);
+  }
+
+  public getUsername(): string {
+    return this.username;
+  }
+
+  public getPassword(): string {
+    return this.password;
+  }
+
+  public getDatabase(): string {
+    return this.database;
+  }
+
+  /**
+   * Gets the base HTTP URL (protocol, host and mapped port) for the ClickHouse container's HTTP interface.
+   * Example: `http://localhost:32768`
+   */
+  public getHttpUrl(): string {
+    const protocol = "http";
+    const host = this.getHost();
+    const port = this.getHttpPort();
+    return `${protocol}://${host}:${port}`;
+  }
+
+  /**
+   * Gets configuration options suitable for passing directly to `createClient({...})`
+   * from `@clickhouse/client`. Uses the HTTP interface.
+   */
+  public getClientOptions(): {
+    url?: string;
+    username: string;
+    password: string;
+    database: string;
+  } {
+    return {
+      url: this.getHttpUrl(),
+      username: this.getUsername(),
+      password: this.getPassword(),
+      database: this.getDatabase(),
+    };
+  }
+
+  /**
+   * Gets a ClickHouse connection URL for the HTTP interface with format:
+   * http://username:password@hostname:port/database
+   * @returns The ClickHouse HTTP URL string.
+   */
+  public getConnectionUrl(): string {
+    const url = new URL(this.getHttpUrl());
+
+    url.username = this.getUsername();
+    url.password = this.getPassword();
+
+    const dbName = this.getDatabase();
+    url.pathname = dbName.startsWith("/") ? dbName : `/${dbName}`;
+
+    return url.toString();
+  }
+}
+
+/**
+ * Resets data between tests on a reused ClickHouse container by truncating every base table
+ * (MergeTree etc.) in the migrated database. Views/materialized views are skipped - their target
+ * tables are base tables and get truncated too, which clears MV state. Cheaper than dropping +
+ * re-migrating, and these migrations aren't version-tracked so they can't simply be re-run.
+ */
+export async function truncateClickhouseTables(client: ClickHouseClient, database = "trigger_dev") {
+  const result = await client.query({
+    query: `SELECT name FROM system.tables WHERE database = '${database}' AND engine NOT LIKE '%View%'`,
+    format: "JSONEachRow",
+  });
+  const tables = await result.json<{ name: string }>();
+
+  for (const { name } of tables) {
+    await client.command({ query: `TRUNCATE TABLE \`${database}\`.\`${name}\`` });
+  }
+}
+
+export async function runClickhouseMigrations(client: ClickHouseClient, migrationsPath: string) {
+  // Get all the *.sql files in the migrations path
+  const queries = await getAllClickhouseMigrationQueries(migrationsPath);
+
+  for (const query of queries) {
+    await client.command({
+      query,
+    });
+  }
+}
+
+async function getAllClickhouseMigrationQueries(migrationsPath: string) {
+  const queries: string[] = [];
+  // Get all the *.sql files in the migrations path
+  const migrations = await readdir(migrationsPath);
+
+  for (const migration of migrations) {
+    const migrationPath = resolve(migrationsPath, migration);
+
+    const migrationContent = await readFile(migrationPath, "utf-8");
+
+    // Split content by goose markers
+    const parts = migrationContent.split(/--\s*\+goose\s+(Up|Down)/i);
+
+    // The array will be: ["", "Up", "up queries", "Down", "down queries"]
+    // We want the "up queries" part which is at index 2
+    if (parts.length >= 3) {
+      const upQueries = parts[2]!.trim();
+      queries.push(
+        ...upQueries
+          .split(";")
+          .filter((q) => q.trim())
+          .map((q) => q.trim())
+      );
+    }
+  }
+
+  return queries;
+}

internal-packages/testcontainers/src/docker.ts

@@ -0,0 +1,160 @@
+import { x } from "tinyexec";
+
+function stringToLines(str: string): string[] {
+  return str.split("\n").filter(Boolean);
+}
+
+function lineToWords(line: string): string[] {
+  return line.trim().split(/\s+/);
+}
+
+async function getDockerNetworks(): Promise<string[]> {
+  try {
+    const result = await x("docker", ["network", "ls" /* , "--no-trunc" */]);
+    return stringToLines(result.stdout);
+  } catch (error) {
+    console.error(error);
+    return ["error: check additional logs for more details"];
+  }
+}
+
+async function getDockerContainers(): Promise<string[]> {
+  try {
+    const result = await x("docker", ["ps", "-a" /* , "--no-trunc" */]);
+    return stringToLines(result.stdout);
+  } catch (error) {
+    console.error(error);
+    return ["error: check additional logs for more details"];
+  }
+}
+
+type DockerResource = { id: string; name: string };
+
+type DockerNetworkAttachment = DockerResource & {
+  containers: string[];
+};
+
+export async function getDockerNetworkAttachments(): Promise<DockerNetworkAttachment[]> {
+  let attachments: DockerNetworkAttachment[] = [];
+  let networks: DockerResource[] = [];
+
+  try {
+    const result = await x("docker", [
+      "network",
+      "ls",
+      "--format",
+      '{{.ID | printf "%.12s"}} {{.Name}}',
+    ]);
+
+    const lines = stringToLines(result.stdout);
+
+    for (const line of lines) {
+      const [id, name] = lineToWords(line);
+
+      if (!id || !name) {
+        continue;
+      }
+
+      networks.push({ id, name });
+    }
+  } catch (err) {
+    console.error("Failed to list docker networks:", err);
+  }
+
+  for (const { id, name } of networks) {
+    try {
+      // Get containers, one per line: id name\n
+      const containersResult = await x("docker", [
+        "network",
+        "inspect",
+        "--format",
+        '{{range $k, $v := .Containers}}{{$k | printf "%.12s"}} {{$v.Name}}\n{{end}}',
+        id,
+      ]);
+
+      const containers = stringToLines(containersResult.stdout);
+
+      attachments.push({ id, name, containers });
+    } catch (err) {
+      console.error(`Failed to inspect network ${id}:`, err);
+      attachments.push({ id, name, containers: [] });
+    }
+  }
+
+  return attachments;
+}
+
+type DockerContainerNetwork = DockerResource & {
+  networks: string[];
+};
+
+export async function getDockerContainerNetworks(): Promise<DockerContainerNetwork[]> {
+  let results: DockerContainerNetwork[] = [];
+  let containers: DockerResource[] = [];
+
+  try {
+    const result = await x("docker", [
+      "ps",
+      "-a",
+      "--format",
+      '{{.ID | printf "%.12s"}} {{.Names}}',
+    ]);
+
+    const lines = stringToLines(result.stdout);
+
+    for (const line of lines) {
+      const [id, name] = lineToWords(line);
+
+      if (!id || !name) {
+        continue;
+      }
+
+      containers.push({ id, name });
+    }
+  } catch (err) {
+    console.error("Failed to list docker containers:", err);
+  }
+
+  for (const { id, name } of containers) {
+    try {
+      const inspectResult = await x("docker", [
+        "inspect",
+        "--format",
+        '{{ range $k, $v := .NetworkSettings.Networks }}{{ $k | printf "%.12s" }} {{ $v.Name }}\n{{ end }}',
+        id,
+      ]);
+
+      const networks = stringToLines(inspectResult.stdout);
+
+      results.push({ id, name, networks });
+    } catch (err) {
+      console.error(`Failed to inspect container ${id}:`, err);
+      results.push({ id, name: String(err), networks: [] });
+    }
+  }
+
+  return results;
+}
+
+export type DockerDiagnostics = {
+  containers?: string[];
+  networks?: string[];
+  containerNetworks?: DockerContainerNetwork[];
+  networkAttachments?: DockerNetworkAttachment[];
+};
+
+export async function getDockerDiagnostics(): Promise<DockerDiagnostics> {
+  const [containers, networks, networkAttachments, containerNetworks] = await Promise.all([
+    getDockerContainers(),
+    getDockerNetworks(),
+    getDockerNetworkAttachments(),
+    getDockerContainerNetworks(),
+  ]);
+
+  return {
+    containers,
+    networks,
+    containerNetworks,
+    networkAttachments,
+  };
+}

internal-packages/testcontainers/src/heteroDedicated.test.ts

@@ -0,0 +1,26 @@
+import { expect } from "vitest";
+import { heteroRunOpsPostgresTest } from "./index.js";
+
+// The dedicated subset (NEW/PG17) has run-ops tables (TaskRun) but NOT control-plane tables
+// (Organization); the legacy side (PG14) keeps the full control-plane schema.
+heteroRunOpsPostgresTest(
+  "NEW (PG17) side has run-ops tables but NOT control-plane tables",
+  async ({ prisma14, prisma17 }) => {
+    // Cast regclass -> text: Prisma can't deserialize a bare `regclass` column.
+    const regclass = async (
+      p: { $queryRawUnsafe: (q: string) => Promise<unknown> },
+      table: string
+    ): Promise<string | null> => {
+      const rows = (await p.$queryRawUnsafe(
+        `SELECT to_regclass('"${table}"')::text AS t`
+      )) as Array<{ t: string | null }>;
+      return rows[0]?.t ?? null;
+    };
+
+    expect(await regclass(prisma14, "Organization")).not.toBeNull();
+    expect(await regclass(prisma17, "TaskRun")).not.toBeNull();
+    expect(await regclass(prisma17, "Organization")).toBeNull();
+  },
+  // Booting two PG containers and pushing two schemas on first run far exceeds vitest's 5s default.
+  120_000
+);

internal-packages/testcontainers/src/index.ts

@@ -0,0 +1,827 @@
+import { type ClickHouseClient, createClient } from "@clickhouse/client";
+import { type StartedPostgreSqlContainer, PostgreSqlContainer } from "@testcontainers/postgresql";
+import type { StartedRedisContainer } from "@testcontainers/redis";
+import { PrismaClient } from "@trigger.dev/database";
+import { RunOpsPrismaClient } from "@internal/run-ops-database";
+import Redis, { type RedisOptions } from "ioredis";
+import path from "path";
+import { type StartedNetwork, Network } from "testcontainers";
+import { type TestContext, test } from "vitest";
+import {
+  type StartedClickHouseContainer,
+  ClickHouseContainer,
+  runClickhouseMigrations,
+  truncateClickhouseTables,
+} from "./clickhouse";
+import { getTaskMetadata, logCleanup, logSetup } from "./logs";
+import { type MinIOConnectionConfig, type StartedMinIOContainer, MinIOContainer } from "./minio";
+import {
+  createClickHouseContainer,
+  createElectricContainer,
+  createPostgresContainer,
+  createRedisContainer,
+  postgresUriWithDatabase,
+  pushDatabaseSchema,
+  pushRunOpsSchema,
+  useContainer,
+  withCiResourceLimits,
+  withContainerSetup,
+} from "./utils";
+
+export { assertNonNullable, createPostgresContainer } from "./utils";
+export { laggingReplica, type LaggingModel } from "./laggingReplica";
+export { logCleanup };
+export type { MinIOConnectionConfig };
+
+type NetworkContext = { network: StartedNetwork };
+
+type PostgresContext = NetworkContext & {
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+};
+
+type RedisContext = NetworkContext & {
+  redisContainer: StartedRedisContainer;
+  redisOptions: RedisOptions;
+};
+
+type ElectricContext = {
+  electricOrigin: string;
+};
+
+export type ContainerContext = NetworkContext & PostgresContext & RedisContext & ClickhouseContext;
+export type PostgresAndRedisContext = NetworkContext & PostgresContext & RedisContext;
+export type ContainerWithElectricAndRedisContext = ContainerContext & ElectricContext;
+export type ContainerWithElectricContext = NetworkContext & PostgresContext & ElectricContext;
+
+export type {
+  StartedClickHouseContainer,
+  StartedMinIOContainer,
+  StartedNetwork,
+  StartedPostgreSqlContainer,
+  StartedRedisContainer,
+};
+
+type Use<T> = (value: T) => Promise<void>;
+
+export const network = async ({ task }: TestContext, use: Use<StartedNetwork>) => {
+  const testName = task.name;
+
+  logSetup("network: starting", { testName });
+
+  const start = Date.now();
+  const network = await new Network().start();
+  const startDurationMs = Date.now() - start;
+
+  const metadata = {
+    ...getTaskMetadata(task),
+    networkId: network.getId().slice(0, 12),
+    networkName: network.getName(),
+    startDurationMs,
+  };
+
+  logSetup("network: started", metadata);
+
+  try {
+    await use(network);
+  } finally {
+    await logCleanup("network", network.stop(), metadata);
+  }
+};
+
+export const postgresContainer = async (
+  { network, task }: { network: StartedNetwork } & TestContext,
+  use: Use<StartedPostgreSqlContainer>
+) => {
+  const { container, metadata: _metadata } = await withContainerSetup({
+    name: "postgresContainer",
+    task,
+    setup: createPostgresContainer(network),
+  });
+
+  await useContainer("postgresContainer", { container, task, use: () => use(container) });
+};
+
+export const prisma = async (
+  { postgresContainer, task }: { postgresContainer: StartedPostgreSqlContainer } & TestContext,
+  use: Use<PrismaClient>
+) => {
+  const testName = task.name;
+  const url = postgresContainer.getConnectionUri();
+
+  console.log("Initializing Prisma with URL:", url);
+
+  const prisma = new PrismaClient({
+    datasources: {
+      db: {
+        url,
+      },
+    },
+  });
+  try {
+    await use(prisma);
+  } finally {
+    await logCleanup("prisma", prisma.$disconnect(), { testName });
+  }
+};
+
+const POSTGRES_TEMPLATE_DB = "template_db";
+let pgCloneCounter = 0;
+
+type PostgresTestContext = {
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+};
+
+// --- Worker-scoped + per-test-isolated fixtures (shared by the standalone *Test and containerTest) ---
+// The pattern: boot each container ONCE per worker; isolate per test cheaply (postgres = template
+// clone, redis = FLUSHALL, clickhouse = TRUNCATE) instead of re-booting. Reset fixtures are `auto`
+// so they run for every test even if it doesn't destructure them.
+
+// Boot postgres ONCE per worker (module singleton, reaped by Ryuk on worker exit) and push the
+// schema into a dedicated template db that nothing else connects to (so CREATE DATABASE ... TEMPLATE
+// never trips on an active session).
+let workerPostgresContainer: Promise<StartedPostgreSqlContainer> | undefined;
+const getWorkerPostgresContainer = () => {
+  if (!workerPostgresContainer) {
+    workerPostgresContainer = (async () => {
+      const container = await withCiResourceLimits(new PostgreSqlContainer("docker.io/postgres:14"))
+        .withCommand(["-c", "listen_addresses=*", "-c", "wal_level=logical"])
+        .start();
+      // Create the template db explicitly via an admin connection (the same primitive the per-test
+      // clone uses) instead of relying on `prisma db push` to create a missing database. That
+      // create-if-missing path behaves differently on CI and - because push errors were swallowed -
+      // surfaced only later as a confusing "template database template_db does not exist" at clone
+      // time. Pushing into an already-existing db is the path the pre-worker-scope code always used.
+      const admin = new PrismaClient({
+        datasources: {
+          db: { url: postgresUriWithDatabase(container.getConnectionUri(), "postgres") },
+        },
+      });
+      await admin.$executeRawUnsafe(`CREATE DATABASE "${POSTGRES_TEMPLATE_DB}"`);
+      await admin.$disconnect();
+      await pushDatabaseSchema(
+        postgresUriWithDatabase(container.getConnectionUri(), POSTGRES_TEMPLATE_DB)
+      );
+      return container;
+    })();
+  }
+  return workerPostgresContainer;
+};
+
+// --- Heterogeneous PG14 + PG17 fixture ---
+// `und-x-icu` is a predefined ICU collation available by default in BOTH PG14 and PG17, so it is the
+// version-symmetric per-column source of truth for cross-version sort equality.
+export const HETERO_PINNED_ICU_COLLATION = "und-x-icu";
+
+// PG17 worker singleton mirroring getWorkerPostgresContainer. PG17 supports the ICU cluster locale
+// provider (PG14 does not - it arrived in PG15), so only this side sets the cluster locale; the real
+// cross-version guarantee is the per-column COLLATE in the proof test.
+async function bootstrapPg17TemplateContainer(
+  pushSchema: (databaseUrl: string) => Promise<unknown>
+): Promise<StartedPostgreSqlContainer> {
+  const container = await withCiResourceLimits(new PostgreSqlContainer("docker.io/postgres:17"))
+    .withCommand(["-c", "listen_addresses=*", "-c", "wal_level=logical"])
+    .withEnvironment({
+      POSTGRES_INITDB_ARGS: "--locale-provider=icu --icu-locale=en-US --encoding=UTF8",
+    })
+    .start();
+  const admin = new PrismaClient({
+    datasources: {
+      db: { url: postgresUriWithDatabase(container.getConnectionUri(), "postgres") },
+    },
+  });
+  await admin.$executeRawUnsafe(`CREATE DATABASE "${POSTGRES_TEMPLATE_DB}"`);
+  await admin.$disconnect();
+  await pushSchema(postgresUriWithDatabase(container.getConnectionUri(), POSTGRES_TEMPLATE_DB));
+  return container;
+}
+
+let workerPostgresContainer17: Promise<StartedPostgreSqlContainer> | undefined;
+const getWorkerPostgresContainer17 = () => {
+  if (!workerPostgresContainer17) {
+    workerPostgresContainer17 = bootstrapPg17TemplateContainer(pushDatabaseSchema);
+  }
+  return workerPostgresContainer17;
+};
+
+// PG17 worker singleton for the DEDICATED run-ops fixture. This is a SEPARATE container from
+// getWorkerPostgresContainer17 on purpose: that one's template db has the FULL @trigger.dev/database
+// schema pushed (consumed by heteroPostgresTest), whereas this one pushes the dedicated run-ops
+// SUBSET schema. Keeping them apart avoids a schema collision in the shared template db.
+let runOpsWorkerPostgresContainer17: Promise<StartedPostgreSqlContainer> | undefined;
+const getRunOpsWorkerPostgresContainer17 = () => {
+  if (!runOpsWorkerPostgresContainer17) {
+    runOpsWorkerPostgresContainer17 = bootstrapPg17TemplateContainer(pushRunOpsSchema);
+  }
+  return runOpsWorkerPostgresContainer17;
+};
+
+// Per test: clone a fresh database from the template (fast filesystem copy), then hand back a view
+// of the shared container whose connection points at the clone. This keeps prisma AND any code that
+// reads postgresContainer.getConnectionUri()/getDatabase() (e.g. logical replication) on the SAME
+// isolated database - and it's parallel-ready (each test owns its db).
+const clonedPostgresContainer = async ({}, use: Use<StartedPostgreSqlContainer>) => {
+  const container = await getWorkerPostgresContainer();
+  const baseUri = container.getConnectionUri();
+  const cloneDb = `test_${pgCloneCounter++}`;
+
+  await createDatabaseFromTemplate(baseUri, cloneDb);
+
+  const cloneUri = postgresUriWithDatabase(baseUri, cloneDb);
+  const view = new Proxy(container, {
+    get(target, prop, receiver) {
+      if (prop === "getConnectionUri") return () => cloneUri;
+      if (prop === "getDatabase") return () => cloneDb;
+      const value = Reflect.get(target, prop, receiver);
+      return typeof value === "function" ? value.bind(target) : value;
+    },
+  });
+
+  try {
+    await use(view);
+  } finally {
+    await dropCloneDatabase(baseUri, cloneDb);
+  }
+};
+
+const createDatabaseFromTemplate = async (baseUri: string, cloneDb: string) => {
+  const admin = new PrismaClient({
+    datasources: { db: { url: postgresUriWithDatabase(baseUri, "postgres") } },
+  });
+  try {
+    await admin.$executeRawUnsafe(
+      `CREATE DATABASE "${cloneDb}" TEMPLATE "${POSTGRES_TEMPLATE_DB}"`
+    );
+  } finally {
+    await admin.$disconnect();
+  }
+};
+
+// Best-effort drop so clones don't pile up in the worker's pg over a long suite. WITH (FORCE)
+// terminates any lingering backends (pg 13+). A failed drop is harmless - the whole container is
+// reaped on worker exit - so we never let cleanup fail the test.
+const dropCloneDatabase = async (baseUri: string, cloneDb: string) => {
+  const cleanup = new PrismaClient({
+    datasources: { db: { url: postgresUriWithDatabase(baseUri, "postgres") } },
+  });
+  try {
+    await cleanup.$executeRawUnsafe(`DROP DATABASE IF EXISTS "${cloneDb}" WITH (FORCE)`);
+  } catch {
+    // ignore - reaped with the container anyway
+  } finally {
+    await cleanup.$disconnect();
+  }
+};
+
+// A second migrated-but-empty database on the same worker postgres, cloned from the schema
+// template. For tests that need to simulate a read replica that hasn't caught up: schema
+// present, rows absent. Lazy - only booted when a test destructures it.
+const schemaOnlyPrismaFixture = async ({}: {}, use: Use<PrismaClient>) => {
+  const container = await getWorkerPostgresContainer();
+  const baseUri = container.getConnectionUri();
+  const cloneDb = `schema_only_${pgCloneCounter++}`;
+
+  await createDatabaseFromTemplate(baseUri, cloneDb);
+
+  const prisma = new PrismaClient({
+    datasources: { db: { url: postgresUriWithDatabase(baseUri, cloneDb) } },
+  });
+  try {
+    await use(prisma);
+  } finally {
+    await logCleanup("schemaOnlyPrisma", prisma.$disconnect());
+    await dropCloneDatabase(baseUri, cloneDb);
+  }
+};
+
+const prismaFromContainer = async (
+  { postgresContainer }: { postgresContainer: StartedPostgreSqlContainer },
+  use: Use<PrismaClient>
+) => {
+  const prisma = new PrismaClient({
+    datasources: { db: { url: postgresContainer.getConnectionUri() } },
+  });
+  try {
+    await use(prisma);
+  } finally {
+    await logCleanup("prisma", prisma.$disconnect());
+  }
+};
+
+export const postgresTest = test.extend<PostgresTestContext>({
+  postgresContainer: clonedPostgresContainer,
+  prisma: prismaFromContainer,
+});
+
+type HeteroPostgresTestContext = {
+  // PG14 (legacy / control-plane DB analog)
+  postgresContainer14: StartedPostgreSqlContainer;
+  prisma14: PrismaClient;
+  uri14: string;
+  // PG17 (new / dedicated run-ops DB analog)
+  postgresContainer17: StartedPostgreSqlContainer;
+  prisma17: PrismaClient;
+  uri17: string;
+  pinnedCollation: string; // === HETERO_PINNED_ICU_COLLATION
+};
+
+// Hands a test two prisma clients + two connection URIs (one PG14, one PG17) over the same migrated
+// schema, each on a fresh per-test clone of its version's template. Consumed only by explicit
+// cross-version test files - never wired into a product fixture.
+export const heteroPostgresTest = test.extend<HeteroPostgresTestContext>({
+  postgresContainer14: async ({}, use) => {
+    await use(await getWorkerPostgresContainer());
+  },
+  postgresContainer17: async ({}, use) => {
+    await use(await getWorkerPostgresContainer17());
+  },
+  uri14: async ({ postgresContainer14 }, use) => {
+    const baseUri = postgresContainer14.getConnectionUri();
+    const cloneDb = `hetero14_${pgCloneCounter++}`;
+    await createDatabaseFromTemplate(baseUri, cloneDb);
+    try {
+      await use(postgresUriWithDatabase(baseUri, cloneDb));
+    } finally {
+      await dropCloneDatabase(baseUri, cloneDb);
+    }
+  },
+  uri17: async ({ postgresContainer17 }, use) => {
+    const baseUri = postgresContainer17.getConnectionUri();
+    const cloneDb = `hetero17_${pgCloneCounter++}`;
+    await createDatabaseFromTemplate(baseUri, cloneDb);
+    try {
+      await use(postgresUriWithDatabase(baseUri, cloneDb));
+    } finally {
+      await dropCloneDatabase(baseUri, cloneDb);
+    }
+  },
+  prisma14: async ({ uri14 }, use) => {
+    const prisma = new PrismaClient({ datasources: { db: { url: uri14 } } });
+    try {
+      await use(prisma);
+    } finally {
+      await prisma.$disconnect();
+    }
+  },
+  prisma17: async ({ uri17 }, use) => {
+    const prisma = new PrismaClient({ datasources: { db: { url: uri17 } } });
+    try {
+      await use(prisma);
+    } finally {
+      await prisma.$disconnect();
+    }
+  },
+  pinnedCollation: async ({}, use) => {
+    await use(HETERO_PINNED_ICU_COLLATION);
+  },
+});
+
+type HeteroRunOpsPostgresTestContext = {
+  // PG14 (legacy / control-plane DB analog) — full @trigger.dev/database control-plane schema.
+  postgresContainer14: StartedPostgreSqlContainer;
+  prisma14: PrismaClient;
+  uri14: string;
+  // PG17 (new / dedicated run-ops DB) — the @internal/run-ops-database SUBSET schema.
+  postgresContainer17: StartedPostgreSqlContainer;
+  prisma17: RunOpsPrismaClient;
+  uri17: string;
+};
+
+// Additive sibling of heteroPostgresTest for the dedicated run-ops migration: prisma14 is the full
+// control-plane schema on PG14 (legacy), prisma17 is a RunOpsPrismaClient over the dedicated SUBSET
+// schema on a SEPARATE PG17 container. Lets a test prove the two sides carry different schemas
+// without disturbing the existing heteroPostgresTest (which keeps the full schema on both sides).
+export const heteroRunOpsPostgresTest = test.extend<HeteroRunOpsPostgresTestContext>({
+  postgresContainer14: async ({}, use) => {
+    await use(await getWorkerPostgresContainer());
+  },
+  postgresContainer17: async ({}, use) => {
+    await use(await getRunOpsWorkerPostgresContainer17());
+  },
+  uri14: async ({ postgresContainer14 }, use) => {
+    const baseUri = postgresContainer14.getConnectionUri();
+    const cloneDb = `heteroRunOps14_${pgCloneCounter++}`;
+    await createDatabaseFromTemplate(baseUri, cloneDb);
+    try {
+      await use(postgresUriWithDatabase(baseUri, cloneDb));
+    } finally {
+      await dropCloneDatabase(baseUri, cloneDb);
+    }
+  },
+  uri17: async ({ postgresContainer17 }, use) => {
+    const baseUri = postgresContainer17.getConnectionUri();
+    const cloneDb = `heteroRunOps17_${pgCloneCounter++}`;
+    await createDatabaseFromTemplate(baseUri, cloneDb);
+    try {
+      await use(postgresUriWithDatabase(baseUri, cloneDb));
+    } finally {
+      await dropCloneDatabase(baseUri, cloneDb);
+    }
+  },
+  prisma14: async ({ uri14 }, use) => {
+    const prisma = new PrismaClient({ datasources: { db: { url: uri14 } } });
+    try {
+      await use(prisma);
+    } finally {
+      await prisma.$disconnect();
+    }
+  },
+  prisma17: async ({ uri17 }, use) => {
+    const prisma = new RunOpsPrismaClient({ datasources: { db: { url: uri17 } } });
+    try {
+      await use(prisma);
+    } finally {
+      await prisma.$disconnect();
+    }
+  },
+});
+
+export const redisContainer = async (
+  { network, task }: { network: StartedNetwork } & TestContext,
+  use: Use<StartedRedisContainer>
+) => {
+  const { container, metadata: _metadata } = await withContainerSetup({
+    name: "redisContainer",
+    task,
+    setup: createRedisContainer({
+      port: 6379,
+      network,
+    }),
+  });
+
+  await useContainer("redisContainer", { container, task, use: () => use(container) });
+};
+
+export const redisOptions = async (
+  { redisContainer }: { redisContainer: StartedRedisContainer },
+  use: Use<RedisOptions>
+) => {
+  const options: RedisOptions = {
+    host: redisContainer.getHost(),
+    port: redisContainer.getPort(),
+    password: redisContainer.getPassword(),
+    maxRetriesPerRequest: 20, // Lower the retry attempts
+    retryStrategy(times) {
+      const delay = Math.min(times * 50, 2000);
+      return delay;
+    },
+    connectTimeout: 10000, // 10 seconds
+    // Add more robust connection options
+    enableOfflineQueue: true,
+    reconnectOnError: (err) => {
+      const targetError = "READONLY";
+      if (err.message.includes(targetError)) {
+        return true;
+      }
+      return false;
+    },
+    enableAutoPipelining: true,
+    autoResubscribe: true,
+    autoResendUnfulfilledCommands: true,
+    lazyConnect: false,
+    showFriendlyErrorStack: true,
+  };
+
+  console.log("Redis options", options);
+
+  await use(options);
+};
+
+// Worker-scoped redis: booted once per worker, FLUSHALL per test. Big win for redis-heavy files
+// (buffer.test.ts: 88 boots -> 1). Safe ONLY for tests that don't leave background redis work
+// (a Worker loop, BatchQueue) running past the test body - use isolatedRedisTest for those.
+const bootWorkerRedis = async ({}, use: Use<StartedRedisContainer>) => {
+  const { container } = await createRedisContainer({ port: 6379 });
+  try {
+    await use(container);
+  } finally {
+    await container.stop({ timeout: 0 });
+  }
+};
+
+const flushRedis = async (
+  { redisContainer }: { redisContainer: StartedRedisContainer },
+  use: Use<void>
+) => {
+  const redis = new Redis({
+    host: redisContainer.getHost(),
+    port: redisContainer.getPort(),
+    password: redisContainer.getPassword(),
+    maxRetriesPerRequest: 3,
+  });
+  try {
+    await redis.flushall();
+  } finally {
+    redis.disconnect();
+  }
+  await use();
+};
+
+type RedisTestContext = {
+  redisContainer: StartedRedisContainer;
+  resetRedis: void;
+  redisOptions: RedisOptions;
+};
+
+// Worker-scoped redis (boots once, FLUSHALL between tests). Use isolatedRedisTest for tests that run
+// background redis work (redis-worker Workers, BatchQueue) past the test body - see its note + README.
+export const redisTest = test.extend<RedisTestContext>({
+  redisContainer: [bootWorkerRedis, { scope: "worker" }],
+  resetRedis: [flushRedis, { auto: true }],
+  redisOptions,
+});
+
+// Per-test redis for tests with background redis work (redis-worker Workers, BatchQueue) that can
+// outlive the test body - a shared redis would let leaked work hit a closed connection / next test
+// ("Connection is closed"). Boot is kept fast (see createRedisContainer).
+export const isolatedRedisTest = test.extend<RedisContext>({
+  network,
+  redisContainer,
+  redisOptions,
+});
+
+const electricOrigin = async (
+  {
+    postgresContainer,
+    network,
+    task,
+  }: { postgresContainer: StartedPostgreSqlContainer; network: StartedNetwork } & TestContext,
+  use: Use<string>
+) => {
+  const {
+    origin,
+    container,
+    metadata: _metadata,
+  } = await withContainerSetup({
+    name: "electricContainer",
+    task,
+    setup: createElectricContainer(postgresContainer, network),
+  });
+
+  await useContainer("electricContainer", { container, task, use: () => use(origin) });
+};
+
+const clickhouseContainer = async (
+  { network, task }: { network: StartedNetwork } & TestContext,
+  use: Use<StartedClickHouseContainer>
+) => {
+  const { container, metadata: _metadata } = await withContainerSetup({
+    name: "clickhouseContainer",
+    task,
+    setup: createClickHouseContainer(network),
+  });
+
+  await useContainer("clickhouseContainer", { container, task, use: () => use(container) });
+};
+
+const clickhouseClient = async (
+  { clickhouseContainer, task }: { clickhouseContainer: StartedClickHouseContainer } & TestContext,
+  use: Use<ClickHouseClient>
+) => {
+  const testName = task.name;
+  const client = createClient({ url: clickhouseContainer.getConnectionUrl() });
+
+  try {
+    await use(client);
+  } finally {
+    await logCleanup("clickhouseClient", client.close(), { testName });
+  }
+};
+
+type ClickhouseContext = {
+  network: StartedNetwork;
+  clickhouseContainer: StartedClickHouseContainer;
+  clickhouseClient: ClickHouseClient;
+};
+
+const clickhouseMigrationsPath = path.resolve(__dirname, "../../clickhouse/schema");
+
+type ClickhouseTestContext = {
+  clickhouseContainer: StartedClickHouseContainer;
+  resetClickhouse: void;
+  clickhouseClient: ClickHouseClient;
+};
+
+// Boot + migrate clickhouse once per worker.
+const bootWorkerClickhouse = async ({}, use: Use<StartedClickHouseContainer>) => {
+  const container = await withCiResourceLimits(new ClickHouseContainer()).start();
+  const client = createClient({ url: container.getConnectionUrl() });
+  await client.ping();
+  await runClickhouseMigrations(client, clickhouseMigrationsPath);
+  await client.close();
+  try {
+    await use(container);
+  } finally {
+    await container.stop({ timeout: 0 });
+  }
+};
+
+// Per test: truncate all tables on the shared clickhouse (auto fixture so it runs for every test).
+const truncateClickhouseFixture = async (
+  { clickhouseContainer }: { clickhouseContainer: StartedClickHouseContainer },
+  use: Use<void>
+) => {
+  const client = createClient({ url: clickhouseContainer.getConnectionUrl() });
+  await truncateClickhouseTables(client);
+  await client.close();
+  await use();
+};
+
+const scopedClickhouseClient = async (
+  { clickhouseContainer }: { clickhouseContainer: StartedClickHouseContainer },
+  use: Use<ClickHouseClient>
+) => {
+  const client = createClient({ url: clickhouseContainer.getConnectionUrl() });
+  try {
+    await use(client);
+  } finally {
+    await logCleanup("clickhouseClient", client.close());
+  }
+};
+
+export const clickhouseTest = test.extend<ClickhouseTestContext>({
+  clickhouseContainer: [bootWorkerClickhouse, { scope: "worker" }],
+  resetClickhouse: [truncateClickhouseFixture, { auto: true }],
+  clickhouseClient: scopedClickhouseClient,
+});
+
+// NOTE: per-test containers (not worker-scoped) - the replication package does logical replication
+// (slots/publications/REPLICA IDENTITY), which doesn't play nicely with a shared container +
+// template-clone. A dedicated container per test is the correct, isolated choice here.
+export const postgresAndRedisTest = test.extend<PostgresAndRedisContext>({
+  network,
+  postgresContainer,
+  prisma,
+  redisContainer,
+  redisOptions,
+});
+
+type ContainerTestContext = {
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+  schemaOnlyPrisma: PrismaClient;
+  redisContainer: StartedRedisContainer;
+  resetRedis: void;
+  redisOptions: RedisOptions;
+  clickhouseContainer: StartedClickHouseContainer;
+  resetClickhouse: void;
+  clickhouseClient: ClickHouseClient;
+};
+
+// The workhorse fixture (~36 files). Postgres (template-clone), Redis (FLUSHALL) and ClickHouse
+// (truncate) all boot once per worker - no per-test container boots. Use containerTestWithIsolatedRedis
+// for tests that run background redis work (BatchQueue, redis-worker Workers) past the test body.
+export const containerTest = test.extend<ContainerTestContext>({
+  postgresContainer: clonedPostgresContainer,
+  prisma: prismaFromContainer,
+  schemaOnlyPrisma: schemaOnlyPrismaFixture,
+  redisContainer: [bootWorkerRedis, { scope: "worker" }],
+  resetRedis: [flushRedis, { auto: true }],
+  redisOptions,
+  clickhouseContainer: [bootWorkerClickhouse, { scope: "worker" }],
+  resetClickhouse: [truncateClickhouseFixture, { auto: true }],
+  clickhouseClient: scopedClickhouseClient,
+});
+
+type ContainerWithIsolatedRedisContext = {
+  network: StartedNetwork;
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+  redisContainer: StartedRedisContainer;
+  redisOptions: RedisOptions;
+  clickhouseContainer: StartedClickHouseContainer;
+  resetClickhouse: void;
+  clickhouseClient: ClickHouseClient;
+};
+
+// Same as containerTest but Redis is PER-TEST - for tests whose background redis work (BatchQueue,
+// Workers) outlives the test body and would otherwise hit a closed/shared connection.
+export const containerTestWithIsolatedRedis = test.extend<ContainerWithIsolatedRedisContext>({
+  network,
+  postgresContainer: clonedPostgresContainer,
+  prisma: prismaFromContainer,
+  redisContainer,
+  redisOptions,
+  clickhouseContainer: [bootWorkerClickhouse, { scope: "worker" }],
+  resetClickhouse: [truncateClickhouseFixture, { auto: true }],
+  clickhouseClient: scopedClickhouseClient,
+});
+
+type ContainerWithIsolatedRedisNoClickhouseContext = {
+  network: StartedNetwork;
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+  redisContainer: StartedRedisContainer;
+  redisOptions: RedisOptions;
+};
+
+// Like containerTestWithIsolatedRedis (template-clone Postgres + per-test Redis) but with no
+// ClickHouse - for suites that touch Postgres + Redis but never ClickHouse, avoiding its boot+migrate.
+export const containerTestWithIsolatedRedisNoClickhouse =
+  test.extend<ContainerWithIsolatedRedisNoClickhouseContext>({
+    network,
+    postgresContainer: clonedPostgresContainer,
+    prisma: prismaFromContainer,
+    redisContainer,
+    redisOptions,
+  });
+
+// For tests that exercise the Postgres -> ClickHouse logical-replication pipeline (WAL slots,
+// publications, REPLICA IDENTITY). These need a dedicated Postgres per test - the worker-scoped +
+// template-clone model used by containerTest doesn't carry logical replication across cloned dbs.
+// Postgres is per-test (the WAL slot/publication lives in the db it writes to); ClickHouse is
+// worker-scoped + truncated (the pipeline writes pg->clickhouse and a shared+truncated clickhouse is
+// fine). Redis is per-test too (background work safety, same as containerTest).
+type ReplicationContainerTestContext = {
+  network: StartedNetwork;
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+  redisContainer: StartedRedisContainer;
+  redisOptions: RedisOptions;
+  clickhouseContainer: StartedClickHouseContainer;
+  resetClickhouse: void;
+  clickhouseClient: ClickHouseClient;
+};
+
+export const replicationContainerTest = test.extend<ReplicationContainerTestContext>({
+  network,
+  postgresContainer,
+  prisma,
+  redisContainer,
+  redisOptions,
+  clickhouseContainer: [bootWorkerClickhouse, { scope: "worker" }],
+  resetClickhouse: [truncateClickhouseFixture, { auto: true }],
+  clickhouseClient: scopedClickhouseClient,
+});
+
+export const containerWithElectricTest = test.extend<ContainerWithElectricContext>({
+  network,
+  postgresContainer,
+  prisma,
+  electricOrigin,
+});
+
+export const containerWithElectricAndRedisTest = test.extend<ContainerWithElectricAndRedisContext>({
+  network,
+  postgresContainer,
+  prisma,
+  redisContainer,
+  redisOptions,
+  electricOrigin,
+  clickhouseContainer,
+  clickhouseClient,
+});
+
+// Boot minio once per worker; reset the bucket per test (auto fixture).
+const bootWorkerMinio = async ({}, use: Use<StartedMinIOContainer>) => {
+  const container = await withCiResourceLimits(new MinIOContainer()).start();
+  try {
+    await use(container);
+  } finally {
+    await container.stop({ timeout: 0 });
+  }
+};
+
+const minioReset = async (
+  { minioContainer }: { minioContainer: StartedMinIOContainer },
+  use: Use<void>
+) => {
+  await minioContainer.resetBucket();
+  await use();
+};
+
+const minioConfig = async (
+  { minioContainer }: { minioContainer: StartedMinIOContainer },
+  use: Use<MinIOConnectionConfig>
+) => {
+  await use(minioContainer.getConnectionConfig());
+};
+
+type MinioTestContext = {
+  minioContainer: StartedMinIOContainer;
+  resetMinio: void;
+  minioConfig: MinIOConnectionConfig;
+};
+
+export const minioTest = test.extend<MinioTestContext>({
+  minioContainer: [bootWorkerMinio, { scope: "worker" }],
+  resetMinio: [minioReset, { auto: true }],
+  minioConfig,
+});
+
+type PostgresAndMinioTestContext = {
+  postgresContainer: StartedPostgreSqlContainer;
+  prisma: PrismaClient;
+  minioContainer: StartedMinIOContainer;
+  resetMinio: void;
+  minioConfig: MinIOConnectionConfig;
+};
+
+export const postgresAndMinioTest = test.extend<PostgresAndMinioTestContext>({
+  postgresContainer: clonedPostgresContainer,
+  prisma: prismaFromContainer,
+  minioContainer: [bootWorkerMinio, { scope: "worker" }],
+  resetMinio: [minioReset, { auto: true }],
+  minioConfig,
+});

internal-packages/testcontainers/src/laggingReplica.ts

@@ -0,0 +1,86 @@
+// A read-replica that has NOT caught up to its primary, for reproducing read-your-writes hazards the
+// zero-lag single-DB test harness can't surface. Wraps a real Prisma client; for the configured
+// models it serves STALE reads instead of live ones. Other models and all writes forward untouched.
+// Modes: "missing" (row not replicated yet -> null/[]/0, *OrThrow throws); "frozen" (row out of date
+// -> returns the provided pre-write snapshot rows, matched on top-level scalar `where` equality).
+// `wasHit` asserts the stale replica was actually consulted.
+
+const READ_METHODS = new Set([
+  "findFirst",
+  "findUnique",
+  "findFirstOrThrow",
+  "findUniqueOrThrow",
+  "findMany",
+  "count",
+]);
+
+export type LaggingModel =
+  | { model: string; mode: "missing" }
+  | { model: string; mode: "frozen"; rows: readonly Record<string, unknown>[] };
+
+// Match a frozen row against a Prisma `where` using top-level scalar equality only (enough for the
+// friendlyId / id / environmentId lookups these reads use); nested filters are treated as "matches".
+function whereMatches(row: Record<string, unknown>, where: unknown): boolean {
+  if (where == null || typeof where !== "object") return true;
+  return Object.entries(where as Record<string, unknown>).every(([key, val]) => {
+    if (val !== null && typeof val === "object") return true;
+    return row[key] === val;
+  });
+}
+
+export function laggingReplica<C extends object>(
+  real: C,
+  configs: readonly LaggingModel[]
+): { client: C; wasHit: (model?: string) => boolean } {
+  const byModel = new Map(configs.map((c) => [c.model, c]));
+  const hits = new Set<string>();
+
+  const makeModelProxy = (modelName: string, realModel: object, cfg: LaggingModel) =>
+    new Proxy(realModel, {
+      get(target, prop) {
+        if (typeof prop === "string" && READ_METHODS.has(prop)) {
+          return async (args?: { where?: unknown }) => {
+            hits.add(modelName);
+            if (cfg.mode === "missing") {
+              if (prop === "findMany") return [];
+              if (prop === "count") return 0;
+              if (prop === "findFirstOrThrow" || prop === "findUniqueOrThrow") {
+                throw new Error(
+                  `laggingReplica: ${modelName}.${prop} - row not visible on replica yet`
+                );
+              }
+              return null;
+            }
+            const matched = cfg.rows.filter((r) => whereMatches(r, args?.where));
+            if (prop === "findMany") return matched;
+            if (prop === "count") return matched.length;
+            const first = matched[0] ?? null;
+            if ((prop === "findFirstOrThrow" || prop === "findUniqueOrThrow") && !first) {
+              throw new Error(`laggingReplica: ${modelName}.${prop} - no frozen row matched`);
+            }
+            return first;
+          };
+        }
+        const value = (target as Record<string | symbol, unknown>)[prop];
+        return typeof value === "function"
+          ? (value as (...a: unknown[]) => unknown).bind(target)
+          : value;
+      },
+    });
+
+  const client = new Proxy(real, {
+    get(target, prop) {
+      const cfg = typeof prop === "string" ? byModel.get(prop) : undefined;
+      const delegate = cfg ? (target as Record<string, unknown>)[prop as string] : undefined;
+      if (cfg && delegate && typeof delegate === "object") {
+        return makeModelProxy(prop as string, delegate, cfg);
+      }
+      const value = (target as Record<string | symbol, unknown>)[prop];
+      return typeof value === "function"
+        ? (value as (...a: unknown[]) => unknown).bind(target)
+        : value;
+    },
+  }) as C;
+
+  return { client, wasHit: (model) => (model ? hits.has(model) : hits.size > 0) };
+}

internal-packages/testcontainers/src/logs.ts

@@ -0,0 +1,105 @@
+import { env, isCI } from "std-env";
+import type { TestContext } from "vitest";
+import type { DockerDiagnostics } from "./docker";
+import { getDockerDiagnostics } from "./docker";
+import type { StartedTestContainer } from "testcontainers";
+
+let setupOrder = 0;
+
+// Emit timing JSON in CI, or locally when TESTCONTAINERS_TIMING is set (drives the local timing harness)
+const emitTimingLogs = isCI || !!env.TESTCONTAINERS_TIMING;
+
+export function logSetup(resource: string, metadata: Record<string, unknown>) {
+  const order = setupOrder++;
+
+  if (!emitTimingLogs) {
+    return;
+  }
+
+  console.log(
+    JSON.stringify({
+      type: "setup",
+      order,
+      resource,
+      timestamp: new Date().toISOString(),
+      ...metadata,
+    })
+  );
+}
+
+export function getContainerMetadata(container: StartedTestContainer) {
+  return {
+    containerName: container.getName(),
+    containerId: container.getId().slice(0, 12),
+    containerNetworkNames: container.getNetworkNames(),
+  };
+}
+
+export function getTaskMetadata(task: TestContext["task"]) {
+  return {
+    testName: task.name,
+  };
+}
+
+let cleanupOrder = 0;
+let activeCleanups = 0;
+
+/**
+ * Logs the cleanup of a resource.
+ * @param resource - The resource that is being cleaned up.
+ * @param promise - The cleanup promise to await..
+ */
+export async function logCleanup(
+  resource: string,
+  promise: Promise<unknown>,
+  metadata: Record<string, unknown> = {}
+) {
+  const start = new Date();
+  const order = cleanupOrder++;
+  const activeAtStart = ++activeCleanups;
+
+  let error: unknown = null;
+
+  try {
+    await promise;
+  } catch (err) {
+    error = err instanceof Error ? err.message : String(err);
+  }
+
+  const end = new Date();
+  const durationMs = end.getTime() - start.getTime();
+  const activeAtEnd = --activeCleanups;
+  const parallel = activeAtStart > 1 || activeAtEnd > 0;
+
+  if (!emitTimingLogs) {
+    return;
+  }
+
+  let dockerDiagnostics: DockerDiagnostics = {};
+
+  // Only run docker diagnostics if there was an error or cleanup took longer than 5s
+  if (error || durationMs > 5000 || env.DOCKER_DIAGNOSTICS) {
+    try {
+      dockerDiagnostics = await getDockerDiagnostics();
+    } catch (diagnosticErr) {
+      console.error("Failed to get docker diagnostics:", diagnosticErr);
+    }
+  }
+
+  console.log(
+    JSON.stringify({
+      type: "cleanup",
+      order,
+      resource,
+      durationMs,
+      start: start.toISOString(),
+      end: end.toISOString(),
+      parallel,
+      error,
+      activeAtStart,
+      activeAtEnd,
+      ...metadata,
+      ...dockerDiagnostics,
+    })
+  );
+}

internal-packages/testcontainers/src/minio.ts

@@ -0,0 +1,145 @@
+import type { StartedTestContainer } from "testcontainers";
+import { AbstractStartedContainer, GenericContainer, Wait } from "testcontainers";
+import { x } from "tinyexec";
+
+const MINIO_PORT = 9000;
+
+export type MinIOConnectionConfig = {
+  baseUrl: string;
+  accessKeyId: string;
+  secretAccessKey: string;
+  region: string;
+};
+
+export class MinIOContainer extends GenericContainer {
+  private accessKeyId = "minioadmin";
+  private secretAccessKey = "minioadmin";
+  private region = "us-east-1";
+
+  constructor(image = "minio/minio:latest") {
+    super(image);
+    this.withExposedPorts(MINIO_PORT);
+    this.withCommand(["server", "/data"]);
+    this.withWaitStrategy(Wait.forLogMessage(/API:/));
+    this.withStartupTimeout(120_000);
+  }
+
+  public withAccessKeyId(accessKeyId: string): this {
+    this.accessKeyId = accessKeyId;
+    return this;
+  }
+
+  public withSecretAccessKey(secretAccessKey: string): this {
+    this.secretAccessKey = secretAccessKey;
+    return this;
+  }
+
+  public withRegion(region: string): this {
+    this.region = region;
+    return this;
+  }
+
+  public override async start(): Promise<StartedMinIOContainer> {
+    this.withEnvironment({
+      MINIO_ROOT_USER: this.accessKeyId,
+      MINIO_ROOT_PASSWORD: this.secretAccessKey,
+    });
+
+    const startedContainer = await super.start();
+
+    // Create the "packets" bucket using MinIO client
+    await x(
+      "docker",
+      [
+        "exec",
+        startedContainer.getId(),
+        "mc",
+        "alias",
+        "set",
+        "local",
+        "http://localhost:9000",
+        this.accessKeyId,
+        this.secretAccessKey,
+      ],
+      { throwOnError: true }
+    );
+
+    await x("docker", ["exec", startedContainer.getId(), "mc", "mb", "local/packets"], {
+      throwOnError: true,
+    });
+
+    return new StartedMinIOContainer(
+      startedContainer,
+      this.accessKeyId,
+      this.secretAccessKey,
+      this.region
+    );
+  }
+}
+
+export class StartedMinIOContainer extends AbstractStartedContainer {
+  constructor(
+    startedTestContainer: StartedTestContainer,
+    private readonly accessKeyId: string,
+    private readonly secretAccessKey: string,
+    private readonly region: string
+  ) {
+    super(startedTestContainer);
+  }
+
+  public getPort(): number {
+    return super.getMappedPort(MINIO_PORT);
+  }
+
+  public getAccessKeyId(): string {
+    return this.accessKeyId;
+  }
+
+  public getSecretAccessKey(): string {
+    return this.secretAccessKey;
+  }
+
+  public getRegion(): string {
+    return this.region;
+  }
+
+  /**
+   * Gets the base URL (protocol, host and mapped port) for the MinIO container.
+   * Example: `http://localhost:32768`
+   */
+  public getBaseUrl(): string {
+    const protocol = "http";
+    const host = this.getHost();
+    const port = this.getPort();
+    return `${protocol}://${host}:${port}`;
+  }
+
+  /**
+   * Empties the bucket between tests on a reused container (the "local" mc alias and the bucket are
+   * created at boot). Recreates the bucket so each test starts from the same empty state.
+   */
+  public async resetBucket(bucket = "packets"): Promise<void> {
+    await x(
+      "docker",
+      ["exec", this.getId(), "mc", "rm", "--recursive", "--force", `local/${bucket}`],
+      {
+        throwOnError: false,
+      }
+    );
+    await x("docker", ["exec", this.getId(), "mc", "mb", "--ignore-existing", `local/${bucket}`], {
+      throwOnError: true,
+    });
+  }
+
+  /**
+   * Gets connection configuration suitable for object storage clients.
+   */
+  public getConnectionConfig(): MinIOConnectionConfig {
+    return {
+      baseUrl: this.getBaseUrl(),
+      accessKeyId: this.getAccessKeyId(),
+      secretAccessKey: this.getSecretAccessKey(),
+      region: this.getRegion(),
+    };
+  }
+}

internal-packages/testcontainers/src/sequencer.cjs

@@ -0,0 +1,129 @@
+// Authored as plain CommonJS (NOT .ts) on purpose. vitest loads each package's vitest.config.ts by
+// bundling it, and it EXTERNALIZES this workspace subpath - node then loads this file verbatim. A .ts
+// here reaches node as raw TypeScript and crashes config loading on CI's pinned node 20 (no type
+// stripping: `SyntaxError`). Keeping it dependency-free JS - and importing nothing from the ESM-only
+// `vitest/node` - makes it loadable on every node. Types for consumers live in sequencer.d.cts.
+
+const { existsSync, readFileSync } = require("node:fs");
+const path = require("node:path");
+
+// Walk up from the package dir (cwd at config-load time) to the monorepo root (pnpm-workspace.yaml).
+function findRepoRoot(start) {
+  let dir = start;
+  for (let i = 0; i < 20; i++) {
+    if (existsSync(path.join(dir, "pnpm-workspace.yaml"))) return dir;
+    const parent = path.dirname(dir);
+    if (parent === dir) break;
+    dir = parent;
+  }
+  return start;
+}
+
+// test-timings.json lives at the monorepo root: { "<repo-relative path>": <ms> }
+const REPO_ROOT = findRepoRoot(process.cwd());
+const TIMINGS_PATH = path.resolve(REPO_ROOT, "test-timings.json");
+
+let cachedTimings;
+
+function loadTimings() {
+  if (!cachedTimings) {
+    // A MISSING file is a legitimate state (no timings configured yet => count-based split). But a
+    // file that EXISTS and won't parse is a real problem with a committed artifact we control - fail
+    // loud rather than silently degrading sharding (silent fallbacks are what hid earlier bugs).
+    if (!existsSync(TIMINGS_PATH)) {
+      cachedTimings = {};
+      return cachedTimings;
+    }
+    try {
+      cachedTimings = JSON.parse(readFileSync(TIMINGS_PATH, "utf-8"));
+    } catch (error) {
+      throw new Error(`Failed to parse ${TIMINGS_PATH}: ${error?.message ?? error}`);
+    }
+  }
+  return cachedTimings;
+}
+
+function median(nums) {
+  if (nums.length === 0) return 1;
+  const sorted = [...nums].sort((a, b) => a - b);
+  const mid = Math.floor(sorted.length / 2);
+  return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
+}
+
+// Stable per-package offset (derived from the package dir) so each package's heaviest file - which
+// LPT always drops into bin 0 - maps to a DIFFERENT shard. Without it, a serial multi-package job
+// (`turbo --concurrency=1 --filter "@internal/*"`) stacks every package's heaviest file into shard 1.
+// It's a rotation of the bin->shard mapping, so coverage stays exact (each file runs once).
+function packageOffset(specs, count) {
+  if (specs.length === 0) return 0;
+  const rel = path.relative(REPO_ROOT, specs[0].moduleId);
+  const key = rel.split(path.sep).slice(0, 2).join("/");
+  // FNV-1a - spreads similar sibling package names (e.g. internal-packages/*) far better than a
+  // simple polynomial hash mod count, which collided run-engine + schedule-engine onto one shard.
+  let h = 2166136261;
+  for (let i = 0; i < key.length; i++) {
+    h ^= key.charCodeAt(i);
+    h = Math.imul(h, 16777619);
+  }
+  return (h >>> 0) % count;
+}
+
+/**
+ * Duration-weighted interpretation of `--shard=i/N`. Instead of vitest's default file-count split,
+ * this greedily bin-packs test files by recorded duration (test-timings.json at the repo root;
+ * unknown/new files get the median) so each shard does roughly equal work.
+ *
+ * The packing is fully deterministic (sort by duration desc, then moduleId) so every shard computes
+ * the identical bins and just takes its own - no file runs twice or gets dropped. Falls back to the
+ * full set when no shard is configured, and to ~count-based when no timings exist.
+ *
+ * Implemented as a standalone TestSequencer (not extending BaseSequencer) so this file never imports
+ * `vitest/node` - see the header note.
+ */
+class DurationShardingSequencer {
+  constructor(ctx) {
+    this.ctx = ctx;
+  }
+
+  // Deterministic order (heaviest first, then moduleId) - stable across shards and a sensible
+  // in-shard run order, replacing BaseSequencer's default sort we no longer inherit.
+  async sort(files) {
+    const timings = loadTimings();
+    const fallback = median(Object.values(timings));
+    return [...files].sort((a, b) => {
+      const am = timings[path.relative(REPO_ROOT, a.moduleId)] ?? fallback;
+      const bm = timings[path.relative(REPO_ROOT, b.moduleId)] ?? fallback;
+      return bm - am || a.moduleId.localeCompare(b.moduleId);
+    });
+  }
+
+  async shard(specs) {
+    const shard = this.ctx.config.shard;
+    if (!shard || specs.length === 0) {
+      return specs;
+    }
+
+    const timings = loadTimings();
+    const fallback = median(Object.values(timings));
+
+    const weighted = specs
+      .map((spec) => ({
+        spec,
+        ms: timings[path.relative(REPO_ROOT, spec.moduleId)] ?? fallback,
+      }))
+      .sort((a, b) => b.ms - a.ms || a.spec.moduleId.localeCompare(b.spec.moduleId));
+
+    const bins = Array.from({ length: shard.count }, () => ({ total: 0, specs: [] }));
+
+    for (const { spec, ms } of weighted) {
+      const lightest = bins.reduce((min, bin) => (bin.total < min.total ? bin : min));
+      lightest.total += ms;
+      lightest.specs.push(spec);
+    }
+
+    const offset = packageOffset(specs, shard.count);
+    return bins[(shard.index - 1 + offset) % shard.count].specs;
+  }
+}
+
+module.exports = { DurationShardingSequencer };

internal-packages/testcontainers/src/sequencer.d.cts

@@ -0,0 +1,13 @@
+import type { TestSequencer, TestSpecification, Vitest } from "vitest/node";
+
+/**
+ * Duration-weighted `--shard=i/N`: bin-packs test files by recorded duration (test-timings.json at
+ * the repo root) so each shard does roughly equal work. The runtime lives in `sequencer.cjs` (plain
+ * JS, so vitest config loading can load it on any node - see that file's header); this declaration
+ * supplies the types for configs that wire it via `sequence: { sequencer: DurationShardingSequencer }`.
+ */
+export declare class DurationShardingSequencer implements TestSequencer {
+  constructor(ctx: Vitest);
+  sort(files: TestSpecification[]): Promise<TestSpecification[]>;
+  shard(files: TestSpecification[]): Promise<TestSpecification[]>;
+}

internal-packages/testcontainers/src/utils.ts

@@ -0,0 +1,369 @@
+import { createClient } from "@clickhouse/client";
+import type { StartedPostgreSqlContainer } from "@testcontainers/postgresql";
+import { PostgreSqlContainer } from "@testcontainers/postgresql";
+import type { StartedRedisContainer } from "@testcontainers/redis";
+import { RedisContainer } from "@testcontainers/redis";
+import { PrismaClient } from "@trigger.dev/database";
+import { tryCatch } from "@trigger.dev/core";
+import Redis from "ioredis";
+import path from "path";
+import { isDebug } from "std-env";
+import type { StartedNetwork, StartedTestContainer } from "testcontainers";
+import { GenericContainer, Wait } from "testcontainers";
+import { x } from "tinyexec";
+import type { TestContext } from "vitest";
+import { ClickHouseContainer, runClickhouseMigrations } from "./clickhouse";
+import { MinIOContainer } from "./minio";
+import { getContainerMetadata, getTaskMetadata, logCleanup, logSetup } from "./logs";
+
+/** Returns the container's connection URI with the database path swapped to `database`. */
+export function postgresUriWithDatabase(uri: string, database: string): string {
+  const url = new URL(uri);
+  url.pathname = `/${database}`;
+  return url.toString();
+}
+
+/** Pushes the Prisma schema into the database at `databaseUrl` (which must already exist). */
+export async function pushDatabaseSchema(databaseUrl: string) {
+  const databasePath = path.resolve(__dirname, "../../database");
+
+  return pushPrismaSchema({
+    prismaBin: `${databasePath}/node_modules/.bin/prisma`,
+    schemaPath: `${databasePath}/prisma/schema.prisma`,
+    // The full @trigger.dev/database datasource reads DATABASE_URL/DIRECT_URL.
+    env: { DATABASE_URL: databaseUrl, DIRECT_URL: databaseUrl },
+  });
+}
+
+/**
+ * Pushes the DEDICATED run-ops subset schema (@internal/run-ops-database) into the database at
+ * `databaseUrl`. The run-ops datasource reads RUN_OPS_DATABASE_URL, and its schema is a subset of
+ * the control-plane schema (run-ops tables, no Organization/Project/etc).
+ */
+export async function pushRunOpsSchema(databaseUrl: string) {
+  // Resolve the schema (and the package's own prisma binary) through the @internal/run-ops-database
+  // package so this keeps working regardless of where pnpm hoists it.
+  const schemaPath = require.resolve("@internal/run-ops-database/prisma/schema.prisma");
+  const runOpsPackagePath = path.resolve(schemaPath, "../..");
+
+  const result = await pushPrismaSchema({
+    prismaBin: `${runOpsPackagePath}/node_modules/.bin/prisma`,
+    schemaPath,
+    env: { RUN_OPS_DATABASE_URL: databaseUrl },
+  });
+
+  // `db push` derives DDL from the schema datamodel and so cannot create the SQL-only partial unique
+  // index Prisma can't express (the NULL-batchIndex dedup). Apply it here so the test DB matches what
+  // `migrate deploy` builds in production; otherwise ON CONFLICT DO NOTHING can't dedupe a re-blocked
+  // NULL-batchIndex edge and the idempotency contract goes untested.
+  await applyRunOpsSqlOnlyIndexes(databaseUrl);
+
+  return result;
+}
+
+async function applyRunOpsSqlOnlyIndexes(databaseUrl: string) {
+  // Reuse @trigger.dev/database's PrismaClient purely as a raw-SQL connection (the established
+  // primitive in this package — see createDatabaseFromTemplate). No model access, so the subset
+  // schema mismatch is irrelevant.
+  const client = new PrismaClient({ datasources: { db: { url: databaseUrl } } });
+  try {
+    await client.$executeRawUnsafe(
+      `CREATE UNIQUE INDEX IF NOT EXISTS "TaskRunWaitpoint_taskRunId_waitpointId_batchIndex_null_key" ON "public"."TaskRunWaitpoint"("taskRunId", "waitpointId") WHERE "batchIndex" IS NULL`
+    );
+  } finally {
+    await client.$disconnect();
+  }
+}
+
+async function pushPrismaSchema({
+  prismaBin,
+  schemaPath,
+  env,
+}: {
+  prismaBin: string;
+  schemaPath: string;
+  env: Record<string, string>;
+}) {
+  // throwOnError is essential: without it tinyexec swallows a non-zero `prisma db push`, so a failed
+  // push looks like success and only surfaces much later as a confusing downstream error.
+  const result = await x(
+    prismaBin,
+    [
+      "db",
+      "push",
+      "--force-reset",
+      "--accept-data-loss",
+      "--skip-generate",
+      "--schema",
+      schemaPath,
+    ],
+    {
+      throwOnError: true,
+      nodeOptions: {
+        env: {
+          ...process.env,
+          ...env,
+        },
+      },
+    }
+  );
+
+  return result;
+}
+
+/**
+ * Caps each container's CPU/memory to approximate the 2-core CI runner locally (for timing + flake
+ * reproduction). Set TESTCONTAINERS_CPU (cores per container, e.g. "2") and/or
+ * TESTCONTAINERS_MEMORY_GB (GB per container). Pair with running the runner under `taskset -c 0,1`.
+ * No-op when neither is set. (testcontainers v11 has no cpuset pinning, only this quota cap.)
+ */
+export function withCiResourceLimits<T extends GenericContainer>(container: T): T {
+  const cpu = parsePositiveNumberEnv("TESTCONTAINERS_CPU");
+  const memory = parsePositiveNumberEnv("TESTCONTAINERS_MEMORY_GB");
+  if (cpu === undefined && memory === undefined) {
+    return container;
+  }
+  return container.withResourcesQuota({
+    ...(cpu !== undefined ? { cpu } : {}),
+    ...(memory !== undefined ? { memory } : {}),
+  });
+}
+
+// Fail fast on a malformed value rather than letting NaN reach the container runtime as a cryptic error.
+function parsePositiveNumberEnv(name: string): number | undefined {
+  const raw = process.env[name];
+  if (!raw) return undefined;
+  const value = Number(raw);
+  if (!Number.isFinite(value) || value <= 0) {
+    throw new Error(`${name} must be a positive number, got "${raw}"`);
+  }
+  return value;
+}
+
+export async function createPostgresContainer(
+  network: StartedNetwork,
+  opts?: { imageTag?: string; initdbArgs?: string }
+) {
+  const imageTag = opts?.imageTag ?? "docker.io/postgres:14";
+  let builder = withCiResourceLimits(new PostgreSqlContainer(imageTag))
+    .withNetwork(network)
+    .withNetworkAliases("database")
+    .withCommand(["-c", "listen_addresses=*", "-c", "wal_level=logical"]);
+  if (opts?.initdbArgs) {
+    // POSTGRES_INITDB_ARGS is the official postgres image hook for initdb flags
+    // (locale/collation provider). No dedicated PostgreSqlContainer method exists.
+    builder = builder.withEnvironment({ POSTGRES_INITDB_ARGS: opts.initdbArgs });
+  }
+  const container = await builder.start();
+
+  await pushDatabaseSchema(container.getConnectionUri());
+
+  return { url: container.getConnectionUri(), container, network };
+}
+
+export async function createClickHouseContainer(network: StartedNetwork) {
+  const container = await withCiResourceLimits(new ClickHouseContainer())
+    .withNetwork(network)
+    .start();
+
+  const client = createClient({
+    url: container.getConnectionUrl(),
+  });
+
+  await client.ping();
+
+  const migrationsPath = path.resolve(__dirname, "../../clickhouse/schema");
+
+  await runClickhouseMigrations(client, migrationsPath);
+
+  return {
+    url: container.getConnectionUrl(),
+    container,
+    network,
+  };
+}
+
+export async function createRedisContainer({
+  port,
+  network,
+}: {
+  port?: number;
+  network?: StartedNetwork;
+}) {
+  let container = withCiResourceLimits(new RedisContainer("redis:7.2"))
+    .withExposedPorts(port ?? 6379)
+    .withStartupTimeout(120_000); // 2 minutes
+
+  if (network) {
+    container = container.withNetwork(network).withNetworkAliases("redis");
+  }
+
+  // Wait only on the readiness log (RedisContainer's default) - the previous Docker healthcheck added
+  // a full poll-cycle of latency per boot, which dominates per-test redis. verifyRedisConnection
+  // below still confirms the container actually accepts connections before we hand it to the test.
+  const startedContainer = await container
+    .withWaitStrategy(Wait.forLogMessage("Ready to accept connections"))
+    .start();
+
+  const [error] = await tryCatch(verifyRedisConnection(startedContainer));
+
+  if (error) {
+    await startedContainer.stop({ timeout: 30_000 });
+    throw new Error("verifyRedisConnection error", { cause: error });
+  }
+
+  return {
+    container: startedContainer,
+  };
+}
+
+async function verifyRedisConnection(container: StartedRedisContainer) {
+  const redis = new Redis({
+    host: container.getHost(),
+    port: container.getPort(),
+    password: container.getPassword(),
+    maxRetriesPerRequest: 20,
+    connectTimeout: 10000,
+    retryStrategy(times) {
+      const delay = Math.min(times * 50, 2000);
+      return delay;
+    },
+  });
+
+  const containerMetadata = {
+    containerId: container.getId().slice(0, 12),
+    containerName: container.getName(),
+    containerNetworkNames: container.getNetworkNames(),
+  };
+
+  redis.on("error", (error) => {
+    if (isDebug) {
+      console.log("verifyRedisConnection: client error", error, containerMetadata);
+    }
+
+    // Don't throw here, we'll do that below if the ping fails
+  });
+
+  try {
+    await redis.ping();
+  } catch (error) {
+    if (isDebug) {
+      console.log("verifyRedisConnection: ping error", error, containerMetadata);
+    }
+
+    throw new Error("verifyRedisConnection: ping error", { cause: error });
+  } finally {
+    await redis.quit();
+  }
+}
+
+export async function createElectricContainer(
+  postgresContainer: StartedPostgreSqlContainer,
+  network: StartedNetwork
+) {
+  const databaseUrl = `postgresql://${postgresContainer.getUsername()}:${postgresContainer.getPassword()}@${postgresContainer.getIpAddress(
+    network.getName()
+  )}:5432/${postgresContainer.getDatabase()}?sslmode=disable`;
+
+  const container = await withCiResourceLimits(
+    new GenericContainer(
+      "electricsql/electric:1.2.4@sha256:20da3d0b0e74926c5623392db67fd56698b9e374c4aeb6cb5cadeb8fea171c36"
+    )
+  )
+    .withExposedPorts(3000)
+    .withNetwork(network)
+    .withEnvironment({
+      DATABASE_URL: databaseUrl,
+      ELECTRIC_INSECURE: "true",
+    })
+    .start();
+
+  return {
+    container,
+    origin: `http://${container.getHost()}:${container.getMappedPort(3000)}`,
+  };
+}
+
+export async function createMinIOContainer(network: StartedNetwork) {
+  const container = await withCiResourceLimits(new MinIOContainer())
+    .withNetwork(network)
+    .withNetworkAliases("minio")
+    .start();
+
+  return {
+    container,
+    network,
+  };
+}
+
+export function assertNonNullable<T>(value: T): asserts value is NonNullable<T> {
+  // Plain throw — *not* `vitest.expect`. Two reasons:
+  //   1. This module is imported by globalSetup files that run before any
+  //      vitest worker exists, so `import { expect }` from "vitest" at
+  //      top level can crash on init.
+  //   2. Lazy-loading via `require("vitest")` (the prior fix) collides
+  //      with OTel auto-instrumentation: `@opentelemetry/instrumentation`
+  //      hooks `require()` via `require-in-the-middle`, and vitest is
+  //      ESM-only — the require() throws "Vitest cannot be imported in
+  //      a CommonJS module using require()", failing every test that
+  //      uses `assertNonNullable` after OTel's been touched.
+  // The plain throw still gives vitest a useful failure (the message is
+  // shown in the stack trace) without the instrumentation hazard.
+  if (value === null || value === undefined) {
+    throw new Error(`assertNonNullable: value was ${value === null ? "null" : "undefined"}`);
+  }
+}
+
+export async function withContainerSetup<T>({
+  name,
+  task,
+  setup,
+}: {
+  name: string;
+  task: TestContext["task"];
+  setup: Promise<T extends { container: StartedTestContainer } ? T : never>;
+}): Promise<T & { metadata: Record<string, unknown> }> {
+  const testName = task.name;
+  logSetup(`${name}: starting`, { testName });
+
+  const start = Date.now();
+  const result = await setup;
+  const startDurationMs = Date.now() - start;
+
+  const metadata = {
+    ...getTaskMetadata(task),
+    ...getContainerMetadata(result.container),
+    startDurationMs,
+  };
+
+  logSetup(`${name}: started`, metadata);
+
+  return { ...result, metadata };
+}
+
+export async function useContainer<TContainer extends StartedTestContainer>(
+  name: string,
+  {
+    container,
+    task,
+    use,
+  }: { container: TContainer; task: TestContext["task"]; use: () => Promise<void> }
+) {
+  const metadata = {
+    ...getTaskMetadata(task),
+    ...getContainerMetadata(container),
+    useDurationMs: 0,
+  };
+
+  try {
+    const start = Date.now();
+    await use();
+    const useDurationMs = Date.now() - start;
+    metadata.useDurationMs = useDurationMs;
+  } finally {
+    // Containers are throwaway, so we force-kill (SIGKILL) instead of waiting for a graceful
+    // shutdown - ClickHouse alone spends ~5s/test gracefully stopping. timeout: 0 = immediate kill.
+    // We still await it (no pileup); logCleanup swallows any teardown-time connection errors.
+    await logCleanup(name, container.stop({ timeout: 0 }), metadata);
+  }
+}

internal-packages/testcontainers/src/webapp.ts

@@ -0,0 +1,253 @@
+import { spawn } from "child_process";
+import { createServer } from "net";
+import { delimiter, resolve } from "path";
+import { Network } from "testcontainers";
+import { PrismaClient } from "@trigger.dev/database";
+import { createPostgresContainer, createRedisContainer } from "./utils";
+
+const WEBAPP_ROOT = resolve(__dirname, "../../../apps/webapp");
+// pnpm hoists transitive deps to node_modules/.pnpm/node_modules but does NOT symlink them
+// to the root node_modules. We need NODE_PATH so the webapp process can find them at runtime.
+const PNPM_HOISTED_MODULES = resolve(__dirname, "../../../node_modules/.pnpm/node_modules");
+
+async function findFreePort(): Promise<number> {
+  return new Promise((res, rej) => {
+    const srv = createServer();
+    srv.listen(0, () => {
+      const port = (srv.address() as { port: number }).port;
+      srv.close((err) => (err ? rej(err) : res(port)));
+    });
+  });
+}
+
+async function waitForHealthcheck(
+  url: string,
+  opts: { timeoutMs?: number; acceptAnyResponse?: boolean } = {}
+): Promise<void> {
+  const { timeoutMs = 60000, acceptAnyResponse = false } = opts;
+  const deadline = Date.now() + timeoutMs;
+  while (Date.now() < deadline) {
+    try {
+      const res = await fetch(url);
+      if (acceptAnyResponse || res.ok) return;
+    } catch {}
+    await new Promise((r) => setTimeout(r, 500));
+  }
+  throw new Error(`Webapp did not become healthy at ${url} within ${timeoutMs}ms`);
+}
+
+export interface WebappInstance {
+  baseUrl: string;
+  fetch(path: string, init?: RequestInit): Promise<Response>;
+}
+
+export interface StartWebappOptions {
+  /**
+   * When true (default), the spawned webapp runs with `RBAC_FORCE_FALLBACK=1`
+   * so the default fallback handles all auth checks. The comprehensive
+   * suite (`*.e2e.full.test.ts`) relies on this — it's pinned to the
+   * fallback so results don't depend on whether `@triggerdotdev/plugins/rbac`
+   * happens to be installed in the local node_modules.
+   *
+   * Set to false to spawn a webapp that loads any installed RBAC
+   * plugin instead, for testing the plugin path.
+   */
+  forceRbacFallback?: boolean;
+
+  /**
+   * When true, spawns the webapp with `REQUIRE_PLUGINS=1` so the plugin
+   * loader throws instead of silently falling back when the plugin
+   * module fails to load. Used by the healthcheck rollback e2e test —
+   * with this set and the plugin not installed, `/healthcheck` should
+   * return 500.
+   *
+   * Implies `forceRbacFallback: false` (you can't observe REQUIRE_PLUGINS
+   * behaviour when the loader is short-circuited).
+   */
+  requirePlugins?: boolean;
+}
+
+export async function startWebapp(
+  databaseUrl: string,
+  redis: { host: string; port: number },
+  options: StartWebappOptions = {}
+): Promise<{
+  instance: WebappInstance;
+  stop: () => Promise<void>;
+}> {
+  // requirePlugins implies forceRbacFallback=false — you can't observe the
+  // REQUIRE_PLUGINS=1 throw if the loader short-circuits before reaching it.
+  const requirePlugins = options.requirePlugins ?? false;
+  const forceRbacFallback = options.forceRbacFallback ?? !requirePlugins;
+  const port = await findFreePort();
+
+  // Merge NODE_PATH so transitive pnpm deps (hoisted to .pnpm/node_modules) are resolvable
+  const existingNodePath = process.env.NODE_PATH;
+  const nodePath = existingNodePath
+    ? `${PNPM_HOISTED_MODULES}${delimiter}${existingNodePath}`
+    : PNPM_HOISTED_MODULES;
+
+  const proc = spawn(process.execPath, ["build/server.js"], {
+    cwd: WEBAPP_ROOT,
+    env: {
+      ...process.env,
+      // Match `pnpm run start` (production-mode boot). NODE_ENV=test
+      // surfaces a circular-init regression in the production bundle
+      // — see TRI-8731 — that production-mode dodges by initialising
+      // modules in a different order. Tests don't depend on test-mode
+      // semantics; they only need an isolated webapp + DB.
+      NODE_ENV: "production",
+      DATABASE_URL: databaseUrl,
+      DIRECT_URL: databaseUrl,
+      PORT: String(port),
+      REMIX_APP_PORT: String(port), // override .env file value (vitest loads .env via Vite)
+      SESSION_SECRET: "test-session-secret-for-e2e-tests",
+      MAGIC_LINK_SECRET: "test-magic-link-secret-32chars!!",
+      ENCRYPTION_KEY: "test-encryption-key-for-e2e!!!!!", // exactly 32 bytes
+      CLICKHOUSE_URL: "http://localhost:19123", // dummy, auth paths never connect
+      DEPLOY_REGISTRY_HOST: "registry.example.com", // dummy, not needed for auth tests
+      ELECTRIC_ORIGIN: "http://localhost:3060",
+      REDIS_HOST: redis.host,
+      REDIS_PORT: String(redis.port),
+      REDIS_TLS_DISABLED: "true", // all *_REDIS_TLS_DISABLED vars default to this; test Redis has no TLS
+      // Disable all background workers. Each worker has its own env var and its own
+      // check idiom ("0" vs "false" vs boolean), so we set all of them explicitly.
+      WORKER_ENABLED: "false", // disables workerQueue.initialize() (checked === "true")
+      RUN_ENGINE_WORKER_ENABLED: "0", // disables run engine workers (checked === "0", default "1")
+      SCHEDULE_WORKER_ENABLED: "0", // disables schedule engine worker (checked === "0")
+      BATCH_QUEUE_WORKER_ENABLED: "false", // disables batch queue consumers (BoolEnv)
+      LEGACY_RUN_ENGINE_WORKER_ENABLED: "0", // disables legacy run engine worker
+      COMMON_WORKER_ENABLED: "0", // disables common worker
+      RUN_ENGINE_TTL_SYSTEM_DISABLED: "true", // disables TTL expiry system (BoolEnv)
+      RUN_ENGINE_TTL_CONSUMERS_DISABLED: "true", // disables TTL consumers (BoolEnv)
+      RUN_REPLICATION_ENABLED: "0",
+      // Force the RBAC loader to use the default fallback in e2e tests
+      // so auth behaviour is deterministic regardless of whether a
+      // plugin is installed in the local node_modules. Set to "0" /
+      // undefined to spawn a webapp that loads any installed plugin.
+      ...(forceRbacFallback ? { RBAC_FORCE_FALLBACK: "1" } : {}),
+      // When requirePlugins is set, explicitly override RBAC_FORCE_FALLBACK
+      // to "0" so a local apps/webapp/.env that sets it to "1" doesn't
+      // short-circuit the loader past the REQUIRE_PLUGINS check.
+      ...(requirePlugins ? { REQUIRE_PLUGINS: "1", RBAC_FORCE_FALLBACK: "0" } : {}),
+      NODE_PATH: nodePath,
+    },
+    stdio: ["ignore", "pipe", "pipe"],
+  });
+
+  const stderr: string[] = [];
+  proc.stderr?.on("data", (d: Buffer) => {
+    const line = d.toString();
+    stderr.push(line);
+    if (process.env.WEBAPP_TEST_VERBOSE) {
+      process.stderr.write(line);
+    }
+  });
+
+  const stdout: string[] = [];
+  proc.stdout?.on("data", (d: Buffer) => {
+    const line = d.toString();
+    stdout.push(line);
+    if (process.env.WEBAPP_TEST_VERBOSE) {
+      process.stdout.write(line);
+    }
+  });
+
+  // Capture spawn errors instead of throwing from inside the listener.
+  // A synchronous throw from an EventEmitter listener bypasses the surrounding
+  // try/catch and surfaces as an uncaughtException, tearing down the test runner.
+  let spawnError: Error | undefined;
+  proc.on("error", (err) => {
+    spawnError = err;
+  });
+
+  const baseUrl = `http://localhost:${port}`;
+
+  try {
+    if (spawnError) throw spawnError;
+    // When requirePlugins is set, /healthcheck is expected to respond with
+    // 500 (the whole point of those tests is to verify that). Accept any
+    // response as "the webapp is up" — the test then asserts the status.
+    await waitForHealthcheck(`${baseUrl}/healthcheck`, {
+      acceptAnyResponse: requirePlugins,
+    });
+    if (spawnError) throw spawnError;
+  } catch (err) {
+    proc.kill("SIGTERM");
+    const output = [...stdout, ...stderr].join("\n");
+    throw new Error(`Webapp failed to start.\nOutput:\n${output}\n\nOriginal error: ${err}`);
+  }
+
+  return {
+    instance: {
+      baseUrl,
+      fetch: (path: string, init?: RequestInit) => fetch(`${baseUrl}${path}`, init),
+    },
+    stop: () =>
+      new Promise<void>((res) => {
+        const timer = setTimeout(() => {
+          proc.kill("SIGKILL");
+          res();
+        }, 10_000);
+        proc.once("exit", () => {
+          clearTimeout(timer);
+          res();
+        });
+        proc.kill("SIGTERM");
+      }),
+  };
+}
+
+export interface TestServer {
+  webapp: WebappInstance;
+  prisma: PrismaClient;
+  // Postgres connection string. Useful when test workers run in separate
+  // processes and need to construct their own clients against the same DB.
+  databaseUrl: string;
+  stop: () => Promise<void>;
+}
+
+/** Convenience helper: starts a postgres + redis container + webapp and returns both for testing. */
+export async function startTestServer(options: StartWebappOptions = {}): Promise<TestServer> {
+  const network = await new Network().start();
+
+  // Track each resource as we acquire it so we can tear it down if a later step fails.
+  let pgContainer: Awaited<ReturnType<typeof createPostgresContainer>>["container"] | undefined;
+  let pgUrl: string | undefined;
+  let redisContainer: Awaited<ReturnType<typeof createRedisContainer>>["container"] | undefined;
+  let prisma: PrismaClient | undefined;
+  let stopWebapp: (() => Promise<void>) | undefined;
+  let webapp: WebappInstance;
+
+  try {
+    const pg = await createPostgresContainer(network);
+    pgContainer = pg.container;
+    pgUrl = pg.url;
+
+    const { container: rc } = await createRedisContainer({ network });
+    redisContainer = rc;
+
+    prisma = new PrismaClient({ datasources: { db: { url: pg.url } } });
+    await prisma.$connect(); // pre-warm pool; surface connection failures before tests start
+    const started = await startWebapp(pg.url, { host: rc.getHost(), port: rc.getPort() }, options);
+    webapp = started.instance;
+    stopWebapp = started.stop;
+  } catch (err) {
+    await stopWebapp?.().catch(() => {});
+    await prisma?.$disconnect().catch(() => {});
+    await pgContainer?.stop().catch(() => {});
+    await redisContainer?.stop().catch(() => {});
+    await network.stop().catch(() => {});
+    throw err;
+  }
+
+  const stop = async () => {
+    await stopWebapp!().catch((err) => console.error("stopWebapp failed:", err));
+    await prisma!.$disconnect().catch((err) => console.error("prisma.$disconnect failed:", err));
+    await pgContainer!.stop().catch((err) => console.error("pgContainer.stop failed:", err));
+    await redisContainer!.stop().catch((err) => console.error("redisContainer.stop failed:", err));
+    await network.stop().catch((err) => console.error("network.stop failed:", err));
+  };
+
+  return { webapp, prisma: prisma!, databaseUrl: pgUrl!, stop };
+}

internal-packages/testcontainers/tsconfig.json

@@ -0,0 +1,25 @@
+{
+  "compilerOptions": {
+    "target": "es2022",
+    "lib": ["ES2022", "DOM", "DOM.Iterable", "DOM.AsyncIterable"],
+    "module": "CommonJS",
+    "moduleResolution": "Node",
+    "moduleDetection": "force",
+    "types": ["vitest/globals"],
+    "esModuleInterop": true,
+    "forceConsistentCasingInFileNames": true,
+    "isolatedModules": true,
+    "preserveWatchOutput": true,
+    "skipLibCheck": true,
+    "noEmit": true,
+    "strict": true,
+    "noUncheckedIndexedAccess": true,
+    "paths": {
+      "@trigger.dev/core": ["../../packages/core/src/index"],
+      "@trigger.dev/core/*": ["../../packages/core/src/*"],
+      "@trigger.dev/database": ["../../internal-packages/database/src/index"],
+      "@trigger.dev/database/*": ["../../internal-packages/database/src/*"]
+    }
+  },
+  "exclude": ["node_modules"]
+}

internal-packages/testcontainers/vitest.config.ts

@@ -0,0 +1,8 @@
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+  test: {
+    include: ["**/*.test.ts"],
+    globals: true,
+  },
+});