chore: import upstream snapshot with attribution

packages/redis-worker/CHANGELOG.md

@@ -0,0 +1,487 @@
+# @trigger.dev/redis-worker
+
+## 4.5.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.3`
+
+## 4.5.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.2`
+
+## 4.5.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.1`
+
+## 4.5.0
+
+### Patch Changes
+
+- Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
+- Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged. ([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
+- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only). ([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
+- Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing. ([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
+
+  MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts.
+
+  MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count.
+
+- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
+- Add a `redis_worker.queue.oldest_message_age` observable gauge (unit `ms`, labeled `worker_name`) reporting the age of the oldest overdue message in each queue. This is a generic queue-stall signal: it stays at 0 while a queue drains healthily and rises only when due work sits undrained (e.g. a blocked dequeue, a dead consumer, or backpressure), even when no items are being processed. Orphaned queue entries are resolved against the items hash so they don't report a phantom stall. Also exposes `SimpleQueue.oldestMessageAge()`. ([#4086](https://github.com/triggerdotdev/trigger.dev/pull/4086))
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0`
+
+## 4.5.0-rc.7
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.7`
+
+## 4.5.0-rc.6
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.6`
+
+## 4.5.0-rc.5
+
+### Patch Changes
+
+- Make mollifier buffer and drainer internals configurable. `MollifierBuffer` now accepts `ackGraceTtlSeconds`, `maxRetriesPerRequest`, `reconnectStepMs`, and `reconnectMaxMs` options, and `MollifierDrainer` accepts `maxBackoffMs` and `backoffFloorMs`. All default to their previous hardcoded values, so existing behaviour is unchanged. ([#3822](https://github.com/triggerdotdev/trigger.dev/pull/3822))
+- `MollifierDrainer` accepts a `drainBatchSize` option (default 1) that controls how many entries are popped per env per tick — in-flight handlers remain capped by the global `concurrency`. `MollifierBuffer` also gains `getDrainingCount()` / `listStaleDraining()`, backed by a new `mollifier:draining` ZSET maintained atomically with pop/ack/fail/requeue (observability-only). ([#3797](https://github.com/triggerdotdev/trigger.dev/pull/3797))
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.5`
+
+## 4.5.0-rc.4
+
+### Minor Changes
+
+- Mollifier buffer extensions: idempotency dedup, an atomic `mutateSnapshot` API, metadata CAS, claim primitives, and a `MollifierSnapshot` type. The buffer's Redis client now reconnects with jittered backoff so a fleet of clients doesn't stampede Redis in lockstep after a blip. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
+- Add `onTerminalFailure` callback to `MollifierDrainerOptions` so the customer's run lands a SYSTEM_FAILURE PG row even when the drainer exhausts `maxAttempts` on a retryable PG error. Previously, retryable-error exhaustion called `buffer.fail()` directly, which atomically marks FAILED + DELs the entry hash with no PG write — silent data loss when PG was unreachable across the full retry budget. The callback fires before `buffer.fail()` on any terminal path (`cause: "non-retryable"` or `"max-attempts-exhausted"`); throwing a retryable error from the callback causes the drainer to requeue rather than fail. ([#3754](https://github.com/triggerdotdev/trigger.dev/pull/3754))
+
+### Patch Changes
+
+- Pipeline the per-entry `HGETALL` fetches in `MollifierBuffer.listEntriesForEnv`. The previous serial implementation issued one Redis round-trip per runId returned by `LRANGE`, which dominated stale-sweep wall-time at any meaningful backlog (at the sweep's default maxCount=1000, this is ~1000 RTTs per env per pass). Behaviour is unchanged — entries are still skipped when the entry hash has been torn down by a concurrent drainer ack/fail between the LRANGE and the HGETALL. ([#3752](https://github.com/triggerdotdev/trigger.dev/pull/3752))
+- Mollifier `mutateSnapshot` now enforces a tag cap: an `append_tags` patch carrying `maxTags` returns `"limit_exceeded"` (writing nothing) when the deduped tag count would exceed the limit, so a buffered run can't accumulate more tags via the tags API than the trigger validator allows at creation. ([#3756](https://github.com/triggerdotdev/trigger.dev/pull/3756))
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.4`
+
+## 4.5.0-rc.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.3`
+
+## 4.5.0-rc.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.2`
+
+## 4.5.0-rc.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.1`
+
+## 4.5.0-rc.0
+
+### Patch Changes
+
+- Add MollifierBuffer and MollifierDrainer primitives for trigger burst smoothing. ([#3614](https://github.com/triggerdotdev/trigger.dev/pull/3614))
+
+  MollifierBuffer (`accept`, `pop`, `ack`, `requeue`, `fail`, `evaluateTrip`) is a per-env FIFO over Redis with atomic Lua transitions for status tracking. `evaluateTrip` is a sliding-window trip evaluator the webapp gate uses to detect per-env trigger bursts.
+
+  MollifierDrainer pops entries through a polling loop with a user-supplied handler. The loop survives transient Redis errors via capped exponential backoff (up to 5s), and per-env pop failures don't poison the rest of the batch — one env's blip is logged and counted as failed for that tick. Rotation is two-level: orgs at the top, envs within each org. The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}` atomically with per-env queues, so the drainer walks orgs → envs directly without an in-memory cache. The `maxOrgsPerTick` option (default 500) caps how many orgs are scheduled per tick; for each picked org, one env is popped (rotating round-robin within the org). An org with N envs gets the same per-tick scheduling slot as an org with 1 env, so tenant-level drainage throughput is determined by org count rather than env count.
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.5.0-rc.0`
+
+## 4.4.6
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.6`
+
+## 4.4.5
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.5`
+
+## 4.4.4
+
+### Patch Changes
+
+- Adapted the CLI API client to propagate the trigger source via http headers. ([#3241](https://github.com/triggerdotdev/trigger.dev/pull/3241))
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.4`
+
+## 4.4.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.3`
+
+## 4.4.2
+
+### Patch Changes
+
+- Fix slow batch queue processing by removing spurious cooloff on concurrency blocks and fixing a race condition where retry attempt counts were not atomically updated during message re-queue. ([#3079](https://github.com/triggerdotdev/trigger.dev/pull/3079))
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.2`
+
+## 4.4.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.1`
+
+## 4.4.0
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.4.0`
+
+## 4.3.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.3.3`
+
+## 4.3.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.3.2`
+
+## 4.3.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.3.1`
+
+## 4.3.0
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.3.0`
+
+## 4.2.0
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.2.0`
+
+## 4.1.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.1.2`
+
+## 4.1.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.1.1`
+
+## 4.1.0
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.1.0`
+
+## 4.0.7
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.7`
+
+## 4.0.6
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.6`
+
+## 4.0.5
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.5`
+
+## 4.0.4
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.4`
+
+## 4.0.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.3`
+
+## 4.0.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.2`
+
+## 4.0.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.1`
+
+## 4.0.0
+
+### Major Changes
+
+- Trigger.dev v4 release. Please see our upgrade to v4 docs to view the full changelog: https://trigger.dev/docs/upgrade-to-v4 ([#1869](https://github.com/triggerdotdev/trigger.dev/pull/1869))
+
+### Patch Changes
+
+- Now each worker gets it's own pLimit concurrency limiter, and we will only ever dequeue items where there is concurrency capacity, preventing incorrectly retried jobs due to visibility timeout expiry ([#2235](https://github.com/triggerdotdev/trigger.dev/pull/2235))
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0`
+
+## 4.0.0-v4-beta.28
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.28`
+
+## 4.0.0-v4-beta.27
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.27`
+
+## 4.0.0-v4-beta.26
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.26`
+
+## 4.0.0-v4-beta.25
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.25`
+
+## 4.0.0-v4-beta.24
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.24`
+
+## 4.0.0-v4-beta.23
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.23`
+
+## 4.0.0-v4-beta.22
+
+### Patch Changes
+
+- Now each worker gets it's own pLimit concurrency limiter, and we will only ever dequeue items where there is concurrency capacity, preventing incorrectly retried jobs due to visibility timeout expiry ([#2235](https://github.com/triggerdotdev/trigger.dev/pull/2235))
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.22`
+
+## 4.0.0-v4-beta.21
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.21`
+
+## 4.0.0-v4-beta.20
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.20`
+
+## 4.0.0-v4-beta.19
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.19`
+
+## 4.0.0-v4-beta.18
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.18`
+
+## 4.0.0-v4-beta.17
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.17`
+
+## 4.0.0-v4-beta.16
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.16`
+
+## 4.0.0-v4-beta.15
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.15`
+
+## 4.0.0-v4-beta.14
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.14`
+
+## 4.0.0-v4-beta.13
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.13`
+
+## 4.0.0-v4-beta.12
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.12`
+
+## 4.0.0-v4-beta.11
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.11`
+
+## 4.0.0-v4-beta.10
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.10`
+
+## 4.0.0-v4-beta.9
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.9`
+
+## 4.0.0-v4-beta.8
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.8`
+
+## 4.0.0-v4-beta.7
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.7`
+
+## 4.0.0-v4-beta.6
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.6`
+
+## 4.0.0-v4-beta.5
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.5`
+
+## 4.0.0-v4-beta.4
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.4`
+
+## 4.0.0-v4-beta.3
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.3`
+
+## 4.0.0-v4-beta.2
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.2`
+
+## 4.0.0-v4-beta.1
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.1`
+
+## 4.0.0-v4-beta.0
+
+### Major Changes
+
+- Trigger.dev v4 release. Please see our upgrade to v4 docs to view the full changelog: https://trigger.dev/docs/upgrade-to-v4 ([#1869](https://github.com/triggerdotdev/trigger.dev/pull/1869))
+
+### Patch Changes
+
+- Updated dependencies:
+  - `@trigger.dev/core@4.0.0-v4-beta.0`

packages/redis-worker/CLAUDE.md

@@ -0,0 +1,19 @@
+# Redis Worker
+
+`@trigger.dev/redis-worker` - custom Redis-based background job system. **This replaces graphile-worker/zodworker** for all new background job needs.
+
+## Key Files
+
+- `src/worker.ts` - Worker loop and job processing with concurrency control
+- `src/queue.ts` - Redis-backed job queue abstraction
+- `src/fair-queue/` - Fair dequeueing algorithm for queue selection
+
+## Usage
+
+Used by the webapp for background jobs (alerting, batch processing, common tasks) and by the run engine for TTL expiration and batch operations.
+
+All new background jobs in the webapp should use redis-worker. Do NOT add new jobs to zodworker (`@internal/zodworker`) or graphile-worker.
+
+## Testing
+
+Uses ioredis. Tests use testcontainers for Redis.

packages/redis-worker/README.md

@@ -0,0 +1,10 @@
+# Redis worker
+
+This is a simple worker that pulls tasks from a Redis queue (also in this package).
+
+Features
+
+- Configurable settings for concurrency and pull speed.
+- Job payloads.
+- A schema so only defined jobs can be added to the queue.
+- The ability to have future dates for jobs.

packages/redis-worker/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@trigger.dev/redis-worker",
+  "version": "4.5.3",
+  "description": "Redis worker for trigger.dev",
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/triggerdotdev/trigger.dev",
+    "directory": "packages/redis-worker"
+  },
+  "type": "module",
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "clean": "rimraf dist .turbo",
+    "build": "tsup",
+    "dev": "tsup --watch",
+    "typecheck": "tsc --noEmit -p tsconfig.src.json",
+    "test": "vitest --sequence.concurrent=false --no-file-parallelism"
+  },
+  "dependencies": {
+    "@trigger.dev/core": "workspace:4.5.3",
+    "lodash.omit": "^4.5.0",
+    "nanoid": "^5.0.7",
+    "p-limit": "^6.2.0",
+    "seedrandom": "^3.0.5",
+    "zod": "3.25.76",
+    "cron-parser": "^4.9.0"
+  },
+  "devDependencies": {
+    "@internal/redis": "workspace:*",
+    "@internal/testcontainers": "workspace:*",
+    "@internal/tracing": "workspace:*",
+    "@types/lodash.omit": "^4.5.7",
+    "@types/seedrandom": "^3.0.8",
+    "rimraf": "6.0.1",
+    "tsup": "^8.4.0",
+    "tsx": "4.17.0"
+  },
+  "engines": {
+    "node": ">=18.20.0"
+  },
+  "main": "./dist/index.cjs",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    }
+  }
+}

packages/redis-worker/src/cron.test.ts

@@ -0,0 +1,130 @@
+import { redisTest } from "@internal/testcontainers";
+import { Logger } from "@trigger.dev/core/logger";
+import { describe } from "node:test";
+import { expect } from "vitest";
+import { Worker, CronSchema } from "./worker.js";
+import { setTimeout } from "node:timers/promises";
+
+describe("Worker with cron", () => {
+  redisTest(
+    "process items on the cron schedule",
+    { timeout: 180_000 },
+    async ({ redisContainer }) => {
+      const processedItems: CronSchema[] = [];
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          cronJob: {
+            cron: "*/5 * * * * *", // Every 5 seconds
+            schema: CronSchema,
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+            jitterInMs: 100,
+          },
+        },
+        jobs: {
+          cronJob: async ({ payload }) => {
+            await setTimeout(30); // Simulate work
+            processedItems.push(payload);
+          },
+        },
+        concurrency: {
+          workers: 2,
+          tasksPerWorker: 3,
+        },
+        logger: new Logger("test", "debug"),
+      }).start();
+
+      await setTimeout(6_000);
+
+      expect(processedItems.length).toBeGreaterThanOrEqual(1);
+
+      const firstItem = processedItems[0];
+
+      expect(firstItem?.timestamp).toBeGreaterThan(0);
+      expect(firstItem?.lastTimestamp).toBeUndefined();
+      expect(firstItem?.cron).toBe("*/5 * * * * *");
+
+      await setTimeout(6_000);
+
+      expect(processedItems.length).toBeGreaterThanOrEqual(2);
+
+      const secondItem = processedItems[1];
+      expect(secondItem?.timestamp).toBeGreaterThan(firstItem!.timestamp);
+      expect(secondItem?.lastTimestamp).toBe(firstItem?.timestamp);
+      expect(secondItem?.cron).toBe("*/5 * * * * *");
+
+      await worker.stop();
+    }
+  );
+
+  redisTest(
+    "continues processing cron items even when job handler throws errors",
+    { timeout: 180_000 },
+    async ({ redisContainer }) => {
+      const processedItems: CronSchema[] = [];
+      let executionCount = 0;
+
+      const worker = new Worker({
+        name: "test-worker-error",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          cronJob: {
+            cron: "*/3 * * * * *", // Every 3 seconds
+            schema: CronSchema,
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 1 }, // Only try once to fail faster
+            jitterInMs: 100,
+          },
+        },
+        jobs: {
+          cronJob: async ({ payload }) => {
+            executionCount++;
+            await setTimeout(30); // Simulate work
+
+            // Throw error on first and third execution
+            if (executionCount === 1 || executionCount === 3) {
+              throw new Error(`Simulated error on execution ${executionCount}`);
+            }
+
+            processedItems.push(payload);
+          },
+        },
+        concurrency: {
+          workers: 2,
+          tasksPerWorker: 3,
+        },
+        logger: new Logger("test", "debug"),
+      }).start();
+
+      // Wait long enough for 4 executions (12 seconds + buffer)
+      await setTimeout(14_000);
+
+      // Should have at least 4 executions total
+      expect(executionCount).toBeGreaterThanOrEqual(4);
+
+      // Should have 2 successful items (executions 2 and 4)
+      expect(processedItems.length).toBeGreaterThanOrEqual(2);
+
+      // Verify that some executions failed (execution count > successful count)
+      // This proves that errors occurred but cron scheduling continued
+      expect(executionCount).toBeGreaterThan(processedItems.length);
+
+      // Verify that successful executions still have correct structure
+      const firstSuccessful = processedItems[0];
+      expect(firstSuccessful?.timestamp).toBeGreaterThan(0);
+      expect(firstSuccessful?.cron).toBe("*/3 * * * * *");
+
+      await worker.stop();
+    }
+  );
+});

packages/redis-worker/src/fair-queue/concurrency.ts

@@ -0,0 +1,304 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import type {
+  ConcurrencyCheckResult,
+  ConcurrencyGroupConfig,
+  ConcurrencyState,
+  FairQueueKeyProducer,
+  QueueDescriptor,
+} from "./types.js";
+
+export interface ConcurrencyManagerOptions {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  groups: ConcurrencyGroupConfig[];
+}
+
+/**
+ * ConcurrencyManager handles multi-level concurrency tracking and limiting.
+ *
+ * Features:
+ * - Multiple concurrent concurrency groups (tenant, org, project, etc.)
+ * - Atomic reserve/release operations using Lua scripts
+ * - Efficient batch checking of all groups
+ */
+export class ConcurrencyManager {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private groups: ConcurrencyGroupConfig[];
+  private groupsByName: Map<string, ConcurrencyGroupConfig>;
+
+  constructor(private options: ConcurrencyManagerOptions) {
+    this.redis = createRedisClient(options.redis);
+    this.keys = options.keys;
+    this.groups = options.groups;
+    this.groupsByName = new Map(options.groups.map((g) => [g.name, g]));
+
+    this.#registerCommands();
+  }
+
+  // ============================================================================
+  // Public Methods
+  // ============================================================================
+
+  /**
+   * Check if a message can be processed given all concurrency constraints.
+   * Checks all configured groups and returns the first one at capacity.
+   */
+  async canProcess(queue: QueueDescriptor): Promise<ConcurrencyCheckResult> {
+    for (const group of this.groups) {
+      const groupId = group.extractGroupId(queue);
+      const isAtCapacity = await this.isAtCapacity(group.name, groupId);
+
+      if (isAtCapacity) {
+        const state = await this.getState(group.name, groupId);
+        return {
+          allowed: false,
+          blockedBy: state,
+        };
+      }
+    }
+
+    return { allowed: true };
+  }
+
+  /**
+   * Reserve concurrency slots for a message across all groups.
+   * Atomic - either all groups are reserved or none.
+   *
+   * @returns true if reservation successful, false if any group is at capacity
+   */
+  async reserve(queue: QueueDescriptor, messageId: string): Promise<boolean> {
+    // Build list of group keys and limits
+    const groupData = await Promise.all(
+      this.groups.map(async (group) => {
+        const groupId = group.extractGroupId(queue);
+        const limit = await group.getLimit(groupId);
+        return {
+          key: this.keys.concurrencyKey(group.name, groupId),
+          limit: limit || group.defaultLimit,
+        };
+      })
+    );
+
+    // Use Lua script for atomic multi-group reservation
+    // Pass keys as KEYS array so ioredis applies keyPrefix correctly
+    const keys = groupData.map((g) => g.key);
+    const limits = groupData.map((g) => g.limit.toString());
+
+    // Args order: messageId, ...limits (keys are passed separately)
+    const result = await this.redis.reserveConcurrency(keys.length, keys, messageId, ...limits);
+
+    return result === 1;
+  }
+
+  /**
+   * Release concurrency slots for a message across all groups.
+   */
+  async release(queue: QueueDescriptor, messageId: string): Promise<void> {
+    const pipeline = this.redis.pipeline();
+
+    for (const group of this.groups) {
+      const groupId = group.extractGroupId(queue);
+      const key = this.keys.concurrencyKey(group.name, groupId);
+      pipeline.srem(key, messageId);
+    }
+
+    await pipeline.exec();
+  }
+
+  /**
+   * Release concurrency slots for multiple messages in a single pipeline.
+   * More efficient than calling release() multiple times.
+   */
+  async releaseBatch(
+    messages: Array<{ queue: QueueDescriptor; messageId: string }>
+  ): Promise<void> {
+    if (messages.length === 0) {
+      return;
+    }
+
+    const pipeline = this.redis.pipeline();
+
+    for (const { queue, messageId } of messages) {
+      for (const group of this.groups) {
+        const groupId = group.extractGroupId(queue);
+        const key = this.keys.concurrencyKey(group.name, groupId);
+        pipeline.srem(key, messageId);
+      }
+    }
+
+    await pipeline.exec();
+  }
+
+  /**
+   * Get current concurrency for a specific group.
+   */
+  async getCurrentConcurrency(groupName: string, groupId: string): Promise<number> {
+    const key = this.keys.concurrencyKey(groupName, groupId);
+    return await this.redis.scard(key);
+  }
+
+  /**
+   * Get available capacity for a queue across all concurrency groups.
+   * Returns the minimum available capacity across all groups.
+   */
+  async getAvailableCapacity(queue: QueueDescriptor): Promise<number> {
+    if (this.groups.length === 0) {
+      return 0;
+    }
+
+    // Build group data for parallel fetching
+    const groupData = this.groups.map((group) => ({
+      group,
+      groupId: group.extractGroupId(queue),
+    }));
+
+    // Fetch all current counts and limits in parallel
+    const [currents, limits] = await Promise.all([
+      Promise.all(
+        groupData.map(({ group, groupId }) =>
+          this.redis.scard(this.keys.concurrencyKey(group.name, groupId))
+        )
+      ),
+      Promise.all(
+        groupData.map(({ group, groupId }) =>
+          group.getLimit(groupId).then((limit) => limit || group.defaultLimit)
+        )
+      ),
+    ]);
+
+    // Calculate minimum available capacity across all groups
+    let minCapacity = Infinity;
+    for (let i = 0; i < groupData.length; i++) {
+      const available = Math.max(0, limits[i]! - currents[i]!);
+      minCapacity = Math.min(minCapacity, available);
+    }
+
+    return minCapacity === Infinity ? 0 : minCapacity;
+  }
+
+  /**
+   * Get concurrency limit for a specific group.
+   */
+  async getConcurrencyLimit(groupName: string, groupId: string): Promise<number> {
+    const group = this.groupsByName.get(groupName);
+    if (!group) {
+      throw new Error(`Unknown concurrency group: ${groupName}`);
+    }
+    return (await group.getLimit(groupId)) || group.defaultLimit;
+  }
+
+  /**
+   * Check if a group is at capacity.
+   */
+  async isAtCapacity(groupName: string, groupId: string): Promise<boolean> {
+    const [current, limit] = await Promise.all([
+      this.getCurrentConcurrency(groupName, groupId),
+      this.getConcurrencyLimit(groupName, groupId),
+    ]);
+    return current >= limit;
+  }
+
+  /**
+   * Get full state for a group.
+   */
+  async getState(groupName: string, groupId: string): Promise<ConcurrencyState> {
+    const [current, limit] = await Promise.all([
+      this.getCurrentConcurrency(groupName, groupId),
+      this.getConcurrencyLimit(groupName, groupId),
+    ]);
+    return {
+      groupName,
+      groupId,
+      current,
+      limit,
+    };
+  }
+
+  /**
+   * Get all active message IDs for a group.
+   */
+  async getActiveMessages(groupName: string, groupId: string): Promise<string[]> {
+    const key = this.keys.concurrencyKey(groupName, groupId);
+    return await this.redis.smembers(key);
+  }
+
+  /**
+   * Force-clear concurrency for a group (use with caution).
+   * Useful for cleanup after crashes.
+   */
+  async clearGroup(groupName: string, groupId: string): Promise<void> {
+    const key = this.keys.concurrencyKey(groupName, groupId);
+    await this.redis.del(key);
+  }
+
+  /**
+   * Remove a specific message from concurrency tracking.
+   * Useful for cleanup.
+   */
+  async removeMessage(messageId: string, queue: QueueDescriptor): Promise<void> {
+    await this.release(queue, messageId);
+  }
+
+  /**
+   * Get configured group names.
+   */
+  getGroupNames(): string[] {
+    return this.groups.map((g) => g.name);
+  }
+
+  /**
+   * Close the Redis connection.
+   */
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  #registerCommands(): void {
+    // Atomic multi-group reservation
+    // KEYS: concurrency set keys for each group (keyPrefix is applied by ioredis)
+    // ARGV[1]: messageId
+    // ARGV[2..n]: limits for each group (in same order as KEYS)
+    this.redis.defineCommand("reserveConcurrency", {
+      lua: `
+local numGroups = #KEYS
+local messageId = ARGV[1]
+
+-- Check all groups first
+for i = 1, numGroups do
+  local key = KEYS[i]
+  local limit = tonumber(ARGV[1 + i])  -- Limits start at ARGV[2]
+  local current = redis.call('SCARD', key)
+  
+  if current >= limit then
+    return 0  -- At capacity
+  end
+end
+
+-- All groups have capacity, add message to all
+for i = 1, numGroups do
+  local key = KEYS[i]
+  redis.call('SADD', key, messageId)
+end
+
+return 1
+      `,
+    });
+  }
+}
+
+// Extend Redis interface for custom commands
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    reserveConcurrency(
+      numKeys: number,
+      keys: string[],
+      messageId: string,
+      ...limits: string[]
+    ): Promise<number>;
+  }
+}

packages/redis-worker/src/fair-queue/keyProducer.ts

@@ -0,0 +1,175 @@
+import type { FairQueueKeyProducer } from "./types.js";
+
+/**
+ * Default key producer for the fair queue system.
+ * Uses a configurable prefix and standard key structure.
+ *
+ * Key structure:
+ * - Master queue: {prefix}:master:{shardId} (legacy, drain-only)
+ * - Dispatch index: {prefix}:dispatch:{shardId} (Level 1: tenantIds)
+ * - Tenant queue index: {prefix}:tenantq:{tenantId} (Level 2: queueIds)
+ * - Queue: {prefix}:queue:{queueId}
+ * - Queue items: {prefix}:queue:{queueId}:items
+ * - Concurrency: {prefix}:concurrency:{groupName}:{groupId}
+ * - In-flight: {prefix}:inflight:{shardId}
+ * - In-flight data: {prefix}:inflight:{shardId}:data
+ * - Worker queue: {prefix}:worker:{consumerId}
+ */
+export class DefaultFairQueueKeyProducer implements FairQueueKeyProducer {
+  private readonly prefix: string;
+  private readonly separator: string;
+
+  constructor(options: { prefix?: string; separator?: string } = {}) {
+    this.prefix = options.prefix ?? "fq";
+    this.separator = options.separator ?? ":";
+  }
+
+  // ============================================================================
+  // Master Queue Keys
+  // ============================================================================
+
+  masterQueueKey(shardId: number): string {
+    return this.#buildKey("master", shardId.toString());
+  }
+
+  // ============================================================================
+  // Queue Keys
+  // ============================================================================
+
+  queueKey(queueId: string): string {
+    return this.#buildKey("queue", queueId);
+  }
+
+  queueItemsKey(queueId: string): string {
+    return this.#buildKey("queue", queueId, "items");
+  }
+
+  // ============================================================================
+  // Concurrency Keys
+  // ============================================================================
+
+  concurrencyKey(groupName: string, groupId: string): string {
+    return this.#buildKey("concurrency", groupName, groupId);
+  }
+
+  // ============================================================================
+  // In-Flight Keys
+  // ============================================================================
+
+  inflightKey(shardId: number): string {
+    return this.#buildKey("inflight", shardId.toString());
+  }
+
+  inflightDataKey(shardId: number): string {
+    return this.#buildKey("inflight", shardId.toString(), "data");
+  }
+
+  // ============================================================================
+  // Worker Queue Keys
+  // ============================================================================
+
+  workerQueueKey(consumerId: string): string {
+    return this.#buildKey("worker", consumerId);
+  }
+
+  // ============================================================================
+  // Tenant Dispatch Keys (Two-Level Index)
+  // ============================================================================
+
+  dispatchKey(shardId: number): string {
+    return this.#buildKey("dispatch", shardId.toString());
+  }
+
+  tenantQueueIndexKey(tenantId: string): string {
+    return this.#buildKey("tenantq", tenantId);
+  }
+
+  // ============================================================================
+  // Dead Letter Queue Keys
+  // ============================================================================
+
+  deadLetterQueueKey(tenantId: string): string {
+    return this.#buildKey("dlq", tenantId);
+  }
+
+  deadLetterQueueDataKey(tenantId: string): string {
+    return this.#buildKey("dlq", tenantId, "data");
+  }
+
+  // ============================================================================
+  // Extraction Methods
+  // ============================================================================
+
+  /**
+   * Extract tenant ID from a queue ID.
+   * Default implementation assumes queue IDs are formatted as: tenant:{tenantId}:...
+   * Override this method for custom queue ID formats.
+   */
+  extractTenantId(queueId: string): string {
+    const parts = queueId.split(this.separator);
+    // Expect format: tenant:{tenantId}:...
+    if (parts.length >= 2 && parts[0] === "tenant" && parts[1]) {
+      return parts[1];
+    }
+    // Fallback: return the first segment
+    return parts[0] ?? "";
+  }
+
+  /**
+   * Extract a group ID from a queue ID.
+   * Default implementation looks for pattern: {groupName}:{groupId}:...
+   * Override this method for custom queue ID formats.
+   */
+  extractGroupId(groupName: string, queueId: string): string {
+    const parts = queueId.split(this.separator);
+
+    // Look for the group name in the queue ID parts
+    for (let i = 0; i < parts.length - 1; i++) {
+      if (parts[i] === groupName) {
+        const nextPart = parts[i + 1];
+        if (nextPart) {
+          return nextPart;
+        }
+      }
+    }
+
+    // Fallback: return an empty string
+    return "";
+  }
+
+  // ============================================================================
+  // Helper Methods
+  // ============================================================================
+
+  #buildKey(...parts: string[]): string {
+    return [this.prefix, ...parts].join(this.separator);
+  }
+}
+
+/**
+ * Key producer with custom extraction logic via callbacks.
+ * Useful when queue IDs don't follow a standard pattern.
+ */
+export class CallbackFairQueueKeyProducer extends DefaultFairQueueKeyProducer {
+  private readonly tenantExtractor: (queueId: string) => string;
+  private readonly groupExtractor: (groupName: string, queueId: string) => string;
+
+  constructor(options: {
+    prefix?: string;
+    separator?: string;
+    extractTenantId: (queueId: string) => string;
+    extractGroupId: (groupName: string, queueId: string) => string;
+  }) {
+    super({ prefix: options.prefix, separator: options.separator });
+    this.tenantExtractor = options.extractTenantId;
+    this.groupExtractor = options.extractGroupId;
+  }
+
+  override extractTenantId(queueId: string): string {
+    return this.tenantExtractor(queueId);
+  }
+
+  override extractGroupId(groupName: string, queueId: string): string {
+    return this.groupExtractor(groupName, queueId);
+  }
+}

packages/redis-worker/src/fair-queue/masterQueue.ts

@@ -0,0 +1,257 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import { jumpHash } from "@trigger.dev/core/v3/serverOnly";
+import type { FairQueueKeyProducer, QueueWithScore } from "./types.js";
+
+export interface MasterQueueOptions {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  shardCount: number;
+}
+
+/**
+ * Master queue manages the top-level queue of queues.
+ *
+ * Features:
+ * - Sharding for horizontal scaling
+ * - Consistent hashing for queue-to-shard assignment
+ * - Queues scored by oldest message timestamp
+ */
+export class MasterQueue {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private shardCount: number;
+
+  constructor(private options: MasterQueueOptions) {
+    this.redis = createRedisClient(options.redis);
+    this.keys = options.keys;
+    this.shardCount = Math.max(1, options.shardCount);
+
+    this.#registerCommands();
+  }
+
+  // ============================================================================
+  // Public Methods
+  // ============================================================================
+
+  /**
+   * Get the shard ID for a queue.
+   * Uses consistent hashing based on queue ID.
+   */
+  getShardForQueue(queueId: string): number {
+    return this.#hashToShard(queueId);
+  }
+
+  /**
+   * Add a queue to its master queue shard.
+   * Updates the score to the oldest message timestamp.
+   *
+   * @param queueId - The queue identifier
+   * @param oldestMessageTimestamp - Timestamp of the oldest message in the queue
+   */
+  async addQueue(queueId: string, oldestMessageTimestamp: number): Promise<void> {
+    const shardId = this.getShardForQueue(queueId);
+    const masterKey = this.keys.masterQueueKey(shardId);
+
+    // Just use plain ZADD - it will add if not exists, or update if exists
+    // The score represents the oldest message timestamp
+    // We rely on the enqueue Lua scripts to set the correct score
+    await this.redis.zadd(masterKey, oldestMessageTimestamp, queueId);
+  }
+
+  /**
+   * Update a queue's score in the master queue.
+   * This is typically called after dequeuing to update to the new oldest message.
+   *
+   * @param queueId - The queue identifier
+   * @param newOldestTimestamp - New timestamp of the oldest message
+   */
+  async updateQueueScore(queueId: string, newOldestTimestamp: number): Promise<void> {
+    const shardId = this.getShardForQueue(queueId);
+    const masterKey = this.keys.masterQueueKey(shardId);
+
+    await this.redis.zadd(masterKey, newOldestTimestamp, queueId);
+  }
+
+  /**
+   * Remove a queue from its master queue shard.
+   * Called when a queue becomes empty.
+   *
+   * @param queueId - The queue identifier
+   */
+  async removeQueue(queueId: string): Promise<void> {
+    const shardId = this.getShardForQueue(queueId);
+    const masterKey = this.keys.masterQueueKey(shardId);
+
+    await this.redis.zrem(masterKey, queueId);
+  }
+
+  /**
+   * Get queues from a shard, ordered by oldest message (lowest score first).
+   *
+   * @param shardId - The shard to query
+   * @param limit - Maximum number of queues to return (default: 1000)
+   * @param maxScore - Maximum score (timestamp) to include (default: now)
+   */
+  async getQueuesFromShard(
+    shardId: number,
+    limit: number = 1000,
+    maxScore?: number
+  ): Promise<QueueWithScore[]> {
+    const masterKey = this.keys.masterQueueKey(shardId);
+    const score = maxScore ?? Date.now();
+
+    // Get queues with scores up to maxScore
+    const results = await this.redis.zrangebyscore(
+      masterKey,
+      "-inf",
+      score,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      limit
+    );
+
+    const queues: QueueWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const queueId = results[i];
+      const scoreStr = results[i + 1];
+      if (queueId && scoreStr) {
+        queues.push({
+          queueId,
+          score: parseFloat(scoreStr),
+          tenantId: this.keys.extractTenantId(queueId),
+        });
+      }
+    }
+
+    return queues;
+  }
+
+  /**
+   * Get the number of queues in a shard.
+   */
+  async getShardQueueCount(shardId: number): Promise<number> {
+    const masterKey = this.keys.masterQueueKey(shardId);
+    return await this.redis.zcard(masterKey);
+  }
+
+  /**
+   * Get total queue count across all shards.
+   */
+  async getTotalQueueCount(): Promise<number> {
+    const counts = await Promise.all(
+      Array.from({ length: this.shardCount }, (_, i) => this.getShardQueueCount(i))
+    );
+    return counts.reduce((sum, count) => sum + count, 0);
+  }
+
+  /**
+   * Atomically add a queue to master queue only if queue has messages.
+   * Uses Lua script for atomicity.
+   *
+   * @param queueId - The queue identifier
+   * @param queueKey - The actual queue sorted set key
+   * @returns Whether the queue was added to the master queue
+   */
+  async addQueueIfNotEmpty(queueId: string, queueKey: string): Promise<boolean> {
+    const shardId = this.getShardForQueue(queueId);
+    const masterKey = this.keys.masterQueueKey(shardId);
+
+    const result = await this.redis.addQueueIfNotEmpty(masterKey, queueKey, queueId);
+    return result === 1;
+  }
+
+  /**
+   * Atomically remove a queue from master queue only if queue is empty.
+   * Uses Lua script for atomicity.
+   *
+   * @param queueId - The queue identifier
+   * @param queueKey - The actual queue sorted set key
+   * @returns Whether the queue was removed from the master queue
+   */
+  async removeQueueIfEmpty(queueId: string, queueKey: string): Promise<boolean> {
+    const shardId = this.getShardForQueue(queueId);
+    const masterKey = this.keys.masterQueueKey(shardId);
+
+    const result = await this.redis.removeQueueIfEmpty(masterKey, queueKey, queueId);
+    return result === 1;
+  }
+
+  /**
+   * Close the Redis connection.
+   */
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  /**
+   * Map queue ID to shard using Jump Consistent Hash.
+   * Provides better distribution than djb2 and minimal reshuffling when shard count changes.
+   */
+  #hashToShard(queueId: string): number {
+    return jumpHash(queueId, this.shardCount);
+  }
+
+  #registerCommands(): void {
+    // Atomically add queue to master if it has messages
+    this.redis.defineCommand("addQueueIfNotEmpty", {
+      numberOfKeys: 2,
+      lua: `
+local masterKey = KEYS[1]
+local queueKey = KEYS[2]
+local queueId = ARGV[1]
+
+-- Check if queue has any messages
+local count = redis.call('ZCARD', queueKey)
+if count == 0 then
+  return 0
+end
+
+-- Get the oldest message timestamp (lowest score)
+local oldest = redis.call('ZRANGE', queueKey, 0, 0, 'WITHSCORES')
+if #oldest == 0 then
+  return 0
+end
+
+local score = oldest[2]
+
+-- Add to master queue with the oldest message score
+redis.call('ZADD', masterKey, score, queueId)
+return 1
+      `,
+    });
+
+    // Atomically remove queue from master if it's empty
+    this.redis.defineCommand("removeQueueIfEmpty", {
+      numberOfKeys: 2,
+      lua: `
+local masterKey = KEYS[1]
+local queueKey = KEYS[2]
+local queueId = ARGV[1]
+
+-- Check if queue is empty
+local count = redis.call('ZCARD', queueKey)
+if count > 0 then
+  return 0
+end
+
+-- Remove from master queue
+redis.call('ZREM', masterKey, queueId)
+return 1
+      `,
+    });
+  }
+}
+
+// Extend Redis interface for custom commands
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    addQueueIfNotEmpty(masterKey: string, queueKey: string, queueId: string): Promise<number>;
+
+    removeQueueIfEmpty(masterKey: string, queueKey: string, queueId: string): Promise<number>;
+  }
+}

packages/redis-worker/src/fair-queue/retry.ts

@@ -0,0 +1,173 @@
+import { calculateNextRetryDelay } from "@trigger.dev/core/v3";
+import type { RetryOptions } from "@trigger.dev/core/v3/schemas";
+
+/**
+ * RetryStrategy interface for pluggable retry logic.
+ */
+export interface RetryStrategy {
+  /**
+   * Calculate the next retry delay in milliseconds.
+   * Return null to indicate the message should be sent to DLQ.
+   *
+   * @param attempt - Current attempt number (1-indexed)
+   * @param error - Optional error from the failed attempt
+   * @returns Delay in milliseconds, or null to send to DLQ
+   */
+  getNextDelay(attempt: number, error?: Error): number | null;
+
+  /**
+   * Maximum number of attempts before moving to DLQ.
+   */
+  maxAttempts: number;
+}
+
+/**
+ * Exponential backoff retry strategy.
+ *
+ * Uses the same algorithm as @trigger.dev/core's calculateNextRetryDelay.
+ */
+export class ExponentialBackoffRetry implements RetryStrategy {
+  readonly maxAttempts: number;
+  private options: RetryOptions;
+
+  constructor(options?: Partial<RetryOptions>) {
+    this.options = {
+      maxAttempts: options?.maxAttempts ?? 12,
+      factor: options?.factor ?? 2,
+      minTimeoutInMs: options?.minTimeoutInMs ?? 1_000,
+      maxTimeoutInMs: options?.maxTimeoutInMs ?? 3_600_000, // 1 hour
+      randomize: options?.randomize ?? true,
+    };
+    this.maxAttempts = this.options.maxAttempts ?? 12;
+  }
+
+  getNextDelay(attempt: number, _error?: Error): number | null {
+    if (attempt >= this.maxAttempts) {
+      return null; // Send to DLQ
+    }
+
+    const delay = calculateNextRetryDelay(this.options, attempt);
+    return delay ?? null;
+  }
+}
+
+/**
+ * Fixed delay retry strategy.
+ *
+ * Always waits the same amount of time between retries.
+ */
+export class FixedDelayRetry implements RetryStrategy {
+  readonly maxAttempts: number;
+  private delayMs: number;
+
+  constructor(options: { maxAttempts: number; delayMs: number }) {
+    this.maxAttempts = options.maxAttempts;
+    this.delayMs = options.delayMs;
+  }
+
+  getNextDelay(attempt: number, _error?: Error): number | null {
+    if (attempt >= this.maxAttempts) {
+      return null; // Send to DLQ
+    }
+    return this.delayMs;
+  }
+}
+
+/**
+ * Linear backoff retry strategy.
+ *
+ * Delay increases linearly with each attempt.
+ */
+export class LinearBackoffRetry implements RetryStrategy {
+  readonly maxAttempts: number;
+  private baseDelayMs: number;
+  private maxDelayMs: number;
+
+  constructor(options: { maxAttempts: number; baseDelayMs: number; maxDelayMs?: number }) {
+    this.maxAttempts = options.maxAttempts;
+    this.baseDelayMs = options.baseDelayMs;
+    this.maxDelayMs = options.maxDelayMs ?? options.baseDelayMs * options.maxAttempts;
+  }
+
+  getNextDelay(attempt: number, _error?: Error): number | null {
+    if (attempt >= this.maxAttempts) {
+      return null; // Send to DLQ
+    }
+    const delay = this.baseDelayMs * attempt;
+    return Math.min(delay, this.maxDelayMs);
+  }
+}
+
+/**
+ * No retry strategy.
+ *
+ * Messages go directly to DLQ on first failure.
+ */
+export class NoRetry implements RetryStrategy {
+  readonly maxAttempts = 1;
+
+  getNextDelay(_attempt: number, _error?: Error): number | null {
+    return null; // Always send to DLQ
+  }
+}
+
+/**
+ * Immediate retry strategy.
+ *
+ * Retries immediately without any delay.
+ */
+export class ImmediateRetry implements RetryStrategy {
+  readonly maxAttempts: number;
+
+  constructor(maxAttempts: number) {
+    this.maxAttempts = maxAttempts;
+  }
+
+  getNextDelay(attempt: number, _error?: Error): number | null {
+    if (attempt >= this.maxAttempts) {
+      return null; // Send to DLQ
+    }
+    return 0; // Immediate retry
+  }
+}
+
+/**
+ * Custom retry strategy that uses a user-provided function.
+ */
+export class CustomRetry implements RetryStrategy {
+  readonly maxAttempts: number;
+  private calculateDelay: (attempt: number, error?: Error) => number | null;
+
+  constructor(options: {
+    maxAttempts: number;
+    calculateDelay: (attempt: number, error?: Error) => number | null;
+  }) {
+    this.maxAttempts = options.maxAttempts;
+    this.calculateDelay = options.calculateDelay;
+  }
+
+  getNextDelay(attempt: number, error?: Error): number | null {
+    if (attempt >= this.maxAttempts) {
+      return null;
+    }
+    return this.calculateDelay(attempt, error);
+  }
+}
+
+/**
+ * Default retry options matching @trigger.dev/core defaults.
+ */
+export const defaultRetryOptions: RetryOptions = {
+  maxAttempts: 12,
+  factor: 2,
+  minTimeoutInMs: 1_000,
+  maxTimeoutInMs: 3_600_000,
+  randomize: true,
+};
+
+/**
+ * Create an exponential backoff retry strategy with default options.
+ */
+export function createDefaultRetryStrategy(): RetryStrategy {
+  return new ExponentialBackoffRetry(defaultRetryOptions);
+}

packages/redis-worker/src/fair-queue/scheduler.ts

@@ -0,0 +1,116 @@
+import type { FairScheduler, SchedulerContext, TenantQueues } from "./types.js";
+
+/**
+ * Re-export scheduler types for convenience.
+ */
+export type { FairScheduler, SchedulerContext, TenantQueues };
+
+/**
+ * Base class for scheduler implementations.
+ * Provides common utilities and default implementations.
+ */
+export abstract class BaseScheduler implements FairScheduler {
+  /**
+   * Select queues for processing from a master queue shard.
+   * Must be implemented by subclasses.
+   */
+  abstract selectQueues(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<TenantQueues[]>;
+
+  /**
+   * Called after processing a message to update scheduler state.
+   * Default implementation does nothing.
+   */
+  async recordProcessed(_tenantId: string, _queueId: string): Promise<void> {
+    // Default: no state tracking
+  }
+
+  /**
+   * Called after processing multiple messages to update scheduler state.
+   * Batch variant for efficiency - reduces Redis calls when processing multiple messages.
+   * Default implementation does nothing.
+   */
+  async recordProcessedBatch(_tenantId: string, _queueId: string, _count: number): Promise<void> {
+    // Default: no state tracking
+  }
+
+  /**
+   * Initialize the scheduler.
+   * Default implementation does nothing.
+   */
+  async initialize(): Promise<void> {
+    // Default: no initialization needed
+  }
+
+  /**
+   * Cleanup scheduler resources.
+   * Default implementation does nothing.
+   */
+  async close(): Promise<void> {
+    // Default: no cleanup needed
+  }
+
+  /**
+   * Helper to group queues by tenant.
+   */
+  protected groupQueuesByTenant(
+    queues: Array<{ queueId: string; tenantId: string }>
+  ): Map<string, string[]> {
+    const grouped = new Map<string, string[]>();
+
+    for (const { queueId, tenantId } of queues) {
+      const existing = grouped.get(tenantId) ?? [];
+      existing.push(queueId);
+      grouped.set(tenantId, existing);
+    }
+
+    return grouped;
+  }
+
+  /**
+   * Helper to convert grouped queues to TenantQueues array.
+   */
+  protected toTenantQueuesArray(grouped: Map<string, string[]>): TenantQueues[] {
+    return Array.from(grouped.entries()).map(([tenantId, queues]) => ({
+      tenantId,
+      queues,
+    }));
+  }
+
+  /**
+   * Helper to filter out tenants at capacity.
+   */
+  protected async filterAtCapacity(
+    tenants: TenantQueues[],
+    context: SchedulerContext,
+    groupName: string = "tenant"
+  ): Promise<TenantQueues[]> {
+    const filtered: TenantQueues[] = [];
+
+    for (const tenant of tenants) {
+      const isAtCapacity = await context.isAtCapacity(groupName, tenant.tenantId);
+      if (!isAtCapacity) {
+        filtered.push(tenant);
+      }
+    }
+
+    return filtered;
+  }
+}
+
+/**
+ * Simple noop scheduler that returns empty results.
+ * Useful for testing or disabling scheduling.
+ */
+export class NoopScheduler extends BaseScheduler {
+  async selectQueues(
+    _masterQueueShard: string,
+    _consumerId: string,
+    _context: SchedulerContext
+  ): Promise<TenantQueues[]> {
+    return [];
+  }
+}

packages/redis-worker/src/fair-queue/schedulers/drr.ts

@@ -0,0 +1,454 @@
+import { createRedisClient, type Redis } from "@internal/redis";
+import { BaseScheduler } from "../scheduler.js";
+import type {
+  DRRSchedulerConfig,
+  DispatchSchedulerContext,
+  FairQueueKeyProducer,
+  SchedulerContext,
+  TenantQueues,
+  QueueWithScore,
+} from "../types.js";
+
+/**
+ * Deficit Round Robin (DRR) Scheduler.
+ *
+ * DRR ensures fair processing across tenants by:
+ * - Allocating a "quantum" of credits to each tenant per round
+ * - Accumulating unused credits as "deficit"
+ * - Processing from tenants with available deficit
+ * - Capping deficit to prevent starvation
+ *
+ * Key improvements over basic implementations:
+ * - Atomic deficit operations using Lua scripts
+ * - Efficient iteration through tenants
+ * - Automatic deficit cleanup for inactive tenants
+ */
+export class DRRScheduler extends BaseScheduler {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private quantum: number;
+  private maxDeficit: number;
+  private masterQueueLimit: number;
+  private logger: NonNullable<DRRSchedulerConfig["logger"]>;
+
+  constructor(private config: DRRSchedulerConfig) {
+    super();
+    this.redis = createRedisClient(config.redis);
+    this.keys = config.keys;
+    this.quantum = config.quantum;
+    this.maxDeficit = config.maxDeficit;
+    this.masterQueueLimit = config.masterQueueLimit ?? 1000;
+    this.logger = config.logger ?? {
+      debug: () => {},
+      error: () => {},
+    };
+
+    this.#registerCommands();
+  }
+
+  // ============================================================================
+  // FairScheduler Implementation
+  // ============================================================================
+
+  /**
+   * Select queues for processing using DRR algorithm.
+   *
+   * Algorithm:
+   * 1. Get all queues from the master shard
+   * 2. Group by tenant
+   * 3. Filter out tenants at concurrency capacity
+   * 4. Add quantum to each tenant's deficit (atomically)
+   * 5. Select queues from tenants with deficit >= 1
+   * 6. Order tenants by deficit (highest first for fairness)
+   */
+  async selectQueues(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<TenantQueues[]> {
+    // Get all queues from the master shard
+    const queues = await this.#getQueuesFromShard(masterQueueShard);
+
+    if (queues.length === 0) {
+      return [];
+    }
+
+    // Group queues by tenant
+    const queuesByTenant = this.groupQueuesByTenant(
+      queues.map((q) => ({ queueId: q.queueId, tenantId: q.tenantId }))
+    );
+
+    // Get unique tenant IDs
+    const tenantIds = Array.from(queuesByTenant.keys());
+
+    // Add quantum to all active tenants atomically
+    const deficits = await this.#addQuantumToTenants(tenantIds);
+
+    // Build tenant data with deficits
+    const tenantData: Array<{
+      tenantId: string;
+      deficit: number;
+      queues: string[];
+      isAtCapacity: boolean;
+    }> = await Promise.all(
+      tenantIds.map(async (tenantId, index) => {
+        const isAtCapacity = await context.isAtCapacity("tenant", tenantId);
+        return {
+          tenantId,
+          deficit: deficits[index] ?? 0,
+          queues: queuesByTenant.get(tenantId) ?? [],
+          isAtCapacity,
+        };
+      })
+    );
+
+    // Filter out tenants at capacity or with no deficit
+    const eligibleTenants = tenantData.filter((t) => !t.isAtCapacity && t.deficit >= 1);
+
+    // Log tenants blocked by capacity
+    const blockedTenants = tenantData.filter((t) => t.isAtCapacity);
+    if (blockedTenants.length > 0) {
+      this.logger.debug("DRR: tenants blocked by concurrency", {
+        blockedCount: blockedTenants.length,
+        blockedTenants: blockedTenants.map((t) => t.tenantId),
+      });
+    }
+
+    // Sort by deficit (highest first for fairness)
+    eligibleTenants.sort((a, b) => b.deficit - a.deficit);
+
+    this.logger.debug("DRR: queue selection complete", {
+      totalQueues: queues.length,
+      totalTenants: tenantIds.length,
+      eligibleTenants: eligibleTenants.length,
+      topTenantDeficit: eligibleTenants[0]?.deficit,
+    });
+
+    // Convert to TenantQueues format
+    return eligibleTenants.map((t) => ({
+      tenantId: t.tenantId,
+      queues: t.queues,
+    }));
+  }
+
+  /**
+   * Select queues using the two-level tenant dispatch index.
+   *
+   * Algorithm:
+   * 1. ZRANGEBYSCORE on dispatch index (gets only tenants with queues - much smaller)
+   * 2. Add quantum to each tenant's deficit (atomically)
+   * 3. Check capacity as safety net (dispatch should only have tenants with capacity)
+   * 4. Select tenants with deficit >= 1, sorted by deficit (highest first)
+   * 5. For each tenant, fetch their queues from Level 2 index
+   */
+  async selectQueuesFromDispatch(
+    dispatchShardKey: string,
+    consumerId: string,
+    context: DispatchSchedulerContext
+  ): Promise<TenantQueues[]> {
+    // Level 1: Get tenants from dispatch index
+    const tenants = await this.#getTenantsFromDispatch(dispatchShardKey);
+
+    if (tenants.length === 0) {
+      return [];
+    }
+
+    const tenantIds = tenants.map((t) => t.tenantId);
+
+    // Add quantum to all active tenants atomically (1 Lua call)
+    const deficits = await this.#addQuantumToTenants(tenantIds);
+
+    // Build candidates sorted by deficit (highest first)
+    const candidates = tenantIds
+      .map((tenantId, index) => ({ tenantId, deficit: deficits[index] ?? 0 }))
+      .filter((t) => t.deficit >= 1);
+
+    candidates.sort((a, b) => b.deficit - a.deficit);
+
+    // Pick the first tenant with available capacity and fetch their queues.
+    // This keeps the scheduler cheap: O(1) in the common case where the
+    // highest-deficit tenant has capacity. The consumer loop iterates fast
+    // (1ms yield between rounds) so we cycle through tenants quickly.
+    for (const { tenantId, deficit } of candidates) {
+      const isAtCapacity = await context.isAtCapacity("tenant", tenantId);
+      if (isAtCapacity) continue;
+
+      // Limit queues fetched to what the tenant can actually process this round.
+      // deficit = max messages this tenant should process, so no point fetching
+      // more queues than that (each queue yields at least 1 message).
+      const queueLimit = Math.ceil(deficit);
+      const queues = await context.getQueuesForTenant(tenantId, queueLimit);
+      if (queues.length > 0) {
+        this.logger.debug("DRR dispatch: selected tenant", {
+          dispatchTenants: tenants.length,
+          candidates: candidates.length,
+          selectedTenant: tenantId,
+          deficit,
+          queueLimit,
+          queuesReturned: queues.length,
+        });
+
+        return [{ tenantId, queues: queues.map((q) => q.queueId) }];
+      }
+    }
+
+    return [];
+  }
+
+  /**
+   * Record that a message was processed from a tenant.
+   * Decrements the tenant's deficit.
+   */
+  override async recordProcessed(tenantId: string, _queueId: string): Promise<void> {
+    await this.#decrementDeficit(tenantId);
+  }
+
+  /**
+   * Record that multiple messages were processed from a tenant.
+   * Decrements the tenant's deficit by count atomically.
+   */
+  override async recordProcessedBatch(
+    tenantId: string,
+    _queueId: string,
+    count: number
+  ): Promise<void> {
+    await this.#decrementDeficitBatch(tenantId, count);
+  }
+
+  override async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Public Methods for Deficit Management
+  // ============================================================================
+
+  /**
+   * Get the current deficit for a tenant.
+   */
+  async getDeficit(tenantId: string): Promise<number> {
+    const key = this.#deficitKey();
+    const value = await this.redis.hget(key, tenantId);
+    return value ? parseFloat(value) : 0;
+  }
+
+  /**
+   * Reset deficit for a tenant.
+   * Used when a tenant has no more active queues.
+   */
+  async resetDeficit(tenantId: string): Promise<void> {
+    const key = this.#deficitKey();
+    await this.redis.hdel(key, tenantId);
+  }
+
+  /**
+   * Get all tenant deficits.
+   */
+  async getAllDeficits(): Promise<Map<string, number>> {
+    const key = this.#deficitKey();
+    const data = await this.redis.hgetall(key);
+    const result = new Map<string, number>();
+    for (const [tenantId, value] of Object.entries(data)) {
+      result.set(tenantId, parseFloat(value));
+    }
+    return result;
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  #deficitKey(): string {
+    // Use a fixed key for DRR deficit tracking
+    return `${this.keys.masterQueueKey(0).split(":")[0]}:drr:deficit`;
+  }
+
+  async #getTenantsFromDispatch(
+    dispatchKey: string
+  ): Promise<Array<{ tenantId: string; score: number }>> {
+    const now = Date.now();
+    const results = await this.redis.zrangebyscore(
+      dispatchKey,
+      "-inf",
+      now,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      this.masterQueueLimit
+    );
+
+    const tenants: Array<{ tenantId: string; score: number }> = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const tenantId = results[i];
+      const scoreStr = results[i + 1];
+      if (tenantId && scoreStr) {
+        tenants.push({
+          tenantId,
+          score: parseFloat(scoreStr),
+        });
+      }
+    }
+
+    return tenants;
+  }
+
+  async #getQueuesFromShard(shardKey: string): Promise<QueueWithScore[]> {
+    const now = Date.now();
+    const results = await this.redis.zrangebyscore(
+      shardKey,
+      "-inf",
+      now,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      this.masterQueueLimit
+    );
+
+    const queues: QueueWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const queueId = results[i];
+      const scoreStr = results[i + 1];
+      if (queueId && scoreStr) {
+        queues.push({
+          queueId,
+          score: parseFloat(scoreStr),
+          tenantId: this.keys.extractTenantId(queueId),
+        });
+      }
+    }
+
+    return queues;
+  }
+
+  /**
+   * Add quantum to multiple tenants atomically.
+   * Returns the new deficit values.
+   */
+  async #addQuantumToTenants(tenantIds: string[]): Promise<number[]> {
+    if (tenantIds.length === 0) {
+      return [];
+    }
+
+    const key = this.#deficitKey();
+
+    // Use Lua script for atomic quantum addition with capping
+    const results = await this.redis.drrAddQuantum(
+      key,
+      this.quantum.toString(),
+      this.maxDeficit.toString(),
+      ...tenantIds
+    );
+
+    return results.map((r) => parseFloat(r));
+  }
+
+  /**
+   * Decrement deficit for a tenant atomically.
+   */
+  async #decrementDeficit(tenantId: string): Promise<number> {
+    const key = this.#deficitKey();
+
+    // Use Lua script to decrement and ensure non-negative
+    const result = await this.redis.drrDecrementDeficit(key, tenantId);
+    return parseFloat(result);
+  }
+
+  /**
+   * Decrement deficit for a tenant by a count atomically.
+   */
+  async #decrementDeficitBatch(tenantId: string, count: number): Promise<number> {
+    const key = this.#deficitKey();
+
+    // Use Lua script to decrement by count and ensure non-negative
+    const result = await this.redis.drrDecrementDeficitBatch(key, tenantId, count.toString());
+    return parseFloat(result);
+  }
+
+  #registerCommands(): void {
+    // Atomic quantum addition with capping for multiple tenants
+    this.redis.defineCommand("drrAddQuantum", {
+      numberOfKeys: 1,
+      lua: `
+local deficitKey = KEYS[1]
+local quantum = tonumber(ARGV[1])
+local maxDeficit = tonumber(ARGV[2])
+local results = {}
+
+for i = 3, #ARGV do
+  local tenantId = ARGV[i]
+  
+  -- Add quantum to deficit
+  local newDeficit = redis.call('HINCRBYFLOAT', deficitKey, tenantId, quantum)
+  newDeficit = tonumber(newDeficit)
+  
+  -- Cap at maxDeficit
+  if newDeficit > maxDeficit then
+    redis.call('HSET', deficitKey, tenantId, maxDeficit)
+    newDeficit = maxDeficit
+  end
+  
+  table.insert(results, tostring(newDeficit))
+end
+
+return results
+      `,
+    });
+
+    // Atomic deficit decrement with floor at 0
+    this.redis.defineCommand("drrDecrementDeficit", {
+      numberOfKeys: 1,
+      lua: `
+local deficitKey = KEYS[1]
+local tenantId = ARGV[1]
+
+local newDeficit = redis.call('HINCRBYFLOAT', deficitKey, tenantId, -1)
+newDeficit = tonumber(newDeficit)
+
+-- Floor at 0
+if newDeficit < 0 then
+  redis.call('HSET', deficitKey, tenantId, 0)
+  newDeficit = 0
+end
+
+return tostring(newDeficit)
+      `,
+    });
+
+    // Atomic deficit decrement by count with floor at 0
+    this.redis.defineCommand("drrDecrementDeficitBatch", {
+      numberOfKeys: 1,
+      lua: `
+local deficitKey = KEYS[1]
+local tenantId = ARGV[1]
+local count = tonumber(ARGV[2])
+
+local newDeficit = redis.call('HINCRBYFLOAT', deficitKey, tenantId, -count)
+newDeficit = tonumber(newDeficit)
+
+-- Floor at 0
+if newDeficit < 0 then
+  redis.call('HSET', deficitKey, tenantId, 0)
+  newDeficit = 0
+end
+
+return tostring(newDeficit)
+      `,
+    });
+  }
+}
+
+// Extend Redis interface for custom commands
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    drrAddQuantum(
+      deficitKey: string,
+      quantum: string,
+      maxDeficit: string,
+      ...tenantIds: string[]
+    ): Promise<string[]>;
+
+    drrDecrementDeficit(deficitKey: string, tenantId: string): Promise<string>;
+
+    drrDecrementDeficitBatch(deficitKey: string, tenantId: string, count: string): Promise<string>;
+  }
+}

packages/redis-worker/src/fair-queue/schedulers/index.ts

@@ -0,0 +1,7 @@
+/**
+ * Scheduler implementations for the fair queue system.
+ */
+
+export { DRRScheduler } from "./drr.js";
+export { WeightedScheduler } from "./weighted.js";
+export { RoundRobinScheduler } from "./roundRobin.js";

packages/redis-worker/src/fair-queue/schedulers/roundRobin.ts

@@ -0,0 +1,156 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import { BaseScheduler } from "../scheduler.js";
+import type {
+  FairQueueKeyProducer,
+  SchedulerContext,
+  TenantQueues,
+  QueueWithScore,
+} from "../types.js";
+
+export interface RoundRobinSchedulerConfig {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  /** Maximum queues to fetch from master queue per iteration */
+  masterQueueLimit?: number;
+}
+
+/**
+ * Round Robin Scheduler.
+ *
+ * Simple scheduler that processes tenants in strict rotation order.
+ * Maintains a "last served" pointer in Redis to track position.
+ *
+ * Features:
+ * - Predictable ordering (good for debugging)
+ * - Fair rotation through all tenants
+ * - No weighting or bias
+ */
+export class RoundRobinScheduler extends BaseScheduler {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private masterQueueLimit: number;
+
+  constructor(private config: RoundRobinSchedulerConfig) {
+    super();
+    this.redis = createRedisClient(config.redis);
+    this.keys = config.keys;
+    this.masterQueueLimit = config.masterQueueLimit ?? 1000;
+  }
+
+  // ============================================================================
+  // FairScheduler Implementation
+  // ============================================================================
+
+  async selectQueues(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<TenantQueues[]> {
+    const now = Date.now();
+
+    // Get all queues from master shard
+    const queues = await this.#getQueuesFromShard(masterQueueShard, now);
+
+    if (queues.length === 0) {
+      return [];
+    }
+
+    // Group queues by tenant
+    const queuesByTenant = new Map<string, string[]>();
+    const tenantOrder: string[] = [];
+
+    for (const queue of queues) {
+      if (!queuesByTenant.has(queue.tenantId)) {
+        queuesByTenant.set(queue.tenantId, []);
+        tenantOrder.push(queue.tenantId);
+      }
+      queuesByTenant.get(queue.tenantId)!.push(queue.queueId);
+    }
+
+    // Get last served index
+    const lastServedIndex = await this.#getLastServedIndex(masterQueueShard);
+
+    // Rotate tenant order based on last served
+    const rotatedTenants = this.#rotateArray(tenantOrder, lastServedIndex);
+
+    // Filter out tenants at capacity
+    const eligibleTenants: TenantQueues[] = [];
+
+    for (const tenantId of rotatedTenants) {
+      const isAtCapacity = await context.isAtCapacity("tenant", tenantId);
+      if (!isAtCapacity) {
+        const tenantQueues = queuesByTenant.get(tenantId) ?? [];
+        // Sort queues by age (oldest first based on original scores)
+        eligibleTenants.push({
+          tenantId,
+          queues: tenantQueues,
+        });
+      }
+    }
+
+    // Update last served index to the first eligible tenant
+    const firstEligible = eligibleTenants[0];
+    if (firstEligible) {
+      const firstTenantIndex = tenantOrder.indexOf(firstEligible.tenantId);
+      await this.#setLastServedIndex(masterQueueShard, firstTenantIndex + 1);
+    }
+
+    return eligibleTenants;
+  }
+
+  override async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  async #getQueuesFromShard(shardKey: string, maxScore: number): Promise<QueueWithScore[]> {
+    const results = await this.redis.zrangebyscore(
+      shardKey,
+      "-inf",
+      maxScore,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      this.masterQueueLimit
+    );
+
+    const queues: QueueWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const queueId = results[i];
+      const scoreStr = results[i + 1];
+      if (queueId && scoreStr) {
+        queues.push({
+          queueId,
+          score: parseFloat(scoreStr),
+          tenantId: this.keys.extractTenantId(queueId),
+        });
+      }
+    }
+
+    return queues;
+  }
+
+  #lastServedKey(shardKey: string): string {
+    return `${shardKey}:rr:lastServed`;
+  }
+
+  async #getLastServedIndex(shardKey: string): Promise<number> {
+    const key = this.#lastServedKey(shardKey);
+    const value = await this.redis.get(key);
+    return value ? parseInt(value, 10) : 0;
+  }
+
+  async #setLastServedIndex(shardKey: string, index: number): Promise<void> {
+    const key = this.#lastServedKey(shardKey);
+    await this.redis.set(key, index.toString());
+  }
+
+  #rotateArray<T>(array: T[], startIndex: number): T[] {
+    if (array.length === 0) return [];
+    const normalizedIndex = startIndex % array.length;
+    return [...array.slice(normalizedIndex), ...array.slice(0, normalizedIndex)];
+  }
+}

packages/redis-worker/src/fair-queue/schedulers/weighted.ts

@@ -0,0 +1,429 @@
+import { createRedisClient, type Redis } from "@internal/redis";
+import seedrandom from "seedrandom";
+import { BaseScheduler } from "../scheduler.js";
+import type {
+  FairQueueKeyProducer,
+  SchedulerContext,
+  TenantQueues,
+  QueueWithScore,
+  WeightedSchedulerBiases,
+  WeightedSchedulerConfig,
+} from "../types.js";
+
+interface TenantConcurrency {
+  current: number;
+  limit: number;
+}
+
+interface TenantSnapshot {
+  tenantId: string;
+  concurrency: TenantConcurrency;
+  queues: Array<{ queueId: string; age: number }>;
+}
+
+interface QueueSnapshot {
+  id: string;
+  tenants: Map<string, TenantSnapshot>;
+  queues: Array<{ queueId: string; tenantId: string; age: number }>;
+}
+
+const defaultBiases: WeightedSchedulerBiases = {
+  concurrencyLimitBias: 0,
+  availableCapacityBias: 0,
+  queueAgeRandomization: 0,
+};
+
+/**
+ * Weighted Shuffle Scheduler.
+ *
+ * Uses weighted random selection to balance between:
+ * - Concurrency limit (higher limits get more weight)
+ * - Available capacity (tenants with more capacity get more weight)
+ * - Queue age (older queues get priority, with configurable randomization)
+ *
+ * Features:
+ * - Snapshot caching to reduce Redis calls
+ * - Configurable biases for fine-tuning
+ * - Maximum tenant count to limit iteration
+ */
+export class WeightedScheduler extends BaseScheduler {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private rng: seedrandom.PRNG;
+  private biases: WeightedSchedulerBiases;
+  private defaultTenantLimit: number;
+  private masterQueueLimit: number;
+  private reuseSnapshotCount: number;
+  private maximumTenantCount: number;
+
+  // Snapshot cache
+  private snapshotCache: Map<string, { snapshot: QueueSnapshot; reuseCount: number }> = new Map();
+
+  constructor(private config: WeightedSchedulerConfig) {
+    super();
+    this.redis = createRedisClient(config.redis);
+    this.keys = config.keys;
+    this.rng = seedrandom(config.seed);
+    this.biases = config.biases ?? defaultBiases;
+    this.defaultTenantLimit = config.defaultTenantConcurrencyLimit ?? 100;
+    this.masterQueueLimit = config.masterQueueLimit ?? 100;
+    this.reuseSnapshotCount = config.reuseSnapshotCount ?? 0;
+    this.maximumTenantCount = config.maximumTenantCount ?? 0;
+  }
+
+  // ============================================================================
+  // FairScheduler Implementation
+  // ============================================================================
+
+  async selectQueues(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<TenantQueues[]> {
+    const snapshot = await this.#getOrCreateSnapshot(masterQueueShard, consumerId, context);
+
+    if (snapshot.queues.length === 0) {
+      return [];
+    }
+
+    // Shuffle tenants based on weights
+    const shuffledTenants = this.#shuffleTenantsByWeight(snapshot);
+
+    // Order queues within each tenant
+    return shuffledTenants.map((tenantId) => ({
+      tenantId,
+      queues: this.#orderQueuesForTenant(snapshot, tenantId),
+    }));
+  }
+
+  override async close(): Promise<void> {
+    this.snapshotCache.clear();
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  async #getOrCreateSnapshot(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<QueueSnapshot> {
+    const cacheKey = `${masterQueueShard}:${consumerId}`;
+
+    // Check cache
+    if (this.reuseSnapshotCount > 0) {
+      const cached = this.snapshotCache.get(cacheKey);
+      if (cached && cached.reuseCount < this.reuseSnapshotCount) {
+        this.snapshotCache.set(cacheKey, {
+          snapshot: cached.snapshot,
+          reuseCount: cached.reuseCount + 1,
+        });
+        return cached.snapshot;
+      }
+    }
+
+    // Create new snapshot
+    const snapshot = await this.#createSnapshot(masterQueueShard, context);
+
+    // Cache if enabled
+    if (this.reuseSnapshotCount > 0) {
+      this.snapshotCache.set(cacheKey, { snapshot, reuseCount: 0 });
+    }
+
+    return snapshot;
+  }
+
+  async #createSnapshot(
+    masterQueueShard: string,
+    context: SchedulerContext
+  ): Promise<QueueSnapshot> {
+    const now = Date.now();
+
+    // Get queues from master shard
+    let rawQueues = await this.#getQueuesFromShard(masterQueueShard, now);
+
+    if (rawQueues.length === 0) {
+      return { id: crypto.randomUUID(), tenants: new Map(), queues: [] };
+    }
+
+    // Apply maximum tenant count if configured
+    if (this.maximumTenantCount > 0) {
+      rawQueues = this.#selectTopTenantQueues(rawQueues);
+    }
+
+    // Build tenant data
+    const tenantIds = new Set<string>();
+    const queuesByTenant = new Map<string, Array<{ queueId: string; age: number }>>();
+
+    for (const queue of rawQueues) {
+      tenantIds.add(queue.tenantId);
+      const tenantQueues = queuesByTenant.get(queue.tenantId) ?? [];
+      tenantQueues.push({
+        queueId: queue.queueId,
+        age: now - queue.score,
+      });
+      queuesByTenant.set(queue.tenantId, tenantQueues);
+    }
+
+    // Get concurrency for each tenant
+    const tenants = new Map<string, TenantSnapshot>();
+    for (const tenantId of tenantIds) {
+      const [current, limit] = await Promise.all([
+        context.getCurrentConcurrency("tenant", tenantId),
+        context.getConcurrencyLimit("tenant", tenantId),
+      ]);
+
+      // Skip tenants at capacity
+      if (current >= limit) {
+        continue;
+      }
+
+      tenants.set(tenantId, {
+        tenantId,
+        concurrency: { current, limit },
+        queues: queuesByTenant.get(tenantId) ?? [],
+      });
+    }
+
+    // Build final queue list (only from non-capacity tenants)
+    const queues = rawQueues
+      .filter((q) => tenants.has(q.tenantId))
+      .map((q) => ({
+        queueId: q.queueId,
+        tenantId: q.tenantId,
+        age: now - q.score,
+      }));
+
+    return {
+      id: crypto.randomUUID(),
+      tenants,
+      queues,
+    };
+  }
+
+  async #getQueuesFromShard(shardKey: string, maxScore: number): Promise<QueueWithScore[]> {
+    const results = await this.redis.zrangebyscore(
+      shardKey,
+      "-inf",
+      maxScore,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      this.masterQueueLimit
+    );
+
+    const queues: QueueWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const queueId = results[i];
+      const scoreStr = results[i + 1];
+      if (queueId && scoreStr) {
+        queues.push({
+          queueId,
+          score: parseFloat(scoreStr),
+          tenantId: this.keys.extractTenantId(queueId),
+        });
+      }
+    }
+
+    return queues;
+  }
+
+  #selectTopTenantQueues(queues: QueueWithScore[]): QueueWithScore[] {
+    // Group by tenant and calculate average age
+    const queuesByTenant = new Map<string, QueueWithScore[]>();
+    for (const queue of queues) {
+      const tenantQueues = queuesByTenant.get(queue.tenantId) ?? [];
+      tenantQueues.push(queue);
+      queuesByTenant.set(queue.tenantId, tenantQueues);
+    }
+
+    // Calculate average age per tenant
+    const tenantAges = Array.from(queuesByTenant.entries()).map(([tenantId, tQueues]) => {
+      const avgAge = tQueues.reduce((sum, q) => sum + q.score, 0) / tQueues.length;
+      return { tenantId, avgAge };
+    });
+
+    // Weighted shuffle to select top N tenants
+    const maxAge = Math.max(...tenantAges.map((t) => t.avgAge));
+    // Guard against division by zero: if maxAge is 0, assign equal weights
+    const weightedTenants =
+      maxAge === 0
+        ? tenantAges.map((t) => ({
+            tenantId: t.tenantId,
+            weight: 1 / tenantAges.length,
+          }))
+        : tenantAges.map((t) => ({
+            tenantId: t.tenantId,
+            weight: t.avgAge / maxAge,
+          }));
+
+    const selectedTenants = new Set<string>();
+    let remaining = [...weightedTenants];
+    let totalWeight = remaining.reduce((sum, t) => sum + t.weight, 0);
+
+    while (selectedTenants.size < this.maximumTenantCount && remaining.length > 0) {
+      let random = this.rng() * totalWeight;
+      let index = 0;
+
+      while (random > 0 && index < remaining.length) {
+        const item = remaining[index];
+        if (item) {
+          random -= item.weight;
+        }
+        index++;
+      }
+      index = Math.max(0, index - 1);
+
+      const selected = remaining[index];
+      if (selected) {
+        selectedTenants.add(selected.tenantId);
+        totalWeight -= selected.weight;
+        remaining.splice(index, 1);
+      }
+    }
+
+    // Return queues only from selected tenants
+    return queues.filter((q) => selectedTenants.has(q.tenantId));
+  }
+
+  #shuffleTenantsByWeight(snapshot: QueueSnapshot): string[] {
+    const tenantIds = Array.from(snapshot.tenants.keys());
+
+    if (tenantIds.length === 0) {
+      return [];
+    }
+
+    const { concurrencyLimitBias, availableCapacityBias } = this.biases;
+
+    // If no biases, do simple shuffle
+    if (concurrencyLimitBias === 0 && availableCapacityBias === 0) {
+      return this.#shuffle(tenantIds);
+    }
+
+    // Calculate weights
+    const maxLimit = Math.max(
+      ...tenantIds.map((id) => snapshot.tenants.get(id)!.concurrency.limit)
+    );
+
+    const weightedTenants = tenantIds.map((tenantId) => {
+      const tenant = snapshot.tenants.get(tenantId)!;
+      let weight = 1;
+
+      // Concurrency limit bias
+      if (concurrencyLimitBias > 0) {
+        // Guard against division by zero: if maxLimit is 0, treat normalizedLimit as 0
+        const normalizedLimit = maxLimit > 0 ? tenant.concurrency.limit / maxLimit : 0;
+        weight *= 1 + Math.pow(normalizedLimit * concurrencyLimitBias, 2);
+      }
+
+      // Available capacity bias
+      if (availableCapacityBias > 0) {
+        // Guard against division by zero: if limit is 0, treat as fully used (no bonus)
+        const usedPercentage =
+          tenant.concurrency.limit > 0 ? tenant.concurrency.current / tenant.concurrency.limit : 1;
+        const availableBonus = 1 - usedPercentage;
+        weight *= 1 + Math.pow(availableBonus * availableCapacityBias, 2);
+      }
+
+      return { tenantId, weight };
+    });
+
+    return this.#weightedShuffle(weightedTenants);
+  }
+
+  #orderQueuesForTenant(snapshot: QueueSnapshot, tenantId: string): string[] {
+    const tenant = snapshot.tenants.get(tenantId);
+    if (!tenant || tenant.queues.length === 0) {
+      return [];
+    }
+
+    const queues = [...tenant.queues];
+    const { queueAgeRandomization } = this.biases;
+
+    // Strict age-based ordering
+    if (queueAgeRandomization === 0) {
+      return queues.sort((a, b) => b.age - a.age).map((q) => q.queueId);
+    }
+
+    // Weighted random based on age
+    const maxAge = Math.max(...queues.map((q) => q.age));
+    // Guard against division by zero: if maxAge is 0, all queues have equal weight
+    const ageDenom = maxAge === 0 ? 1 : maxAge;
+    const weightedQueues = queues.map((q) => ({
+      queue: q,
+      weight: 1 + (q.age / ageDenom) * queueAgeRandomization,
+    }));
+
+    const result: string[] = [];
+    let remaining = [...weightedQueues];
+    let totalWeight = remaining.reduce((sum, q) => sum + q.weight, 0);
+
+    while (remaining.length > 0) {
+      let random = this.rng() * totalWeight;
+      let index = 0;
+
+      while (random > 0 && index < remaining.length) {
+        const item = remaining[index];
+        if (item) {
+          random -= item.weight;
+        }
+        index++;
+      }
+      index = Math.max(0, index - 1);
+
+      const selected = remaining[index];
+      if (selected) {
+        result.push(selected.queue.queueId);
+        totalWeight -= selected.weight;
+        remaining.splice(index, 1);
+      }
+    }
+
+    return result;
+  }
+
+  #shuffle<T>(array: T[]): T[] {
+    const result = [...array];
+    for (let i = result.length - 1; i > 0; i--) {
+      const j = Math.floor(this.rng() * (i + 1));
+      const temp = result[i];
+      const swapValue = result[j];
+      if (temp !== undefined && swapValue !== undefined) {
+        result[i] = swapValue;
+        result[j] = temp;
+      }
+    }
+    return result;
+  }
+
+  #weightedShuffle(items: Array<{ tenantId: string; weight: number }>): string[] {
+    const result: string[] = [];
+    let remaining = [...items];
+    let totalWeight = remaining.reduce((sum, item) => sum + item.weight, 0);
+
+    while (remaining.length > 0) {
+      let random = this.rng() * totalWeight;
+      let index = 0;
+
+      while (random > 0 && index < remaining.length) {
+        const item = remaining[index];
+        if (item) {
+          random -= item.weight;
+        }
+        index++;
+      }
+      index = Math.max(0, index - 1);
+
+      const selected = remaining[index];
+      if (selected) {
+        result.push(selected.tenantId);
+        totalWeight -= selected.weight;
+        remaining.splice(index, 1);
+      }
+    }
+
+    return result;
+  }
+}

packages/redis-worker/src/fair-queue/telemetry.ts

@@ -0,0 +1,765 @@
+import type {
+  Attributes,
+  Counter,
+  Histogram,
+  Meter,
+  ObservableGauge,
+  Span,
+  SpanKind,
+  SpanOptions,
+  Tracer,
+  Context,
+} from "@internal/tracing";
+import { context, trace, SpanStatusCode, ROOT_CONTEXT } from "@internal/tracing";
+
+/**
+ * Semantic attributes for fair queue messaging operations.
+ */
+export const FairQueueAttributes = {
+  QUEUE_ID: "fairqueue.queue_id",
+  TENANT_ID: "fairqueue.tenant_id",
+  MESSAGE_ID: "fairqueue.message_id",
+  SHARD_ID: "fairqueue.shard_id",
+  WORKER_QUEUE: "fairqueue.worker_queue",
+  CONSUMER_ID: "fairqueue.consumer_id",
+  ATTEMPT: "fairqueue.attempt",
+  CONCURRENCY_GROUP: "fairqueue.concurrency_group",
+  MESSAGE_COUNT: "fairqueue.message_count",
+  RESULT: "fairqueue.result",
+} as const;
+
+/**
+ * Standard messaging semantic attributes.
+ */
+export const MessagingAttributes = {
+  SYSTEM: "messaging.system",
+  OPERATION: "messaging.operation",
+  MESSAGE_ID: "messaging.message_id",
+  DESTINATION_NAME: "messaging.destination.name",
+} as const;
+
+/**
+ * FairQueue metrics collection.
+ */
+export interface FairQueueMetrics {
+  // Counters
+  messagesEnqueued: Counter;
+  messagesCompleted: Counter;
+  messagesFailed: Counter;
+  messagesRetried: Counter;
+  messagesToDLQ: Counter;
+
+  // Histograms
+  processingTime: Histogram;
+  queueTime: Histogram;
+
+  // Observable gauges (registered with callbacks)
+  queueLength: ObservableGauge;
+  masterQueueLength: ObservableGauge;
+  dispatchLength: ObservableGauge;
+  inflightCount: ObservableGauge;
+  dlqLength: ObservableGauge;
+}
+
+/**
+ * Options for creating FairQueue telemetry.
+ */
+export interface TelemetryOptions {
+  tracer?: Tracer;
+  meter?: Meter;
+  /** Custom name for metrics prefix */
+  name?: string;
+}
+
+/**
+ * Telemetry helper for FairQueue.
+ *
+ * Provides:
+ * - Span creation with proper attributes
+ * - Metric recording
+ * - Context propagation helpers
+ */
+export class FairQueueTelemetry {
+  private tracer?: Tracer;
+  private meter?: Meter;
+  private metrics?: FairQueueMetrics;
+  private name: string;
+
+  constructor(options: TelemetryOptions) {
+    this.tracer = options.tracer;
+    this.meter = options.meter;
+    this.name = options.name ?? "fairqueue";
+
+    if (this.meter) {
+      this.#initializeMetrics();
+    }
+  }
+
+  // ============================================================================
+  // Tracing
+  // ============================================================================
+
+  /**
+   * Create a traced span for an operation.
+   * Returns the result of the function, or throws any error after recording it.
+   */
+  async trace<T>(
+    name: string,
+    fn: (span: Span) => Promise<T>,
+    options?: {
+      kind?: SpanKind;
+      attributes?: Attributes;
+    }
+  ): Promise<T> {
+    if (!this.tracer) {
+      // No tracer, just execute the function with a no-op span
+      return fn(noopSpan);
+    }
+
+    const spanOptions: SpanOptions = {
+      kind: options?.kind,
+      attributes: {
+        [MessagingAttributes.SYSTEM]: this.name,
+        ...options?.attributes,
+      },
+    };
+
+    return this.tracer.startActiveSpan(`${this.name}.${name}`, spanOptions, async (span) => {
+      try {
+        const result = await fn(span);
+        return result;
+      } catch (error) {
+        if (error instanceof Error) {
+          span.recordException(error);
+        } else {
+          span.recordException(new Error(String(error)));
+        }
+        throw error;
+      } finally {
+        span.end();
+      }
+    });
+  }
+
+  /**
+   * Synchronous version of trace.
+   */
+  traceSync<T>(
+    name: string,
+    fn: (span: Span) => T,
+    options?: {
+      kind?: SpanKind;
+      attributes?: Attributes;
+    }
+  ): T {
+    if (!this.tracer) {
+      return fn(noopSpan);
+    }
+
+    const spanOptions: SpanOptions = {
+      kind: options?.kind,
+      attributes: {
+        [MessagingAttributes.SYSTEM]: this.name,
+        ...options?.attributes,
+      },
+    };
+
+    return this.tracer.startActiveSpan(`${this.name}.${name}`, spanOptions, (span) => {
+      try {
+        return fn(span);
+      } catch (error) {
+        if (error instanceof Error) {
+          span.recordException(error);
+        } else {
+          span.recordException(new Error(String(error)));
+        }
+        throw error;
+      } finally {
+        span.end();
+      }
+    });
+  }
+
+  // ============================================================================
+  // Metrics
+  // ============================================================================
+
+  /**
+   * Record a message enqueued.
+   */
+  recordEnqueue(attributes?: Attributes): void {
+    this.metrics?.messagesEnqueued.add(1, attributes);
+  }
+
+  /**
+   * Record a batch of messages enqueued.
+   */
+  recordEnqueueBatch(count: number, attributes?: Attributes): void {
+    this.metrics?.messagesEnqueued.add(count, attributes);
+  }
+
+  /**
+   * Record a message completed successfully.
+   */
+  recordComplete(attributes?: Attributes): void {
+    this.metrics?.messagesCompleted.add(1, attributes);
+  }
+
+  /**
+   * Record a message processing failure.
+   */
+  recordFailure(attributes?: Attributes): void {
+    this.metrics?.messagesFailed.add(1, attributes);
+  }
+
+  /**
+   * Record a message retry.
+   */
+  recordRetry(attributes?: Attributes): void {
+    this.metrics?.messagesRetried.add(1, attributes);
+  }
+
+  /**
+   * Record a message sent to DLQ.
+   */
+  recordDLQ(attributes?: Attributes): void {
+    this.metrics?.messagesToDLQ.add(1, attributes);
+  }
+
+  /**
+   * Record message processing time.
+   *
+   * @param durationMs - Processing duration in milliseconds
+   */
+  recordProcessingTime(durationMs: number, attributes?: Attributes): void {
+    this.metrics?.processingTime.record(durationMs, attributes);
+  }
+
+  /**
+   * Record time a message spent waiting in queue.
+   *
+   * @param durationMs - Queue wait time in milliseconds
+   */
+  recordQueueTime(durationMs: number, attributes?: Attributes): void {
+    this.metrics?.queueTime.record(durationMs, attributes);
+  }
+
+  /**
+   * Register observable gauge callbacks.
+   * Call this after FairQueue is initialized to register the gauge callbacks.
+   */
+  registerGaugeCallbacks(callbacks: {
+    getQueueLength?: (queueId: string) => Promise<number>;
+    getMasterQueueLength?: (shardId: number) => Promise<number>;
+    getDispatchLength?: (shardId: number) => Promise<number>;
+    getInflightCount?: (shardId: number) => Promise<number>;
+    getDLQLength?: (tenantId: string) => Promise<number>;
+    shardCount?: number;
+    observedQueues?: string[];
+    observedTenants?: string[];
+  }): void {
+    if (!this.metrics) return;
+
+    // Queue length gauge
+    if (callbacks.getQueueLength && callbacks.observedQueues) {
+      const getQueueLength = callbacks.getQueueLength;
+      const queues = callbacks.observedQueues;
+
+      this.metrics.queueLength.addCallback(async (observableResult) => {
+        for (const queueId of queues) {
+          const length = await getQueueLength(queueId);
+          observableResult.observe(length, {
+            [FairQueueAttributes.QUEUE_ID]: queueId,
+          });
+        }
+      });
+    }
+
+    // Legacy master queue length gauge (draining, should trend to 0)
+    if (callbacks.getMasterQueueLength && callbacks.shardCount) {
+      const getMasterQueueLength = callbacks.getMasterQueueLength;
+      const shardCount = callbacks.shardCount;
+
+      this.metrics.masterQueueLength.addCallback(async (observableResult) => {
+        for (let shardId = 0; shardId < shardCount; shardId++) {
+          const length = await getMasterQueueLength(shardId);
+          observableResult.observe(length, {
+            [FairQueueAttributes.SHARD_ID]: shardId.toString(),
+          });
+        }
+      });
+    }
+
+    // Dispatch index length gauge (new two-level dispatch, tenant count per shard)
+    if (callbacks.getDispatchLength && callbacks.shardCount) {
+      const getDispatchLength = callbacks.getDispatchLength;
+      const shardCount = callbacks.shardCount;
+
+      this.metrics.dispatchLength.addCallback(async (observableResult) => {
+        for (let shardId = 0; shardId < shardCount; shardId++) {
+          const length = await getDispatchLength(shardId);
+          observableResult.observe(length, {
+            [FairQueueAttributes.SHARD_ID]: shardId.toString(),
+          });
+        }
+      });
+    }
+
+    // Inflight count gauge
+    if (callbacks.getInflightCount && callbacks.shardCount) {
+      const getInflightCount = callbacks.getInflightCount;
+      const shardCount = callbacks.shardCount;
+
+      this.metrics.inflightCount.addCallback(async (observableResult) => {
+        for (let shardId = 0; shardId < shardCount; shardId++) {
+          const count = await getInflightCount(shardId);
+          observableResult.observe(count, {
+            [FairQueueAttributes.SHARD_ID]: shardId.toString(),
+          });
+        }
+      });
+    }
+
+    // DLQ length gauge
+    if (callbacks.getDLQLength && callbacks.observedTenants) {
+      const getDLQLength = callbacks.getDLQLength;
+      const tenants = callbacks.observedTenants;
+
+      this.metrics.dlqLength.addCallback(async (observableResult) => {
+        for (const tenantId of tenants) {
+          const length = await getDLQLength(tenantId);
+          observableResult.observe(length, {
+            [FairQueueAttributes.TENANT_ID]: tenantId,
+          });
+        }
+      });
+    }
+  }
+
+  // ============================================================================
+  // Helper Methods
+  // ============================================================================
+
+  /**
+   * Create standard attributes for a message operation (for spans/traces).
+   * Use this for span attributes where high cardinality is acceptable.
+   */
+  messageAttributes(params: {
+    queueId?: string;
+    tenantId?: string;
+    messageId?: string;
+    attempt?: number;
+    workerQueue?: string;
+    consumerId?: string;
+  }): Attributes {
+    const attrs: Attributes = {};
+
+    if (params.queueId) attrs[FairQueueAttributes.QUEUE_ID] = params.queueId;
+    if (params.tenantId) attrs[FairQueueAttributes.TENANT_ID] = params.tenantId;
+    if (params.messageId) attrs[FairQueueAttributes.MESSAGE_ID] = params.messageId;
+    if (params.attempt !== undefined) attrs[FairQueueAttributes.ATTEMPT] = params.attempt;
+    if (params.workerQueue) attrs[FairQueueAttributes.WORKER_QUEUE] = params.workerQueue;
+    if (params.consumerId) attrs[FairQueueAttributes.CONSUMER_ID] = params.consumerId;
+
+    return attrs;
+  }
+
+  /**
+   * Check if telemetry is enabled.
+   */
+  get isEnabled(): boolean {
+    return !!this.tracer || !!this.meter;
+  }
+
+  /**
+   * Check if tracing is enabled.
+   */
+  get hasTracer(): boolean {
+    return !!this.tracer;
+  }
+
+  /**
+   * Check if metrics are enabled.
+   */
+  get hasMetrics(): boolean {
+    return !!this.meter;
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  #initializeMetrics(): void {
+    if (!this.meter) return;
+
+    this.metrics = {
+      // Counters
+      messagesEnqueued: this.meter.createCounter(`${this.name}.messages.enqueued`, {
+        description: "Number of messages enqueued",
+        unit: "messages",
+      }),
+      messagesCompleted: this.meter.createCounter(`${this.name}.messages.completed`, {
+        description: "Number of messages completed successfully",
+        unit: "messages",
+      }),
+      messagesFailed: this.meter.createCounter(`${this.name}.messages.failed`, {
+        description: "Number of messages that failed processing",
+        unit: "messages",
+      }),
+      messagesRetried: this.meter.createCounter(`${this.name}.messages.retried`, {
+        description: "Number of message retries",
+        unit: "messages",
+      }),
+      messagesToDLQ: this.meter.createCounter(`${this.name}.messages.dlq`, {
+        description: "Number of messages sent to dead letter queue",
+        unit: "messages",
+      }),
+
+      // Histograms
+      processingTime: this.meter.createHistogram(`${this.name}.message.processing_time`, {
+        description: "Message processing time",
+        unit: "ms",
+      }),
+      queueTime: this.meter.createHistogram(`${this.name}.message.queue_time`, {
+        description: "Time message spent waiting in queue",
+        unit: "ms",
+      }),
+
+      // Observable gauges
+      queueLength: this.meter.createObservableGauge(`${this.name}.queue.length`, {
+        description: "Number of messages in a queue",
+        unit: "messages",
+      }),
+      masterQueueLength: this.meter.createObservableGauge(`${this.name}.master_queue.length`, {
+        description: "Number of queues in legacy master queue shard (draining)",
+        unit: "queues",
+      }),
+      dispatchLength: this.meter.createObservableGauge(`${this.name}.dispatch.length`, {
+        description: "Number of tenants in dispatch index shard",
+        unit: "tenants",
+      }),
+      inflightCount: this.meter.createObservableGauge(`${this.name}.inflight.count`, {
+        description: "Number of messages currently being processed",
+        unit: "messages",
+      }),
+      dlqLength: this.meter.createObservableGauge(`${this.name}.dlq.length`, {
+        description: "Number of messages in dead letter queue",
+        unit: "messages",
+      }),
+    };
+  }
+}
+
+// ============================================================================
+// Batched Span Manager
+// ============================================================================
+
+/**
+ * State for tracking a consumer loop's batched span.
+ */
+export interface ConsumerLoopState {
+  /** Countdown of iterations before starting a new span */
+  perTraceCountdown: number;
+  /** When the current trace started */
+  traceStartedAt: Date;
+  /** The current batched span */
+  currentSpan?: Span;
+  /** The context for the current batched span */
+  currentSpanContext?: Context;
+  /** Number of iterations in the current span */
+  iterationsCount: number;
+  /** Total iterations across all spans */
+  totalIterationsCount: number;
+  /** Running duration in milliseconds for the current span */
+  runningDurationInMs: number;
+  /** Stats counters for the current span */
+  stats: Record<string, number>;
+  /** Flag to force span end on next iteration */
+  endSpanInNextIteration: boolean;
+}
+
+/**
+ * Configuration for the BatchedSpanManager.
+ */
+export interface BatchedSpanManagerOptions {
+  /** The tracer to use for creating spans */
+  tracer?: Tracer;
+  /** Name prefix for spans */
+  name: string;
+  /** Maximum iterations before rotating the span */
+  maxIterations: number;
+  /** Maximum seconds before rotating the span */
+  timeoutSeconds: number;
+  /** Optional callback to get dynamic attributes when starting a new batched span */
+  getDynamicAttributes?: () => Attributes;
+}
+
+/**
+ * Manages batched spans for consumer loops.
+ *
+ * This allows multiple iterations to be grouped into a single parent span,
+ * reducing the volume of spans while maintaining observability.
+ */
+export class BatchedSpanManager {
+  private tracer?: Tracer;
+  private name: string;
+  private maxIterations: number;
+  private timeoutSeconds: number;
+  private loopStates = new Map<string, ConsumerLoopState>();
+  private getDynamicAttributes?: () => Attributes;
+
+  constructor(options: BatchedSpanManagerOptions) {
+    this.tracer = options.tracer;
+    this.name = options.name;
+    this.maxIterations = options.maxIterations;
+    this.timeoutSeconds = options.timeoutSeconds;
+    this.getDynamicAttributes = options.getDynamicAttributes;
+  }
+
+  /**
+   * Initialize state for a consumer loop.
+   */
+  initializeLoop(loopId: string): void {
+    this.loopStates.set(loopId, {
+      perTraceCountdown: this.maxIterations,
+      traceStartedAt: new Date(),
+      iterationsCount: 0,
+      totalIterationsCount: 0,
+      runningDurationInMs: 0,
+      stats: {},
+      endSpanInNextIteration: false,
+    });
+  }
+
+  /**
+   * Get the state for a consumer loop.
+   */
+  getState(loopId: string): ConsumerLoopState | undefined {
+    return this.loopStates.get(loopId);
+  }
+
+  /**
+   * Increment a stat counter for a loop.
+   */
+  incrementStat(loopId: string, statName: string, value: number = 1): void {
+    const state = this.loopStates.get(loopId);
+    if (state) {
+      state.stats[statName] = (state.stats[statName] ?? 0) + value;
+    }
+  }
+
+  /**
+   * Mark that the span should end on the next iteration.
+   */
+  markForRotation(loopId: string): void {
+    const state = this.loopStates.get(loopId);
+    if (state) {
+      state.endSpanInNextIteration = true;
+    }
+  }
+
+  /**
+   * Check if the span should be rotated (ended and a new one started).
+   */
+  shouldRotate(loopId: string): boolean {
+    const state = this.loopStates.get(loopId);
+    if (!state) return true;
+
+    return (
+      state.perTraceCountdown <= 0 ||
+      Date.now() - state.traceStartedAt.getTime() > this.timeoutSeconds * 1000 ||
+      state.currentSpanContext === undefined ||
+      state.endSpanInNextIteration
+    );
+  }
+
+  /**
+   * End the current span for a loop and record stats.
+   */
+  endCurrentSpan(loopId: string): void {
+    const state = this.loopStates.get(loopId);
+    if (!state?.currentSpan) return;
+
+    // Record stats as span attributes
+    for (const [statName, count] of Object.entries(state.stats)) {
+      state.currentSpan.setAttribute(`stats.${statName}`, count);
+    }
+
+    state.currentSpan.end();
+    state.currentSpan = undefined;
+    state.currentSpanContext = undefined;
+  }
+
+  /**
+   * Start a new batched span for a loop.
+   */
+  startNewSpan(loopId: string, attributes?: Attributes): void {
+    if (!this.tracer) return;
+
+    const state = this.loopStates.get(loopId);
+    if (!state) return;
+
+    // End any existing span first
+    this.endCurrentSpan(loopId);
+
+    // Calculate metrics from previous span period
+    const traceDurationInMs = state.traceStartedAt
+      ? Date.now() - state.traceStartedAt.getTime()
+      : undefined;
+    const iterationsPerSecond =
+      traceDurationInMs && traceDurationInMs > 0
+        ? state.iterationsCount / (traceDurationInMs / 1000)
+        : undefined;
+
+    // Get dynamic attributes if callback is provided
+    const dynamicAttributes = this.getDynamicAttributes?.() ?? {};
+
+    // Start new span
+    state.currentSpan = this.tracer.startSpan(
+      `${this.name}.consumerLoop`,
+      {
+        kind: 1, // SpanKind.CONSUMER
+        attributes: {
+          loop_id: loopId,
+          max_iterations: this.maxIterations,
+          timeout_seconds: this.timeoutSeconds,
+          previous_iterations: state.iterationsCount,
+          previous_duration_ms: traceDurationInMs,
+          previous_iterations_per_second: iterationsPerSecond,
+          total_iterations: state.totalIterationsCount,
+          ...dynamicAttributes,
+          ...attributes,
+        },
+      },
+      ROOT_CONTEXT
+    );
+
+    // Set up context for child spans
+    state.currentSpanContext = trace.setSpan(ROOT_CONTEXT, state.currentSpan);
+
+    // Reset counters
+    state.perTraceCountdown = this.maxIterations;
+    state.traceStartedAt = new Date();
+    state.iterationsCount = 0;
+    state.runningDurationInMs = 0;
+    state.stats = {};
+    state.endSpanInNextIteration = false;
+  }
+
+  /**
+   * Execute a function within the batched span context.
+   * Automatically handles span rotation and iteration tracking.
+   */
+  async withBatchedSpan<T>(
+    loopId: string,
+    fn: (span: Span) => Promise<T>,
+    options?: {
+      iterationSpanName?: string;
+      attributes?: Attributes;
+    }
+  ): Promise<T> {
+    let state = this.loopStates.get(loopId);
+
+    // Initialize state if not present
+    if (!state) {
+      this.initializeLoop(loopId);
+      state = this.loopStates.get(loopId)!;
+    }
+
+    // Check if we need to rotate the span
+    if (this.shouldRotate(loopId)) {
+      this.startNewSpan(loopId);
+    }
+
+    const startTime = performance.now();
+
+    try {
+      // If no tracer, just execute the function
+      if (!this.tracer || !state.currentSpanContext) {
+        return await fn(noopSpan);
+      }
+
+      // Execute within the batched span context
+      return await context.with(state.currentSpanContext, async () => {
+        // Create an iteration span within the batched span
+        const iterationSpanName = options?.iterationSpanName ?? "iteration";
+
+        return await this.tracer!.startActiveSpan(
+          `${this.name}.${iterationSpanName}`,
+          {
+            attributes: {
+              loop_id: loopId,
+              iteration: state.iterationsCount,
+              ...options?.attributes,
+            },
+          },
+          async (iterationSpan) => {
+            try {
+              return await fn(iterationSpan);
+            } catch (error) {
+              if (error instanceof Error) {
+                iterationSpan.recordException(error);
+                state.currentSpan?.recordException(error);
+              }
+              iterationSpan.setStatus({ code: SpanStatusCode.ERROR });
+              state.endSpanInNextIteration = true;
+              throw error;
+            } finally {
+              iterationSpan.end();
+            }
+          }
+        );
+      });
+    } finally {
+      // Update iteration tracking
+      const duration = performance.now() - startTime;
+      state.runningDurationInMs += duration;
+      state.iterationsCount++;
+      state.totalIterationsCount++;
+      state.perTraceCountdown--;
+    }
+  }
+
+  /**
+   * Clean up state for a loop when it's stopped.
+   */
+  cleanup(loopId: string): void {
+    this.endCurrentSpan(loopId);
+    this.loopStates.delete(loopId);
+  }
+
+  /**
+   * Clean up all loop states.
+   */
+  cleanupAll(): void {
+    for (const loopId of this.loopStates.keys()) {
+      this.cleanup(loopId);
+    }
+  }
+}
+
+/**
+ * No-op span implementation for when telemetry is disabled.
+ */
+const noopSpan: Span = {
+  spanContext: () => ({
+    traceId: "",
+    spanId: "",
+    traceFlags: 0,
+  }),
+  setAttribute: () => noopSpan,
+  setAttributes: () => noopSpan,
+  addEvent: () => noopSpan,
+  addLink: () => noopSpan,
+  addLinks: () => noopSpan,
+  setStatus: () => noopSpan,
+  updateName: () => noopSpan,
+  end: () => {},
+  isRecording: () => false,
+  recordException: () => {},
+};
+
+/**
+ * No-op telemetry instance for when telemetry is disabled.
+ */
+export const noopTelemetry = new FairQueueTelemetry({});

packages/redis-worker/src/fair-queue/tenantDispatch.ts

@@ -0,0 +1,183 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import { jumpHash } from "@trigger.dev/core/v3/serverOnly";
+import type { FairQueueKeyProducer, QueueWithScore } from "./types.js";
+
+export interface TenantDispatchOptions {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  shardCount: number;
+}
+
+export interface TenantWithScore {
+  tenantId: string;
+  score: number;
+}
+
+/**
+ * TenantDispatch manages the two-level tenant dispatch index.
+ *
+ * Level 1 - Dispatch Index (per shard):
+ *   Key: {prefix}:dispatch:{shardId}
+ *   ZSET of tenantIds scored by oldest message timestamp across their queues.
+ *   Only tenants with queues containing messages appear here.
+ *
+ * Level 2 - Per-Tenant Queue Index:
+ *   Key: {prefix}:tenantq:{tenantId}
+ *   ZSET of queueIds scored by oldest message timestamp in that queue.
+ *
+ * This replaces the flat master queue for new enqueues, isolating each tenant's
+ * queue backlog so the scheduler iterates tenants (not queues) at Level 1.
+ */
+export class TenantDispatch {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private shardCount: number;
+
+  constructor(private options: TenantDispatchOptions) {
+    this.redis = createRedisClient(options.redis);
+    this.keys = options.keys;
+    this.shardCount = Math.max(1, options.shardCount);
+  }
+
+  /**
+   * Get the dispatch shard ID for a tenant.
+   * Uses jump consistent hash on the tenant ID so each tenant
+   * always maps to exactly one dispatch shard.
+   */
+  getShardForTenant(tenantId: string): number {
+    return jumpHash(tenantId, this.shardCount);
+  }
+
+  /**
+   * Get eligible tenants from a dispatch shard (Level 1).
+   * Returns tenants ordered by oldest message (lowest score first).
+   */
+  async getTenantsFromShard(
+    shardId: number,
+    limit: number = 1000,
+    maxScore?: number
+  ): Promise<TenantWithScore[]> {
+    const dispatchKey = this.keys.dispatchKey(shardId);
+    const score = maxScore ?? Date.now();
+
+    const results = await this.redis.zrangebyscore(
+      dispatchKey,
+      "-inf",
+      score,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      limit
+    );
+
+    const tenants: TenantWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const tenantId = results[i];
+      const scoreStr = results[i + 1];
+      if (tenantId && scoreStr) {
+        tenants.push({
+          tenantId,
+          score: parseFloat(scoreStr),
+        });
+      }
+    }
+
+    return tenants;
+  }
+
+  /**
+   * Get queues for a specific tenant (Level 2).
+   * Returns queues ordered by oldest message (lowest score first).
+   */
+  async getQueuesForTenant(
+    tenantId: string,
+    limit: number = 1000,
+    maxScore?: number
+  ): Promise<QueueWithScore[]> {
+    const tenantQueueKey = this.keys.tenantQueueIndexKey(tenantId);
+    const score = maxScore ?? Date.now();
+
+    const results = await this.redis.zrangebyscore(
+      tenantQueueKey,
+      "-inf",
+      score,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      limit
+    );
+
+    const queues: QueueWithScore[] = [];
+    for (let i = 0; i < results.length; i += 2) {
+      const queueId = results[i];
+      const scoreStr = results[i + 1];
+      if (queueId && scoreStr) {
+        queues.push({
+          queueId,
+          score: parseFloat(scoreStr),
+          tenantId,
+        });
+      }
+    }
+
+    return queues;
+  }
+
+  /**
+   * Get the number of tenants in a dispatch shard.
+   */
+  async getShardTenantCount(shardId: number): Promise<number> {
+    const dispatchKey = this.keys.dispatchKey(shardId);
+    return await this.redis.zcard(dispatchKey);
+  }
+
+  /**
+   * Get total tenant count across all dispatch shards.
+   * Note: tenants may appear in multiple shards, so this may overcount.
+   */
+  async getTotalTenantCount(): Promise<number> {
+    const counts = await Promise.all(
+      Array.from({ length: this.shardCount }, (_, i) => this.getShardTenantCount(i))
+    );
+    return counts.reduce((sum, count) => sum + count, 0);
+  }
+
+  /**
+   * Get the number of queues for a tenant.
+   */
+  async getTenantQueueCount(tenantId: string): Promise<number> {
+    const tenantQueueKey = this.keys.tenantQueueIndexKey(tenantId);
+    return await this.redis.zcard(tenantQueueKey);
+  }
+
+  /**
+   * Remove a tenant from a specific dispatch shard.
+   */
+  async removeTenantFromShard(shardId: number, tenantId: string): Promise<void> {
+    const dispatchKey = this.keys.dispatchKey(shardId);
+    await this.redis.zrem(dispatchKey, tenantId);
+  }
+
+  /**
+   * Add a tenant to a dispatch shard with the given score.
+   */
+  async addTenantToShard(shardId: number, tenantId: string, score: number): Promise<void> {
+    const dispatchKey = this.keys.dispatchKey(shardId);
+    await this.redis.zadd(dispatchKey, score, tenantId);
+  }
+
+  /**
+   * Remove a queue from a tenant's queue index.
+   */
+  async removeQueueFromTenant(tenantId: string, queueId: string): Promise<void> {
+    const tenantQueueKey = this.keys.tenantQueueIndexKey(tenantId);
+    await this.redis.zrem(tenantQueueKey, queueId);
+  }
+
+  /**
+   * Close the Redis connection.
+   */
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+}

packages/redis-worker/src/fair-queue/tests/concurrency.test.ts

@@ -0,0 +1,739 @@
+import { describe, expect } from "vitest";
+import { redisTest } from "@internal/testcontainers";
+import { ConcurrencyManager } from "../concurrency.js";
+import { DefaultFairQueueKeyProducer } from "../keyProducer.js";
+import type { FairQueueKeyProducer, QueueDescriptor } from "../types.js";
+
+describe("ConcurrencyManager", () => {
+  let keys: FairQueueKeyProducer;
+
+  describe("single group concurrency", () => {
+    redisTest(
+      "should allow processing when under limit",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        const result = await manager.canProcess(queue);
+        expect(result.allowed).toBe(true);
+
+        await manager.close();
+      }
+    );
+
+    redisTest("should block when at capacity", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+      const manager = new ConcurrencyManager({
+        redis: redisOptions,
+        keys,
+        groups: [
+          {
+            name: "tenant",
+            extractGroupId: (q) => q.tenantId,
+            getLimit: async () => 5,
+            defaultLimit: 5,
+          },
+        ],
+      });
+
+      const queue: QueueDescriptor = {
+        id: "queue-1",
+        tenantId: "t1",
+        metadata: {},
+      };
+
+      // Reserve 5 slots (the limit)
+      for (let i = 0; i < 5; i++) {
+        await manager.reserve(queue, `msg-${i}`);
+      }
+
+      const result = await manager.canProcess(queue);
+      expect(result.allowed).toBe(false);
+      expect(result.blockedBy?.groupName).toBe("tenant");
+      expect(result.blockedBy?.current).toBe(5);
+      expect(result.blockedBy?.limit).toBe(5);
+
+      await manager.close();
+    });
+
+    redisTest("should allow after release", { timeout: 15000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+      const manager = new ConcurrencyManager({
+        redis: redisOptions,
+        keys,
+        groups: [
+          {
+            name: "tenant",
+            extractGroupId: (q) => q.tenantId,
+            getLimit: async () => 5,
+            defaultLimit: 5,
+          },
+        ],
+      });
+
+      const queue: QueueDescriptor = {
+        id: "queue-1",
+        tenantId: "t1",
+        metadata: {},
+      };
+
+      // Fill up
+      for (let i = 0; i < 5; i++) {
+        await manager.reserve(queue, `msg-${i}`);
+      }
+
+      // Should be blocked
+      let result = await manager.canProcess(queue);
+      expect(result.allowed).toBe(false);
+
+      // Release one
+      await manager.release(queue, "msg-0");
+
+      // Should be allowed now
+      result = await manager.canProcess(queue);
+      expect(result.allowed).toBe(true);
+
+      await manager.close();
+    });
+  });
+
+  describe("multi-group concurrency", () => {
+    redisTest("should check all groups", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+      const manager = new ConcurrencyManager({
+        redis: redisOptions,
+        keys,
+        groups: [
+          {
+            name: "tenant",
+            extractGroupId: (q) => q.tenantId,
+            getLimit: async () => 5,
+            defaultLimit: 5,
+          },
+          {
+            name: "organization",
+            extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+            getLimit: async () => 10,
+            defaultLimit: 10,
+          },
+        ],
+      });
+
+      const queue: QueueDescriptor = {
+        id: "queue-1",
+        tenantId: "t1",
+        metadata: { orgId: "org1" },
+      };
+
+      // Fill up org level (10)
+      for (let i = 0; i < 10; i++) {
+        await manager.reserve(queue, `msg-${i}`);
+      }
+
+      // Tenant is at 10, over limit of 5
+      // Org is at 10, at limit of 10
+      const result = await manager.canProcess(queue);
+      expect(result.allowed).toBe(false);
+
+      // Should be blocked by tenant first (checked first, limit 5)
+      expect(result.blockedBy?.groupName).toBe("tenant");
+
+      await manager.close();
+    });
+
+    redisTest(
+      "should block if any group is at capacity",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+            {
+              name: "organization",
+              extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        // Use different queue with different tenant but same org
+        const queue1: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: { orgId: "org1" },
+        };
+
+        const queue2: QueueDescriptor = {
+          id: "queue-2",
+          tenantId: "t2",
+          metadata: { orgId: "org1" }, // Same org
+        };
+
+        // Fill up org with messages from both tenants
+        for (let i = 0; i < 5; i++) {
+          await manager.reserve(queue1, `msg-t1-${i}`);
+        }
+        for (let i = 0; i < 5; i++) {
+          await manager.reserve(queue2, `msg-t2-${i}`);
+        }
+
+        // t1 tenant is at 5/5, org is at 10/10
+        let result = await manager.canProcess(queue1);
+        expect(result.allowed).toBe(false);
+
+        // t2 tenant is at 5/5
+        result = await manager.canProcess(queue2);
+        expect(result.allowed).toBe(false);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("getAvailableCapacity", () => {
+    redisTest(
+      "should return available capacity for single group",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        // Initial capacity should be full
+        let capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(10);
+
+        // Reserve 3 slots
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+        await manager.reserve(queue, "msg-3");
+
+        // Capacity should be reduced
+        capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(7);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should return minimum capacity across multiple groups",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+            {
+              name: "organization",
+              extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+              getLimit: async () => 20,
+              defaultLimit: 20,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: { orgId: "org1" },
+        };
+
+        // Initial capacity should be minimum (5 for tenant, 20 for org)
+        let capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(5);
+
+        // Reserve 3 slots
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+        await manager.reserve(queue, "msg-3");
+
+        // Now tenant has 2 left, org has 17 left - minimum is 2
+        capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(2);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should return 0 when any group is at capacity",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 3,
+              defaultLimit: 3,
+            },
+            {
+              name: "organization",
+              extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: { orgId: "org1" },
+        };
+
+        // Fill up tenant capacity
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+        await manager.reserve(queue, "msg-3");
+
+        // Tenant is at 3/3, org is at 3/10
+        const capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(0);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should return 0 when no groups are configured",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        const capacity = await manager.getAvailableCapacity(queue);
+        expect(capacity).toBe(0);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("atomic reservation", () => {
+    redisTest(
+      "should atomically reserve across groups",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+            {
+              name: "organization",
+              extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: { orgId: "org1" },
+        };
+
+        const result = await manager.reserve(queue, "msg-1");
+        expect(result).toBe(true);
+
+        const tenantCurrent = await manager.getCurrentConcurrency("tenant", "t1");
+        const orgCurrent = await manager.getCurrentConcurrency("organization", "org1");
+
+        expect(tenantCurrent).toBe(1);
+        expect(orgCurrent).toBe(1);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should not reserve if any group is at capacity",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        // Fill up tenant
+        for (let i = 0; i < 5; i++) {
+          await manager.reserve(queue, `msg-${i}`);
+        }
+
+        // Try to reserve one more
+        const result = await manager.reserve(queue, "msg-extra");
+        expect(result).toBe(false);
+
+        // Should still be at 5
+        const current = await manager.getCurrentConcurrency("tenant", "t1");
+        expect(current).toBe(5);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("get active messages", () => {
+    redisTest(
+      "should return all active message IDs",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+        await manager.reserve(queue, "msg-3");
+
+        const active = await manager.getActiveMessages("tenant", "t1");
+        expect(active).toHaveLength(3);
+        expect(active).toContain("msg-1");
+        expect(active).toContain("msg-2");
+        expect(active).toContain("msg-3");
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("clear group", () => {
+    redisTest(
+      "should clear all messages for a group",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+
+        await manager.clearGroup("tenant", "t1");
+
+        const current = await manager.getCurrentConcurrency("tenant", "t1");
+        expect(current).toBe(0);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("get state", () => {
+    redisTest(
+      "should return full concurrency state",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        await manager.reserve(queue, "msg-1");
+        await manager.reserve(queue, "msg-2");
+
+        const state = await manager.getState("tenant", "t1");
+        expect(state.groupName).toBe("tenant");
+        expect(state.groupId).toBe("t1");
+        expect(state.current).toBe(2);
+        expect(state.limit).toBe(5);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("group names", () => {
+    redisTest(
+      "should return configured group names",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: redisOptions,
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 5,
+              defaultLimit: 5,
+            },
+            {
+              name: "organization",
+              extractGroupId: (q) => (q.metadata.orgId as string) ?? "default",
+              getLimit: async () => 10,
+              defaultLimit: 10,
+            },
+          ],
+        });
+
+        const names = manager.getGroupNames();
+        expect(names).toEqual(["tenant", "organization"]);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("keyPrefix handling", () => {
+    redisTest(
+      "should correctly reserve and release with keyPrefix",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "myprefix" });
+
+        // Create manager with keyPrefix - this simulates real-world usage
+        const manager = new ConcurrencyManager({
+          redis: {
+            ...redisOptions,
+            keyPrefix: "engine:batch-queue:",
+          },
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 2,
+              defaultLimit: 2,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        // Reserve slots
+        const reserved1 = await manager.reserve(queue, "msg-1");
+        const reserved2 = await manager.reserve(queue, "msg-2");
+        expect(reserved1).toBe(true);
+        expect(reserved2).toBe(true);
+
+        // Should be at capacity
+        let result = await manager.canProcess(queue);
+        expect(result.allowed).toBe(false);
+
+        // Release one - this must use the SAME key as reserve (with keyPrefix)
+        await manager.release(queue, "msg-1");
+
+        // Should now be allowed - this proves reserve and release use the same key
+        result = await manager.canProcess(queue);
+        expect(result.allowed).toBe(true);
+
+        // Verify concurrency count is correct
+        const current = await manager.getCurrentConcurrency("tenant", "t1");
+        expect(current).toBe(1);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should handle reserve/release cycle multiple times with keyPrefix",
+      { timeout: 15000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new ConcurrencyManager({
+          redis: {
+            ...redisOptions,
+            keyPrefix: "myapp:",
+          },
+          keys,
+          groups: [
+            {
+              name: "tenant",
+              extractGroupId: (q) => q.tenantId,
+              getLimit: async () => 1, // Concurrency of 1
+              defaultLimit: 1,
+            },
+          ],
+        });
+
+        const queue: QueueDescriptor = {
+          id: "queue-1",
+          tenantId: "t1",
+          metadata: {},
+        };
+
+        // Simulate processing multiple messages one at a time
+        for (let i = 0; i < 5; i++) {
+          const msgId = `msg-${i}`;
+
+          // Reserve
+          const reserved = await manager.reserve(queue, msgId);
+          expect(reserved).toBe(true);
+
+          // Should be at capacity now
+          const check = await manager.canProcess(queue);
+          expect(check.allowed).toBe(false);
+
+          // Release
+          await manager.release(queue, msgId);
+
+          // Should be free again
+          const checkAfter = await manager.canProcess(queue);
+          expect(checkAfter.allowed).toBe(true);
+        }
+
+        // Final state should be 0 concurrent
+        const current = await manager.getCurrentConcurrency("tenant", "t1");
+        expect(current).toBe(0);
+
+        await manager.close();
+      }
+    );
+  });
+});

packages/redis-worker/src/fair-queue/tests/drr.test.ts

@@ -0,0 +1,459 @@
+import { describe, expect } from "vitest";
+import { redisTest } from "@internal/testcontainers";
+import { createRedisClient } from "@internal/redis";
+import { DRRScheduler } from "../schedulers/drr.js";
+import { DefaultFairQueueKeyProducer } from "../keyProducer.js";
+import type { FairQueueKeyProducer, SchedulerContext } from "../types.js";
+
+describe("DRRScheduler", () => {
+  let keys: FairQueueKeyProducer;
+
+  describe("deficit management", () => {
+    redisTest(
+      "should initialize deficit to 0 for new tenants",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const deficit = await scheduler.getDeficit("new-tenant");
+        expect(deficit).toBe(0);
+
+        await scheduler.close();
+      }
+    );
+
+    redisTest(
+      "should add quantum atomically with capping",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        // Setup: put queues in the master shard
+        const masterKey = keys.masterQueueKey(0);
+        const now = Date.now();
+
+        await redis.zadd(masterKey, now, "tenant:t1:queue:q1");
+
+        // Create context mock
+        const context: SchedulerContext = {
+          getCurrentConcurrency: async () => 0,
+          getConcurrencyLimit: async () => 100,
+          isAtCapacity: async () => false,
+          getQueueDescriptor: (queueId) => ({
+            id: queueId,
+            tenantId: keys.extractTenantId(queueId),
+            metadata: {},
+          }),
+        };
+
+        // Run multiple iterations to accumulate deficit
+        for (let i = 0; i < 15; i++) {
+          await scheduler.selectQueues(masterKey, "consumer-1", context);
+        }
+
+        // Deficit should be capped at maxDeficit (50)
+        const deficit = await scheduler.getDeficit("t1");
+        expect(deficit).toBeLessThanOrEqual(50);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should decrement deficit when processing",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        // Manually set some deficit
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "10");
+
+        // Record processing
+        await scheduler.recordProcessed("t1", "queue:q1");
+
+        const deficit = await scheduler.getDeficit("t1");
+        expect(deficit).toBe(9);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should not go below 0 on decrement",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "0.5");
+
+        await scheduler.recordProcessed("t1", "queue:q1");
+
+        const deficit = await scheduler.getDeficit("t1");
+        expect(deficit).toBe(0);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should decrement deficit by count when using recordProcessedBatch",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        // Manually set deficit
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "15");
+
+        // Record batch processing of 7 messages
+        await scheduler.recordProcessedBatch("t1", "queue:q1", 7);
+
+        const deficit = await scheduler.getDeficit("t1");
+        expect(deficit).toBe(8);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should not go below 0 when recordProcessedBatch decrements more than available",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        // Manually set deficit to 3
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "3");
+
+        // Record batch processing of 10 messages (more than deficit)
+        await scheduler.recordProcessedBatch("t1", "queue:q1", 10);
+
+        const deficit = await scheduler.getDeficit("t1");
+        expect(deficit).toBe(0);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest("should reset deficit for tenant", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+      const redis = createRedisClient(redisOptions);
+
+      const scheduler = new DRRScheduler({
+        redis: redisOptions,
+        keys,
+        quantum: 5,
+        maxDeficit: 50,
+      });
+
+      const deficitKey = `test:drr:deficit`;
+      await redis.hset(deficitKey, "t1", "25");
+
+      await scheduler.resetDeficit("t1");
+
+      const deficit = await scheduler.getDeficit("t1");
+      expect(deficit).toBe(0);
+
+      await scheduler.close();
+      await redis.quit();
+    });
+  });
+
+  describe("queue selection", () => {
+    redisTest(
+      "should return queues grouped by tenant",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const masterKey = keys.masterQueueKey(0);
+        const now = Date.now();
+
+        // Add queues for different tenants (all timestamps in the past)
+        await redis.zadd(
+          masterKey,
+          now - 200,
+          "tenant:t1:queue:q1",
+          now - 100,
+          "tenant:t1:queue:q2",
+          now - 50,
+          "tenant:t2:queue:q1"
+        );
+
+        const context: SchedulerContext = {
+          getCurrentConcurrency: async () => 0,
+          getConcurrencyLimit: async () => 100,
+          isAtCapacity: async () => false,
+          getQueueDescriptor: (queueId) => ({
+            id: queueId,
+            tenantId: keys.extractTenantId(queueId),
+            metadata: {},
+          }),
+        };
+
+        const result = await scheduler.selectQueues(masterKey, "consumer-1", context);
+
+        // Should have both tenants
+        const tenantIds = result.map((r) => r.tenantId);
+        expect(tenantIds).toContain("t1");
+        expect(tenantIds).toContain("t2");
+
+        // t1 should have 2 queues
+        const t1 = result.find((r) => r.tenantId === "t1");
+        expect(t1?.queues).toHaveLength(2);
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should filter out tenants at capacity",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const masterKey = keys.masterQueueKey(0);
+        const now = Date.now();
+
+        await redis.zadd(
+          masterKey,
+          now - 100,
+          "tenant:t1:queue:q1",
+          now - 50,
+          "tenant:t2:queue:q1"
+        );
+
+        const context: SchedulerContext = {
+          getCurrentConcurrency: async () => 0,
+          getConcurrencyLimit: async () => 100,
+          isAtCapacity: async (_, groupId) => groupId === "t1", // t1 at capacity
+          getQueueDescriptor: (queueId) => ({
+            id: queueId,
+            tenantId: keys.extractTenantId(queueId),
+            metadata: {},
+          }),
+        };
+
+        const result = await scheduler.selectQueues(masterKey, "consumer-1", context);
+
+        // Only t2 should be returned
+        const tenantIds = result.map((r) => r.tenantId);
+        expect(tenantIds).not.toContain("t1");
+        expect(tenantIds).toContain("t2");
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should skip tenants with insufficient deficit",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const masterKey = keys.masterQueueKey(0);
+        const now = Date.now();
+
+        await redis.zadd(
+          masterKey,
+          now - 100,
+          "tenant:t1:queue:q1",
+          now - 50,
+          "tenant:t2:queue:q1"
+        );
+
+        // Set t1 deficit to 0 (no credits)
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "0");
+
+        const context: SchedulerContext = {
+          getCurrentConcurrency: async () => 0,
+          getConcurrencyLimit: async () => 100,
+          isAtCapacity: async () => false,
+          getQueueDescriptor: (queueId) => ({
+            id: queueId,
+            tenantId: keys.extractTenantId(queueId),
+            metadata: {},
+          }),
+        };
+
+        // First call adds quantum to both tenants
+        // t1: 0 + 5 = 5, t2: 0 + 5 = 5
+        const result = await scheduler.selectQueues(masterKey, "consumer-1", context);
+
+        // Both should be returned (both have deficit >= 1 after quantum added)
+        const tenantIds = result.map((r) => r.tenantId);
+        expect(tenantIds).toContain("t1");
+        expect(tenantIds).toContain("t2");
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should order tenants by deficit (highest first)",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+        const redis = createRedisClient(redisOptions);
+
+        const scheduler = new DRRScheduler({
+          redis: redisOptions,
+          keys,
+          quantum: 5,
+          maxDeficit: 50,
+        });
+
+        const masterKey = keys.masterQueueKey(0);
+        const now = Date.now();
+
+        await redis.zadd(
+          masterKey,
+          now - 300,
+          "tenant:t1:queue:q1",
+          now - 200,
+          "tenant:t2:queue:q1",
+          now - 100,
+          "tenant:t3:queue:q1"
+        );
+
+        // Set different deficits
+        const deficitKey = `test:drr:deficit`;
+        await redis.hset(deficitKey, "t1", "10");
+        await redis.hset(deficitKey, "t2", "30");
+        await redis.hset(deficitKey, "t3", "20");
+
+        const context: SchedulerContext = {
+          getCurrentConcurrency: async () => 0,
+          getConcurrencyLimit: async () => 100,
+          isAtCapacity: async () => false,
+          getQueueDescriptor: (queueId) => ({
+            id: queueId,
+            tenantId: keys.extractTenantId(queueId),
+            metadata: {},
+          }),
+        };
+
+        const result = await scheduler.selectQueues(masterKey, "consumer-1", context);
+
+        // Should be ordered by deficit: t2 (35), t3 (25), t1 (15)
+        // (original + quantum of 5)
+        expect(result[0]?.tenantId).toBe("t2");
+        expect(result[1]?.tenantId).toBe("t3");
+        expect(result[2]?.tenantId).toBe("t1");
+
+        await scheduler.close();
+        await redis.quit();
+      }
+    );
+  });
+
+  describe("get all deficits", () => {
+    redisTest("should return all tenant deficits", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+      const redis = createRedisClient(redisOptions);
+
+      const scheduler = new DRRScheduler({
+        redis: redisOptions,
+        keys,
+        quantum: 5,
+        maxDeficit: 50,
+      });
+
+      const deficitKey = `test:drr:deficit`;
+      await redis.hset(deficitKey, "t1", "10");
+      await redis.hset(deficitKey, "t2", "20");
+      await redis.hset(deficitKey, "t3", "30");
+
+      const deficits = await scheduler.getAllDeficits();
+
+      expect(deficits.get("t1")).toBe(10);
+      expect(deficits.get("t2")).toBe(20);
+      expect(deficits.get("t3")).toBe(30);
+
+      await scheduler.close();
+      await redis.quit();
+    });
+  });
+});

packages/redis-worker/src/fair-queue/tests/retry.test.ts

@@ -0,0 +1,182 @@
+import { describe, expect, it } from "vitest";
+import {
+  ExponentialBackoffRetry,
+  FixedDelayRetry,
+  LinearBackoffRetry,
+  NoRetry,
+  ImmediateRetry,
+  CustomRetry,
+} from "../retry.js";
+
+describe("RetryStrategy", () => {
+  describe("ExponentialBackoffRetry", () => {
+    it("should return increasing delays", () => {
+      const strategy = new ExponentialBackoffRetry({
+        maxAttempts: 5,
+        factor: 2,
+        minTimeoutInMs: 100,
+        maxTimeoutInMs: 10000,
+        randomize: false,
+      });
+
+      const delay1 = strategy.getNextDelay(1);
+      const delay2 = strategy.getNextDelay(2);
+      const delay3 = strategy.getNextDelay(3);
+
+      // Delays should increase
+      expect(delay1).not.toBeNull();
+      expect(delay2).not.toBeNull();
+      expect(delay3).not.toBeNull();
+      expect(delay2!).toBeGreaterThan(delay1!);
+      expect(delay3!).toBeGreaterThan(delay2!);
+    });
+
+    it("should return null when max attempts reached", () => {
+      const strategy = new ExponentialBackoffRetry({ maxAttempts: 3 });
+
+      expect(strategy.getNextDelay(1)).not.toBeNull();
+      expect(strategy.getNextDelay(2)).not.toBeNull();
+      expect(strategy.getNextDelay(3)).toBeNull();
+    });
+
+    it("should have correct maxAttempts", () => {
+      const strategy = new ExponentialBackoffRetry({ maxAttempts: 7 });
+      expect(strategy.maxAttempts).toBe(7);
+    });
+  });
+
+  describe("FixedDelayRetry", () => {
+    it("should return same delay for all attempts", () => {
+      const strategy = new FixedDelayRetry({ maxAttempts: 5, delayMs: 500 });
+
+      expect(strategy.getNextDelay(1)).toBe(500);
+      expect(strategy.getNextDelay(2)).toBe(500);
+      expect(strategy.getNextDelay(3)).toBe(500);
+      expect(strategy.getNextDelay(4)).toBe(500);
+    });
+
+    it("should return null when max attempts reached", () => {
+      const strategy = new FixedDelayRetry({ maxAttempts: 3, delayMs: 500 });
+
+      expect(strategy.getNextDelay(1)).toBe(500);
+      expect(strategy.getNextDelay(2)).toBe(500);
+      expect(strategy.getNextDelay(3)).toBeNull();
+    });
+  });
+
+  describe("LinearBackoffRetry", () => {
+    it("should return linearly increasing delays", () => {
+      const strategy = new LinearBackoffRetry({
+        maxAttempts: 5,
+        baseDelayMs: 100,
+      });
+
+      expect(strategy.getNextDelay(1)).toBe(100);
+      expect(strategy.getNextDelay(2)).toBe(200);
+      expect(strategy.getNextDelay(3)).toBe(300);
+      expect(strategy.getNextDelay(4)).toBe(400);
+    });
+
+    it("should cap at maxDelayMs", () => {
+      const strategy = new LinearBackoffRetry({
+        maxAttempts: 10,
+        baseDelayMs: 100,
+        maxDelayMs: 250,
+      });
+
+      expect(strategy.getNextDelay(1)).toBe(100);
+      expect(strategy.getNextDelay(2)).toBe(200);
+      expect(strategy.getNextDelay(3)).toBe(250);
+      expect(strategy.getNextDelay(5)).toBe(250);
+    });
+
+    it("should return null when max attempts reached", () => {
+      const strategy = new LinearBackoffRetry({
+        maxAttempts: 3,
+        baseDelayMs: 100,
+      });
+
+      expect(strategy.getNextDelay(3)).toBeNull();
+    });
+  });
+
+  describe("NoRetry", () => {
+    it("should always return null", () => {
+      const strategy = new NoRetry();
+
+      expect(strategy.getNextDelay(1)).toBeNull();
+      expect(strategy.getNextDelay(0)).toBeNull();
+    });
+
+    it("should have maxAttempts of 1", () => {
+      const strategy = new NoRetry();
+      expect(strategy.maxAttempts).toBe(1);
+    });
+  });
+
+  describe("ImmediateRetry", () => {
+    it("should return 0 delay for all attempts", () => {
+      const strategy = new ImmediateRetry(5);
+
+      expect(strategy.getNextDelay(1)).toBe(0);
+      expect(strategy.getNextDelay(2)).toBe(0);
+      expect(strategy.getNextDelay(4)).toBe(0);
+    });
+
+    it("should return null when max attempts reached", () => {
+      const strategy = new ImmediateRetry(3);
+
+      expect(strategy.getNextDelay(3)).toBeNull();
+    });
+  });
+
+  describe("CustomRetry", () => {
+    it("should use custom calculation function", () => {
+      const strategy = new CustomRetry({
+        maxAttempts: 5,
+        calculateDelay: (attempt) => attempt * attempt * 100,
+      });
+
+      expect(strategy.getNextDelay(1)).toBe(100);
+      expect(strategy.getNextDelay(2)).toBe(400);
+      expect(strategy.getNextDelay(3)).toBe(900);
+      expect(strategy.getNextDelay(4)).toBe(1600);
+    });
+
+    it("should pass error to calculation function", () => {
+      const errors: Error[] = [];
+      const strategy = new CustomRetry({
+        maxAttempts: 5,
+        calculateDelay: (_attempt, error) => {
+          if (error) errors.push(error);
+          return 100;
+        },
+      });
+
+      const testError = new Error("test error");
+      strategy.getNextDelay(1, testError);
+
+      expect(errors).toHaveLength(1);
+      expect(errors[0]).toBe(testError);
+    });
+
+    it("should return null when max attempts reached", () => {
+      const strategy = new CustomRetry({
+        maxAttempts: 3,
+        calculateDelay: () => 100,
+      });
+
+      expect(strategy.getNextDelay(3)).toBeNull();
+    });
+
+    it("should allow custom function to return null for DLQ", () => {
+      const strategy = new CustomRetry({
+        maxAttempts: 5,
+        calculateDelay: (attempt) => (attempt === 2 ? null : 100),
+      });
+
+      expect(strategy.getNextDelay(1)).toBe(100);
+      expect(strategy.getNextDelay(2)).toBeNull(); // Custom function says DLQ
+    });
+  });
+});

packages/redis-worker/src/fair-queue/tests/visibility.test.ts

@@ -0,0 +1,916 @@
+import { describe, expect } from "vitest";
+import { redisTest } from "@internal/testcontainers";
+import { createRedisClient } from "@internal/redis";
+import { VisibilityManager, DefaultFairQueueKeyProducer } from "../index.js";
+import type { FairQueueKeyProducer, ReclaimedMessageInfo } from "../types.js";
+
+describe("VisibilityManager", () => {
+  let keys: FairQueueKeyProducer;
+
+  describe("heartbeat", () => {
+    redisTest(
+      "should return true when message exists in in-flight set",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:heartbeat-exists";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add a message to the queue
+        const messageId = "heartbeat-test-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1, value: "test" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        // Claim the message (moves it to in-flight set)
+        const claimResult = await manager.claim(
+          queueId,
+          queueKey,
+          queueItemsKey,
+          "consumer-1",
+          5000
+        );
+        expect(claimResult.claimed).toBe(true);
+
+        // Heartbeat should succeed since message is in-flight
+        const heartbeatResult = await manager.heartbeat(messageId, queueId, 5000);
+        expect(heartbeatResult).toBe(true);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should return false when message does not exist in in-flight set",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        // Heartbeat for a message that was never claimed
+        const heartbeatResult = await manager.heartbeat(
+          "non-existent-msg",
+          "tenant:t1:queue:non-existent",
+          5000
+        );
+        expect(heartbeatResult).toBe(false);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should return false after message is completed",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:heartbeat-after-complete";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add and claim a message
+        const messageId = "completed-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1, value: "test" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        const claimResult = await manager.claim(
+          queueId,
+          queueKey,
+          queueItemsKey,
+          "consumer-1",
+          5000
+        );
+        expect(claimResult.claimed).toBe(true);
+
+        // Heartbeat should work before complete
+        const heartbeatBefore = await manager.heartbeat(messageId, queueId, 5000);
+        expect(heartbeatBefore).toBe(true);
+
+        // Complete the message
+        await manager.complete(messageId, queueId);
+
+        // Heartbeat should fail after complete
+        const heartbeatAfter = await manager.heartbeat(messageId, queueId, 5000);
+        expect(heartbeatAfter).toBe(false);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should correctly update the deadline score",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 1000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:heartbeat-deadline";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add and claim a message with short timeout
+        const messageId = "deadline-test-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1, value: "test" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        // Claim with 1 second timeout
+        await manager.claim(queueId, queueKey, queueItemsKey, "consumer-1", 1000);
+
+        // Get initial deadline
+        const inflightKey = keys.inflightKey(0);
+        const member = `${messageId}:${queueId}`;
+        const initialScore = await redis.zscore(inflightKey, member);
+        expect(initialScore).not.toBeNull();
+
+        // Wait a bit
+        await new Promise((resolve) => setTimeout(resolve, 100));
+
+        // Extend deadline by 10 seconds
+        const beforeHeartbeat = Date.now();
+        const heartbeatSuccess = await manager.heartbeat(messageId, queueId, 10000);
+        expect(heartbeatSuccess).toBe(true);
+
+        // Check that deadline was extended
+        const newScore = await redis.zscore(inflightKey, member);
+        expect(newScore).not.toBeNull();
+
+        // New deadline should be approximately now + 10 seconds
+        const newDeadline = parseFloat(newScore!);
+        expect(newDeadline).toBeGreaterThan(parseFloat(initialScore!));
+        expect(newDeadline).toBeGreaterThanOrEqual(beforeHeartbeat + 10000);
+        // Allow some tolerance for execution time
+        expect(newDeadline).toBeLessThan(beforeHeartbeat + 11000);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should handle multiple consecutive heartbeats",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 1000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:multi-heartbeat";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add and claim a message
+        const messageId = "multi-heartbeat-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1, value: "test" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        await manager.claim(queueId, queueKey, queueItemsKey, "consumer-1", 1000);
+
+        // Multiple heartbeats should all succeed
+        for (let i = 0; i < 5; i++) {
+          const result = await manager.heartbeat(messageId, queueId, 1000);
+          expect(result).toBe(true);
+          await new Promise((resolve) => setTimeout(resolve, 50));
+        }
+
+        // Message should still be in-flight
+        const inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(1);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+  });
+
+  describe("claimBatch", () => {
+    redisTest(
+      "should claim multiple messages atomically",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:claim-batch";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add multiple messages to the queue
+        for (let i = 1; i <= 5; i++) {
+          const messageId = `msg-${i}`;
+          const storedMessage = {
+            id: messageId,
+            queueId,
+            tenantId: "t1",
+            payload: { value: `test-${i}` },
+            timestamp: Date.now() - (6 - i) * 1000,
+            attempt: 1,
+          };
+          await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+          await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+        }
+
+        // Claim batch of 3 messages
+        const claimed = await manager.claimBatch(queueId, queueKey, queueItemsKey, "consumer-1", 3);
+
+        expect(claimed).toHaveLength(3);
+        expect(claimed[0]!.messageId).toBe("msg-1");
+        expect(claimed[1]!.messageId).toBe("msg-2");
+        expect(claimed[2]!.messageId).toBe("msg-3");
+
+        // Verify messages are in in-flight set
+        const inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(3);
+
+        // Verify messages are removed from queue
+        const remainingCount = await redis.zcard(queueKey);
+        expect(remainingCount).toBe(2);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should return empty array when queue is empty",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const queueId = "tenant:t1:queue:empty";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        const claimed = await manager.claimBatch(queueId, queueKey, queueItemsKey, "consumer-1", 5);
+        expect(claimed).toHaveLength(0);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should claim all available messages when queue has fewer than maxCount",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:partial-batch";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add only 3 messages to the queue
+        for (let i = 1; i <= 3; i++) {
+          const messageId = `msg-${i}`;
+          const storedMessage = {
+            id: messageId,
+            queueId,
+            tenantId: "t1",
+            payload: { value: `test-${i}` },
+            timestamp: Date.now() - (4 - i) * 1000,
+            attempt: 1,
+          };
+          await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+          await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+        }
+
+        // Request 10 messages but only 3 exist
+        const claimed = await manager.claimBatch(
+          queueId,
+          queueKey,
+          queueItemsKey,
+          "consumer-1",
+          10
+        );
+
+        expect(claimed).toHaveLength(3);
+        expect(claimed[0]!.messageId).toBe("msg-1");
+        expect(claimed[1]!.messageId).toBe("msg-2");
+        expect(claimed[2]!.messageId).toBe("msg-3");
+
+        // Verify all messages are in in-flight set
+        const inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(3);
+
+        // Verify queue is empty
+        const remainingCount = await redis.zcard(queueKey);
+        expect(remainingCount).toBe(0);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should skip corrupted messages and continue claiming valid ones",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:corrupted-batch";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+
+        // Add valid message 1
+        const storedMessage1 = {
+          id: "msg-1",
+          queueId,
+          tenantId: "t1",
+          payload: { value: "test-1" },
+          timestamp: Date.now() - 3000,
+          attempt: 1,
+        };
+        await redis.zadd(queueKey, storedMessage1.timestamp, "msg-1");
+        await redis.hset(queueItemsKey, "msg-1", JSON.stringify(storedMessage1));
+
+        // Add corrupted message 2 (invalid JSON)
+        await redis.zadd(queueKey, Date.now() - 2000, "msg-2");
+        await redis.hset(queueItemsKey, "msg-2", "not-valid-json{{{");
+
+        // Add valid message 3
+        const storedMessage3 = {
+          id: "msg-3",
+          queueId,
+          tenantId: "t1",
+          payload: { value: "test-3" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+        await redis.zadd(queueKey, storedMessage3.timestamp, "msg-3");
+        await redis.hset(queueItemsKey, "msg-3", JSON.stringify(storedMessage3));
+
+        // Claim all 3 messages
+        const claimed = await manager.claimBatch(queueId, queueKey, queueItemsKey, "consumer-1", 5);
+
+        // Should only return the 2 valid messages
+        expect(claimed).toHaveLength(2);
+        expect(claimed[0]!.messageId).toBe("msg-1");
+        expect(claimed[1]!.messageId).toBe("msg-3");
+
+        // Corrupted message should have been removed from in-flight
+        // Valid messages should be in in-flight
+        const inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(2);
+
+        // Queue should be empty (all messages processed or removed)
+        const remainingCount = await redis.zcard(queueKey);
+        expect(remainingCount).toBe(0);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+  });
+
+  describe("releaseBatch", () => {
+    redisTest(
+      "should release multiple messages back to queue atomically",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:release-batch";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Add messages to queue and claim them
+        for (let i = 1; i <= 5; i++) {
+          const messageId = `msg-${i}`;
+          const storedMessage = {
+            id: messageId,
+            queueId,
+            tenantId: "t1",
+            payload: { value: `test-${i}` },
+            timestamp: Date.now() - (6 - i) * 1000,
+            attempt: 1,
+          };
+          await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+          await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+        }
+
+        // Claim all 5 messages
+        const claimed = await manager.claimBatch(queueId, queueKey, queueItemsKey, "consumer-1", 5);
+        expect(claimed).toHaveLength(5);
+
+        // Verify all messages are in-flight
+        let inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(5);
+
+        // Queue should be empty
+        let queueCount = await redis.zcard(queueKey);
+        expect(queueCount).toBe(0);
+
+        // Release messages 3, 4, 5 back to queue (batch release)
+        const messagesToRelease = claimed.slice(2);
+        await manager.releaseBatch(
+          messagesToRelease,
+          queueId,
+          queueKey,
+          queueItemsKey,
+          tenantQueueIndexKey,
+          dispatchKey,
+          "t1"
+        );
+
+        // Verify 2 messages still in-flight
+        inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(2);
+
+        // Verify 3 messages back in queue
+        queueCount = await redis.zcard(queueKey);
+        expect(queueCount).toBe(3);
+
+        // Verify the correct messages are back in queue
+        const queueMembers = await redis.zrange(queueKey, 0, -1);
+        expect(queueMembers).toContain("msg-3");
+        expect(queueMembers).toContain("msg-4");
+        expect(queueMembers).toContain("msg-5");
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should handle empty messages array",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const queueId = "tenant:t1:queue:empty-release";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Should not throw when releasing empty array
+        await manager.releaseBatch(
+          [],
+          queueId,
+          queueKey,
+          queueItemsKey,
+          tenantQueueIndexKey,
+          dispatchKey,
+          "t1"
+        );
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should update dispatch indexes with oldest message timestamp",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 5000,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:dispatch-update";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Add and claim messages
+        const baseTime = Date.now();
+        for (let i = 1; i <= 3; i++) {
+          const messageId = `msg-${i}`;
+          const storedMessage = {
+            id: messageId,
+            queueId,
+            tenantId: "t1",
+            payload: { value: `test-${i}` },
+            timestamp: baseTime + i * 1000, // Different timestamps
+            attempt: 1,
+          };
+          await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+          await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+        }
+
+        const claimed = await manager.claimBatch(queueId, queueKey, queueItemsKey, "consumer-1", 3);
+
+        // Release all messages back
+        await manager.releaseBatch(
+          claimed,
+          queueId,
+          queueKey,
+          queueItemsKey,
+          tenantQueueIndexKey,
+          dispatchKey,
+          "t1"
+        );
+
+        // Tenant queue index should have the queue with correct score
+        const tenantQueueScore = await redis.zscore(tenantQueueIndexKey, queueId);
+        expect(tenantQueueScore).not.toBeNull();
+
+        // Dispatch index should have the tenant
+        const dispatchScore = await redis.zscore(dispatchKey, "t1");
+        expect(dispatchScore).not.toBeNull();
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+  });
+
+  describe("reclaimTimedOut", () => {
+    redisTest(
+      "should return reclaimed message info with tenantId for concurrency release",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 100, // Very short timeout
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:reclaim-test";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const _tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Add and claim a message
+        const messageId = "reclaim-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1, value: "test" },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+          metadata: { orgId: "org-123" },
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        // Claim with very short timeout
+        const claimResult = await manager.claim(
+          queueId,
+          queueKey,
+          queueItemsKey,
+          "consumer-1",
+          100
+        );
+        expect(claimResult.claimed).toBe(true);
+
+        // Wait for timeout to expire
+        await new Promise((resolve) => setTimeout(resolve, 150));
+
+        // Reclaim should return the message info
+        const reclaimedMessages = await manager.reclaimTimedOut(0, (qId) => ({
+          queueKey: keys.queueKey(qId),
+          queueItemsKey: keys.queueItemsKey(qId),
+          tenantQueueIndexKey: keys.tenantQueueIndexKey(keys.extractTenantId(qId)),
+          dispatchKey,
+          tenantId: keys.extractTenantId(qId),
+        }));
+
+        expect(reclaimedMessages).toHaveLength(1);
+        expect(reclaimedMessages[0]).toEqual({
+          messageId,
+          queueId,
+          tenantId: "t1",
+          metadata: { orgId: "org-123" },
+        });
+
+        // Verify message is back in queue
+        const queueCount = await redis.zcard(queueKey);
+        expect(queueCount).toBe(1);
+
+        // Verify message is back in queue with its original timestamp (not the deadline)
+        const queueMessages = await redis.zrange(queueKey, 0, -1, "WITHSCORES");
+        expect(queueMessages[0]).toBe(messageId);
+        expect(parseInt(queueMessages[1]!)).toBe(storedMessage.timestamp);
+
+        // Verify message is no longer in-flight
+        const inflightCount = await manager.getTotalInflightCount();
+        expect(inflightCount).toBe(0);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should return empty array when no messages have timed out",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 60000, // Long timeout
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:no-timeout";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const _tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Add and claim a message with long timeout
+        const messageId = "long-timeout-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1 },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        await manager.claim(queueId, queueKey, queueItemsKey, "consumer-1");
+
+        // Reclaim should return empty array (message hasn't timed out)
+        const reclaimedMessages = await manager.reclaimTimedOut(0, (qId) => ({
+          queueKey: keys.queueKey(qId),
+          queueItemsKey: keys.queueItemsKey(qId),
+          tenantQueueIndexKey: keys.tenantQueueIndexKey(keys.extractTenantId(qId)),
+          dispatchKey,
+          tenantId: keys.extractTenantId(qId),
+        }));
+
+        expect(reclaimedMessages).toHaveLength(0);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should reclaim multiple timed-out messages and return all their info",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 100,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const _tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+
+        // Add and claim messages for two different tenants
+        for (const tenant of ["t1", "t2"]) {
+          const queueId = `tenant:${tenant}:queue:multi-reclaim`;
+          const queueKey = keys.queueKey(queueId);
+          const queueItemsKey = keys.queueItemsKey(queueId);
+
+          const messageId = `msg-${tenant}`;
+          const storedMessage = {
+            id: messageId,
+            queueId,
+            tenantId: tenant,
+            payload: { id: 1 },
+            timestamp: Date.now() - 1000,
+            attempt: 1,
+          };
+
+          await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+          await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+          await manager.claim(queueId, queueKey, queueItemsKey, "consumer-1", 100);
+        }
+
+        // Wait for timeout
+        await new Promise((resolve) => setTimeout(resolve, 150));
+
+        // Reclaim should return both messages
+        const reclaimedMessages = await manager.reclaimTimedOut(0, (qId) => ({
+          queueKey: keys.queueKey(qId),
+          queueItemsKey: keys.queueItemsKey(qId),
+          tenantQueueIndexKey: keys.tenantQueueIndexKey(keys.extractTenantId(qId)),
+          dispatchKey,
+          tenantId: keys.extractTenantId(qId),
+        }));
+
+        expect(reclaimedMessages).toHaveLength(2);
+
+        // Verify both tenants are represented
+        const tenantIds = reclaimedMessages.map((m: ReclaimedMessageInfo) => m.tenantId).sort();
+        expect(tenantIds).toEqual(["t1", "t2"]);
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+
+    redisTest(
+      "should use fallback tenantId extraction when message data is missing or corrupted",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new VisibilityManager({
+          redis: redisOptions,
+          keys,
+          shardCount: 1,
+          defaultTimeoutMs: 100,
+        });
+
+        const redis = createRedisClient(redisOptions);
+        const queueId = "tenant:t1:queue:fallback-test";
+        const queueKey = keys.queueKey(queueId);
+        const queueItemsKey = keys.queueItemsKey(queueId);
+        const _tenantQueueIndexKey = keys.tenantQueueIndexKey("t1");
+        const dispatchKey = keys.dispatchKey(0);
+        const inflightDataKey = keys.inflightDataKey(0);
+
+        // Add and claim a message
+        const messageId = "fallback-msg";
+        const storedMessage = {
+          id: messageId,
+          queueId,
+          tenantId: "t1",
+          payload: { id: 1 },
+          timestamp: Date.now() - 1000,
+          attempt: 1,
+          metadata: { orgId: "org-123" },
+        };
+
+        await redis.zadd(queueKey, storedMessage.timestamp, messageId);
+        await redis.hset(queueItemsKey, messageId, JSON.stringify(storedMessage));
+
+        // Claim the message
+        const claimResult = await manager.claim(
+          queueId,
+          queueKey,
+          queueItemsKey,
+          "consumer-1",
+          100
+        );
+        expect(claimResult.claimed).toBe(true);
+
+        // Corrupt the in-flight data by setting invalid JSON
+        await redis.hset(inflightDataKey, messageId, "not-valid-json{{{");
+
+        // Wait for timeout
+        await new Promise((resolve) => setTimeout(resolve, 150));
+
+        // Reclaim should still work using fallback extraction
+        const reclaimedMessages = await manager.reclaimTimedOut(0, (qId) => ({
+          queueKey: keys.queueKey(qId),
+          queueItemsKey: keys.queueItemsKey(qId),
+          tenantQueueIndexKey: keys.tenantQueueIndexKey(keys.extractTenantId(qId)),
+          dispatchKey,
+          tenantId: keys.extractTenantId(qId),
+        }));
+
+        expect(reclaimedMessages).toHaveLength(1);
+        expect(reclaimedMessages[0]).toEqual({
+          messageId,
+          queueId,
+          tenantId: "t1", // Extracted from queueId via fallback
+          metadata: {}, // Empty metadata since we couldn't parse the stored message
+        });
+
+        await manager.close();
+        await redis.quit();
+      }
+    );
+  });
+});

packages/redis-worker/src/fair-queue/tests/workerQueue.test.ts

@@ -0,0 +1,237 @@
+import { describe, expect } from "vitest";
+import { redisTest } from "@internal/testcontainers";
+import { WorkerQueueManager } from "../workerQueue.js";
+import { DefaultFairQueueKeyProducer } from "../keyProducer.js";
+import type { FairQueueKeyProducer } from "../types.js";
+
+describe("WorkerQueueManager", () => {
+  let keys: FairQueueKeyProducer;
+
+  describe("push and pop", () => {
+    redisTest(
+      "should push and pop a single message",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Push a message
+        await manager.push("worker-1", "msg-1:queue-1");
+
+        // Pop should return the message
+        const result = await manager.pop("worker-1");
+        expect(result).not.toBeNull();
+        expect(result!.messageKey).toBe("msg-1:queue-1");
+        expect(result!.queueLength).toBe(0);
+
+        await manager.close();
+      }
+    );
+
+    redisTest(
+      "should push and pop messages in FIFO order",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Push messages
+        await manager.push("worker-1", "msg-1:queue-1");
+        await manager.push("worker-1", "msg-2:queue-1");
+        await manager.push("worker-1", "msg-3:queue-1");
+
+        // Pop should return in FIFO order
+        let result = await manager.pop("worker-1");
+        expect(result!.messageKey).toBe("msg-1:queue-1");
+        expect(result!.queueLength).toBe(2);
+
+        result = await manager.pop("worker-1");
+        expect(result!.messageKey).toBe("msg-2:queue-1");
+        expect(result!.queueLength).toBe(1);
+
+        result = await manager.pop("worker-1");
+        expect(result!.messageKey).toBe("msg-3:queue-1");
+        expect(result!.queueLength).toBe(0);
+
+        // Queue should be empty
+        result = await manager.pop("worker-1");
+        expect(result).toBeNull();
+
+        await manager.close();
+      }
+    );
+
+    redisTest("should push batch of messages", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+      const manager = new WorkerQueueManager({
+        redis: redisOptions,
+        keys,
+      });
+
+      // Push batch
+      await manager.pushBatch("worker-1", ["msg-1:queue-1", "msg-2:queue-1", "msg-3:queue-1"]);
+
+      // Check length
+      const length = await manager.getLength("worker-1");
+      expect(length).toBe(3);
+
+      await manager.close();
+    });
+  });
+
+  describe("getLength", () => {
+    redisTest(
+      "should return correct queue length",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Initially empty
+        let length = await manager.getLength("worker-1");
+        expect(length).toBe(0);
+
+        // Push messages
+        await manager.push("worker-1", "msg-1:queue-1");
+        await manager.push("worker-1", "msg-2:queue-1");
+
+        length = await manager.getLength("worker-1");
+        expect(length).toBe(2);
+
+        // Pop one
+        await manager.pop("worker-1");
+
+        length = await manager.getLength("worker-1");
+        expect(length).toBe(1);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("peek", () => {
+    redisTest(
+      "should peek at messages without removing them",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Push messages
+        await manager.push("worker-1", "msg-1:queue-1");
+        await manager.push("worker-1", "msg-2:queue-1");
+
+        // Peek
+        const messages = await manager.peek("worker-1");
+        expect(messages).toEqual(["msg-1:queue-1", "msg-2:queue-1"]);
+
+        // Messages should still be there
+        const length = await manager.getLength("worker-1");
+        expect(length).toBe(2);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("remove", () => {
+    redisTest("should remove a specific message", { timeout: 10000 }, async ({ redisOptions }) => {
+      keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+      const manager = new WorkerQueueManager({
+        redis: redisOptions,
+        keys,
+      });
+
+      // Push messages
+      await manager.push("worker-1", "msg-1:queue-1");
+      await manager.push("worker-1", "msg-2:queue-1");
+      await manager.push("worker-1", "msg-3:queue-1");
+
+      // Remove the middle one
+      const removed = await manager.remove("worker-1", "msg-2:queue-1");
+      expect(removed).toBe(1);
+
+      // Check remaining
+      const messages = await manager.peek("worker-1");
+      expect(messages).toEqual(["msg-1:queue-1", "msg-3:queue-1"]);
+
+      await manager.close();
+    });
+  });
+
+  describe("clear", () => {
+    redisTest(
+      "should clear all messages from queue",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Push messages
+        await manager.push("worker-1", "msg-1:queue-1");
+        await manager.push("worker-1", "msg-2:queue-1");
+
+        // Clear
+        await manager.clear("worker-1");
+
+        // Should be empty
+        const length = await manager.getLength("worker-1");
+        expect(length).toBe(0);
+
+        await manager.close();
+      }
+    );
+  });
+
+  describe("separate worker queues", () => {
+    redisTest(
+      "should maintain separate queues for different workers",
+      { timeout: 10000 },
+      async ({ redisOptions }) => {
+        keys = new DefaultFairQueueKeyProducer({ prefix: "test" });
+
+        const manager = new WorkerQueueManager({
+          redis: redisOptions,
+          keys,
+        });
+
+        // Push to different worker queues
+        await manager.push("worker-1", "msg-1-1:queue-1");
+        await manager.push("worker-2", "msg-2-1:queue-1");
+        await manager.push("worker-1", "msg-1-2:queue-1");
+        await manager.push("worker-2", "msg-2-2:queue-1");
+
+        // Each worker should have its own messages
+        const worker1Messages = await manager.peek("worker-1");
+        expect(worker1Messages).toEqual(["msg-1-1:queue-1", "msg-1-2:queue-1"]);
+
+        const worker2Messages = await manager.peek("worker-2");
+        expect(worker2Messages).toEqual(["msg-2-1:queue-1", "msg-2-2:queue-1"]);
+
+        await manager.close();
+      }
+    );
+  });
+});

packages/redis-worker/src/fair-queue/types.ts

@@ -0,0 +1,652 @@
+import type { RedisOptions } from "@internal/redis";
+import type { Logger } from "@trigger.dev/core/logger";
+import type { Tracer, Meter } from "@internal/tracing";
+import type { z } from "zod";
+import type { RetryStrategy } from "./retry.js";
+
+// ============================================================================
+// Global Rate Limiter
+// ============================================================================
+
+/**
+ * Interface for a global rate limiter that limits processing across all consumers.
+ * When configured, consumers will check this before processing each message.
+ */
+export interface GlobalRateLimiter {
+  /**
+   * Check if processing is allowed under the rate limit.
+   * @returns Object with allowed flag and optional resetAt timestamp (ms since epoch)
+   */
+  limit(): Promise<{ allowed: boolean; resetAt?: number }>;
+}
+
+// ============================================================================
+// Core Queue Types
+// ============================================================================
+
+/**
+ * Descriptor for a queue in the fair queue system.
+ * Contains all the metadata needed to identify and route a queue.
+ */
+export interface QueueDescriptor {
+  /** Unique queue identifier */
+  id: string;
+  /** Tenant this queue belongs to */
+  tenantId: string;
+  /** Additional metadata for concurrency group extraction */
+  metadata: Record<string, unknown>;
+}
+
+/**
+ * A message in the queue with its metadata.
+ */
+export interface QueueMessage<TPayload = unknown> {
+  /** Unique message identifier */
+  id: string;
+  /** The queue this message belongs to */
+  queueId: string;
+  /** Message payload */
+  payload: TPayload;
+  /** Timestamp when message was enqueued */
+  timestamp: number;
+  /** Current attempt number (1-indexed, for retries) */
+  attempt: number;
+  /** Optional metadata */
+  metadata?: Record<string, unknown>;
+}
+
+/**
+ * Internal message format stored in Redis.
+ * Includes additional fields for tracking and routing.
+ */
+export interface StoredMessage<TPayload = unknown> {
+  /** Message ID */
+  id: string;
+  /** Queue ID */
+  queueId: string;
+  /** Tenant ID */
+  tenantId: string;
+  /** Message payload */
+  payload: TPayload;
+  /** Timestamp when enqueued */
+  timestamp: number;
+  /** Current attempt number */
+  attempt: number;
+  /** Worker queue to route to */
+  workerQueue?: string;
+  /** Additional metadata */
+  metadata?: Record<string, unknown>;
+}
+
+/**
+ * Queue with its score (oldest message timestamp) from the master queue.
+ */
+export interface QueueWithScore {
+  /** Queue identifier */
+  queueId: string;
+  /** Score (typically oldest message timestamp) */
+  score: number;
+  /** Tenant ID extracted from queue */
+  tenantId: string;
+}
+
+// ============================================================================
+// Concurrency Types
+// ============================================================================
+
+/**
+ * Configuration for a concurrency group.
+ * Allows defining arbitrary levels of concurrency (tenant, org, project, etc.)
+ */
+export interface ConcurrencyGroupConfig {
+  /** Group name (e.g., "tenant", "organization", "project") */
+  name: string;
+  /** Extract the group ID from a queue descriptor */
+  extractGroupId: (queue: QueueDescriptor) => string;
+  /** Get the concurrency limit for a specific group ID */
+  getLimit: (groupId: string) => Promise<number>;
+  /** Default limit if not specified */
+  defaultLimit: number;
+}
+
+/**
+ * Current concurrency state for a group.
+ */
+export interface ConcurrencyState {
+  /** Group name */
+  groupName: string;
+  /** Group ID */
+  groupId: string;
+  /** Current active count */
+  current: number;
+  /** Configured limit */
+  limit: number;
+}
+
+/**
+ * Result of a concurrency check.
+ */
+export interface ConcurrencyCheckResult {
+  /** Whether processing is allowed */
+  allowed: boolean;
+  /** If not allowed, which group is blocking */
+  blockedBy?: ConcurrencyState;
+}
+
+// ============================================================================
+// Visibility Types
+// ============================================================================
+
+/**
+ * Information about a reclaimed message from visibility timeout.
+ * Used to release concurrency after a message is returned to the queue.
+ */
+export interface ReclaimedMessageInfo {
+  /** Message ID */
+  messageId: string;
+  /** Queue ID */
+  queueId: string;
+  /** Tenant ID for concurrency release */
+  tenantId: string;
+  /** Additional metadata for concurrency group extraction */
+  metadata?: Record<string, unknown>;
+}
+
+// ============================================================================
+// Scheduler Types
+// ============================================================================
+
+/**
+ * Queues grouped by tenant for the scheduler.
+ */
+export interface TenantQueues {
+  /** Tenant identifier */
+  tenantId: string;
+  /** Queue IDs belonging to this tenant, in priority order */
+  queues: string[];
+}
+
+/**
+ * Context provided to the scheduler for making decisions.
+ */
+export interface SchedulerContext {
+  /** Get current concurrency for a group */
+  getCurrentConcurrency(groupName: string, groupId: string): Promise<number>;
+  /** Get concurrency limit for a group */
+  getConcurrencyLimit(groupName: string, groupId: string): Promise<number>;
+  /** Check if a group is at capacity */
+  isAtCapacity(groupName: string, groupId: string): Promise<boolean>;
+  /** Get queue descriptor by ID */
+  getQueueDescriptor(queueId: string): QueueDescriptor;
+}
+
+/**
+ * Extended context for two-level dispatch scheduling.
+ */
+export interface DispatchSchedulerContext extends SchedulerContext {
+  /** Get queues for a specific tenant from the per-tenant queue index (Level 2) */
+  getQueuesForTenant(tenantId: string, limit?: number): Promise<QueueWithScore[]>;
+}
+
+/**
+ * Pluggable scheduler interface for fair queue selection.
+ */
+export interface FairScheduler {
+  /**
+   * Select queues for processing from a master queue shard.
+   * Returns queues grouped by tenant, ordered by the fairness algorithm.
+   *
+   * @param masterQueueShard - The master queue shard key
+   * @param consumerId - The consumer making the request
+   * @param context - Context for concurrency checks
+   * @returns Queues grouped by tenant in priority order
+   */
+  selectQueues(
+    masterQueueShard: string,
+    consumerId: string,
+    context: SchedulerContext
+  ): Promise<TenantQueues[]>;
+
+  /**
+   * Select queues using the two-level tenant dispatch index.
+   * Level 1: reads tenantIds from dispatch shard.
+   * Level 2: reads queueIds from per-tenant index.
+   * Optional - falls back to selectQueues with flat queue list if not implemented.
+   */
+  selectQueuesFromDispatch?(
+    dispatchShardKey: string,
+    consumerId: string,
+    context: DispatchSchedulerContext
+  ): Promise<TenantQueues[]>;
+
+  /**
+   * Called after processing a message to update scheduler state.
+   * Optional - not all schedulers need to track state.
+   */
+  recordProcessed?(tenantId: string, queueId: string): Promise<void>;
+
+  /**
+   * Called after processing multiple messages to update scheduler state.
+   * Batch variant for efficiency - reduces Redis calls when processing multiple messages.
+   * Optional - falls back to calling recordProcessed multiple times if not implemented.
+   */
+  recordProcessedBatch?(tenantId: string, queueId: string, count: number): Promise<void>;
+
+  /**
+   * Initialize the scheduler (called once on startup).
+   */
+  initialize?(): Promise<void>;
+
+  /**
+   * Cleanup scheduler resources.
+   */
+  close?(): Promise<void>;
+}
+
+// ============================================================================
+// Visibility Timeout Types
+// ============================================================================
+
+/**
+ * An in-flight message being processed.
+ */
+export interface InFlightMessage<TPayload = unknown> {
+  /** Message ID */
+  messageId: string;
+  /** Queue ID */
+  queueId: string;
+  /** Message payload */
+  payload: TPayload;
+  /** When visibility timeout expires */
+  deadline: number;
+  /** Consumer that claimed this message */
+  consumerId: string;
+}
+
+/**
+ * Result of claiming a message.
+ */
+export interface ClaimResult<TPayload = unknown> {
+  /** Whether the claim was successful */
+  claimed: boolean;
+  /** The claimed message if successful */
+  message?: InFlightMessage<TPayload>;
+}
+
+// ============================================================================
+// Key Producer Interface
+// ============================================================================
+
+/**
+ * Interface for generating Redis keys for the fair queue system.
+ * Implementations can customize key prefixes and structures.
+ */
+export interface FairQueueKeyProducer {
+  // Master queue keys
+  /** Get the master queue key for a shard */
+  masterQueueKey(shardId: number): string;
+
+  // Individual queue keys
+  /** Get the queue key for storing messages */
+  queueKey(queueId: string): string;
+  /** Get the queue items hash key */
+  queueItemsKey(queueId: string): string;
+
+  // Concurrency tracking keys
+  /** Get the concurrency set key for a group */
+  concurrencyKey(groupName: string, groupId: string): string;
+
+  // In-flight tracking keys
+  /** Get the in-flight sorted set key for a shard */
+  inflightKey(shardId: number): string;
+  /** Get the in-flight message data hash key */
+  inflightDataKey(shardId: number): string;
+
+  // Worker queue keys
+  /** Get the worker queue key for a consumer */
+  workerQueueKey(consumerId: string): string;
+
+  // Dead letter queue keys
+  /** Get the dead letter queue key for a tenant */
+  deadLetterQueueKey(tenantId: string): string;
+  /** Get the dead letter queue data hash key for a tenant */
+  deadLetterQueueDataKey(tenantId: string): string;
+
+  // Tenant dispatch keys (two-level index)
+  /** Get the dispatch index key for a shard (Level 1: tenantIds with capacity) */
+  dispatchKey(shardId: number): string;
+  /** Get the per-tenant queue index key (Level 2: queueIds for a tenant) */
+  tenantQueueIndexKey(tenantId: string): string;
+
+  // Extraction methods
+  /** Extract tenant ID from a queue ID */
+  extractTenantId(queueId: string): string;
+  /** Extract a specific group ID from a queue ID */
+  extractGroupId(groupName: string, queueId: string): string;
+}
+
+// ============================================================================
+// FairQueue Options
+// ============================================================================
+
+/**
+ * Worker queue configuration options.
+ * Worker queues are always enabled - FairQueue routes messages to worker queues,
+ * and external consumers are responsible for consuming from those queues.
+ */
+export interface WorkerQueueOptions<TPayload = unknown> {
+  /**
+   * Function to resolve which worker queue a message should go to.
+   * This is called during the claim-and-push phase to determine the target queue.
+   */
+  resolveWorkerQueue: (message: StoredMessage<TPayload>) => string;
+}
+
+/**
+ * Retry and dead letter queue configuration.
+ */
+export interface RetryOptions {
+  /** Retry strategy for failed messages */
+  strategy: RetryStrategy;
+  /** Whether to enable dead letter queue (default: true) */
+  deadLetterQueue?: boolean;
+}
+
+/**
+ * Queue cooloff configuration to avoid repeatedly polling concurrency-limited queues.
+ */
+export interface CooloffOptions {
+  /** Whether cooloff is enabled (default: true) */
+  enabled?: boolean;
+  /** Number of consecutive empty dequeues before entering cooloff (default: 10) */
+  threshold?: number;
+  /** Duration of cooloff period in milliseconds (default: 10000) */
+  periodMs?: number;
+  /** Maximum number of cooloff state entries before triggering cleanup (default: 1000) */
+  maxStatesSize?: number;
+}
+
+/**
+ * Options for creating a FairQueue instance.
+ *
+ * @typeParam TPayloadSchema - Zod schema for message payload validation
+ */
+export interface FairQueueOptions<TPayloadSchema extends z.ZodTypeAny = z.ZodUnknown> {
+  /** Redis connection options */
+  redis: RedisOptions;
+
+  /** Key producer for Redis keys */
+  keys: FairQueueKeyProducer;
+
+  /** Scheduler for fair queue selection */
+  scheduler: FairScheduler;
+
+  // Payload validation
+  /** Zod schema for message payload validation */
+  payloadSchema?: TPayloadSchema;
+  /** Whether to validate payloads on enqueue (default: false) */
+  validateOnEnqueue?: boolean;
+
+  // Sharding
+  /** Number of master queue shards (default: 1) */
+  shardCount?: number;
+
+  // Concurrency
+  /** Concurrency group configurations */
+  concurrencyGroups?: ConcurrencyGroupConfig[];
+
+  // Worker queue (required)
+  /**
+   * Worker queue configuration.
+   * FairQueue routes messages to worker queues; external consumers handle consumption.
+   */
+  workerQueue: WorkerQueueOptions<z.infer<TPayloadSchema>>;
+
+  // Retry and DLQ
+  /** Retry and dead letter queue configuration */
+  retry?: RetryOptions;
+
+  // Visibility timeout
+  /** Visibility timeout in milliseconds (default: 30000) */
+  visibilityTimeoutMs?: number;
+  /** Heartbeat interval in milliseconds (default: visibilityTimeoutMs / 3) */
+  heartbeatIntervalMs?: number;
+  /** Interval for reclaiming timed-out messages (default: 5000) */
+  reclaimIntervalMs?: number;
+
+  // Consumers
+  /** Number of consumer loops to run (default: 1) */
+  consumerCount?: number;
+  /** Interval between consumer iterations in milliseconds (default: 100) */
+  consumerIntervalMs?: number;
+  /** Whether to start consumers on initialization (default: true) */
+  startConsumers?: boolean;
+
+  // Batch claiming
+  /** Maximum number of messages to claim in a single batch operation (default: 10) */
+  batchClaimSize?: number;
+
+  // Consumer tracing
+  /** Maximum iterations before starting a new trace span (default: 500) */
+  consumerTraceMaxIterations?: number;
+  /** Maximum seconds before starting a new trace span (default: 60) */
+  consumerTraceTimeoutSeconds?: number;
+
+  // Cooloff
+  /** Queue cooloff configuration */
+  cooloff?: CooloffOptions;
+
+  // Observability
+  /** Logger instance */
+  logger?: Logger;
+  /** OpenTelemetry tracer */
+  tracer?: Tracer;
+  /** OpenTelemetry meter */
+  meter?: Meter;
+  /** Name for metrics/tracing (default: "fairqueue") */
+  name?: string;
+
+  // Worker queue backpressure
+  /**
+   * Maximum number of items allowed in a worker queue before claiming pauses.
+   * When set, the claim phase checks worker queue depth and skips claiming if
+   * the queue is at or above this limit. This prevents unbounded worker queue
+   * growth which could cause visibility timeouts (claimed messages have a
+   * visibility timeout that ticks while they sit in the worker queue).
+   * Requires `workerQueueDepthCheckId` to know which queue to check.
+   * Disabled by default (0 = no limit).
+   */
+  workerQueueMaxDepth?: number;
+  /**
+   * The worker queue ID to check depth against when workerQueueMaxDepth is set.
+   * Required when workerQueueMaxDepth > 0 and the system uses a single shared worker queue.
+   * If not set, depth checking is disabled even if workerQueueMaxDepth is set.
+   */
+  workerQueueDepthCheckId?: string;
+}
+
+// ============================================================================
+// Message Handler Types
+// ============================================================================
+
+/**
+ * Context passed to the message handler.
+ */
+export interface MessageHandlerContext<TPayload = unknown> {
+  /** The message being processed */
+  message: QueueMessage<TPayload>;
+  /** Queue descriptor */
+  queue: QueueDescriptor;
+  /** Consumer ID processing this message */
+  consumerId: string;
+  /** Extend the visibility timeout */
+  heartbeat(): Promise<boolean>;
+  /** Mark message as successfully processed */
+  complete(): Promise<void>;
+  /** Release message back to the queue for retry */
+  release(): Promise<void>;
+  /** Mark message as failed (triggers retry or DLQ) */
+  fail(error?: Error): Promise<void>;
+}
+
+/**
+ * Handler function for processing messages.
+ */
+export type MessageHandler<TPayload = unknown> = (
+  context: MessageHandlerContext<TPayload>
+) => Promise<void>;
+
+// ============================================================================
+// Dead Letter Queue Types
+// ============================================================================
+
+/**
+ * A message in the dead letter queue.
+ */
+export interface DeadLetterMessage<TPayload = unknown> {
+  /** Message ID */
+  id: string;
+  /** Original queue ID */
+  queueId: string;
+  /** Tenant ID */
+  tenantId: string;
+  /** Message payload */
+  payload: TPayload;
+  /** Timestamp when moved to DLQ */
+  deadLetteredAt: number;
+  /** Number of attempts before DLQ */
+  attempts: number;
+  /** Last error message if available */
+  lastError?: string;
+  /** Original message timestamp */
+  originalTimestamp: number;
+}
+
+// ============================================================================
+// Cooloff State Types
+// ============================================================================
+
+/**
+ * Cooloff state for a queue.
+ */
+export type QueueCooloffState =
+  | { tag: "normal"; consecutiveFailures: number }
+  | { tag: "cooloff"; expiresAt: number };
+
+// ============================================================================
+// Enqueue Options
+// ============================================================================
+
+/**
+ * Options for enqueueing a message.
+ */
+export interface EnqueueOptions<TPayload = unknown> {
+  /** Queue to add the message to */
+  queueId: string;
+  /** Tenant ID for the queue */
+  tenantId: string;
+  /** Message payload */
+  payload: TPayload;
+  /** Optional message ID (auto-generated if not provided) */
+  messageId?: string;
+  /** Optional timestamp (defaults to now) */
+  timestamp?: number;
+  /** Optional metadata for concurrency group extraction */
+  metadata?: Record<string, string>;
+}
+
+/**
+ * Options for enqueueing multiple messages.
+ */
+export interface EnqueueBatchOptions<TPayload = unknown> {
+  /** Queue to add messages to */
+  queueId: string;
+  /** Tenant ID for the queue */
+  tenantId: string;
+  /** Messages to enqueue */
+  messages: Array<{
+    payload: TPayload;
+    messageId?: string;
+    timestamp?: number;
+  }>;
+  /** Optional metadata for concurrency group extraction */
+  metadata?: Record<string, string>;
+}
+
+// ============================================================================
+// DRR Scheduler Types
+// ============================================================================
+
+/**
+ * Configuration for the Deficit Round Robin scheduler.
+ */
+export interface DRRSchedulerConfig {
+  /** Credits allocated per tenant per round */
+  quantum: number;
+  /** Maximum accumulated deficit (prevents starvation) */
+  maxDeficit: number;
+  /** Maximum queues to fetch from master queue (default: 1000) */
+  masterQueueLimit?: number;
+  /** Redis options for state storage */
+  redis: RedisOptions;
+  /** Key producer */
+  keys: FairQueueKeyProducer;
+  /** Optional logger */
+  logger?: {
+    debug: (message: string, context?: Record<string, unknown>) => void;
+    error: (message: string, context?: Record<string, unknown>) => void;
+  };
+}
+
+// ============================================================================
+// Weighted Scheduler Types
+// ============================================================================
+
+/**
+ * Bias configuration for weighted shuffle scheduler.
+ */
+export interface WeightedSchedulerBiases {
+  /**
+   * How much to bias towards tenants with higher concurrency limits.
+   * 0 = no bias, 1 = full bias based on limit differences
+   */
+  concurrencyLimitBias: number;
+
+  /**
+   * How much to bias towards tenants with more available capacity.
+   * 0 = no bias, 1 = full bias based on available capacity
+   */
+  availableCapacityBias: number;
+
+  /**
+   * Controls randomization of queue ordering within tenants.
+   * 0 = strict age-based ordering (oldest first)
+   * 1 = completely random ordering
+   * Values between 0-1 blend between age-based and random ordering
+   */
+  queueAgeRandomization: number;
+}
+
+/**
+ * Configuration for the weighted shuffle scheduler.
+ */
+export interface WeightedSchedulerConfig {
+  /** Redis options */
+  redis: RedisOptions;
+  /** Key producer */
+  keys: FairQueueKeyProducer;
+  /** Default tenant concurrency limit */
+  defaultTenantConcurrencyLimit?: number;
+  /** Maximum queues to consider from master queue */
+  masterQueueLimit?: number;
+  /** Bias configuration */
+  biases?: WeightedSchedulerBiases;
+  /** Number of iterations to reuse a snapshot */
+  reuseSnapshotCount?: number;
+  /** Maximum number of tenants to consider */
+  maximumTenantCount?: number;
+  /** Random seed for reproducibility */
+  seed?: string;
+  /** Optional tracer */
+  tracer?: Tracer;
+}

packages/redis-worker/src/fair-queue/visibility.ts

@@ -0,0 +1,881 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import { jumpHash } from "@trigger.dev/core/v3/serverOnly";
+import type {
+  ClaimResult,
+  FairQueueKeyProducer,
+  InFlightMessage,
+  ReclaimedMessageInfo,
+  StoredMessage,
+} from "./types.js";
+
+export interface VisibilityManagerOptions {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  shardCount: number;
+  defaultTimeoutMs: number;
+  logger?: {
+    debug: (message: string, context?: Record<string, unknown>) => void;
+    error: (message: string, context?: Record<string, unknown>) => void;
+  };
+}
+
+/**
+ * VisibilityManager handles message visibility timeouts for safe message processing.
+ *
+ * Features:
+ * - Claim messages with visibility timeout
+ * - Heartbeat to extend timeout
+ * - Automatic reclaim of timed-out messages
+ * - Per-shard in-flight tracking
+ *
+ * Data structures:
+ * - In-flight sorted set: score = deadline timestamp, member = "{messageId}:{queueId}"
+ * - In-flight data hash: field = messageId, value = JSON message data
+ */
+export class VisibilityManager {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private shardCount: number;
+  private defaultTimeoutMs: number;
+  private logger: NonNullable<VisibilityManagerOptions["logger"]>;
+
+  constructor(private options: VisibilityManagerOptions) {
+    this.redis = createRedisClient(options.redis);
+    this.keys = options.keys;
+    this.shardCount = options.shardCount;
+    this.defaultTimeoutMs = options.defaultTimeoutMs;
+    this.logger = options.logger ?? {
+      debug: () => {},
+      error: () => {},
+    };
+
+    this.#registerCommands();
+  }
+
+  // ============================================================================
+  // Public Methods
+  // ============================================================================
+
+  /**
+   * Claim a message for processing.
+   * Moves the message from its queue to the in-flight set with a visibility timeout.
+   *
+   * @param queueId - The queue to claim from
+   * @param queueKey - The Redis key for the queue sorted set
+   * @param queueItemsKey - The Redis key for the queue items hash
+   * @param consumerId - ID of the consumer claiming the message
+   * @param timeoutMs - Visibility timeout in milliseconds
+   * @returns Claim result with the message if successful
+   */
+  async claim<TPayload = unknown>(
+    queueId: string,
+    queueKey: string,
+    queueItemsKey: string,
+    consumerId: string,
+    timeoutMs?: number
+  ): Promise<ClaimResult<TPayload>> {
+    const timeout = timeoutMs ?? this.defaultTimeoutMs;
+    const deadline = Date.now() + timeout;
+    const shardId = this.#getShardForQueue(queueId);
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+
+    // Use Lua script to atomically:
+    // 1. Pop oldest message from queue
+    // 2. Add to in-flight set with deadline
+    // 3. Store message data
+    const result = await this.redis.claimMessage(
+      queueKey,
+      queueItemsKey,
+      inflightKey,
+      inflightDataKey,
+      queueId,
+      consumerId,
+      deadline.toString()
+    );
+
+    if (!result) {
+      return { claimed: false };
+    }
+
+    const [messageId, payloadJson] = result;
+
+    try {
+      const payload = JSON.parse(payloadJson) as TPayload;
+      const message: InFlightMessage<TPayload> = {
+        messageId,
+        queueId,
+        payload,
+        deadline,
+        consumerId,
+      };
+
+      this.logger.debug("Message claimed", {
+        messageId,
+        queueId,
+        consumerId,
+        deadline,
+      });
+
+      return { claimed: true, message };
+    } catch (error) {
+      // JSON parse error - message data is corrupted
+      this.logger.error("Failed to parse claimed message", {
+        messageId,
+        queueId,
+        error: error instanceof Error ? error.message : String(error),
+      });
+
+      // Remove the corrupted message from in-flight
+      await this.#removeFromInflight(shardId, messageId, queueId);
+
+      return { claimed: false };
+    }
+  }
+
+  /**
+   * Claim multiple messages for processing (batch claim).
+   * Moves up to maxCount messages from the queue to the in-flight set.
+   *
+   * @param queueId - The queue to claim from
+   * @param queueKey - The Redis key for the queue sorted set
+   * @param queueItemsKey - The Redis key for the queue items hash
+   * @param consumerId - ID of the consumer claiming the messages
+   * @param maxCount - Maximum number of messages to claim
+   * @param timeoutMs - Visibility timeout in milliseconds
+   * @returns Array of claimed messages
+   */
+  async claimBatch<TPayload = unknown>(
+    queueId: string,
+    queueKey: string,
+    queueItemsKey: string,
+    consumerId: string,
+    maxCount: number,
+    timeoutMs?: number
+  ): Promise<Array<InFlightMessage<TPayload>>> {
+    const timeout = timeoutMs ?? this.defaultTimeoutMs;
+    const deadline = Date.now() + timeout;
+    const shardId = this.#getShardForQueue(queueId);
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+
+    // Use Lua script to atomically claim up to maxCount messages
+    const result = await this.redis.claimMessageBatch(
+      queueKey,
+      queueItemsKey,
+      inflightKey,
+      inflightDataKey,
+      queueId,
+      deadline.toString(),
+      maxCount.toString()
+    );
+
+    if (!result || result.length === 0) {
+      return [];
+    }
+
+    const messages: Array<InFlightMessage<TPayload>> = [];
+
+    // Results come in pairs: [messageId1, payload1, messageId2, payload2, ...]
+    for (let i = 0; i < result.length; i += 2) {
+      const messageId = result[i];
+      const payloadJson = result[i + 1];
+
+      // Skip if either value is missing
+      if (!messageId || !payloadJson) {
+        continue;
+      }
+
+      try {
+        const payload = JSON.parse(payloadJson) as TPayload;
+        messages.push({
+          messageId,
+          queueId,
+          payload,
+          deadline,
+          consumerId,
+        });
+      } catch (error) {
+        // JSON parse error - skip this message
+        this.logger.error("Failed to parse claimed message in batch", {
+          messageId,
+          queueId,
+          error: error instanceof Error ? error.message : String(error),
+        });
+        // Remove the corrupted message from in-flight
+        await this.#removeFromInflight(shardId, messageId, queueId);
+      }
+    }
+
+    if (messages.length > 0) {
+      this.logger.debug("Batch claimed messages", {
+        queueId,
+        consumerId,
+        count: messages.length,
+        deadline,
+      });
+    }
+
+    return messages;
+  }
+
+  /**
+   * Extend the visibility timeout for a message (heartbeat).
+   *
+   * @param messageId - The message ID
+   * @param queueId - The queue ID
+   * @param extendMs - Additional milliseconds to add to the deadline
+   * @returns true if the heartbeat was successful
+   */
+  async heartbeat(messageId: string, queueId: string, extendMs: number): Promise<boolean> {
+    const shardId = this.#getShardForQueue(queueId);
+    const inflightKey = this.keys.inflightKey(shardId);
+    const member = this.#makeMember(messageId, queueId);
+    const newDeadline = Date.now() + extendMs;
+
+    // Use Lua script to atomically check existence and update score
+    // ZADD XX returns 0 even on successful updates, so we use a custom command
+    const result = await this.redis.heartbeatMessage(inflightKey, member, newDeadline.toString());
+
+    const success = result === 1;
+
+    if (success) {
+      this.logger.debug("Heartbeat successful", {
+        messageId,
+        queueId,
+        newDeadline,
+      });
+    }
+
+    return success;
+  }
+
+  /**
+   * Mark a message as successfully processed.
+   * Removes the message from in-flight tracking.
+   *
+   * @param messageId - The message ID
+   * @param queueId - The queue ID
+   */
+  async complete(messageId: string, queueId: string): Promise<void> {
+    const shardId = this.#getShardForQueue(queueId);
+    await this.#removeFromInflight(shardId, messageId, queueId);
+
+    this.logger.debug("Message completed", {
+      messageId,
+      queueId,
+    });
+  }
+
+  /**
+   * Release a message back to its queue.
+   * Used when processing fails or consumer wants to retry later.
+   *
+   * @param messageId - The message ID
+   * @param queueId - The queue ID
+   * @param queueKey - The Redis key for the queue
+   * @param queueItemsKey - The Redis key for the queue items hash
+   * @param tenantQueueIndexKey - The Redis key for the tenant queue index (Level 2)
+   * @param dispatchKey - The Redis key for the dispatch index (Level 1)
+   * @param tenantId - The tenant ID
+   * @param score - Optional score for the message (defaults to now)
+   */
+  async release<_TPayload = unknown>(
+    messageId: string,
+    queueId: string,
+    queueKey: string,
+    queueItemsKey: string,
+    tenantQueueIndexKey: string,
+    dispatchKey: string,
+    tenantId: string,
+    score?: number,
+    updatedData?: string
+  ): Promise<void> {
+    const shardId = this.#getShardForQueue(queueId);
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+    const member = this.#makeMember(messageId, queueId);
+    const messageScore = score ?? Date.now();
+
+    // Use Lua script to atomically:
+    // 1. Get message data from in-flight (or use updatedData if provided)
+    // 2. Remove from in-flight
+    // 3. Add back to queue
+    // 4. Update dispatch indexes to ensure queue is picked up
+    await this.redis.releaseMessage(
+      inflightKey,
+      inflightDataKey,
+      queueKey,
+      queueItemsKey,
+      tenantQueueIndexKey,
+      dispatchKey,
+      member,
+      messageId,
+      messageScore.toString(),
+      queueId,
+      updatedData ?? "",
+      tenantId
+    );
+
+    this.logger.debug("Message released", {
+      messageId,
+      queueId,
+      score: messageScore,
+    });
+  }
+
+  /**
+   * Release multiple messages back to their queue in a single operation.
+   * Used when processing fails or consumer wants to retry later.
+   * All messages must belong to the same queue.
+   *
+   * @param messages - Array of messages to release (must all have same queueId)
+   * @param queueId - The queue ID
+   * @param queueKey - The Redis key for the queue
+   * @param queueItemsKey - The Redis key for the queue items hash
+   * @param tenantQueueIndexKey - The Redis key for the tenant queue index (Level 2)
+   * @param dispatchKey - The Redis key for the dispatch index (Level 1)
+   * @param tenantId - The tenant ID
+   * @param score - Optional score for the messages (defaults to now)
+   */
+  async releaseBatch(
+    messages: Array<{ messageId: string }>,
+    queueId: string,
+    queueKey: string,
+    queueItemsKey: string,
+    tenantQueueIndexKey: string,
+    dispatchKey: string,
+    tenantId: string,
+    score?: number
+  ): Promise<void> {
+    if (messages.length === 0) {
+      return;
+    }
+
+    const shardId = this.#getShardForQueue(queueId);
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+    const messageScore = score ?? Date.now();
+
+    // Build arrays of members and messageIds for the Lua script
+    const messageIds = messages.map((m) => m.messageId);
+    const members = messages.map((m) => this.#makeMember(m.messageId, queueId));
+
+    await this.redis.releaseMessageBatch(
+      inflightKey,
+      inflightDataKey,
+      queueKey,
+      queueItemsKey,
+      tenantQueueIndexKey,
+      dispatchKey,
+      messageScore.toString(),
+      queueId,
+      tenantId,
+      ...members,
+      ...messageIds
+    );
+
+    this.logger.debug("Batch messages released", {
+      queueId,
+      count: messages.length,
+      score: messageScore,
+    });
+  }
+
+  /**
+   * Reclaim timed-out messages from a shard.
+   * Returns messages to their original queues.
+   *
+   * @param shardId - The shard to check
+   * @param getQueueKeys - Function to get queue keys for a queue ID
+   * @returns Array of reclaimed message info for concurrency release
+   */
+  async reclaimTimedOut(
+    shardId: number,
+    getQueueKeys: (queueId: string) => {
+      queueKey: string;
+      queueItemsKey: string;
+      tenantQueueIndexKey: string;
+      dispatchKey: string;
+      tenantId: string;
+    }
+  ): Promise<ReclaimedMessageInfo[]> {
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+    const now = Date.now();
+
+    // Get all messages past their deadline
+    const timedOut = await this.redis.zrangebyscore(
+      inflightKey,
+      "-inf",
+      now,
+      "WITHSCORES",
+      "LIMIT",
+      0,
+      100 // Process in batches
+    );
+
+    const reclaimedMessages: ReclaimedMessageInfo[] = [];
+
+    for (let i = 0; i < timedOut.length; i += 2) {
+      const member = timedOut[i];
+      const _deadlineScore = timedOut[i + 1]; // This is the visibility deadline, not the original timestamp
+      if (!member || !_deadlineScore) {
+        continue;
+      }
+      const { messageId, queueId } = this.#parseMember(member);
+      const { queueKey, queueItemsKey, tenantQueueIndexKey, dispatchKey, tenantId } =
+        getQueueKeys(queueId);
+
+      try {
+        // Get message data BEFORE releasing so we can extract tenantId for concurrency release
+        const dataJson = await this.redis.hget(inflightDataKey, messageId);
+        let storedMessage: StoredMessage | null = null;
+        if (dataJson) {
+          try {
+            storedMessage = JSON.parse(dataJson);
+          } catch {
+            // Ignore parse error, proceed with reclaim
+          }
+        }
+
+        // Re-add to queue with original timestamp to preserve priority
+        // Fall back to now if we can't get the original timestamp
+        const score = storedMessage?.timestamp ?? now;
+        await this.redis.releaseMessage(
+          inflightKey,
+          inflightDataKey,
+          queueKey,
+          queueItemsKey,
+          tenantQueueIndexKey,
+          dispatchKey,
+          member,
+          messageId,
+          score.toString(),
+          queueId,
+          "",
+          tenantId
+        );
+
+        // Track reclaimed message for concurrency release
+        // Always add to reclaimedMessages to avoid concurrency leaks
+        if (storedMessage) {
+          reclaimedMessages.push({
+            messageId,
+            queueId,
+            tenantId: storedMessage.tenantId,
+            metadata: storedMessage.metadata,
+          });
+        } else {
+          // Fallback: extract tenantId from queueId when message data is missing or corrupted
+          // This ensures concurrency is released even if we can't get the full metadata
+          this.logger.error("Missing or corrupted message data during reclaim, using fallback", {
+            messageId,
+            queueId,
+          });
+          reclaimedMessages.push({
+            messageId,
+            queueId,
+            tenantId: this.keys.extractTenantId(queueId),
+            metadata: {},
+          });
+        }
+
+        this.logger.debug("Reclaimed timed-out message", {
+          messageId,
+          queueId,
+          deadline: _deadlineScore,
+        });
+      } catch (error) {
+        this.logger.error("Failed to reclaim message", {
+          messageId,
+          queueId,
+          error: error instanceof Error ? error.message : String(error),
+        });
+      }
+    }
+
+    return reclaimedMessages;
+  }
+
+  /**
+   * Get all in-flight messages for a shard.
+   */
+  async getInflightMessages(shardId: number): Promise<
+    Array<{
+      messageId: string;
+      queueId: string;
+      deadline: number;
+    }>
+  > {
+    const inflightKey = this.keys.inflightKey(shardId);
+    const results = await this.redis.zrange(inflightKey, 0, -1, "WITHSCORES");
+
+    const messages: Array<{ messageId: string; queueId: string; deadline: number }> = [];
+
+    for (let i = 0; i < results.length; i += 2) {
+      const member = results[i];
+      const deadlineStr = results[i + 1];
+      if (!member || !deadlineStr) {
+        continue;
+      }
+      const deadline = parseFloat(deadlineStr);
+      const { messageId, queueId } = this.#parseMember(member);
+
+      messages.push({ messageId, queueId, deadline });
+    }
+
+    return messages;
+  }
+
+  /**
+   * Get count of in-flight messages for a shard.
+   */
+  async getInflightCount(shardId: number): Promise<number> {
+    const inflightKey = this.keys.inflightKey(shardId);
+    return await this.redis.zcard(inflightKey);
+  }
+
+  /**
+   * Get total in-flight count across all shards.
+   */
+  async getTotalInflightCount(): Promise<number> {
+    const counts = await Promise.all(
+      Array.from({ length: this.shardCount }, (_, i) => this.getInflightCount(i))
+    );
+    return counts.reduce((sum, count) => sum + count, 0);
+  }
+
+  /**
+   * Close the Redis connection.
+   */
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private Methods
+  // ============================================================================
+
+  /**
+   * Map queue ID to shard using Jump Consistent Hash.
+   * Must use same algorithm as MasterQueue for consistency.
+   */
+  #getShardForQueue(queueId: string): number {
+    return jumpHash(queueId, this.shardCount);
+  }
+
+  #makeMember(messageId: string, queueId: string): string {
+    return `${messageId}:${queueId}`;
+  }
+
+  #parseMember(member: string): { messageId: string; queueId: string } {
+    const colonIndex = member.indexOf(":");
+    if (colonIndex === -1) {
+      return { messageId: member, queueId: "" };
+    }
+    return {
+      messageId: member.substring(0, colonIndex),
+      queueId: member.substring(colonIndex + 1),
+    };
+  }
+
+  async #removeFromInflight(shardId: number, messageId: string, queueId: string): Promise<void> {
+    const inflightKey = this.keys.inflightKey(shardId);
+    const inflightDataKey = this.keys.inflightDataKey(shardId);
+    const member = this.#makeMember(messageId, queueId);
+
+    const pipeline = this.redis.pipeline();
+    pipeline.zrem(inflightKey, member);
+    pipeline.hdel(inflightDataKey, messageId);
+    await pipeline.exec();
+  }
+
+  #registerCommands(): void {
+    // Atomic claim: pop from queue, add to in-flight
+    this.redis.defineCommand("claimMessage", {
+      numberOfKeys: 4,
+      lua: `
+local queueKey = KEYS[1]
+local queueItemsKey = KEYS[2]
+local inflightKey = KEYS[3]
+local inflightDataKey = KEYS[4]
+
+local queueId = ARGV[1]
+local consumerId = ARGV[2]
+local deadline = tonumber(ARGV[3])
+
+-- Get oldest message from queue
+local items = redis.call('ZRANGE', queueKey, 0, 0)
+if #items == 0 then
+  return nil
+end
+
+local messageId = items[1]
+
+-- Get message data
+local payload = redis.call('HGET', queueItemsKey, messageId)
+if not payload then
+  -- Message data missing, remove from queue and return nil
+  redis.call('ZREM', queueKey, messageId)
+  return nil
+end
+
+-- Remove from queue
+redis.call('ZREM', queueKey, messageId)
+redis.call('HDEL', queueItemsKey, messageId)
+
+-- Add to in-flight set with deadline
+local member = messageId .. ':' .. queueId
+redis.call('ZADD', inflightKey, deadline, member)
+
+-- Store message data for potential release
+redis.call('HSET', inflightDataKey, messageId, payload)
+
+return {messageId, payload}
+      `,
+    });
+
+    // Atomic batch claim: pop up to N messages from queue, add to in-flight
+    this.redis.defineCommand("claimMessageBatch", {
+      numberOfKeys: 4,
+      lua: `
+local queueKey = KEYS[1]
+local queueItemsKey = KEYS[2]
+local inflightKey = KEYS[3]
+local inflightDataKey = KEYS[4]
+
+local queueId = ARGV[1]
+local deadline = tonumber(ARGV[2])
+local maxCount = tonumber(ARGV[3])
+
+-- Get up to maxCount oldest messages from queue
+local items = redis.call('ZRANGE', queueKey, 0, maxCount - 1)
+if #items == 0 then
+  return {}
+end
+
+local results = {}
+
+for i, messageId in ipairs(items) do
+  -- Get message data
+  local payload = redis.call('HGET', queueItemsKey, messageId)
+  
+  if payload then
+    -- Remove from queue
+    redis.call('ZREM', queueKey, messageId)
+    redis.call('HDEL', queueItemsKey, messageId)
+    
+    -- Add to in-flight set with deadline
+    local member = messageId .. ':' .. queueId
+    redis.call('ZADD', inflightKey, deadline, member)
+    
+    -- Store message data for potential release
+    redis.call('HSET', inflightDataKey, messageId, payload)
+    
+    -- Add to results
+    table.insert(results, messageId)
+    table.insert(results, payload)
+  else
+    -- Message data missing, remove from queue
+    redis.call('ZREM', queueKey, messageId)
+  end
+end
+
+return results
+      `,
+    });
+
+    // Atomic release: remove from in-flight, add back to queue, update dispatch indexes
+    this.redis.defineCommand("releaseMessage", {
+      numberOfKeys: 6,
+      lua: `
+local inflightKey = KEYS[1]
+local inflightDataKey = KEYS[2]
+local queueKey = KEYS[3]
+local queueItemsKey = KEYS[4]
+local tenantQueueIndexKey = KEYS[5]
+local dispatchKey = KEYS[6]
+
+local member = ARGV[1]
+local messageId = ARGV[2]
+local score = tonumber(ARGV[3])
+local queueId = ARGV[4]
+local updatedData = ARGV[5]
+local tenantId = ARGV[6]
+
+-- Get message data from in-flight
+local payload = redis.call('HGET', inflightDataKey, messageId)
+if not payload then
+  -- Message not in in-flight or already released
+  return 0
+end
+
+-- Use updatedData if provided (e.g. incremented attempt count for retries),
+-- otherwise use the original in-flight data
+if updatedData and updatedData ~= "" then
+  payload = updatedData
+end
+
+-- Remove from in-flight
+redis.call('ZREM', inflightKey, member)
+redis.call('HDEL', inflightDataKey, messageId)
+
+-- Add back to queue
+redis.call('ZADD', queueKey, score, messageId)
+redis.call('HSET', queueItemsKey, messageId, payload)
+
+-- Update tenant queue index (Level 2) with queue's oldest message
+local oldest = redis.call('ZRANGE', queueKey, 0, 0, 'WITHSCORES')
+if #oldest >= 2 then
+  redis.call('ZADD', tenantQueueIndexKey, oldest[2], queueId)
+end
+
+-- Update dispatch index (Level 1) with tenant's oldest across all queues
+local tenantOldest = redis.call('ZRANGE', tenantQueueIndexKey, 0, 0, 'WITHSCORES')
+if #tenantOldest >= 2 then
+  redis.call('ZADD', dispatchKey, tenantOldest[2], tenantId)
+end
+
+return 1
+      `,
+    });
+
+    // Atomic batch release: release multiple messages back to queue
+    this.redis.defineCommand("releaseMessageBatch", {
+      numberOfKeys: 6,
+      lua: `
+local inflightKey = KEYS[1]
+local inflightDataKey = KEYS[2]
+local queueKey = KEYS[3]
+local queueItemsKey = KEYS[4]
+local tenantQueueIndexKey = KEYS[5]
+local dispatchKey = KEYS[6]
+
+local score = tonumber(ARGV[1])
+local queueId = ARGV[2]
+local tenantId = ARGV[3]
+
+-- Remaining args are: members..., messageIds...
+-- Calculate how many messages we have
+local numMessages = (table.getn(ARGV) - 3) / 2
+local membersStart = 4
+local messageIdsStart = membersStart + numMessages
+
+local releasedCount = 0
+
+for i = 0, numMessages - 1 do
+  local member = ARGV[membersStart + i]
+  local messageId = ARGV[messageIdsStart + i]
+
+  -- Get message data from in-flight
+  local payload = redis.call('HGET', inflightDataKey, messageId)
+  if payload then
+    -- Remove from in-flight
+    redis.call('ZREM', inflightKey, member)
+    redis.call('HDEL', inflightDataKey, messageId)
+
+    -- Add back to queue
+    redis.call('ZADD', queueKey, score, messageId)
+    redis.call('HSET', queueItemsKey, messageId, payload)
+
+    releasedCount = releasedCount + 1
+  end
+end
+
+-- Update dispatch indexes (only once at the end)
+if releasedCount > 0 then
+  -- Update tenant queue index (Level 2)
+  local oldest = redis.call('ZRANGE', queueKey, 0, 0, 'WITHSCORES')
+  if #oldest >= 2 then
+    redis.call('ZADD', tenantQueueIndexKey, oldest[2], queueId)
+  end
+  -- Update dispatch index (Level 1)
+  local tenantOldest = redis.call('ZRANGE', tenantQueueIndexKey, 0, 0, 'WITHSCORES')
+  if #tenantOldest >= 2 then
+    redis.call('ZADD', dispatchKey, tenantOldest[2], tenantId)
+  end
+end
+
+return releasedCount
+      `,
+    });
+
+    // Atomic heartbeat: check if member exists and update score
+    // ZADD XX returns 0 even on successful updates (it counts new additions only)
+    // So we need to check existence first with ZSCORE
+    this.redis.defineCommand("heartbeatMessage", {
+      numberOfKeys: 1,
+      lua: `
+local inflightKey = KEYS[1]
+local member = ARGV[1]
+local newDeadline = tonumber(ARGV[2])
+
+-- Check if member exists in the in-flight set
+local score = redis.call('ZSCORE', inflightKey, member)
+if not score then
+  return 0
+end
+
+-- Update the deadline
+redis.call('ZADD', inflightKey, 'XX', newDeadline, member)
+return 1
+      `,
+    });
+  }
+}
+
+// Extend Redis interface for custom commands
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    claimMessage(
+      queueKey: string,
+      queueItemsKey: string,
+      inflightKey: string,
+      inflightDataKey: string,
+      queueId: string,
+      consumerId: string,
+      deadline: string
+    ): Promise<[string, string] | null>;
+
+    claimMessageBatch(
+      queueKey: string,
+      queueItemsKey: string,
+      inflightKey: string,
+      inflightDataKey: string,
+      queueId: string,
+      deadline: string,
+      maxCount: string
+    ): Promise<string[]>;
+
+    releaseMessage(
+      inflightKey: string,
+      inflightDataKey: string,
+      queueKey: string,
+      queueItemsKey: string,
+      tenantQueueIndexKey: string,
+      dispatchKey: string,
+      member: string,
+      messageId: string,
+      score: string,
+      queueId: string,
+      updatedData: string,
+      tenantId: string
+    ): Promise<number>;
+
+    releaseMessageBatch(
+      inflightKey: string,
+      inflightDataKey: string,
+      queueKey: string,
+      queueItemsKey: string,
+      tenantQueueIndexKey: string,
+      dispatchKey: string,
+      score: string,
+      queueId: string,
+      tenantId: string,
+      ...membersAndMessageIds: string[]
+    ): Promise<number>;
+
+    heartbeatMessage(inflightKey: string, member: string, newDeadline: string): Promise<number>;
+  }
+}

packages/redis-worker/src/fair-queue/workerQueue.ts

@@ -0,0 +1,292 @@
+import { createRedisClient, type Redis, type RedisOptions } from "@internal/redis";
+import type { FairQueueKeyProducer } from "./types.js";
+
+export interface WorkerQueueManagerOptions {
+  redis: RedisOptions;
+  keys: FairQueueKeyProducer;
+  logger?: {
+    debug: (message: string, context?: Record<string, unknown>) => void;
+    error: (message: string, context?: Record<string, unknown>) => void;
+  };
+}
+
+/**
+ * WorkerQueueManager handles the intermediate worker queue layer.
+ *
+ * This provides:
+ * - Low-latency message delivery via blocking pop (BLPOP)
+ * - Routing of messages to specific workers/consumers
+ * - Efficient waiting without polling
+ *
+ * Flow:
+ * 1. Master queue consumer claims message from message queue
+ * 2. Message key is pushed to worker queue
+ * 3. Worker queue consumer does blocking pop to receive message
+ */
+export class WorkerQueueManager {
+  private redis: Redis;
+  private keys: FairQueueKeyProducer;
+  private logger: NonNullable<WorkerQueueManagerOptions["logger"]>;
+
+  constructor(private options: WorkerQueueManagerOptions) {
+    this.redis = createRedisClient(options.redis);
+    this.keys = options.keys;
+    this.logger = options.logger ?? {
+      debug: () => {},
+      error: () => {},
+    };
+    this.#registerCommands();
+  }
+
+  // ============================================================================
+  // Public Methods
+  // ============================================================================
+
+  /**
+   * Push a message key to a worker queue.
+   * Called after claiming a message from the message queue.
+   *
+   * @param workerQueueId - The worker queue identifier
+   * @param messageKey - The message key to push (typically "messageId:queueId")
+   */
+  async push(workerQueueId: string, messageKey: string): Promise<void> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    await this.redis.rpush(workerQueueKey, messageKey);
+
+    this.logger.debug("Pushed to worker queue", {
+      workerQueueId,
+      workerQueueKey,
+      messageKey,
+    });
+  }
+
+  /**
+   * Push multiple message keys to a worker queue.
+   *
+   * @param workerQueueId - The worker queue identifier
+   * @param messageKeys - The message keys to push
+   */
+  async pushBatch(workerQueueId: string, messageKeys: string[]): Promise<void> {
+    if (messageKeys.length === 0) {
+      return;
+    }
+
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    await this.redis.rpush(workerQueueKey, ...messageKeys);
+
+    this.logger.debug("Pushed batch to worker queue", {
+      workerQueueId,
+      workerQueueKey,
+      count: messageKeys.length,
+    });
+  }
+
+  /**
+   * Blocking pop from a worker queue.
+   * Waits until a message is available or timeout expires.
+   *
+   * @param workerQueueId - The worker queue identifier
+   * @param timeoutSeconds - Maximum time to wait (0 = wait forever)
+   * @param signal - Optional abort signal to cancel waiting
+   * @returns The message key, or null if timeout
+   */
+  async blockingPop(
+    workerQueueId: string,
+    timeoutSeconds: number,
+    signal?: AbortSignal
+  ): Promise<string | null> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+
+    // Create a separate client for blocking operation
+    // This is required because BLPOP blocks the connection
+    const blockingClient = this.redis.duplicate();
+
+    // Define cleanup outside try so it's accessible in finally
+    // This prevents listener accumulation on the AbortSignal
+    const cleanup = signal
+      ? () => {
+          blockingClient.disconnect();
+        }
+      : null;
+
+    try {
+      // Set up abort handler
+      if (signal && cleanup) {
+        signal.addEventListener("abort", cleanup, { once: true });
+
+        if (signal.aborted) {
+          return null;
+        }
+      }
+
+      const result = await blockingClient.blpop(workerQueueKey, timeoutSeconds);
+
+      if (!result) {
+        return null;
+      }
+
+      // BLPOP returns [key, value]
+      const [, messageKey] = result;
+
+      this.logger.debug("Blocking pop received message", {
+        workerQueueId,
+        workerQueueKey,
+        messageKey,
+      });
+
+      return messageKey;
+    } catch (error) {
+      // Handle abort/disconnect
+      if (signal?.aborted) {
+        return null;
+      }
+
+      this.logger.error("Blocking pop error", {
+        workerQueueId,
+        error: error instanceof Error ? error.message : String(error),
+      });
+
+      throw error;
+    } finally {
+      // Always remove the listener to prevent accumulation on the AbortSignal
+      // (once: true only removes if abort fires, not on normal completion)
+      if (cleanup && signal) {
+        signal.removeEventListener("abort", cleanup);
+      }
+      await blockingClient.quit().catch(() => {
+        // Ignore quit errors (may already be disconnected)
+      });
+    }
+  }
+
+  /**
+   * Non-blocking pop from a worker queue.
+   *
+   * @param workerQueueId - The worker queue identifier
+   * @returns The message key and queue length, or null if empty
+   */
+  async pop(workerQueueId: string): Promise<{ messageKey: string; queueLength: number } | null> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+
+    const result = await this.redis.popWithLength(workerQueueKey);
+
+    if (!result) {
+      return null;
+    }
+
+    const [messageKey, queueLength] = result;
+
+    this.logger.debug("Non-blocking pop received message", {
+      workerQueueId,
+      workerQueueKey,
+      messageKey,
+      queueLength,
+    });
+
+    return { messageKey, queueLength: Number(queueLength) };
+  }
+
+  /**
+   * Get the current length of a worker queue.
+   */
+  async getLength(workerQueueId: string): Promise<number> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    return await this.redis.llen(workerQueueKey);
+  }
+
+  /**
+   * Peek at all messages in a worker queue without removing them.
+   * Useful for debugging and tests.
+   */
+  async peek(workerQueueId: string): Promise<string[]> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    return await this.redis.lrange(workerQueueKey, 0, -1);
+  }
+
+  /**
+   * Remove a specific message from the worker queue.
+   * Used when a message needs to be removed without processing.
+   *
+   * @param workerQueueId - The worker queue identifier
+   * @param messageKey - The message key to remove
+   * @returns Number of removed items
+   */
+  async remove(workerQueueId: string, messageKey: string): Promise<number> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    return await this.redis.lrem(workerQueueKey, 0, messageKey);
+  }
+
+  /**
+   * Clear all messages from a worker queue.
+   */
+  async clear(workerQueueId: string): Promise<void> {
+    const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
+    await this.redis.del(workerQueueKey);
+  }
+
+  /**
+   * Close the Redis connection.
+   */
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  // ============================================================================
+  // Private - Register Commands
+  // ============================================================================
+
+  /**
+   * Initialize custom Redis commands.
+   */
+  #registerCommands(): void {
+    // Non-blocking pop with queue length
+    this.redis.defineCommand("popWithLength", {
+      numberOfKeys: 1,
+      lua: `
+local workerQueueKey = KEYS[1]
+
+-- Pop the first message
+local messageKey = redis.call('LPOP', workerQueueKey)
+if not messageKey then
+  return nil
+end
+
+-- Get remaining queue length
+local queueLength = redis.call('LLEN', workerQueueKey)
+
+return {messageKey, queueLength}
+      `,
+    });
+  }
+
+  /**
+   * Register custom commands on an external Redis client.
+   * Use this when initializing FairQueue with worker queues.
+   */
+  registerCommands(redis: Redis): void {
+    redis.defineCommand("popWithLength", {
+      numberOfKeys: 1,
+      lua: `
+local workerQueueKey = KEYS[1]
+
+-- Pop the first message
+local messageKey = redis.call('LPOP', workerQueueKey)
+if not messageKey then
+  return nil
+end
+
+-- Get remaining queue length
+local queueLength = redis.call('LLEN', workerQueueKey)
+
+return {messageKey, queueLength}
+      `,
+    });
+  }
+}
+
+// Extend Redis interface for custom commands
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    popWithLength(workerQueueKey: string): Promise<[string, string] | null>;
+  }
+}

packages/redis-worker/src/index.ts

@@ -0,0 +1,7 @@
+export * from "./queue.js";
+export * from "./worker.js";
+export * from "./utils.js";
+
+// Fair Queue System
+export * from "./fair-queue/index.js";
+export * from "./mollifier/index.js";

packages/redis-worker/src/mollifier/drainer.ts

@@ -0,0 +1,444 @@
+import { Logger } from "@trigger.dev/core/logger";
+import type { MollifierBuffer } from "./buffer.js";
+import type { BufferEntry } from "./schemas.js";
+import { deserialiseSnapshot } from "./schemas.js";
+
+export type MollifierDrainerHandler<TPayload> = (input: {
+  runId: string;
+  envId: string;
+  orgId: string;
+  payload: TPayload;
+  attempts: number;
+  createdAt: Date;
+}) => Promise<void>;
+
+// Invoked once per entry before `buffer.fail()` on any terminal path —
+// non-retryable error OR retryable error after maxAttempts. Lets the caller
+// land a SYSTEM_FAILURE PG row so the customer sees the run instead of it
+// silently disappearing alongside the buffer entry. Throwing a retryable
+// error from the callback causes the drainer to requeue rather than fail
+// (so the PG write itself gets another chance once PG recovers); throwing
+// anything else falls through to `buffer.fail()` to avoid an infinite loop
+// on a genuinely bad payload.
+export type MollifierDrainerTerminalFailureCause = "non-retryable" | "max-attempts-exhausted";
+export type MollifierDrainerTerminalFailureHandler<TPayload> = (input: {
+  runId: string;
+  envId: string;
+  orgId: string;
+  payload: TPayload;
+  attempts: number;
+  createdAt: Date;
+  error: { code: string; message: string };
+  cause: MollifierDrainerTerminalFailureCause;
+}) => Promise<void>;
+
+export type MollifierDrainerOptions<TPayload> = {
+  buffer: MollifierBuffer;
+  handler: MollifierDrainerHandler<TPayload>;
+  onTerminalFailure?: MollifierDrainerTerminalFailureHandler<TPayload>;
+  concurrency: number;
+  maxAttempts: number;
+  isRetryable: (err: unknown) => boolean;
+  pollIntervalMs?: number;
+  // Cap on how many ORGS `runOnce` processes per tick. The drainer rotates
+  // through orgs at the top level and picks one env per org per tick, so
+  // the actual per-tick pop count is at most `maxOrgsPerTick`. Tune for
+  // "typical orgs with pending entries" rather than total system org
+  // count. Defaults to 500.
+  //
+  // The buffer maintains `mollifier:orgs` and `mollifier:org-envs:${orgId}`
+  // atomically with per-env queues, so the drainer can walk orgs → envs
+  // directly. An org with N envs gets the same per-tick scheduling slot
+  // as an org with 1 env — tenant-level drainage throughput is determined
+  // by org count, not env count.
+  maxOrgsPerTick?: number;
+  // Per-env per-tick pop cap. Default 1 preserves the original
+  // one-pop-per-env-per-tick behaviour. Setting it higher lets a single
+  // env drain at handler-parallelism speed: each tick the drainer pops
+  // up to `drainBatchSize` entries from the env's queue, then dispatches
+  // them all through the shared `concurrency`-bounded pLimit. For a
+  // single-env burst this turns N sequential ticks into one tick of N
+  // parallel handler calls, capped by `concurrency`. Org/env fairness
+  // still holds — each org still contributes exactly one env per tick.
+  //
+  // Memory: per-tick in-flight entries ≤ `maxOrgsPerTick × drainBatchSize`.
+  // Operators sizing this should ensure their PG pool / engine handler
+  // can sustain `concurrency` parallel writes; popping more than the
+  // handler can process per tick just queues entries in JS waiting on
+  // pLimit.
+  drainBatchSize?: number;
+  // Cap on the exponential backoff applied after consecutive `runOnce`
+  // errors. Defaults to 5000ms. The backoff base is `max(pollIntervalMs,
+  // backoffFloorMs)` and doubles per consecutive error up to this cap.
+  maxBackoffMs?: number;
+  // Floor for the exponential-backoff base, so a tiny `pollIntervalMs`
+  // doesn't collapse the backoff to near-zero on a sustained outage.
+  // Defaults to 100ms.
+  backoffFloorMs?: number;
+  logger?: Logger;
+};
+
+export type DrainResult = {
+  drained: number;
+  failed: number;
+};
+
+export class MollifierDrainer<TPayload = unknown> {
+  private readonly buffer: MollifierBuffer;
+  private readonly handler: MollifierDrainerHandler<TPayload>;
+  private readonly onTerminalFailure?: MollifierDrainerTerminalFailureHandler<TPayload>;
+  private readonly maxAttempts: number;
+  private readonly isRetryable: (err: unknown) => boolean;
+  private readonly pollIntervalMs: number;
+  private readonly maxOrgsPerTick: number;
+  private readonly drainBatchSize: number;
+  private readonly concurrency: number;
+  private readonly maxBackoffMs: number;
+  private readonly backoffFloorMs: number;
+  private readonly logger: Logger;
+  // Rotation state. `orgCursor` advances through the active-orgs list.
+  // Each org has its own internal cursor in `perOrgEnvCursors` for
+  // cycling through that org's envs. Both reset on `start()`.
+  private orgCursor = 0;
+  private perOrgEnvCursors = new Map<string, number>();
+  private isRunning = false;
+  private stopping = false;
+  private loopPromise: Promise<void> | null = null;
+
+  constructor(options: MollifierDrainerOptions<TPayload>) {
+    this.buffer = options.buffer;
+    this.handler = options.handler;
+    this.onTerminalFailure = options.onTerminalFailure;
+    this.maxAttempts = options.maxAttempts;
+    this.isRetryable = options.isRetryable;
+    this.pollIntervalMs = options.pollIntervalMs ?? 100;
+    this.maxOrgsPerTick = options.maxOrgsPerTick ?? 500;
+    this.drainBatchSize = Math.max(1, options.drainBatchSize ?? 1);
+    this.concurrency = Math.max(1, options.concurrency);
+    this.maxBackoffMs = options.maxBackoffMs ?? 5_000;
+    this.backoffFloorMs = Math.max(1, options.backoffFloorMs ?? 100);
+    this.logger = options.logger ?? new Logger("MollifierDrainer", "debug");
+  }
+
+  async runOnce(): Promise<DrainResult> {
+    const orgs = await this.buffer.listOrgs();
+    if (orgs.length === 0) return { drained: 0, failed: 0 };
+
+    const orgSlice = this.takeOrgSlice(orgs);
+
+    // Fan the per-org SMEMBERS out in a single pipelined round-trip. Serial
+    // awaits would otherwise add `orgSlice.length × RTT` of dead time before
+    // pops start — at the default `maxOrgsPerTick=500` and a ~1ms ElastiCache
+    // RTT that's a ~500ms per-tick floor. ioredis auto-pipelines concurrent
+    // commands into one batch, so the burst is cheap; SMEMBERS on a small set
+    // is O(N) per org and trivial at this scale. `Promise.all` preserves
+    // order, so the org→envs pairing below stays deterministic.
+    const envsByOrg = await Promise.all(orgSlice.map((orgId) => this.buffer.listEnvsForOrg(orgId)));
+    const targets: string[] = [];
+    for (let i = 0; i < orgSlice.length; i++) {
+      const orgId = orgSlice[i]!;
+      const envsForOrg = envsByOrg[i]!;
+      if (envsForOrg.length === 0) continue;
+      const envId = this.pickEnvForOrg(orgId, envsForOrg);
+      targets.push(envId);
+    }
+
+    if (targets.length === 0) return { drained: 0, failed: 0 };
+
+    // Worker-pool draining. We spawn up to `concurrency` workers; each
+    // worker repeatedly:
+    //   1. Picks the next env with budget remaining (round-robin),
+    //      atomically claiming one slot of that env's per-tick budget.
+    //   2. Pops one entry and processes it.
+    //   3. Repeats until pickNextEnv returns null.
+    //
+    // This pattern gives us both invariants the prior two designs traded
+    // off:
+    //   - Single-env bursts use the full `concurrency` budget. All
+    //     workers can pull from one env, processing `concurrency` entries
+    //     in parallel.
+    //   - The number of entries in "popped-but-not-acked" (DRAINING)
+    //     state at any moment is bounded by the worker count, i.e.
+    //     `concurrency` — same blast radius as the pre-batch
+    //     one-pop-per-env model. A process crash mid-tick strands at
+    //     most `concurrency` entries for stale-sweep to recover, not
+    //     `maxOrgsPerTick × drainBatchSize`.
+    //
+    // Fairness: pickNextEnv advances a cursor by 1 each successful pick,
+    // so workers round-robin across envs at the entry level. Combined
+    // with the per-env budget cap, an env contributes at most
+    // `drainBatchSize` entries per tick regardless of how many workers
+    // are free — a heavy env can't starve siblings within a tick.
+    const remaining = new Map<string, number>();
+    const skip = new Set<string>(); // envs with empty queue or pop failure this tick
+    for (const envId of targets) remaining.set(envId, this.drainBatchSize);
+
+    let cursor = 0;
+    const pickNextEnv = (): string | null => {
+      for (let i = 0; i < targets.length; i++) {
+        const idx = (cursor + i) % targets.length;
+        const envId = targets[idx]!;
+        if (skip.has(envId)) continue;
+        const r = remaining.get(envId) ?? 0;
+        if (r > 0) {
+          remaining.set(envId, r - 1);
+          cursor = (idx + 1) % targets.length;
+          return envId;
+        }
+      }
+      return null;
+    };
+
+    let drained = 0;
+    let failed = 0;
+
+    const worker = async (): Promise<void> => {
+      while (true) {
+        const envId = pickNextEnv();
+        if (envId === null) return;
+        let entry: BufferEntry | null;
+        try {
+          entry = await this.buffer.pop(envId);
+        } catch (err) {
+          // A pop failure on one env aborts that env's batch for this
+          // tick (don't keep hammering a broken Redis) and counts as
+          // exactly one failure — same as the pre-batch path on a pop
+          // blowup. Other envs continue.
+          //
+          // `pickNextEnv` decrements `remaining` before the pop settles,
+          // so multiple workers can race into the same env and all hit
+          // a throwing pop before the first catch lands. Guarding the
+          // failure increment on `!skip.has(envId)` keeps the per-env
+          // failure count at exactly one even under that race —
+          // matching the documented contract.
+          this.logger.error("MollifierDrainer.pop failed", { envId, err });
+          if (!skip.has(envId)) {
+            skip.add(envId);
+            failed += 1;
+          }
+          continue;
+        }
+        if (!entry) {
+          // Queue exhausted between scheduling and this pop. Mark the
+          // env skipped so siblings aren't held up by repeated empty pops.
+          skip.add(envId);
+          continue;
+        }
+        try {
+          const outcome = await this.processEntry(entry);
+          if (outcome === "drained") drained += 1;
+          else failed += 1;
+        } catch (err) {
+          this.logger.error("MollifierDrainer.processEntry failed", {
+            envId,
+            runId: entry.runId,
+            err,
+          });
+          failed += 1;
+        }
+      }
+    };
+
+    const totalBudget = targets.length * this.drainBatchSize;
+    const workerCount = Math.min(this.concurrency, totalBudget);
+    await Promise.all(Array.from({ length: workerCount }, () => worker()));
+
+    return { drained, failed };
+  }
+
+  start(): void {
+    if (this.isRunning) return;
+    this.isRunning = true;
+    this.stopping = false;
+    // Reset rotation state on each (re)start. A stop+start cycle means
+    // operator intent to "begin clean" — between-restart cursor drift
+    // would otherwise carry implicit state across what should look like
+    // a fresh boot.
+    this.orgCursor = 0;
+    this.perOrgEnvCursors = new Map();
+    this.loopPromise = this.loop();
+  }
+
+  // Signal the loop to exit (`stopping = true`) and wait for it. With no
+  // timeout, wait indefinitely for the in-flight `runOnce` and its handlers
+  // to settle — same semantic as FairQueue / BatchQueue's `stop()`. With a
+  // timeout, race the loop promise against a deadline so a hung handler
+  // can't wedge the process past its termination grace period.
+  async stop(options: { timeoutMs?: number } = {}): Promise<void> {
+    if (!this.isRunning || !this.loopPromise) return;
+    this.stopping = true;
+    if (options.timeoutMs == null) {
+      await this.loopPromise;
+      return;
+    }
+    // Hold the timer handle so we can clearTimeout() it after the race.
+    // Without this, when the loop wins the race, the discarded timer is
+    // still ref'd and pins the Node event loop for up to `timeoutMs`,
+    // delaying process shutdown by exactly the slack we were trying to
+    // bound. try/finally clears the handle in every exit path (loop-won,
+    // timeout-won, or exception).
+    const timeoutSentinel = Symbol("mollifier.stop.timeout");
+    let timeoutHandle: ReturnType<typeof setTimeout> | undefined;
+    const timeoutPromise = new Promise<typeof timeoutSentinel>((resolve) => {
+      timeoutHandle = setTimeout(() => resolve(timeoutSentinel), options.timeoutMs);
+    });
+    try {
+      const winner = await Promise.race([
+        this.loopPromise.then(() => "done" as const),
+        timeoutPromise,
+      ]);
+      if (winner === timeoutSentinel) {
+        this.logger.warn(
+          "MollifierDrainer.stop: deadline exceeded; returning while loop iteration is in flight",
+          { timeoutMs: options.timeoutMs }
+        );
+      }
+    } finally {
+      if (timeoutHandle) clearTimeout(timeoutHandle);
+    }
+  }
+
+  // Transient Redis errors (e.g. a connection blip in `listOrgs` /
+  // `listEnvsForOrg` / `pop`) must not kill the polling loop permanently.
+  // We log each `runOnce` failure, back off so we don't spin tight on a
+  // sustained outage, and resume. The loop only exits when `stop()` flips
+  // `stopping`.
+  private async loop(): Promise<void> {
+    try {
+      let consecutiveErrors = 0;
+      while (!this.stopping) {
+        try {
+          const result = await this.runOnce();
+          consecutiveErrors = 0;
+          if (result.drained === 0 && result.failed === 0) {
+            await this.delay(this.pollIntervalMs);
+          }
+        } catch (err) {
+          consecutiveErrors += 1;
+          this.logger.error("MollifierDrainer.runOnce failed; backing off", {
+            err,
+            consecutiveErrors,
+          });
+          await this.delay(this.backoffMs(consecutiveErrors));
+        }
+      }
+    } finally {
+      this.isRunning = false;
+    }
+  }
+
+  // Exponential backoff capped at 5s. Keeps the loop responsive after a
+  // brief blip while preventing a tight retry loop during a long Redis
+  // outage. 1 → 200ms, 2 → 400ms, 3 → 800ms, 4 → 1.6s, 5 → 3.2s, 6+ → 5s.
+  private backoffMs(consecutiveErrors: number): number {
+    const base = Math.max(this.pollIntervalMs, this.backoffFloorMs);
+    const capped = Math.min(base * 2 ** (consecutiveErrors - 1), this.maxBackoffMs);
+    return capped;
+  }
+
+  private delay(ms: number): Promise<void> {
+    return new Promise((resolve) => setTimeout(resolve, ms));
+  }
+
+  // Take up to `maxOrgsPerTick` orgs starting at the current cursor, with
+  // wrap-around. Cursor advances by 1 each tick so every org reaches
+  // every slot position (0..sliceSize-1) over a full cycle — no
+  // head-of-line bias within the slice. Orgs are sorted before slicing
+  // so rotation is deterministic regardless of Redis SET iteration order.
+  private takeOrgSlice(orgs: string[]): string[] {
+    const sorted = [...orgs].sort();
+    const n = sorted.length;
+    const sliceSize = Math.min(this.maxOrgsPerTick, n);
+    const start = this.orgCursor % n;
+    this.orgCursor = (this.orgCursor + 1) % Math.max(n, 1);
+    const end = start + sliceSize;
+    if (end <= n) return sorted.slice(start, end);
+    return [...sorted.slice(start), ...sorted.slice(0, end - n)];
+  }
+
+  // Pick one env from the org's active-envs list, rotating per org via
+  // the per-org cursor. Each org's cursor advances by 1 each visit, so
+  // an org with N envs cycles through them across N visits.
+  private pickEnvForOrg(orgId: string, envsForOrg: string[]): string {
+    const sorted = [...envsForOrg].sort();
+    const cursor = this.perOrgEnvCursors.get(orgId) ?? 0;
+    const idx = cursor % sorted.length;
+    this.perOrgEnvCursors.set(orgId, (cursor + 1) % sorted.length);
+    return sorted[idx]!;
+  }
+
+  private async processEntry(entry: BufferEntry): Promise<"drained" | "failed"> {
+    try {
+      const payload = deserialiseSnapshot<TPayload>(entry.payload);
+      await this.handler({
+        runId: entry.runId,
+        envId: entry.envId,
+        orgId: entry.orgId,
+        payload,
+        attempts: entry.attempts,
+        createdAt: entry.createdAt,
+      });
+      await this.buffer.ack(entry.runId);
+      return "drained";
+    } catch (err) {
+      const nextAttempts = entry.attempts + 1;
+      if (this.isRetryable(err) && nextAttempts < this.maxAttempts) {
+        await this.buffer.requeue(entry.runId);
+        this.logger.warn("MollifierDrainer: retryable error, requeued", {
+          runId: entry.runId,
+          attempts: nextAttempts,
+        });
+        return "failed";
+      }
+      const cause: MollifierDrainerTerminalFailureCause = this.isRetryable(err)
+        ? "max-attempts-exhausted"
+        : "non-retryable";
+      const code = err instanceof Error ? err.name : "Unknown";
+      const message = err instanceof Error ? err.message : String(err);
+      // Run the terminal-failure callback BEFORE buffer.fail() so a
+      // SYSTEM_FAILURE PG row can land while the entry is still around to
+      // read from (and so we don't lose the run if the callback's own
+      // write itself needs a retry). If the callback throws a retryable
+      // error, requeue the entry instead of fail()ing — PG is still
+      // unreachable, give it another tick. Any other callback failure
+      // falls through to buffer.fail() so a genuinely bad snapshot
+      // doesn't loop forever.
+      if (this.onTerminalFailure) {
+        try {
+          await this.onTerminalFailure({
+            runId: entry.runId,
+            envId: entry.envId,
+            orgId: entry.orgId,
+            payload: deserialiseSnapshot<TPayload>(entry.payload),
+            attempts: nextAttempts,
+            createdAt: entry.createdAt,
+            error: { code, message },
+            cause,
+          });
+        } catch (writeErr) {
+          if (this.isRetryable(writeErr)) {
+            await this.buffer.requeue(entry.runId);
+            this.logger.warn("MollifierDrainer: terminal-failure callback retryable; requeued", {
+              runId: entry.runId,
+              attempts: nextAttempts,
+              writeErr,
+            });
+            return "failed";
+          }
+          this.logger.error("MollifierDrainer: terminal-failure callback failed", {
+            runId: entry.runId,
+            writeErr,
+          });
+        }
+      }
+      await this.buffer.fail(entry.runId, { code, message });
+      this.logger.error("MollifierDrainer: terminal failure", {
+        runId: entry.runId,
+        code,
+        message,
+        cause,
+      });
+      return "failed";
+    }
+  }
+}

packages/redis-worker/src/mollifier/index.ts

@@ -0,0 +1,28 @@
+export {
+  MollifierBuffer,
+  type MollifierBufferOptions,
+  type SnapshotPatch,
+  type AcceptResult,
+  type MutateSnapshotResult,
+  type CasSetMetadataResult,
+  type IdempotencyClaimResult,
+  type IdempotencyLookupInput,
+  idempotencyLookupKeyFor,
+  makeIdempotencyClaimKey,
+} from "./buffer.js";
+export {
+  MollifierDrainer,
+  type MollifierDrainerOptions,
+  type MollifierDrainerHandler,
+  type MollifierDrainerTerminalFailureHandler,
+  type MollifierDrainerTerminalFailureCause,
+  type DrainResult,
+} from "./drainer.js";
+export {
+  BufferEntrySchema,
+  BufferEntryStatus,
+  BufferEntryError,
+  serialiseSnapshot,
+  deserialiseSnapshot,
+  type BufferEntry,
+} from "./schemas.js";

packages/redis-worker/src/mollifier/schemas.ts

@@ -0,0 +1,86 @@
+import { z } from "zod";
+
+export const BufferEntryStatus = z.enum(["QUEUED", "DRAINING", "FAILED"]);
+export type BufferEntryStatus = z.infer<typeof BufferEntryStatus>;
+
+export const BufferEntryError = z.object({
+  code: z.string(),
+  message: z.string(),
+});
+export type BufferEntryError = z.infer<typeof BufferEntryError>;
+
+const stringToInt = z.string().transform((v, ctx) => {
+  const n = Number(v);
+  if (!Number.isInteger(n) || n < 0) {
+    ctx.addIssue({ code: z.ZodIssueCode.custom, message: "expected non-negative integer string" });
+    return z.NEVER;
+  }
+  return n;
+});
+
+const stringToDate = z.string().transform((v, ctx) => {
+  const d = new Date(v);
+  if (Number.isNaN(d.getTime())) {
+    ctx.addIssue({ code: z.ZodIssueCode.custom, message: "expected ISO date string" });
+    return z.NEVER;
+  }
+  return d;
+});
+
+const stringToBool = z
+  .union([z.literal("true"), z.literal("false")])
+  .transform((v) => v === "true");
+
+const stringToError = z.string().transform((v, ctx) => {
+  try {
+    return BufferEntryError.parse(JSON.parse(v));
+  } catch {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      message: "expected JSON-encoded BufferEntryError",
+    });
+    return z.NEVER;
+  }
+});
+
+export const BufferEntrySchema = z.object({
+  runId: z.string().min(1),
+  envId: z.string().min(1),
+  orgId: z.string().min(1),
+  payload: z.string(),
+  status: BufferEntryStatus,
+  attempts: stringToInt,
+  createdAt: stringToDate,
+  // Microsecond epoch of accept time, kept as a hash field for dwell
+  // metrics. Not a queue sort key (the queue is a FIFO LIST). Defaulted
+  // so an entry written by an accept Lua predating this field — or one
+  // surviving across the deploy that introduced it — still parses instead
+  // of being silently dropped on pop.
+  createdAtMicros: stringToInt.default("0"),
+  // Drainer-ack flag: `true` once the drainer has materialised this run
+  // into PG. The hash persists for a short grace TTL after ack so direct
+  // reads (retrieve, trace, etc.) still resolve while PG replica lag
+  // settles. Absent on pre-ack entries.
+  materialised: stringToBool.default("false"),
+  // Denormalised pointer to the Redis idempotency lookup key (set when
+  // the run was accepted with an idempotency key, empty otherwise). The
+  // ack Lua reads this to DEL the lookup atomically with marking the
+  // entry materialised.
+  idempotencyLookupKey: z.string().optional().default(""),
+  // Optimistic-lock counter for the snapshot's `metadata` field.
+  // Incremented atomically by the CAS metadata Lua. Matches the
+  // semantic of `TaskRun.metadataVersion` on the PG side (which the
+  // UpdateMetadataService uses for the same retry-on-conflict pattern).
+  metadataVersion: stringToInt.default("0"),
+  lastError: stringToError.optional(),
+});
+
+export type BufferEntry = z.infer<typeof BufferEntrySchema>;
+
+export function serialiseSnapshot(snapshot: unknown): string {
+  return JSON.stringify(snapshot);
+}
+
+export function deserialiseSnapshot<T = unknown>(serialised: string): T {
+  return JSON.parse(serialised) as T;
+}

packages/redis-worker/src/queue.test.ts

@@ -0,0 +1,606 @@
+import { redisTest } from "@internal/testcontainers";
+import { describe } from "node:test";
+import { expect } from "vitest";
+import { z } from "zod";
+import { SimpleQueue } from "./queue.js";
+import { Logger } from "@trigger.dev/core/logger";
+import { createRedisClient } from "@internal/redis";
+
+describe("SimpleQueue", () => {
+  redisTest("enqueue/dequeue", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+      expect(await queue.size()).toBe(1);
+
+      await queue.enqueue({ id: "2", job: "test", item: { value: 2 }, visibilityTimeoutMs: 2000 });
+      expect(await queue.size()).toBe(2);
+
+      const [first] = await queue.dequeue(1);
+
+      if (!first) {
+        throw new Error("No item dequeued");
+      }
+
+      expect(first).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+      expect(await queue.size()).toBe(1);
+      expect(await queue.size({ includeFuture: true })).toBe(2);
+
+      await queue.ack(first.id);
+      expect(await queue.size({ includeFuture: true })).toBe(1);
+
+      const [second] = await queue.dequeue(1);
+
+      if (!second) {
+        throw new Error("No item dequeued");
+      }
+
+      expect(second).toEqual(
+        expect.objectContaining({
+          id: "2",
+          job: "test",
+          item: { value: 2 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      await queue.ack(second.id);
+      expect(await queue.size({ includeFuture: true })).toBe(0);
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("getJob", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+      const job1 = await queue.getJob("1");
+
+      expect(job1).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      await queue.enqueue({ id: "2", job: "test", item: { value: 2 }, visibilityTimeoutMs: 2000 });
+      const job2 = await queue.getJob("2");
+
+      expect(job2).toEqual(
+        expect.objectContaining({
+          id: "2",
+          job: "test",
+          item: { value: 2 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      const job3 = await queue.getJob("3");
+      expect(job3).toBeNull();
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("no items", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      const missOne = await queue.dequeue(1);
+      expect(missOne).toEqual([]);
+
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+      const [hitOne] = await queue.dequeue(1);
+      expect(hitOne).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      const missTwo = await queue.dequeue(1);
+      expect(missTwo).toEqual([]);
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("future item", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      await queue.enqueue({
+        id: "1",
+        job: "test",
+        item: { value: 1 },
+        availableAt: new Date(Date.now() + 50),
+        visibilityTimeoutMs: 2000,
+        attempt: 0,
+      });
+
+      const miss = await queue.dequeue(1);
+      expect(miss).toEqual([]);
+
+      await new Promise((resolve) => setTimeout(resolve, 50));
+
+      const [first] = await queue.dequeue();
+      expect(first).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("oldestMessageAge", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      // empty queue → 0
+      expect(await queue.oldestMessageAge()).toBe(0);
+
+      // only a future-scheduled item → 0 (not yet overdue)
+      await queue.enqueue({
+        id: "future",
+        job: "test",
+        item: { value: 1 },
+        availableAt: new Date(Date.now() + 60_000),
+        visibilityTimeoutMs: 2000,
+      });
+      expect(await queue.oldestMessageAge()).toBe(0);
+
+      // an overdue item → age > 0
+      await queue.enqueue({
+        id: "overdue",
+        job: "test",
+        item: { value: 2 },
+        availableAt: new Date(Date.now() - 5_000),
+        visibilityTimeoutMs: 2000,
+      });
+      const age = await queue.oldestMessageAge();
+      expect(age).toBeGreaterThanOrEqual(5_000);
+      expect(age).toBeLessThan(60_000);
+
+      // an orphaned queue entry (no payload in the items hash), older than the
+      // real overdue item, must be ignored — it can't be dequeued, so it isn't a
+      // real stall. Age should still reflect the real overdue item (~5s), not the
+      // orphan's ~999s.
+      const redisClient = createRedisClient({
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      });
+      await redisClient.zadd(`{queue:test-1:}queue`, Date.now() - 999_000, "orphaned-id");
+      const ageWithOrphan = await queue.oldestMessageAge();
+      expect(ageWithOrphan).toBeGreaterThanOrEqual(5_000);
+      expect(ageWithOrphan).toBeLessThan(60_000);
+
+      // once dequeued, the item is invisible (future-scored) → back to 0 (the
+      // orphan is cleaned by the dequeue scan, the real item goes in-flight)
+      const [first] = await queue.dequeue(2);
+      expect(first?.id).toBe("overdue");
+      expect(await queue.oldestMessageAge()).toBe(0);
+
+      await redisClient.quit();
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("invisibility timeout", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 1_000 });
+
+      const [first] = await queue.dequeue();
+      expect(first).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 1_000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      const missImmediate = await queue.dequeue(1);
+      expect(missImmediate).toEqual([]);
+
+      await new Promise((resolve) => setTimeout(resolve, 1_000));
+
+      const [second] = await queue.dequeue();
+      expect(second).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 1_000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      // Acknowledge the item and verify it's removed
+      await queue.ack(second!.id);
+      expect(await queue.size({ includeFuture: true })).toBe(0);
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("dequeue multiple items", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-1",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+      await queue.enqueue({ id: "2", job: "test", item: { value: 2 }, visibilityTimeoutMs: 2000 });
+      await queue.enqueue({ id: "3", job: "test", item: { value: 3 }, visibilityTimeoutMs: 2000 });
+
+      expect(await queue.size()).toBe(3);
+
+      const dequeued = await queue.dequeue(2);
+      expect(dequeued).toHaveLength(2);
+      expect(dequeued[0]).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+      expect(dequeued[1]).toEqual(
+        expect.objectContaining({
+          id: "2",
+          job: "test",
+          item: { value: 2 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      expect(await queue.size()).toBe(1);
+      expect(await queue.size({ includeFuture: true })).toBe(3);
+
+      if (!dequeued[0] || !dequeued[1]) {
+        throw new Error("No items dequeued");
+      }
+
+      await queue.ack(dequeued[0].id);
+      await queue.ack(dequeued[1].id);
+
+      expect(await queue.size({ includeFuture: true })).toBe(1);
+
+      const [last] = await queue.dequeue(1);
+      expect(last).toEqual(
+        expect.objectContaining({
+          id: "3",
+          job: "test",
+          item: { value: 3 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      if (!last) {
+        throw new Error("No item dequeued");
+      }
+
+      await queue.ack(last.id);
+      expect(await queue.size({ includeFuture: true })).toBe(0);
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("Dead Letter Queue", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-dlq",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      // Enqueue an item
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+      expect(await queue.size()).toBe(1);
+      expect(await queue.sizeOfDeadLetterQueue()).toBe(0);
+
+      // Move item to DLQ
+      await queue.moveToDeadLetterQueue("1", "Test error message");
+      expect(await queue.size()).toBe(0);
+      expect(await queue.sizeOfDeadLetterQueue()).toBe(1);
+
+      // Attempt to dequeue from the main queue should return empty
+      const dequeued = await queue.dequeue(1);
+      expect(dequeued).toEqual([]);
+
+      // Redrive item from DLQ
+      await queue.redriveFromDeadLetterQueue("1");
+      await new Promise((resolve) => setTimeout(resolve, 200));
+      expect(await queue.size()).toBe(1);
+      expect(await queue.sizeOfDeadLetterQueue()).toBe(0);
+
+      // Dequeue the redriven item
+      const [redrivenItem] = await queue.dequeue(1);
+
+      if (!redrivenItem) {
+        throw new Error("No item dequeued");
+      }
+
+      expect(redrivenItem).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      // Acknowledge the item
+      await queue.ack(redrivenItem.id);
+      expect(await queue.size()).toBe(0);
+      expect(await queue.sizeOfDeadLetterQueue()).toBe(0);
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest("cleanup orphaned queue entries", { timeout: 20_000 }, async ({ redisContainer }) => {
+    const queue = new SimpleQueue({
+      name: "test-orphaned",
+      schema: {
+        test: z.object({
+          value: z.number(),
+        }),
+      },
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      logger: new Logger("test", "log"),
+    });
+
+    try {
+      // First, add a normal item
+      await queue.enqueue({ id: "1", job: "test", item: { value: 1 }, visibilityTimeoutMs: 2000 });
+
+      const redisClient = createRedisClient({
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      });
+
+      // Manually add an orphaned item to the queue (without corresponding hash entry)
+      await redisClient.zadd(`{queue:test-orphaned:}queue`, Date.now(), "orphaned-id");
+
+      // Verify both items are in the queue
+      expect(await queue.size()).toBe(2);
+
+      // Dequeue should process both items, but only return the valid one
+      // and clean up the orphaned entry
+      const dequeued = await queue.dequeue(2);
+
+      // Should only get the valid item
+      expect(dequeued).toHaveLength(1);
+      expect(dequeued[0]).toEqual(
+        expect.objectContaining({
+          id: "1",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          attempt: 0,
+          timestamp: expect.any(Date),
+        })
+      );
+
+      // The orphaned item should have been removed
+      expect(await queue.size({ includeFuture: true })).toBe(1);
+
+      // Verify the orphaned ID is no longer in the queue
+      const orphanedScore = await redisClient.zscore(`{queue:test-orphaned:}queue`, "orphaned-id");
+      expect(orphanedScore).toBeNull();
+    } finally {
+      await queue.close();
+    }
+  });
+
+  redisTest(
+    "enqueueOnce only enqueues the first message with a given ID",
+    { timeout: 20_000 },
+    async ({ redisContainer }) => {
+      const queue = new SimpleQueue({
+        name: "test-once",
+        schema: {
+          test: z.object({
+            value: z.number(),
+          }),
+        },
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        logger: new Logger("test", "log"),
+      });
+
+      try {
+        const now = Date.now();
+        const availableAt1 = new Date(now + 1000);
+        const availableAt2 = new Date(now + 5000);
+
+        // First enqueueOnce should succeed
+        const first = await queue.enqueueOnce({
+          id: "unique-id",
+          job: "test",
+          item: { value: 1 },
+          visibilityTimeoutMs: 2000,
+          availableAt: availableAt1,
+        });
+        expect(first).toBe(true);
+        expect(await queue.size({ includeFuture: true })).toBe(1);
+
+        // Second enqueueOnce with same ID but different value and availableAt should do nothing
+        const second = await queue.enqueueOnce({
+          id: "unique-id",
+          job: "test",
+          item: { value: 999 },
+          visibilityTimeoutMs: 2000,
+          availableAt: availableAt2,
+        });
+        expect(second).toBe(false);
+        expect(await queue.size({ includeFuture: true })).toBe(1);
+
+        // Dequeue after 1s should get the original item, not the second
+        await new Promise((resolve) => setTimeout(resolve, 1100));
+        const [item] = await queue.dequeue(1);
+        expect(item).toBeDefined();
+        expect(item?.id).toBe("unique-id");
+        expect(item?.item).toEqual({ value: 1 });
+        // Should not be the second value
+        expect(item?.item).not.toEqual({ value: 999 });
+      } finally {
+        await queue.close();
+      }
+    }
+  );
+});

packages/redis-worker/src/queue.ts

@@ -0,0 +1,768 @@
+import {
+  createRedisClient,
+  type Redis,
+  type Callback,
+  type RedisOptions,
+  type Result,
+} from "@internal/redis";
+import { Logger } from "@trigger.dev/core/logger";
+import { nanoid } from "nanoid";
+import type { z } from "zod";
+
+export interface MessageCatalogSchema {
+  [key: string]: z.ZodFirstPartySchemaTypes | z.ZodDiscriminatedUnion<any, any>;
+}
+
+export type MessageCatalogKey<TMessageCatalog extends MessageCatalogSchema> = keyof TMessageCatalog;
+export type MessageCatalogValue<
+  TMessageCatalog extends MessageCatalogSchema,
+  TKey extends MessageCatalogKey<TMessageCatalog>,
+> = z.infer<TMessageCatalog[TKey]>;
+
+export type AnyMessageCatalog = MessageCatalogSchema;
+export type QueueItem<TMessageCatalog extends MessageCatalogSchema> = {
+  id: string;
+  job: MessageCatalogKey<TMessageCatalog>;
+  item: MessageCatalogValue<TMessageCatalog, MessageCatalogKey<TMessageCatalog>>;
+  visibilityTimeoutMs: number;
+  attempt: number;
+  timestamp: Date;
+  deduplicationKey?: string;
+};
+
+export type AnyQueueItem = {
+  id: string;
+  job: string;
+  item: any;
+  visibilityTimeoutMs: number;
+  attempt: number;
+  timestamp: Date;
+  deduplicationKey?: string;
+};
+
+export class SimpleQueue<TMessageCatalog extends MessageCatalogSchema> {
+  name: string;
+  private redis: Redis;
+  private schema: TMessageCatalog;
+  private logger: Logger;
+
+  constructor({
+    name,
+    schema,
+    redisOptions,
+    logger,
+  }: {
+    name: string;
+    schema: TMessageCatalog;
+    redisOptions: RedisOptions;
+    logger?: Logger;
+  }) {
+    this.name = name;
+    this.logger = logger ?? new Logger("SimpleQueue", "debug");
+
+    this.redis = createRedisClient(
+      {
+        ...redisOptions,
+        keyPrefix: `${redisOptions.keyPrefix ?? ""}{queue:${name}:}`,
+        retryStrategy(times) {
+          const delay = Math.min(times * 50, 1000);
+          return delay;
+        },
+        maxRetriesPerRequest: 20,
+      },
+      {
+        onError: (error) => {
+          this.logger.error(`RedisWorker queue redis client error:`, {
+            error,
+            keyPrefix: redisOptions.keyPrefix,
+          });
+        },
+      }
+    );
+    this.#registerCommands();
+    this.schema = schema;
+  }
+
+  async cancel(cancellationKey: string): Promise<void> {
+    await this.redis.set(`cancellationKey:${cancellationKey}`, "1", "EX", 60 * 60 * 24); // 1 day
+  }
+
+  async enqueue({
+    id,
+    job,
+    item,
+    attempt,
+    availableAt,
+    visibilityTimeoutMs,
+    cancellationKey,
+  }: {
+    id?: string;
+    job: MessageCatalogKey<TMessageCatalog>;
+    item: MessageCatalogValue<TMessageCatalog, MessageCatalogKey<TMessageCatalog>>;
+    attempt?: number;
+    availableAt?: Date;
+    visibilityTimeoutMs: number;
+    cancellationKey?: string;
+  }): Promise<void> {
+    try {
+      const score = availableAt ? availableAt.getTime() : Date.now();
+      const deduplicationKey = nanoid();
+      const serializedItem = JSON.stringify({
+        job,
+        item,
+        visibilityTimeoutMs,
+        attempt,
+        deduplicationKey,
+      });
+
+      const result = cancellationKey
+        ? await this.redis.enqueueItemWithCancellationKey(
+            `queue`,
+            `items`,
+            `cancellationKey:${cancellationKey}`,
+            id ?? nanoid(),
+            score,
+            serializedItem
+          )
+        : await this.redis.enqueueItem(`queue`, `items`, id ?? nanoid(), score, serializedItem);
+
+      if (result !== 1) {
+        throw new Error("Enqueue operation failed");
+      }
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.enqueue(): error enqueuing`, {
+        queue: this.name,
+        error: e,
+        id,
+        item,
+      });
+      throw e;
+    }
+  }
+
+  async enqueueOnce({
+    id,
+    job,
+    item,
+    attempt,
+    availableAt,
+    visibilityTimeoutMs,
+  }: {
+    id: string;
+    job: MessageCatalogKey<TMessageCatalog>;
+    item: MessageCatalogValue<TMessageCatalog, MessageCatalogKey<TMessageCatalog>>;
+    attempt?: number;
+    availableAt?: Date;
+    visibilityTimeoutMs: number;
+  }): Promise<boolean> {
+    if (!id) {
+      throw new Error("enqueueOnce requires an id");
+    }
+    try {
+      const score = availableAt ? availableAt.getTime() : Date.now();
+      const deduplicationKey = nanoid();
+      const serializedItem = JSON.stringify({
+        job,
+        item,
+        visibilityTimeoutMs,
+        attempt,
+        deduplicationKey,
+      });
+      const result = await this.redis.enqueueItemOnce(`queue`, `items`, id, score, serializedItem);
+      // 1 if inserted, 0 if already exists
+      return result === 1;
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.enqueueOnce(): error enqueuing`, {
+        queue: this.name,
+        error: e,
+        id,
+        item,
+      });
+      throw e;
+    }
+  }
+
+  async dequeue(count: number = 1): Promise<Array<QueueItem<TMessageCatalog>>> {
+    const now = Date.now();
+
+    try {
+      const results = await this.redis.dequeueItems(`queue`, `items`, now, count);
+
+      if (!results || results.length === 0) {
+        return [];
+      }
+
+      const dequeuedItems: Array<QueueItem<TMessageCatalog>> = [];
+
+      for (const [id, serializedItem, score] of results) {
+        const parsedItem = JSON.parse(serializedItem) as any;
+        if (typeof parsedItem.job !== "string") {
+          this.logger.error(`Invalid item in queue`, { queue: this.name, id, item: parsedItem });
+          continue;
+        }
+
+        const timestamp = new Date(Number(score));
+
+        const schema = this.schema[parsedItem.job];
+
+        if (!schema) {
+          this.logger.error(`Invalid item in queue, schema not found`, {
+            queue: this.name,
+            id,
+            item: parsedItem,
+            job: parsedItem.job,
+            timestamp,
+            availableJobs: Object.keys(this.schema),
+          });
+          continue;
+        }
+
+        const validatedItem = schema.safeParse(parsedItem.item);
+
+        if (!validatedItem.success) {
+          this.logger.error("Invalid item in queue", {
+            queue: this.name,
+            id,
+            item: parsedItem,
+            errors: validatedItem.error,
+            attempt: parsedItem.attempt,
+            timestamp,
+          });
+          continue;
+        }
+
+        const visibilityTimeoutMs = parsedItem.visibilityTimeoutMs as number;
+
+        dequeuedItems.push({
+          id,
+          job: parsedItem.job,
+          item: validatedItem.data,
+          visibilityTimeoutMs,
+          attempt: parsedItem.attempt ?? 0,
+          timestamp,
+          deduplicationKey: parsedItem.deduplicationKey,
+        });
+      }
+
+      return dequeuedItems;
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.dequeue(): error dequeuing`, {
+        queue: this.name,
+        error: e,
+        count,
+      });
+      throw e;
+    }
+  }
+
+  async ack(id: string, deduplicationKey?: string): Promise<void> {
+    try {
+      const result = await this.redis.ackItem(`queue`, `items`, id, deduplicationKey ?? "");
+      if (result !== 1) {
+        this.logger.debug(
+          `SimpleQueue ${this.name}.ack(): ack operation returned ${result}. This means it was not removed from the queue.`,
+          {
+            queue: this.name,
+            id,
+            deduplicationKey,
+            result,
+          }
+        );
+      }
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.ack(): error acknowledging item`, {
+        queue: this.name,
+        error: e,
+        id,
+        deduplicationKey,
+      });
+      throw e;
+    }
+  }
+
+  async reschedule(id: string, availableAt: Date): Promise<void> {
+    await this.redis.zadd(`queue`, "XX", availableAt.getTime(), id);
+  }
+
+  async size({ includeFuture = false }: { includeFuture?: boolean } = {}): Promise<number> {
+    try {
+      if (includeFuture) {
+        // If includeFuture is true, return the total count of all items
+        return await this.redis.zcard(`queue`);
+      } else {
+        // If includeFuture is false, return the count of items available now
+        const now = Date.now();
+        return await this.redis.zcount(`queue`, "-inf", now);
+      }
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.size(): error getting queue size`, {
+        queue: this.name,
+        error: e,
+        includeFuture,
+      });
+      throw e;
+    }
+  }
+
+  /**
+   * Age (in ms) of the oldest *overdue* message — the oldest item whose scheduled
+   * time has already passed (score <= now). Returns 0 when the queue is empty or
+   * only holds future/delayed or in-flight (future-scored) items.
+   *
+   * Resolves the candidate against the `items` hash so orphaned `queue` entries
+   * (a member whose payload is missing — the same stale state `dequeueItems`
+   * cleans up) don't report a phantom stall for work that can't be dequeued. The
+   * Lua scans due items oldest-first and returns the first score whose payload
+   * still exists.
+   *
+   * This is the generic stall signal: it stays at 0 while a queue drains healthily
+   * and rises only when due work sits undrained (poison block, dead consumer,
+   * backpressure).
+   */
+  async oldestMessageAge(): Promise<number> {
+    try {
+      const now = Date.now();
+      // -1 sentinel = nothing due, or every due entry is orphaned.
+      const score = Number(await this.redis.getOldestDueScore(`queue`, `items`, now));
+
+      if (!Number.isFinite(score) || score < 0) {
+        return 0;
+      }
+
+      return Math.max(0, now - score);
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.oldestMessageAge(): error getting oldest age`, {
+        queue: this.name,
+        error: e,
+      });
+      // Swallow: a transient Redis error must not break observable metric collection.
+      return 0;
+    }
+  }
+
+  async getJob(id: string): Promise<QueueItem<TMessageCatalog> | null> {
+    const result = await this.redis.getJob(`queue`, `items`, id);
+
+    if (!result) {
+      return null;
+    }
+
+    const [_, score, serializedItem] = result;
+    const item = JSON.parse(serializedItem) as QueueItem<TMessageCatalog>;
+
+    return {
+      id,
+      job: item.job,
+      item: item.item,
+      visibilityTimeoutMs: item.visibilityTimeoutMs,
+      attempt: item.attempt ?? 0,
+      timestamp: new Date(Number(score)),
+      deduplicationKey: item.deduplicationKey ?? undefined,
+    };
+  }
+
+  async moveToDeadLetterQueue(id: string, errorMessage: string): Promise<void> {
+    try {
+      this.logger.debug(`SimpleQueue ${this.name}.moveToDeadLetterQueue(): moving item to DLQ`, {
+        queue: this.name,
+        id,
+        errorMessage,
+      });
+
+      const result = await this.redis.moveToDeadLetterQueue(
+        `queue`,
+        `items`,
+        `dlq`,
+        `dlq:items`,
+        id,
+        errorMessage
+      );
+
+      if (result !== 1) {
+        throw new Error("Move to Dead Letter Queue operation failed");
+      }
+    } catch (e) {
+      this.logger.error(
+        `SimpleQueue ${this.name}.moveToDeadLetterQueue(): error moving item to DLQ`,
+        {
+          queue: this.name,
+          error: e,
+          id,
+          errorMessage,
+        }
+      );
+      throw e;
+    }
+  }
+
+  async sizeOfDeadLetterQueue(): Promise<number> {
+    try {
+      return await this.redis.zcard(`dlq`);
+    } catch (e) {
+      this.logger.error(`SimpleQueue ${this.name}.dlqSize(): error getting DLQ size`, {
+        queue: this.name,
+        error: e,
+      });
+      throw e;
+    }
+  }
+
+  async redriveFromDeadLetterQueue(id: string): Promise<void> {
+    try {
+      const result = await this.redis.redriveFromDeadLetterQueue(
+        `queue`,
+        `items`,
+        `dlq`,
+        `dlq:items`,
+        id
+      );
+
+      if (result !== 1) {
+        throw new Error("Redrive from Dead Letter Queue operation failed");
+      }
+    } catch (e) {
+      this.logger.error(
+        `SimpleQueue ${this.name}.redriveFromDeadLetterQueue(): error redriving item from DLQ`,
+        {
+          queue: this.name,
+          error: e,
+          id,
+        }
+      );
+      throw e;
+    }
+  }
+
+  async close(): Promise<void> {
+    await this.redis.quit();
+  }
+
+  #registerCommands() {
+    this.redis.defineCommand("enqueueItem", {
+      numberOfKeys: 2,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local id = ARGV[1]
+        local score = ARGV[2]
+        local serializedItem = ARGV[3]
+
+        redis.call('ZADD', queue, score, id)
+        redis.call('HSET', items, id, serializedItem)
+
+        return 1
+      `,
+    });
+
+    this.redis.defineCommand("enqueueItemWithCancellationKey", {
+      numberOfKeys: 3,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local cancellationKey = KEYS[3]
+
+        local id = ARGV[1]
+        local score = ARGV[2]
+        local serializedItem = ARGV[3]
+
+        -- if the cancellation key exists, return 1
+        if redis.call('EXISTS', cancellationKey) == 1 then
+          return 1
+        end
+
+        redis.call('ZADD', queue, score, id)
+        redis.call('HSET', items, id, serializedItem)
+
+        return 1
+      `,
+    });
+
+    this.redis.defineCommand("dequeueItems", {
+      numberOfKeys: 2,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local now = tonumber(ARGV[1])
+        local count = tonumber(ARGV[2])
+
+        local result = redis.call('ZRANGEBYSCORE', queue, '-inf', now, 'WITHSCORES', 'LIMIT', 0, count)
+
+        if #result == 0 then
+          return {}
+        end
+
+        local dequeued = {}
+
+        for i = 1, #result, 2 do
+          local id = result[i]
+          local score = tonumber(result[i + 1])
+
+          if score > now then
+            break
+          end
+
+          local serializedItem = redis.call('HGET', items, id)
+
+          if serializedItem then
+            local item = cjson.decode(serializedItem)
+            local visibilityTimeoutMs = tonumber(item.visibilityTimeoutMs)
+            local invisibleUntil = now + visibilityTimeoutMs
+
+            redis.call('ZADD', queue, invisibleUntil, id)
+            table.insert(dequeued, {id, serializedItem, score})
+          else
+            -- Remove the orphaned queue entry if no corresponding item exists
+            redis.call('ZREM', queue, id)
+          end
+        end
+
+        return dequeued
+      `,
+    });
+
+    this.redis.defineCommand("getOldestDueScore", {
+      numberOfKeys: 2,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local now = tonumber(ARGV[1])
+
+        -- Oldest-first scan of due items, bounded so a long prefix of orphans can't
+        -- make this O(n). Orphans are rare (dequeueItems removes them), so in the
+        -- common case this returns on the first iteration. Read-only: unlike
+        -- dequeueItems we don't ZREM orphans here — a metric probe must not mutate.
+        local result = redis.call('ZRANGEBYSCORE', queue, '-inf', now, 'WITHSCORES', 'LIMIT', 0, 100)
+
+        for i = 1, #result, 2 do
+          local id = result[i]
+          if redis.call('HEXISTS', items, id) == 1 then
+            return result[i + 1]
+          end
+        end
+
+        return -1
+      `,
+    });
+
+    this.redis.defineCommand("getJob", {
+      numberOfKeys: 2,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local jobId = ARGV[1]
+
+        local serializedItem = redis.call('HGET', items, jobId)
+
+        if serializedItem == false then
+          return nil
+        end
+
+        -- get the score from the queue sorted set
+        local score = redis.call('ZSCORE', queue, jobId)
+
+        return { jobId, score, serializedItem }
+      `,
+    });
+
+    this.redis.defineCommand("ackItem", {
+      numberOfKeys: 2,
+      lua: `
+        local queueKey = KEYS[1]
+        local itemsKey = KEYS[2]
+        local id = ARGV[1]
+        local deduplicationKey = ARGV[2]
+
+        -- Get the item from the hash
+        local item = redis.call('HGET', itemsKey, id)
+        if not item then
+          return -1
+        end
+
+        -- Only check deduplicationKey if a non-empty one was passed in
+        if deduplicationKey and deduplicationKey ~= "" then
+          local success, parsed = pcall(cjson.decode, item)
+          if success then
+            if parsed.deduplicationKey and parsed.deduplicationKey ~= deduplicationKey then
+              return 0
+            end
+          end
+        end
+
+        -- Remove from sorted set and hash
+        redis.call('ZREM', queueKey, id)
+        redis.call('HDEL', itemsKey, id)
+        return 1
+      `,
+    });
+
+    this.redis.defineCommand("moveToDeadLetterQueue", {
+      numberOfKeys: 4,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local dlq = KEYS[3]
+        local dlqItems = KEYS[4]
+        local id = ARGV[1]
+        local errorMessage = ARGV[2]
+
+        local item = redis.call('HGET', items, id)
+        if not item then
+          return 0
+        end
+
+        local parsedItem = cjson.decode(item)
+        parsedItem.errorMessage = errorMessage
+
+        local time = redis.call('TIME')
+        local now = tonumber(time[1]) * 1000 + math.floor(tonumber(time[2]) / 1000)
+
+        redis.call('ZREM', queue, id)
+        redis.call('HDEL', items, id)
+
+        redis.call('ZADD', dlq, now, id)
+        redis.call('HSET', dlqItems, id, cjson.encode(parsedItem))
+
+        return 1
+      `,
+    });
+
+    this.redis.defineCommand("redriveFromDeadLetterQueue", {
+      numberOfKeys: 4,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local dlq = KEYS[3]
+        local dlqItems = KEYS[4]
+        local id = ARGV[1]
+
+        local item = redis.call('HGET', dlqItems, id)
+        if not item then
+          return 0
+        end
+
+        local parsedItem = cjson.decode(item)
+        parsedItem.errorMessage = nil
+
+        local time = redis.call('TIME')
+        local now = tonumber(time[1]) * 1000 + math.floor(tonumber(time[2]) / 1000)
+
+        redis.call('ZREM', dlq, id)
+        redis.call('HDEL', dlqItems, id)
+
+        redis.call('ZADD', queue, now, id)
+        redis.call('HSET', items, id, cjson.encode(parsedItem))
+
+        return 1
+      `,
+    });
+
+    this.redis.defineCommand("enqueueItemOnce", {
+      numberOfKeys: 2,
+      lua: `
+        local queue = KEYS[1]
+        local items = KEYS[2]
+        local id = ARGV[1]
+        local score = ARGV[2]
+        local serializedItem = ARGV[3]
+
+        -- Only add if not exists
+        local added = redis.call('HSETNX', items, id, serializedItem)
+        if added == 1 then
+          redis.call('ZADD', queue, 'NX', score, id)
+          return 1
+        else
+          return 0
+        end
+      `,
+    });
+  }
+}
+
+declare module "@internal/redis" {
+  interface RedisCommander<Context> {
+    enqueueItem(
+      //keys
+      queue: string,
+      items: string,
+      //args
+      id: string,
+      score: number,
+      serializedItem: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    enqueueItemWithCancellationKey(
+      //keys
+      queue: string,
+      items: string,
+      cancellationKey: string,
+      //args
+      id: string,
+      score: number,
+      serializedItem: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    dequeueItems(
+      //keys
+      queue: string,
+      items: string,
+      //args
+      now: number,
+      count: number,
+      callback?: Callback<Array<[string, string, string]>>
+    ): Result<Array<[string, string, string]>, Context>;
+
+    ackItem(
+      queue: string,
+      items: string,
+      id: string,
+      deduplicationKey: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    redriveFromDeadLetterQueue(
+      queue: string,
+      items: string,
+      dlq: string,
+      dlqItems: string,
+      id: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    moveToDeadLetterQueue(
+      queue: string,
+      items: string,
+      dlq: string,
+      dlqItems: string,
+      id: string,
+      errorMessage: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    enqueueItemOnce(
+      queue: string,
+      items: string,
+      id: string,
+      score: number,
+      serializedItem: string,
+      callback?: Callback<number>
+    ): Result<number, Context>;
+
+    getJob(
+      queue: string,
+      items: string,
+      id: string,
+      callback?: Callback<[string, string, string] | null>
+    ): Result<[string, string, string] | null, Context>;
+
+    getOldestDueScore(
+      //keys
+      queue: string,
+      items: string,
+      //args
+      now: number,
+      callback?: Callback<string | number>
+    ): Result<string | number, Context>;
+  }
+}

packages/redis-worker/src/utils.ts

@@ -0,0 +1,10 @@
+/**
+ * Check if an error is an AbortError.
+ *
+ * This handles both:
+ * - Custom abort errors created with `new Error("AbortError")` (sets .message)
+ * - Native Node.js AbortError from timers/promises (sets .name)
+ */
+export function isAbortError(error: unknown): boolean {
+  return error instanceof Error && (error.name === "AbortError" || error.message === "AbortError");
+}

packages/redis-worker/src/worker.test.ts

@@ -0,0 +1,630 @@
+import { isolatedRedisTest as redisTest } from "@internal/testcontainers";
+import { Logger } from "@trigger.dev/core/logger";
+import { describe } from "node:test";
+import { expect } from "vitest";
+import { z } from "zod";
+import { Worker } from "./worker.js";
+import { createRedisClient } from "@internal/redis";
+
+describe("Worker", () => {
+  redisTest("Process items that don't throw", { timeout: 30_000 }, async ({ redisContainer }) => {
+    const processedItems: number[] = [];
+    const worker = new Worker({
+      name: "test-worker",
+      redisOptions: {
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      },
+      catalog: {
+        testJob: {
+          schema: z.object({ value: z.number() }),
+          visibilityTimeoutMs: 5000,
+          retry: { maxAttempts: 3 },
+        },
+      },
+      jobs: {
+        testJob: async ({ payload }) => {
+          await new Promise((resolve) => setTimeout(resolve, 30)); // Simulate work
+          processedItems.push(payload.value);
+        },
+      },
+      concurrency: {
+        workers: 2,
+        tasksPerWorker: 3,
+      },
+      logger: new Logger("test", "log"),
+    }).start();
+
+    // Enqueue 10 items
+    for (let i = 0; i < 10; i++) {
+      await worker.enqueue({
+        id: `item-${i}`,
+        job: "testJob",
+        payload: { value: i },
+        visibilityTimeoutMs: 5000,
+      });
+    }
+
+    // Wait for items to be processed
+    await new Promise((resolve) => setTimeout(resolve, 2000));
+
+    expect(processedItems.length).toBe(10);
+    expect(new Set(processedItems).size).toBe(10); // Ensure all items were processed uniquely
+  });
+
+  redisTest(
+    "Process items that throw an error",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedItems: number[] = [];
+      const hadAttempt = new Set<string>();
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.number() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3, minTimeoutInMs: 10, maxTimeoutInMs: 10 },
+          },
+        },
+        jobs: {
+          testJob: async ({ id, payload }) => {
+            if (!hadAttempt.has(id)) {
+              hadAttempt.add(id);
+              throw new Error("Test error");
+            }
+
+            await new Promise((resolve) => setTimeout(resolve, 30)); // Simulate work
+            processedItems.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 2,
+          tasksPerWorker: 3,
+        },
+        pollIntervalMs: 50,
+        logger: new Logger("test", "error"),
+      }).start();
+
+      // Enqueue 10 items
+      for (let i = 0; i < 10; i++) {
+        await worker.enqueue({
+          id: `item-${i}`,
+          job: "testJob",
+          payload: { value: i },
+          visibilityTimeoutMs: 5000,
+        });
+      }
+
+      // Wait for items to be processed
+      await new Promise((resolve) => setTimeout(resolve, 500));
+
+      expect(processedItems.length).toBe(10);
+      expect(new Set(processedItems).size).toBe(10); // Ensure all items were processed uniquely
+    }
+  );
+
+  redisTest(
+    "Process an item that permanently fails and ends up in DLQ",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedItems: number[] = [];
+      const failedItemId = "permanent-fail-item";
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.number() }),
+            visibilityTimeoutMs: 1000,
+            retry: { maxAttempts: 3, minTimeoutInMs: 10, maxTimeoutInMs: 50 },
+          },
+        },
+        jobs: {
+          testJob: async ({ id, payload }) => {
+            if (id === failedItemId) {
+              throw new Error("Permanent failure");
+            }
+            processedItems.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 50,
+        logger: new Logger("test", "error"),
+      }).start();
+
+      // Enqueue the item that will permanently fail
+      await worker.enqueue({
+        id: failedItemId,
+        job: "testJob",
+        payload: { value: 999 },
+      });
+
+      // Enqueue a normal item
+      await worker.enqueue({
+        id: "normal-item",
+        job: "testJob",
+        payload: { value: 1 },
+      });
+
+      // Wait for items to be processed and retried
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Check that the normal item was processed
+      expect(processedItems).toEqual([1]);
+
+      // Check that the failed item is in the DLQ
+      const dlqSize = await worker.queue.sizeOfDeadLetterQueue();
+      expect(dlqSize).toBe(1);
+    }
+  );
+
+  redisTest(
+    "Redrive an item from DLQ and process it successfully",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedItems: number[] = [];
+      const failedItemId = "fail-then-redrive-item";
+      let attemptCount = 0;
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.number() }),
+            visibilityTimeoutMs: 1000,
+            retry: { maxAttempts: 3, minTimeoutInMs: 10, maxTimeoutInMs: 50 },
+          },
+        },
+        jobs: {
+          testJob: async ({ id, payload }) => {
+            if (id === failedItemId && attemptCount < 3) {
+              attemptCount++;
+              throw new Error("Temporary failure");
+            }
+            processedItems.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 50,
+        logger: new Logger("test", "error"),
+      }).start();
+
+      // Enqueue the item that will fail 3 times
+      await worker.enqueue({
+        id: failedItemId,
+        job: "testJob",
+        payload: { value: 999 },
+      });
+
+      // Wait for the item to be processed and moved to DLQ
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Check that the item is in the DLQ
+      let dlqSize = await worker.queue.sizeOfDeadLetterQueue();
+      expect(dlqSize).toBe(1);
+
+      // Create a Redis client to publish the redrive message
+      const redisClient = createRedisClient({
+        host: redisContainer.getHost(),
+        port: redisContainer.getPort(),
+        password: redisContainer.getPassword(),
+      });
+
+      // Publish redrive message
+      await redisClient.publish("test-worker:redrive", JSON.stringify({ id: failedItemId }));
+
+      // Wait for the item to be redrived and processed
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Check that the item was processed successfully
+      expect(processedItems).toEqual([999]);
+
+      // Check that the DLQ is now empty
+      dlqSize = await worker.queue.sizeOfDeadLetterQueue();
+      expect(dlqSize).toBe(0);
+
+      await redisClient.quit();
+    }
+  );
+
+  redisTest(
+    "Should process a job with the same ID only once when rescheduled",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedPayloads: string[] = [];
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.string() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+          },
+        },
+        jobs: {
+          testJob: async ({ payload }) => {
+            await new Promise((resolve) => setTimeout(resolve, 30)); // Simulate work
+            processedPayloads.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 10, // Ensure quick polling to detect the scheduled item
+        logger: new Logger("test", "log"),
+      }).start();
+
+      // Unique ID to use for both enqueues
+      const testJobId = "duplicate-job-id";
+
+      // Enqueue the first item immediately
+      await worker.enqueue({
+        id: testJobId,
+        job: "testJob",
+        payload: { value: "first-attempt" },
+        availableAt: new Date(Date.now() + 50),
+      });
+
+      // Enqueue another item with the same ID but scheduled 50ms in the future
+      await worker.enqueue({
+        id: testJobId,
+        job: "testJob",
+        payload: { value: "second-attempt" },
+        availableAt: new Date(Date.now() + 50),
+      });
+
+      // Wait enough time for both jobs to be processed if they were going to be
+      await new Promise((resolve) => setTimeout(resolve, 300));
+
+      // Verify that only one job was processed (the second one should have replaced the first)
+      expect(processedPayloads.length).toBe(1);
+
+      // Verify that the second job's payload was the one processed
+      expect(processedPayloads[0]).toBe("second-attempt");
+
+      await worker.stop();
+    }
+  );
+
+  redisTest(
+    "Should process second job with same ID when enqueued during first job execution with future availableAt",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedPayloads: string[] = [];
+      const jobStarted: string[] = [];
+      let firstJobCompleted = false;
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.string() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+          },
+        },
+        jobs: {
+          testJob: async ({ payload }) => {
+            // Record when the job starts processing
+            jobStarted.push(payload.value);
+
+            if (payload.value === "first-attempt") {
+              // First job takes a long time to process
+              await new Promise((resolve) => setTimeout(resolve, 1_000));
+              firstJobCompleted = true;
+            }
+
+            // Record when the job completes
+            processedPayloads.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 10,
+        logger: new Logger("test", "log"),
+      }).start();
+
+      const testJobId = "long-running-job-id";
+
+      // Queue the first job
+      await worker.enqueue({
+        id: testJobId,
+        job: "testJob",
+        payload: { value: "first-attempt" },
+      });
+
+      // Verify initial queue size
+      const size1 = await worker.queue.size({ includeFuture: true });
+      expect(size1).toBe(1);
+
+      // Wait until we know the first job has started processing
+      while (jobStarted.length === 0) {
+        await new Promise((resolve) => setTimeout(resolve, 10));
+      }
+
+      // Now that first job is running, queue second job with same ID
+      // Set availableAt to be 1.5 seconds in the future (after first job completes)
+      await worker.enqueue({
+        id: testJobId,
+        job: "testJob",
+        payload: { value: "second-attempt" },
+        availableAt: new Date(Date.now() + 1500),
+      });
+
+      // Verify queue size after second enqueue
+      const size2 = await worker.queue.size({ includeFuture: true });
+      const _size2Present = await worker.queue.size({ includeFuture: false });
+      expect(size2).toBe(1); // Should still be 1 as it's the same ID
+
+      // Wait for the first job to complete
+      while (!firstJobCompleted) {
+        await new Promise((resolve) => setTimeout(resolve, 10));
+      }
+
+      // Check queue size right after first job completes
+      const _size3 = await worker.queue.size({ includeFuture: true });
+      const _size3Present = await worker.queue.size({ includeFuture: false });
+
+      // Wait long enough for the second job to become available and potentially run
+      await new Promise((resolve) => setTimeout(resolve, 2000));
+
+      // Final queue size
+      const _size4 = await worker.queue.size({ includeFuture: true });
+      const _size4Present = await worker.queue.size({ includeFuture: false });
+
+      // First job should have run
+      expect(processedPayloads).toContain("first-attempt");
+
+      // These assertions should fail - demonstrating the bug
+      // The second job should run after its availableAt time, but doesn't because
+      // the ack from the first job removed it from Redis entirely
+      expect(jobStarted).toContain("second-attempt");
+      expect(processedPayloads).toContain("second-attempt");
+      expect(processedPayloads.length).toBe(2);
+
+      await worker.stop();
+    }
+  );
+
+  redisTest(
+    "Should properly remove future-scheduled job after completion",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedPayloads: string[] = [];
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.string() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+          },
+        },
+        jobs: {
+          testJob: async ({ payload }) => {
+            processedPayloads.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 10,
+        logger: new Logger("test", "debug"), // Use debug to see all logs
+      }).start();
+
+      // Schedule a job 500ms in the future
+      await worker.enqueue({
+        id: "future-job",
+        job: "testJob",
+        payload: { value: "test" },
+        availableAt: new Date(Date.now() + 500),
+      });
+
+      // Verify it's in the future queue
+      const initialSize = await worker.queue.size();
+      const initialSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(initialSize).toBe(0);
+      expect(initialSizeWithFuture).toBe(1);
+
+      // Wait for job to be processed
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Verify job was processed
+      expect(processedPayloads).toContain("test");
+
+      // Verify queue is completely empty
+      const finalSize = await worker.queue.size();
+      const finalSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(finalSize).toBe(0);
+      expect(finalSizeWithFuture).toBe(0);
+
+      await worker.stop();
+    }
+  );
+
+  redisTest(
+    "Should properly remove immediate job after completion",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedPayloads: string[] = [];
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.string() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+          },
+        },
+        jobs: {
+          testJob: async ({ payload }) => {
+            processedPayloads.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 10,
+        logger: new Logger("test", "debug"), // Use debug to see all logs
+      }).start();
+
+      // Enqueue a job to run immediately
+      await worker.enqueue({
+        id: "immediate-job",
+        job: "testJob",
+        payload: { value: "test" },
+      });
+
+      // Verify it's in the present queue
+      const initialSize = await worker.queue.size();
+      const initialSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(initialSize).toBe(1);
+      expect(initialSizeWithFuture).toBe(1);
+
+      // Wait for job to be processed
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Verify job was processed
+      expect(processedPayloads).toContain("test");
+
+      // Verify queue is completely empty
+      const finalSize = await worker.queue.size();
+      const finalSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(finalSize).toBe(0);
+      expect(finalSizeWithFuture).toBe(0);
+
+      await worker.stop();
+    }
+  );
+
+  redisTest(
+    "Should allow cancelling a job before it's enqueued, but only if the enqueue.cancellationKey is provided",
+    { timeout: 30_000 },
+    async ({ redisContainer }) => {
+      const processedPayloads: string[] = [];
+
+      const worker = new Worker({
+        name: "test-worker",
+        redisOptions: {
+          host: redisContainer.getHost(),
+          port: redisContainer.getPort(),
+          password: redisContainer.getPassword(),
+        },
+        catalog: {
+          testJob: {
+            schema: z.object({ value: z.string() }),
+            visibilityTimeoutMs: 5000,
+            retry: { maxAttempts: 3 },
+          },
+        },
+        jobs: {
+          testJob: async ({ payload }) => {
+            processedPayloads.push(payload.value);
+          },
+        },
+        concurrency: {
+          workers: 1,
+          tasksPerWorker: 1,
+        },
+        pollIntervalMs: 10,
+        logger: new Logger("test", "debug"), // Use debug to see all logs
+      }).start();
+
+      // Enqueue a job to run immediately
+      await worker.enqueue({
+        id: "immediate-job",
+        job: "testJob",
+        payload: { value: "test" },
+        cancellationKey: "test-cancellation-key",
+      });
+
+      // Verify it's in the present queue
+      const initialSize = await worker.queue.size();
+      const initialSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(initialSize).toBe(1);
+      expect(initialSizeWithFuture).toBe(1);
+
+      // Wait for job to be processed
+      await new Promise((resolve) => setTimeout(resolve, 1000));
+
+      // Verify job was processed
+      expect(processedPayloads).toContain("test");
+
+      // Verify queue is completely empty
+      const finalSize = await worker.queue.size();
+      const finalSizeWithFuture = await worker.queue.size({ includeFuture: true });
+      expect(finalSize).toBe(0);
+      expect(finalSizeWithFuture).toBe(0);
+
+      // Now cancel a key
+      await worker.cancel("test-cancellation-key-2");
+
+      await worker.enqueue({
+        id: "immediate-job",
+        job: "testJob",
+        payload: { value: "test" },
+        cancellationKey: "test-cancellation-key-2",
+      });
+
+      // Verify it's not in the queue (since it's been cancelled)
+      const finalSize2 = await worker.queue.size();
+      expect(finalSize2).toBe(0);
+      const finalSize2WithFuture = await worker.queue.size({ includeFuture: true });
+      expect(finalSize2WithFuture).toBe(0);
+
+      await worker.stop();
+    }
+  );
+});

packages/redis-worker/tsconfig.build.json

@@ -0,0 +1,21 @@
+{
+  "include": ["src/**/*.ts"],
+  "exclude": ["src/**/*.test.ts"],
+  "compilerOptions": {
+    "composite": true,
+    "target": "ES2019",
+    "lib": ["ES2019", "DOM", "DOM.Iterable", "DOM.AsyncIterable"],
+    "outDir": "dist",
+    "module": "Node16",
+    "moduleResolution": "Node16",
+    "moduleDetection": "force",
+    "verbatimModuleSyntax": false,
+    "esModuleInterop": true,
+    "forceConsistentCasingInFileNames": true,
+    "isolatedModules": true,
+    "preserveWatchOutput": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "declaration": true
+  }
+}

packages/redis-worker/tsconfig.json

@@ -0,0 +1,11 @@
+{
+  "extends": "../../.configs/tsconfig.base.json",
+  "references": [
+    {
+      "path": "./tsconfig.src.json"
+    },
+    {
+      "path": "./tsconfig.test.json"
+    }
+  ]
+}

packages/redis-worker/tsconfig.src.json

@@ -0,0 +1,10 @@
+{
+  "extends": "./tsconfig.json",
+  "include": ["./src/**/*.ts"],
+  "compilerOptions": {
+    "isolatedDeclarations": false,
+    "composite": true,
+    "sourceMap": true,
+    "customConditions": ["@triggerdotdev/source"]
+  }
+}

packages/redis-worker/tsconfig.test.json

@@ -0,0 +1,11 @@
+{
+  "extends": "./tsconfig.json",
+  "include": ["./test/**/*.ts"],
+  "references": [{ "path": "./tsconfig.src.json" }],
+  "compilerOptions": {
+    "isolatedDeclarations": false,
+    "composite": true,
+    "sourceMap": true,
+    "types": ["vitest/globals"]
+  }
+}

packages/redis-worker/tsup.config.ts

@@ -0,0 +1,27 @@
+import { defineConfig } from "tsup";
+
+export default defineConfig({
+  entry: ["src/index.ts"],
+  format: ["cjs", "esm"],
+  dts: true,
+  splitting: false,
+  sourcemap: true,
+  clean: true,
+  treeshake: true,
+  bundle: true,
+  minify: false,
+  noExternal: [
+    // Always bundle internal packages
+    /^@internal/,
+    // Always bundle ESM-only packages
+    "nanoid",
+    "p-limit",
+  ],
+  banner: ({ format }) => {
+    if (format !== "esm") return;
+
+    return {
+      js: `import { createRequire } from 'module'; const require = createRequire(import.meta.url || process.cwd() + '/index.js');`,
+    };
+  },
+});

packages/redis-worker/vitest.config.ts

@@ -0,0 +1,13 @@
+import { defineConfig } from "vitest/config";
+import { DurationShardingSequencer } from "@internal/testcontainers/sequencer";
+
+export default defineConfig({
+  test: {
+    sequence: { sequencer: DurationShardingSequencer },
+    include: ["**/*.test.ts"],
+    globals: true,
+    // CI-only: absorbs timing races (real-clock waits vs worker poll interval) under shard CPU contention
+    retry: process.env.CI ? 2 : 0,
+    fileParallelism: false,
+  },
+});