chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,82 @@
|
||||
# REVIEW.md — Trigger.dev OSS
|
||||
|
||||
Repo-specific signal for anyone (human or agent) reviewing a PR in this codebase. Calibrates what counts as critical, what to always check, and what to skip.
|
||||
|
||||
## What makes a 🔴 Important finding here
|
||||
|
||||
Reserve 🔴 for things that would page someone or block a rollback. In this codebase, that means:
|
||||
|
||||
- **Rolling-deploy breakage.** Old and new versions of the webapp/supervisor run side-by-side during deploys. A change is broken if:
|
||||
- A Lua script's behavior changes for a given key set without versioning (rename the script with a behavior-descriptive suffix like `Tracked` rather than `V2` — both versions must coexist safely).
|
||||
- A Redis data shape used by both versions changes in place. New shapes need a new key namespace.
|
||||
- A migration is not backward-compatible with the prior image.
|
||||
- **Schema / migration safety.** Prisma migrations must be backward-compatible with the prior deploy. Adding NOT NULL without a default, dropping a column an old image still reads, renaming a column — all 🔴.
|
||||
- **ClickHouse migration ordering + idempotency.** Goose runs in strict mode in the deploy pipeline and refuses to apply a missing version below the current version — slotting a new file in below the latest already-applied version blocks the deploy. New ClickHouse migration files MUST use the next available number (`max(files in internal-packages/clickhouse/schema/) + 1`); if main has added migrations while you've been on a branch, renumber yours. DDL must also be idempotent (`ADD COLUMN IF NOT EXISTS`, `DROP COLUMN IF EXISTS`, `CREATE TABLE IF NOT EXISTS`, `ADD INDEX IF NOT EXISTS`) so a partial / `--allow-missing` apply elsewhere doesn't fail on retry. Either fault is 🔴 — both break test/prod deploys. Rules live in `internal-packages/clickhouse/CLAUDE.md`.
|
||||
- **Queue / concurrency correctness.** RunQueue, MarQS (V1, legacy), redis-worker — any change to enqueue / dequeue / locking semantics. Re-derive the invariant on paper before flagging or accepting.
|
||||
- **Missing index on a hot table.** New Prisma queries against `TaskRun`, `TaskRunExecutionSnapshot`, `JobRun`, `Project`, etc. must use an existing index. Check `internal-packages/database/prisma/schema.prisma` for the relevant `@@index` lines — don't guess and don't propose `EXPLAIN`.
|
||||
- **Recovery-path queries.** Any `TaskRun.findFirst` / `findMany` added to a schedule, run-recovery, or restart loop. Recovery fan-outs (Redis crash, restart storms) turn "rare indexed query" into a DB incident. 🔴 even if indexed.
|
||||
- **Aggregations on hot tables.** No `COUNT` / `GROUP BY` on `TaskRun` or other tables that can reach billions of rows. Use Redis or ClickHouse for counts.
|
||||
- **Prod Redis blast-radius.** New code paths that `SCAN` with broad patterns (`*foo*`) on prod-shaped Redis, or `EVAL` Lua with `SCAN` loops inside. Both are 🔴.
|
||||
- **`@trigger.dev/core` direct import** from anywhere outside the SDK package. Always import from `@trigger.dev/sdk`. Core direct imports are 🔴 — they break the public API contract.
|
||||
- **Heavy execute-deps imported into request-handler bundles.** Specifically `chat.handover` and similar split-bundle entry points must not transitively import the agent task's execute path. Watch for new imports added at module top-level of route files.
|
||||
- **V1 engine code modified in a "V2 only" PR.** The `apps/webapp/app/v3/` directory contains both. If the PR description says V2-only but it touches `triggerTaskV1`, `cancelTaskRunV1`, `MarQS`, etc. — 🔴.
|
||||
|
||||
## Performance (always review)
|
||||
|
||||
Every PR gets a performance pass — not just the ones that look perf-sensitive. For each new query or unit of work, weigh three things: (a) the size of the table it hits, (b) whether it sits on a hot path, (c) whether the data it walks can be deep or wide (run trees, batches). The 🔴 bullets above on indexes, recovery-path queries, aggregations, and Redis `SCAN` are part of this pass — the rest below extends it.
|
||||
|
||||
**Treat these tables as large — no scans, no `COUNT` / `GROUP BY`, no unbounded fetch:**
|
||||
|
||||
- **Postgres — the `TaskRun` family:** `TaskRun`, `TaskRunExecutionSnapshot`, `Waitpoint`, `BatchTaskRun` and their join tables. Assume billions of rows.
|
||||
- **ClickHouse — `task_events_v1` / `task_events_v2`.** Partitioned by `toDate(inserted_at)`; `ORDER BY (environment_id, toUnixTimestamp(start_time), trace_id)`. Note `span_id` / `parent_span_id` are NOT in the sort key — span-id lookups can't skip granules, only `environment_id` + a `start_time` window can.
|
||||
|
||||
**Hot paths — extra scrutiny on any added query or work:**
|
||||
|
||||
- **Trigger + batch trigger** (`triggerTask.server.ts`, `batchTriggerV3.server.ts`) — see `apps/webapp/CLAUDE.md`; do not add DB queries to these.
|
||||
- **Dequeue / RunQueue** (`dequeueSystem.ts`, run-queue read/lock paths) — runs on every execution.
|
||||
- **Execution-snapshot creation in the run engine** — any engine function that writes a `TaskRunExecutionSnapshot` runs per state transition; a new query there multiplies by run volume.
|
||||
- **OTEL ingestion** (`otel.v1.traces.ts`, `otel.v1.logs.ts`) — write volume scales with customer span counts.
|
||||
- **Trace + run-list reads** (trace view, run list, span detail) — read paths over the large tables above.
|
||||
|
||||
**Deep / wide shapes — one run can explode into a huge tree or batch; code that walks them is the trap:**
|
||||
|
||||
- Trace span subtrees (deeply nested child runs → deep span trees).
|
||||
- Batch + parent/child fan-out (one run triggers thousands of children).
|
||||
- Waitpoint / run-dependency chains.
|
||||
- Tag / attribute many-to-many joins against the run/event tables.
|
||||
|
||||
**Anti-patterns (severity):**
|
||||
|
||||
- **Per-level fan-out that re-scans a large table once per tree depth** → 🔴. A BFS issuing one query per level (e.g. `parent_span_id IN {thisLevel}`) re-reads the same granules D times for a depth-D tree. Prefer one windowed query + an in-memory tree build.
|
||||
- **Dropping the partition-pruning predicate** — `inserted_at` for ClickHouse, the `createdAt` window for partitioned Postgres — to "widen" a lookup → 🔴. Without it the query scans every partition. Keep a bounded window even for ancestor / backfill lookups.
|
||||
- **Unbounded `IN (...)` built from a result set** (a BFS frontier, a batch's child ids) → 🟡. It can reach the row cap (`MAXIMUM_TRACE_SUMMARY_VIEW_COUNT` defaults to 25k). Cap or chunk to ≤1–2k ids per query.
|
||||
- **Sequential per-level round-trips** where one recursive or windowed query would do → 🟡. N levels = N round-trip latencies stacked.
|
||||
- **Replacing a single bounded query with a multi-query walk for _every_ call** (not just a rare fallback) → 🔴 on a hot read path, 🟡 elsewhere. Keep the cheap single-query path; branch into the expensive walk only when the cheap one comes up short.
|
||||
|
||||
## Always check
|
||||
|
||||
- **Tests use testcontainers, not mocks.** Vitest with `redisTest` / `postgresTest` / `containerTest` from `@internal/testcontainers`. Any new `vi.mock(...)` on Redis, Postgres, BullMQ, or other infra is wrong here — 🔴 if added in production-path tests, 🟡 if isolated unit test.
|
||||
- **Public-package changes have a changeset.** `pnpm run changeset:add` produces `.changeset/*.md`. Required for any edit under `packages/*`. Missing → 🟡; missing on a breaking change → 🔴.
|
||||
- **Server-only changes have `.server-changes/*.md`.** Required for `apps/webapp/`, `apps/supervisor/` edits with no public-package change. Body should be 1-2 sentences (it has to fit as one bullet in a future changelog). Missing → 🟡.
|
||||
- **Lua script naming.** Coexisting scripts use behavior-descriptive suffixes (`Tracked`), never `V2`. Old name must keep working until the next deploy clears it.
|
||||
- **RunQueue payload shape.** V2 run-queue payload's `projectId` is consumed by `workerQueueResolver` for override matching. If a PR drops it from the payload, 🔴.
|
||||
- **`safeSend` scope.** Defensive IPC wrappers belong on loop / interval / handler contexts, not one-shot terminal sends. If the PR adds `safeSend` to a single terminal call for consistency, 🟡 with a "remove this" suggestion.
|
||||
- **Zod version.** Pinned to `3.25.76` monorepo-wide. New package adding zod with a different version or range — 🔴.
|
||||
|
||||
## Skip (do NOT flag)
|
||||
|
||||
- Anything oxfmt / oxlint catches. CI enforces both via the `code-quality` check.
|
||||
- TypeScript style preferences (`type` vs `interface`) — already covered by repo standards.
|
||||
- Test coverage exhortations as a generic suggestion. Only flag missing tests when a specific code path is genuinely untested and the path has prior incidents.
|
||||
- `agentcrumbs` markers (`// @crumbs`, `// #region @crumbs`) and `agentcrumbs` imports — these are temporary debug instrumentation stripped before merge.
|
||||
- `// removed comments for removed code`, renamed `_unused` vars, re-exported types as "backwards compatibility shims" — also covered by repo standards.
|
||||
- Suggestions to "add error handling" without naming a specific scenario that breaks.
|
||||
- Documentation prose nitpicks in `docs/*` MDX files unless factually wrong.
|
||||
|
||||
## Things V1/legacy that should NOT block a PR
|
||||
|
||||
The `apps/webapp/app/v3/` directory name is misleading — most code there is V2. Only specific files are V1-only legacy: `MarQS` queue, `triggerTaskV1`, `cancelTaskRunV1`, and a handful of others (see `apps/webapp/CLAUDE.md` for the exact list). Don't flag "you should refactor this to use V2" on those — they're frozen.
|
||||
|
||||
## Confidence calibration for this repo
|
||||
|
||||
The most common false-positive pattern: speculating about race conditions in code paths the agent doesn't have runtime visibility into. If the only evidence is "this *could* race", drop it. If you can point to a specific interleaving with file:line for each step, surface it.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
paths:
|
||||
- "internal-packages/database/**"
|
||||
---
|
||||
|
||||
# Database Migration Safety
|
||||
|
||||
- When adding indexes to **existing tables**, use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` to avoid table locks. These must be in their own separate migration file (one index per file).
|
||||
- Indexes on **newly created tables** (same migration as `CREATE TABLE`) do not need CONCURRENTLY.
|
||||
- When indexing a **new column on an existing table**, split into two migrations: first `ADD COLUMN IF NOT EXISTS`, then `CREATE INDEX CONCURRENTLY IF NOT EXISTS` in a separate file.
|
||||
- After generating a migration with Prisma, remove extraneous lines for: `_BackgroundWorkerToBackgroundWorkerFile`, `_BackgroundWorkerToTaskQueue`, `_TaskRunToTaskRunTag`, `_WaitpointRunConnections`, `_completedWaitpoints`, `SecretStore_key_idx`, and unrelated TaskRun indexes.
|
||||
- Never drop columns or tables without explicit approval.
|
||||
- New code should target `RunEngineVersion.V2` only.
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
paths:
|
||||
- "docs/**"
|
||||
---
|
||||
|
||||
# Documentation Writing Rules
|
||||
|
||||
- Use Mintlify MDX format. Frontmatter: `title`, `description`, `sidebarTitle` (optional).
|
||||
- After creating a new page, add it to `docs.json` navigation under the correct group.
|
||||
- Use Mintlify components: `<Note>`, `<Warning>`, `<Info>`, `<Tip>`, `<CodeGroup>`, `<Expandable>`, `<Steps>`/`<Step>`.
|
||||
- Code examples should be complete and runnable where possible.
|
||||
- Always import from `@trigger.dev/sdk`, never `@trigger.dev/sdk/v3`.
|
||||
- Keep paragraphs short. Use headers to break up content.
|
||||
- Link to related pages using relative paths (e.g., `[Tasks](/tasks/overview)`).
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/webapp/app/v3/**"
|
||||
---
|
||||
|
||||
# Legacy V1 Engine Code in `app/v3/`
|
||||
|
||||
The `v3/` directory name is misleading - most code here is actively used by the current V2 engine. Only the specific files below are legacy V1-only code.
|
||||
|
||||
## V1-Only Files - Never Modify
|
||||
|
||||
- `marqs/` directory (entire MarQS queue system: sharedQueueConsumer, devQueueConsumer, fairDequeuingStrategy, devPubSub)
|
||||
- `legacyRunEngineWorker.server.ts` (V1 background job worker)
|
||||
- `services/triggerTaskV1.server.ts` (deprecated V1 task triggering)
|
||||
- `services/cancelTaskRunV1.server.ts` (deprecated V1 cancellation)
|
||||
- `authenticatedSocketConnection.server.ts` (V1 dev WebSocket using DevQueueConsumer)
|
||||
- `sharedSocketConnection.ts` (V1 shared queue socket using SharedQueueConsumer)
|
||||
|
||||
## V1/V2 Branching Pattern
|
||||
|
||||
Some services act as routers that branch on `RunEngineVersion`:
|
||||
- `services/cancelTaskRun.server.ts` - calls V1 service or `engine.cancelRun()` for V2
|
||||
- `services/batchTriggerV3.server.ts` - uses marqs for V1 path, run-engine for V2
|
||||
|
||||
When editing these shared services, only modify V2 code paths.
|
||||
|
||||
## V2 Modern Stack
|
||||
|
||||
- **Run lifecycle**: `@internal/run-engine` (internal-packages/run-engine)
|
||||
- **Background jobs**: `@trigger.dev/redis-worker` (not graphile-worker/zodworker)
|
||||
- **Queue operations**: RunQueue inside run-engine (not MarQS)
|
||||
- **V2 engine singleton**: `runEngine.server.ts`, `runEngineHandlers.server.ts`
|
||||
- **V2 workers**: `commonWorker.server.ts`, `alertsWorker.server.ts`, `batchTriggerWorker.server.ts`
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
paths:
|
||||
- "**/package.json"
|
||||
---
|
||||
|
||||
# Installing Packages
|
||||
|
||||
When adding a new dependency to any package.json in the monorepo:
|
||||
|
||||
1. **Look up the latest version** on npm before adding:
|
||||
```bash
|
||||
pnpm view <package-name> version
|
||||
```
|
||||
If unsure which version to use (e.g. major version compatibility), confirm with the user.
|
||||
|
||||
2. **Edit the package.json directly** — do NOT use `pnpm add` as it can cause issues in the monorepo. Add the dependency with the correct version range (typically `^x.y.z`).
|
||||
|
||||
3. **Run `pnpm i` from the repo root** after editing to install and update the lockfile:
|
||||
```bash
|
||||
pnpm i
|
||||
```
|
||||
Always run from the repo root, not from the package directory.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
paths:
|
||||
- "packages/**"
|
||||
---
|
||||
|
||||
# Public Package Rules
|
||||
|
||||
- Changes to `packages/` are **customer-facing**. Always add a changeset: `pnpm run changeset:add`
|
||||
- Default to **patch**. Get maintainer approval for minor. Never select major without explicit approval.
|
||||
- `@trigger.dev/core`: **Never import the root**. Always use subpath imports (e.g., `@trigger.dev/core/v3`).
|
||||
- Do NOT update `rules/` or `.claude/skills/trigger-dev-tasks/` unless explicitly asked. These are maintained in separate dedicated passes.
|
||||
- Test changes using the `hello-world` project in the [`triggerdotdev/references`](https://github.com/triggerdotdev/references) repo.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
paths:
|
||||
- "apps/**"
|
||||
---
|
||||
|
||||
# Server App Changes
|
||||
|
||||
When modifying server apps (webapp, supervisor, etc.) with **no package changes**, add a `.server-changes/` file instead of a changeset:
|
||||
|
||||
```bash
|
||||
cat > .server-changes/descriptive-name.md << 'EOF'
|
||||
---
|
||||
area: webapp
|
||||
type: fix
|
||||
---
|
||||
|
||||
Brief description of what changed and why.
|
||||
EOF
|
||||
```
|
||||
|
||||
- **area**: `webapp` | `supervisor`
|
||||
- **type**: `feature` | `fix` | `improvement` | `breaking`
|
||||
- If the PR also touches `packages/`, just the changeset is sufficient (no `.server-changes/` needed).
|
||||
@@ -0,0 +1,191 @@
|
||||
---
|
||||
name: drizzle
|
||||
description: Use this skill when writing or modifying Drizzle ORM schemas, queries, or migrations in this repo — specifically the `@internal/dashboard-agent-db` package (the dashboard agent's conversation datastore). Covers pg-core schema definition, the postgres-js driver, drizzle-kit migrations, and this repo's conventions: a dedicated Postgres schema, foreign-key-free cross-database design, pooler-safe connections, and the access-pattern query layer. Drizzle is NOT the main database — that's Prisma.
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# Drizzle ORM (this repo)
|
||||
|
||||
Drizzle is used in exactly one place: **`internal-packages/dashboard-agent-db`** (`@internal/dashboard-agent-db`), the in-dashboard agent's conversation store. Everything else in the monorepo is **Prisma** (`@trigger.dev/database`). Keep them separate.
|
||||
|
||||
Pinned versions: **`drizzle-orm` ^0.45**, **`drizzle-kit` ^0.31** (dev), **`postgres` ^3.4** (postgres.js driver). drizzle-orm and drizzle-kit are intentionally on different version lines — 0.31.x is the correct companion for 0.45.x, there is no peer dependency between them.
|
||||
|
||||
## Critical rules
|
||||
|
||||
1. **Drizzle is only the agent's own datastore.** The agent (and its task bundle) must have **no access to the main Prisma database or ClickHouse**. Never import the Prisma client into the agent task or into `@internal/dashboard-agent-db`. Main data is reached via the API, not Drizzle.
|
||||
2. **Foreign-key-free.** In cloud this DB is a *separate* PlanetScale database, so it can't FK into the main DB. Reference main entities (`organizationId`, `userId`, …) **by id only — never `.references()`**. Joins happen in app code; tenant scoping is enforced in the query layer.
|
||||
3. **One dedicated Postgres schema.** All tables live under `pgSchema("trigger_dashboard_agent")` so they're schema-qualified and isolated from Prisma's `public` schema (this is what makes the OSS single-database fallback safe).
|
||||
4. **Pooler-safe connections.** Connections go through a transaction-mode pooler (PlanetScale / PgBouncer-style), so postgres.js must run with **`prepare: false`** — prepared statements don't survive a connection being handed to another client between checkouts.
|
||||
5. **Node16 module resolution.** Relative imports need explicit **`.js`** extensions (`import { chats } from "./schema.js"`), even though the source is `.ts`.
|
||||
6. **Scope every user query.** All queries that touch user data go through `src/queries.ts` and are scoped by `organizationId` / `userId`, so callers can't forget the `where`. Don't write ad-hoc cross-tenant queries elsewhere.
|
||||
|
||||
## Package layout
|
||||
|
||||
```text
|
||||
internal-packages/dashboard-agent-db/
|
||||
drizzle.config.ts # drizzle-kit config (schema path, out dir, schemaFilter)
|
||||
drizzle/ # generated migrations (committed)
|
||||
src/
|
||||
schema.ts # pgSchema + table definitions
|
||||
client.ts # createDashboardAgentDb() — postgres.js + drizzle
|
||||
queries.ts # the access-pattern layer (org/user-scoped)
|
||||
index.ts # barrel: re-exports schema, client, queries
|
||||
```
|
||||
|
||||
`package.json` points `main`/`types` at `./src/index.ts` (consumed as source, no build step) — same as other simple internal packages.
|
||||
|
||||
## Schema (pg-core)
|
||||
|
||||
Use `pgSchema(...).table(...)`, not the bare `pgTable`, so tables land in the dedicated schema. ([schemas](https://orm.drizzle.team/docs/schemas), [pg column types](https://orm.drizzle.team/docs/column-types/pg), [indexes](https://orm.drizzle.team/docs/indexes-constraints))
|
||||
|
||||
```ts
|
||||
import { sql } from "drizzle-orm";
|
||||
import { index, jsonb, pgSchema, text, timestamp } from "drizzle-orm/pg-core";
|
||||
|
||||
export const dashboardAgentSchema = pgSchema("trigger_dashboard_agent");
|
||||
|
||||
export const chats = dashboardAgentSchema.table(
|
||||
"chats",
|
||||
{
|
||||
id: text("id").primaryKey(),
|
||||
organizationId: text("organization_id").notNull(), // FK-free: id only, no .references()
|
||||
userId: text("user_id").notNull(),
|
||||
title: text("title").notNull().default("New chat"),
|
||||
// JSONB with a typed view; .default([]) / .default({}) emit '[]'::jsonb / '{}'::jsonb
|
||||
messages: jsonb("messages").$type<unknown[]>().notNull().default([]),
|
||||
metadata: jsonb("metadata").$type<Record<string, unknown>>().notNull().default({}),
|
||||
deletedAt: timestamp("deleted_at", { withTimezone: true }), // soft delete
|
||||
lastMessageAt: timestamp("last_message_at", { withTimezone: true }),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
// Extra config returns an ARRAY in drizzle-orm 0.36+ (not an object).
|
||||
(t) => [
|
||||
// Partial + ordered composite index. `.desc()` on the column, `.where(sql`...`)` for partial.
|
||||
index("chats_org_user_last_msg_idx")
|
||||
.on(t.organizationId, t.userId, t.lastMessageAt.desc())
|
||||
.where(sql`${t.deletedAt} is null`),
|
||||
]
|
||||
);
|
||||
|
||||
// Inferred row types for the query layer + consumers.
|
||||
export type Chat = typeof chats.$inferSelect;
|
||||
export type NewChat = typeof chats.$inferInsert;
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `timestamp(..., { withTimezone: true })` → `timestamp with time zone`. Use `.defaultNow()` for `DEFAULT now()`.
|
||||
- For a "newest first, nulls last" sort the partial index uses `.desc()`; the *query* uses raw `sql` for `NULLS LAST` (see below).
|
||||
- Don't add `.references()` — see critical rule 2.
|
||||
|
||||
## Client (postgres.js + drizzle)
|
||||
|
||||
([connect overview](https://orm.drizzle.team/docs/connect-overview)) One small pool, `prepare: false`. In the agent task create it once in `onBoot` (per-process); in the webapp wrap it in the `singleton(...)` helper.
|
||||
|
||||
```ts
|
||||
import { drizzle, type PostgresJsDatabase } from "drizzle-orm/postgres-js";
|
||||
import postgres, { type Sql } from "postgres";
|
||||
import * as schema from "./schema.js";
|
||||
|
||||
export type DashboardAgentDb = PostgresJsDatabase<typeof schema>;
|
||||
|
||||
export function createDashboardAgentDb(connectionString: string, opts: { max?: number } = {}) {
|
||||
const sql: Sql = postgres(connectionString, {
|
||||
max: opts.max ?? 5, // small — the pooler does the real pooling
|
||||
idle_timeout: 20, // release conns when an agent run suspends
|
||||
prepare: false, // REQUIRED for transaction-mode poolers
|
||||
});
|
||||
return { db: drizzle(sql, { schema }), sql, close: () => sql.end() };
|
||||
}
|
||||
```
|
||||
|
||||
## Queries (the access-pattern layer)
|
||||
|
||||
([select](https://orm.drizzle.team/docs/select), [insert](https://orm.drizzle.team/docs/insert), [operators](https://orm.drizzle.team/docs/operators), [transactions](https://orm.drizzle.team/docs/transactions), [joins](https://orm.drizzle.team/docs/joins))
|
||||
|
||||
```ts
|
||||
import { and, desc, eq, isNull, sql } from "drizzle-orm";
|
||||
|
||||
// Select EXPLICIT columns for list views — never select a large blob (messages)
|
||||
// or a secret (tokens) you don't need. `NULLS LAST` needs raw sql in orderBy.
|
||||
await db
|
||||
.select({ id: chats.id, title: chats.title, lastMessageAt: chats.lastMessageAt })
|
||||
.from(chats)
|
||||
.where(and(eq(chats.organizationId, orgId), eq(chats.userId, userId), isNull(chats.deletedAt)))
|
||||
.orderBy(sql`${chats.pinnedAt} desc nulls last`, desc(chats.lastMessageAt))
|
||||
.limit(50);
|
||||
|
||||
// Idempotent create (avoids a duplicate-key race between two writers).
|
||||
await db.insert(chats).values({ id, organizationId: orgId, userId }).onConflictDoNothing();
|
||||
|
||||
// Upsert.
|
||||
await db
|
||||
.insert(chatSessions)
|
||||
.values({ chatId, publicAccessToken })
|
||||
.onConflictDoUpdate({ target: chatSessions.chatId, set: { publicAccessToken, updatedAt: sql`now()` } });
|
||||
|
||||
// Owner-scope a join (this DB is FK-free, so enforce ownership in the query).
|
||||
await db
|
||||
.select({ /* session cols */ })
|
||||
.from(chatSessions)
|
||||
.innerJoin(chats, eq(chats.id, chatSessions.chatId))
|
||||
.where(and(eq(chatSessions.chatId, chatId), eq(chats.userId, userId)));
|
||||
|
||||
// Multi-write that must be consistent on the next read → one transaction.
|
||||
await db.transaction(async (tx) => {
|
||||
await tx.update(chats).set({ messages, updatedAt: sql`now()` }).where(eq(chats.id, chatId));
|
||||
await tx.insert(chatSessions).values({ /* ... */ }).onConflictDoUpdate({ /* ... */ });
|
||||
});
|
||||
```
|
||||
|
||||
Use `sql\`now()\`` for DB-side timestamps in updates.
|
||||
|
||||
## Migrations (drizzle-kit)
|
||||
|
||||
([kit overview](https://orm.drizzle.team/docs/kit-overview), [generate](https://orm.drizzle.team/docs/drizzle-kit-generate), [migrate](https://orm.drizzle.team/docs/drizzle-kit-migrate))
|
||||
|
||||
`drizzle.config.ts` must set **`schemaFilter`** so drizzle-kit only ever manages our schema — never Prisma's `public` (critical in the OSS single-DB fallback):
|
||||
|
||||
```ts
|
||||
import { defineConfig } from "drizzle-kit";
|
||||
export default defineConfig({
|
||||
schema: "./src/schema.ts",
|
||||
out: "./drizzle",
|
||||
dialect: "postgresql",
|
||||
schemaFilter: ["trigger_dashboard_agent"],
|
||||
dbCredentials: { url: process.env.DASHBOARD_AGENT_DATABASE_URL ?? process.env.DATABASE_URL ?? "postgres://placeholder" },
|
||||
});
|
||||
```
|
||||
|
||||
Workflow:
|
||||
|
||||
```bash
|
||||
cd internal-packages/dashboard-agent-db
|
||||
pnpm run db:generate # diff schema.ts → emit SQL into drizzle/. OFFLINE (no DB needed).
|
||||
# review the generated drizzle/000N_*.sql before committing
|
||||
pnpm run db:migrate # apply pending migrations. Needs a real DATABASE URL.
|
||||
```
|
||||
|
||||
- `db:generate` is **offline** — it only reads `schema.ts`, so you can verify a schema change compiles to valid DDL with no database. Use it as a fast check.
|
||||
- drizzle-kit names migration files with a **random suffix** (`0000_magenta_lilandra.sql`). Don't regenerate a committed migration just to "refresh" it — that churns the filename. After the first migration is committed, schema changes produce a **new** `000N_*.sql`; commit that.
|
||||
- Generated DDL for a new schema is one `CREATE SCHEMA` + schema-qualified `CREATE TABLE`s + indexes, **no foreign keys** (by design here).
|
||||
|
||||
## Common gotchas
|
||||
|
||||
- **`prepare: false`** is not optional with a pooler — without it you'll get prepared-statement errors under load.
|
||||
- **Missing `.js` extension** on a relative import → TS2835 under Node16 resolution.
|
||||
- **Extra-config callback returns an array** `(t) => [ ... ]` in drizzle-orm 0.36+. The old object form `(t) => ({ ... })` is deprecated.
|
||||
- **`NULLS LAST` / `NULLS FIRST`** aren't on the `desc()` helper — use raw `sql\`col desc nulls last\`` in `orderBy`.
|
||||
- **Don't `SELECT *` into list views** — explicitly pick columns so you never ship a megabyte `messages` blob or a session token to a list query.
|
||||
- **Adding a dependency**: edit `package.json`, then `pnpm i` from the repo root (never `pnpm add`). Mind the repo's `minimumReleaseAge` (3 days) — pin with a caret range and let pnpm resolve an old-enough version.
|
||||
|
||||
## Reference (official docs)
|
||||
|
||||
- Schema declaration — https://orm.drizzle.team/docs/sql-schema-declaration
|
||||
- PostgreSQL column types — https://orm.drizzle.team/docs/column-types/pg
|
||||
- Schemas (`pgSchema`) — https://orm.drizzle.team/docs/schemas
|
||||
- Indexes & constraints — https://orm.drizzle.team/docs/indexes-constraints
|
||||
- Connect (postgres-js) — https://orm.drizzle.team/docs/connect-overview
|
||||
- Select / Insert / Update / Delete — https://orm.drizzle.team/docs/select · /insert · /update · /delete
|
||||
- Joins / Operators — https://orm.drizzle.team/docs/joins · /operators
|
||||
- Transactions — https://orm.drizzle.team/docs/transactions
|
||||
- drizzle-kit (generate / migrate / push) — https://orm.drizzle.team/docs/kit-overview
|
||||
@@ -0,0 +1,199 @@
|
||||
---
|
||||
name: errors-api-e2e
|
||||
description: End-to-end smoke test for the public Errors HTTP API (error groups). Seeds failed runs into ClickHouse so the error materialized views populate, then drives the real endpoints against the running webapp — list (with filters + pagination), retrieve, resolve/ignore/unresolve, the `filter[error]` runs filter, user attribution via the `trigger.dev mint-token` -> JWT exchange, and the 401/403/404 negatives. Use for "smoke test the errors API", "test the errors API e2e", "prove the errors endpoints work", or to re-verify after changes.
|
||||
allowed-tools: Read, Bash
|
||||
---
|
||||
|
||||
# Errors API — end-to-end smoke test
|
||||
|
||||
Proves the public Errors API against the **running** webapp with real HTTP. No
|
||||
mocks. The error data plane is ClickHouse (`errors_v1` + `error_occurrences_v1`,
|
||||
both materialized-view-fed from `task_runs_v2`) plus Postgres `ErrorGroupState`
|
||||
for lifecycle status; this skill seeds straight into `task_runs_v2` and lets the
|
||||
MVs do the rest.
|
||||
|
||||
Code under test:
|
||||
- `apps/webapp/app/routes/api.v1.errors.ts` — `GET /api/v1/errors` (list).
|
||||
- `apps/webapp/app/routes/api.v1.errors.$errorId.ts` — `GET /api/v1/errors/:errorId` (detail).
|
||||
- `apps/webapp/app/routes/api.v1.errors.$errorId.{resolve,ignore,unresolve}.ts` — state actions.
|
||||
- `apps/webapp/app/presenters/v3/ApiErrorListPresenter.server.ts` / `ApiErrorGroupPresenter.server.ts`.
|
||||
- `apps/webapp/app/presenters/v3/ApiRunListPresenter.server.ts` — the `filter[error]` addition on `GET /api/v1/runs`.
|
||||
- `apps/webapp/app/v3/services/errorGroupActions.server.ts` — resolve/ignore/unresolve (nullable `userId`).
|
||||
- Attribution: `api.v1.projects.$projectRef.$env.jwt.ts` stamps `act:{sub}` for PAT **and** UAT exchanges; `@trigger.dev/rbac` surfaces `act.sub` through bearer auth; the action handlers read `authentication.actor?.sub`.
|
||||
|
||||
`errorId` is `error_<fingerprint>` (round-trips via `ErrorId` in `@trigger.dev/core/v3/isomorphic`).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Webapp running on http://localhost:3030 (`pnpm run dev --filter webapp`). Confirm `curl -s http://localhost:3030/healthcheck`.
|
||||
- DB seeded (`pnpm run db:seed`), and a local ClickHouse reachable at `CLICKHOUSE_URL` (the `pnpm run docker` stack).
|
||||
- The CLI built + logged in to localhost:3030 (`pnpm run build --filter trigger.dev`; profile `default` points at localhost:3030). Needed only for the attribution leg.
|
||||
|
||||
> Important wiring facts the seed relies on (verified):
|
||||
> - The MVs read the error type/message from `error.data.*`, so the seeded
|
||||
> `error` JSON column **must** be wrapped: `{"data": {"type": ..., "message": ..., "stack": ...}}`.
|
||||
> - The MVs only fire for failed statuses: `SYSTEM_FAILURE | CRASHED | INTERRUPTED | COMPLETED_WITH_ERRORS | TIMED_OUT`, and require a non-empty `error_fingerprint`.
|
||||
> - `GET /api/v1/runs` lists run **ids** from ClickHouse but **hydrates from Postgres** `TaskRun`. So the error-list/detail/action legs work from a ClickHouse-only seed, but the `filter[error]` leg needs a **paired** Postgres `TaskRun` row whose `id` equals the ClickHouse `run_id`.
|
||||
|
||||
Run everything from the repo root in one shell. Invoke the built CLI via a
|
||||
function (a `CLI="node …"` variable won't word-split under zsh):
|
||||
```bash
|
||||
cli() { node packages/cli-v3/dist/esm/index.js "$@"; }
|
||||
PROFILE=default
|
||||
```
|
||||
|
||||
## Setup — resolve a dev environment + connection strings
|
||||
|
||||
```bash
|
||||
cd apps/webapp
|
||||
CHURL=$(grep -E "^CLICKHOUSE_URL=" .env | head -1 | cut -d= -f2- | tr -d '"')
|
||||
DBURL=$(grep -E "^DATABASE_URL=" .env | head -1 | cut -d= -f2- | tr -d '"' | tr -d "'" | sed 's/?.*//')
|
||||
|
||||
# Pick the seeded hello-world dev env (proj_rrkpdguyagvsoktglnod). Adjust the
|
||||
# WHERE if you want a different project.
|
||||
read ENV ORG PROJ REF < <(psql "$DBURL" -t -A -F' ' -c "
|
||||
SELECT re.id, re.\"organizationId\", re.\"projectId\", p.\"externalRef\"
|
||||
FROM \"RuntimeEnvironment\" re
|
||||
JOIN \"Project\" p ON p.id = re.\"projectId\"
|
||||
WHERE re.slug='dev' AND p.\"externalRef\"='proj_rrkpdguyagvsoktglnod' LIMIT 1;")
|
||||
APIKEY=$(psql "$DBURL" -t -A -c "SELECT \"apiKey\" FROM \"RuntimeEnvironment\" WHERE id='$ENV';")
|
||||
cd ..
|
||||
H="Authorization: Bearer $APIKEY"
|
||||
B="http://localhost:3030"
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Seed two error groups (ClickHouse, MV-fed)
|
||||
|
||||
```bash
|
||||
RUN=$(node -e 'console.log(Date.now().toString(36))')
|
||||
TASK="errors-api-e2e-$RUN"; FP_A="fpA${RUN}"; FP_B="fpB${RUN}"
|
||||
ERRID_A="error_$FP_A"; ERRID_B="error_$FP_B"
|
||||
NOW_CH=$(node -e 'console.log(new Date().toISOString().replace("T"," ").replace("Z","").slice(0,23))')
|
||||
NOW_MS=$(node -e 'console.log(Date.now())')
|
||||
Q=$(python3 -c "import urllib.parse;print(urllib.parse.quote('INSERT INTO trigger_dev.task_runs_v2 FORMAT JSONEachRow'))")
|
||||
|
||||
mkrow() { # status fingerprint errorType message runId
|
||||
echo "{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$5\",\"friendly_id\":\"run_$5\",\"status\":\"$1\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"$3\",\"message\":\"$4\",\"stack\":\"at x (a.ts:1:1)\"}},\"error_fingerprint\":\"$2\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
|
||||
}
|
||||
ROWS="$(mkrow COMPLETED_WITH_ERRORS $FP_A AlphaBoom 'alpha boom happened' r_a1_$RUN)
|
||||
$(mkrow COMPLETED_WITH_ERRORS $FP_A AlphaBoom 'alpha boom happened' r_a2_$RUN)
|
||||
$(mkrow CRASHED $FP_B BetaCrash 'beta crash happened' r_b1_$RUN)"
|
||||
printf '%s' "$ROWS" | curl -s "$CHURL/?query=$Q" --data-binary @-
|
||||
|
||||
# Poll until both fingerprints appear in errors_v1 (the MV is near-instant locally).
|
||||
for i in $(seq 1 10); do
|
||||
N=$(curl -s "$CHURL" --data-binary "SELECT count() FROM (SELECT 1 FROM trigger_dev.errors_v1 WHERE environment_id='$ENV' AND error_fingerprint IN ('$FP_A','$FP_B') GROUP BY error_fingerprint)")
|
||||
[ "$N" = "2" ] && break; sleep 1
|
||||
done
|
||||
echo "seeded fingerprints in errors_v1: $N (want 2)"
|
||||
```
|
||||
PASS: `N = 2`. Alpha has 2 occurrences, beta 1.
|
||||
|
||||
### 2. List + filters + pagination
|
||||
|
||||
```bash
|
||||
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bperiod%5D=1d" -H "$H" \
|
||||
| python3 -c "import sys,json;d=json.load(sys.stdin);print('count',len(d['data']),[(e['id'],e['status'],e['count']) for e in d['data']])"
|
||||
```
|
||||
PASS: 2 groups, both `status=unresolved`, alpha `count=2`, beta `count=1`, ids `error_<fp>`.
|
||||
|
||||
Assert each filter narrows correctly (each should return the noted shape):
|
||||
```bash
|
||||
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=unresolved&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print('unresolved:',len(json.load(sys.stdin)['data']))" # 2
|
||||
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bsearch%5D=AlphaBoom&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print('search:',[e['errorType'] for e in json.load(sys.stdin)['data']])" # ['AlphaBoom']
|
||||
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bperiod%5D=1d&page%5Bsize%5D=1" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('page size 1:',len(d['data']),'next?',bool(d['pagination'].get('next')))" # 1 / True
|
||||
```
|
||||
PASS: `unresolved: 2`, `search: ['AlphaBoom']`, `page size 1: 1 / next? True`.
|
||||
|
||||
### 3. Retrieve detail
|
||||
|
||||
```bash
|
||||
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" \
|
||||
| python3 -c "import sys,json;d=json.load(sys.stdin);print(d['id'],d['errorType'],d['status'],d['count'],d['affectedVersions'],d['resolvedBy'])"
|
||||
```
|
||||
PASS: `error_<fpA> AlphaBoom unresolved 2 ['20240101.1'] None`.
|
||||
|
||||
### 4. Resolve / ignore / unresolve (env API key — `resolvedBy` null)
|
||||
|
||||
```bash
|
||||
st(){ python3 -c "import sys,json;d=json.load(sys.stdin);print('status',d['status'],'| resolvedInVersion',d['resolvedInVersion'],'| resolvedBy',d['resolvedBy'],'| ignoredUntil',bool(d['ignoredUntil']),'| reason',d['ignoredReason'])"; }
|
||||
|
||||
curl -s -X POST "$B/api/v1/errors/$ERRID_A/resolve" -H "$H" -H 'Content-Type: application/json' -d '{"resolvedInVersion":"20240101.1"}' >/dev/null
|
||||
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | st # status resolved | resolvedInVersion 20240101.1 | resolvedBy None
|
||||
|
||||
curl -s -X POST "$B/api/v1/errors/$ERRID_B/ignore" -H "$H" -H 'Content-Type: application/json' -d '{"duration":3600000,"reason":"known flake"}' >/dev/null
|
||||
curl -s "$B/api/v1/errors/$ERRID_B" -H "$H" | st # status ignored | ignoredUntil True | reason known flake
|
||||
|
||||
curl -s -X POST "$B/api/v1/errors/$ERRID_A/unresolve" -H "$H" >/dev/null
|
||||
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | st # status unresolved
|
||||
```
|
||||
PASS: each transition reflected; `filter[status]=ignored` returns only beta:
|
||||
```bash
|
||||
curl -s "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK&filter%5Bstatus%5D=ignored&filter%5Bperiod%5D=1d" -H "$H" | python3 -c "import sys,json;print([e['id'] for e in json.load(sys.stdin)['data']])" # [error_<fpB>]
|
||||
```
|
||||
|
||||
### 5. `filter[error]` on the runs list (paired PG + CH seed)
|
||||
|
||||
The runs list hydrates from Postgres, so seed a matching `TaskRun` row + a CH row
|
||||
that share `run_id`/`id` and carry a fingerprint:
|
||||
```bash
|
||||
RID="re2e${RUN}"; FRID="run_${RID}"; FP_R="fpR${RUN}"
|
||||
psql "$DBURL" -v ON_ERROR_STOP=1 -c "
|
||||
INSERT INTO \"TaskRun\" (id, \"friendlyId\", \"taskIdentifier\", payload, \"traceId\", \"spanId\", \"runtimeEnvironmentId\", \"projectId\", queue, status, \"createdAt\", \"updatedAt\")
|
||||
VALUES ('$RID','$FRID','$TASK','{}','trace_$RID','span_$RID','$ENV','$PROJ','task/$TASK','COMPLETED_WITH_ERRORS', now(), now())
|
||||
ON CONFLICT (id) DO NOTHING;" >/dev/null
|
||||
ROW="{\"environment_id\":\"$ENV\",\"organization_id\":\"$ORG\",\"project_id\":\"$PROJ\",\"run_id\":\"$RID\",\"friendly_id\":\"$FRID\",\"status\":\"COMPLETED_WITH_ERRORS\",\"environment_type\":\"DEVELOPMENT\",\"engine\":\"V2\",\"task_identifier\":\"$TASK\",\"created_at\":\"$NOW_CH\",\"updated_at\":\"$NOW_CH\",\"error\":{\"data\":{\"type\":\"RunsFilterErr\",\"message\":\"for runs filter\",\"stack\":\"at x\"}},\"error_fingerprint\":\"$FP_R\",\"task_version\":\"20240101.1\",\"_version\":\"$NOW_MS\",\"_is_deleted\":0}"
|
||||
printf '%s' "$ROW" | curl -s "$CHURL/?query=$Q" --data-binary @-
|
||||
sleep 1
|
||||
curl -s "$B/api/v1/runs?filter%5Berror%5D=error_$FP_R" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('runs:',[r['id'] for r in d['data']])"
|
||||
```
|
||||
PASS: one run, `run_<RID>` (status maps to `FAILED`). Proves `filter[error]` -> fingerprint -> CH -> PG hydration.
|
||||
|
||||
### 6. Attribution — `mint-token` -> JWT exchange records the acting user
|
||||
|
||||
```bash
|
||||
TOKEN=$(cli mint-token --profile $PROFILE --client errors-api-e2e 2>/dev/null) # UAT
|
||||
ENVJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors","write:errors"]}}' \
|
||||
| python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
|
||||
# Decoded env JWT carries act.sub = the user id.
|
||||
node -e 'const p=JSON.parse(Buffer.from(process.argv[1].split(".")[1],"base64url").toString());console.log("act:",JSON.stringify(p.act))' "$ENVJWT"
|
||||
|
||||
curl -s -X POST "$B/api/v1/errors/$ERRID_A/resolve" -H "Authorization: Bearer $ENVJWT" \
|
||||
-H 'Content-Type: application/json' -d '{"resolvedInVersion":"20240101.2"}' >/dev/null
|
||||
curl -s "$B/api/v1/errors/$ERRID_A" -H "$H" | python3 -c "import sys,json;d=json.load(sys.stdin);print('resolvedBy:',d['resolvedBy'])"
|
||||
```
|
||||
PASS: `act.sub` is the user id (matches `cli whoami`), and `detail.resolvedBy` equals that user id (not null). A plain env key leaves it null (step 4). A **PAT** exchanged the same way also stamps `act` — repeat with the stored PAT to confirm `ignoredByUserId` attribution.
|
||||
|
||||
### 7. Negatives
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w 'unknown id: %{http_code} (404)\n' "$B/api/v1/errors/error_doesnotexist0000" -H "$H"
|
||||
curl -s -o /dev/null -w 'no auth list: %{http_code} (401)\n' "$B/api/v1/errors"
|
||||
curl -s -o /dev/null -w 'no auth resolve: %{http_code} (401)\n' -X POST "$B/api/v1/errors/$ERRID_B/resolve" -H 'Content-Type: application/json' -d '{}'
|
||||
|
||||
# read-only JWT must be denied on write, allowed on read
|
||||
READJWT=$(curl -sS -X POST "$B/api/v1/projects/$REF/dev/jwt" -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"claims":{"scopes":["read:errors"]}}' | python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
|
||||
curl -s -o /dev/null -w 'read JWT write: %{http_code} (403)\n' -X POST "$B/api/v1/errors/$ERRID_B/resolve" -H "Authorization: Bearer $READJWT" -H 'Content-Type: application/json' -d '{}'
|
||||
curl -s -o /dev/null -w 'read JWT read: %{http_code} (200)\n' "$B/api/v1/errors?filter%5BtaskIdentifier%5D=$TASK" -H "Authorization: Bearer $READJWT"
|
||||
```
|
||||
PASS: `404`, `401`, `401`, `403`, `200` respectively.
|
||||
|
||||
## Result
|
||||
|
||||
Report PASS only if: step 1 lands 2 groups in `errors_v1`; step 2's filters and
|
||||
pagination narrow correctly; step 3 returns the detail; step 4's resolve/ignore/
|
||||
unresolve flip status (and `filter[status]` follows); step 5's `filter[error]`
|
||||
returns the paired run; step 6 records `resolvedBy` = the acting user via the
|
||||
JWT exchange (null with a plain env key); and step 7 returns 404/401/401/403/200.
|
||||
A red leg is a bug or a missing prereq — report the exact status + body and file
|
||||
a Linear issue, don't tune around it.
|
||||
|
||||
## Notes / gotchas
|
||||
|
||||
- Run files use a unique `$RUN` suffix per invocation, so reruns don't collide and seeded rows stay isolated by their unique task identifier. They are local-dev test rows (90-day ClickHouse TTL); no cleanup required.
|
||||
- After **adding** the route files, the classic Remix dev compiler may not register them until a dev-server restart (a stale manifest returns Remix's HTML 404 on the new paths). If `POST …/resolve` returns a 404 HTML page rather than 401/200, restart `pnpm run dev --filter webapp`.
|
||||
- The rbac `act` extraction lives in `@trigger.dev/rbac` (a built dep). After editing it, `pnpm run build --filter @trigger.dev/rbac` and restart the webapp so the attribution leg (step 6) reflects the change.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: span-timeline-events
|
||||
description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events.
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# Span Timeline Events
|
||||
|
||||
The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
|
||||
|
||||
## How They Work
|
||||
|
||||
1. **Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
|
||||
2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
|
||||
3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
|
||||
4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
|
||||
|
||||
## ClickHouse Storage Constraint
|
||||
|
||||
When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
|
||||
|
||||
## Timeline Rendering (SpanTimeline component)
|
||||
|
||||
The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
|
||||
|
||||
1. **Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
|
||||
2. **"Started"** marker (thick cap) - at the span's `startTime`
|
||||
3. **Duration bar** (thick 7px line) - from "Started" to "Finished"
|
||||
4. **"Finished"** marker (thick cap) - at `startTime + duration`
|
||||
|
||||
The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
|
||||
|
||||
## Trace View Sort Order
|
||||
|
||||
Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
|
||||
|
||||
## Event Structure
|
||||
|
||||
```typescript
|
||||
// OTel span event format
|
||||
{
|
||||
name: "trigger.dev/run", // Must start with "trigger.dev/" to render
|
||||
timeUnixNano: "1711200000000000000",
|
||||
attributes: [
|
||||
{ key: "event", value: { stringValue: "dequeue" } }, // The actual event type
|
||||
{ key: "duration", value: { intValue: 150 } }, // Optional: duration in ms
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Admin-Only Events
|
||||
|
||||
`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
|
||||
|
||||
| Event | Admin-only | Friendly name |
|
||||
|-------|-----------|---------------|
|
||||
| `dequeue` | No | Dequeued |
|
||||
| `fork` | No | Launched |
|
||||
| `import` | No (if no fork event) | Importing task file |
|
||||
| `create_attempt` | Yes | Attempt created |
|
||||
| `lazy_payload` | Yes | Lazy attempt initialized |
|
||||
| `pod_scheduled` | Yes | Pod scheduled |
|
||||
| (default) | Yes | (raw event name) |
|
||||
|
||||
## Adding New Timeline Events
|
||||
|
||||
1. Add OTLP span event with `name: "trigger.dev/<scope>"` and `properties.event: "<type>"`
|
||||
2. Event timestamp must be strictly after the parent span's `startTimeUnixNano` (ClickHouse drops earlier events)
|
||||
3. Add friendly name in `getFriendlyNameForEvent()` in `app/utils/timelineSpanEvents.ts`
|
||||
4. Set admin visibility in `getAdminOnlyForEvent()`
|
||||
5. Optionally add help text in `getHelpTextForEvent()`
|
||||
|
||||
## Key Files
|
||||
|
||||
- `app/utils/timelineSpanEvents.ts` - filtering, naming, admin logic
|
||||
- `app/components/run/RunTimeline.tsx` - `SpanTimeline` component (thin line + thick bar rendering)
|
||||
- `app/presenters/v3/SpanPresenter.server.ts` - loads span data including events
|
||||
- `app/v3/eventRepository/clickhouseEventRepository.server.ts` - `spanEventsToTaskEventV1Input()` (storage filter), `#mergeRecordsIntoSpanDetail` (reassembly)
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
name: trigger-dev-tasks
|
||||
description: Use this skill when writing, designing, or optimizing Trigger.dev background tasks and workflows. This includes creating reliable async tasks, implementing AI workflows, setting up scheduled jobs, structuring complex task hierarchies with subtasks, configuring build extensions for tools like ffmpeg or Puppeteer/Playwright, and handling task schemas with Zod validation.
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# Trigger.dev Task Expert
|
||||
|
||||
You are an expert Trigger.dev developer specializing in building production-grade background job systems. Tasks deployed to Trigger.dev run in Node.js 21+ and use the `@trigger.dev/sdk` package.
|
||||
|
||||
## Critical Rules
|
||||
|
||||
1. **Always use `@trigger.dev/sdk`** - Never use `@trigger.dev/sdk/v3` or deprecated `client.defineJob` pattern
|
||||
2. **Never use `node-fetch`** - Use the built-in `fetch` function
|
||||
3. **Export all tasks** - Every task must be exported, including subtasks
|
||||
4. **Never wrap wait/trigger calls in Promise.all** - `triggerAndWait`, `batchTriggerAndWait`, and `wait.*` calls cannot be wrapped in `Promise.all` or `Promise.allSettled`
|
||||
|
||||
## Basic Task Pattern
|
||||
|
||||
```ts
|
||||
import { task } from "@trigger.dev/sdk";
|
||||
|
||||
export const processData = task({
|
||||
id: "process-data",
|
||||
retry: {
|
||||
maxAttempts: 10,
|
||||
factor: 1.8,
|
||||
minTimeoutInMs: 500,
|
||||
maxTimeoutInMs: 30_000,
|
||||
},
|
||||
run: async (payload: { userId: string; data: any[] }) => {
|
||||
console.log(`Processing ${payload.data.length} items`);
|
||||
return { processed: payload.data.length };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Schema Task (with validation)
|
||||
|
||||
```ts
|
||||
import { schemaTask } from "@trigger.dev/sdk";
|
||||
import { z } from "zod";
|
||||
|
||||
export const validatedTask = schemaTask({
|
||||
id: "validated-task",
|
||||
schema: z.object({
|
||||
name: z.string(),
|
||||
email: z.string().email(),
|
||||
}),
|
||||
run: async (payload) => {
|
||||
// Payload is automatically validated and typed
|
||||
return { message: `Hello ${payload.name}` };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Triggering Tasks
|
||||
|
||||
### From Backend Code (type-only import to prevent dependency leakage)
|
||||
|
||||
```ts
|
||||
import { tasks } from "@trigger.dev/sdk";
|
||||
import type { processData } from "./trigger/tasks";
|
||||
|
||||
const handle = await tasks.trigger<typeof processData>("process-data", {
|
||||
userId: "123",
|
||||
data: [{ id: 1 }],
|
||||
});
|
||||
```
|
||||
|
||||
### From Inside Tasks
|
||||
|
||||
```ts
|
||||
export const parentTask = task({
|
||||
id: "parent-task",
|
||||
run: async (payload) => {
|
||||
// Trigger and wait - returns Result object, NOT direct output
|
||||
const result = await childTask.triggerAndWait({ data: "value" });
|
||||
if (result.ok) {
|
||||
console.log("Output:", result.output);
|
||||
} else {
|
||||
console.error("Failed:", result.error);
|
||||
}
|
||||
|
||||
// Or unwrap directly (throws on error)
|
||||
const output = await childTask.triggerAndWait({ data: "value" }).unwrap();
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Idempotency (Critical for Retries)
|
||||
|
||||
Always use idempotency keys when triggering tasks from inside other tasks:
|
||||
|
||||
```ts
|
||||
import { idempotencyKeys } from "@trigger.dev/sdk";
|
||||
|
||||
export const paymentTask = task({
|
||||
id: "process-payment",
|
||||
run: async (payload: { orderId: string }) => {
|
||||
// Scoped to current run - survives retries
|
||||
const key = await idempotencyKeys.create(`payment-${payload.orderId}`);
|
||||
|
||||
await chargeCustomer.trigger(payload, {
|
||||
idempotencyKey: key,
|
||||
idempotencyKeyTTL: "24h",
|
||||
});
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Trigger Options
|
||||
|
||||
```ts
|
||||
await myTask.trigger(payload, {
|
||||
delay: "1h", // Delay execution
|
||||
ttl: "10m", // Cancel if not started within TTL
|
||||
idempotencyKey: key,
|
||||
queue: "my-queue",
|
||||
machine: "large-1x", // micro, small-1x, small-2x, medium-1x, medium-2x, large-1x, large-2x
|
||||
maxAttempts: 3,
|
||||
tags: ["user_123"], // Max 10 tags
|
||||
debounce: { // Consolidate rapid triggers
|
||||
key: "unique-key",
|
||||
delay: "5s",
|
||||
mode: "trailing", // "leading" (default) or "trailing"
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Debouncing
|
||||
|
||||
Consolidate multiple triggers into a single execution:
|
||||
|
||||
```ts
|
||||
// Rapid triggers with same key = single execution
|
||||
await myTask.trigger({ userId: "123" }, {
|
||||
debounce: {
|
||||
key: "user-123-update",
|
||||
delay: "5s",
|
||||
},
|
||||
});
|
||||
|
||||
// Trailing mode: use payload from LAST trigger
|
||||
await myTask.trigger({ data: "latest" }, {
|
||||
debounce: {
|
||||
key: "my-key",
|
||||
delay: "10s",
|
||||
mode: "trailing",
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Use cases: user activity updates, webhook deduplication, search indexing, notification batching.
|
||||
|
||||
## Batch Triggering
|
||||
|
||||
Up to 1,000 items per batch, 3MB per payload:
|
||||
|
||||
```ts
|
||||
const results = await myTask.batchTriggerAndWait([
|
||||
{ payload: { userId: "1" } },
|
||||
{ payload: { userId: "2" } },
|
||||
]);
|
||||
|
||||
for (const result of results) {
|
||||
if (result.ok) console.log(result.output);
|
||||
}
|
||||
```
|
||||
|
||||
## Machine Presets
|
||||
|
||||
| Preset | vCPU | Memory |
|
||||
|-------------|------|--------|
|
||||
| micro | 0.25 | 0.25GB |
|
||||
| small-1x | 0.5 | 0.5GB |
|
||||
| small-2x | 1 | 1GB |
|
||||
| medium-1x | 1 | 2GB |
|
||||
| medium-2x | 2 | 4GB |
|
||||
| large-1x | 4 | 8GB |
|
||||
| large-2x | 8 | 16GB |
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Break complex workflows into subtasks** that can be independently retried and made idempotent
|
||||
2. **Don't over-complicate** - Sometimes `Promise.allSettled` inside a single task is better than many subtasks (each task has dedicated process and is charged by millisecond)
|
||||
3. **Always configure retries** - Set appropriate `maxAttempts` based on the operation
|
||||
4. **Use idempotency keys** - Especially for payment/critical operations
|
||||
5. **Group related subtasks** - Keep subtasks only used by one parent in the same file, don't export them
|
||||
6. **Use logger** - Log at key execution points with `logger.info()`, `logger.error()`, etc.
|
||||
|
||||
## Reference Documentation
|
||||
|
||||
For detailed documentation on specific topics, read these files:
|
||||
|
||||
- `basic-tasks.md` - Task basics, triggering, waits
|
||||
- `advanced-tasks.md` - Tags, queues, concurrency, metadata, error handling
|
||||
- `scheduled-tasks.md` - Cron schedules, declarative and imperative
|
||||
- `realtime.md` - Real-time subscriptions, streams, React hooks
|
||||
- `config.md` - trigger.config.ts, build extensions (Prisma, Playwright, FFmpeg, etc.)
|
||||
@@ -0,0 +1,485 @@
|
||||
# Trigger.dev Advanced Tasks (v4)
|
||||
|
||||
**Advanced patterns and features for writing tasks**
|
||||
|
||||
## Tags & Organization
|
||||
|
||||
```ts
|
||||
import { task, tags } from "@trigger.dev/sdk";
|
||||
|
||||
export const processUser = task({
|
||||
id: "process-user",
|
||||
run: async (payload: { userId: string; orgId: string }, { ctx }) => {
|
||||
// Add tags during execution
|
||||
await tags.add(`user_${payload.userId}`);
|
||||
await tags.add(`org_${payload.orgId}`);
|
||||
|
||||
return { processed: true };
|
||||
},
|
||||
});
|
||||
|
||||
// Trigger with tags
|
||||
await processUser.trigger(
|
||||
{ userId: "123", orgId: "abc" },
|
||||
{ tags: ["priority", "user_123", "org_abc"] } // Max 10 tags per run
|
||||
);
|
||||
|
||||
// Subscribe to tagged runs
|
||||
for await (const run of runs.subscribeToRunsWithTag("user_123")) {
|
||||
console.log(`User task ${run.id}: ${run.status}`);
|
||||
}
|
||||
```
|
||||
|
||||
**Tag Best Practices:**
|
||||
|
||||
- Use prefixes: `user_123`, `org_abc`, `video:456`
|
||||
- Max 10 tags per run, 1-64 characters each
|
||||
- Tags don't propagate to child tasks automatically
|
||||
|
||||
## Batch Triggering v2
|
||||
|
||||
Enhanced batch triggering with larger payloads and streaming ingestion.
|
||||
|
||||
### Limits
|
||||
|
||||
- **Maximum batch size**: 1,000 items (increased from 500)
|
||||
- **Payload per item**: 3MB each (increased from 1MB combined)
|
||||
- Payloads > 512KB automatically offload to object storage
|
||||
|
||||
### Rate Limiting (per environment)
|
||||
|
||||
| Tier | Bucket Size | Refill Rate |
|
||||
|------|-------------|-------------|
|
||||
| Free | 1,200 runs | 100 runs/10 sec |
|
||||
| Hobby | 5,000 runs | 500 runs/5 sec |
|
||||
| Pro | 5,000 runs | 500 runs/5 sec |
|
||||
|
||||
### Concurrent Batch Processing
|
||||
|
||||
| Tier | Concurrent Batches |
|
||||
|------|-------------------|
|
||||
| Free | 1 |
|
||||
| Hobby | 10 |
|
||||
| Pro | 10 |
|
||||
|
||||
### Usage
|
||||
|
||||
```ts
|
||||
import { myTask } from "./trigger/myTask";
|
||||
|
||||
// Basic batch trigger (up to 1,000 items)
|
||||
const runs = await myTask.batchTrigger([
|
||||
{ payload: { userId: "user-1" } },
|
||||
{ payload: { userId: "user-2" } },
|
||||
{ payload: { userId: "user-3" } },
|
||||
]);
|
||||
|
||||
// Batch trigger with wait
|
||||
const results = await myTask.batchTriggerAndWait([
|
||||
{ payload: { userId: "user-1" } },
|
||||
{ payload: { userId: "user-2" } },
|
||||
]);
|
||||
|
||||
for (const result of results) {
|
||||
if (result.ok) {
|
||||
console.log("Result:", result.output);
|
||||
}
|
||||
}
|
||||
|
||||
// With per-item options
|
||||
const batchHandle = await myTask.batchTrigger([
|
||||
{
|
||||
payload: { userId: "123" },
|
||||
options: {
|
||||
idempotencyKey: "user-123-batch",
|
||||
tags: ["priority"],
|
||||
},
|
||||
},
|
||||
{
|
||||
payload: { userId: "456" },
|
||||
options: {
|
||||
idempotencyKey: "user-456-batch",
|
||||
},
|
||||
},
|
||||
]);
|
||||
```
|
||||
|
||||
## Debouncing
|
||||
|
||||
Consolidate multiple triggers into a single execution by debouncing task runs with a unique key and delay window.
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **User activity updates**: Batch rapid user actions into a single run
|
||||
- **Webhook deduplication**: Handle webhook bursts without redundant processing
|
||||
- **Search indexing**: Combine document updates instead of processing individually
|
||||
- **Notification batching**: Group notifications to prevent user spam
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```ts
|
||||
await myTask.trigger(
|
||||
{ userId: "123" },
|
||||
{
|
||||
debounce: {
|
||||
key: "user-123-update", // Unique identifier for debounce group
|
||||
delay: "5s", // Wait duration ("5s", "1m", or milliseconds)
|
||||
},
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
### Execution Modes
|
||||
|
||||
**Leading Mode** (default): Uses payload/options from the first trigger; subsequent triggers only reschedule execution time.
|
||||
|
||||
```ts
|
||||
// First trigger sets the payload
|
||||
await myTask.trigger({ action: "first" }, {
|
||||
debounce: { key: "my-key", delay: "10s" }
|
||||
});
|
||||
|
||||
// Second trigger only reschedules - payload remains "first"
|
||||
await myTask.trigger({ action: "second" }, {
|
||||
debounce: { key: "my-key", delay: "10s" }
|
||||
});
|
||||
// Task executes with { action: "first" }
|
||||
```
|
||||
|
||||
**Trailing Mode**: Uses payload/options from the most recent trigger.
|
||||
|
||||
```ts
|
||||
await myTask.trigger(
|
||||
{ data: "latest-value" },
|
||||
{
|
||||
debounce: {
|
||||
key: "trailing-example",
|
||||
delay: "10s",
|
||||
mode: "trailing",
|
||||
},
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
In trailing mode, these options update with each trigger:
|
||||
- `payload` — task input data
|
||||
- `metadata` — run metadata
|
||||
- `tags` — run tags (replaces existing)
|
||||
- `maxAttempts` — retry attempts
|
||||
- `maxDuration` — maximum compute time
|
||||
- `machine` — machine preset
|
||||
|
||||
### Important Notes
|
||||
|
||||
- Idempotency keys take precedence over debounce settings
|
||||
- Compatible with `triggerAndWait()` — parent runs block correctly on debounced execution
|
||||
- Debounce key is scoped to the task
|
||||
|
||||
## Concurrency & Queues
|
||||
|
||||
```ts
|
||||
import { task, queue } from "@trigger.dev/sdk";
|
||||
|
||||
// Shared queue for related tasks
|
||||
const emailQueue = queue({
|
||||
name: "email-processing",
|
||||
concurrencyLimit: 5, // Max 5 emails processing simultaneously
|
||||
});
|
||||
|
||||
// Task-level concurrency
|
||||
export const oneAtATime = task({
|
||||
id: "sequential-task",
|
||||
queue: { concurrencyLimit: 1 }, // Process one at a time
|
||||
run: async (payload) => {
|
||||
// Critical section - only one instance runs
|
||||
},
|
||||
});
|
||||
|
||||
// Per-user concurrency
|
||||
export const processUserData = task({
|
||||
id: "process-user-data",
|
||||
run: async (payload: { userId: string }) => {
|
||||
// Override queue with user-specific concurrency
|
||||
await childTask.trigger(payload, {
|
||||
queue: {
|
||||
name: `user-${payload.userId}`,
|
||||
concurrencyLimit: 2,
|
||||
},
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
export const emailTask = task({
|
||||
id: "send-email",
|
||||
queue: emailQueue, // Use shared queue
|
||||
run: async (payload: { to: string }) => {
|
||||
// Send email logic
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Error Handling & Retries
|
||||
|
||||
```ts
|
||||
import { task, retry, AbortTaskRunError } from "@trigger.dev/sdk";
|
||||
|
||||
export const resilientTask = task({
|
||||
id: "resilient-task",
|
||||
retry: {
|
||||
maxAttempts: 10,
|
||||
factor: 1.8, // Exponential backoff multiplier
|
||||
minTimeoutInMs: 500,
|
||||
maxTimeoutInMs: 30_000,
|
||||
randomize: false,
|
||||
},
|
||||
catchError: async ({ error, ctx }) => {
|
||||
// Custom error handling
|
||||
if (error.code === "FATAL_ERROR") {
|
||||
throw new AbortTaskRunError("Cannot retry this error");
|
||||
}
|
||||
|
||||
// Log error details
|
||||
console.error(`Task ${ctx.task.id} failed:`, error);
|
||||
|
||||
// Allow retry by returning nothing
|
||||
return { retryAt: new Date(Date.now() + 60000) }; // Retry in 1 minute
|
||||
},
|
||||
run: async (payload) => {
|
||||
// Retry specific operations
|
||||
const result = await retry.onThrow(
|
||||
async () => {
|
||||
return await unstableApiCall(payload);
|
||||
},
|
||||
{ maxAttempts: 3 }
|
||||
);
|
||||
|
||||
// Conditional HTTP retries
|
||||
const response = await retry.fetch("https://api.example.com", {
|
||||
retry: {
|
||||
maxAttempts: 5,
|
||||
condition: (response, error) => {
|
||||
return response?.status === 429 || response?.status >= 500;
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
return result;
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Machines & Performance
|
||||
|
||||
```ts
|
||||
export const heavyTask = task({
|
||||
id: "heavy-computation",
|
||||
machine: { preset: "large-2x" }, // 8 vCPU, 16 GB RAM
|
||||
maxDuration: 1800, // 30 minutes timeout
|
||||
run: async (payload, { ctx }) => {
|
||||
// Resource-intensive computation
|
||||
if (ctx.machine.preset === "large-2x") {
|
||||
// Use all available cores
|
||||
return await parallelProcessing(payload);
|
||||
}
|
||||
|
||||
return await standardProcessing(payload);
|
||||
},
|
||||
});
|
||||
|
||||
// Override machine when triggering
|
||||
await heavyTask.trigger(payload, {
|
||||
machine: { preset: "medium-1x" }, // Override for this run
|
||||
});
|
||||
```
|
||||
|
||||
**Machine Presets:**
|
||||
|
||||
- `micro`: 0.25 vCPU, 0.25 GB RAM
|
||||
- `small-1x`: 0.5 vCPU, 0.5 GB RAM (default)
|
||||
- `small-2x`: 1 vCPU, 1 GB RAM
|
||||
- `medium-1x`: 1 vCPU, 2 GB RAM
|
||||
- `medium-2x`: 2 vCPU, 4 GB RAM
|
||||
- `large-1x`: 4 vCPU, 8 GB RAM
|
||||
- `large-2x`: 8 vCPU, 16 GB RAM
|
||||
|
||||
## Idempotency
|
||||
|
||||
```ts
|
||||
import { task, idempotencyKeys } from "@trigger.dev/sdk";
|
||||
|
||||
export const paymentTask = task({
|
||||
id: "process-payment",
|
||||
retry: {
|
||||
maxAttempts: 3,
|
||||
},
|
||||
run: async (payload: { orderId: string; amount: number }) => {
|
||||
// Automatically scoped to this task run, so if the task is retried, the idempotency key will be the same
|
||||
const idempotencyKey = await idempotencyKeys.create(`payment-${payload.orderId}`);
|
||||
|
||||
// Ensure payment is processed only once
|
||||
await chargeCustomer.trigger(payload, {
|
||||
idempotencyKey,
|
||||
idempotencyKeyTTL: "24h", // Key expires in 24 hours
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
// Payload-based idempotency
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
function createPayloadHash(payload: any): string {
|
||||
const hash = createHash("sha256");
|
||||
hash.update(JSON.stringify(payload));
|
||||
return hash.digest("hex");
|
||||
}
|
||||
|
||||
export const deduplicatedTask = task({
|
||||
id: "deduplicated-task",
|
||||
run: async (payload) => {
|
||||
const payloadHash = createPayloadHash(payload);
|
||||
const idempotencyKey = await idempotencyKeys.create(payloadHash);
|
||||
|
||||
await processData.trigger(payload, { idempotencyKey });
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Metadata & Progress Tracking
|
||||
|
||||
```ts
|
||||
import { task, metadata } from "@trigger.dev/sdk";
|
||||
|
||||
export const batchProcessor = task({
|
||||
id: "batch-processor",
|
||||
run: async (payload: { items: any[] }, { ctx }) => {
|
||||
const totalItems = payload.items.length;
|
||||
|
||||
// Initialize progress metadata
|
||||
metadata
|
||||
.set("progress", 0)
|
||||
.set("totalItems", totalItems)
|
||||
.set("processedItems", 0)
|
||||
.set("status", "starting");
|
||||
|
||||
const results = [];
|
||||
|
||||
for (let i = 0; i < payload.items.length; i++) {
|
||||
const item = payload.items[i];
|
||||
|
||||
// Process item
|
||||
const result = await processItem(item);
|
||||
results.push(result);
|
||||
|
||||
// Update progress
|
||||
const progress = ((i + 1) / totalItems) * 100;
|
||||
metadata
|
||||
.set("progress", progress)
|
||||
.increment("processedItems", 1)
|
||||
.append("logs", `Processed item ${i + 1}/${totalItems}`)
|
||||
.set("currentItem", item.id);
|
||||
}
|
||||
|
||||
// Final status
|
||||
metadata.set("status", "completed");
|
||||
|
||||
return { results, totalProcessed: results.length };
|
||||
},
|
||||
});
|
||||
|
||||
// Update parent metadata from child task
|
||||
export const childTask = task({
|
||||
id: "child-task",
|
||||
run: async (payload, { ctx }) => {
|
||||
// Update parent task metadata
|
||||
metadata.parent.set("childStatus", "processing");
|
||||
metadata.root.increment("childrenCompleted", 1);
|
||||
|
||||
return { processed: true };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Logging & Tracing
|
||||
|
||||
```ts
|
||||
import { task, logger } from "@trigger.dev/sdk";
|
||||
|
||||
export const tracedTask = task({
|
||||
id: "traced-task",
|
||||
run: async (payload, { ctx }) => {
|
||||
logger.info("Task started", { userId: payload.userId });
|
||||
|
||||
// Custom trace with attributes
|
||||
const user = await logger.trace(
|
||||
"fetch-user",
|
||||
async (span) => {
|
||||
span.setAttribute("user.id", payload.userId);
|
||||
span.setAttribute("operation", "database-fetch");
|
||||
|
||||
const userData = await database.findUser(payload.userId);
|
||||
span.setAttribute("user.found", !!userData);
|
||||
|
||||
return userData;
|
||||
},
|
||||
{ userId: payload.userId }
|
||||
);
|
||||
|
||||
logger.debug("User fetched", { user: user.id });
|
||||
|
||||
try {
|
||||
const result = await processUser(user);
|
||||
logger.info("Processing completed", { result });
|
||||
return result;
|
||||
} catch (error) {
|
||||
logger.error("Processing failed", {
|
||||
error: error.message,
|
||||
userId: payload.userId,
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Hidden Tasks
|
||||
|
||||
```ts
|
||||
// Hidden task - not exported, only used internally
|
||||
const internalProcessor = task({
|
||||
id: "internal-processor",
|
||||
run: async (payload: { data: string }) => {
|
||||
return { processed: payload.data.toUpperCase() };
|
||||
},
|
||||
});
|
||||
|
||||
// Public task that uses hidden task
|
||||
export const publicWorkflow = task({
|
||||
id: "public-workflow",
|
||||
run: async (payload: { input: string }) => {
|
||||
// Use hidden task internally
|
||||
const result = await internalProcessor.triggerAndWait({
|
||||
data: payload.input,
|
||||
});
|
||||
|
||||
if (result.ok) {
|
||||
return { output: result.output.processed };
|
||||
}
|
||||
|
||||
throw new Error("Internal processing failed");
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Concurrency**: Use queues to prevent overwhelming external services
|
||||
- **Retries**: Configure exponential backoff for transient failures
|
||||
- **Idempotency**: Always use for payment/critical operations
|
||||
- **Metadata**: Track progress for long-running tasks
|
||||
- **Machines**: Match machine size to computational requirements
|
||||
- **Tags**: Use consistent naming patterns for filtering
|
||||
- **Debouncing**: Use for user activity, webhooks, and notification batching
|
||||
- **Batch triggering**: Use for bulk operations up to 1,000 items
|
||||
- **Error Handling**: Distinguish between retryable and fatal errors
|
||||
|
||||
Design tasks to be stateless, idempotent, and resilient to failures. Use metadata for state tracking and queues for resource management.
|
||||
@@ -0,0 +1,199 @@
|
||||
# Trigger.dev Basic Tasks (v4)
|
||||
|
||||
**MUST use `@trigger.dev/sdk`, NEVER `client.defineJob`**
|
||||
|
||||
## Basic Task
|
||||
|
||||
```ts
|
||||
import { task } from "@trigger.dev/sdk";
|
||||
|
||||
export const processData = task({
|
||||
id: "process-data",
|
||||
retry: {
|
||||
maxAttempts: 10,
|
||||
factor: 1.8,
|
||||
minTimeoutInMs: 500,
|
||||
maxTimeoutInMs: 30_000,
|
||||
randomize: false,
|
||||
},
|
||||
run: async (payload: { userId: string; data: any[] }) => {
|
||||
// Task logic - runs for long time, no timeouts
|
||||
console.log(`Processing ${payload.data.length} items for user ${payload.userId}`);
|
||||
return { processed: payload.data.length };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Schema Task (with validation)
|
||||
|
||||
```ts
|
||||
import { schemaTask } from "@trigger.dev/sdk";
|
||||
import { z } from "zod";
|
||||
|
||||
export const validatedTask = schemaTask({
|
||||
id: "validated-task",
|
||||
schema: z.object({
|
||||
name: z.string(),
|
||||
age: z.number(),
|
||||
email: z.string().email(),
|
||||
}),
|
||||
run: async (payload) => {
|
||||
// Payload is automatically validated and typed
|
||||
return { message: `Hello ${payload.name}, age ${payload.age}` };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Triggering Tasks
|
||||
|
||||
### From Backend Code
|
||||
|
||||
```ts
|
||||
import { tasks } from "@trigger.dev/sdk";
|
||||
import type { processData } from "./trigger/tasks";
|
||||
|
||||
// Single trigger
|
||||
const handle = await tasks.trigger<typeof processData>("process-data", {
|
||||
userId: "123",
|
||||
data: [{ id: 1 }, { id: 2 }],
|
||||
});
|
||||
|
||||
// Batch trigger (up to 1,000 items, 3MB per payload)
|
||||
const batchHandle = await tasks.batchTrigger<typeof processData>("process-data", [
|
||||
{ payload: { userId: "123", data: [{ id: 1 }] } },
|
||||
{ payload: { userId: "456", data: [{ id: 2 }] } },
|
||||
]);
|
||||
```
|
||||
|
||||
### Debounced Triggering
|
||||
|
||||
Consolidate multiple triggers into a single execution:
|
||||
|
||||
```ts
|
||||
// Multiple rapid triggers with same key = single execution
|
||||
await myTask.trigger(
|
||||
{ userId: "123" },
|
||||
{
|
||||
debounce: {
|
||||
key: "user-123-update", // Unique key for debounce group
|
||||
delay: "5s", // Wait before executing
|
||||
},
|
||||
}
|
||||
);
|
||||
|
||||
// Trailing mode: use payload from LAST trigger
|
||||
await myTask.trigger(
|
||||
{ data: "latest-value" },
|
||||
{
|
||||
debounce: {
|
||||
key: "trailing-example",
|
||||
delay: "10s",
|
||||
mode: "trailing", // Default is "leading" (first payload)
|
||||
},
|
||||
}
|
||||
);
|
||||
```
|
||||
|
||||
**Debounce modes:**
|
||||
- `leading` (default): Uses payload from first trigger, subsequent triggers only reschedule
|
||||
- `trailing`: Uses payload from most recent trigger
|
||||
|
||||
### From Inside Tasks (with Result handling)
|
||||
|
||||
```ts
|
||||
export const parentTask = task({
|
||||
id: "parent-task",
|
||||
run: async (payload) => {
|
||||
// Trigger and continue
|
||||
const handle = await childTask.trigger({ data: "value" });
|
||||
|
||||
// Trigger and wait - returns Result object, NOT task output
|
||||
const result = await childTask.triggerAndWait({ data: "value" });
|
||||
if (result.ok) {
|
||||
console.log("Task output:", result.output); // Actual task return value
|
||||
} else {
|
||||
console.error("Task failed:", result.error);
|
||||
}
|
||||
|
||||
// Quick unwrap (throws on error)
|
||||
const output = await childTask.triggerAndWait({ data: "value" }).unwrap();
|
||||
|
||||
// Batch trigger and wait
|
||||
const results = await childTask.batchTriggerAndWait([
|
||||
{ payload: { data: "item1" } },
|
||||
{ payload: { data: "item2" } },
|
||||
]);
|
||||
|
||||
for (const run of results) {
|
||||
if (run.ok) {
|
||||
console.log("Success:", run.output);
|
||||
} else {
|
||||
console.log("Failed:", run.error);
|
||||
}
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
export const childTask = task({
|
||||
id: "child-task",
|
||||
run: async (payload: { data: string }) => {
|
||||
return { processed: payload.data };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> Never wrap triggerAndWait or batchTriggerAndWait calls in a Promise.all or Promise.allSettled as this is not supported in Trigger.dev tasks.
|
||||
|
||||
## Waits
|
||||
|
||||
```ts
|
||||
import { task, wait } from "@trigger.dev/sdk";
|
||||
|
||||
export const taskWithWaits = task({
|
||||
id: "task-with-waits",
|
||||
run: async (payload) => {
|
||||
console.log("Starting task");
|
||||
|
||||
// Wait for specific duration
|
||||
await wait.for({ seconds: 30 });
|
||||
await wait.for({ minutes: 5 });
|
||||
await wait.for({ hours: 1 });
|
||||
await wait.for({ days: 1 });
|
||||
|
||||
// Wait until specific date
|
||||
await wait.until({ date: new Date("2024-12-25") });
|
||||
|
||||
// Wait for token (from external system)
|
||||
await wait.forToken({
|
||||
token: "user-approval-token",
|
||||
timeoutInSeconds: 3600, // 1 hour timeout
|
||||
});
|
||||
|
||||
console.log("All waits completed");
|
||||
return { status: "completed" };
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> Never wrap wait calls in a Promise.all or Promise.allSettled as this is not supported in Trigger.dev tasks.
|
||||
|
||||
## Key Points
|
||||
|
||||
- **Result vs Output**: `triggerAndWait()` returns a `Result` object with `ok`, `output`, `error` properties - NOT the direct task output
|
||||
- **Type safety**: Use `import type` for task references when triggering from backend
|
||||
- **Waits > 5 seconds**: Automatically checkpointed, don't count toward compute usage
|
||||
- **Debounce + idempotency**: Idempotency keys take precedence over debounce settings
|
||||
|
||||
## NEVER Use (v2 deprecated)
|
||||
|
||||
```ts
|
||||
// BREAKS APPLICATION
|
||||
client.defineJob({
|
||||
id: "job-id",
|
||||
run: async (payload, io) => {
|
||||
/* ... */
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
Use SDK (`@trigger.dev/sdk`), check `result.ok` before accessing `result.output`
|
||||
@@ -0,0 +1,346 @@
|
||||
# Trigger.dev Configuration
|
||||
|
||||
**Complete guide to configuring `trigger.config.ts` with build extensions**
|
||||
|
||||
## Basic Configuration
|
||||
|
||||
```ts
|
||||
import { defineConfig } from "@trigger.dev/sdk";
|
||||
|
||||
export default defineConfig({
|
||||
project: "<project-ref>", // Required: Your project reference
|
||||
dirs: ["./trigger"], // Task directories
|
||||
runtime: "node", // "node", "node-22", or "bun"
|
||||
logLevel: "info", // "debug", "info", "warn", "error"
|
||||
|
||||
// Default retry settings
|
||||
retries: {
|
||||
enabledInDev: false,
|
||||
default: {
|
||||
maxAttempts: 3,
|
||||
minTimeoutInMs: 1000,
|
||||
maxTimeoutInMs: 10000,
|
||||
factor: 2,
|
||||
randomize: true,
|
||||
},
|
||||
},
|
||||
|
||||
// Build configuration
|
||||
build: {
|
||||
autoDetectExternal: true,
|
||||
keepNames: true,
|
||||
minify: false,
|
||||
extensions: [], // Build extensions go here
|
||||
},
|
||||
|
||||
// Global lifecycle hooks
|
||||
onStartAttempt: async ({ payload, ctx }) => {
|
||||
console.log("Global task start");
|
||||
},
|
||||
onSuccess: async ({ payload, output, ctx }) => {
|
||||
console.log("Global task success");
|
||||
},
|
||||
onFailure: async ({ payload, error, ctx }) => {
|
||||
console.log("Global task failure");
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Build Extensions
|
||||
|
||||
### Database & ORM
|
||||
|
||||
#### Prisma
|
||||
|
||||
```ts
|
||||
import { prismaExtension } from "@trigger.dev/build/extensions/prisma";
|
||||
|
||||
extensions: [
|
||||
prismaExtension({
|
||||
schema: "prisma/schema.prisma",
|
||||
version: "5.19.0", // Optional: specify version
|
||||
migrate: true, // Run migrations during build
|
||||
directUrlEnvVarName: "DIRECT_DATABASE_URL",
|
||||
typedSql: true, // Enable TypedSQL support
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
#### TypeScript Decorators (for TypeORM)
|
||||
|
||||
```ts
|
||||
import { emitDecoratorMetadata } from "@trigger.dev/build/extensions/typescript";
|
||||
|
||||
extensions: [
|
||||
emitDecoratorMetadata(), // Enables decorator metadata
|
||||
];
|
||||
```
|
||||
|
||||
### Scripting Languages
|
||||
|
||||
#### Python
|
||||
|
||||
```ts
|
||||
import { pythonExtension } from "@trigger.dev/build/extensions/python";
|
||||
|
||||
extensions: [
|
||||
pythonExtension({
|
||||
scripts: ["./python/**/*.py"], // Copy Python files
|
||||
requirementsFile: "./requirements.txt", // Install packages
|
||||
devPythonBinaryPath: ".venv/bin/python", // Dev mode binary
|
||||
}),
|
||||
];
|
||||
|
||||
// Usage in tasks
|
||||
const result = await python.runInline(`print("Hello, world!")`);
|
||||
const output = await python.runScript("./python/script.py", ["arg1"]);
|
||||
```
|
||||
|
||||
### Browser Automation
|
||||
|
||||
#### Playwright
|
||||
|
||||
```ts
|
||||
import { playwright } from "@trigger.dev/build/extensions/playwright";
|
||||
|
||||
extensions: [
|
||||
playwright({
|
||||
browsers: ["chromium", "firefox", "webkit"], // Default: ["chromium"]
|
||||
headless: true, // Default: true
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
#### Puppeteer
|
||||
|
||||
```ts
|
||||
import { puppeteer } from "@trigger.dev/build/extensions/puppeteer";
|
||||
|
||||
extensions: [puppeteer()];
|
||||
|
||||
// Environment variable needed:
|
||||
// PUPPETEER_EXECUTABLE_PATH: "/usr/bin/google-chrome-stable"
|
||||
```
|
||||
|
||||
#### Lightpanda
|
||||
|
||||
```ts
|
||||
import { lightpanda } from "@trigger.dev/build/extensions/lightpanda";
|
||||
|
||||
extensions: [
|
||||
lightpanda({
|
||||
version: "latest", // or "nightly"
|
||||
disableTelemetry: false,
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
### Media Processing
|
||||
|
||||
#### FFmpeg
|
||||
|
||||
```ts
|
||||
import { ffmpeg } from "@trigger.dev/build/extensions/core";
|
||||
|
||||
extensions: [
|
||||
ffmpeg({ version: "7" }), // Static build, or omit for Debian version
|
||||
];
|
||||
|
||||
// Automatically sets FFMPEG_PATH and FFPROBE_PATH
|
||||
// Add fluent-ffmpeg to external packages if using
|
||||
```
|
||||
|
||||
#### Audio Waveform
|
||||
|
||||
```ts
|
||||
import { audioWaveform } from "@trigger.dev/build/extensions/audioWaveform";
|
||||
|
||||
extensions: [
|
||||
audioWaveform(), // Installs Audio Waveform 1.1.0
|
||||
];
|
||||
```
|
||||
|
||||
### System & Package Management
|
||||
|
||||
#### System Packages (apt-get)
|
||||
|
||||
```ts
|
||||
import { aptGet } from "@trigger.dev/build/extensions/core";
|
||||
|
||||
extensions: [
|
||||
aptGet({
|
||||
packages: ["ffmpeg", "imagemagick", "curl=7.68.0-1"], // Can specify versions
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
#### Additional NPM Packages
|
||||
|
||||
Only use this for installing CLI tools, NOT packages you import in your code.
|
||||
|
||||
```ts
|
||||
import { additionalPackages } from "@trigger.dev/build/extensions/core";
|
||||
|
||||
extensions: [
|
||||
additionalPackages({
|
||||
packages: ["wrangler"], // CLI tools and specific versions
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
#### Additional Files
|
||||
|
||||
```ts
|
||||
import { additionalFiles } from "@trigger.dev/build/extensions/core";
|
||||
|
||||
extensions: [
|
||||
additionalFiles({
|
||||
files: ["wrangler.toml", "./assets/**", "./fonts/**"], // Glob patterns supported
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
### Environment & Build Tools
|
||||
|
||||
#### Environment Variable Sync
|
||||
|
||||
```ts
|
||||
import { syncEnvVars } from "@trigger.dev/build/extensions/core";
|
||||
|
||||
extensions: [
|
||||
syncEnvVars(async (ctx) => {
|
||||
// ctx contains: environment, projectRef, env
|
||||
return [
|
||||
{ name: "SECRET_KEY", value: await getSecret(ctx.environment) },
|
||||
{ name: "API_URL", value: ctx.environment === "prod" ? "api.prod.com" : "api.dev.com" },
|
||||
];
|
||||
}),
|
||||
];
|
||||
```
|
||||
|
||||
#### ESBuild Plugins
|
||||
|
||||
```ts
|
||||
import { esbuildPlugin } from "@trigger.dev/build/extensions";
|
||||
import { sentryEsbuildPlugin } from "@sentry/esbuild-plugin";
|
||||
|
||||
extensions: [
|
||||
esbuildPlugin(
|
||||
sentryEsbuildPlugin({
|
||||
org: process.env.SENTRY_ORG,
|
||||
project: process.env.SENTRY_PROJECT,
|
||||
authToken: process.env.SENTRY_AUTH_TOKEN,
|
||||
}),
|
||||
{ placement: "last", target: "deploy" } // Optional config
|
||||
),
|
||||
];
|
||||
```
|
||||
|
||||
## Custom Build Extensions
|
||||
|
||||
```ts
|
||||
import { defineConfig } from "@trigger.dev/sdk";
|
||||
|
||||
const customExtension = {
|
||||
name: "my-custom-extension",
|
||||
|
||||
externalsForTarget: (target) => {
|
||||
return ["some-native-module"]; // Add external dependencies
|
||||
},
|
||||
|
||||
onBuildStart: async (context) => {
|
||||
console.log(`Build starting for ${context.target}`);
|
||||
// Register esbuild plugins, modify build context
|
||||
},
|
||||
|
||||
onBuildComplete: async (context, manifest) => {
|
||||
console.log("Build complete, adding layers");
|
||||
// Add build layers, modify deployment
|
||||
context.addLayer({
|
||||
id: "my-layer",
|
||||
files: [{ source: "./custom-file", destination: "/app/custom" }],
|
||||
commands: ["chmod +x /app/custom"],
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
export default defineConfig({
|
||||
project: "my-project",
|
||||
build: {
|
||||
extensions: [customExtension],
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Telemetry
|
||||
|
||||
```ts
|
||||
import { PrismaInstrumentation } from "@prisma/instrumentation";
|
||||
import { OpenAIInstrumentation } from "@langfuse/openai";
|
||||
|
||||
export default defineConfig({
|
||||
// ... other config
|
||||
telemetry: {
|
||||
instrumentations: [new PrismaInstrumentation(), new OpenAIInstrumentation()],
|
||||
exporters: [customExporter], // Optional custom exporters
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Machine & Performance
|
||||
|
||||
```ts
|
||||
export default defineConfig({
|
||||
// ... other config
|
||||
defaultMachine: "large-1x", // Default machine for all tasks
|
||||
maxDuration: 300, // Default max duration (seconds)
|
||||
enableConsoleLogging: true, // Console logging in development
|
||||
});
|
||||
```
|
||||
|
||||
## Common Extension Combinations
|
||||
|
||||
### Full-Stack Web App
|
||||
|
||||
```ts
|
||||
extensions: [
|
||||
prismaExtension({ schema: "prisma/schema.prisma", migrate: true }),
|
||||
additionalFiles({ files: ["./public/**", "./assets/**"] }),
|
||||
syncEnvVars(async (ctx) => [...envVars]),
|
||||
];
|
||||
```
|
||||
|
||||
### AI/ML Processing
|
||||
|
||||
```ts
|
||||
extensions: [
|
||||
pythonExtension({
|
||||
scripts: ["./ai/**/*.py"],
|
||||
requirementsFile: "./requirements.txt",
|
||||
}),
|
||||
ffmpeg({ version: "7" }),
|
||||
additionalPackages({ packages: ["wrangler"] }),
|
||||
];
|
||||
```
|
||||
|
||||
### Web Scraping
|
||||
|
||||
```ts
|
||||
extensions: [
|
||||
playwright({ browsers: ["chromium"] }),
|
||||
puppeteer(),
|
||||
additionalFiles({ files: ["./selectors.json", "./proxies.txt"] }),
|
||||
];
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Use specific versions**: Pin extension versions for reproducible builds
|
||||
- **External packages**: Add modules with native addons to the `build.external` array
|
||||
- **Environment sync**: Use `syncEnvVars` for dynamic secrets
|
||||
- **File paths**: Use glob patterns for flexible file inclusion
|
||||
- **Debug builds**: Use `--log-level debug --dry-run` for troubleshooting
|
||||
|
||||
Extensions only affect deployment, not local development. Use `external` array for packages that shouldn't be bundled.
|
||||
@@ -0,0 +1,244 @@
|
||||
# Trigger.dev Realtime
|
||||
|
||||
**Real-time monitoring and updates for runs**
|
||||
|
||||
## Core Concepts
|
||||
|
||||
Realtime allows you to:
|
||||
|
||||
- Subscribe to run status changes, metadata updates, and streams
|
||||
- Build real-time dashboards and UI updates
|
||||
- Monitor task progress from frontend and backend
|
||||
|
||||
## Authentication
|
||||
|
||||
### Public Access Tokens
|
||||
|
||||
```ts
|
||||
import { auth } from "@trigger.dev/sdk";
|
||||
|
||||
// Read-only token for specific runs
|
||||
const publicToken = await auth.createPublicToken({
|
||||
scopes: {
|
||||
read: {
|
||||
runs: ["run_123", "run_456"],
|
||||
tasks: ["my-task-1", "my-task-2"],
|
||||
},
|
||||
},
|
||||
expirationTime: "1h", // Default: 15 minutes
|
||||
});
|
||||
```
|
||||
|
||||
### Trigger Tokens (Frontend only)
|
||||
|
||||
```ts
|
||||
// Single-use token for triggering tasks
|
||||
const triggerToken = await auth.createTriggerPublicToken("my-task", {
|
||||
expirationTime: "30m",
|
||||
});
|
||||
```
|
||||
|
||||
## Backend Usage
|
||||
|
||||
### Subscribe to Runs
|
||||
|
||||
```ts
|
||||
import { runs, tasks } from "@trigger.dev/sdk";
|
||||
|
||||
// Trigger and subscribe
|
||||
const handle = await tasks.trigger("my-task", { data: "value" });
|
||||
|
||||
// Subscribe to specific run
|
||||
for await (const run of runs.subscribeToRun<typeof myTask>(handle.id)) {
|
||||
console.log(`Status: ${run.status}, Progress: ${run.metadata?.progress}`);
|
||||
if (run.status === "COMPLETED") break;
|
||||
}
|
||||
|
||||
// Subscribe to runs with tag
|
||||
for await (const run of runs.subscribeToRunsWithTag("user-123")) {
|
||||
console.log(`Tagged run ${run.id}: ${run.status}`);
|
||||
}
|
||||
|
||||
// Subscribe to batch
|
||||
for await (const run of runs.subscribeToBatch(batchId)) {
|
||||
console.log(`Batch run ${run.id}: ${run.status}`);
|
||||
}
|
||||
```
|
||||
|
||||
### Realtime Streams v2
|
||||
|
||||
```ts
|
||||
import { streams, InferStreamType } from "@trigger.dev/sdk";
|
||||
|
||||
// 1. Define streams (shared location)
|
||||
export const aiStream = streams.define<string>({
|
||||
id: "ai-output",
|
||||
});
|
||||
|
||||
export type AIStreamPart = InferStreamType<typeof aiStream>;
|
||||
|
||||
// 2. Pipe from task
|
||||
export const streamingTask = task({
|
||||
id: "streaming-task",
|
||||
run: async (payload) => {
|
||||
const completion = await openai.chat.completions.create({
|
||||
model: "gpt-4",
|
||||
messages: [{ role: "user", content: payload.prompt }],
|
||||
stream: true,
|
||||
});
|
||||
|
||||
const { waitUntilComplete } = aiStream.pipe(completion);
|
||||
await waitUntilComplete();
|
||||
},
|
||||
});
|
||||
|
||||
// 3. Read from backend
|
||||
const stream = await aiStream.read(runId, {
|
||||
timeoutInSeconds: 300,
|
||||
startIndex: 0, // Resume from specific chunk
|
||||
});
|
||||
|
||||
for await (const chunk of stream) {
|
||||
console.log("Chunk:", chunk); // Fully typed
|
||||
}
|
||||
```
|
||||
|
||||
## React Frontend Usage
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
npm add @trigger.dev/react-hooks
|
||||
```
|
||||
|
||||
### Triggering Tasks
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
import { useTaskTrigger, useRealtimeTaskTrigger } from "@trigger.dev/react-hooks";
|
||||
import type { myTask } from "../trigger/tasks";
|
||||
|
||||
function TriggerComponent({ accessToken }: { accessToken: string }) {
|
||||
// Basic trigger
|
||||
const { submit, handle, isLoading } = useTaskTrigger<typeof myTask>("my-task", {
|
||||
accessToken,
|
||||
});
|
||||
|
||||
// Trigger with realtime updates
|
||||
const {
|
||||
submit: realtimeSubmit,
|
||||
run,
|
||||
isLoading: isRealtimeLoading,
|
||||
} = useRealtimeTaskTrigger<typeof myTask>("my-task", { accessToken });
|
||||
|
||||
return (
|
||||
<div>
|
||||
<button onClick={() => submit({ data: "value" })} disabled={isLoading}>
|
||||
Trigger Task
|
||||
</button>
|
||||
|
||||
<button onClick={() => realtimeSubmit({ data: "realtime" })} disabled={isRealtimeLoading}>
|
||||
Trigger with Realtime
|
||||
</button>
|
||||
|
||||
{run && <div>Status: {run.status}</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### Subscribing to Runs
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
import { useRealtimeRun, useRealtimeRunsWithTag } from "@trigger.dev/react-hooks";
|
||||
import type { myTask } from "../trigger/tasks";
|
||||
|
||||
function SubscribeComponent({ runId, accessToken }: { runId: string; accessToken: string }) {
|
||||
// Subscribe to specific run
|
||||
const { run, error } = useRealtimeRun<typeof myTask>(runId, {
|
||||
accessToken,
|
||||
onComplete: (run) => {
|
||||
console.log("Task completed:", run.output);
|
||||
},
|
||||
});
|
||||
|
||||
// Subscribe to tagged runs
|
||||
const { runs } = useRealtimeRunsWithTag("user-123", { accessToken });
|
||||
|
||||
if (error) return <div>Error: {error.message}</div>;
|
||||
if (!run) return <div>Loading...</div>;
|
||||
|
||||
return (
|
||||
<div>
|
||||
<div>Status: {run.status}</div>
|
||||
<div>Progress: {run.metadata?.progress || 0}%</div>
|
||||
{run.output && <div>Result: {JSON.stringify(run.output)}</div>}
|
||||
|
||||
<h3>Tagged Runs:</h3>
|
||||
{runs.map((r) => (
|
||||
<div key={r.id}>
|
||||
{r.id}: {r.status}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### Realtime Streams with React
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
import { useRealtimeStream } from "@trigger.dev/react-hooks";
|
||||
import { aiStream } from "../trigger/streams";
|
||||
|
||||
function StreamComponent({ runId, accessToken }: { runId: string; accessToken: string }) {
|
||||
// Pass defined stream directly for type safety
|
||||
const { parts, error } = useRealtimeStream(aiStream, runId, {
|
||||
accessToken,
|
||||
timeoutInSeconds: 300,
|
||||
throttleInMs: 50, // Control re-render frequency
|
||||
});
|
||||
|
||||
if (error) return <div>Error: {error.message}</div>;
|
||||
if (!parts) return <div>Loading...</div>;
|
||||
|
||||
const text = parts.join(""); // parts is typed as AIStreamPart[]
|
||||
|
||||
return <div>Streamed Text: {text}</div>;
|
||||
}
|
||||
```
|
||||
|
||||
### Wait Tokens
|
||||
|
||||
```tsx
|
||||
"use client";
|
||||
import { useWaitToken } from "@trigger.dev/react-hooks";
|
||||
|
||||
function WaitTokenComponent({ tokenId, accessToken }: { tokenId: string; accessToken: string }) {
|
||||
const { complete } = useWaitToken(tokenId, { accessToken });
|
||||
|
||||
return <button onClick={() => complete({ approved: true })}>Approve Task</button>;
|
||||
}
|
||||
```
|
||||
|
||||
## Run Object Properties
|
||||
|
||||
Key properties available in run subscriptions:
|
||||
|
||||
- `id`: Unique run identifier
|
||||
- `status`: `QUEUED`, `EXECUTING`, `COMPLETED`, `FAILED`, `CANCELED`, etc.
|
||||
- `payload`: Task input data (typed)
|
||||
- `output`: Task result (typed, when completed)
|
||||
- `metadata`: Real-time updatable data
|
||||
- `createdAt`, `updatedAt`: Timestamps
|
||||
- `costInCents`: Execution cost
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Use Realtime over SWR**: Recommended for most use cases due to rate limits
|
||||
- **Scope tokens properly**: Only grant necessary read/trigger permissions
|
||||
- **Handle errors**: Always check for errors in hooks and subscriptions
|
||||
- **Type safety**: Use task types for proper payload/output typing
|
||||
- **Cleanup subscriptions**: Backend subscriptions auto-complete, frontend hooks auto-cleanup
|
||||
@@ -0,0 +1,113 @@
|
||||
# Scheduled Tasks (Cron)
|
||||
|
||||
Recurring tasks using cron. For one-off future runs, use the **delay** option.
|
||||
|
||||
## Define a Scheduled Task
|
||||
|
||||
```ts
|
||||
import { schedules } from "@trigger.dev/sdk";
|
||||
|
||||
export const task = schedules.task({
|
||||
id: "first-scheduled-task",
|
||||
run: async (payload) => {
|
||||
payload.timestamp; // Date (scheduled time, UTC)
|
||||
payload.lastTimestamp; // Date | undefined
|
||||
payload.timezone; // IANA, e.g. "America/New_York" (default "UTC")
|
||||
payload.scheduleId; // string
|
||||
payload.externalId; // string | undefined
|
||||
payload.upcoming; // Date[]
|
||||
|
||||
payload.timestamp.toLocaleString("en-US", { timeZone: payload.timezone });
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> Scheduled tasks need at least one schedule attached to run.
|
||||
|
||||
## Attach Schedules
|
||||
|
||||
**Declarative (sync on dev/deploy):**
|
||||
|
||||
```ts
|
||||
schedules.task({
|
||||
id: "every-2h",
|
||||
cron: "0 */2 * * *", // UTC
|
||||
run: async () => {},
|
||||
});
|
||||
|
||||
schedules.task({
|
||||
id: "tokyo-5am",
|
||||
cron: { pattern: "0 5 * * *", timezone: "Asia/Tokyo", environments: ["PRODUCTION", "STAGING"] },
|
||||
run: async () => {},
|
||||
});
|
||||
```
|
||||
|
||||
**Imperative (SDK or dashboard):**
|
||||
|
||||
```ts
|
||||
await schedules.create({
|
||||
task: task.id,
|
||||
cron: "0 0 * * *",
|
||||
timezone: "America/New_York", // DST-aware
|
||||
externalId: "user_123",
|
||||
deduplicationKey: "user_123-daily", // updates if reused
|
||||
});
|
||||
```
|
||||
|
||||
### Dynamic / Multi-tenant Example
|
||||
|
||||
```ts
|
||||
// /trigger/reminder.ts
|
||||
export const reminderTask = schedules.task({
|
||||
id: "todo-reminder",
|
||||
run: async (p) => {
|
||||
if (!p.externalId) throw new Error("externalId is required");
|
||||
const user = await db.getUser(p.externalId);
|
||||
await sendReminderEmail(user);
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
// app/reminders/route.ts
|
||||
export async function POST(req: Request) {
|
||||
const data = await req.json();
|
||||
return Response.json(
|
||||
await schedules.create({
|
||||
task: reminderTask.id,
|
||||
cron: "0 8 * * *",
|
||||
timezone: data.timezone,
|
||||
externalId: data.userId,
|
||||
deduplicationKey: `${data.userId}-reminder`,
|
||||
})
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Cron Syntax (no seconds)
|
||||
|
||||
```
|
||||
* * * * *
|
||||
| | | | └ day of week (0–7 or 1L–7L; 0/7=Sun; L=last)
|
||||
| | | └── month (1–12)
|
||||
| | └──── day of month (1–31 or L)
|
||||
| └────── hour (0–23)
|
||||
└──────── minute (0–59)
|
||||
```
|
||||
|
||||
## When Schedules Won't Trigger
|
||||
|
||||
- **Dev:** only when the dev CLI is running.
|
||||
- **Staging/Production:** only for tasks in the **latest deployment**.
|
||||
|
||||
## SDK Management
|
||||
|
||||
```ts
|
||||
await schedules.retrieve(id);
|
||||
await schedules.list();
|
||||
await schedules.update(id, { cron: "0 0 1 * *", externalId: "ext", deduplicationKey: "key" });
|
||||
await schedules.deactivate(id);
|
||||
await schedules.activate(id);
|
||||
await schedules.del(id);
|
||||
await schedules.timezones(); // list of IANA timezones
|
||||
```
|
||||
Reference in New Issue
Block a user