wehub-resource-sync
384 lines
12 KiB
Plaintext
384 lines
12 KiB
Plaintext
---
|
|
title: "Errors & Retrying"
|
|
description: "How to deal with errors and write reliable tasks."
|
|
---
|
|
|
|
import OpenaiRetry from "/snippets/code/openai-retry.mdx";
|
|
|
|
When an uncaught error is thrown inside your task, that task attempt will fail.
|
|
|
|
You can configure retrying in two ways:
|
|
|
|
1. In your [trigger.config file](/config/config-file) you can set the default retrying behavior for all tasks.
|
|
2. On each task you can set the retrying behavior.
|
|
|
|
<Note>Task-level retry settings override the defaults in your `trigger.config` file.</Note>
|
|
|
|
<Note>
|
|
By default when you create your project using the CLI init command we disabled retrying in the DEV
|
|
environment. You can enable it in your [trigger.config file](/config/config-file).
|
|
</Note>
|
|
|
|
## A simple example with OpenAI
|
|
|
|
This task will retry 10 times with exponential backoff.
|
|
|
|
- `openai.chat.completions.create()` can throw an error.
|
|
- The result can be empty and we want to try again. So we manually throw an error.
|
|
|
|
<OpenaiRetry />
|
|
|
|
## Combining tasks
|
|
|
|
One way to gain reliability is to break your work into smaller tasks and [trigger](/triggering) them from each other. Each task can have its own retrying behavior:
|
|
|
|
```ts /trigger/multiple-tasks.ts
|
|
import { task } from "@trigger.dev/sdk";
|
|
|
|
export const myTask = task({
|
|
id: "my-task",
|
|
retry: {
|
|
maxAttempts: 10,
|
|
},
|
|
run: async (payload: string) => {
|
|
const result = await otherTask.triggerAndWait("some data");
|
|
//...do other stuff
|
|
},
|
|
});
|
|
|
|
export const otherTask = task({
|
|
id: "other-task",
|
|
retry: {
|
|
maxAttempts: 5,
|
|
},
|
|
run: async (payload: string) => {
|
|
return {
|
|
foo: "bar",
|
|
};
|
|
},
|
|
});
|
|
```
|
|
|
|
Another benefit of this approach is that you can view the logs and retry each task independently from the dashboard.
|
|
|
|
## Retrying smaller parts of a task
|
|
|
|
Another complimentary strategy is to perform retrying inside of your task.
|
|
|
|
We provide some useful functions that you can use to retry smaller parts of a task. Of course, you can also write your own logic or use other packages.
|
|
|
|
### retry.onThrow()
|
|
|
|
You can retry a block of code that can throw an error, with the same retry settings as a task.
|
|
|
|
```ts /trigger/retry-on-throw.ts
|
|
import { task, logger, retry } from "@trigger.dev/sdk";
|
|
|
|
export const retryOnThrow = task({
|
|
id: "retry-on-throw",
|
|
run: async (payload: any) => {
|
|
//Will retry up to 3 times. If it fails 3 times it will throw.
|
|
const result = await retry.onThrow(
|
|
async ({ attempt }) => {
|
|
//throw on purpose the first 2 times, obviously this is a contrived example
|
|
if (attempt < 3) throw new Error("failed");
|
|
//...
|
|
return {
|
|
foo: "bar",
|
|
};
|
|
},
|
|
{ maxAttempts: 3, randomize: false }
|
|
);
|
|
|
|
//this will log out after 3 attempts of retry.onThrow
|
|
logger.info("Result", { result });
|
|
},
|
|
});
|
|
```
|
|
|
|
<Note>
|
|
If all of the attempts with `retry.onThrow` fail, an error will be thrown. You can catch this or
|
|
let it cause a retry of the entire task.
|
|
</Note>
|
|
|
|
### retry.fetch()
|
|
|
|
You can use `fetch`, `axios`, or any other library in your code.
|
|
|
|
But we do provide a convenient function to perform HTTP requests with conditional retrying based on the response:
|
|
|
|
```ts /trigger/retry-fetch.ts
|
|
import { task, logger, retry } from "@trigger.dev/sdk";
|
|
|
|
export const taskWithFetchRetries = task({
|
|
id: "task-with-fetch-retries",
|
|
run: async ({ payload, ctx }) => {
|
|
//if the Response is a 429 (too many requests), it will retry using the data from the response. A lot of good APIs send these headers.
|
|
const headersResponse = await retry.fetch("http://my.host/test-headers", {
|
|
retry: {
|
|
byStatus: {
|
|
"429": {
|
|
strategy: "headers",
|
|
limitHeader: "x-ratelimit-limit",
|
|
remainingHeader: "x-ratelimit-remaining",
|
|
resetHeader: "x-ratelimit-reset",
|
|
resetFormat: "unix_timestamp_in_ms",
|
|
},
|
|
},
|
|
},
|
|
});
|
|
const json = await headersResponse.json();
|
|
logger.info("Fetched headers response", { json });
|
|
|
|
//if the Response is a 500-599 (issue with the server you're calling), it will retry up to 10 times with exponential backoff
|
|
const backoffResponse = await retry.fetch("http://my.host/test-backoff", {
|
|
retry: {
|
|
byStatus: {
|
|
"500-599": {
|
|
strategy: "backoff",
|
|
maxAttempts: 10,
|
|
factor: 2,
|
|
minTimeoutInMs: 1_000,
|
|
maxTimeoutInMs: 30_000,
|
|
randomize: false,
|
|
},
|
|
},
|
|
},
|
|
});
|
|
const json2 = await backoffResponse.json();
|
|
logger.info("Fetched backoff response", { json2 });
|
|
|
|
//You can additionally specify a timeout. In this case if the response takes longer than 1 second, it will retry up to 5 times with exponential backoff
|
|
const timeoutResponse = await retry.fetch("https://httpbin.org/delay/2", {
|
|
timeoutInMs: 1000,
|
|
retry: {
|
|
timeout: {
|
|
maxAttempts: 5,
|
|
factor: 1.8,
|
|
minTimeoutInMs: 500,
|
|
maxTimeoutInMs: 30_000,
|
|
randomize: false,
|
|
},
|
|
},
|
|
});
|
|
const json3 = await timeoutResponse.json();
|
|
logger.info("Fetched timeout response", { json3 });
|
|
|
|
return {
|
|
result: "success",
|
|
payload,
|
|
json,
|
|
json2,
|
|
json3,
|
|
};
|
|
},
|
|
});
|
|
```
|
|
|
|
<Note>
|
|
If all of the attempts with `retry.fetch` fail, an error will be thrown. You can catch this or let
|
|
it cause a retry of the entire task.
|
|
</Note>
|
|
|
|
## Advanced error handling and retrying
|
|
|
|
We provide a `catchError` callback on the task and in your `trigger.config` file. This gets called when an uncaught error is thrown in your task.
|
|
|
|
You can
|
|
|
|
- Inspect the error, log it, and return a different error if you'd like.
|
|
- Modify the retrying behavior based on the error, payload, context, etc.
|
|
|
|
If you don't return anything from the function it will use the settings on the task (or inherited from the config). So you only need to use this to override things.
|
|
|
|
### OpenAI error handling example
|
|
|
|
OpenAI calls can fail for a lot of reasons and the ideal retry behavior is different for each.
|
|
|
|
In this complicated example:
|
|
|
|
- We skip retrying if there's no Response status.
|
|
- We skip retrying if you've run out of credits.
|
|
- If there are no Response headers we let the normal retrying logic handle it (return undefined).
|
|
- If we've run out of requests or tokens we retry at the time specified in the headers.
|
|
|
|
<CodeGroup>
|
|
|
|
```ts tasks.ts
|
|
import { task } from "@trigger.dev/sdk";
|
|
import { calculateISO8601DurationOpenAIVariantResetAt, openai } from "./openai.js";
|
|
|
|
export const openaiTask = task({
|
|
id: "openai-task",
|
|
retry: {
|
|
maxAttempts: 1,
|
|
},
|
|
run: async (payload: { prompt: string }) => {
|
|
const chatCompletion = await openai.chat.completions.create({
|
|
messages: [{ role: "user", content: payload.prompt }],
|
|
model: "gpt-3.5-turbo",
|
|
});
|
|
|
|
return chatCompletion.choices[0].message.content;
|
|
},
|
|
catchError: async ({ payload, error, ctx, retryAt }) => {
|
|
if (error instanceof OpenAI.APIError) {
|
|
if (!error.status) {
|
|
return {
|
|
skipRetrying: true,
|
|
};
|
|
}
|
|
|
|
if (error.status === 429 && error.type === "insufficient_quota") {
|
|
return {
|
|
skipRetrying: true,
|
|
};
|
|
}
|
|
|
|
if (!error.headers) {
|
|
//returning undefined means the normal retrying logic will be used
|
|
return;
|
|
}
|
|
|
|
const remainingRequests = error.headers["x-ratelimit-remaining-requests"];
|
|
const requestResets = error.headers["x-ratelimit-reset-requests"];
|
|
|
|
if (typeof remainingRequests === "string" && Number(remainingRequests) === 0) {
|
|
return {
|
|
retryAt: calculateISO8601DurationOpenAIVariantResetAt(requestResets),
|
|
};
|
|
}
|
|
|
|
const remainingTokens = error.headers["x-ratelimit-remaining-tokens"];
|
|
const tokensResets = error.headers["x-ratelimit-reset-tokens"];
|
|
|
|
if (typeof remainingTokens === "string" && Number(remainingTokens) === 0) {
|
|
return {
|
|
retryAt: calculateISO8601DurationOpenAIVariantResetAt(tokensResets),
|
|
};
|
|
}
|
|
}
|
|
},
|
|
});
|
|
```
|
|
|
|
```ts openai.ts
|
|
import { OpenAI } from "openai";
|
|
|
|
export const openai = new OpenAI({ apiKey: env.OPENAI_API_KEY });
|
|
|
|
export function calculateISO8601DurationOpenAIVariantResetAt(
|
|
resets: string,
|
|
now: Date = new Date()
|
|
): Date | undefined {
|
|
// Check if the input is null or undefined
|
|
if (!resets) return undefined;
|
|
|
|
// Regular expression to match the duration string pattern
|
|
const pattern = /^(?:(\d+)d)?(?:(\d+)h)?(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?(?:(\d+)ms)?$/;
|
|
const match = resets.match(pattern);
|
|
|
|
// If the string doesn't match the expected format, return undefined
|
|
if (!match) return undefined;
|
|
|
|
// Extract days, hours, minutes, seconds, and milliseconds from the string
|
|
const days = parseInt(match[1] ?? "0", 10) || 0;
|
|
const hours = parseInt(match[2] ?? "0", 10) || 0;
|
|
const minutes = parseInt(match[3] ?? "0", 10) || 0;
|
|
const seconds = parseFloat(match[4] ?? "0") || 0;
|
|
const milliseconds = parseInt(match[5] ?? "0", 10) || 0;
|
|
|
|
// Calculate the future date based on the current date plus the extracted time
|
|
const resetAt = new Date(now);
|
|
resetAt.setDate(resetAt.getDate() + days);
|
|
resetAt.setHours(resetAt.getHours() + hours);
|
|
resetAt.setMinutes(resetAt.getMinutes() + minutes);
|
|
resetAt.setSeconds(resetAt.getSeconds() + Math.floor(seconds));
|
|
resetAt.setMilliseconds(
|
|
resetAt.getMilliseconds() + (seconds - Math.floor(seconds)) * 1000 + milliseconds
|
|
);
|
|
|
|
return resetAt;
|
|
}
|
|
```
|
|
|
|
</CodeGroup>
|
|
|
|
## Preventing retries
|
|
|
|
### Using `AbortTaskRunError`
|
|
|
|
You can prevent retries by throwing an `AbortTaskRunError`. This will fail the task attempt and disable retrying.
|
|
|
|
```ts /trigger/myTasks.ts
|
|
import { task, AbortTaskRunError } from "@trigger.dev/sdk";
|
|
|
|
export const openaiTask = task({
|
|
id: "openai-task",
|
|
run: async (payload: { prompt: string }) => {
|
|
//if this fails, it will throw an error and stop retrying
|
|
const chatCompletion = await openai.chat.completions.create({
|
|
messages: [{ role: "user", content: payload.prompt }],
|
|
model: "gpt-3.5-turbo",
|
|
});
|
|
|
|
if (chatCompletion.choices[0]?.message.content === undefined) {
|
|
// If OpenAI returns an empty response, abort retrying
|
|
throw new AbortTaskRunError("OpenAI call failed");
|
|
}
|
|
|
|
return chatCompletion.choices[0].message.content;
|
|
},
|
|
});
|
|
```
|
|
|
|
### Using try/catch
|
|
|
|
Sometimes you want to catch an error and don't want to retry the task. You can use try/catch as you normally would. In this example we fallback to using Replicate if OpenAI fails.
|
|
|
|
```ts /trigger/myTasks.ts
|
|
import { task } from "@trigger.dev/sdk";
|
|
|
|
export const openaiTask = task({
|
|
id: "openai-task",
|
|
run: async (payload: { prompt: string }) => {
|
|
try {
|
|
//if this fails, it will throw an error and retry
|
|
const chatCompletion = await openai.chat.completions.create({
|
|
messages: [{ role: "user", content: payload.prompt }],
|
|
model: "gpt-3.5-turbo",
|
|
});
|
|
|
|
if (chatCompletion.choices[0]?.message.content === undefined) {
|
|
//sometimes OpenAI returns an empty response, let's retry by throwing an error
|
|
throw new Error("OpenAI call failed");
|
|
}
|
|
|
|
return chatCompletion.choices[0].message.content;
|
|
} catch (error) {
|
|
//use Replicate if OpenAI fails
|
|
const prediction = await replicate.run(
|
|
"meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
|
|
{
|
|
input: {
|
|
prompt: payload.prompt,
|
|
max_new_tokens: 250,
|
|
},
|
|
}
|
|
);
|
|
|
|
if (prediction.output === undefined) {
|
|
//retry if Replicate fails
|
|
throw new Error("Replicate call failed");
|
|
}
|
|
|
|
return prediction.output;
|
|
}
|
|
},
|
|
});
|
|
```
|
|
|
|
## Replay failed runs in bulk
|
|
|
|
After you deploy a fix, use [bulk actions](/bulk-actions) to replay multiple failed runs from the dashboard, or [bulk actions with the SDK](/runs/bulk-actions) to replay them from backend code. Bulk replay creates an asynchronous action that targets selected run IDs or a `runs.list()` filter, so you can retry a known failure set without replaying each run individually.
|