When an uncaught error is thrown inside your task, that task attempt will fail.
You can configure retrying in two ways:
- In your trigger.config file you can set the default retrying behavior for all tasks.
- On each task you can set the retrying behavior.
By default when you create your project using the CLI init command we disabled retrying in the DEV
environment. You can enable it in your trigger.config file.
A simple example with OpenAI
This task will retry 10 times with exponential backoff.
openai.chat.completions.create() can throw an error.- The result can be empty and we want to try again. So we manually throw an error.
import { task } from "@trigger.dev/sdk/v3";
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
export const openaiTask = task({
id: "openai-task",
retry: {
maxAttempts: 10,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
run: async (payload: { prompt: string }) => {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
if (chatCompletion.choices[0]?.message.content === undefined) {
throw new Error("OpenAI call failed");
}
return chatCompletion.choices[0].message.content;
},
});
Combining tasks
One way to gain reliability is to break your work into smaller tasks and trigger them from each other. Each task can have its own retrying behavior:
/trigger/multiple-tasks.ts
export const myTask = task({
id: "my-task",
retry: {
maxAttempts: 10,
},
run: async (payload: string) => {
const result = await otherTask.triggerAndWait("some data");
},
});
export const otherTask = task({
id: "other-task",
retry: {
maxAttempts: 5,
},
run: async (payload: string) => {
return {
foo: "bar",
};
},
});
Another benefit of this approach is that you can view the logs and retry each task independently from the dashboard.
Retrying smaller parts of a task
Another complimentary strategy is to perform retrying inside of your task.
We provide some useful functions that you can use to retry smaller parts of a task. Of course, you can also write your own logic or use other packages.
retry.onThrow()
You can retry a block of code that can throw an error, with the same retry settings as a task.
/trigger/retry-on-throw.ts
export const retryOnThrow = task({
id: "retry-on-throw",
run: async (payload: any) => {
const result = await retry.onThrow(
async ({ attempt }) => {
if (attempt < 3) throw new Error("failed");
return {
foo: "bar",
};
},
{ maxAttempts: 3, randomize: false }
);
logger.info("Result", { result });
},
});
If all of the attempts with retry.onThrow fail, an error will be thrown. You can catch this or
let it cause a retry of the entire task.
retry.fetch()
You can use fetch, axios, or any other library in your code.
But we do provide a convenient function to perform HTTP requests with conditional retrying based on the response:
export const taskWithFetchRetries = task({
id: "task-with-fetch-retries",
run: async ({ payload, ctx }) => {
const headersResponse = await retry.fetch("http://my.host/test-headers", {
retry: {
byStatus: {
"429": {
strategy: "headers",
limitHeader: "x-ratelimit-limit",
remainingHeader: "x-ratelimit-remaining",
resetHeader: "x-ratelimit-reset",
resetFormat: "unix_timestamp_in_ms",
},
},
},
});
const json = await headersResponse.json();
logger.info("Fetched headers response", { json });
const backoffResponse = await retry.fetch("http://my.host/test-backoff", {
retry: {
byStatus: {
"500-599": {
strategy: "backoff",
maxAttempts: 10,
factor: 2,
minTimeoutInMs: 1_000,
maxTimeoutInMs: 30_000,
randomize: false,
},
},
},
});
const json2 = await backoffResponse.json();
logger.info("Fetched backoff response", { json2 });
const timeoutResponse = await retry.fetch("https://httpbin.org/delay/2", {
timeoutInMs: 1000,
retry: {
timeout: {
maxAttempts: 5,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
},
});
const json3 = await timeoutResponse.json();
logger.info("Fetched timeout response", { json3 });
return {
result: "success",
payload,
json,
json2,
json3,
};
},
});
If all of the attempts with retry.fetch fail, an error will be thrown. You can catch this or let
it cause a retry of the entire task.
Advanced error handling and retrying
We provide a handleError callback on the task and in your trigger.config file. This gets called when an uncaught error is thrown in your task.
You can
- Inspect the error, log it, and return a different error if you’d like.
- Modify the retrying behavior based on the error, payload, context, etc.
If you don’t return anything from the function it will use the settings on the task (or inherited from the config). So you only need to use this to override things.
OpenAI error handling example
OpenAI calls can fail for a lot of reasons and the ideal retry behavior is different for each.
In this complicated example:
- We skip retrying if there’s no Response status.
- We skip retrying if you’ve run out of credits.
- If there are no Response headers we let the normal retrying logic handle it (return undefined).
- If we’ve run out of requests or tokens we retry at the time specified in the headers.
export const openaiTask = task({
id: "openai-task",
retry: {
maxAttempts: 1,
},
run: async (payload: { prompt: string }) => {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
return chatCompletion.choices[0].message.content;
},
handleError: async (payload, error, { ctx, retryAt }) => {
if (error instanceof OpenAI.APIError) {
if (!error.status) {
return {
skipRetrying: true,
};
}
if (error.status === 429 && error.type === "insufficient_quota") {
return {
skipRetrying: true,
};
}
if (!error.headers) {
return;
}
const remainingRequests = error.headers["x-ratelimit-remaining-requests"];
const requestResets = error.headers["x-ratelimit-reset-requests"];
if (typeof remainingRequests === "string" && Number(remainingRequests) === 0) {
return {
retryAt: calculateISO8601DurationOpenAIVariantResetAt(requestResets),
};
}
const remainingTokens = error.headers["x-ratelimit-remaining-tokens"];
const tokensResets = error.headers["x-ratelimit-reset-tokens"];
if (typeof remainingTokens === "string" && Number(remainingTokens) === 0) {
return {
retryAt: calculateISO8601DurationOpenAIVariantResetAt(tokensResets),
};
}
}
},
});
Using try/catch to prevent retries
Sometimes you want to catch an error and don’t want to retry the task. You can use try/catch as you normally would. In this example we fallback to using Replicate if OpenAI fails.
import { task } from "@trigger.dev/sdk/v3";
export const openaiTask = task({
id: "openai-task",
run: async (payload: { prompt: string }) => {
try {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
if (chatCompletion.choices[0]?.message.content === undefined) {
throw new Error("OpenAI call failed");
}
return chatCompletion.choices[0].message.content;
} catch (error) {
const prediction = await replicate.run(
"meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
{
input: {
prompt: payload.prompt,
max_new_tokens: 250,
},
}
);
if (prediction.output === undefined) {
throw new Error("Replicate call failed");
}
return prediction.output;
}
},
});
