Retries and backoff

A retry policy is two things: how many attempts a job gets in total, and how long to wait before each one. Both are configured in the attribute, on the queue or on the job.

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Queues)]
enum AppQueues {
    #[queue(retry(max_attempts = 3, backoff = "exponential", base = "1s", max = "2m"))]
    Images,
}

#[derive(Debug, Serialize, Deserialize, Job)]
#[job(queue = AppQueues::Images, retry(max_attempts = 5))]
struct ResizeImage { path: String }

The grammar

retry(
    max_attempts = 3,            // total attempts including the first; 1 means no retries
    backoff = "exponential",     // "none" | "fixed" | "exponential" (default)
    delay = "1s",                // fixed only, required
    base = "1s",                 // exponential only, default "1s", must be <= max
    factor = 2.0,                // exponential only, default 2.0
    max = "5m",                  // exponential only, default "5m"
    jitter = true,               // exponential only, default true
)

retry() with an empty body is valid and means “every default”: three attempts, exponential, base 1s, factor 2.0, capped at 5m, with jitter.

Durations are string literals parsed at compile time: an integer followed by ms, s, m, h or d. A bare integer means seconds. "500ms", "30s", "2 m" and "30" are all valid. Zero is rejected everywhere a duration is accepted.

Precedence

1. The job’s own retry(...), if it has one.
2. Otherwise the queue’s retry(...), if it has one.
3. Otherwise no retries: the job is dead-lettered on its first failure.

No retry attribute means no retries

This is the part that surprises people. A queue with no retry(...) gets RetryPolicy::default(), which is max_attempts: 1 and Backoff::None. One retryable failure and the job is in the dead-letter queue. Put a policy on the queue if you want a floor for everything on it.

The effective policy is resolved per delivery, so a job can override its queue in either direction: more attempts than the queue allows, fewer, or a completely different backoff shape.

How long each wait is

Exponential backoff waits min(max, base * factor^(attempt - 1)) before the next attempt. With jitter, the wait is drawn uniformly from [0, computed] — full jitter, not a small random offset.

With the defaults, and jitter off so the numbers are legible:

Failed attempt	Computed wait
1	1s
2	2s
3	4s
4	8s
9	256s
10 and later	300s (the cap)

Jitter matters more than it looks. Without it, a downstream outage that fails a thousand jobs at once produces a thousand retries at the same instant, then again two seconds later. Full jitter spreads them across the whole interval, which is why it is the default.

Choosing a policy

A flaky network call: exponential with jitter, a handful of attempts, capped somewhere below how long you are willing to wait for the work. retry(max_attempts = 5) and the defaults are a reasonable starting point.

A rate-limited API: not a retry at all. The API told you how long to wait, so defer for exactly that long instead of guessing with a backoff.

An operation that must not be repeated quickly: backoff = "fixed" with a delay long enough that the second attempt is meaningfully different from the first.

Work where a failure is permanent: no retry policy, and return JobError::fatal from the handler so it does not even spend the first retry.

Running out of attempts

RetryPolicy::decide(failed_attempt) returns GiveUp when failed_attempt >= max_attempts, and the worker then dead-letters the job with a reason naming both numbers:

max attempts (2) exhausted after attempt 2: job failed (retryable): service is down

That string lands in the dead-letter queue’s x-death-reason header, and in MemoryBackend::dead_letters() in tests.

A retry is not a new job

The envelope is re-published with attempt incremented and the same job_id, so a job that succeeds on its third attempt has one id and three deliveries. Because each retry acks the envelope it replaces, MemoryBackend::acked() counts attempts rather than jobs, which is what makes retry behaviour assertable:

assert_eq!(h.log.attempts_of("flaky@example.com"), vec![1, 2, 3]);

let acked = h.backend.acked(EMAILS);
assert_eq!(acked.iter().map(|e| e.attempt).collect::<Vec<_>>(), vec![1, 2, 3]);
assert!(acked.iter().all(|e| e.job_id == id), "same job id throughout");

A retry also resets the message priority to 0. A deferral earns a place at the front of the queue for exactly one round; a retry does not.

Where the waiting happens

The waiting is done by the broker, not by the worker: the envelope is published into a hold queue whose TTL is the delay, and the broker dead-letters it back onto the work queue when the TTL expires. The worker holds nothing and consumes no concurrency slot while a job is waiting, and a worker restart does not lose the retry.

One consequence worth knowing: delays are rounded up to retry_granularity, 1 second by default, so each distinct delay gets its own hold queue. Exponential backoff with jitter produces a different delay every time, so this is the knob that bounds how many hold queues a busy failing queue can have at once. With a five-minute cap, 1s allows up to 300 of them and 10s up to 30. See Broker topology.

The arithmetic never panics

Backoff::delay_for is written to be total. failed_attempt is clamped to at least 1, the exponentiation happens in f64 and saturates to max on overflow, a non-finite or non-positive factor degenerates to a constant base, and a zero base or max yields Duration::ZERO. The result is always less than or equal to max.

Configuring a policy in code

Most code uses the attribute, but RetryPolicy is an ordinary type, which is what tests and hand-built QueueConfigs use:

RetryPolicy::none()                                   // 1 attempt, Backoff::None
RetryPolicy::exponential(5)                           // 5 attempts, Backoff::exponential()
RetryPolicy::fixed(3, Duration::from_millis(500))     // 3 attempts, constant 500ms
RetryPolicy::new(4, Backoff::Exponential {
    base: Duration::from_secs(2),
    factor: 3.0,
    max: Duration::from_secs(60),
    jitter: true,
})

Signatures are in RetryPolicy and Backoff.