Skip to content

Retries and failure

What an Err from a #[process] method means — the retry budget and its backoff, panic handling, and where a dead-lettered job goes.

The contract is small: a #[process] method returns anyhow::Result<()>, Err(_) marks the attempt failed, and Ok(()) completes the job — once the attempt’s transaction has settled, which is the one way a body that returned Ok can still fail (see below). The budget is the queue port’s, the same on every backend: a retryable failure runs again after a backoff, as the same job, until the budget is spent.

crates/features/src/audio/queue/processor.rs
#[process(queue = AudioQueue, retries = 3)]
async fn transcode(&self, job: TranscodeCommand) -> Result<()> {
self.svc.transcode(job.file).await?;
Ok(())
}
  • Ok(()) ⇒ the attempt’s transaction commits and the job is done. If the commit cannot be honoured, nothing it wrote landed and the attempt fails instead — the one case where Ok is not the end.
  • Err(e) ⇒ the attempt failed. While the budget holds another attempt, the job runs again after the backoff below, under the same job_id with attempt one higher. After retries failed retries it is dead-lettered (see Where failed jobs go).
  • A panic inside the method is caught, logged as job dead-lettered: handler panicked at error with the panic’s message, and the job dead-letters at once. The attempt’s queue.job line reports outcome="panic" beside it, under the same span_id. The worker keeps running — one bad job doesn’t stop the queue’s consumer.

The panic is caught by the queue port itself, around the handler, which is what keeps the event inside the attempt’s trace. Each delivery also runs on a task of its own, so the worker goes on to its next job.

That event is the panic’s one record: under #[nest_rs::main] the process hook stays quiet for a panic the attempt contains, so no raw thread … panicked at … line reaches stderr, and the event names where the handler panicked under panic_location (a panic is said once).

retries = N is N + 1 attempts in all: the first try and N retries. The default is 0 — the job is tried once. The budget counts attempts that answered. A delivery that ends without an answer — its process killed, frozen or cut off past its lease — spends none: the job runs again at the attempt it was at. That has its own bound, the stall limit: a job whose deliveries end that way three times in a row (STALL_LIMIT) is taking its workers down — an abort, an OOM kill, a hang until the pod is killed — and is dead-lettered without running, as job dead-lettered: its deliveries ended without an answer past the stall limit. See Crashes, and the stall limit.

After attempt n fails, the job waits min(5 min, 1 s · 2^(n − 1)) before attempt n + 1, scaled by a jitter between 0.8 and 1.2. The jitter is derived from the job’s id and n rather than drawn, so jobs a shared outage failed together come back apart, and one job waits the same on every replay. It is the one retry policy the framework runs — nest_rs::core::Backoff, the shape of gRPC’s connection backoff — and the worker’s own calls to its backend, Redis’s reconnections and retry_on_conflict wait on it too, each from a seed of its own:

After attemptWait before jitterWait
11 s0.8 – 1.2 s
22 s1.6 – 2.4 s
34 s3.2 – 4.8 s
9256 s204.8 – 307.2 s
10 and after5 min4 – 6 min

The worker says how long it chose, on the line that reports the failure:

Terminal window
WARN nest_rs::queue: job failed; will retry within the budget error=transcode failed retry_after_ms=1143 trace_id=01a0deaacf9670f8a924efd95a2ec64b span_id=a83a4d58c89c3d5d

A failure can ask for longer. A Problem carrying with_retry_after(30) — the upstream’s 429 or 503 passed on — holds the next attempt back at least that long, so the job does not spend its budget while the upstream is still refusing. The wait is the longer of the two, what it asks counts up to an hour and is stretched by up to a fifth — 30 to 36 seconds here, drawn from the job’s id like the jitter — so the jobs one 429 refused together do not all come back at the instant it named, and none comes back before it. The line adds retry_after_asked_ms=30000 beside the wait it chose.

On Redis the next attempt waits off the stream, due on Redis’s clock, until a running worker files it back — see Delayed jobs — and the worker that ran the failed attempt moves on: no permit is held while a job waits, and a shutdown during the wait loses nothing. The port counts the budget once; the backend keeps no retry count of its own.

A failure a retry cannot fix is non-retryable and dead-letters on the attempt it happens: a payload that does not deserialize, a pipe rejection, a missing provider, a client error (below). Every other Err your method returns is retryable — the budget is how many times a transient fault gets another chance.

A job that fails for a reason no retry clears says so the way an HTTP handler does, with a Problem. A 4xx is the client’s error (RFC 9110 §15.5) and dead-letters the job on the attempt it happens; 408 and 429 invite a later try and retry, as does a 5xx:

crates/features/src/audio/queue/processor.rs
#[process(queue = AudioQueue, retries = 3)]
async fn transcode(&self, job: TranscodeCommand) -> Result<()> {
if job.file.get_version_num() != 7 {
return Err(Problem::new(422, code::INVALID_ARGUMENT).into());
}
self.svc.transcode(job.file).await?;
Ok(())
}

What decides, in order:

  1. The problem an exception filter answered, when one catches the failure: a filter mapping a domain error to Problem::new(422, …) dead-letters at attempt one, one answering 503 retries.
  2. A JobError the failure carries keeps its disposition — JobError::abort(…) returned from the method dead-letters, retry retries.
  3. The method’s error, read by its type: a method returning a ToProblem error — Result<(), ServiceError> — by the problem it says (ServiceError::Conflict is a 409, so it dead-letters), and any other — an anyhow::Error above — by a Problem in its chain.
  4. Anything else retries within the budget.

A method returning Result<(), ServiceError> dead-letters a NotFound, a 404, too. A job filed at once — pushed on the pool, whatever transaction you opened yourself beside it — can run before that write commits and find nothing: push from a listener on the event the commit announces, or return the error as JobError::retry. One pushed inside the unit’s own transaction waits in the outbox for its commit (A push inside a transaction). An exception filter may also answer a failed attempt with job.acknowledge() through QueueView: the job settles as done, what it wrote rolls back, and the failure is said at warn as job failed; a layer acknowledged it, so it settles done and its writes rolled back.

How long an attempt may run, and how a long one resumes where it stopped, are on Long-running jobs; what a worker does with a job another release sealed is on Wire format.

A dead-lettered job leaves one entry on the queue’s nestrs:queue:{<queue>}:dead stream: its job id, its record as stored, and the reason — the sentence the job dead-lettered line carries: the failure and every cause beneath it, a .context(…) included, with any decode failure said without its value and a database error by its SQLSTATE and the names the server reported (database error 22P02), never by the server’s message, which quotes the input it refused:

Terminal window
$ valkey-cli XRANGE 'nestrs:queue:{audio}:dead' - +
1) 1) "1790442140102-0"
2) 1) "job"
2) "01a0deaa-cf96-70f8-a924-efd8fbbe3d38"
3) "record"
4) "{\"v\":1,…}"
5) "reason"
6) "transcode failed"

The job is the id the push returned; the job’s lines name the stream entry each attempt ran from as backend_id. The stream keeps a week of dead letters and about the last 10,000. The framework does not ship a dead-letter routing layer: the dead letters are the storage’s, and inspecting them is an operational task — filing one back, once its cause is fixed, is the one command under Dead letters.

If you need richer dead-lettering — alerting, replay, archival — catch the Err in the method and republish the payload to a dedicated queue before returning the error. A #[processor] on that queue then runs your dead-letter logic, and the pattern is the same on any backend:

crates/features/src/audio/queue/processor.rs
#[process(queue = AudioQueue, retries = 3)]
async fn transcode(&self, job: TranscodeCommand) -> Result<()> {
match self.svc.transcode(job.file).await {
Ok(_) => Ok(()),
Err(e) => {
self.svc.report_dead(&job, &e).await?;
Err(e.into())
}
}
}

Panics, deserialization errors, container misses

Section titled “Panics, deserialization errors, container misses”

Four failure modes the framework converts into the same Err path, so you don’t have to think about them separately:

  • Panics in the method body — caught, with a job dead-lettered: handler panicked event carrying the panic message. The job dead-letters immediately (regardless of remaining retries), the worker stays up.
  • Deserialization errors — the macro-emitted handler decodes the job from the payload in place. A payload that doesn’t match the method’s job type returns Err before your method even runs. Retrying won’t help: the payload shape is wrong, so it is classified non-retryable and dead-letters on the first attempt, logging job dead-lettered: non-retryable failure at error. The retries budget is not consumed. The reason names where the payload failed and what kind of value it found — invalid type: a string, expected u64 — and never the value: serde’s own sentence quotes it, and the reason lands in the log line and the dead-letter record, where a payload’s data does not belong. A decode your method does itself — serde_json::from_str(&reply)? in an anyhow::Result — is said the same way when it fails, on the retry warn, the dead-letter line and the record alike.
  • Container resolution failures — Container::get of the #[processor] host finds nothing when the host is not registered under its own type. The handler aborts naming it, and the job dead-letters on the first attempt: a configuration mistake rather than a transient, so investigate it.
  • A transaction that could not be settled — the body returned Ok, and the attempt’s transaction could not be committed, so nothing it wrote landed. The attempt fails rather than reporting writes that were never made, and the database decides whether it is retried — with the answer depending on where it failed. A statement inside the attempt failed on a serialization conflict, a deadlock, a pool that timed out, or a connection the server closed: retryable, because nothing is durable before COMMIT and the framework therefore knows the attempt wrote nothing. A constraint checked at COMMIT, or a commit whose outcome is unknown because the connection dropped mid-flight, is not — it dead-letters at once, because that one may have landed and replaying it would write twice. Retrying replays the whole body, side effects included, so spending the budget on a failure that repeats identically costs several times over and dead-letters anyway. See Job transactions.
  • The dead letters are not routed anywhere. Nothing alerts or archives on your behalf, and the stream keeps a week of them; wire the wrapper above if you need more.
  • A retry replays the whole body. The transaction bounds what an attempt leaves behind in the database, never an email sent or an API called before the failure — keep those behind an idempotency key.
  • Delivery is at least once, so a handler is idempotent. A second delivery comes only once a lease lapses — a worker that crashed, froze, or was cut off from Redis for longer than its lease — and runs the job again, while every write the first delivery makes after that lands nowhere. See At least once, and the fence.
  • crates/nest-rs-queue/src/consume.rs — attempt: the budget, the backoff, the stall limit, the events and the operation line.
  • crates/nest-rs-queue/src/backoff.rs — the wait before each retry, and the waits pinned for one job.
  • crates/nest-rs-redis/src/queue/scripts.rs — how an outcome settles on Redis, fenced on its lease: acknowledged, filed for later, or dead-lettered.
  • crates/nest-rs-redis/tests/e2e/queue/consumer.rs — the behaviour kit’s retry, dead letter and stall limit, a dead letter’s record, and a checkpoint across a retry and a dead replica, against live Redis.