Retries and failure
What an Err from a #[process] method means — the retry budget and its backoff, panic handling, and where a dead-lettered job goes.
The contract is small: a #[process] method returns anyhow::Result<()>,
Err(_) marks the attempt failed, and Ok(()) completes the job — once the
attempt’s transaction has settled, which is the one way a body that returned
Ok can still fail (see
below). The budget is the
queue port’s, the same on every backend: a retryable failure runs again after
a backoff, as the same job, until the budget is spent.
The basic shape
Section titled “The basic shape”#[process(queue = AudioQueue, retries = 3)]async fn transcode(&self, job: TranscodeCommand) -> Result<()> { self.svc.transcode(job.file).await?; Ok(())}Ok(())⇒ the attempt’s transaction commits and the job is done. If the commit cannot be honoured, nothing it wrote landed and the attempt fails instead — the one case whereOkis not the end.Err(e)⇒ the attempt failed. While the budget holds another attempt, the job runs again after the backoff below, under the samejob_idwithattemptone higher. Afterretriesfailed retries it is dead-lettered (see Where failed jobs go).- A panic inside the method is caught, logged as
job dead-lettered: handler panickedaterrorwith the panic’s message, and the job dead-letters at once. The attempt’squeue.jobline reportsoutcome="panic"beside it, under the samespan_id. The worker keeps running — one bad job doesn’t stop the queue’s consumer.
The panic is caught by the queue port itself, around the handler, which is what keeps the event inside the attempt’s trace. Each delivery also runs on a task of its own, so the worker goes on to its next job.
That event is the panic’s one record: under #[nest_rs::main] the process
hook stays quiet for a panic the attempt contains, so no raw
thread … panicked at … line reaches stderr, and the event names where the
handler panicked under panic_location
(a panic is said once).
The retry shape
Section titled “The retry shape”retries = N is N + 1 attempts in all: the first try and N retries. The
default is 0 — the job is tried once. The budget counts attempts that
answered. A delivery that ends without an answer — its process killed, frozen
or cut off past its lease — spends none: the job runs again at the attempt it
was at. That has its own bound, the stall limit: a job whose deliveries end
that way three times in a row (STALL_LIMIT) is taking its workers down — an
abort, an OOM kill, a hang until the pod is killed — and is dead-lettered
without running, as
job dead-lettered: its deliveries ended without an answer past the stall limit.
See Crashes, and the stall limit.
After attempt n fails, the job waits min(5 min, 1 s · 2^(n − 1)) before
attempt n + 1, scaled by a jitter between 0.8 and 1.2. The jitter is derived
from the job’s id and n rather than drawn, so jobs a shared outage failed
together come back apart, and one job waits the same on every replay. It is the
one retry policy the framework runs — nest_rs::core::Backoff, the shape of
gRPC’s connection backoff — and the worker’s own calls to its backend, Redis’s
reconnections and retry_on_conflict wait on it too, each from a seed of its
own:
| After attempt | Wait before jitter | Wait |
|---|---|---|
| 1 | 1 s | 0.8 – 1.2 s |
| 2 | 2 s | 1.6 – 2.4 s |
| 3 | 4 s | 3.2 – 4.8 s |
| 9 | 256 s | 204.8 – 307.2 s |
| 10 and after | 5 min | 4 – 6 min |
The worker says how long it chose, on the line that reports the failure:
WARN nest_rs::queue: job failed; will retry within the budget error=transcode failed retry_after_ms=1143 trace_id=01a0deaacf9670f8a924efd95a2ec64b span_id=a83a4d58c89c3d5dA failure can ask for longer. A Problem carrying with_retry_after(30) — the
upstream’s 429 or 503 passed on — holds the next attempt back at least that
long, so the job does not spend its budget while the upstream is still refusing.
The wait is the longer of the two, what it asks counts up to an hour and is
stretched by up to a fifth — 30 to 36 seconds here, drawn from the job’s id
like the jitter — so the jobs one 429 refused together do not all come back at
the instant it named, and none comes back before it. The line adds
retry_after_asked_ms=30000 beside the wait it chose.
On Redis the next attempt waits off the stream, due on Redis’s clock, until a running worker files it back — see Delayed jobs — and the worker that ran the failed attempt moves on: no permit is held while a job waits, and a shutdown during the wait loses nothing. The port counts the budget once; the backend keeps no retry count of its own.
A failure a retry cannot fix is non-retryable and dead-letters on the
attempt it happens: a payload that does not deserialize, a pipe rejection, a
missing provider, a client error (below).
Every other Err your method returns is retryable — the budget is how many
times a transient fault gets another chance.
A client error dead-letters at once
Section titled “A client error dead-letters at once”A job that fails for a reason no retry clears says so the way an HTTP handler
does, with a Problem. A 4xx is the client’s error (RFC 9110 §15.5) and
dead-letters the job on the attempt it happens; 408 and 429 invite a later
try and retry, as does a 5xx:
#[process(queue = AudioQueue, retries = 3)]async fn transcode(&self, job: TranscodeCommand) -> Result<()> { if job.file.get_version_num() != 7 { return Err(Problem::new(422, code::INVALID_ARGUMENT).into()); } self.svc.transcode(job.file).await?; Ok(())}What decides, in order:
- The problem an exception filter answered, when one catches the failure:
a filter mapping a domain error to
Problem::new(422, …)dead-letters at attempt one, one answering503retries. - A
JobErrorthe failure carries keeps its disposition —JobError::abort(…)returned from the method dead-letters,retryretries. - The method’s error, read by its type: a method returning a
ToProblemerror —Result<(), ServiceError>— by the problem it says (ServiceError::Conflictis a409, so it dead-letters), and any other — ananyhow::Errorabove — by aProblemin its chain. - Anything else retries within the budget.
A method returning Result<(), ServiceError> dead-letters a NotFound, a
404, too. A job filed at once — pushed on the pool, whatever transaction you opened
yourself beside it — can run before that write commits and find nothing: push from a
listener on the event the commit announces, or return the error as JobError::retry.
One pushed inside the unit’s own transaction waits in the outbox for its commit
(A push inside a transaction).
An exception filter may also answer a failed attempt with
job.acknowledge() through QueueView: the job settles as done, what it
wrote rolls back, and the failure is said at warn as
job failed; a layer acknowledged it, so it settles done and its writes rolled back.
How long an attempt may run, and how a long one resumes where it stopped, are on Long-running jobs; what a worker does with a job another release sealed is on Wire format.
Where failed jobs go
Section titled “Where failed jobs go”A dead-lettered job leaves one entry on the queue’s
nestrs:queue:{<queue>}:dead stream: its job id, its record as stored, and
the reason — the sentence the job dead-lettered line carries: the failure
and every cause beneath it, a .context(…) included, with any decode failure
said without its value and a database error by its SQLSTATE and the names the
server reported (database error 22P02), never by the server’s message, which
quotes the input it refused:
$ valkey-cli XRANGE 'nestrs:queue:{audio}:dead' - +1) 1) "1790442140102-0" 2) 1) "job" 2) "01a0deaa-cf96-70f8-a924-efd8fbbe3d38" 3) "record" 4) "{\"v\":1,…}" 5) "reason" 6) "transcode failed"The job is the id the push returned; the job’s lines name the stream entry
each attempt ran from as backend_id. The stream keeps a week of dead letters
and about the last 10,000. The framework does not ship a dead-letter
routing layer: the dead letters are the storage’s, and inspecting them is an
operational task — filing one back, once its cause is fixed, is the one command
under Dead letters.
If you need richer dead-lettering — alerting, replay, archival — catch the
Err in the method and republish the payload to a dedicated queue before
returning the error. A #[processor] on that queue then runs your
dead-letter logic, and the pattern is the same on any backend:
#[process(queue = AudioQueue, retries = 3)]async fn transcode(&self, job: TranscodeCommand) -> Result<()> { match self.svc.transcode(job.file).await { Ok(_) => Ok(()), Err(e) => { self.svc.report_dead(&job, &e).await?; Err(e.into()) } }}Panics, deserialization errors, container misses
Section titled “Panics, deserialization errors, container misses”Four failure modes the framework converts into the same Err path,
so you don’t have to think about them separately:
- Panics in the method body — caught, with a
job dead-lettered: handler panickedevent carrying the panic message. The job dead-letters immediately (regardless of remaining retries), the worker stays up. - Deserialization errors — the macro-emitted handler decodes the job from the
payload in place. A payload that doesn’t match the method’s
job type returns
Errbefore your method even runs. Retrying won’t help: the payload shape is wrong, so it is classified non-retryable and dead-letters on the first attempt, loggingjob dead-lettered: non-retryable failureaterror. Theretriesbudget is not consumed. The reason names where the payload failed and what kind of value it found —invalid type: a string, expected u64— and never the value: serde’s own sentence quotes it, and the reason lands in the log line and the dead-letter record, where a payload’s data does not belong. A decode your method does itself —serde_json::from_str(&reply)?in ananyhow::Result— is said the same way when it fails, on the retrywarn, the dead-letter line and the record alike. - Container resolution failures —
Container::getof the#[processor]host finds nothing when the host is not registered under its own type. The handler aborts naming it, and the job dead-letters on the first attempt: a configuration mistake rather than a transient, so investigate it. - A transaction that could not be settled — the body returned
Ok, and the attempt’s transaction could not be committed, so nothing it wrote landed. The attempt fails rather than reporting writes that were never made, and the database decides whether it is retried — with the answer depending on where it failed. A statement inside the attempt failed on a serialization conflict, a deadlock, a pool that timed out, or a connection the server closed: retryable, because nothing is durable beforeCOMMITand the framework therefore knows the attempt wrote nothing. A constraint checked atCOMMIT, or a commit whose outcome is unknown because the connection dropped mid-flight, is not — it dead-letters at once, because that one may have landed and replaying it would write twice. Retrying replays the whole body, side effects included, so spending the budget on a failure that repeats identically costs several times over and dead-letters anyway. See Job transactions.
Limits
Section titled “Limits”- The dead letters are not routed anywhere. Nothing alerts or archives on your behalf, and the stream keeps a week of them; wire the wrapper above if you need more.
- A retry replays the whole body. The transaction bounds what an attempt leaves behind in the database, never an email sent or an API called before the failure — keep those behind an idempotency key.
- Delivery is at least once, so a handler is idempotent. A second delivery comes only once a lease lapses — a worker that crashed, froze, or was cut off from Redis for longer than its lease — and runs the job again, while every write the first delivery makes after that lands nowhere. See At least once, and the fence.
Reference
Section titled “Reference”crates/nest-rs-queue/src/consume.rs—attempt: the budget, the backoff, the stall limit, the events and the operation line.crates/nest-rs-queue/src/backoff.rs— the wait before each retry, and the waits pinned for one job.crates/nest-rs-redis/src/queue/scripts.rs— how an outcome settles on Redis, fenced on its lease: acknowledged, filed for later, or dead-lettered.crates/nest-rs-redis/tests/e2e/queue/consumer.rs— the behaviour kit’s retry, dead letter and stall limit, a dead letter’s record, and a checkpoint across a retry and a dead replica, against live Redis.
Going further
Section titled “Going further”- Leases and redelivery — leases, redeliveries and shutdown.
- Observability — what shows up in spans and logs when a job retries or fails.
- OpenTelemetry — wiring the queue spans into a trace backend.