Skip to content

Leases and redelivery

How a Redis worker holds a job — a lease, and a fence on every write — and what a crash, a slow Redis and a shutdown do to a job in flight.

Every job is delivered at least once. On Redis that rests on a lease each delivery holds and a fence every write it makes passes; this page is that mechanism, and what a crash, a slow Redis and a shutdown do to a job in flight.

A worker reads its queue with XREADGROUP, which hands an entry over, records the worker as its owner and starts its idle clock in one command — so a job is never handed over without a lease. While the attempt runs, the port renews the lease every third of its length, by claiming the entry again without counting a delivery. A lease no renewal reached for a whole lease has lapsed: the next worker that asks, with a permit free, takes the entry over, which counts a delivery. A worker looks for lapsed leases once a second: Redis filters a pending list of up to ten thousand entries whole, and a longer one is read a page of a thousand at a time from where the last look stopped, so no look costs Redis more however many deliveries run elsewhere; behind such a list, a lapsed entry waits up to a look per page.

Every write a delivery makes — its outcome, a renewal, a checkpoint — is fenced: the script first reads the entry’s owner and delivery count, and writes nothing unless both are still what the worker received. So once a lease has lapsed and another worker holds the job, the first one’s outcome, renewal and checkpoint land nowhere; its attempt is cut, filed cancelled, and the delivery holding the job decides.

A job’s outcome — completed, filed for its retry, dead-lettered — acknowledges and deletes its entry in the script that writes it, so a job that ended is never delivered again. A second delivery happens only when a lease lapses under a worker: the worker crashed, froze past its lease, or was cut off from Redis for longer than it — and then the first delivery’s writes land nowhere.

A push is filed once per job, too. The script that files it skips a job its queue still holds — on the stream, running, or held back in …:delayed — so a push the client sends again after its answer was lost files nothing twice. Once the job ended, the same job is filed as a new one.

A replica that crashes mid-attempt leaves its job leased to it. Once the lease lapses — <PREFIX>_REDIS__QUEUE__LEASE_SECS, thirty seconds by default — the next replica asking for jobs takes it over, and the job runs again.

A delivery that ends without an answer does not spend the retry budget. retries counts the attempts that answered; a delivery that ended without one — its worker killed, frozen, or cut off past its lease — is counted by the entry’s own delivery count instead, and every filing of the job’s record starts it over. A job whose deliveries end that way three times in a row (STALL_LIMIT) takes its workers down every time — an abort, an OOM kill, a hang until the pod is killed — and is dead-lettered without running, as job dead-lettered: its deliveries ended without an answer past the stall limit.

Every command a worker or a producer sends answers or fails within <PREFIX>_REDIS__CONNECT_TIMEOUT_SECS, ten seconds by default, so an outage fails calls rather than holding them. The boot refuses a budget of eighteen seconds or more, the queue port’s own net on every call: the port would stop waiting first, and a failure would reach its caller as a bare timeout instead of Redis’s own cause. A worker’s read blocks on a connection of its own — on the shared one it would stall every other caller — for at most a second, so its bound is that second plus the budget. A timeout says the answer did not arrive in time, never that the command did not run: a settle whose answer is lost may have ended the job all the same, which is why the line says not confirmed.

The call that failedWhat happens to the jobThe line, on nest_rs::queue
a readnothing was handed over, or what was waits leased until its lease lapses; the read is tried again after a backoff doubling from 250 ms to 5 s, each wait within a fifth either way so replicas come back apartqueue receive failed; retrying, warn
renewing leasesthe attempts run on, and the renewal is tried again; one no renewal reached for a whole lease is cut and its job handed backjob leases not renewed; retrying, warn
an outcometried again while the lease holds; past it, said, and left to its lease — the next delivery runs the job again unless the outcome landedjob outcome not confirmed; unless it was written, the job runs again once its lease lapses, error
filing due jobsthey wait, and the next upkeep files themqueue upkeep failed; retrying, warn

A backend call that panics — a driver’s defect, not an outage — is treated as one that did not answer: the worker files the panic once, queue backend panicked answering the worker; the call counts as unanswered, at error with the queue and the call, then goes on as the table says.

A failover is followed. A failover demotes the primary under its open connections; the first that answers READONLY is opened again, once, through the URL’s name — which now reaches the new primary — and says so at warn, redis connection opened again: the server it reached answered as a read-only replica. The writes refused meanwhile fail as any refused call does.

A push, a cancel and a checkpoint’s read or save fail within one budget too, as the connection’s own failure, before the queue port’s net — eighteen seconds — gives up on the call; see When a push fails.

An attempt ends within its timeout. A handler still running at its method’s timeout — ten minutes unless #[process] says otherwise — is cut where it waits and fails retryably, so a call that never answers holds its permit and its renewed lease that long and no longer; see How long an attempt runs.

A shutdown stops asking for jobs at once, and hands back what a read brought after it. Running attempts get the shutdown window, less a reserve, to finish where they run, their leases renewed until each ends. What still runs when the window closes is interrupted — its queue.job line filed with outcome="cancelled" — and its job handed back to the stream inside the reserve, due at once for another replica, so the orchestrator’s SIGKILL never lands on a job the worker still holds. The window is <PREFIX>_QUEUE__SHUTDOWN_TIMEOUT_SECS, 20 seconds by default, which leaves the shutdown hooks’ five, the telemetry flush’s three and the console’s one inside Kubernetes’ default 30-second grace period; a window you raise needs the pod’s grace period at least 9.5 seconds above it, plus any preStop delay (the way down).

The reserve is one Redis call long: the connection’s budget, or five seconds when that is shorter, and never more than half the window. A delivery still waiting on Redis once the whole window has passed is said at error and left to its lease: the next delivery runs its job.

  • crates/nest-rs-redis/src/queue/consumer.rs — the read, the reclaim, the renewal, the outcome and the upkeep.
  • crates/nest-rs-redis/src/queue/scripts.rs — every transition, fenced.