Wire format
A worker reads only the envelope versions it knows: a newer release's job waits unread for one that does, no attempt spent, and is dead-lettered after a day.
A processor reads only payloads it understands. The wire envelope carries a
version, and the port checks it before the handler runs.
A future release that bumps WIRE_FORMAT_VERSION to 2 makes a v: 1 worker
leave v: 2 jobs unread rather than misinterpret them: the job goes back to the
queue exactly as it was stored, due again in a minute, with no attempt spent,
and the worker files one line per delivery at warn, in the trace the newer
producer sealed:
WARN nest_rs::queue: job sealed by a newer release handed back unread queue=audio job_id=01a0… version=2 supported=1 retry_after_ms=60000 waited_ms=0 patience_ms=86400000 hint="a newer release sealed this job and only its consumers can run it: finish rolling them forward, or, rolling back, keep one running until the queue holds none of its jobs — a job still unread a day after it was first handed back is dead-lettered, naming both versions, and kept with the dead letters for a consumer of that release" trace_id=4bf9… span_id=…So during a rolling deploy jobs from the new producers wait, and the new
workers run them as they come up — under the id their push returned, which the
line names. A rollout that stopped does not hand its jobs back forever.
waited_ms counts from the job’s first hand-back — never from its push, so a
job pushed with a delay is not charged for the delay — and a job still unread a
day later (NEWER_RELEASE_PATIENCE) is dead-lettered, once, at
error:
ERROR nest_rs::queue: job dead-lettered: a newer release sealed it, and none of its consumers ran it in time version=2 supported=1 waited_ms=86400412 patience_ms=86400000 hint="…"Rolling the producers back stops new such jobs and recovers none already
sealed: those need a consumer of the release that sealed them. Keep one
running until the queue holds none of them — a job it runs is cleared of its
wait — or let them reach the dead letters, where each keeps its record a week,
naming both versions, for a consumer of that release. An older version — a v: 1 job
met by a v: 2 worker — is dead-lettered naming the direction of the gap,
since a newer consumer reading an older shape is a decision the release that
bumps the version makes, not a default.
Envelopes from before versions
Section titled “Envelopes from before versions”An envelope with no id or attempt — what 6.x sealed — still runs, as
attempt 1 under an id minted for its delivery. Unversioned legacy payloads
(a bare job JSON, no v key) are decoded directly with a tracing::warn, so
jobs left from a release before envelopes existed don’t get stuck. New
producers always seal.
Going further
Section titled “Going further”- Retries and failure — the dead letters a job past its patience lands in.
- Upgrading queues from 6.x — draining 6.x’s jobs before 7.0 runs.
- Observability — the line each hand-back files.