Long-running jobs
How long an attempt runs before its timeout cuts it, and a Checkpoint that lets the next attempt resume where the last one stopped.
A job that runs long meets two questions: how long an attempt may run before it is cut, and where the next attempt starts.
How long an attempt runs
Section titled “How long an attempt runs”An attempt runs for at most ten minutes unless its decorator says otherwise.
timeout takes a whole number of ms, s, m or h, up to "24h":
#[process(queue = AudioQueue, retries = 3, timeout = "30m")]async fn transcode(&self, job: TranscodeCommand) -> Result<()> { self.svc.transcode(job.file).await?; Ok(())}No edge deadline sits above a job, so the timeout is what ends an attempt
waiting on a call that never answers — an outbound request with no timeout of
its own. It bounds the method’s body: past it, the attempt is cut where it
waits and fails retryably, since the next attempt may meet the answer the
first waited on in vain, and what it wrote in its transaction is rolled back.
The failure names the deadline, on the retry warn, the dead-letter line and
its record alike:
WARN nest_rs::queue: job failed; will retry within the budget error=the attempt did not end within its 1800s timeout, and was cut — the `timeout` key on its job decorator sets it retry_after_ms=1143 trace_id=… span_id=…A body that returned within its timeout is never retried for what comes after
it: its COMMIT runs to its answer, and a listener on what it committed runs
on past the timeout rather than failing the attempt. A commit that does not
answer within the driver’s bound may have landed, so that attempt is
dead-lettered, never replayed — running it again could write twice:
ERROR nest_rs::queue: job dead-lettered: its commit did not answer in time, and what it wrote may have landed outcome="in_doubt" error=the unit's commit did not answer within its bound; its writes may have landed trace_id=… span_id=…The default is the dispatch deadline of Google Cloud Tasks, and the same key,
default and ceiling hold on #[every], #[cron] and #[after] — see
How long a run lasts.
Resuming where an attempt stopped
Section titled “Resuming where an attempt stopped”A retry replays the whole body. A job that works through a long file can keep
its place instead, with a Checkpoint<S> parameter beside the job:
#[process(queue = AudioQueue, retries = 3, transactional = false)]async fn transcode(&self, job: TranscodeCommand, mut progress: Checkpoint<TranscodeProgress>) -> Result<()> { let first = progress.get().map_or(0, |saved| saved.next_chunk); for chunk in first..self.svc.chunks(&job.file).await? { self.svc.transcode_chunk(&job.file, chunk).await?; progress.save(TranscodeProgress { next_chunk: chunk + 1 }).await?; } Ok(())}get answers the last state saved for this job — by an earlier attempt, or by
a replica that died holding it — and save stores a new one at once. The
state is any Serialize + DeserializeOwned value, lasts through the job’s
retries and redeliveries, and is cleared when the job completes, dead-letters
or is cancelled.
The method must declare transactional = false, and the decorator refuses a
Checkpoint without it: a save reaches the backend at once, while a
transactional attempt rolls its database work back when it fails, so the retry
would resume past rows that were undone. A checkpoint needs a backend declaring
Capability::Checkpoint; on Redis it is one field beside the job, written
only by the delivery holding it and removed when the job ends — see
Delivery on Redis.
A store that does not answer within the port’s net — BACKEND_TIMEOUT, eighteen
seconds — or panics answering fails the attempt, retryably, the panic said once
at error: the read before your method runs does so on its own, and a save does
when you return its error with ?. The clear at the job’s end is no call of its
own: the backend lets the checkpoint go in the step that settles the job, so it lands
or fails with the outcome.
Reference
Section titled “Reference”crates/nest-rs-queue/src/checkpoint.rs—Checkpoint<S>, and theCheckpointStorea backend keeps it in.
Going further
Section titled “Going further”- Retries and failure — the budget a cut attempt spends.
- Leases and redelivery — a delivery whose worker died mid-attempt.
- Running a tick — the same timeout on a scheduled job.