Skip to content

Long-running jobs

How long an attempt runs before its timeout cuts it, and a Checkpoint that lets the next attempt resume where the last one stopped.

A job that runs long meets two questions: how long an attempt may run before it is cut, and where the next attempt starts.

An attempt runs for at most ten minutes unless its decorator says otherwise. timeout takes a whole number of ms, s, m or h, up to "24h":

#[process(queue = AudioQueue, retries = 3, timeout = "30m")]
async fn transcode(&self, job: TranscodeCommand) -> Result<()> {
self.svc.transcode(job.file).await?;
Ok(())
}

No edge deadline sits above a job, so the timeout is what ends an attempt waiting on a call that never answers — an outbound request with no timeout of its own. It bounds the method’s body: past it, the attempt is cut where it waits and fails retryably, since the next attempt may meet the answer the first waited on in vain, and what it wrote in its transaction is rolled back. The failure names the deadline, on the retry warn, the dead-letter line and its record alike:

Terminal window
WARN nest_rs::queue: job failed; will retry within the budget error=the attempt did not end within its 1800s timeout, and was cut — the `timeout` key on its job decorator sets it retry_after_ms=1143 trace_id=… span_id=…

A body that returned within its timeout is never retried for what comes after it: its COMMIT runs to its answer, and a listener on what it committed runs on past the timeout rather than failing the attempt. A commit that does not answer within the driver’s bound may have landed, so that attempt is dead-lettered, never replayed — running it again could write twice:

Terminal window
ERROR nest_rs::queue: job dead-lettered: its commit did not answer in time, and what it wrote may have landed outcome="in_doubt" error=the unit's commit did not answer within its bound; its writes may have landed trace_id=… span_id=…

The default is the dispatch deadline of Google Cloud Tasks, and the same key, default and ceiling hold on #[every], #[cron] and #[after] — see How long a run lasts.

A retry replays the whole body. A job that works through a long file can keep its place instead, with a Checkpoint<S> parameter beside the job:

crates/features/src/audio/queue/processor.rs
#[process(queue = AudioQueue, retries = 3, transactional = false)]
async fn transcode(&self, job: TranscodeCommand, mut progress: Checkpoint<TranscodeProgress>) -> Result<()> {
let first = progress.get().map_or(0, |saved| saved.next_chunk);
for chunk in first..self.svc.chunks(&job.file).await? {
self.svc.transcode_chunk(&job.file, chunk).await?;
progress.save(TranscodeProgress { next_chunk: chunk + 1 }).await?;
}
Ok(())
}

get answers the last state saved for this job — by an earlier attempt, or by a replica that died holding it — and save stores a new one at once. The state is any Serialize + DeserializeOwned value, lasts through the job’s retries and redeliveries, and is cleared when the job completes, dead-letters or is cancelled.

The method must declare transactional = false, and the decorator refuses a Checkpoint without it: a save reaches the backend at once, while a transactional attempt rolls its database work back when it fails, so the retry would resume past rows that were undone. A checkpoint needs a backend declaring Capability::Checkpoint; on Redis it is one field beside the job, written only by the delivery holding it and removed when the job ends — see Delivery on Redis.

A store that does not answer within the port’s net — BACKEND_TIMEOUT, eighteen seconds — or panics answering fails the attempt, retryably, the panic said once at error: the read before your method runs does so on its own, and a save does when you return its error with ?. The clear at the job’s end is no call of its own: the backend lets the checkpoint go in the step that settles the job, so it lands or fails with the outcome.

  • crates/nest-rs-queue/src/checkpoint.rs — Checkpoint<S>, and the CheckpointStore a backend keeps it in.