Skip to content

Writing a queue driver

Plug a storage into the queue port: the contract a backend implements — declare it, file jobs, and hand them over under a lease.

A queue driver connects the queue port to a storage — a broker, a stream, a database table. The port owns what consuming a job is: the attempt, its retry budget and backoff, the trace, each method’s permits, the lease’s renewal, the drain. Its QueueWorker runs all of it on every backend, so a driver only moves records and keeps leases. At the end of this page and Binding a queue driver, your backend runs every #[process] method an app declares, and the port’s behaviour kit proves it keeps the contract.

Every excerpt below is lifted from nest-rs-queue’s in-memory test backend or from the Redis binding, so each compiles and runs in the framework’s suite.

Terminal window
cargo add nest-rs --no-default-features --features queue
cargo add nest-rs --dev --no-default-features --features queue,testing

Your crate is nestrs-<storage>, and it depends on the umbrella alone, so neither line links an HTTP stack. The naming rule roots every name your driver owns at that crate — its target, its keys, its <PREFIX>_<STORAGE>__* settings.

  1. Declare the backend. One QueueBackend constant names your storage — its OpenTelemetry messaging.system — and each optional capability it honours, one at a time, never Capabilities::ALL:

    from nest-rs-queue's test backend
    pub(crate) static DELAYING: QueueBackend = QueueBackend::new(
    "memory",
    Capabilities::NONE
    .with(Capability::DelayedPush)
    .with(Capability::Checkpoint),
    );

    The retry budget, one transaction per attempt and a method’s concurrency are not capabilities: the port keeps all three on every backend. Checkpoint: a method declaring throttle(..) on this backend fails the boot, naming the capability and the backend.

  2. File jobs. Implement JobProducer. The port has refused every option your backend does not declare, minted each job’s id and sealed each envelope; store the record as the bytes it gives you, keyed by envelope.id(), and never read inside it. Skip an envelope whose job your storage still holds — waiting, held back or running — and answer Ok for it, so a call sent again after its answer was lost files nothing twice:

    from nest-rs-queue's test backend, abridged
    #[async_trait]
    impl JobProducer for Memory {
    fn backend(&self) -> &'static QueueBackend {
    self.inner.backend
    }
    async fn enqueue(
    &self,
    queue: &QueueName,
    envelopes: Vec<Envelope>,
    options: &PushOptions,
    ) -> Result<(), QueueError> {
    // …
    for envelope in envelopes {
    if state.queue(queue.as_str()).holds_job(envelope.id()) {
    continue;
    }
    let job = Some(envelope.id().clone());
    let entry = state.entry(job, envelope.into_json().to_string().into_bytes());
    // …
    }
    // …
    }
    }

    A backend declaring UniquePush makes that check first: a held envelope’s own job holds its key, which never refuses it. A backend declaring Cancellation implements remove, and remove_unique beside UniquePush. Checkpoint: a push answers a receipt naming the job.

    The skip is also what the outbox leans on: a job pushed inside a transaction reaches enqueue from the port’s relay after the commit, and a relay that dies before deleting its record sends the same envelope again. Your backend owes the outbox nothing more.

  3. Hand jobs over under a lease. Implement JobConsumer. The worker asks; your storage answers in the backend’s own words:

    CallWhat it owes
    prepareReady the storage for the methods, once, at boot, and say how long a call takes to answer at most — under BACKEND_TIMEOUT, or the boot refuses the backend, naming it.
    receiveAt most ask.max deliveries, waiting at most ask.wait for the first — each leased in the step that hands it over, with its delivery count: 1 the first time, one more each time it is handed over again.
    renewExtend each lease still yours, without counting a delivery; one LeaseHold per lease, in order.
    settleEnd one delivery as its Disposition says, in the step that checks the lease is still yours — and answer Lost, writing nothing, when it is not.
    maintain, close, checkpointOptional: upkeep while nothing runs (a delayed record falling due), letting go at the end of the drain, and a checkpoint store for a backend declaring Checkpoint.

    The fence on settle is the heart of it. A worker whose lease lapsed — frozen, or cut off from the storage — must write nothing once another delivery holds the job:

    from nest-rs-queue's test backend, abridged
    async fn settle(
    &self,
    _method: &'static ProcessMethod,
    lease: &MemoryLease,
    disposition: Disposition<'_>,
    ) -> Result<LeaseHold, QueueError> {
    let mut state = self.memory.lock();
    if !state.queue(&lease.queue).holds(lease) {
    return Ok(LeaseHold::Lost);
    }
    // …
    match disposition {
    Disposition::Complete => {
    // …
    }
    Disposition::Retry { after, record, .. } | Disposition::Defer { after, record, .. } => {
    refile(&mut state, record, after);
    }
    Disposition::Requeue { record, .. } => refile(&mut state, record, Duration::ZERO),
    Disposition::DeadLetter { reason, record, .. } => {
    // …
    }
    other => panic!("a disposition the in-memory backend was not written for: {other:?}"),
    }
    // …
    Ok(LeaseHold::Held)
    }

    Complete and DeadLetter end the job: let every record of it go — its unique key and its checkpoint with it — and keep a dead letter with the value-free reason. Retry, Defer and Requeue file the record given, due once after passed, and a record filed again starts its delivery count over. A backend without DelayedPush is never handed a wait: the port waits in process. Disposition is non-exhaustive, and a variant added later goes only to a backend declaring the capability that names it. Checkpoint: a delivery the worker never settles comes back once its lease lapses, with its count one higher.

The module an app imports, and what it owes in production — its budget, its failures, its TLS, its proof at boot, its health, the behaviour kit that tests it — continue on Binding a queue driver.