Writing a queue driver
Plug a storage into the queue port: the contract a backend implements — declare it, file jobs, and hand them over under a lease.
A queue driver connects the queue port to a storage — a broker, a stream, a
database table. The port owns what consuming a job is: the attempt, its
retry budget and backoff, the trace, each method’s permits, the lease’s
renewal, the drain. Its QueueWorker runs all of it on every backend, so a
driver only moves records and keeps leases. At the end of this page and
Binding a queue driver, your backend runs every
#[process] method an app declares, and the port’s behaviour kit proves it
keeps the contract.
Every excerpt below is lifted from nest-rs-queue’s in-memory test backend or
from the Redis binding, so each compiles and runs in the framework’s suite.
Install
Section titled “Install”cargo add nest-rs --no-default-features --features queuecargo add nest-rs --dev --no-default-features --features queue,testingYour crate is nestrs-<storage>, and it depends on the umbrella alone, so
neither line links an HTTP stack. The naming rule
roots every name your driver owns at that crate — its target, its keys, its
<PREFIX>_<STORAGE>__* settings.
The contract
Section titled “The contract”-
Declare the backend. One
QueueBackendconstant names your storage — its OpenTelemetrymessaging.system— and each optional capability it honours, one at a time, neverCapabilities::ALL:from nest-rs-queue's test backend pub(crate) static DELAYING: QueueBackend = QueueBackend::new("memory",Capabilities::NONE.with(Capability::DelayedPush).with(Capability::Checkpoint),);The retry budget, one transaction per attempt and a method’s
concurrencyare not capabilities: the port keeps all three on every backend. Checkpoint: a method declaringthrottle(..)on this backend fails the boot, naming the capability and the backend. -
File jobs. Implement
JobProducer. The port has refused every option your backend does not declare, minted each job’s id and sealed each envelope; store the record as the bytes it gives you, keyed byenvelope.id(), and never read inside it. Skip an envelope whose job your storage still holds — waiting, held back or running — and answerOkfor it, so a call sent again after its answer was lost files nothing twice:from nest-rs-queue's test backend, abridged #[async_trait]impl JobProducer for Memory {fn backend(&self) -> &'static QueueBackend {self.inner.backend}async fn enqueue(&self,queue: &QueueName,envelopes: Vec<Envelope>,options: &PushOptions,) -> Result<(), QueueError> {// …for envelope in envelopes {if state.queue(queue.as_str()).holds_job(envelope.id()) {continue;}let job = Some(envelope.id().clone());let entry = state.entry(job, envelope.into_json().to_string().into_bytes());// …}// …}}A backend declaring
UniquePushmakes that check first: a held envelope’s own job holds its key, which never refuses it. A backend declaringCancellationimplementsremove, andremove_uniquebesideUniquePush. Checkpoint: a push answers a receipt naming the job.The skip is also what the outbox leans on: a job pushed inside a transaction reaches
enqueuefrom the port’s relay after the commit, and a relay that dies before deleting its record sends the same envelope again. Your backend owes the outbox nothing more. -
Hand jobs over under a lease. Implement
JobConsumer. The worker asks; your storage answers in the backend’s own words:Call What it owes prepareReady the storage for the methods, once, at boot, and say how long a call takes to answer at most — under BACKEND_TIMEOUT, or the boot refuses the backend, naming it.receiveAt most ask.maxdeliveries, waiting at mostask.waitfor the first — each leased in the step that hands it over, with its delivery count: 1 the first time, one more each time it is handed over again.renewExtend each lease still yours, without counting a delivery; one LeaseHoldper lease, in order.settleEnd one delivery as its Dispositionsays, in the step that checks the lease is still yours — and answerLost, writing nothing, when it is not.maintain,close,checkpointOptional: upkeep while nothing runs (a delayed record falling due), letting go at the end of the drain, and a checkpoint store for a backend declaring Checkpoint.The fence on
settleis the heart of it. A worker whose lease lapsed — frozen, or cut off from the storage — must write nothing once another delivery holds the job:from nest-rs-queue's test backend, abridged async fn settle(&self,_method: &'static ProcessMethod,lease: &MemoryLease,disposition: Disposition<'_>,) -> Result<LeaseHold, QueueError> {let mut state = self.memory.lock();if !state.queue(&lease.queue).holds(lease) {return Ok(LeaseHold::Lost);}// …match disposition {Disposition::Complete => {// …}Disposition::Retry { after, record, .. } | Disposition::Defer { after, record, .. } => {refile(&mut state, record, after);}Disposition::Requeue { record, .. } => refile(&mut state, record, Duration::ZERO),Disposition::DeadLetter { reason, record, .. } => {// …}other => panic!("a disposition the in-memory backend was not written for: {other:?}"),}// …Ok(LeaseHold::Held)}CompleteandDeadLetterend the job: let every record of it go — its unique key and its checkpoint with it — and keep a dead letter with the value-freereason.Retry,DeferandRequeuefile the record given, due onceafterpassed, and a record filed again starts its delivery count over. A backend withoutDelayedPushis never handed a wait: the port waits in process.Dispositionis non-exhaustive, and a variant added later goes only to a backend declaring the capability that names it. Checkpoint: a delivery the worker never settles comes back once its lease lapses, with its count one higher.
The binding
Section titled “The binding”The module an app imports, and what it owes in production — its budget, its failures, its TLS, its proof at boot, its health, the behaviour kit that tests it — continue on Binding a queue driver.
Going further
Section titled “Going further”- Delivery on Redis — the reference driver: one Lua script per transition, each fenced.
- Retries and failure — what the port does with each outcome.
- Extending NestRS — every contract a driver’s crate may implement.