Skip to content

Producing jobs

How a feature enqueues — the typed queue handle, the service as the choke point, and what a push answers when it fails.

Pushing a job is a one-line call — jobs.push(AudioQueue, job).await? — but where that line lives decides whether the feature stays auditable. The rule is the same one that holds Repo calls behind a service: controllers, resolvers, and gateways ask the service to enqueue. The service names the queue; the transport adapter doesn’t.

A job is any Serialize + DeserializeOwned + Clone + Send + Sync + Unpin + 'static type — the marker nest_rs::queue::Job is auto-implemented for every type that satisfies those bounds. No derive, no trait to write. The convention is to keep payloads small: a reference to the work, not the work itself.

crates/features/src/audio/command.rs (from the demo)
use nest_rs::queue::{Queue, queue};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TranscodeCommand {
pub file: Uuid,
}
#[queue(name = "audio", job = TranscodeCommand)]
pub struct AudioQueue;
pub const AUDIO_QUEUE: &str = <AudioQueue as Queue>::NAME;

Plain serde derives, not #[input] — the one place the wire-DTO shorthand is the wrong tool. #[input] carries deny_unknown_fields, which is right at an edge a client controls and wrong on a producer↔worker contract. A producer one deploy ahead that adds a field would have every job it enqueues refused by the older worker. And a decode failure aborts rather than retries, so the job dead-letters on its first attempt. Adding a field to a payload has to be survivable, and that means accepting one you do not know yet.

A queue payload is named for the boundary it crosses: an imperative “do X” handled by one processor is a Command (command.rs at the feature port); a published past-tense fact with potentially many consumers is an Event (event.rs). The HTTP body that triggers the enqueue is a separate TranscodeDto — each boundary speaks its own vocabulary.

Producer and consumer agree on both facts by importing AudioQueue: the wire name and the payload type travel together, so a renamed queue or a changed payload breaks the build on whichever side did not follow. The #[queue] marker sits at the port beside the payload, not in the consumer-side queue/ adapter — the producer is one directory up, and a marker it cannot see is a marker it cannot use.

A producer is any code that asks the service to enqueue. The HTTP controller, the GraphQL resolver, the scheduled producer — all of them delegate. They do not hold the queue’s handle and do not know the queue name.

crates/features/src/audio/service.rs (from the demo, abridged)
use std::sync::Arc;
use nest_rs::core::{error_message, injectable};
use nest_rs::queue::Jobs;
use uuid::Uuid;
use super::command::{AUDIO_QUEUE, AudioQueue, TranscodeCommand};
use super::error::AudioError;
#[injectable]
pub struct AudioService {
#[inject]
jobs: Arc<Jobs>,
}
impl AudioService {
pub async fn enqueue_transcode(&self, file: Uuid) -> Result<(), AudioError> {
self.jobs
.push(AudioQueue, TranscodeCommand { file })
.await?;
tracing::debug!(target: crate::audio::TARGET, queue = AUDIO_QUEUE, "enqueued transcode job");
Ok(())
}
}

push takes both the wire name and the accepted payload from the marker, so enqueueing onto the wrong queue, or with the wrong payload, does not compile, and answers with a PushReceipt naming the queue and the job’s id. push_with takes what a push declares besides — a delay, a unique key — and push_many an iterator of jobs for one queue, answering one receipt per job, in order (push_many_with for a delay on all of them). All are async because the backend is.

The port does the rest before the backend sees the job: it checks the options, mints the job’s id — a UUID v7, the job_id every attempt at the job reports — and seals your payload in the wire envelope with your trace context, your actor and your tenant, so the job runs under the trace that enqueued it and for the same tenant: current_tenant_id() answers in the processor what it answered where you pushed. You never see either — see Correlation and Multi-tenant SaaS.

A queue another deployment drains — its worker in another binary — takes a marker of its own on this side, naming that queue and the payload both sides agree on, #[queue(name = "billing-invoices", job = InvoiceCommand)]; the push is then as typed as any other. A payload lives at the port, so the producer links the type its consumer reads.

Arc<nest_rs::queue::Jobs> is the form to inject: the feature names no backend, so it compiles unchanged against a different driver. RedisQueueModule binds it over the producer it opens on the connection RedisModule opens for its instance (a second Valkey has one of its own).

A push is refused before anything is filed when its options cannot be honoured — QueueError::Unsupported for an option the backend does not declare, QueueError::InvalidUniqueKey for a key no backend could file, QueueError::InvalidOptions for a delay ending after 9999-12-31T23:59:59.999Z, the last instant RFC 3339 writes (Delay::LATEST_DUE) — or when a unique key is held. Past those, an error from the backend is a failure to answer: the job may be queued and its reply lost, so a caller that retries such a push can queue it twice unless it carries a unique key.

No backend holds a push past eighteen seconds. The port waits BACKEND_TIMEOUT for the backend’s answer to a push or a cancel, then gives up on the call and fails it with QueueError::Unanswered, naming the queue and the call — which, like any failure to answer, may still have filed the job. It is a net under the backend, never its budget: on Redis each command a push sends waits at most <PREFIX>_REDIS__CONNECT_TIMEOUT_SECS, ten seconds by default, so a Redis that stopped answering fails the push at that budget, with its own cause, well before the net. Eighteen seconds stays under the HTTP edge’s request timeout, so a push made inside a request fails with an error naming its queue while the request can still answer. A backend that panics answering a push or a cancel fails it the same way, as QueueError::Panicked naming the queue and the call — the panic itself is filed once at error on nest_rs::queue, never handed to your handler.

push_many hands the backend its jobs a hundred at a time (ENQUEUE_BATCH), each call under the net, so the net holds a push of any size. It is not atomic. When a call fails after the backend accepted earlier ones, the error is QueueError::PartiallyQueued { receipts, source }: receipts names every job of the accepted calls — queued, and to be left out of a retry or cancelled — and source is the failure that stopped the push. The failing call’s own jobs may be queued too, and no receipt says which, so retrying those can queue one twice. A push failing at its first call returns that failure alone.

A push written inside a transaction is written into it, and filed once it commits: The outbox.

A push can hold its job back or keep one job per key, and a waiting job can be taken back: Delays, unique keys and cancels.

The controller is a thin transport. It accepts a TranscodeDto — the REST body, a distinct type from the queue’s TranscodeCommand — calls the service, and answers 202: the job is queued, not done, and the reply echoes nothing the client sent. No queue knowledge of its own.

Declare the DTO with #[input]: a Json<T> handler argument requires schemars::JsonSchema (see extractors), and #[input] derives it along with Deserialize and Validate. A hand-derived DTO needs #[derive(schemars::JsonSchema)] and the schemars dependency to go with it.

crates/features/src/audio/dtos/transcode_dto.rs (from the demo, abridged)
use nest_rs::core::input;
use uuid::Uuid;
#[input]
pub struct TranscodeDto {
pub file: Uuid,
}

file is the key an upload minted, parsed as a UUID at the edge: a name the client chose is a 400 before the service runs.

crates/features/src/audio/http/controller.rs (from the demo, abridged)
use std::sync::Arc;
use nest_rs::authz::Manage;
use nest_rs::http::{
Body, Header, HeaderValue, HttpError, Json, Opaque, Query, Response, Result, SseEvent,
SseStream, StatusCode, controller, routes,
};
use nest_rs::throttler::{Throttle, ThrottlerGuard};
use crate::audio::{
AudioService, DirectUploadDto, PresignedUrlDto, StreamResumeDto, TranscodeDto, TranscodeState,
Transcoding,
};
use crate::authn::AuthnGuard;
use crate::authz::AuthzGuard;
#[controller(path = "/audio")]
#[use_guards(AuthnGuard, AuthzGuard, ThrottlerGuard)]
pub struct AudioController {
#[inject]
svc: Arc<AudioService>,
}
#[routes]
impl AudioController {
#[post("/transcode")]
#[authorize(Manage, Transcoding)]
#[meta(Throttle::per_minute(20))]
// … #[api(…)]
#[http_code(202)]
async fn transcode(&self, Json(body): Json<TranscodeDto>) -> Result<()> {
self.svc.enqueue_transcode(body.file).await.opaque()?;
Ok(())
}
}

Swap this for a #[resolver] mutation, a #[gateway] message, or a #[scheduled] method on a timer — the controller is the only file that changes. The service stays the single audited entry point, every transport hits it through enqueue_transcode.

Three reasons the push call hides behind a service:

  1. One audit point per queue. AudioQueue is named in exactly one place per side. A grep for enqueue_transcode shows every producer.
  2. Pre-flight logic survives a swap of transport. If enqueueing needs an ability check, an idempotency key, a tracing field — the service holds it. Adding a second producer doesn’t duplicate that.
  3. The consumer can change without touching producers. Renaming the queue, splitting audio into audio.fast and audio.slow, moving from Redis to another backend — one file changes, every adapter follows. And because the name lives on the #[queue] type, a rename that misses a call site is a compile error rather than a queue nobody drains.

The same shape the rest of the framework uses for Repo, just on the push side. The service is the contract; transports adapt.

  • crates/features/src/audio/command.rs — TranscodeCommand, AudioQueue.
  • crates/features/src/audio/service.rs — enqueue_transcode.
  • crates/features/src/audio/http/controller.rs — the HTTP producer.
  • crates/features/src/audio/schedule/tasks.rs — the scheduled producer (a timer that calls the same service).
  • crates/nest-rs-queue/src/jobs.rs — Jobs: push, push_with, push_many, push_many_with, cancel and cancel_unique.
  • crates/nest-rs-queue/src/producer.rs — JobProducer, what a backend implements, and the path every push takes to it.
  • Wiring — producer-only vs worker apps sharing a features crate.
  • Retries and failure — what happens once a worker drains the job.
  • Queue — the section landing and worker overview.