Producing jobs
How a feature enqueues — the typed queue handle, the service as the choke point, and what a push answers when it fails.
Pushing a job is a one-line call — jobs.push(AudioQueue, job).await? —
but where that line lives decides whether the feature stays auditable.
The rule is the same one that holds Repo calls behind a service:
controllers, resolvers, and gateways ask the service to enqueue. The
service names the queue; the transport adapter doesn’t.
The payload
Section titled “The payload”A job is any Serialize + DeserializeOwned + Clone + Send + Sync + Unpin + 'static
type — the marker nest_rs::queue::Job is auto-implemented for every
type that satisfies those bounds. No derive, no trait to write. The
convention is to keep payloads small: a reference to the work, not the
work itself.
use nest_rs::queue::{Queue, queue};use serde::{Deserialize, Serialize};use uuid::Uuid;
#[derive(Debug, Clone, Serialize, Deserialize)]pub struct TranscodeCommand { pub file: Uuid,}
#[queue(name = "audio", job = TranscodeCommand)]pub struct AudioQueue;
pub const AUDIO_QUEUE: &str = <AudioQueue as Queue>::NAME;Plain serde derives, not #[input] — the one place the wire-DTO
shorthand is the wrong tool. #[input] carries deny_unknown_fields, which is right at an edge a client
controls and wrong on a producer↔worker contract. A producer one deploy
ahead that adds a field would have every job it enqueues refused by the
older worker. And a decode failure aborts rather than retries, so the job
dead-letters on its first attempt.
Adding a field to a payload has to be survivable, and that means accepting one
you do not know yet.
A queue payload is named for the boundary it crosses: an imperative
“do X” handled by one processor is a Command (command.rs at the
feature port); a published past-tense fact with potentially many
consumers is an Event (event.rs). The HTTP body that triggers
the enqueue is a separate TranscodeDto — each boundary speaks its
own vocabulary.
Producer and consumer agree on both facts by importing AudioQueue:
the wire name and the payload type travel together, so a renamed queue or
a changed payload breaks the build on whichever side did not follow. The
#[queue] marker sits at the port beside the payload, not in the
consumer-side queue/ adapter — the producer is one directory up, and a
marker it cannot see is a marker it cannot use.
The service is the choke point
Section titled “The service is the choke point”A producer is any code that asks the service to enqueue. The HTTP controller, the GraphQL resolver, the scheduled producer — all of them delegate. They do not hold the queue’s handle and do not know the queue name.
use std::sync::Arc;
use nest_rs::core::{error_message, injectable};use nest_rs::queue::Jobs;use uuid::Uuid;
use super::command::{AUDIO_QUEUE, AudioQueue, TranscodeCommand};use super::error::AudioError;
#[injectable]pub struct AudioService { #[inject] jobs: Arc<Jobs>,}
impl AudioService { pub async fn enqueue_transcode(&self, file: Uuid) -> Result<(), AudioError> { self.jobs .push(AudioQueue, TranscodeCommand { file }) .await?; tracing::debug!(target: crate::audio::TARGET, queue = AUDIO_QUEUE, "enqueued transcode job"); Ok(()) }}push takes both the wire name and the accepted payload from the marker, so
enqueueing onto the wrong queue, or with the wrong payload, does not compile,
and answers with a PushReceipt naming the queue and the job’s id. push_with
takes what a push declares besides — a delay, a unique key — and push_many
an iterator of jobs for one queue, answering one receipt per job, in order
(push_many_with for a delay on all of them). All are async because the
backend is.
The port does the rest before the backend sees the job: it checks the options,
mints the job’s id — a UUID v7, the job_id every attempt at the job reports —
and seals your payload in the wire envelope with your trace context, your actor
and your tenant, so the job runs under the trace that enqueued it and for the
same tenant: current_tenant_id() answers in the processor what it answered
where you pushed. You never see either — see
Correlation and
Multi-tenant SaaS.
A queue another deployment drains — its worker in another binary — takes a
marker of its own on this side, naming that queue and the payload both sides
agree on, #[queue(name = "billing-invoices", job = InvoiceCommand)]; the
push is then as typed as any other. A payload lives at the port, so the
producer links the type its consumer reads.
Arc<nest_rs::queue::Jobs> is the form to inject: the feature names no
backend, so it compiles unchanged against a different driver.
RedisQueueModule binds it over the producer it opens on the connection
RedisModule opens for its instance (a second
Valkey has one of its own).
When a push fails
Section titled “When a push fails”A push is refused before anything is filed when its options cannot be honoured
— QueueError::Unsupported for an option the backend does not declare,
QueueError::InvalidUniqueKey for a key no backend could file,
QueueError::InvalidOptions for a delay ending after 9999-12-31T23:59:59.999Z,
the last instant RFC 3339 writes (Delay::LATEST_DUE) — or when a unique key is
held. Past those, an error from the backend is a failure to
answer: the job may be queued and its reply lost, so a caller that retries such
a push can queue it twice unless it carries a unique key.
No backend holds a push past eighteen seconds. The port waits
BACKEND_TIMEOUT for the backend’s answer to a push or a cancel, then gives up
on the call and fails it with QueueError::Unanswered, naming the queue and the
call — which, like any failure to answer, may still have filed the job. It is a
net under the backend, never its budget: on Redis each command a push sends
waits at most <PREFIX>_REDIS__CONNECT_TIMEOUT_SECS, ten seconds by default, so a
Redis that stopped answering fails the push at that budget, with its own cause,
well before the net. Eighteen seconds stays under the HTTP edge’s request timeout,
so a push made inside a request fails with an error naming its queue while the
request can still answer. A backend that panics answering a push or a cancel
fails it the same way, as QueueError::Panicked naming the queue and the call —
the panic itself is filed once at error on nest_rs::queue, never handed to
your handler.
push_many hands the backend its jobs a hundred at a time (ENQUEUE_BATCH),
each call under the net, so the net holds a push of any size. It is not atomic.
When a call fails after the backend accepted earlier ones, the error is
QueueError::PartiallyQueued { receipts, source }: receipts names every job
of the accepted calls — queued, and to be left out of a retry or cancelled —
and source is the failure that stopped the push. The failing call’s own jobs
may be queued too, and no receipt says which, so retrying those can queue one
twice. A push failing at its first call returns that failure alone.
A push inside a transaction
Section titled “A push inside a transaction”A push written inside a transaction is written into it, and filed once it commits: The outbox.
Delays, unique keys and cancels
Section titled “Delays, unique keys and cancels”A push can hold its job back or keep one job per key, and a waiting job can be taken back: Delays, unique keys and cancels.
The HTTP adapter
Section titled “The HTTP adapter”The controller is a thin transport. It accepts a TranscodeDto — the
REST body, a distinct type from the queue’s TranscodeCommand — calls
the service, and answers 202: the job is queued, not done, and the reply
echoes nothing the client sent. No queue knowledge of its own.
Declare the DTO with #[input]: a Json<T> handler argument requires
schemars::JsonSchema (see extractors), and #[input]
derives it along with Deserialize and Validate. A hand-derived DTO needs
#[derive(schemars::JsonSchema)] and the schemars dependency to go with it.
use nest_rs::core::input;use uuid::Uuid;
#[input]pub struct TranscodeDto { pub file: Uuid,}file is the key an upload minted, parsed as a UUID at the edge: a name the
client chose is a 400 before the service runs.
use std::sync::Arc;
use nest_rs::authz::Manage;use nest_rs::http::{ Body, Header, HeaderValue, HttpError, Json, Opaque, Query, Response, Result, SseEvent, SseStream, StatusCode, controller, routes,};use nest_rs::throttler::{Throttle, ThrottlerGuard};
use crate::audio::{ AudioService, DirectUploadDto, PresignedUrlDto, StreamResumeDto, TranscodeDto, TranscodeState, Transcoding,};use crate::authn::AuthnGuard;use crate::authz::AuthzGuard;
#[controller(path = "/audio")]#[use_guards(AuthnGuard, AuthzGuard, ThrottlerGuard)]pub struct AudioController { #[inject] svc: Arc<AudioService>,}
#[routes]impl AudioController { #[post("/transcode")] #[authorize(Manage, Transcoding)] #[meta(Throttle::per_minute(20))] // … #[api(…)] #[http_code(202)] async fn transcode(&self, Json(body): Json<TranscodeDto>) -> Result<()> { self.svc.enqueue_transcode(body.file).await.opaque()?; Ok(()) }}Swap this for a #[resolver] mutation, a #[gateway] message, or a
#[scheduled] method on a timer — the controller is the only file that
changes. The service stays the single audited entry point, every
transport hits it through enqueue_transcode.
Why this rule
Section titled “Why this rule”Three reasons the push call hides behind a service:
- One audit point per queue.
AudioQueueis named in exactly one place per side. A grep forenqueue_transcodeshows every producer. - Pre-flight logic survives a swap of transport. If enqueueing needs an ability check, an idempotency key, a tracing field — the service holds it. Adding a second producer doesn’t duplicate that.
- The consumer can change without touching producers. Renaming the
queue, splitting
audiointoaudio.fastandaudio.slow, moving from Redis to another backend — one file changes, every adapter follows. And because the name lives on the#[queue]type, a rename that misses a call site is a compile error rather than a queue nobody drains.
The same shape the rest of the framework uses for Repo, just on the
push side. The service is the contract; transports adapt.
Reference
Section titled “Reference”crates/features/src/audio/command.rs—TranscodeCommand,AudioQueue.crates/features/src/audio/service.rs—enqueue_transcode.crates/features/src/audio/http/controller.rs— the HTTP producer.crates/features/src/audio/schedule/tasks.rs— the scheduled producer (a timer that calls the same service).crates/nest-rs-queue/src/jobs.rs—Jobs:push,push_with,push_many,push_many_with,cancelandcancel_unique.crates/nest-rs-queue/src/producer.rs—JobProducer, what a backend implements, and the path every push takes to it.
Going further
Section titled “Going further”- Wiring — producer-only vs worker apps sharing a features crate.
- Retries and failure — what happens once a worker drains the job.
- Queue — the section landing and worker overview.