Concurrency and scaling
How many of a method's jobs run at once and how fast they may start — per replica, across the deployment — and adding replicas on queue depth.
Two #[process] keys decide how a method’s jobs are scheduled, and the platform
decides the rest. concurrency bounds how many attempts run at once in one
worker replica, throttle caps how many may start per window across the whole
deployment, and the number of replicas is what an autoscaler changes:
#[processor]impl NotificationsProcessor { #[process(queue = NotifyQueue, retries = 3, concurrency = 4, throttle(limit = 120, window = "1m"))] async fn notify(&self, job: NotifyCommand) -> Result<()> { self.svc.persist(job).await?; Ok(()) }}The worker states both at boot, beside the retry budget:
INFO nest_rs::queue: mounted host="NotificationsProcessor" method="notify" queue="notifications" retries=3 concurrency=4 throttle_limit=120 throttle_window_ms=60000Where each key applies
Section titled “Where each key applies”| Key | Default | What it counts | Across |
|---|---|---|---|
concurrency = N | 1 | attempts running at once | one method, in one worker replica |
throttle(limit = N, window = "…") | none | attempts starting per window | one queue, every replica together |
retries = N | 0 | attempts of one job | the job, wherever it runs — see Retries and failure |
Concurrency
Section titled “Concurrency”concurrency = N runs at most N attempts of the method at once in one worker
replica — one when the key is absent, so a method that declares nothing runs its
jobs one after another. The bound belongs to the method: each method has permits
of its own, so a slow queue never takes another’s. And it belongs to the
replica: three replicas run up to three times N.
A job holds its permit for its whole delivery, and a replica whose permits are all taken asks for nothing: the jobs stay in Redis, where a peer replica reads them. A replica asks only for as many jobs as it has permits free — see Fetching on Redis — so it never holds a job it has not started.
concurrency is not a capability: every backend honours it, so a method runs
the same bound whichever driver is underneath it.
Sizing against the database pool
Section titled “Sizing against the database pool”The pool that caps a worker is the database’s. An attempt is one transaction by
default, and that transaction holds one connection from the job’s first Repo
call until it commits or rolls back. A replica whose methods all reach the
database at the same moment wants as many connections as the sum of their
concurrency values. Keep that sum within <PREFIX>_SEAORM__MAX_CONNECTIONS, or
the attempts past it wait for a connection — and one that waits past
<PREFIX>_SEAORM__CONNECT_TIMEOUT_SECS fails, retryably. A method declared
transactional = false borrows a connection per statement instead.
Redis is not what bounds it: a replica reaches Redis over one multiplexed connection, plus one per method for its blocking read, and a running job holds none of its own.
Fetching on Redis
Section titled “Fetching on Redis”On Redis, each method of each replica asks for as many jobs as it has permits
free, with a blocking XREADGROUP on a connection of its own that waits at most
a second. There is no poll: a job pushed while the read waits wakes it at once,
so a method starts jobs as fast as its permits free, never concurrency per
interval. Past what one replica’s permits run, throughput is a higher
concurrency or more replicas.
A delayed job or a retry reaches the stream within a second of falling due.
It waits off the stream, in …:due, until a running worker files it: each
method of each replica runs an upkeep script at least once a second — sooner
when the next job falls due sooner — and files up to a hundred due jobs per
upkeep, so a burst of due retries is filed in steps rather than in one long
script. See Delayed jobs.
An idle method costs Redis a fixed few calls: per method and per replica, one blocking read a second with a look for lapsed leases beside it — a look costs at most what ten thousand pending entries cost, however many run — one upkeep script a second, and one sweep of the consumer group a minute. Multiply by methods and replicas to size what an idle deployment sends.
Throttling starts
Section titled “Throttling starts”throttle(limit = N, window = "…") caps how many of the method’s attempts may
start in a window, counted across every replica draining the queue. Both
keys are required, and the window is a whole number above zero with an ms,
s, m or h suffix — "500ms", "30s", "5m", "1h".
Reach for a throttle when the constraint is a rate someone else imposes, such as
a provider’s requests per minute; reach for concurrency when it is a resource
the job holds while it runs. Both apply at once: a throttled method still runs at
most its concurrency attempts at a time.
- A retry is an attempt like any other, and counts against the limit. So does an attempt a drain interrupts: it started, and did whatever it did downstream before the cut.
- A start over the limit is never read. The window is counted before a read asks for jobs, and the read asks for no more than the window has left, so a job past the limit stays where it waits, keeps its attempt number, costs the retry budget nothing, and is never dropped.
- It limits the deployment. On Redis the count is one key per queue, counted in one script before each read, so every replica counts against one number and no two reads race. A read gives back, while the window lasts, the starts it counted and found no job for.
- A full window holds the reads. Once the window is full, that replica’s
reads for the method wait until it ends — the instant the count’s key expires
— and say so at
debug,queue receive waits for its throttle's next window. The backlog waits on the stream, where peers and an autoscaler see it, and no replica holds a job it read for a window to open.
After 125 pushes to the method above on one replica, 120 started and the window’s count had 42 seconds left; the other five wait on the stream for it to end, and nothing was held back:
$ valkey-cli GET 'nestrs:queue:{notifications}:throttle'"120"$ valkey-cli PTTL 'nestrs:queue:{notifications}:throttle'(integer) 42022$ valkey-cli XLEN 'nestrs:queue:{notifications}:jobs'(integer) 5$ valkey-cli ZCARD 'nestrs:queue:{notifications}:due'(integer) 0The Redis window is fixed and opens with its first start: the key expires
window later, and the count starts again with the next start after that — the
scheme the HTTP rate limiter uses. Nothing exceeds the limit
within one window, and starts on either side of a window’s end can reach twice
the limit within a short span. The window is kept on Redis’s clock, in
milliseconds, so a window under a second holds as declared.
A throttle needs a backend declaring Capability::Throttle. Redis declares it;
on a backend that does not, the worker’s boot fails naming the declaration, the
method and the backend.
Adding replicas
Section titled “Adding replicas”Throughput past one replica’s permits comes from running more replicas of the worker app — the unit the platform already schedules, meters and restarts:
$ kubectl scale deploy/worker --replicas=6Every replica reads each queue as a consumer of its own in the queue’s consumer group, and no two of them receive the same job. A replica starting takes nothing from its peers: it reads what waits on the stream, and a job moves to another replica only once its lease lapses. See Leases and redelivery.
On queue depth, with KEDA
Section titled “On queue depth, with KEDA”A worker whose jobs wait on I/O — a storage round-trip, one INSERT — sits near
0% CPU however deep its backlog, so a CPU-driven autoscaler never fires. Queue
depth is the signal that moves with the work: nestrs:queue:{<queue>}:jobs is
the stream a worker reads, and KEDA’s redis-streams scaler
reads its length.
apiVersion: keda.sh/v1alpha1kind: ScaledObjectmetadata: name: workerspec: scaleTargetRef: name: worker minReplicaCount: 1 maxReplicaCount: 10 triggers: - type: redis-streams metadata: address: redis:6379 enableTLS: "true" stream: "nestrs:queue:{audio}:jobs" streamLength: "20"streamLength is the backlog one replica is sized for, so it moves with the
method’s concurrency: at four at once, 20 is five jobs per running slot, and
40 on the stream ask for two replicas. Declare one trigger per queue the worker
drains, and KEDA follows the busiest. KEDA dials Redis itself, so address is
host:port rather than the URL the framework reads; a password, TLS and the
database index of the URL’s path (databaseIndex) are the scaler’s own
metadata, and its user takes the KEDA rule.
Under Sentinel and on a Cluster, KEDA has a scaler of its own for each, which takes every address the URL names:
- type: redis-sentinel-streams metadata: addresses: sentinel-a:26379,sentinel-b:26379,sentinel-c:26379 sentinelMaster: orders enableTLS: "true" stream: "nestrs:queue:{audio}:jobs" streamLength: "20" - type: redis-cluster-streams metadata: addresses: node-a:6379,node-b:6379,node-c:6379 enableTLS: "true" stream: "nestrs:queue:{audio}:jobs" streamLength: "20"sentinelMaster is the URL’s sentinelServiceName, and the sentinels’ own
credentials go in sentinelUsername and sentinelPassword. KEDA’s cluster
client takes no database, so a queue it scales on a Cluster sits in database 0.
What KEDA’s user adds on each topology is on
ACL on Sentinel and Cluster.
The trigger counts waiting and running jobs. The stream holds a job until it
ends — its outcome deletes its entry — so a replica busy with long jobs still
counts them, and the length falls only as jobs finish. Once the backlog drains,
KEDA scales down, and the pod it stops drains: running attempts get the shutdown
window, and what still runs when it closes is interrupted and handed back to the
stream, to run again on another replica from the start — see
Shutdown. Nothing is lost, but the work done so far
is, unless the method keeps a
checkpoint. Pair the trigger with the settings
that bound this: a scale-down stabilization window at least as long as a job
runs (advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.stabilizationWindowSeconds
on the ScaledObject, and a cooldownPeriod for the last replica), and a pod
terminationGracePeriodSeconds at least 9.5 seconds above the drain window,
<PREFIX>_QUEUE__SHUTDOWN_TIMEOUT_SECS (20 by default), plus any preStop
delay — the shutdown hooks and the two flushes follow the drain — raised as far
as a job’s length when jobs are long.
What fails if you get it wrong
Section titled “What fails if you get it wrong”concurrency = 0does not compile:#[process] `concurrency` takes a whole number of at least 1 — how many jobs of this method one worker replica runs at once; 0 would never run a job.throttle(limit = 10)with no window does not compile:#[process] `throttle` needs both `limit` and `window` — write `throttle(limit = 10, window = "1m")`.window = "60"does not compile: the window takes a duration literal, and a number with no suffix is not one.- A drain window under a second or above an hour fails the boot, from the
environment or pinned in code, naming
<PREFIX>_QUEUE__SHUTDOWN_TIMEOUT_SECSand why the bound is there; so does a lease out of the same range, naming<PREFIX>_REDIS__QUEUE__LEASE_SECS. - A trigger naming a stream the worker does not read — a key without its
braces, or a 6.x list — reads zero forever and holds the deployment at
minReplicaCount, saying nothing.
Reference
Section titled “Reference”crates/nest-rs-queue/src/process_options.rs—ProcessOptionsandThrottle.crates/nest-rs-queue/src/worker.rs— each method’s permits, the read for as many jobs as are free, and the upkeep’s cadence.crates/nest-rs-redis/src/queue/consumer.rs— the blocking read on its own connection, the throttle counted before it, and the upkeep that files due jobs.crates/nest-rs-redis/src/queue/scripts.rs— the throttle’s window, counted and given back.crates/nest-rs-redis/tests/e2e/queue/consumer.rs— the behaviour kit’s concurrency and delayed push, and a throttle across replicas, against live Redis.
Going further
Section titled “Going further”- Leases and redelivery — what a replica starting, dying or stopping does to the jobs it holds.
- Retries and failure — the budget a retry spends.
- Repo and the executor — what one transaction per attempt holds.