Skip to content

Concurrency and scaling

How many of a method's jobs run at once and how fast they may start — per replica, across the deployment — and adding replicas on queue depth.

Two #[process] keys decide how a method’s jobs are scheduled, and the platform decides the rest. concurrency bounds how many attempts run at once in one worker replica, throttle caps how many may start per window across the whole deployment, and the number of replicas is what an autoscaler changes:

crates/features/src/notifications/queue/processor.rs
#[processor]
impl NotificationsProcessor {
#[process(queue = NotifyQueue, retries = 3, concurrency = 4, throttle(limit = 120, window = "1m"))]
async fn notify(&self, job: NotifyCommand) -> Result<()> {
self.svc.persist(job).await?;
Ok(())
}
}

The worker states both at boot, beside the retry budget:

Terminal window
INFO nest_rs::queue: mounted host="NotificationsProcessor" method="notify" queue="notifications" retries=3 concurrency=4 throttle_limit=120 throttle_window_ms=60000
KeyDefaultWhat it countsAcross
concurrency = N1attempts running at onceone method, in one worker replica
throttle(limit = N, window = "…")noneattempts starting per windowone queue, every replica together
retries = N0attempts of one jobthe job, wherever it runs — see Retries and failure

concurrency = N runs at most N attempts of the method at once in one worker replica — one when the key is absent, so a method that declares nothing runs its jobs one after another. The bound belongs to the method: each method has permits of its own, so a slow queue never takes another’s. And it belongs to the replica: three replicas run up to three times N.

A job holds its permit for its whole delivery, and a replica whose permits are all taken asks for nothing: the jobs stay in Redis, where a peer replica reads them. A replica asks only for as many jobs as it has permits free — see Fetching on Redis — so it never holds a job it has not started.

concurrency is not a capability: every backend honours it, so a method runs the same bound whichever driver is underneath it.

The pool that caps a worker is the database’s. An attempt is one transaction by default, and that transaction holds one connection from the job’s first Repo call until it commits or rolls back. A replica whose methods all reach the database at the same moment wants as many connections as the sum of their concurrency values. Keep that sum within <PREFIX>_SEAORM__MAX_CONNECTIONS, or the attempts past it wait for a connection — and one that waits past <PREFIX>_SEAORM__CONNECT_TIMEOUT_SECS fails, retryably. A method declared transactional = false borrows a connection per statement instead.

Redis is not what bounds it: a replica reaches Redis over one multiplexed connection, plus one per method for its blocking read, and a running job holds none of its own.

On Redis, each method of each replica asks for as many jobs as it has permits free, with a blocking XREADGROUP on a connection of its own that waits at most a second. There is no poll: a job pushed while the read waits wakes it at once, so a method starts jobs as fast as its permits free, never concurrency per interval. Past what one replica’s permits run, throughput is a higher concurrency or more replicas.

A delayed job or a retry reaches the stream within a second of falling due. It waits off the stream, in …:due, until a running worker files it: each method of each replica runs an upkeep script at least once a second — sooner when the next job falls due sooner — and files up to a hundred due jobs per upkeep, so a burst of due retries is filed in steps rather than in one long script. See Delayed jobs.

An idle method costs Redis a fixed few calls: per method and per replica, one blocking read a second with a look for lapsed leases beside it — a look costs at most what ten thousand pending entries cost, however many run — one upkeep script a second, and one sweep of the consumer group a minute. Multiply by methods and replicas to size what an idle deployment sends.

throttle(limit = N, window = "…") caps how many of the method’s attempts may start in a window, counted across every replica draining the queue. Both keys are required, and the window is a whole number above zero with an ms, s, m or h suffix — "500ms", "30s", "5m", "1h".

Reach for a throttle when the constraint is a rate someone else imposes, such as a provider’s requests per minute; reach for concurrency when it is a resource the job holds while it runs. Both apply at once: a throttled method still runs at most its concurrency attempts at a time.

  • A retry is an attempt like any other, and counts against the limit. So does an attempt a drain interrupts: it started, and did whatever it did downstream before the cut.
  • A start over the limit is never read. The window is counted before a read asks for jobs, and the read asks for no more than the window has left, so a job past the limit stays where it waits, keeps its attempt number, costs the retry budget nothing, and is never dropped.
  • It limits the deployment. On Redis the count is one key per queue, counted in one script before each read, so every replica counts against one number and no two reads race. A read gives back, while the window lasts, the starts it counted and found no job for.
  • A full window holds the reads. Once the window is full, that replica’s reads for the method wait until it ends — the instant the count’s key expires — and say so at debug, queue receive waits for its throttle's next window. The backlog waits on the stream, where peers and an autoscaler see it, and no replica holds a job it read for a window to open.

After 125 pushes to the method above on one replica, 120 started and the window’s count had 42 seconds left; the other five wait on the stream for it to end, and nothing was held back:

Terminal window
$ valkey-cli GET 'nestrs:queue:{notifications}:throttle'
"120"
$ valkey-cli PTTL 'nestrs:queue:{notifications}:throttle'
(integer) 42022
$ valkey-cli XLEN 'nestrs:queue:{notifications}:jobs'
(integer) 5
$ valkey-cli ZCARD 'nestrs:queue:{notifications}:due'
(integer) 0

The Redis window is fixed and opens with its first start: the key expires window later, and the count starts again with the next start after that — the scheme the HTTP rate limiter uses. Nothing exceeds the limit within one window, and starts on either side of a window’s end can reach twice the limit within a short span. The window is kept on Redis’s clock, in milliseconds, so a window under a second holds as declared.

A throttle needs a backend declaring Capability::Throttle. Redis declares it; on a backend that does not, the worker’s boot fails naming the declaration, the method and the backend.

Throughput past one replica’s permits comes from running more replicas of the worker app — the unit the platform already schedules, meters and restarts:

Terminal window
$ kubectl scale deploy/worker --replicas=6

Every replica reads each queue as a consumer of its own in the queue’s consumer group, and no two of them receive the same job. A replica starting takes nothing from its peers: it reads what waits on the stream, and a job moves to another replica only once its lease lapses. See Leases and redelivery.

A worker whose jobs wait on I/O — a storage round-trip, one INSERT — sits near 0% CPU however deep its backlog, so a CPU-driven autoscaler never fires. Queue depth is the signal that moves with the work: nestrs:queue:{<queue>}:jobs is the stream a worker reads, and KEDA’s redis-streams scaler reads its length.

worker-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: worker
spec:
scaleTargetRef:
name: worker
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: redis-streams
metadata:
address: redis:6379
enableTLS: "true"
stream: "nestrs:queue:{audio}:jobs"
streamLength: "20"

streamLength is the backlog one replica is sized for, so it moves with the method’s concurrency: at four at once, 20 is five jobs per running slot, and 40 on the stream ask for two replicas. Declare one trigger per queue the worker drains, and KEDA follows the busiest. KEDA dials Redis itself, so address is host:port rather than the URL the framework reads; a password, TLS and the database index of the URL’s path (databaseIndex) are the scaler’s own metadata, and its user takes the KEDA rule.

Under Sentinel and on a Cluster, KEDA has a scaler of its own for each, which takes every address the URL names:

Sentinel
- type: redis-sentinel-streams
metadata:
addresses: sentinel-a:26379,sentinel-b:26379,sentinel-c:26379
sentinelMaster: orders
enableTLS: "true"
stream: "nestrs:queue:{audio}:jobs"
streamLength: "20"
Cluster
- type: redis-cluster-streams
metadata:
addresses: node-a:6379,node-b:6379,node-c:6379
enableTLS: "true"
stream: "nestrs:queue:{audio}:jobs"
streamLength: "20"

sentinelMaster is the URL’s sentinelServiceName, and the sentinels’ own credentials go in sentinelUsername and sentinelPassword. KEDA’s cluster client takes no database, so a queue it scales on a Cluster sits in database 0. What KEDA’s user adds on each topology is on ACL on Sentinel and Cluster.

The trigger counts waiting and running jobs. The stream holds a job until it ends — its outcome deletes its entry — so a replica busy with long jobs still counts them, and the length falls only as jobs finish. Once the backlog drains, KEDA scales down, and the pod it stops drains: running attempts get the shutdown window, and what still runs when it closes is interrupted and handed back to the stream, to run again on another replica from the start — see Shutdown. Nothing is lost, but the work done so far is, unless the method keeps a checkpoint. Pair the trigger with the settings that bound this: a scale-down stabilization window at least as long as a job runs (advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.stabilizationWindowSeconds on the ScaledObject, and a cooldownPeriod for the last replica), and a pod terminationGracePeriodSeconds at least 9.5 seconds above the drain window, <PREFIX>_QUEUE__SHUTDOWN_TIMEOUT_SECS (20 by default), plus any preStop delay — the shutdown hooks and the two flushes follow the drain — raised as far as a job’s length when jobs are long.

  • concurrency = 0 does not compile: #[process] `concurrency` takes a whole number of at least 1 — how many jobs of this method one worker replica runs at once; 0 would never run a job.
  • throttle(limit = 10) with no window does not compile: #[process] `throttle` needs both `limit` and `window` — write `throttle(limit = 10, window = "1m")`.
  • window = "60" does not compile: the window takes a duration literal, and a number with no suffix is not one.
  • A drain window under a second or above an hour fails the boot, from the environment or pinned in code, naming <PREFIX>_QUEUE__SHUTDOWN_TIMEOUT_SECS and why the bound is there; so does a lease out of the same range, naming <PREFIX>_REDIS__QUEUE__LEASE_SECS.
  • A trigger naming a stream the worker does not read — a key without its braces, or a 6.x list — reads zero forever and holds the deployment at minReplicaCount, saying nothing.
  • crates/nest-rs-queue/src/process_options.rs — ProcessOptions and Throttle.
  • crates/nest-rs-queue/src/worker.rs — each method’s permits, the read for as many jobs as are free, and the upkeep’s cadence.
  • crates/nest-rs-redis/src/queue/consumer.rs — the blocking read on its own connection, the throttle counted before it, and the upkeep that files due jobs.
  • crates/nest-rs-redis/src/queue/scripts.rs — the throttle’s window, counted and given back.
  • crates/nest-rs-redis/tests/e2e/queue/consumer.rs — the behaviour kit’s concurrency and delayed push, and a throttle across replicas, against live Redis.