Skip to main content

Scaling a consumer

On the previous page, one worker consumed the shipping consumer: a continuous pull loop that ships each order and acks. That kept up while Acme shipped a few orders an hour.

Then Acme's order volume climbed. Orders arrive faster than one worker can ship them, and the unshipped ones pile up in ORDERS.

You clear that backlog by adding workers. Point several processes at the same shipping consumer and they share the load: the server hands each stored order to exactly one worker. You reuse the ORDERS stream and shipping consumer you already have.

One consumer, many workers​

Each worker names the same shipping consumer and runs the same pull loop. Start that loop in several processes at once:

#!/bin/bash
# A worker loop for the shipping consumer.
# Run this same loop in three separate terminals to form a pool of
# three workers. All three share the one "shipping" consumer, and the
# server splits the stored messages across them.

# Each pass: pull one message, "process" it, and acknowledge it.
# --count 1 pulls a single message at a time.
# --ack acknowledges the message after it is received.
while true; do
  nats consumer next ORDERS shipping --count 1 --ack
done

# Publish work from a fourth terminal and watch it spread across the
# three running loops:
#   nats pub orders.shipped '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
#
# To see in-flight redelivery: stop one worker (Ctrl-C) right after it
# pulls a message but before it acks. After AckWait (30s default), the
# server redelivers that message to one of the surviving workers.

Open three terminals and run that loop in each. The ORDERS stream already holds the orders from earlier pages; now add a fresh handful of orders.shipped messages from a fourth terminal so all three workers have something to compete for:

nats pub orders.shipped '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
nats pub orders.shipped '{"order_id":"ord_2zr9","customer":"globex","total_cents":7800,"ts":"2026-05-22T10:14:25Z"}'
nats pub orders.shipped '{"order_id":"ord_5k1m","customer":"initech","total_cents":1500,"ts":"2026-05-22T10:14:29Z"}'

The server hands each waiting order to one worker, rotating round-robin through the workers that have a pull request open. No two workers get the same order, so the three split the backlog evenly:

A worker only takes a turn while it's actually asking. One that's still shipping an order has no pull request open, so the server skips it and gives the order to the next worker in line. Distribution follows demand: a faster worker pulls more often and ships more.

The consumer still tracks a single position through all of this. Ask the server and the acked count climbs as one number:

nats consumer info ORDERS shipping
State:

  Last Delivered Message: Consumer sequence: 3 Stream sequence: 6
     Acknowledgment Floor: Consumer sequence: 3 Stream sequence: 6
         Outstanding Acks: 0 out of maximum 1,000

The position belongs to the consumer, not to each worker, so it advances the same whether one process pulls or three.

This works on the log you already have​

ORDERS is the same Limits-retention stream from page one; nothing about it changed to make this work. Sharing is a property of the consumer, not the stream: point many workers at one consumer and the server splits the work on any stream.

One consequence matters here. An ack advances the position, it doesn't remove the order. The order stays in ORDERS for billing, analytics, and any consumer that reads the log later. The workers share a position, not the messages.

How this differs from a queue group​

If you read the Core Concepts, this looks like a queue group, but the two balance different things.

A queue group splits live messages across core NATS subscribers as each one arrives, with no storage behind it. A subscriber that's offline when a message arrives misses it for good.

Workers sharing a consumer split stored messages against the stream. A message waits in the stream until some worker pulls it and acks it, so a worker that's offline just leaves its share for the others.

Only the stream-backed split survives a worker dropping out or a restart.

When a worker crashes mid-message​

A worker pulls an order and starts shipping it. Before it acks, the process dies. On the consumer that order is still in progress: the server delivered it and is waiting on the ack. When AckWait runs out (30 seconds by default), the server hands the order to another worker. This is the redelivery loop from the acknowledgment page, now spread across the pool.

Watch it happen. Kill one worker mid-order, wait out AckWait, and the order reappears on a surviving worker. It ships once, because some worker eventually acks it.

Capping how much is in progress​

Each worker can hold an order in progress, so more workers mean more in progress at once. There's a ceiling on that.

The ceiling is MaxAckPending: how many delivered-but-unacked messages the consumer allows at once, default 1000. Hit it and the server stops delivering new orders until some get acked. The cap is shared across the whole consumer, not per worker: five workers get 1000 between them, not 1000 each.

You set it when you create or update the consumer:

nats consumer add ORDERS shipping --max-pending 1000

Set it too low and workers sit idle waiting for a slot. Set it too high and a slow ack leaves a big in-progress backlog that all redelivers at once if many workers die together.

Pitfalls​

Running several workers makes two consumer settings, AckWait and MaxAckPending, matter in practice.

A redelivered order can arrive at a second worker. When one worker crashes mid-order, the server gives that order to another worker after AckWait, so the work still gets done. The same order can then run twice, so key side effects by order_id. A redelivery shows up in the message's delivery count:

#!/bin/bash
# Redelivery means a worker can see the same order more than once: a
# crashed worker's in-flight message comes back after AckWait. Keep
# processing idempotent so shipping ord_8w2k twice is harmless.

# Whole-pool view of redelivery. Look for "Redelivered Messages".
# A climbing count is normal under churn; a large jump means a batch of
# workers died and their in-flight work came back to surviving workers.
nats consumer info ORDERS shipping

# Pull one order WITHOUT acking so you can inspect it first.
# If your worker keys side effects by order_id, a second delivery of
# ord_8w2k is a no-op instead of a double shipment.
nats consumer next ORDERS shipping --count 1 --no-ack

The pool shares work by demand, so you don't choose which worker gets an order. For that control, use priority groups: send everything to one worker until it fails, or keep a standby worker idle until the pool falls behind.

A low MaxAckPending starves a large set of workers. The cap is shared across the whole consumer, not per worker. Set it to 3 and only three messages are ever in progress, so ten workers leave seven of them idle no matter how much is stored. Set the cap to at least your worker count, with room to spare:

#!/bin/bash
# MaxAckPending is the ceiling on delivered-but-unacked messages across
# the WHOLE pool, not per worker. Set it too low and a large pool sits
# idle waiting for a slot. Size it to at least your worker count.

# Check the current cap. Look for "Outstanding Acks: N out of maximum M"
# in the output. M is MaxAckPending.
nats consumer info ORDERS shipping

# Raise the cap so every worker can hold a message at once, with room
# to spare. For ten shipping workers, a few thousand is comfortable.
nats consumer edit ORDERS shipping --max-pending 5000

A crashed worker holds its order until AckWait. The server can't tell a crash from slow work; it only knows the ack hasn't come. Until the timer runs out (30 seconds by default), that order stays in progress and goes to no one else. Set AckWait to your normal processing time: too short redelivers while a healthy worker is still working, too long leaves real failures stuck. The full set of in-progress and redelivery options (AckWait, MaxDeliver, backoff arrays) is in Reference → Consumer Configuration.

Where you are​

You now have:

  • One shipping consumer, unchanged from earlier pages.
  • Several workers pulling from it, splitting the stored messages between them.
  • A crashed worker's message redelivered after AckWait, not lost.
  • A name for the ceiling on how much runs at once: MaxAckPending, shared across every worker.

You scale by starting more processes; the consumer is unchanged. And because ORDERS is a log, it keeps every order after a worker handles it.

What's next​

Several workers on the ORDERS log still leave every order in the stream after one of them handles it. That's right for an audit log, but wrong for a backlog of jobs that should disappear once they're done. The next page builds the FULFILLMENT queue: a WorkQueue stream where each order to ship is claimed by one worker and removed on ack, the home these shipping workers belong on.

See also​