Skip to main content

Retention policies

The previous page shaped ORDERS with limits: how many messages it keeps, for how long, in how many bytes. Those limits decide when a message leaves the stream because the stream ran out of room.

There's a second, separate question: should a message ever leave the stream because a consumer finished with it? Limits don't ask that. A message capped only by age or size stays until its limit hits, read or unread, acked or not.

Some workloads need the other behavior. A job queue wants a message gone once a worker completes it. A fan-out wants a message gone once every interested consumer has seen it. Limits can't express either.

The retention policy is the field that does. It decides what makes a message ready to leave the stream. You picked one already, by accepting a default, when you created ORDERS.

The three policies​

A stream has exactly one retention policy, fixed by the retention field. There are three values.

Limits is the default, and the one ORDERS has. Messages stay until a limit is reached: MaxMsgs, MaxBytes, or MaxAge, whichever comes first. Consumers reading and acking messages has no effect on what the stream keeps. The stream is a log, and the log holds everything inside its limits.

Interest keeps a message only while some consumer still wants it. A message is removed once every consumer on the stream has acked it. If no consumer is interested in a subject, a message on that subject is removed right away.

WorkQueue keeps a message only until one consumer acks it. The first ack removes the message for everyone. Each message is delivered once and then removed.

The three policies differ in who decides a message is finished. Under Limits, the limits decide. Under Interest, every consumer must ack before the message is removed. Under WorkQueue, the first consumer to ack removes it.

Pick the policy from the kind of work​

Choose a retention policy from the kind of work the stream does. The policy follows from the work, not the other way around.

An audit log or event history → Limits. You want to keep every message for a window of time no matter who read it, and you want to replay from any point. ORDERS is this kind of stream. Late consumers, re-reads, and the replay on the reading-back page all depend on messages staying after they're consumed. Limits is the only policy that allows that, which is why it's the default and why ORDERS keeps it.

A fan-out where every consumer must process each message → Interest. Several independent services each need to handle every order once, and once they all have, the message is no longer needed. The stream stays small because it drops a message once the last interested consumer is done. You get fan-out delivery without an ever-growing log.

A job queue where each message is work for one worker → WorkQueue. This is the home for Acme's shipping workers from Scaling a consumer. Each message is an order to ship. One shipping worker claims it, ships it, and acks, and then the task is removed so no one ships the same order twice. The queue drains as the workers keep up, where a log would only grow.

A WorkQueue stream for the shipping work​

ORDERS stays Limits; don't change it — it's the record of what customers did. But Acme also has work to do on each order: when one is paid, a shipping worker has to ship it, exactly once. Ship the same order twice and a second parcel goes out the door. That's a job queue, not a log — each task goes to one worker and is gone once it's done. Give it its own FULFILLMENT stream with WorkQueue retention:

#!/bin/bash

# A WorkQueue stream for the shipping work. Leave ORDERS alone — that's
# the record. FULFILLMENT is a separate queue of paid orders waiting to
# ship, worked by the shipping workers from the Scaling a consumer page.

# --retention work sets the WorkQueue policy. natscli accepts "work" and
# "workq" as aliases. Under WorkQueue a message is removed the moment the
# first consumer acks it, so the queue drains as the workers keep up.
nats stream add FULFILLMENT \
  --subjects "fulfill.>" \
  --retention work \
  --defaults

# Inspect it — the Options block now reads Retention: WorkQueue, where
# ORDERS reads Retention: Limits.
nats stream info FULFILLMENT

# Enqueue one order to ship, then have a shipping worker pull and ack it.
# After the ack the task is gone — the message count drops back to zero,
# which no limit on a Limits stream like ORDERS would ever do.
nats pub fulfill.us '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
nats consumer add FULFILLMENT shippers --pull --ack explicit --defaults
nats consumer next FULFILLMENT shippers --count 1 --ack

# Confirm the ack removed the task: Messages is back to 0.
nats stream info FULFILLMENT

The --retention work flag is the only change from how you built ORDERS. nats stream info FULFILLMENT shows it in the Options block:

nats stream info FULFILLMENT
             Subjects: fulfill.>

Options:

            Retention: WorkQueue
       Discard Policy: Old

Enqueue one order to ship and have a shipping worker ack it, and the stream's message count drops back to zero. The ack removed the task, which no limit on ORDERS does. The worker publishes orders.shipped back to ORDERS as it finishes, so the record lands in the log while the task drains from the queue.

Switching retention on a live stream​

Set retention when you create the stream, and leave it there.

The server allows exactly one live change: it swaps Limits and Interest, in either direction. Anything involving WorkQueue is locked — see the pitfall below. And even the allowed swap applies to messages already stored, right away. Say a stream has been collecting an audit history under Limits and you switch it to Interest. From that moment the server removes any message every consumer has already acked, and any message on a subject no consumer is interested in — including history you meant to keep.

Treat the policy as fixed at creation. If you want a different policy than the stream has, create a new stream with the right policy rather than editing the running one. ORDERS was created as Limits on purpose, and it stays Limits.

How Interest and WorkQueue can go wrong​

Interest and WorkQueue each have a way they can go wrong. Know it before you use them.

Interest can fill the disk. A message is only removed once all consumers ack it. The stream tracks the lowest ack position across every consumer and only deletes up to that point. So a single slow consumer holds up cleanup for the whole stream. If a consumer stalls (a stuck worker, a service that's down), its unacked messages never become ready to leave, and the stream grows until it hits its limits or runs out of room. Interest retention still needs limits set, and it makes watching consumer health more important.

WorkQueue delivers each message once. The first ack removes the message for everyone. So two separate consumers on a WorkQueue stream don't each get a full copy. They split the messages between them, and neither sees the whole stream. For several consumers that each see every message, use Interest or Limits. A worker pool sharing one consumer (the worker-pool page) works on WorkQueue. Several separate consumers that each expect the full stream does not.

The full set of retention behavior, including how Interest and WorkQueue interact with stream republish, mirrors, and sources, is in Reference → Stream Configuration. This page uses only the three retention values.

Pitfalls​

Both of these are WorkQueue constraints. The server checks them when you create or edit a stream, so you find out right away rather than in production.

Retention to or from WorkQueue is locked after creation. The earlier section covered the Limits–Interest swap the server allows, and how even that rewrites your history. The rule underneath it is stricter: the server rejects any change that adds or removes WorkQueue. A stream that isn't WorkQueue at creation can't become one, and a WorkQueue stream can't change to another policy.

Don't plan a migration path that edits retention into or out of WorkQueue. Create a new stream with the policy you want and move the data. The edit below is rejected with stream configuration update can not change retention policy to/from workqueue.

#!/bin/bash

# Pitfall: you cannot switch a stream's retention to or from WorkQueue
# on a live stream. Limits and Interest can swap, but WorkQueue is fixed
# at creation. Try to move FULFILLMENT (WorkQueue) to Limits and the
# server refuses:
#   stream configuration update can not change retention policy to/from
#   workqueue
# The command exits non-zero — FULFILLMENT keeps its WorkQueue policy.
nats stream edit FULFILLMENT --retention limits --force

# The safe move is a new stream with the policy you want, then migrate.
# ORDERS itself stays Limits — never edit a live stream's retention to
# WorkQueue hoping to convert it.
nats stream info FULFILLMENT

WorkQueue rejects consumers that overlap. The first ack removes a message for everyone, so the server won't let two consumers claim the same message. Adding a second unfiltered consumer, or two consumers whose filters overlap, fails the create: multiple non-filtered consumers not allowed on workqueue stream, or filtered consumer not unique on workqueue stream for overlapping filters.

Give each consumer a filter that splits the subjects between them, so no message belongs to two consumers. A worker pool sharing one consumer is the other valid setup; see A pool of workers.

#!/bin/bash

# Pitfall: a WorkQueue stream rejects a second consumer whose subjects
# overlap an existing one. The server won't let two consumers fight over
# the same task, so it refuses the create up front.

# FULFILLMENT is the WorkQueue stream from earlier on this page. Add one
# unfiltered consumer — fine, it owns the whole queue.
nats consumer add FULFILLMENT shippers --pull --ack explicit --defaults

# Now try to add a second unfiltered consumer. The server rejects it:
#   multiple non-filtered consumers not allowed on workqueue stream
# (error 10099). The command exits non-zero.
nats consumer add FULFILLMENT eu-shippers --pull --ack explicit --defaults

# The fix is non-overlapping filters so each task belongs to exactly one
# consumer — here, one shipper consumer per region. Delete the broad
# consumer, then split by subject.
nats consumer rm FULFILLMENT shippers --force

nats consumer add FULFILLMENT us-shippers --pull --ack explicit --filter "fulfill.us" --defaults
nats consumer add FULFILLMENT eu-shippers --pull --ack explicit --filter "fulfill.eu" --defaults

# These two coexist because fulfill.us and fulfill.eu never collide.
# A wildcard like fulfill.> would overlap both and be rejected with
# "filtered consumer not unique on workqueue stream" (error 10100).

Where you are​

ORDERS is unchanged. It's a Limits stream that holds its order history and lets late and repeat consumers replay.

You now have:

  • The three retention policies (Limits, Interest, WorkQueue) and the one question that separates them: who decides a message is finished.
  • Which policy fits which kind of work: audit log → Limits, fan-out → Interest, job queue → WorkQueue.
  • A FULFILLMENT WorkQueue stream — the shipping workers' queue — that dropped a task on ack while ORDERS kept the record.
  • The rule that retention is fixed at creation, not switched on a live stream.

What's next​

Retention removes messages on a schedule the server runs. The next page covers removing them by hand: deleting a single message and purging the stream.

See also​