Skip to main content

Reconnection

On the last page order-svc opened a connection with a name, a server pool, and a sane connect timeout. That connection works right up until the server it landed on goes away: a restart, a deploy, a node lost in the n1/n2/n3 cluster. A naive client stops there.

This page makes the connection survive that. When the link drops, the client cycles the server pool, waits between tries, and rejoins. While it's away, the publishes the application keeps making are retained: they queue and flush once the link is back. By the end, order-svc and the JetStream consumers continue through a server going away without the application being affected.

Two new ideas carry the page: reconnect with backoff and jitter, and the reconnect buffer. We define each before we use it. Backoff builds on the state a dropped connection moves into, the RECONNECTING state, so that's where we start.

The disconnect, and the RECONNECTING state​

A disconnect is the moment the client notices the link is gone. The client's read loop sees the socket close, or a keepalive PING goes unanswered, and the connection leaves the CONNECTED state. The client fires a disconnect callback so the application can log it, but it doesn't close. Instead it moves to a new state: RECONNECTING.

In RECONNECTING the client is trying to re-establish the connection. It walks the server pool (the same list of URLs from the connecting page), dialing each in turn. The cluster is server-side; from the client's seat it's just a pool of addresses to try. When one of them answers and the handshake completes, the client returns to CONNECTED and fires a reconnect callback.

CONNECTED ──disconnect──▶ RECONNECTING ──+OK──▶ CONNECTED
                              │
                              └──pool exhausted, retries spent──▶ CLOSED

Reconnect is on by default in every client. The option that governs it is AllowReconnect; leave it true. Turning it off makes the connection close on the first disconnect, which is rarely what a service wants.

Backoff and jitter​

The client doesn't retry the pool at full rate. After it has tried every URL once and none answered, it pauses before the next sweep. That pause is backoff: the growing wait between reconnect attempts. Without it, a client whose server just died would spin a tight dial loop and consume CPU against a pool that isn't ready yet.

The base wait is ReconnectWait, default two seconds. On top of that the client adds jitter, a small random amount mixed into the wait so a thousand clients that all lost the same server don't retry in lockstep and overload the remaining server. The default jitter is 100ms on a plaintext link and 1s on a TLS link, because a TLS handshake costs more and you want the retries spread wider.

MaxReconnect bounds how many attempts the client makes before it gives up and moves to CLOSED. The default is 60. For a long-lived service that's the wrong value: you want it to keep trying forever. A negative MaxReconnect means retry without limit. Set it to -1 for order-svc so a long outage never closes the connection out from under the service.

Here's order-svc connecting to the pool with reconnect tuned for a long-lived service: unlimited retries, a two-second base wait, and jitter left at its sensible default.

#!/bin/bash

# Connect order-svc to the n1/n2/n3 server pool with reconnect tuned
# for a long-lived service, then publish one order.
#
# The CLI reconnects automatically and cannot script every client
# option (ReconnectWait, ReconnectJitter, a negative MaxReconnect for
# unlimited retries) the way the library tabs do. What it can show is
# the pool itself: pass the whole cluster as the server list so the
# client has somewhere to fail over to when one node goes away.
#
# --server takes a comma-separated pool. The client randomizes it by
# default and, on a disconnect, cycles the list with backoff + jitter
# until one node answers. --connection-name makes order-svc show up by
# name in `nats server report connections`.

# Publish a single order to orders.created against the pool. If the node
# the client first lands on goes away, the client reconnects to another
# member of the pool on its own and the publish rides through.
nats pub orders.created \
  '{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}' \
  --server "nats://n1:4222,nats://n2:4222,nats://n3:4222" \
  --connection-name order-svc

The published message is the same canonical order shape used everywhere in this chapter:

{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}

The full set of connection options lives in Reference. Here we cover only the ones that change how a connection behaves under fault.

Observe a reconnect​

You don't need a cluster to see this. Point a subscriber at a single nats-server, then kill and restart that server underneath it. The client logs the disconnect, cycles its pool, and rejoins on its own.

In one terminal, start a server:

nats-server

In a second terminal, subscribe with reconnect logging on:

nats sub "orders.>" --connection-name order-svc

Now go back to the first terminal, stop the server with Ctrl+C, and start it again. The subscriber prints a disconnect, then a reconnect a moment later, and resumes receiving without you restarting it. The subscription was re-sent automatically on the new connection; you didn't re-issue nats sub.

Against the real n1/n2/n3 pool the same thing happens, except the client rejoins on a different server while the failed one is still down. Why the node went away, and how the surviving nodes carry the load, is the cluster's job; see Topologies. From the client's seat it's one disconnect and one reconnect.

The reconnect buffer​

A reconnecting client has a problem the subscriber demo doesn't show: what about the publishes the application makes while the link is down? order-svc doesn't stop creating orders just because its server restarted.

The answer is the reconnect buffer: a client-side queue that holds outbound publishes during RECONNECTING and flushes them, in order, the moment the connection is back. The application calls publish as normal; the client holds the publishes through the gap. The default size is 8 MB (ReconnectBufSize).

Flush is not a delivery guarantee. On reconnect the client first restores its subscriptions, then re-sends the buffered publishes over the new connection in the order they were made. Each one is still a plain core publish, carrying the same at-most-once guarantee as any other. The buffer holds the publishes through the gap, but it doesn't turn a publish into a durable, acknowledged write. For the exact flush sequence and option knobs, see Reference.

That buffer is bounded on purpose. If the outage runs long enough that publishes accumulate past 8 MB, the next publish fails rather than letting the client consume unbounded memory. In Go the error is ErrReconnectBufExceeded; every client surfaces the same condition under its own name. Treat that error as a signal to slow down or shed load; it shouldn't be ignored.

One thing the reconnect buffer does not do is restore a JetStream consumer's place in the stream. When the connection comes back the consumer re-subscribes, but where it was up to (what it had acknowledged) is the JetStream layer's bookkeeping, not the connection's. That boundary lives in JetStream → Acknowledgment.

Pitfalls​

A few traps turn a working reconnect into a silent failure. Each comes back to this page's two ideas: backoff and the reconnect buffer.

The default retry limit gives up on a long outage. MaxReconnect defaults to 60. A service that hits a 60-attempt run of bad luck (a rolling cluster upgrade, a long network partition) exhausts the count, the connection moves to CLOSED, and it never comes back on its own. For anything long-lived, set MaxReconnect to -1 so the client retries forever, and watch the reconnect-error callback so the outage is visible in your logs rather than silent.

Wire up unlimited retries and a callback that records every failed attempt, so a long outage is logged rather than silently fatal:

#!/bin/bash

# Make a long outage loud instead of lethal: keep retrying forever and
# make every disconnect/reconnect visible.
#
# The library tabs set MaxReconnect to -1 (retry without limit) and
# register a reconnect-error callback that logs each failed attempt.
# The CLI has no callback to register, but it does keep reconnecting on
# its own and prints disconnect/reconnect lines while it runs — which is
# the observable half of the same behavior.
#
# To SEE it: run this against a single local nats-server, then stop and
# restart that server. The subscriber logs a disconnect, cycles its
# pool, and resumes — you never re-run the command. Pointed at the
# n1/n2/n3 pool, it rejoins on a surviving node instead.

# Start a local server in another terminal first:
#
#   nats-server
#
# Then subscribe across the pool. Leave it running, kill the server it
# is on, and watch it reconnect on its own.
nats sub "orders.>" \
  --server "nats://n1:4222,nats://n2:4222,nats://n3:4222" \
  --connection-name order-svc

A zero or fixed backoff overloads the remaining server. If you replace the default backoff with a custom delay of zero, the client spins a tight CPU loop against a pool that isn't ready. If you make it a fixed delay with no jitter, a fleet of clients that all lost the same server retry in lockstep and send requests to the one remaining server at the same instant. Always keep a non-zero wait, and always keep jitter, which is what spreads those retries across time.

A full reconnect buffer drops your next publish. The reconnect buffer is 8 MB by default, not infinite. A long outage with a busy publisher overflows it, and the publish that overflows fails. Don't ignore that error. Catch it, back off publishing, and let the buffer drain once the link returns instead of retrying the publish in a tight loop.

A stale connection on an overloaded server goes undetected for two minutes. A disconnect is usually instant, but a connection that's merely wedged (the server alive but not reading) is only caught by the missed-keepalive path, which can take up to two minutes at the default ping interval. Under heavy load, lower the ping interval so a wedged link is detected in seconds instead of minutes. The full set of keepalive options is in Reference.

Where you are​

order-svc and the JetStream consumers now survive a server going away. You have:

  • a connection that detects a disconnect and enters RECONNECTING instead of dying
  • backoff plus jitter cycling the n1/n2/n3 pool, with MaxReconnect set to -1 for unlimited retries on a long-lived service
  • subscriptions that auto-restore, and publishes that buffer through the gap and flush on reconnect

The connection rides through a fault. What it doesn't yet do is exit cleanly: a Ctrl+C today drops whatever is in flight.

What's next​

The next mechanism is drain and shutdown: telling a connection to unsubscribe, deliver the last in-flight messages to your handlers, flush pending publishes, and only then close, so a deploy or a SIGTERM never loses work.

Continue to Drain & Shutdown.

See also​