Resilient Clients Deep Dive
Core NATS taught you to publish and subscribe on a single server running on your laptop. This chapter takes those same publishers and subscribers and asks the harder question: what happens when the server goes away, the network stalls, or the handler can't keep up?
The answer isn't new application code but the connection, the live link from your client to a server, configured to survive faults. A resilient client detects when its connection breaks, buffers work while it's broken, recovers without losing the application's place, and shuts down without losing in-flight work.
One idea holds the whole chapter together: the connection is a state machine. It moves through a small set of states (DISCONNECTED, CONNECTING, CONNECTED, RECONNECTING, DRAINING, CLOSED), and every fault is just a transition between them. A server dying moves a CONNECTED client to RECONNECTING. A clean shutdown moves it to DRAINING and then CLOSED. Once you can picture that machine, each mechanism in this chapter is one well-defined edge on it.
We harden one running system: the Acme ORDERS app you already know.
The order-svc publisher and the warehouse, notifications, and
analytics subscribers keep their code; what changes is the connection
each one opens. Every example uses the same order event:
{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}
By the end you'll have
- An
order-svcconnection that opens with a name, a server pool (the list of server URLs the client may connect to), and a sane connect timeout, instead of a bare default connection. - A connection that reconnects with backoff and jitter when a server
goes away, cycling the
n1/n2/n3pool and flushing buffered publishes the moment it rejoins. - A clean shutdown that drains in-flight work instead of dropping it, triggered on a SIGTERM.
- Subscribers that bound their memory with pending limits and surface a backlog through the async error callback instead of growing until the process is killed.
- A request-reply call that tells "the responder is absent" apart from "the responder is slow" and retries each one differently and safely.
- A connection that presents the
order-svccredentials over a CA-validated TLS link.
Who this is for
You've read the Core NATS deep dive, so publish,
subscribe, and request-reply are familiar, and ideally the
JetStream deep dive, so you know what a stream and a
consumer are. This chapter reuses the ORDERS stream and its consumers
rather than re-introducing them.
You don't need to have run NATS clients in production before. We start from "open a connection the right way" and grow one fault at a time.
Unlike the other Learn chapters, this one has no Core Concepts primer to read first; the connection lifecycle is taught here and nowhere else. So every term gets defined in the page that first uses it: DISCONNECTED, drain, slow consumer, no responders, and the rest.
How to read it
Each page introduces at most two new concepts and carries the same
order-svc connection forward. You open the connection,
make it reconnect,
drain it, and so on: one
resilience option per page, never a fresh example.
This chapter only ever talks about what the client does. When a page reaches the edge of the client (why a server went away, how a credential was issued, what happens to a JetStream consumer's acknowledgment position across a reconnect), it names the gap in one sentence and links out rather than re-explaining it here. Server-side faults live in Topologies; consumer acknowledgment lives in JetStream → Acknowledgment; issuing credentials lives in Security.
Where a feature has a long list of options, the page covers only the ones that change how a connection behaves under fault. You'll find the full set of connection options in Reference.
Map
| Page | What you learn |
|---|---|
| Resilient Clients Deep Dive | The connection as a state machine, and the seven faults this chapter survives |
| Connecting | Open order-svc with a name, a server pool, and a connect timeout, and read the connect handshake |
| Reconnection | Reconnect with backoff and jitter, cycle the server pool, and buffer publishes while disconnected |
| Drain & Shutdown | Drain in-flight work on shutdown instead of dropping it with a bare close |
| Slow Consumers | Bound a subscription's pending buffer and detect overflow before it OOMs |
| Request-Reply Resilience | Tell no-responders apart from a timeout, and retry each one safely |
| TLS & Auth | Consume a credentials file and trust a CA so the connection is authenticated and encrypted |
| Where Next | A production checklist and a map of what's beyond the client |
Prerequisites
You'll need:
- A working
nats-server. The early pages use a singlenats-server; once reconnection enters, the pages point the client at then1/n2/n3cluster from the Topologies deep dive, used only as a server pool the client connects to. - The
natsCLI installed and pointed at your server. The CLI carries the first tab of each example; the JavaScript, Go, Python, Java, Rust, and C# tabs carry the client options the CLI can't express, such as pending limits and the reconnect callbacks.
Open a terminal, start your server, and turn to Connecting.
See also
- Core NATS deep dive — the publishers, subscribers, and request-reply calls this chapter hardens.
- JetStream deep dive — the
ORDERSstream and consumers whose connections this chapter makes resilient. - Topologies deep dive — the
n1/n2/n3cluster this chapter treats only as a server pool.