Skip to main content

Where to go next

You started this chapter with nothing running, and by the end you have three servers (n1-east, n2-east, n3-east) that found each other from a single seed route, elected leaders for every RAFT group, and hold the ORDERS stream at R=3. A write from order-svc now lands on the leader, commits once a quorum has it, and survives one server dying. That covers the full arc of the chapter.

This page collects the mechanism you built into one place and points you at the chapters and Reference that take it further.

The five core ideas​

Every page in this chapter turned on the same five ideas.

Routes are the server-to-server connections that form the cluster. You configure one explicit seed route, and gossip does the rest: each server shares the peers it knows in its INFO, so one seed grows into a full mesh without you listing every server.

RAFT groups are how the cluster agrees. There's one meta group across the whole cluster plus one group per stream, and each group runs an election to pick a single leader. The leader is the only member that accepts writes; the followers replicate from it.

A quorum is a majority of a group's peers: for R=3, two of three. The leader appends a write to its log and commits it the moment a quorum has the entry. Quorum is what lets the group make progress while a minority is down, and lose nothing when it returns.

Placement decides where the replicas live. You constrain a stream to a cluster and to servers carrying matching tags, and the meta leader assigns the peers from the servers that qualify.

Peer management grows or shrinks the set. You add a peer and wait for its catchup to bring lag to zero before it counts toward quorum; you remove a peer one at a time so the group never drops below a majority.

Those five ideas are routes, RAFT, quorum, placement, and peers. Everything else in this chapter (terms, elections, append entries, apply, preferred leader, migration) is a refinement of those five.

Where the details live now​

The chapter is unversioned and concept-first. The exact election timers, the WAL format, the full cluster {} field list, and every StreamConfig option live in Reference, which is versioned and exhaustive. When you need the precise type of a config field or the wire format of a route, that's where to look.

The Reference root is the entry point. The handoff phrases throughout this chapter ("the full set of options is documented in Reference") all point into it. The pages you'll reach for most:

Sibling deep dives​

This chapter is the mechanism beneath two others, so it depends on them and they depend on it.

The Topologies deep dive covers the shapes: when to run a single server, when to grow into a cluster, when to reach for a super-cluster or leaf nodes. This chapter ran beneath its east cluster; Topologies decides what shape to build, and this chapter explains how the shape agrees and replicates once it's built.

The JetStream deep dive created the ORDERS stream this chapter replicated. Its page on surviving node loss is the one-page operator intro to R=3; this chapter went deeper, into the election and the quorum commit that make "lose a node, keep serving" actually work. For copies across clusters, JetStream's mirrors and sources page covers the DR story this chapter leaves to it.

The Deployment deep dive covers running this on real infrastructure: Kubernetes, rolling upgrades, and sizing the servers you formed a cluster from here.

The Backup & Recovery deep dive covers the operational protection: snapshotting a stream before a risky peer change, and restoring it if a migration goes wrong.

Where you are​

This is the end of the chapter. The arc is complete, and this page adds no new scenario state. The east cluster, its elected leaders, and the ORDERS stream at R=3 are still running in your session exactly as you left them on the previous page. You can keep experimenting (kill a server and watch a re-election, add a fifth peer, move placement) or tear it all down with nats stream rm ORDERS and stop the three servers when you're done.

You hold the core model: routes form the mesh, RAFT groups agree, a quorum commits each write, placement decides where the replicas live, and peer management grows the set without losing agreement. That model is the minimum you need for operating any NATS cluster in production.

Production checklist​

Every page in this chapter closed with a Pitfalls section. This collects the action items from all of them in one place: a last pass before you trust a cluster with real orders. Each group links back to the page that explains the why.

Forming a cluster — see Pitfalls​

  • Give every server the same cluster.name; a mismatch silently forms two clusters that look like one until a message fails to cross.
  • Point routes at the route port (6222), not the client port (4222); aiming at the client listener never establishes the route.
  • List two or three seed routes so the cluster still forms if one seed server is down at boot; gossip needs only one to reach, but only if that one is up.

Raft and leaders — see Pitfalls​

  • Treat a brief "no leader" window during failover as normal; the election timer is 4–9s, so let the client retry instead of failing the write.
  • Use nats stream cluster step-down to move leadership off a server, not to pick its successor; the next election is still quorum-based, so read nats stream info to learn who won.
  • Track the meta leader and a stream leader as different RAFT groups; check nats server info for one and nats stream info ORDERS for the other, because losing one is not losing the other.

Replication and R=3 — see Pitfalls​

  • Run real orders at R≥3, never R=1; a single copy is gone with its server's disk, with no failover and no recovery.
  • Read from the leader when you need read-after-write; a follower can lag, so a Direct Get from one returns data that's correct but not the newest.
  • Read a PubAck as quorum held, not full replication; before deliberately taking a server down, verify each replica shows current in nats stream info.

Placement — see Pitfalls​

  • Read a server's tags back before placing against them; tags are an intersection and a missing one fails with no suitable peers for placement rather than falling back to any server.
  • Spell tags exactly; matching folds case (ssd equals SSD) but sdd matches nothing.
  • Treat preferred as a hint for the initial leader, not a lock; if that server dies the next election is quorum-based and random, so move leadership explicitly with step-down --preferred when you need it somewhere specific now.

Scaling and peer management — see Pitfalls​

  • Remove peers one at a time; pulling two from an R=3 group at once loses quorum and the stream goes leaderless. Remove one, wait for a named leader and zero lag, then the next.
  • Wait for a freshly added peer to show current with zero lag before trusting it; while it catches up it's an observer, so don't kill another server mid-catchup.
  • Know the replica count from nats stream info before removing anything; peer-remove doesn't warn you that dropping the last peer destroys the replica.

See also​