Skip to main content

Raft and leaders

The previous page left you with a live three-server cluster east (n1-east, n2-east, and n3-east) that discovered itself from one seed route. The servers know about each other and forward messages. But knowing about each other isn't the same as agreeing with each other.

Plain orders.created traffic needs no agreement: a publish lands on one server, gets forwarded, and is gone. Stored data is different. When the ORDERS stream keeps three copies of every order, the three servers holding those copies must agree on which orders exist and in what order, even while one of them is restarting or unreachable.

This page is about that agreement. It introduces two ideas: a RAFT group is a set of servers that keep an identical log by consensus, and a leader election is how a group picks the one server that drives the log. Everything the next page does with replication rests on these two.

RAFT groups​

NATS reaches agreement with RAFT, a consensus algorithm. A RAFT group is a fixed set of servers (its peers) that maintain a single shared log that all of them agree on. One peer is the leader; the rest are followers. The leader is the only peer that accepts new entries; it copies each entry to the followers, and the group advances together.

A cluster runs many RAFT groups at once, layered.

The first is the meta group: one cluster-wide RAFT group whose peers are the servers themselves. Its log holds the cluster's assignments rather than your data: which streams exist, how many replicas each has, and which servers hold them. Its leader is the meta leader, the server that decides where new streams and consumers are placed. Every server in east is a peer of the meta group.

The rest are per-asset groups: every replicated stream gets its own RAFT group, and so does every replicated consumer. The ORDERS stream running with three copies is one RAFT group whose three peers are the three servers holding those copies. Its leader, the stream leader, is the server that accepts writes to ORDERS and replicates them.

So east holding one replicated ORDERS stream runs at least two RAFT groups at once: the meta group across all three servers, and the ORDERS stream group across its three peers. They're independent. The meta leader and the ORDERS stream leader may be the same server or different servers, and a change to one doesn't move the other.

You can see both. Ask any server what it knows about itself and the meta group:

nats server info n1-east

The output names the server, its cluster east, and whether it currently holds JetStream's meta leadership. To see a stream's group, ask the stream:

nats stream info ORDERS

The Cluster Information block names the stream's RAFT group, listing its Leader and each Replica peer:

Cluster Information:

           Name: east
         Leader: n1-east
        Replica: n2-east, current, seen 0.05s ago
        Replica: n3-east, current, seen 0.07s ago

Leader is the stream leader. Each Replica line shows a follower peer, whether it's current (caught up), and how recently the leader heard from it. This is the ORDERS RAFT group, viewed from the outside.

For a live view of a group's RAFT state (current term, who the leader is, each peer's status), check the /raftz monitoring endpoint. The full set of RAFT internals it exposes is documented in Reference → /raftz: log compaction, the $NRG.* subjects peers vote over, snapshot timing. We only need the group, the leader, and the followers here.

Leader election​

A group has one leader at a time. When the leader is healthy, the steady-state behavior is straightforward: the leader sends a periodic heartbeat to its followers (by default about once a second), and as long as that heartbeat arrives, the followers stay followers and do nothing but accept the leader's entries.

The interesting case is when the heartbeat stops, because the leader crashed, hung, or got cut off by the network. The followers can't tell why the heartbeat stopped, only that it did. So they don't wait forever. Each follower runs an election timer, and if no heartbeat arrives before it fires, that follower assumes the leader is gone and starts an election to replace it.

An election needs one more idea: the term, a number that counts elections and only ever goes up. Every entry and every vote is stamped with a term, so the group can always tell a stale message from a current one. A leader from an older term is automatically obsolete the moment a newer term exists.

Here's the sequence when a follower's election timer fires.

The follower becomes a candidate, the third RAFT role. It increments the term to one higher than any it's seen, votes for itself, and asks every other peer to vote for it in this new term.

Each peer grants its vote if it hasn't already voted in this term and the candidate's log is at least as up to date as its own. A peer votes for at most one candidate per term, which is what stops two leaders from emerging at once.

The candidate becomes the leader the instant it collects a quorum of votes: a majority of the group's peers, (N+1)/2. For the three-peer ORDERS group that's two: the candidate's own vote plus one more. With the majority in hand, the new leader immediately starts sending heartbeats, the other peers return to being followers, and the group is whole again under the new term.

The quorum rule is why a majority must survive for a group to elect a leader at all. A three-peer group keeps a leader as long as two peers are up; lose two and the survivor can't reach a majority, so it can't become leader and the group goes leaderless until a peer returns. This is the consensus math behind the odd-server-count advice the Topologies chapter gives as a deployment choice: an even count buys no extra majority.

Observing an election​

You can observe this directly. With east running and ORDERS replicated, find the current stream leader, kill it, and watch the survivors elect a new one.

First, find the leader:

nats stream info ORDERS | grep Leader

Say it reports Leader: n1-east. Stop that server: Ctrl-C its terminal, or kill its process. For a moment the ORDERS group has no leader: its heartbeat has stopped and the two survivors are running their election timers.

Within a few seconds, ask one of the survivors:

nats stream info ORDERS --server nats://127.0.0.1:4223

The Cluster Information block now names a different leader, n2-east or n3-east, and the term has advanced. The remaining two peers held a quorum (two of three), so they elected a new leader and ORDERS is writable again, even with n1-east down.

Moving a leader manually​

Sometimes you want to move leadership without killing anything: to drain a server before maintenance, or to rebalance after a restart. A stepdown is a leader voluntarily yielding its role so the group elects a new one.

For a stream leader, ask the stream's group to step down:

nats stream cluster step-down ORDERS

The current leader yields, the group runs an election, and a different peer takes over. The meta group has its own stepdown, scoped to the whole cluster:

nats server cluster step-down

That moves the meta leader, independently of any stream leader. Use the stream form to move a single stream, the server form to move cluster-wide assignment duty.

Pitfalls​

RAFT is robust, but its timing and its layering are common sources of confusion. Each pitfall below is scoped to this page's two concepts: groups and elections.

An election takes seconds, not milliseconds. The election timer fires between four and nine seconds after the last heartbeat, deliberately staggered so two followers don't become candidates at the exact same instant. So when you kill a leader, expect a short window where nats stream info shows no leader and writes are refused. That window is RAFT working as designed, not a bug. Don't build a client that treats a brief "no leader" as a fatal error. Have it retry, since a new leader arrives within seconds.

The correct handling is to retry the write rather than fail it. A nats stream info during the gap confirms what's happening:

# During the election window, the leader line is briefly empty:
nats stream info ORDERS | grep Leader
# Leader:
# Re-run a few seconds later and a new leader appears:
nats stream info ORDERS | grep Leader
# Leader: n2-east

Stepdown moves leadership, but does not pick the successor. nats stream cluster step-down makes the current leader yield, but the next leader is still chosen by a normal quorum election among the remaining peers. There's no flag that hands leadership to a specific server. Don't run stepdown expecting n3-east to win. Run it to move leadership off the current server, then read nats stream info to learn who actually won. (Choosing where a leader prefers to land at creation time is placement, covered on Placement, and even that is a hint, not a lock.)

The meta leader and a stream leader are different groups. Losing the meta leader doesn't lose the ORDERS stream leader, and vice versa; they're separate RAFT groups with separate elections. A common mistake is to see "the leader is down," panic, and assume the stream is unavailable when only the meta leader moved (or the reverse). Check the right group: nats server info for the meta leader, nats stream info ORDERS for the stream leader.

Where you are​

Your cluster now has names for its moving parts:

  • The meta group spans all three servers and holds the cluster's assignments; its meta leader decides placement.
  • The ORDERS stream is its own RAFT group of three peers with its own stream leader, independent of the meta leader.
  • You've killed a leader and watched the survivors run a leader election (a candidate, a bumped term, and a quorum of votes), and you've moved a leader on purpose with stepdown.

What you haven't done yet is follow a single write through the group: how the leader gets an order onto all three peers and decides it's safe.

What's next​

The next page traces exactly that. It follows one orders.created write from the leader's log to a quorum of peers, shows where the write commits, and explains the consistency you get from R=3: Replication and R=3.

See also​