Raft from scratch, part 1: leader election & beating split-brain

For two weeks I’ve had a single server. This week I gave it siblings — and hit the oldest problem in
distributed systems: how do a bunch of equal machines, with no boss, agree on a boss? And do it
while messages drop and nodes vanish. This is consensus, and I built the election half of Raft from
scratch to feel exactly why it’s hard.

What I built

A cluster of peers that, from a cold start, elects exactly one leader — and re-elects a new one when
the leader dies, all over the same unreliable network from last week. No coordinator, no shared clock.

term 4:  A(follower)   B(follower)   C(follower)
         └─ B times out first ─┐
term 5:      B(candidate) ──RequestVote──► A, C
             A,C vote yes  ──►  B has a majority  ──►  B(leader)
             B ──heartbeat every 100ms──► A, C   (resets their election timers)

The concepts that made it work

• Terms are a logical clock, and a higher term always wins. Time is sliced into numbered terms,
  each with at most one leader. Any peer that sees a term greater than its own instantly steps down and
  adopts it. That single reflex is the whole reason two leaders can’t coexist.
• Majorities can’t overlap into two leaders. Winning needs a majority, and any two majorities of
  the same cluster share at least one member — who only votes once per term. So ≤1 leader per term,
  guaranteed by counting, not coordination.
• Randomized timeouts are what make it terminate. If every follower timed out at once they’d
  split the vote forever. A little jitter (I used 300–600ms) means someone almost always goes first and
  wins cleanly. Safety comes from majorities; liveness comes from randomness.
• Heartbeats are just elections suppressed. The leader sends empty messages on a timer; each one
  resets a follower’s clock. Silence is the only signal a follower needs to challenge for the throne.

The thing that bit me

Two traps, both about concurrency, not consensus. First: it is astonishingly easy to hold a mutex
while making an RPC — and the instant two peers request each other’s vote, they deadlock, each waiting
for a lock the other holds. The fix is a discipline you must never break: lock, read/modify state,
unlock, then call. Second, subtler: when a vote finally comes back, you have to re-ask “am I even
still a candidate in this term?” — because while that message was in flight you may have already
stepped down. Count that stale vote and you conjure a phantom leader. Neither bug fails cleanly; they
show up as flakiness, which is why I only trusted it after it survived repeated runs under the race
detector.

Tradeoffs & the road not taken

What I built is deliberately the election only:

• No replicated log yet. Right now any peer can win, because there’s no data that could be
  inconsistent. Real Raft adds a rule — don’t vote for a candidate whose log is behind yours — that
  guarantees the new leader has every committed entry. That’s next week’s problem.
• No persistence. Term and vote live in memory; a crash-restart could vote twice in one term. Also
  next week.

Where this sits in the system

Last week: one server, made honest with retries and dedup. This week: many servers that agree on who’s
in charge, and survive that server dying. Next: replicate the log — the leader accepts commands and
copies them to a majority before calling them committed. That’s when this stops being an election and
becomes a fault-tolerant database.

Part of a series building a distributed database from scratch.