1. A Replicated Database

Consider a key-value database running on three nodes: A, B, and C. Each node has its own copy of the data. A client sends SET x = 10; the database must preserve that write even if one node fails and must prevent replicas from applying conflicting writes.

Raft coordinates this through a replicated log. One node acts as leader, orders writes, and sends log entries to followers. Once a write is committed, each node applies it to its own database. We will start with A already elected as leader, follow a write and a read, then examine what happens when A fails.

Raft assumes crash failures: nodes may stop, restart, or lose connectivity, but do not deliberately send false information. It is used in databases, metadata stores, configuration services such as etcd and Consul, and coordination systems.

What Each Node Contains

The storage engine is part of each database node. Its data is already the replica’s database state; there is no later transfer to a separate database server.

An illustrative disk layout is:

raft-log.dat       -> log entries
raft-metadata.dat  -> current term and vote
snapshot.dat       -> application state at a recorded log position
state/             -> storage-engine files containing the applied data

state/ could contain database pages, SSTables, manifests, and a storage-engine WAL. An embedded engine such as RocksDB manages these files and in-memory structures; caches need not survive a restart. Implementations may share storage between the Raft log and application state. Query planning, shard placement, and cluster rebalancing belong to other database components.

The state machine must be deterministic: the same starting state and ordered commands must produce the same logical result. Replicas can have different file layouts or cache contents while storing the same values.

System Overview

2. Write Path: Follow One Command

Starting State

A is the leader; B and C are followers. All three have applied entries through index 100 and store x = 5. Entry 100 was created in term 7, which is also the current term.

A term is an election epoch. A node increments its current term when it starts an election. An election may fail, so a term can end without a leader.

An index is a position in the log. Each entry records its index, creation term, and command. Terms identify which leader epoch produced an entry; indexes determine application order.

Example of a log entry:

index: 100
term: 7
command: SET x = 5

We will track four values on every node:

In this example, A commits entry 101 from its current term after confirming durable copies on A and B—a majority of the three nodes. Followers advance their commit index when the leader announces commitment.

lastLogIndex tracks stored commands, commitIndex tracks commands known to be committed, and lastApplied tracks commands already executed. For entry 101 containing SET x = 10, these positions change separately:

Commitment does not execute the command. The application loop processes committed entries in order:

while lastApplied < commitIndex:
    apply log[lastApplied + 1] to the database
    advance lastApplied

Application may lag while earlier commands are being processed or the storage engine is busy. Thus lastApplied <= commitIndex; a read that must include entry 101 waits until lastApplied reaches at least 101.

For the write walkthrough, all three nodes start at index 100:

The client sends SET x = 10 to A. If it contacts a follower instead, the service may redirect or proxy the request to the leader.

Step 1: A Appends the Entry

A assigns the next index and records:

index: 101
term: 7
command: SET x = 10

In this example A persists the entry before sending it to followers. Only A’s log changes; its database still contains x = 5.

Step 2: A Sends AppendEntries

A sends this request to B and C. Delivery to C is delayed.

term: 7
prevLogIndex: 100
prevLogTerm: 7
entries: [{ index: 101, term: 7, command: SET x = 10 }]
leaderCommit: 100

term and prevLogTerm happen to match. A leader in term 7 could append after an entry created in term 6; then prevLogTerm would be 6.

Sending the request alone changes no replica state:

Step 3: B Persists and Acknowledges

B has index 100 with term 7, so the preceding-entry check succeeds. It appends entry 101, persists the required log changes, and acknowledges replication through 101.

B does not apply the command: the request advertised leaderCommit = 100.

The acknowledgement means entry 101 is durably stored on B, not that B has changed x. If the acknowledgement is lost, A can retry and B can confirm the existing entry.

Step 4: A Establishes Commitment

A receives B’s acknowledgement. Entry 101 now has durable copies on two of the three nodes: A and B. This is a majority. Since the entry belongs to A’s current term, A advances commitIndex to 101.

Entry 101 is now committed, but no database value has changed in this schedule. B has not yet learned A’s new commit index. C is not required for this write to commit.

Step 5: A Applies and Responds

A applies entry 101 to its state machine. The storage engine changes x from 5 to 10, and lastApplied advances to 101. A returns success to the client.

The write has succeeded even though both followers still store x = 5. A success response does not mean that every replica has applied the write.

Step 6: B Learns the Commit Index

A sends B another AppendEntries. It can be an empty heartbeat:

term: 7
prevLogIndex: 101
prevLogTerm: 7
entries: []
leaderCommit: 101

B verifies its entry at 101 and advances commitIndex. Before its application loop runs:

B then applies entry 101:

B learns commitment from A; it does not collect acknowledgements from other nodes.

Step 7: C Receives the Missing Entry

C cannot accept the heartbeat above because it lacks entry 101. A can instead send C a request anchored at its matching entry 100:

term: 7
prevLogIndex: 100
prevLogTerm: 7
entries: [{ index: 101, term: 7, command: SET x = 10 }]
leaderCommit: 101

This request carries both the missing entry and the commit notification. After C persists the entry and updates its commit index, but before applying it:

After C applies the command:

All three databases now contain x = 10. This is one possible schedule: A may notify followers before replying to the client, and followers may apply at different times.

3. Read Path: Which Value Can Be Returned?

A read queries the local storage engine. It does not replay the Raft log to reconstruct x on every request.

Return to the state immediately after Step 5:

The client has received success for SET x = 10 and now sends GET x. There are no subsequent writes in this example.

Reading a Follower Without Coordination

B’s database contains 5. Its log contains the new command, but B has neither learned commitment nor applied it. An explicitly stale follower read may therefore return 5.

For a linearizable read, this is insufficient: a read started after the successful write must reflect that write. B must establish a safe read position and apply through it before returning a value.

Reading Through the Leader

A currently stores 10, but simply calling itself leader is insufficient. A could have lost contact with the others while they elected a replacement.

A ReadIndex-style path works as follows. Assume the leader has already committed an entry from its current term, as A did in Step 4:

A follower can also serve a linearizable read after obtaining a safe read index from the leader and applying through it. Leader leases are another approach, but require explicit clock and timing assumptions.

If A is isolated from B and C, it cannot complete the quorum check. It must wait or fail the read instead of returning an unchecked local value. The leadership and applied-position requirements are described in the Raft paper, Section 8.

4. Leader Election: Replacing A

A Fails Before C Catches Up

Branch from the write example after Step 5: A crashes, B has entry 101 but still believes only 100 is committed, and C has entries through 100.

Followers normally receive AppendEntries heartbeats. Without valid leader communication, their election timers eventually expire. Suppose B times out first.

B Requests Votes in Term 8

B becomes a candidate, increments its term to 8, persists its vote for itself, and sends:

RequestVote:
  term: 8
  candidateId: B
  lastLogIndex: 101
  lastLogTerm: 7

Immediately after B starts the election:

C receives the request, updates its term to 8, and checks whether it has already voted in that term and whether B’s log is sufficiently up to date. C has not voted in term 8. Both last entries have term 7, and B’s index 101 is ahead of C’s 100, so C persists a vote for B.

After B receives C’s vote:

B has two votes out of three. The election changes roles and terms; it does not itself apply entry 101. B and C still store x = 5.

currentTerm and votedFor are durable. Otherwise, a node could vote, restart, and vote for another candidate in the same term. A node that observes a higher term updates its term and becomes a follower. Thus A steps down when it returns and learns of term 8.

Why C Cannot Win This Election First

If C had timed out first, its request would advertise last index 100, term 7. B would reject it because B has index 101 from the same term. With A offline, C could obtain only its own vote.

The general freshness rule compares the last entry’s term first, then its index:

candidate.lastLogTerm > voter.lastLogTerm
or (candidate.lastLogTerm == voter.lastLogTerm
    and candidate.lastLogIndex >= voter.lastLogIndex)

For example, a candidate ending at index 98 in term 8 is considered more up to date than a voter ending at index 101 in term 7. A longer log alone does not win. The candidate’s election term is a separate value: participating in newer elections does not make old log contents fresher.

How B Applies the Inherited Entry

Entry 101 was committed by A, but B has not learned that fact. B cannot commit an older-term entry merely by counting its copies. It can append a no-op entry in its own term:

101, term 7: SET x = 10
102, term 8: no-op

B sends C the missing entries. Once B and C durably store entry 102, B can commit that current-term entry and the preceding prefix through 101. It applies 101 and then the no-op; C follows after learning the commit index.

The previously successful write survives A’s crash even though neither surviving node had applied it before the election.

Split Votes

In the same three-node group, B and C may time out together and each vote for itself in term 8. With A offline, neither gets a majority. They retry in a later term with randomized election timeouts. If B’s next timeout expires first, it may collect C’s vote before C starts another election.

For a fixed group of N voters, a majority is floor(N / 2) + 1. Any two majorities overlap; one durable vote per term prevents two candidates from both winning that term. Preserving committed history additionally depends on the freshness and commitment rules above.

5. Repairing a Follower’s Log

A Follower Is Missing Entries

Consider a separate catch-up case in term 7. A has committed through 101; C has only entries through 98. Assume their entry 98 matches.

A: ... 98, 99, 100, 101
C: ... 98

The leader tracks two values per follower:

The leader calculates quorum replication from this progress and its own log. It does not poll a separate quorum service. Conflict hints can reduce the number of retries.

A Follower Has Conflicting Entries

A mismatch can also mean that a follower has an uncommitted entry from an earlier leader. Each pair below is index:term; assume the current leader is operating in a later term than those shown:

Leader:             98:6  99:7  100:7  101:7
Follower:           98:6  99:6  100:6

A request expecting 100:7 fails even though the follower has index 100. After finding the match at 98:6, the follower replaces its conflicting suffix:

Follower repaired:  98:6  99:7  100:7  101:7

Only uncommitted conflicting entries can be replaced. This is why replicas apply commands after commitment: an applied command must not later disappear during log repair.

Bounding the Follower’s Commit Index

A follower must not commit beyond the matching prefix established by a successful AppendEntries. For contiguous entries, that prefix ends at prevLogIndex + len(entries):

if leaderCommit > commitIndex:
    commitIndex = min(leaderCommit, prevLogIndex + len(entries))

An empty heartbeat establishes a prefix only through prevLogIndex. A rejected request does not advance commitment. This prevents a newer commit notification from committing a missing entry or an unverified local suffix.

6. Crashes, Retries, and Snapshots

Where the Write Stops

Return to SET x = 10 at index 101 in term 7:

The client cannot distinguish these cases from a timeout. For SET x = 10, repeating the command has the same effect in this isolated example. For INCREMENT x, replaying a new request could increment twice. Services use request IDs or client sequence numbers, with replicated deduplication state, to recognize retries.

Recovering Local Application State

The Raft log supports replicated ordering and recovery. A storage-engine WAL, when present, supports local recovery of the engine’s updates. Whether the integration uses both logs and when it flushes them are implementation choices.

A restart must recover a consistent pair: application state and the log position already reflected in that state. If the durable data includes an increment from index 101 but recovery incorrectly replays 101, the increment could be applied twice. A service can coordinate state and applied-position persistence atomically, or restore a snapshot and replay the committed suffix.

A successful write does not require every modified page or SSTable to have reached its final on-disk layout. It requires sufficient durable recovery information. Background flushes and compaction may continue later.

When Log Replay Is No Longer Available

Snapshots bound log growth. Suppose the application has applied through index 10,000 and safely stores:

lastIncludedIndex: 10000
lastIncludedTerm: <term of entry 10000>
application state: <state after applying through 10000>
cluster membership: <configuration at the snapshot boundary>

The service can then discard the covered log prefix. If C returns with entries only through 2,000, the leader can no longer send entries 2,001 onward from that deleted history:

Snapshot creation, installation, and log truncation must preserve a consistent recovery boundary. Normal replication transfers commands; snapshot installation transfers a representation of application state.

7. Partitions and Safety

A Is Isolated from B and C

A cannot obtain a majority for new writes or complete quorum-confirmed reads. B and C can elect a new leader and continue. A may still believe it is leader until it learns the newer term, but that belief does not allow it to commit entries alone.

With fewer than two reachable voters in this three-node group, new writes stop. Raft preserves committed history rather than allowing both sides of a partition to commit conflicting operations.

Protocol Guarantees

For example, after one replica applies SET x = 10 at index 101, another cannot apply DELETE x at that index. It may temporarily be behind, but it cannot choose a different committed command.

8. Operational Limits

Fault-Model Boundary

Raft assumes a follower truthfully acknowledges durable storage. It does not handle a node that lies about its log or sends conflicting fabricated messages.

Byzantine fault-tolerant protocols address arbitrary or malicious behavior. A common BFT model requires 3f + 1 replicas to tolerate f Byzantine faults: four for one fault, seven for two. That stronger model generally requires additional communication and verification. Examples include Tendermint/CometBFT, HotStuff-style protocols, and SmartBFT-style systems, used where mutually distrustful participants or compromise are part of the threat model.

9. Revision Summary

• Stored: an entry exists in a node’s durable log.
• Committed: the protocol has established that the entry must be preserved; a follower learns this through leaderCommit.
• Applied: the node has executed the committed command against its local state machine.
• Read: query applied state, with leadership and application-progress checks when linearizability is required.
• Election: choose a new leader through a majority, constrained by log freshness.
• Recovery: repair an uncommitted suffix, replay retained entries, or install a snapshot.

The write, election, and recovery rules follow the Raft paper. The example schedules separate events for clarity; implementations may batch or overlap them while preserving the same ordering constraints.