1. A Replicated Database

Consider a key-value database running on three nodes: A, B, and C. Each node has its own copy of the data. A client sends SET x = 10; the database must preserve that write even if one node fails and must prevent replicas from applying conflicting writes.

Raft coordinates this through a replicated log. One node acts as leader, orders writes, and sends log entries to followers. Once a write is committed, each node applies it to its own database. We will start with A already elected as leader, follow a write and a read, then examine what happens when A fails.

Raft assumes crash failures: nodes may stop, restart, or lose connectivity, but do not deliberately send false information. It is used in databases, metadata stores, configuration services such as etcd and Consul, and coordination systems.

What Each Node Contains

The storage engine is part of each database node. Its data is already the replica’s database state; there is no later transfer to a separate database server.

An illustrative disk layout is:

raft-log.dat       -> log entries
raft-metadata.dat  -> current term and vote
snapshot.dat       -> application state at a recorded log position
state/             -> storage-engine files containing the applied data

state/ could contain database pages, SSTables, manifests, and a storage-engine WAL. An embedded engine such as RocksDB manages these files and in-memory structures; caches need not survive a restart. Implementations may share storage between the Raft log and application state. Query planning, shard placement, and cluster rebalancing belong to other database components.

The state machine must be deterministic: the same starting state and ordered commands must produce the same logical result. Replicas can have different file layouts or cache contents while storing the same values.

System Overview

2. Write Path: Follow One Command

To understand how Raft processes a write, a node is separated into two independent components:

1. The Consensus Engine (Raft Log): Responsible for replicating ordered log entries across nodes and establishing durable cluster consensus.
2. The Application Engine (State Machine): The database or key-value store that executes committed log entries to produce final user-visible data (e.g., setting x = 10).

[ Client Request ] ──► [ Raft Log Engine ] ──(Consensus)──► [ State Machine ] ──► [ DB Disk ]
│                                   │
lastLogIndex                        commitIndex / lastApplied

The Three Progress Indices

Every node tracks three monotonic sequence numbers to manage this pipeline:

• lastLogIndex (Log Horizon): The index of the highest entry appended to the node’s local Raft log. It answers: “What is written to the local log file on disk?”
  • Note: Uncommitted entries at lastLogIndex may be overwritten or discarded if a leader changes.
• commitIndex (Safety Horizon): The highest log entry that Raft guarantees is permanent because a majority of nodes have persisted it. It answers: “What has the cluster safely agreed upon?”
  • Note: Driven by the leader’s consensus quorum. commitIndex strictly moves forward.
• lastApplied (Execution Horizon): The highest log entry whose operation has been executed against the local state machine/database. It answers: “How far has my local database state caught up?”

Because database application runs asynchronously in the background, a entry progresses through three explicit stages:

The application loop runs continuously in the background, ensuring lastApplied eventually reaches commitIndex:

while lastApplied < commitIndex:
    entry = log[lastApplied + 1]
    apply_to_database(entry.command)
    lastApplied += 1

Starting State

All three nodes (A, B, C) start fully synchronized at index 100 in Term 7:

Client sends SET x = 10 to Leader A.

Step 1: Leader Appends to Local Log

Leader A writes the command to its local Raft log at index 101.

lastLogIndex advances on A, but commitIndex remains 100 because no follower has received it yet.

Step 2: Leader Replicates to Followers

A sends AppendEntries RPC to followers B and C containing entry 101. Message to C is delayed in the network.

Follower B appends entry 101 to its local log and returns a success response to A. B does not execute the write yet because leaderCommit in the RPC is still 100.

Step 3: Leader Reaches Quorum and Commits

Leader A receives B’s ACK. Entry 101 is now stored on a majority of nodes (A and B).

A advances its commitIndex to 101. The write is now durable and committed cluster-wide.

Step 4: Leader Applies Command and Responds

Leader A’s background application loop sees lastApplied (100) < commitIndex (101). It executes SET x = 10 against its database, advances lastApplied to 101, and responds to the client.

Step 5: Follower Catches Up

On the next heartbeat or subsequent RPC, Leader A sends leaderCommit = 101 to B.

B advances commitIndex to 101, triggers its application loop, advances lastApplied to 101, and updates its database state to x = 10.

3. Read Path: Which Value Can Be Returned?

A read queries the local storage engine. It does not replay the Raft log to reconstruct x on every request.

Return to the state immediately after Step 5:

The client has received success for SET x = 10 and now sends GET x. There are no subsequent writes in this example.

Reading a Follower Without Coordination

B’s database contains 5. Its log contains the new command, but B has neither learned commitment nor applied it. An explicitly stale follower read may therefore return 5.

For a linearizable read, this is insufficient: a read started after the successful write must reflect that write. B must establish a safe read position and apply through it before returning a value.

Reading Through the Leader

A currently stores 10, but simply calling itself leader is insufficient. A could have lost contact with the others while they elected a replacement.

A ReadIndex-style path works as follows. Assume the leader has already committed an entry from its current term, as A did in Step 4:

A follower can also serve a linearizable read after obtaining a safe read index from the leader and applying through it. Leader leases are another approach, but require explicit clock and timing assumptions.

If A is isolated from B and C, it cannot complete the quorum check. It must wait or fail the read instead of returning an unchecked local value. The leadership and applied-position requirements are described in the Raft paper, Section 8.

4. Leader Election: Replacing A

A Fails Before C Catches Up

Branch from the write example after Step 5: A crashes, B has entry 101 but still believes only 100 is committed, and C has entries through 100.

Followers normally receive AppendEntries heartbeats. Without valid leader communication, their election timers eventually expire. Suppose B times out first.

B Requests Votes in Term 8

B becomes a candidate, increments its term to 8, persists its vote for itself, and sends:

RequestVote:
  term: 8
  candidateId: B
  lastLogIndex: 101
  lastLogTerm: 7

Immediately after B starts the election:

C receives the request, updates its term to 8, and checks whether it has already voted in that term and whether B’s log is sufficiently up to date. C has not voted in term 8. Both last entries have term 7, and B’s index 101 is ahead of C’s 100, so C persists a vote for B.

After B receives C’s vote:

B has two votes out of three. The election changes roles and terms; it does not itself apply entry 101. B and C still store x = 5.

currentTerm and votedFor are durable. Otherwise, a node could vote, restart, and vote for another candidate in the same term. A node that observes a higher term updates its term and becomes a follower. Thus A steps down when it returns and learns of term 8.

Why C Cannot Win This Election First

If C had timed out first, its request would advertise last index 100, term 7. B would reject it because B has index 101 from the same term. With A offline, C could obtain only its own vote.

The general freshness rule compares the last entry’s term first, then its index:

candidate.lastLogTerm > voter.lastLogTerm
or (candidate.lastLogTerm == voter.lastLogTerm
    and candidate.lastLogIndex >= voter.lastLogIndex)

For example, a candidate ending at index 98 in term 8 is considered more up to date than a voter ending at index 101 in term 7. A longer log alone does not win. The candidate’s election term is a separate value: participating in newer elections does not make old log contents fresher.

How B Applies the Inherited Entry

Entry 101 was committed by A, but B has not learned that fact. B cannot commit an older-term entry merely by counting its copies. It can append a no-op entry in its own term:

101, term 7: SET x = 10
102, term 8: no-op

B sends C the missing entries. Once B and C durably store entry 102, B can commit that current-term entry and the preceding prefix through 101. It applies 101 and then the no-op; C follows after learning the commit index.

The previously successful write survives A’s crash even though neither surviving node had applied it before the election.

Split Votes

In the same three-node group, B and C may time out together and each vote for itself in term 8. With A offline, neither gets a majority. They retry in a later term with randomized election timeouts. If B’s next timeout expires first, it may collect C’s vote before C starts another election.

For a fixed group of N voters, a majority is floor(N / 2) + 1. Any two majorities overlap; one durable vote per term prevents two candidates from both winning that term. Preserving committed history additionally depends on the freshness and commitment rules above.

5. Repairing a Follower’s Log

A Follower Is Missing Entries

Consider a separate catch-up case in term 7. A has committed through 101; C has only entries through 98. Assume their entry 98 matches.

A: ... 98, 99, 100, 101
C: ... 98

The leader tracks two values per follower:

The leader calculates quorum replication from this progress and its own log. It does not poll a separate quorum service. Conflict hints can reduce the number of retries.

A Follower Has Conflicting Entries

A mismatch can also mean that a follower has an uncommitted entry from an earlier leader. Each pair below is index:term; assume the current leader is operating in a later term than those shown:

Leader:             98:6  99:7  100:7  101:7
Follower:           98:6  99:6  100:6

A request expecting 100:7 fails even though the follower has index 100. After finding the match at 98:6, the follower replaces its conflicting suffix:

Follower repaired:  98:6  99:7  100:7  101:7

Only uncommitted conflicting entries can be replaced. This is why replicas apply commands after commitment: an applied command must not later disappear during log repair.

Bounding the Follower’s Commit Index

A follower must not commit beyond the matching prefix established by a successful AppendEntries. For contiguous entries, that prefix ends at prevLogIndex + len(entries):

if leaderCommit > commitIndex:
    commitIndex = min(leaderCommit, prevLogIndex + len(entries))

An empty heartbeat establishes a prefix only through prevLogIndex. A rejected request does not advance commitment. This prevents a newer commit notification from committing a missing entry or an unverified local suffix.

6. Crashes, Retries, and Snapshots

Where the Write Stops

Return to SET x = 10 at index 101 in term 7:

The client cannot distinguish these cases from a timeout. For SET x = 10, repeating the command has the same effect in this isolated example. For INCREMENT x, replaying a new request could increment twice. Services use request IDs or client sequence numbers, with replicated deduplication state, to recognize retries.

Recovering Local Application State

The Raft log supports replicated ordering and recovery. A storage-engine WAL, when present, supports local recovery of the engine’s updates. Whether the integration uses both logs and when it flushes them are implementation choices.

A restart must recover a consistent pair: application state and the log position already reflected in that state. If the durable data includes an increment from index 101 but recovery incorrectly replays 101, the increment could be applied twice. A service can coordinate state and applied-position persistence atomically, or restore a snapshot and replay the committed suffix.

A successful write does not require every modified page or SSTable to have reached its final on-disk layout. It requires sufficient durable recovery information. Background flushes and compaction may continue later.

When Log Replay Is No Longer Available

Snapshots bound log growth. Suppose the application has applied through index 10,000 and safely stores:

lastIncludedIndex: 10000
lastIncludedTerm: <term of entry 10000>
application state: <state after applying through 10000>
cluster membership: <configuration at the snapshot boundary>

The service can then discard the covered log prefix. If C returns with entries only through 2,000, the leader can no longer send entries 2,001 onward from that deleted history:

Snapshot creation, installation, and log truncation must preserve a consistent recovery boundary. Normal replication transfers commands; snapshot installation transfers a representation of application state.

7. Partitions and Safety

A Is Isolated from B and C

A cannot obtain a majority for new writes or complete quorum-confirmed reads. B and C can elect a new leader and continue. A may still believe it is leader until it learns the newer term, but that belief does not allow it to commit entries alone.

With fewer than two reachable voters in this three-node group, new writes stop. Raft preserves committed history rather than allowing both sides of a partition to commit conflicting operations.

Protocol Guarantees

For example, after one replica applies SET x = 10 at index 101, another cannot apply DELETE x at that index. It may temporarily be behind, but it cannot choose a different committed command.

8. Operational Limits

Fault-Model Boundary

Raft assumes a follower truthfully acknowledges durable storage. It does not handle a node that lies about its log or sends conflicting fabricated messages.

Byzantine fault-tolerant protocols address arbitrary or malicious behavior. A common BFT model requires 3f + 1 replicas to tolerate f Byzantine faults: four for one fault, seven for two. That stronger model generally requires additional communication and verification. Examples include Tendermint/CometBFT, HotStuff-style protocols, and SmartBFT-style systems, used where mutually distrustful participants or compromise are part of the threat model.

9. Revision Summary

• Stored: an entry exists in a node’s durable log.
• Committed: the protocol has established that the entry must be preserved; a follower learns this through leaderCommit.
• Applied: the node has executed the committed command against its local state machine.
• Read: query applied state, with leadership and application-progress checks when linearizability is required.
• Election: choose a new leader through a majority, constrained by log freshness.
• Recovery: repair an uncommitted suffix, replay retained entries, or install a snapshot.

The write, election, and recovery rules follow the Raft paper. The example schedules separate events for clarity; implementations may batch or overlap them while preserving the same ordering constraints.