Consensus: One Decision History from Unreliable Participants

Distributed nodes can crash, restart, receive messages late, and disagree about which peers are reachable.

Some systems still need one dependable ordered history for configuration, metadata, locks, or database writes.

Consensus lets a group agree on values or an ordered log while tolerating a defined number of participant failures.

It does not make networks reliable; it defines when a decision is safe despite their unreliability.

A concrete example: replicated configuration

Five nodes store critical cluster configuration.

A leader proposes:

set active_region = south

After a majority persists the entry, it becomes committed. Failed followers can catch up later, and a minority cannot commit a conflicting value alone.

The consensus problem

Participants must satisfy properties such as:

• agreement: correct nodes do not decide different values,
• validity: decisions come from valid proposals,
• termination: progress occurs under required conditions.

Failure and timing assumptions determine what is possible.

Asynchronous uncertainty

A silent node may be:

• crashed,
• partitioned,
• slow,
• or paused.

No fixed timeout can distinguish these perfectly in an asynchronous network. Protocols use timeouts to drive progress while quorums protect safety.

Safety and liveness

Safety means nothing bad happens, such as two conflicting committed logs.

Liveness means progress eventually happens.

During severe partition, a protocol may preserve safety by refusing progress on a minority side.

Replicated log

Many consensus systems agree on an ordered log:

1: create user A
2: set quota 10
3: change owner B

Each node applies committed entries to the same deterministic state machine.

Agreement on order produces agreement on state.

Leader-based consensus

Protocols such as Raft commonly elect a leader.

The leader:

• receives proposals,
• assigns log positions,
• replicates entries,
• and advances commit.

Leadership simplifies ordering but can change after failure.

Terms

Time is divided into logical terms.

Each term has at most one elected leader. Nodes reject stale-term messages and update when they learn a newer term.

Terms order leadership without relying on synchronized wall clocks.

Quorum

A majority quorum intersects every other majority.

This overlap carries knowledge of committed entries into future elections. A candidate with an outdated log should not become leader over a more current majority.

Quorum is the mathematical bridge across failures.

Commit

An entry is committed only after the protocol's replication rule is satisfied.

A leader writing locally is not enough. If it crashes before quorum, a future leader may overwrite the uncommitted proposal.

Clients should receive success only after the required commit point.

Apply

Committed entries are applied to the state machine in order.

Application should be deterministic: the same command and prior state must produce the same result on every replica.

External nondeterminism such as current time should be included in the committed command.

Client retries

A client can time out after a command commits.

Retrying without an operation identifier may append a duplicate command. Clients or state machines need idempotency and result lookup.

Consensus orders commands; it does not infer duplicate business intent.

Read consistency

A follower can serve stale reads cheaply.

Linearizable reads may require:

• leader confirmation,
• quorum check,
• lease under safe assumptions,
• or a committed read barrier.

Choose read guarantees according to operation.

Membership changes

Adding or removing nodes changes quorum.

Unsafe instantaneous configuration can create two non-overlapping majorities. Joint-consensus or staged mechanisms preserve overlap during transition.

Membership is itself consensus-controlled state.

Snapshots

Logs grow indefinitely.

Nodes periodically snapshot state and compact old entries. A lagging node can install a snapshot then continue from a later log position.

Snapshot creation and transfer must preserve a committed boundary.

Failure tolerance

A majority-based group of:

• 3 nodes tolerates 1 unavailable,
• 5 tolerates 2,
• 7 tolerates 3.

More nodes increase failure tolerance and coordination cost.

Place them across independent failure domains.

Consensus is not replication alone

Asynchronous replication copies data but may not define one safe committed order during failover.

Consensus combines replication with elections, quorum, and commit rules.

The distinction matters for split-brain protection.

Consensus is not a distributed transaction

Consensus agrees on the group's log.

A business workflow spanning a payment provider, inventory, and email still needs local transactions, idempotency, and compensation.

Consensus can support a coordinator but does not make all external side effects atomic.

Performance

Consensus writes require network round trips to quorum and durable storage.

Latency depends on:

• leader location,
• follower delay,
• batching,
• disk,
• and contention.

Keep consensus groups focused on state that truly needs one order.

Coordination services

Systems use consensus-backed stores for:

• leader election,
• configuration,
• service discovery,
• locks,
• and metadata.

Applications should not place high-volume ordinary data there without understanding limits.

Testing

Test:

• leader crash,
• follower lag,
• network partition,
• stale message,
• process pause,
• disk failure,
• membership change,
• snapshot recovery,
• and duplicate client request.

Verify safety invariants after recovery, not only availability.

Knowledge check

1. How do safety and liveness differ?
2. Why does a majority quorum preserve knowledge across elections?
3. When can an uncommitted log entry disappear?
4. Why must state-machine application be deterministic?
5. What business problem does consensus not solve by itself?

The one idea to remember

Consensus builds one committed decision history from nodes that can fail and communicate unpredictably. Terms, quorums, leader rules, durable logs, deterministic application, and controlled membership preserve safety while allowing progress when enough nodes can communicate.