Leader Election: Time-Bounded Authority in a Distributed Group

Some distributed tasks need one coordinator:

assign jobs,
order writes,
run a singleton schedule,
or manage membership.

Hard-coding one permanent machine creates a failure point.

Leader election establishes one node's temporary authority through agreement, not through a permanent machine identity.

The hardest part is preventing an old leader from continuing after the group selects a new one.

A concrete example: a scheduler cluster

Three scheduler nodes run.

One is leader and assigns jobs. It stops communicating. The other two form a majority, elect a new leader in a higher term, and continue.

If the old node returns, it sees the higher term and remains follower rather than assigning duplicate jobs.

Why one leader?

Leadership simplifies ordered decisions:

one writer,
one sequence,
one assignment owner.

Without coordination, two nodes can schedule the same job or allocate the same unique resource.

Not every workload needs a leader; partitioned independent work can scale better.

Followers

Non-leader nodes:

replicate state,
monitor the leader,
vote,
and remain ready to take over.

They may serve stale reads or forward requests depending on the system.

Follower readiness determines recovery speed.

Terms and epochs

An election increments a logical term or epoch.

Messages carry the term. Nodes reject messages from older terms, distinguishing current authority from delayed network traffic.

Terms provide logical order independent of wall clocks.

Heartbeats

The leader sends periodic heartbeats.

Followers start an election when heartbeats stop for long enough. The timeout must balance:

fast recovery,
network jitter,
temporary pauses,
and false elections.

Randomized election timeouts reduce simultaneous candidates.

Elections

A candidate:

advances its term,
votes for itself,
requests votes,
becomes leader after quorum,
announces authority.

Protocols add log-freshness rules so a stale node cannot win and lose committed state.

Quorum

A majority quorum ensures two valid leaders in the same term cannot each have disjoint support.

In a five-node group, three votes elect a leader. Two remaining isolated nodes cannot elect another majority.

Quorum sacrifices availability on minority partitions.

Split brain

Split brain occurs when multiple nodes believe they are leader and accept conflicting work.

Causes include:

partition,
stale lease,
weak external lock,
delayed heartbeat,
or misconfigured quorum.

Use consensus and fencing rather than trust self-declared leadership.

Leases

A lease grants authority for a bounded time.

The leader renews it before expiry. Leases depend on bounded clock and delay assumptions, so implementation must account for clock drift and process pauses.

Lease expiry alone may not stop an old leader already writing to an external resource.

Fencing tokens

Each leadership term receives an increasing fencing token.

The protected resource accepts operations only with a token at least as new as the latest seen. A paused old leader resumes with an older token and is rejected.

Fencing enforces authority at the side-effect boundary.

External locks

A distributed lock can elect a leader when one node holds it.

The lock service must provide:

strong ownership,
expiry,
atomic acquisition,
and fencing.

A database row or cache key without correct failure semantics can create false exclusivity.

Leader failure

When a leader fails:

writes may pause,
an election runs,
a new leader catches up,
clients reconnect or redirect.

Recovery time includes detection and state synchronization, not only voting.

Leader step-down

A leader should step down when:

it loses quorum,
term becomes stale,
lease cannot renew,
or local health is unsafe.

Continuing alone favors availability at the cost of conflicting authority.

Planned handoff

During maintenance, leadership can transfer to a healthy up-to-date follower before shutdown.

This reduces interruption and avoids a full timeout-based election.

Handoff still needs term and quorum rules.

Election safety versus speed

Short failure timeouts elect a replacement quickly but mistake temporary pauses for failure more often.

Long timeouts reduce unnecessary elections but extend write unavailability after a real crash. Measure network and process-pause distributions, then use quorum and fencing so an aggressive recovery choice cannot create conflicting authority.

Client behavior

Clients may:

discover the leader,
receive redirects,
send through a proxy,
or try known nodes.

They need retry limits and idempotency because leadership can change after an operation commits but before response arrives.

Leader bottleneck

One leader can limit write throughput.

Scale by:

batching,
partitioning into several independent leader groups,
reducing coordination,
or choosing leaderless operations where semantics allow.

One global leader for unrelated work creates unnecessary serialization.

Singleton jobs

Running one cron job across replicated application instances needs more than "if primary."

Use:

durable scheduler,
leader election with fencing,
idempotent job,
and execution record.

A leader can fail after performing work but before recording completion.

Observability

Track:

current leader,
term,
election count,
heartbeat delay,
quorum health,
step-down reason,
and leadership duration.

Frequent elections can signal overload or network instability.

Testing

Test:

leader crash,
process pause,
network partition,
clock drift where leases apply,
stale leader resume,
simultaneous candidates,
and resource fencing.

Killing one process alone does not cover split brain.

Knowledge check

Why is leadership temporary rather than tied to one machine?
How do terms reject stale messages?
Why does majority quorum prevent two elected leaders in one term?
What does a fencing token protect?
Why can a leader become a throughput bottleneck?

The one idea to remember

Leader election gives one node temporary coordination authority through terms and quorum. Safe systems detect loss, step down without quorum, fence stale leaders at the resource boundary, and make clients tolerate authority changes and uncertain responses.