Availability and Redundancy: Designing for Component Failure

Hardware fails, software crashes, networks partition, credentials expire, and people make mistakes.

Reliable systems are not built by pretending those events will never occur.

Redundancy provides alternate components or paths; availability is the user-visible result of a service being usable when needed.

Redundancy is one technique. It improves availability only when the copies are independent, functional, and able to take over.

A concrete example: two servers in one rack

An application runs on two servers, so it appears redundant.

Both servers share:

• One rack power unit
• One network switch
• One database
• One load balancer

A rack power failure takes both servers offline. A database failure stops both even when they remain powered.

Counting application instances misses the shared dependencies. Availability design must map the complete request path and its failure domains.

Availability is measured over time

Availability is often expressed as:

availability = successful service time / total measured time

For a 30-day month:

• 99% allows about 7 hours 12 minutes unavailable.
• 99.9% allows about 43 minutes.
• 99.99% allows about 4 minutes 19 seconds.

The calculation depends on what counts as unavailable. Is the service down only when every request fails, or also when latency exceeds an agreed limit?

A meaningful measure reflects user experience.

SLI, SLO, and SLA

A service-level indicator, or SLI, is a measurement such as successful request percentage or latency.

A service-level objective, or SLO, is the internal target for that indicator, such as 99.9% successful checkout requests over 28 days.

A service-level agreement, or SLA, is a contractual commitment and may define remedies when missed.

These terms should not be used interchangeably. Engineering generally operates against SLOs, while an SLA is a business and legal promise.

Single points of failure

A single point of failure is one component whose failure stops the required service.

Examples:

• One database server
• One network route
• One identity provider
• One DNS configuration
• One encryption key
• One administrator with unique knowledge

Removing a single point may involve replication, alternate paths, documented recovery, delegated access, or a reduced-function fallback.

Not every component requires duplication. The investment should follow the service's impact and recovery goals.

Active-active and active-passive

In active-active designs, multiple components serve traffic simultaneously.

Benefits:

• Capacity is already in use.
• Failure may require only traffic redistribution.
• Regular activity proves the copies are functional.

Challenges:

• Coordinating writes
• Preventing duplicate work
• Keeping configuration aligned

In active-passive designs, a standby takes over after failure.

This can simplify data ownership, but failover may be slower and the standby may contain unnoticed problems because it is rarely exercised.

Failure domains need independence

A failure domain is a set of components likely to fail together.

Examples include:

• One process
• One host
• One rack
• One power zone
• One data center
• One cloud region
• One provider account

Two instances in separate zones protect against more failures than two instances on one host.

Greater separation adds latency, data replication, cost, and operational complexity. Design for the failures the service has promised to tolerate.

Failover is a process

Failover requires:

1. Detecting a problem.
2. Deciding the primary is truly unavailable.
3. Promoting or selecting an alternative.
4. Redirecting traffic.
5. Preventing two primaries from making conflicting changes.
6. Recovering or replacing the failed component.

Detection that is too slow extends downtime. Detection that is too aggressive can cause needless failovers during brief network delays.

Distributed systems may use quorum or fencing to prevent split brain, where two sides both believe they are primary.

Redundancy can replicate mistakes

Replication quickly copies state, including:

• Accidental deletion
• Corrupt data
• Bad configuration
• Malicious changes

Redundancy improves continuity after component failure. Backups and version history recover earlier states.

Likewise, deploying the same software defect to every region creates consistent failure. Diversity in versions or providers can reduce some correlated risk but increases operational complexity and may introduce inconsistent behavior.

Understand which failure each technique addresses.

Graceful degradation

A service does not always need to be fully functional or completely down.

During a recommendation-system outage, a shop might:

• Show popular products instead of personalized results.
• Preserve cart and checkout.
• Delay analytics events.
• Disable nonessential previews.

Graceful degradation protects the most important user journey.

It requires explicit priorities and tested fallback behavior. A fallback path that is never used may fail when needed.

Error budgets guide change

An SLO below 100% implies a permitted amount of failure, called an error budget.

Teams can use the budget to balance:

• Feature release speed
• Reliability work
• Risky migrations
• Operational load

When the service consumes its budget too quickly, the team may pause risky changes and focus on stability.

The goal is not to celebrate errors. It is to make reliability tradeoffs visible rather than demand impossible perfection without cost.

Mean time measures

Common operational measures include:

• Mean time between failures: Typical operating interval between incidents
• Mean time to detect: Time before a problem is noticed
• Mean time to recover: Time to restore useful service

A system can improve availability by failing less often or recovering faster.

Automation, clear ownership, runbooks, and observability can reduce recovery time even when underlying components remain imperfect.

Averages should be paired with distributions because one severe incident can matter more than many brief ones.

Test failure behavior

Redundancy that has never failed over is an untested assumption.

Testing may include:

• Stopping an instance
• Removing a network path
• Failing a zone in a controlled environment
• Restoring from backup
• Expiring a credential
• Simulating a slow dependency

Chaos engineering is disciplined experimentation on resilience, not random production breakage.

Start with a hypothesis, limit blast radius, define abort conditions, observe results, and fix weaknesses.

Humans and procedures are dependencies

Availability depends on:

• Accurate alerts
• On-call coverage
• Accessible documentation
• Permission to act
• Communication channels
• Practiced incident roles

A technically recoverable system may remain down if no one has the credentials or knows the sequence.

Operational readiness belongs in the architecture.

Cost grows rapidly near perfection

Higher availability can require:

• More facilities
• More data copies
• Faster failover
• Continuous staffing
• Simpler changes
• Extensive testing

Moving from 99.9% to 99.99% can be much harder than moving from 99% to 99.9%.

Set targets from user and business impact. Not every internal report needs the availability of emergency communication.

Knowledge check

1. Why do two application servers not necessarily remove single points of failure?
2. How does an SLO differ from an SLA?
3. What tradeoff separates active-active and active-passive designs?
4. Why is replication not a replacement for backups?
5. What does graceful degradation protect?

The one idea to remember

Redundancy adds alternate parts; availability comes from the whole service continuing to meet user needs. Independence, tested failover, recovery procedures, and sensible targets turn extra components into real resilience.