Distributed Systems: When One Application Spans Computers

Inside one process, a function call either returns, throws an error, or the process stops.

Across a network, the caller can wait without knowing whether the request arrived, whether the remote computer finished, or whether the response was lost on its way back.

Distribution turns ordinary-looking calls into uncertain interactions between components that can fail independently.

That uncertainty is the central fact behind timeouts, retries, duplicate work, consistency models, queues, and many operational practices.

A concrete example: checkout during a partial failure

A checkout service asks a payment service to create a charge.

The order computer is healthy. The payment computer is healthy. A network device between them drops traffic for thirty seconds.

Checkout sees no response, but it cannot immediately know whether:

• payment never received the request,
• payment charged the card but its response was lost,
• payment is still processing,
• or payment failed internally.

The system is partly working and partly unreachable. That is a partial failure.

What makes a system distributed?

A distributed system contains components on multiple computers that coordinate through communication.

Examples include:

• a web server and database,
• microservices,
• replicated storage,
• mobile clients and cloud APIs,
• worker fleets,
• and globally distributed caches.

Even a modest application becomes distributed when it depends on a remote database or third-party API.

Network calls are not local calls

A local call normally has predictable overhead and shared process state.

A network call adds:

• serialization,
• routing,
• variable delay,
• connection limits,
• authentication,
• packet loss,
• remote overload,
• and independent deployment.

An API may look like a function in code, but its failure model is fundamentally different.

Latency

Communication takes time, and that time varies.

Distance, congestion, queueing, encryption, server load, and retries all contribute. A call that usually takes 20 milliseconds may occasionally take two seconds.

Designs must consider latency distributions and user deadlines rather than one average value.

Partial failure

One component can fail while others continue.

A database replica may be unavailable while the primary works. One region may lose connectivity. A payment provider may fail while product browsing remains healthy.

The system needs to isolate damage and communicate degraded behavior instead of assuming everything is either fully up or fully down.

Imperfect knowledge

Silence is ambiguous.

If a caller receives no response, it cannot distinguish a lost request from a lost reply without more protocol. Monitoring also sees the system from particular viewpoints; one health check may succeed while real users on another route fail.

Distributed systems make decisions with incomplete and delayed information.

Concurrency

Multiple computers can update related state at nearly the same time.

Two shoppers may buy the last item. Two workers may process the same message. A profile edit can race with an account deletion.

Correctness requires explicit rules for ordering, conflicts, uniqueness, and repeated work.

No perfectly shared clock

Computer clocks drift and synchronize imperfectly.

Timestamp order does not always equal causal order, especially across machines. A log entry stamped later may describe an event that happened first.

Distributed designs use sequence numbers, logical clocks, database ordering, or explicit versioning when order matters.

Replication

Replication stores copies of data or services on multiple machines.

It can improve:

• availability,
• read capacity,
• geographic latency,
• and disaster tolerance.

It also creates a coordination problem: how quickly must copies agree, and what should happen when they cannot communicate?

Availability and correctness

Different operations tolerate uncertainty differently.

A social reaction counter may accept updates during a temporary partition and reconcile later. A money transfer may reject an operation if the current balance cannot be established safely.

The business consequence determines which compromises are acceptable.

Timeouts

Every remote wait needs a limit.

Without a timeout, blocked calls consume threads, connections, memory, and user patience. A timeout ends waiting, but does not prove the remote operation did not happen.

The caller needs a plan for the resulting uncertain state.

Retries

Retrying can recover from brief network loss or overload.

Retries can also:

• duplicate side effects,
• amplify traffic,
• worsen an outage,
• and extend user latency.

Safe retry design uses limits, backoff, jitter, idempotency, and a clear classification of transient versus permanent errors.

Idempotency

An idempotent operation can be repeated without creating additional intended effects.

For payment creation, a client can send a unique idempotency key. If the response is lost and the request is retried, the server returns the original result instead of charging again.

Idempotency converts duplicate delivery from disaster into manageable protocol behavior.

Asynchronous work

Not every task must finish inside one request.

A video upload can return after durable acceptance, while workers create resolutions later. This improves responsiveness and absorbs bursts.

The product must then expose status, failure, retry, cancellation, and completion.

Queues and events

Queues buffer work between producers and consumers. Publish-subscribe systems distribute events to several interested consumers.

These patterns reduce direct timing dependency, but introduce:

• duplicate delivery,
• ordering limits,
• backlog,
• schema evolution,
• and poison messages.

Decoupling changes where complexity lives; it does not erase complexity.

Consistency

Replicas and derived views may not update simultaneously.

Some systems coordinate before responding to provide stronger consistency. Others accept temporary disagreement to improve availability or scale.

The product must define what users can observe, how long disagreement may last, and which operations require stronger guarantees.

Observability

One user request may cross many services.

Useful diagnosis needs:

• correlation identifiers,
• distributed traces,
• service-level metrics,
• structured logs,
• queue-depth monitoring,
• and deployment records.

Without connected telemetry, teams see scattered symptoms rather than one failing workflow.

Failure testing

Distributed behavior should be tested under:

• delay,
• timeout,
• dropped responses,
• duplicate messages,
• service restart,
• stale data,
• unavailable dependencies,
• and partial completion.

Happy-path tests on a perfect local network do not exercise the defining risks.

Designing for uncertainty

For every remote operation, ask:

1. What is the deadline?
2. Can it be retried?
3. Is it idempotent?
4. What happens if the response is lost?
5. Which state is authoritative?
6. Can work complete later?
7. How is failure observed and recovered?

These questions turn network uncertainty into explicit engineering decisions.

Knowledge check

1. Why is a network call fundamentally different from a local function call?
2. What makes silence from a remote service ambiguous?
3. Why can timestamps from different computers misrepresent event order?
4. How does replication create a consistency problem?
5. Which controls make retries safer?

The one idea to remember

A distributed system coordinates components that communicate with delay and fail independently. Treat every remote interaction as uncertain, define deadlines and repeated-delivery behavior, and make partial failure visible and recoverable.