Load Balancers: Coordinating Traffic Across Healthy Servers

Once an application runs on several servers, clients need a stable way to reach the service without choosing a server themselves.

A load balancer provides that entry point.

A load balancer receives requests or connections and forwards each one to an appropriate healthy backend.

It coordinates traffic, but the application still determines whether requests can safely run on interchangeable instances.

A concrete example: one server fails

Suppose three web servers sit behind a load balancer:

Client -> Load balancer -> Web A
                        -> Web B
                        -> Web C

The load balancer checks each server periodically. If Web B stops responding correctly, it removes B from rotation and sends new requests to A and C.

Users continue using one public service address. They do not need to learn that an individual backend changed.

Existing connections to the failed server may still break, so clients and applications need suitable retry behavior.

Layer 4 and Layer 7 balancing

A Layer 4 load balancer makes decisions using transport information such as IP addresses and TCP or UDP ports.

It can forward connections without understanding HTTP paths or headers.

A Layer 7 load balancer understands an application protocol such as HTTP. It can route based on:

• Host name
• URL path
• Header
• Cookie
• Request method

For example:

/api/*    -> API service
/images/* -> image service

Application-aware routing offers flexibility but requires more processing and protocol knowledge.

Distribution algorithms

Common algorithms include:

• Round robin: Rotate through backends.
• Least connections: Prefer the backend with fewer active connections.
• Weighted routing: Send more traffic to larger or preferred backends.
• Hash-based routing: Map a client or key consistently to a backend.
• Latency-aware routing: Prefer a location or backend with better measured response.

No algorithm perfectly knows future work. Two requests may have very different costs.

Choose based on connection length, workload variation, backend capacity, and whether requests are independent.

Health checks determine eligibility

A health check asks whether a backend should receive traffic.

A shallow TCP check proves only that a port accepts connections. An HTTP readiness check can verify that the process has initialized and can serve requests.

Checks should be:

• Fast
• Reliable
• Representative enough
• Protected from overload
• Clear about timeout and success criteria

A check that depends on every downstream service may remove all backends during one shared dependency failure. A check that only returns a hard-coded 200 may leave broken instances in rotation.

Health policy should match what rerouting can actually improve.

Connection draining supports safe removal

During deployment or maintenance, a backend should stop receiving new requests while finishing work already in progress.

This is called connection draining or graceful deregistration.

The process may:

1. Mark the instance unavailable for new traffic.
2. Wait for existing connections or requests.
3. Apply a maximum drain timeout.
4. Stop the instance.

Applications need graceful shutdown handling. Otherwise the platform removes traffic correctly but the process exits before active work completes.

Long-lived WebSocket connections require different drain behavior from short HTTP requests.

TLS termination

A load balancer often terminates TLS:

1. The client establishes an encrypted HTTPS connection to the load balancer.
2. The load balancer presents the certificate.
3. It decrypts the request and forwards it to a backend.

Central termination simplifies certificate management and can offload cryptographic work.

Traffic between the balancer and backends may also need encryption, especially across untrusted networks or where policy requires end-to-end protection.

The backend may use forwarded headers to learn the original protocol and client address. Those headers must be trusted only from known proxies.

Session persistence

If session state exists only in one server's memory, a returning user may need to reach the same backend.

Sticky sessions use a cookie or connection property to preserve that affinity.

Stickiness can be practical, but it creates:

• Uneven traffic
• Lost sessions when one backend fails
• Harder deployments
• Reduced flexibility

Storing necessary session state in a shared system or using signed self-contained tokens often makes instances more interchangeable.

Affinity should solve a deliberate need rather than hide accidental local state.

Load balancing does not create capacity

A load balancer distributes existing capacity. It cannot make overloaded backends faster.

If demand exceeds the combined capacity, requests still queue or fail.

Autoscaling can add instances, while the load balancer discovers and uses them. These are complementary mechanisms:

• Load balancing assigns traffic.
• Autoscaling changes capacity.

Both can be limited by a shared database, network, quota, or downstream service.

The load balancer must also be available

A single load-balancer process on one machine becomes a new single point of failure.

Production load-balancing services use redundancy, failover, distributed addresses, or provider-managed fleets.

Clients may reach different balancers through:

• DNS
• Anycast routing
• Floating virtual addresses
• Active-passive failover
• Active-active designs

The public endpoint must remain stable while internal components are replaced.

Global load balancing

Global systems can direct users among regions.

Decisions may consider:

• Geographic proximity
• Measured latency
• Region health
• Capacity
• Legal data-location requirements
• User tenancy

DNS-based routing is common but affected by resolver caching. Network anycast can announce the same address from multiple locations.

Cross-region routing needs a data strategy. Sending a request to another region does not automatically make the required database state available there.

Load shedding and limits

When capacity is exhausted, a load balancer or gateway may reject excess requests quickly.

This can protect:

• Backend connection pools
• Memory
• Worker threads
• Downstream dependencies

Controls include:

• Rate limits
• Connection limits
• Request-size limits
• Priority classes
• Queue timeouts

A clear 429 Too Many Requests or 503 Service Unavailable can be healthier than accepting work that times out after consuming resources.

Observing load-balancer behavior

Monitor:

• Request and connection counts
• Backend latency
• Error rates
• Healthy backend count
• Rejected traffic
• TLS failures
• Queue time
• Bytes transferred
• Distribution among backends

Separate errors generated by the load balancer from errors returned by the application.

Logs should preserve request identifiers so a request can be traced from the edge through the selected backend.

Knowledge check

1. What stable role does a load balancer provide to clients?
2. How does Layer 7 routing differ from Layer 4 routing?
3. Why can an overly deep health check remove every backend?
4. What problem does connection draining solve?
5. How does load balancing differ from autoscaling?

The one idea to remember

A load balancer is a traffic coordinator. It presents a stable entry point, selects healthy backends, and supports replacement, but it cannot create capacity or fix application state that prevents servers from being interchangeable.