Scalability: Adding Capacity Without Losing Correctness

A service that works for one hundred users may fail for one million. Requests queue, databases contend, networks saturate, and downstream services reject traffic.

Adding hardware helps only when the system can use it.

Scalability is the ability to increase capacity while preserving acceptable correctness, performance, and cost.

It is not a promise of infinite growth. Every architecture has limits and bottlenecks.

A concrete example: concert tickets

A ticketing site normally serves modest traffic. When sales open for a popular concert, hundreds of thousands of people arrive within seconds.

The system must:

• Display event information
• Authenticate users
• Search available seats
• Prevent two buyers from receiving one seat
• Process payments
• Confirm orders

Adding more web servers can help page requests, but it does not automatically increase inventory-locking capacity or a payment provider's quota.

Scaling requires understanding the entire path.

Scale up

Vertical scaling, or scaling up, gives one machine more resources:

• More CPU
• More memory
• Faster storage
• Greater network bandwidth

Benefits include:

• Minimal application change
• Simple data consistency
• Fewer machines to operate

Limits include:

• Maximum available machine size
• Increasing cost
• Maintenance interruption
• A larger single failure
• Diminishing returns

Scaling up is often the sensible first step. Distributed complexity should solve a demonstrated need.

Scale out

Horizontal scaling, or scaling out, adds more machines or instances.

If ten interchangeable application servers each handle part of the traffic, the service can increase capacity by adding instances.

Scaling out requires work to be distributable. Shared state, local files, in-memory sessions, or one database lock can prevent linear gains.

More machines also create:

• Network communication
• Partial failures
• Coordination
• Deployment complexity
• Observability needs

Horizontal scale exchanges machine limits for distributed-system challenges.

Find the bottleneck

Throughput is constrained by the narrowest important resource.

Potential bottlenecks include:

• CPU
• Memory
• Disk input/output
• Database locks
• Connection pools
• Network bandwidth
• External APIs
• Serial code paths
• Human approval

If the database handles 1,000 writes per second, adding application servers that generate 5,000 writes per second increases waiting and failure.

Measure latency, utilization, queue depth, errors, and saturation at each stage before choosing a remedy.

Stateless services are easier to replicate

A stateless request handler does not require the next request from one user to reach the same instance.

Shared state is stored in appropriate systems such as:

• Databases
• Distributed caches
• Object storage
• Queues
• Signed client tokens

Any healthy instance can then handle a request, which simplifies load balancing and replacement.

Stateless does not mean the application has no state. It means individual service instances do not privately own state required for request continuity.

Partitioning divides data or work

When one system cannot handle all data, partitioning divides it.

Examples:

• Customers A–M on one partition and N–Z on another
• Records divided by customer ID hash
• Events separated by geographic region
• Time-series data split by month

A good partition key distributes load and supports common queries.

Poor keys create hot partitions. If one celebrity account generates half the traffic, partitioning only by account ID may overload one shard while others remain idle.

Cross-partition queries and transactions become harder, so partitioning should follow measured access patterns.

Replication scales reads

Databases can copy data to read replicas.

Writes go to a primary; read-only queries can use replicas. This increases read capacity and can improve geographic latency.

Replicas may lag behind the primary. A user who updates a profile and immediately reads from a delayed replica may see the old value.

Applications must choose where freshness matters and route requests accordingly.

Replication does not divide write ownership by itself.

Caches remove repeated work

Frequently requested, expensive-to-compute data can be cached.

A product page cache may prevent thousands of identical database queries.

Caching improves:

• Latency
• Backend load
• Network use

It introduces:

• Stale values
• Invalidation logic
• Memory limits
• Cold-start behavior
• Uneven hot keys

The underlying system must still survive cache misses. A cache that hides an undersized database can turn a cache restart into a major outage.

Queues smooth bursts

Not every task must finish during the request.

An application can place email generation, image processing, or report creation onto a queue. Workers process jobs at a controlled rate.

Queues absorb bursts, but they do not create infinite capacity. Monitor:

• Queue depth
• Oldest job age
• Processing rate
• Failure and retry counts

When arrival rate remains higher than completion rate, the backlog grows without bound.

The system needs backpressure, capacity changes, load shedding, or reduced incoming work.

Backpressure protects the system

Backpressure tells upstream producers to slow down when downstream capacity is exhausted.

Methods include:

• Bounded queues
• Rate limits
• Concurrency limits
• Retry-after responses
• Admission control
• Dropping low-priority work

Without backpressure, requests accumulate in memory or threads until the entire service fails slowly.

Rejecting some work quickly can preserve useful service for the requests the system can handle.

Autoscaling follows signals

Autoscaling adds or removes instances based on metrics.

Signals may include:

• CPU utilization
• Request rate
• Queue depth
• Response latency
• Scheduled events
• Custom business measures

Scaling takes time. New instances may need to start, download images, warm caches, and establish connections.

Policies need minimum capacity, maximum cost controls, cooldowns, and protection against oscillation.

Autoscaling one layer can overload another, so whole-system limits matter.

Measure throughput and latency together

Throughput is work completed per unit time. Latency is time taken for one operation.

A system may maintain high throughput while individual requests wait longer in queues.

Averages hide painful outliers. Track percentile latency, such as the 95th or 99th percentile, along with error rates and saturation.

Define acceptable service behavior before load testing. "Handle more users" is not measurable enough.

Scaling must preserve correctness

Parallel work can introduce:

• Duplicate processing
• Lost updates
• Out-of-order events
• Conflicting writes
• Inconsistent reads

A system that responds quickly but sells one seat twice has not scaled successfully.

Use transactions, idempotency, unique constraints, version checks, ordering guarantees, and clear ownership where required.

Performance and correctness are one design problem, not separate phases.

Cost is part of scalability

A design that handles twice the load at ten times the cost may be technically scalable but economically unusable.

Track:

• Cost per request
• Cost per customer
• Idle capacity
• Data-transfer cost
• Storage growth
• Operational labor

Sometimes optimization, batching, or removing unnecessary work is better than adding infrastructure.

The cheapest request is often the one the system correctly avoids performing.

Knowledge check

1. How does vertical scaling differ from horizontal scaling?
2. Why does adding application servers not necessarily increase end-to-end capacity?
3. What does stateless mean for a service instance?
4. How can a poor partition key create a hot partition?
5. Why is backpressure healthier than allowing unlimited queues?

The one idea to remember

Scalability is not simply adding servers. It is removing bottlenecks and distributing work while preserving correctness, acceptable latency, and sustainable cost as demand grows.