Autoscaling: Turning Demand Signals into Capacity

Demand changes over time.

A payroll product is busy near month-end, a store spikes during a sale, and a background queue grows after a large import. Keeping maximum capacity running constantly wastes resources, while fixed average capacity fails at peaks.

Autoscaling applies a policy that turns observed signals into capacity changes.

It can respond quickly and consistently, but poor signals automate poor decisions.

A concrete example: scaling queue workers

A worker group processes image jobs.

The policy observes:

pending job count,
age of the oldest job,
current worker count,
and average processing time.

When backlog age rises, it adds workers. After the queue remains low for a cooldown period, it removes them gradually.

Queue age connects scaling to user delay better than CPU alone.

Horizontal scaling

Horizontal scaling changes the number of instances, containers, or workers.

It works well when:

work can be divided,
instances are interchangeable,
state is externalized,
and load distributes correctly.

Adding application instances does not help a saturated single database unless the bottleneck is elsewhere.

Vertical scaling

Vertical scaling changes the size of a resource:

more CPU,
more memory,
faster storage,
or larger database class.

It may be simpler for stateful workloads but can require restart and has an upper limit.

Some platforms combine vertical recommendations with horizontal scaling.

Reactive scaling

Reactive policies respond after a metric crosses a threshold.

Examples:

CPU above 70 percent,
queue age above 30 seconds,
or requests per instance above a target.

The system is already under pressure when the signal appears, so startup time and spare capacity matter.

Predictive and scheduled scaling

Scheduled scaling adds capacity before a known event such as business opening.

Predictive systems model recurring demand. Forecasts can reduce reaction delay but may fail when behavior changes unexpectedly.

Keep reactive safeguards for surprises and validate prediction against actual outcomes.

Choosing a signal

The metric should reflect the constrained work.

Useful examples:

requests per instance,
concurrent sessions,
queue age,
active connections,
custom work units,
or latency with capacity guardrails.

CPU can be misleading for I/O-bound applications, and average CPU can hide one hot instance.

Target tracking

A target-tracking policy tries to maintain a metric near a desired value.

For example:

50 requests per second per instance

As total request rate changes, the controller adjusts instance count. The relationship between metric and capacity should be reasonably predictable.

Step scaling

Step policies apply different changes by severity:

small breach: add two instances,
large breach: add ten,
critical backlog: jump to a higher minimum.

Large steps respond quickly but can overshoot. Observe how the workload and platform behave during each transition.

Startup time

New capacity is not useful until it is ready.

Startup can include:

provisioning,
image download,
application boot,
cache warm-up,
connection creation,
and readiness checks.

A five-minute startup cannot rescue a request with a thirty-second deadline unless capacity is added earlier.

Readiness

Do not send traffic merely because a process exists.

Readiness should verify the instance can serve required behavior. During warm-up, it may need to load models, compile code, or establish pools.

Premature traffic can cause failures that trigger even more scaling.

Minimum and maximum capacity

A minimum avoids cold-start delay and preserves redundancy.

A maximum protects:

cost,
downstream databases,
external quotas,
and account limits.

Maximum capacity should trigger alerts before demand reaches it, because the policy cannot scale further.

Cooldowns and stabilization

Metrics fluctuate.

Without stabilization, the controller can add and remove capacity repeatedly, called thrashing. Cooldowns and scale-down windows let recent changes take effect before reversing them.

Scale-up often should be faster than scale-down because removing too much capacity creates immediate risk.

Scaling in safely

Before terminating an instance:

stop new work,
drain active requests,
finish or return queue messages,
close connections,
and preserve required state.

Abrupt removal can interrupt user requests or duplicate background work.

Statelessness

Horizontal scaling assumes requests can reach different instances.

Local-only sessions, uploaded files, or in-memory jobs create affinity and loss during replacement. Store durable shared state appropriately or use intentional session affinity with understood limitations.

Autoscaling exposes hidden statefulness.

Downstream bottlenecks

More frontend instances create more:

database connections,
cache requests,
third-party calls,
logs,
and messages.

Scale policies should respect downstream capacity. Admission control may be safer than scaling callers into a fixed dependency.

Database autoscaling

Some managed databases scale storage, compute, or read replicas automatically.

Database changes can be slower and consistency-sensitive. Read scaling does not fix write contention, and storage growth may be irreversible.

Understand exact triggers, lag, and billing.

Queue consumers

Scale consumers based on completion goals:

backlog per worker,
oldest-message age,
processing duration,
and retry rate.

If a dependency is failing, adding workers may increase retries rather than throughput. Health-aware scaling can cap demand during downstream incidents.

Cost control

Autoscaling makes spending variable.

Use:

budgets,
maximums,
anomaly alerts,
per-team ownership,
and cost-per-work-unit metrics.

A traffic attack or software loop can otherwise scale the bill as effectively as legitimate success.

Load testing

Test scaling with realistic demand:

gradual growth,
sudden spike,
long peak,
downstream slowdown,
and recovery.

Verify not only instance count but user latency, error rate, readiness, database impact, and scale-in safety.

Knowledge check

How do horizontal and vertical scaling differ?
Why can CPU be a poor scaling signal?
How does startup time affect reactive scaling?
Why should scale-in drain work?
How can autoscaling worsen a downstream outage?

The one idea to remember

Autoscaling is a feedback controller that converts metrics into capacity. Choose signals tied to useful work, account for startup and dependencies, bound cost, stabilize scale-down, and test the complete system response rather than only instance count.