Performance Testing: Workloads, Targets, and Bottlenecks

"The application is fast" is incomplete.

Fast for how many users, performing which actions, with what data, on which infrastructure, and according to which success target?

Performance is meaningful only relative to a defined workload, environment, and objective.

Performance testing supplies measurements, while investigation explains why the system behaves that way.

A concrete example: a sale event

An online store expects 1,000 simultaneous shoppers.

A load test models browsing, search, cart updates, and checkout. Its targets might require:

95 percent of browse responses below 300 milliseconds,
checkout below one second at the 95th percentile,
an error rate below 0.1 percent,
no inventory overselling,
and database CPU below a safe threshold.

The result is actionable because workload and success are defined.

Latency

Latency is the time one operation takes.

Average latency can hide bad experiences. If most requests take 100 milliseconds and a few take ten seconds, the average looks better than the affected users feel.

Percentiles such as p50, p95, and p99 show the distribution. A p95 of 300 milliseconds means 95 percent completed at or below that time.

Throughput

Throughput measures completed work per unit of time:

requests per second,
messages per minute,
files per hour,
or transactions per second.

High throughput is not useful if latency or failures exceed acceptable limits. These measurements must be interpreted together.

Concurrency

Concurrency describes work in progress at the same time.

One thousand requests per second can come from quick operations with low concurrency or slow operations with many outstanding requests. Connection pools, thread limits, and queues respond to concurrency as well as rate.

A workload model should represent user pacing and operation duration.

Load testing

Load testing checks behavior under expected and peak demand.

It answers:

Are targets met?
Which resource approaches saturation?
Does autoscaling respond?
Does the system remain correct?

The goal is not merely to generate the largest possible number.

Stress testing

Stress testing pushes beyond expected capacity.

It explores:

where the system fails,
whether failure is gradual,
which component fails first,
whether recovery occurs,
and whether data remains correct.

A system that rejects excess work clearly is safer than one that becomes slow, inconsistent, and difficult to recover.

Spike testing

Spike testing applies sudden changes in load.

This can reveal:

slow autoscaling,
cold caches,
connection storms,
queue surges,
and rate-limit behavior.

Gradual load alone may hide problems caused by abrupt demand.

Soak testing

A soak or endurance test runs for an extended period.

It finds slow accumulation:

memory leaks,
unreleased connections,
growing logs,
queue backlog,
cache drift,
or periodic job interference.

A system can pass a ten-minute load test and degrade after twelve hours.

Realistic workload mix

Production traffic is rarely one endpoint repeated evenly.

A realistic model includes:

common reads,
less frequent writes,
authenticated and anonymous users,
cache hits and misses,
different payload sizes,
and pauses between human actions.

Measure production patterns where privacy and safety allow, then model representative distributions.

Realistic data

Data volume and shape influence query plans, indexes, serialization, and caches.

A database with 100 rows may respond differently from one with 100 million. Tests need representative cardinality, value distribution, and relationships without copying unsafe personal data.

Synthetic generation should preserve performance-relevant characteristics.

Environment fidelity

Laptop results cannot predict production capacity directly.

Important factors include:

CPU and memory,
network distance,
database configuration,
storage,
service limits,
and competing workloads.

Lower environments can still compare changes, but capacity claims require a sufficiently representative setup.

Coordinated omission

A flawed load generator may wait for a slow response before scheduling the next request.

During the delay, it sends less load and fails to record requests that real users would still attempt. This coordinated omission makes severe latency look better.

Tools and workload models should preserve the intended arrival rate where appropriate.

Resource saturation

Measure system internals alongside user-visible results:

CPU,
memory,
disk I/O,
database connections,
thread or event-loop delay,
queue depth,
cache hit rate,
and network throughput.

The first saturated resource often explains the shape of rising latency.

Bottlenecks

A bottleneck is the limiting part of the system.

After one bottleneck is improved, another may become dominant. Performance optimization is iterative:

measure,
identify constraint,
change one meaningful factor,
retest,
verify correctness and cost.

Guessing often optimizes code that contributes little to total time.

Baselines and regression

A baseline captures expected measurements for a known version and environment.

Future results can detect regressions, but thresholds need tolerance for natural variation. Compare multiple runs and investigate significant trends rather than failing on meaningless noise.

Record code, configuration, data, and infrastructure with each result.

Client-side performance

Server latency is only part of experience.

Web and mobile testing may measure:

startup,
input responsiveness,
rendering,
layout stability,
memory,
battery use,
and download size.

Test representative devices and networks, not only powerful development machines.

Correctness under load

A fast system that loses data or double-charges users has failed.

Performance tests should verify invariants:

accepted requests are accounted for,
duplicates are controlled,
balances remain correct,
and recovery does not corrupt state.

Load can expose races that ordinary functional tests rarely trigger.

Safe execution

Performance tests can overload systems and create costs.

Use isolated accounts, controlled environments, rate boundaries, clear ownership, and cleanup. Production tests require especially careful limits to avoid affecting real users or external providers.

Test traffic should be identifiable in monitoring.

Knowledge check

Why is average latency insufficient?
How do load, stress, spike, and soak tests differ?
Why does realistic data volume matter?
What is coordinated omission?
Why must performance tests check correctness as well as speed?

The one idea to remember

Performance testing measures latency, throughput, resource use, and correctness for a defined workload and environment. Set explicit targets, model realistic demand, observe bottlenecks, and treat every result as contextual evidence.