Rate Limiting: Protecting Capacity, Cost, and Fair Access

An API has finite capacity.

One caller, bug, retry loop, or attack should not be allowed to consume all of it.

Rate limiting restricts work within a time, concurrency, or resource budget to protect reliability, cost, fairness, and abuse boundaries.

It complements authentication and capacity planning. It does not replace either.

A concrete example: login attempts

A login endpoint receives repeated password guesses.

A policy might limit:

• Attempts per account
• Attempts per network
• Attempts per device signal
• Global authentication load

After the threshold, the service delays or rejects attempts.

Limiting only by IP can block many legitimate users behind one network and can be bypassed by distributed attackers.

Several signals and progressive controls produce better protection.

What rate limits protect

Rate limits can protect:

• CPU
• Database connections
• Expensive searches
• Third-party API quota
• Email or SMS cost
• Login security
• Fair multi-tenant use
• Business inventory

The unit may be:

• Requests
• Tokens
• Bytes
• Compute units
• Concurrent jobs
• Messages sent

One request can cost far more than another.

Resource-weighted limits may be more accurate than raw request count.

Identify the subject

A limit can apply per:

• IP address
• User
• API key
• Tenant
• Device
• Endpoint
• Region
• Global service

No one key is perfect.

Unauthenticated traffic may need IP and fingerprint signals. Authenticated traffic can use account or client identity.

A tenant with thousands of users needs a different budget from one individual.

Layer limits so a global safety ceiling remains even if per-user controls are bypassed.

Fixed-window counters

A fixed-window limit counts requests in intervals:

100 requests from 12:00:00 through 12:00:59

It is simple and storage-efficient.

At the boundary, a caller can send 100 requests at 12:00:59 and another 100 at 12:01:00.

That produces 200 requests in two seconds while respecting each window.

Fixed windows fit coarse quotas but allow boundary bursts.

Sliding-window logs

A sliding log stores timestamps of recent requests and counts those within the last interval.

It enforces:

no more than 100 in any rolling 60 seconds

This is accurate but can require substantial memory and cleanup for high-volume clients.

Optimized sliding-window counters approximate the behavior using neighboring buckets.

Choose accuracy according to impact and scale.

Token bucket

A token bucket contains tokens that refill at a steady rate.

Each request consumes tokens.

If the bucket has enough tokens, the request proceeds. Otherwise it is limited.

Parameters:

• Refill rate: sustained allowance
• Bucket capacity: allowed burst
• Request cost: tokens consumed

This supports short bursts while controlling long-term average.

It is a common fit for APIs and network traffic.

Leaky bucket

A leaky-bucket model drains work at a steady rate.

Incoming requests enter a bounded queue.

If the queue fills, excess work is rejected.

This smooths bursts for downstream systems that need stable throughput.

Queueing adds latency. If the request cannot wait, rejection may be healthier.

The term is implemented differently across systems, so document actual behavior rather than rely only on the name.

Quotas

A quota limits usage over a longer period:

• 10,000 requests per day
• 1 TB transfer per month
• 100 report jobs per tenant

Quotas manage cost and product plans.

They differ from burst limits.

A client may be under monthly quota but still overload the service with all requests in one second.

Use both sustained quotas and short-window or concurrency protection.

Concurrency limits

Concurrency limits restrict simultaneous work.

They are useful when:

• Requests are long-running.
• Each request holds a database connection.
• Work consumes large memory.
• Downstream capacity is fixed.

Ten slow requests may be more dangerous than one thousand cached requests.

When at the limit, reject, queue briefly, or prioritize according to policy.

Concurrency controls act directly on in-flight resource use.

Distributed rate limiting

A service with many instances needs shared or coordinated counters.

Approaches include:

• Central fast data store
• Gateway-level limiting
• Consistent partitioning by identity
• Local limits plus global limits
• Approximate distributed algorithms

Strongly consistent global counters add latency and can become a bottleneck.

Local counters are fast but may allow the total to exceed the intended limit.

Choose the acceptable precision and failure behavior.

Fail-open versus fail-closed

What happens if the rate-limit store is unavailable?

• Fail open: Allow requests, preserving availability but losing protection.
• Fail closed: Reject requests, preserving the boundary but causing outage.

The choice depends on the endpoint.

A public content read may fail open with local emergency limits.

A password-reset or expensive money-transfer operation may need stronger fail-closed behavior.

Document and test the degraded mode.

Return useful signals

When limiting, return:

429 Too Many Requests
Retry-After: 30

APIs may also expose:

• Limit
• Remaining budget
• Reset time
• Policy identifier

Standards and header names vary.

Do not expose details that materially help an attacker bypass anti-abuse systems.

Legitimate clients need enough information to back off and schedule work.

Client retry behavior

A well-behaved client:

1. Honors Retry-After.
2. Uses exponential backoff.
3. Adds jitter.
4. Caps attempts.
5. Stops obsolete work.
6. Reduces concurrency.

Immediate synchronized retries create a thundering herd when the window resets.

Client libraries should not hide endless retries.

For nonidempotent requests, retry requires an idempotency mechanism.

Fairness and prioritization

Equal request count is not always fair.

A platform may allocate:

• Guaranteed baseline per tenant
• Larger purchased quota
• Priority for interactive traffic
• Separate budgets for batch work
• Emergency operational traffic

Weighted fair queuing and admission control can preserve important workflows during overload.

Priority must be bounded so low-priority work eventually proceeds and one premium tenant cannot destroy global reliability.

Cost controls

Rate limiting can prevent runaway cost from:

• SMS sending
• Generative-model tokens
• Image processing
• Third-party searches
• Large exports

Use a cost model:

simple lookup = 1 unit
full export = 500 units

Rejecting after expensive work is performed is too late.

Estimate and reserve budget before execution, then reconcile actual use when needed.

Rate limiting and abuse

Rate limits slow abuse but do not determine whether a request is legitimate.

Attackers can distribute traffic across:

• IPs
• Accounts
• Devices
• Stolen keys

Combine limiting with:

• Authentication
• Bot detection
• Behavioral analysis
• Verification challenges
• Fraud controls
• Authorization

Avoid controls that disproportionately block shared networks, accessibility tools, or users with unstable connections.

Login-specific design

Simple account lockout can let an attacker deny service by repeatedly attempting another user's login.

Safer approaches include:

• Progressive delay
• Risk-based verification
• IP and account combination
• MFA
• Notification
• Known-device signals
• Password breach checks

Do not reveal whether an account exists unnecessarily.

Monitor success and failure patterns and give legitimate users a secure recovery path.

Observability

Track:

• Allowed and rejected work
• Limit by policy and subject
• Near-limit usage
• Hot tenants or keys
• Retry volume
• Store latency and errors
• Cost saved
• User-impact complaints

A high rejection rate may indicate attack, a broken client, insufficient capacity, or a poorly chosen policy.

Review limits as usage evolves.

A static threshold from launch may become either irrelevant or harmful.

Test the limits

Test:

• Boundary behavior
• Burst allowance
• Window reset
• Distributed instances
• Clock skew
• Counter-store failure
• Retry headers
• Exemptions
• Very expensive operations
• Tenant isolation

Avoid relying only on unit tests for a distributed algorithm.

Load testing should confirm that limiting activates before the protected dependency collapses.

Knowledge check

1. Why is limiting login only by IP insufficient?
2. How does a token bucket permit bursts while controlling average rate?
3. Why are quotas and concurrency limits different?
4. What tradeoff exists between fail-open and fail-closed behavior?
5. Why should clients add jitter to backoff?

The one idea to remember

Rate limits allocate finite capacity across time, callers, and work cost. Use layered identities, burst and concurrency controls, clear retry signals, and tested degraded behavior to protect both the service and legitimate users.