Caching: Trading Freshness and Complexity for Speed

Many systems repeat the same expensive work: downloading an unchanged image, calculating the same report, or reading the same popular product from a database.

A cache keeps a reusable result closer to where it is needed.

A cache is temporary storage for data or computation that can be obtained again from a more authoritative source.

Caching improves speed and reduces load, but creates a second question: when is the copy too old to trust?

A concrete example: a website logo

The first time a browser visits a site, it downloads logo.png.

The server may respond with instructions allowing the browser to reuse that file for a day. On later pages, the browser reads the local copy rather than download identical bytes again.

Benefits include:

• Faster pages
• Lower network use
• Less origin-server traffic

If the logo changes while the old copy remains valid, some users still see the earlier image. That is the freshness tradeoff.

Caches exist at many layers

A request may encounter:

• CPU caches
• Operating-system disk caches
• Browser caches
• Service-worker caches
• CDN edge caches
• Reverse-proxy caches
• Application memory caches
• Distributed caches
• Database buffer caches

Each layer stores different objects and has different invalidation rules.

Unexpected behavior often comes from looking at one cache while another layer serves the stale value.

Cache hits and misses

A cache hit occurs when the requested value is present and usable.

A cache miss requires fetching or calculating the value from the source.

The hit ratio is:

hits / total cache lookups

A high hit ratio can be useful, but not sufficient. Serving easy, cheap requests from cache while every expensive request misses may produce an impressive ratio with little benefit.

Measure latency saved, backend load reduced, and correctness as well.

Time to live

A time to live, or TTL, defines how long a cached item remains valid.

Short TTL:

• Fresher data
• More source requests

Long TTL:

• Better reuse
• Greater staleness risk

The correct TTL follows how quickly data changes and how harmful stale values are.

A product image can be cached for months when its URL changes with its content hash. An account balance may need immediate consistency and should not use the same policy.

Invalidation

Cache invalidation removes or updates entries when source data changes.

Strategies include:

• Wait for TTL expiration.
• Delete the key after a successful write.
• Publish an invalidation event.
• Use versioned keys or URLs.
• Refresh in the background.

Invalidating too early reduces benefit. Invalidating too late serves incorrect data.

Distributed systems make invalidation harder because events can be delayed, duplicated, lost, or processed out of order.

Cache-aside

In the cache-aside pattern, application code:

1. Looks in the cache.
2. On a hit, returns the value.
3. On a miss, reads the database.
4. Stores the result in cache.
5. Returns it.

This is simple and caches only requested data.

After a database update, the application usually deletes or updates the corresponding cache entry.

A failure between database write and invalidation can leave a stale entry. Design must define whether temporary staleness is acceptable and how it ends.

Read-through and write strategies

With a read-through cache, the cache layer fetches missing data from the source.

Write approaches include:

• Write-through: Update cache and source synchronously.
• Write-behind: Update cache first and persist later.
• Write-around: Write to source and let future reads populate cache.

Write-behind can reduce latency but risks data loss if queued writes disappear.

Each strategy changes consistency and failure behavior. The cache should not accidentally become an undocumented source of truth.

Eviction makes room

Caches have finite capacity. They evict entries according to a policy.

Common policies include:

• Least recently used
• Least frequently used
• First in, first out
• Random
• Explicit priority
• TTL expiration

The ideal policy depends on access patterns. A one-time scan through a large dataset can evict genuinely hot values under a simple least-recently-used policy.

Monitor eviction rate and memory pressure, not only hit ratio.

Cache stampedes

Suppose a popular entry expires. Thousands of requests miss simultaneously and all query the database.

This is a cache stampede.

Mitigations include:

• Allow one requester to refresh while others wait.
• Refresh before hard expiration.
• Add randomness to TTLs so many keys do not expire together.
• Serve slightly stale data during refresh.
• Limit concurrent source requests.

The source must also have enough capacity for cold starts and unavoidable misses.

Negative caching

A cache can temporarily store "not found" results.

Without negative caching, repeated requests for a nonexistent user or URL can hammer the source.

Negative entries usually need shorter TTLs because the object may be created soon.

Be careful with authorization. A result that is absent for one user may exist for another. Cache keys must include every property that affects the result.

Cache keys define correctness

A key identifies which requests may share a cached value.

If a page varies by language and user role, a key containing only the URL can serve English administrator content to another user.

Relevant key dimensions may include:

• Resource ID
• Version
• Locale
• Tenant
• Authorization scope
• Query parameters
• Content encoding

Missing a dimension can leak or corrupt behavior. Adding unnecessary dimensions fragments the cache and lowers reuse.

Key design is part of the application's data model.

Cold caches and warm-up

After restart or deployment, a cache may be empty.

The resulting miss surge can overload databases and increase latency.

Options include:

• Gradual traffic shifting
• Preloading known hot keys
• Reusing an external cache
• Rate-limiting expensive misses
• Keeping old instances during warm-up

A system that works only with a fully warm cache is fragile. Test cold-cache behavior and recovery time.

Caching and consistency

Cached data is a copy. It can disagree with the source.

Applications should define:

• Maximum acceptable staleness
• Which operations require read-after-write behavior
• How updates invalidate copies
• What happens when the cache is unavailable
• Whether stale data is safer than an error

For a news article, slight staleness may be fine. For permission revocation, stale authorization can be dangerous.

Use cache policies based on meaning, not one organization-wide TTL.

Observe cache value

Monitor:

• Hit and miss rates
• Hit and miss latency
• Evictions
• Memory use
• Source load
• Refresh errors
• Stale responses
• Hot keys
• Connection saturation

A cache can increase total latency if network calls to it are slower than recomputing cheap values.

Measure end-to-end benefit before adding another distributed dependency.

Knowledge check

1. What distinguishes a cache from the authoritative source?
2. How does TTL affect freshness and source load?
3. What failure creates a cache stampede?
4. Why must authorization-related values influence a cache key?
5. Why should a system be tested with a cold cache?

The one idea to remember

Caching stores temporary reusable copies to avoid repeated work. The speed comes with responsibilities for keys, expiration, invalidation, capacity, cold starts, and a clearly defined tolerance for stale data.