Engineering problems / Amazon

The thundering herd: when a 99% hit rate still takes down your database

One popular key expires. Every request that arrives in the next 300 milliseconds misses, and all of them ask the database the same question at once. The cache was working perfectly right up until it wasn't.

MockStruct engineering notes~6 min readKnow what a cache TTL is

Your cache hit rate is 99%. Your database is idle. At 03:00 one popular key reaches the end of its TTL, and within the next three hundred milliseconds every request that needs it misses, and every one of them asks the database the same question. The cache did not fail. It worked perfectly, right up to the microsecond it stopped.

Prefer to learn by doing?

Send twenty clients at an expiring key and count the database queries under three strategies — including the one that still fails.

Open the 5-minute lesson →

The 1% that arrives all at once

A cache hit rate is an average, and averages hide concurrency. Those misses are not sprinkled evenly through the hour; they are bunched into the instant after an entry expires. If the value takes 300ms to recompute and you serve 200 requests a second for it, then roughly 60 requests pile in before the first one finishes and writes the result back. None of them finds a cached value, so all 60 recompute it.

Every one of those 60 produces an identical answer. Fifty-nine of them are pure waste, and they arrive as a burst precisely when the database is least able to absorb one. This is why a stampede is best understood as a concurrency bug in your caching layer rather than a capacity problem in your database — adding database capacity treats the symptom and raises the ceiling you will hit next time.

What the people running large fleets watch for

Amazon's Builders' Library describes the thundering herd as many clients making requests that need the same uncached downstream resource at approximately the same time. Its named fix is request coalescing: making sure only one request for an uncached resource is in flight at a time. The article dates from December 2019 and is written by Jas Chhabra.

The same article names two causes that have nothing to do with a TTL, and they are the ones that catch teams out. A newly launched server starts with a completely empty cache, so scaling up under load sends a burst at your dependency. And a fleet-wide cache flush — a deploy, usually — does the same thing across every host simultaneously. Your TTL strategy does not protect you from either.

Primary reading: Caching challenges and strategies · Redis: cache stampede prevention

Three defences, and what each one costs

Coalescing, sometimes called single-flight or a mutex, is the straightforward one: the first request to miss takes a lock and rebuilds the value, and everyone else waits for it. One database query instead of sixty. The cost is latency — the other fifty-nine now wait the full recompute time — and a new dependency on the lock itself.

Probabilistic early expiration sidesteps the wait. Each read, as the expiry approaches, has a small and rising chance of refreshing the value early, so in practice one request refreshes while the rest are still being served from a valid cache. Nobody ever waits on a miss. Redis's own write-up on stampede prevention describes this alongside the mutex approach and suggests starting with the mutex for most applications, keeping early expiration for very high-traffic keys.

The third is to decouple serving from freshness: a soft TTL that triggers a background refresh and a hard TTL that is the real deadline. The Builders' Library frames this as resilience — you keep serving slightly stale data when the dependency is unavailable. It is not a stampede fix on its own, but it means a slow dependency degrades freshness instead of availability.

# probabilistic early expiration
# delta  = how long the value took to compute last time
# beta   = tuning knob, 1.0 is the usual starting point
if now - delta * beta * log(random()) >= expiry:
    refresh_in_background()
return cached_value

Every fix moves the failure somewhere else

Put a mutex in and you have made your availability depend on lock release. If the holder crashes mid-rebuild, everyone else waits for a lock that will never be released — you have converted a load spike into a hang. So the lock needs its own expiry, and now you have a second problem: a slow rebuild outlives its lock, a second client takes the lock, and the first client finishes and deletes a lock it no longer owns.

That specific bug is why Redis's guidance is to write a unique token into the lock and release it with a script that checks the token before deleting. It is a small detail, and it is the difference between a defence that holds under load and one that quietly stops being a lock at exactly the moment it matters.

There is one more trap worth naming: do not cache only successes. The Builders' Library warns against amplifying an outage by repeatedly asking for the same downstream resource and discarding the errors. If the thing you are calling is failing, an uncached error is a stampede with extra steps.

Turn the idea into a small project

Build a fake slow endpoint that takes 300ms and counts how many times it is called. Put a cache in front with a 2-second TTL, then fire 50 concurrent requests for one key across several TTL cycles. Record the call count for no protection, for a mutex, and for early expiration. Then make it realistic: kill the lock holder mid-rebuild and see what the other 49 do.

One question to take with you

Your cache is empty after a deploy, so nothing is expiring — which of these three defences still protects your database?

Test your understanding →

For the related problem of a write that runs twice, read our webhook idempotency walkthrough.

Read next / Algorithms

The LRU cache you were asked to implement — and how Redis really evicts keys

Read it →

Published edition 2026-10-31.