Cache Consistency, Invalidation, and Stampede Prevention TODO
Concept
A cache starts carrying a consistency problem the moment the source of truth and the copy diverge; policies broadly split into expiration-based (TTL) and invalidation-based (delete or update on write). Updating the cache on the write path can let concurrent writes land out of order and leave a stale value behind, so deleting is usually safer than updating. A cache stampede happens when a popular key expires and many requests hit the origin simultaneously, which can momentarily take the origin down. The countermeasures are single-flight (letting only one request query the origin), adding random jitter to expiration times, proactive early recomputation before expiry, and stale-while-revalidate (serving the stale value briefly while a refresh is in flight). Whichever policy you pick, you first have to define 'how much staleness is acceptable' before a choice is even possible.
A large share of outages don't come from missing a cache — they come from the cache emptying all at once and the backend collapsing under the exposed load.
Code & Formula
# 캐시 일관성·무효화·스탬피드 방지 — TTL 만료 순간 다수 요청이 몰리는 스탬피드를
# single-flight(락)로 한 요청만 원본을 조회하게 막는 것을 시연한다.
import time
import threading
origin_calls = 0
origin_lock = threading.Lock()
def slow_origin_fetch(key):
global origin_calls
with origin_lock:
origin_calls += 1
time.sleep(0.05) # 원본 DB/서비스 호출을 흉내
return f"value-for-{key}"
class SingleFlightCache:
def __init__(self):
self.store = {} # key -> (value, expires_at)
self.inflight = {} # key -> threading.Event (동시 요청 합류용)
self.lock = threading.Lock()
def get(self, key, ttl=1.0):
with self.lock:
entry = self.store.get(key)
if entry and entry[1] > time.time():
return entry[0]
if key in self.inflight:
event = self.inflight[key]
else:
event = threading.Event()
self.inflight[key] = event
event = None # 이 스레드가 원본을 조회할 담당자
if event is not None:
event.wait()
return self.store[key][0]
value = slow_origin_fetch(key)
with self.lock:
self.store[key] = (value, time.time() + ttl)
waiter = self.inflight.pop(key)
waiter.set()
return value
cache = SingleFlightCache()
results = []
def worker():
results.append(cache.get("hot-key"))
threads = [threading.Thread(target=worker) for _ in range(20)]
for t in threads: t.start()
for t in threads: t.join()
print("concurrent requests:", len(threads))
print("origin fetches actually made:", origin_calls) # 1 이어야 stampede 방지 성공
print("all results identical:", len(set(results)) == 1)
docs/code/algorithms/algorithms-79.py
Exercise
Build a load test that hits the same key concurrently, then compare origin query counts and p99 latency between a plain TTL cache and one using single-flight.
Practical Connection
Values like prices, order-book snapshots, or oracle responses — where the origin is expensive and access is concentrated — carry the highest stampede risk, and here 'acceptable staleness' maps directly to the price accuracy users actually see.
Where it lands in Jayverse
- Verex: put single-flight in front of the order-book snapshot and oracle price cache specifically. A stampede there directly degrades the price accuracy users see, which is the highest-stakes case the PoC names.
- Number: use stale-while-revalidate for distributed readings. A popular reading's cache expiry shouldn't hammer the origin computation when many licensed consumers hit it at once.
- gitboard: track p99 latency and origin-query count per cached key as a standard test. Run the exercise's load test against any new cache added to a service, not just Verex's, before it ships.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| cache stampede | 캐시 쏠림(인기 키 만료 시 요청이 한꺼번에 몰리는 현상) · 캐시 장애의 대표적 원인. "A cache stampede happens when a popular key expires" |
| single-flight | (동시 요청을) 하나로 묶어 처리하는 기법 · 중복 요청 중 하나만 원본에 보내는 방어 기법. "single-flight (letting only one request query the origin)" |
| jitter | 지터(무작위로 섞은 편차) · 만료 시각을 흩뿌려 동시 만료를 막는 기법. "adding random jitter to expiration times" |
| stale-while-revalidate | 갱신 중에는 오래된 값을 잠시 보여주는 기법 · 최신화 동안 지연을 감추는 캐시 전략. "stale-while-revalidate (serving the stale value briefly" |
| take down | (서버 등을) 다운시키다·마비시키다 · 몰린 요청이 원본 서버를 멈추게 하는 상황. "can momentarily take the origin down" |
| exposed load | (캐시가 사라져) 그대로 노출된 부하 · 백엔드가 감당 못 하는 트래픽을 가리킴. "the backend collapsing under the exposed load" |
| acceptable staleness | 허용 가능한 데이터 지연(오래됨) 정도 · 캐시 전략을 고르기 전에 먼저 정의해야 하는 기준. "how much staleness is acceptable" |
| TTL | 생존 시간(Time To Live) · 캐시 정책 중 만료 시간 기반 방식을 가리킴. "policies broadly split into expiration-based (TTL)" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.