Tail Latency — Hedged Requests and Load Shedding TODO
Concept
Tail latency refers to response times in the tail of the distribution — p99, p999 — not the average. When a single request fans out to multiple backends, the overall response is bound by the slowest one, so a rare delay on one individual server gets amplified into a common delay at the aggregate level. A hedged request is a technique where, if the first request hasn't gotten a response by, say, its p95 latency, a second identical request goes out to another replica and whichever response comes back first is used — extra load rises only a few percent, but the tail shrinks substantially. Load shedding runs the opposite direction: instead of queueing requests beyond capacity, it rejects them quickly at the door, in order to protect the latency target for requests already accepted. Because wait time worsens exponentially as the queue grows, this is usually paired with mechanisms like discarding requests past their deadline or adaptive concurrency limits.
A dashboard that only tracks average response time misses user-facing failures entirely, and a service that queues without bound under overload collapses further once retries pile on top. Managing the tail and shedding load are basic building blocks of availability design.
Code & Formula
# 테일 레이턴시 — hedged request(느리면 복제본에 한 번 더 요청)로 p99 꼬리를 줄이는 걸 시뮬레이션한다.
# 백엔드 3대 중 하나가 가끔 크게 느려질 때, 단일 요청 대비 hedge 를 걸면 총 요청은 조금 늘지만 꼬리는 크게 줄어든다.
import random
random.seed(11)
def backend_latency_ms(server_id):
# 대부분 10ms 내외, 가끔(5%) 200ms 근처로 튀는 "롱테일" 서버를 흉내낸다.
if random.random() < 0.05:
return random.uniform(150, 250)
return random.uniform(5, 15)
def single_request(num_backends=3):
server = random.randrange(num_backends)
return backend_latency_ms(server), 1 # (지연, 사용한 요청 수)
def hedged_request(hedge_delay_ms=20, num_backends=3):
"""첫 요청이 hedge_delay_ms 안에 안 끝나면 다른 서버에 한 번 더 보내고, 먼저 끝난 쪽을 쓴다."""
server1 = random.randrange(num_backends)
latency1 = backend_latency_ms(server1)
if latency1 <= hedge_delay_ms:
return latency1, 1 # 첫 응답이 충분히 빨랐다 -> hedge 발동 안 함
server2 = (server1 + 1) % num_backends
latency2 = backend_latency_ms(server2)
finish = min(latency1, hedge_delay_ms + latency2) # 두 번째 요청은 hedge_delay 이후에 출발
return finish, 2
def percentile(values, p):
s = sorted(values)
idx = int(len(s) * p) - 1
return s[max(0, idx)]
N = 5000
single_latencies, single_calls = [], 0
hedged_latencies, hedged_calls = [], 0
for _ in range(N):
lat, calls = single_request()
single_latencies.append(lat)
single_calls += calls
for _ in range(N):
lat, calls = hedged_request()
hedged_latencies.append(lat)
hedged_calls += calls
print(f"{'metric':>14} {'single':>10} {'hedged':>10}")
print(f"{'p50 (ms)':>14} {percentile(single_latencies, 0.50):>10.1f} {percentile(hedged_latencies, 0.50):>10.1f}")
print(f"{'p99 (ms)':>14} {percentile(single_latencies, 0.99):>10.1f} {percentile(hedged_latencies, 0.99):>10.1f}")
print(f"{'total calls':>14} {single_calls:>10} {hedged_calls:>10}")
print(f"\nextra request overhead: {(hedged_calls / single_calls - 1) * 100:.1f}% (요청 수는 조금만 늘었다)")
docs/code/algorithms/algorithms-44.py
Exercise
Set up three identical backends, artificially inject delay into one, then measure and tabulate p50, p99, and total request count for a single request versus a hedged request issued after p95 latency.
Practical Connection
A service like Verex that talks to multiple RPC providers can hedge eth_call and receipt lookups to keep one node's momentary delay from turning into delayed order fills or settlement transactions.
Where it lands in Jayverse
- Verex: implement hedged requests for eth_call and receipt polling as planned, and set the hedge delay at each RPC provider's measured p95, not a static timeout.
- Devnet: since the hosted Anvil node is a single backend, decide whether the API layer needs a second RPC endpoint specifically to make hedging possible, because there's nothing to hedge against with one node.
- Rabbit: apply load shedding to the bundler/relayer for AA UserOperations — reject new ops fast once queued past a deadline, instead of letting session-key transactions queue unbounded during congestion.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| tail latency | 꼬리 지연, 분포의 극단(p99 등)에서의 응답 시간 · 평균이 아니라 최악 사례를 재는 지표. "Tail latency refers to response times in the tail" |
| fan out | 하나의 요청이 여러 곳으로 갈라져 나가다 · 여러 백엔드에 동시에 요청을 보낼 때. "a single request fans out to multiple backends" |
| amplify into | ~로 증폭되어 나타나다 · 작은 지연이 전체 지연으로 확대될 때. "gets amplified into a common delay" |
| hedged request | 헤지 요청, 이중으로 보내 지연에 대비하는 요청 · 첫 응답이 늦으면 두 번째를 추가로 보내는 기법. "A hedged request is a technique where..." |
| at the door | 들어오기 전 초입에서(비유) · 과부하 요청을 아예 받지 않고 거절할 때. "rejects them quickly at the door" |
| pile on top | 위에 겹쳐 쌓이다, 가중되다 · 재시도가 겹쳐 상황이 더 악화될 때. "collapses further once retries pile on top" |
| building blocks | 기본 구성 요소, 토대 · 가용성 설계의 핵심 기법들을 가리킬 때. "basic building blocks of availability design" |
| p99/p999 | 응답시간 분포의 99번째/99.9번째 백분위수 · 평균이 아니라 최악에 가까운 사용자 경험을 나타내는 표기. "the tail of the distribution — p99, p999" |
| adaptive concurrency limits | 적응형 동시성 제한 · 부하 상황에 맞춰 자동으로 처리 가능한 요청 수를 조절하는 기법. "discarding requests past their deadline or adaptive concurrency limits" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.