Chaos Engineering and Failure-Injection Design TODO
Concept
Chaos engineering is a methodology that deliberately injects controlled failures into a production-like environment to test, by experiment, the hypothesis that a system keeps behaving normally. The procedure is: define a steady state using observable metrics, form a hypothesis that this metric holds even under a given failure, inject an actual failure (instance termination, added latency, packet loss, an error response from a dependency), and observe whether the hypothesis breaks. The core principles are to start with a small blast radius and widen it gradually, and to define abort conditions in advance that stop the experiment immediately. The goal isn't to create failures — it's to surface latent defects that were already there (missing timeouts, retry storms, circular dependencies, bad fallbacks) within a controlled window.
Real failures in distributed systems come less from individual components than from the combination of timeouts, retries, and fallbacks between them, and unit tests never catch that kind of interaction.
Code & Formula
# 카오스 엔지니어링·장애 주입 설계 — 정상 상태를 정의하고, 장애(지연 주입)를 걸어 가설이 깨지는지 관찰한다.
# 폭발 반경을 작게 시작하고, abort 조건을 넘으면 즉시 실험을 중단한다.
import random
random.seed(5)
def call_dependency(latency_ms, fail_rate=0.0):
"""외부 의존(RPC/DB) 호출을 흉내: 지연과 실패율을 파라미터로 받음"""
ok = random.random() > fail_rate
return ok, latency_ms + random.uniform(-2, 2)
def measure_steady_state(n_calls, injected_latency_ms=0, injected_fail_rate=0.0):
latencies, errors = [], 0
for _ in range(n_calls):
ok, lat = call_dependency(10 + injected_latency_ms, injected_fail_rate)
latencies.append(lat)
if not ok:
errors += 1
error_rate = errors / n_calls
avg_latency = sum(latencies) / len(latencies)
return avg_latency, error_rate
# 1. 정상 상태(steady state) 정의: 평균 지연 < 20ms, 에러율 < 1%
baseline_latency, baseline_error = measure_steady_state(200)
print(f"[정상 상태] 평균 지연 {baseline_latency:.1f}ms, 에러율 {baseline_error*100:.1f}%")
# 2. 가설: "RPC 노드에 100ms 지연이 추가돼도 평균 지연은 150ms 미만, 에러율은 5% 미만이다"
def run_experiment(injected_latency_ms, injected_fail_rate, blast_radius_calls):
ABORT_ERROR_RATE = 0.20 # 폭발 반경을 넘는 피해가 감지되면 즉시 중단
latency, error_rate = measure_steady_state(
blast_radius_calls, injected_latency_ms, injected_fail_rate
)
aborted = error_rate > ABORT_ERROR_RATE
return latency, error_rate, aborted
for label, inj_latency, inj_fail in [
("작은 폭발 반경: RPC 지연 +100ms", 100, 0.02),
("의존 서비스 오류 응답 20%", 0, 0.20),
]:
latency, error_rate, aborted = run_experiment(inj_latency, inj_fail, blast_radius_calls=50)
hypothesis_holds = latency < 150 and error_rate < 0.05
status = "ABORT (폭발 반경 초과)" if aborted else (
"가설 유지" if hypothesis_holds else "가설 깨짐 -> 결함 발견"
)
print(f"[{label}] 지연 {latency:.1f}ms, 에러율 {error_rate*100:.1f}% -> {status}")
docs/code/algorithms/algorithms-50.py
Exercise
In staging, inject artificial latency and intermittent errors into one external dependency (an RPC node or a database), record how the error rate and latency metrics change, then fix the timeout and retry policy and repeat the same experiment to confirm the improvement numerically.
Practical Connection
A prediction-market service depends on chain RPC, an oracle, and an indexer all at once, so experimenting in advance with whether RPC or oracle latency halts the entire settlement pipeline can substantially cut real operational risk.
Where it lands in Jayverse
- Verex: run a staging chaos experiment injecting latency/errors into RPC and oracle separately. Record whether either dependency alone halts the settlement pipeline — a steady-state hypothesis worth falsifying before mainnet volume.
- Devnet: use the hosted Anvil as the controlled blast-radius environment. Start with one dependency, widen gradually, and define abort conditions in advance, since Devnet is already the shared target every service points at.
- Auditor: record the before/after improvement number after fixing a timeout/retry policy. That pair belongs in the methodology notes, not just a changelog line, the same "write it down" instinct as the Kaiko-style rule.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| blast radius | (장애의) 파급 범위·폭발 반경 · 실험이나 장애가 영향을 미치는 범위를 비유적으로 말할 때. "start with a small blast radius and widen it gradually" |
| steady state | 정상 상태·안정 상태 · 시스템이 평소 유지하는 관측 가능한 기준 상태. "define a steady state using observable metrics" |
| abort conditions | 중단 조건 · 실험을 즉시 멈춰야 하는 사전 정의된 기준. "define abort conditions in advance" |
| latent defects | 잠재적 결함 · 겉으로는 안 보이지만 이미 내재해 있던 문제. "surface latent defects that were already there" |
| retry storms | 재시도 폭주 · 실패한 요청이 한꺼번에 재시도되며 부하가 몰리는 현상. "missing timeouts, retry storms, circular dependencies" |
| surface (동사) | 겉으로 드러나게 하다·표면화시키다 · 숨어 있던 결함을 실험으로 찾아낼 때. "to surface latent defects that were already there" |
| halt | 완전히 멈추게 하다·정지시키다 · 지연 하나가 전체 파이프라인을 세워버리는 상황. "halts the entire settlement pipeline" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.