Benchmark Methodology — Warm-up, Variance, and Automatic Regression Detection TODO
Concept
A benchmark isn't about measuring one number — it's an experiment that estimates a distribution of measurements. JIT compilation, caches, branch predictors, and connection pools all mean early runs differ from steady state, so the warm-up period should be excluded from measurement. What users actually feel is closer to the median and tail percentiles (p95, p99) than the average, and reporting the variance across repeated runs is what makes comparing two versions valid. Regression detection comes down to whether the difference from a baseline exceeds the noise band, so on noisy environments like shared CI, you either widen the threshold generously or use relative comparisons within the same run.
A benchmark that ignores warm-up and variance lets real performance regressions through while flagging harmless changes as regressions — and eventually nobody trusts the results anymore.
Code & Formula
# 벤치마크 방법론 — 워밍업 구간을 제외하고, 분산까지 함께 보고해 회귀를 판정한다.
# 노이즈 범위(표준편차 기반 임계값)를 넘어선 차이만 "회귀"로 인정한다.
import random
import statistics
random.seed(7)
def run_samples(n, base_ms, warmup=3):
"""워밍업 n_warmup개는 버리고, 정상 상태 표본만 반환."""
raw = [base_ms + random.gauss(0, base_ms * 0.05) for _ in range(n + warmup)]
# 초반 워밍업 구간은 JIT/캐시 예열 때문에 더 느리다고 가정
for i in range(warmup):
raw[i] *= 1.6
return raw[warmup:]
def summarize(samples):
mean = statistics.mean(samples)
stdev = statistics.stdev(samples)
p95 = sorted(samples)[int(len(samples) * 0.95)]
return mean, stdev, p95
def is_regression(baseline, candidate, z_threshold=2.0):
"""두 집단 평균 차이가 결합 표준오차의 z_threshold배를 넘으면 회귀로 판정."""
m0, s0, _ = summarize(baseline)
m1, s1, _ = summarize(candidate)
se = ((s0 ** 2) / len(baseline) + (s1 ** 2) / len(candidate)) ** 0.5
z = (m1 - m0) / se if se else float("inf")
return z > z_threshold, z
baseline = run_samples(30, base_ms=10.0)
candidate_ok = run_samples(30, base_ms=10.2) # 무해한 변경
candidate_bad = run_samples(30, base_ms=13.0) # 실제 회귀
for name, cand in [("무해한 변경", candidate_ok), ("실제 회귀", candidate_bad)]:
m0, s0, p95_0 = summarize(baseline)
m1, s1, p95_1 = summarize(cand)
flagged, z = is_regression(baseline, cand)
print(f"[{name}] baseline mean={m0:.2f}ms p95={p95_0:.2f}ms | "
f"candidate mean={m1:.2f}ms p95={p95_1:.2f}ms | z={z:.2f} -> "
f"{'REGRESSION' if flagged else 'OK'}")
docs/code/algorithms/algorithms-46.py
Exercise
Run a single function many times with Go's testing.B, extract percentiles and standard deviation, and tabulate how much the numbers change with and without warm-up.
Practical Connection
Contract gas consumption is deterministic, so regressions can be pinned to an exact value, but off-chain matching-engine and API latency have to be managed as distributions — Verex's CI needs two different kinds of regression gates for these two cases.
Where it lands in Jayverse
- Verex/CI: build the two regression gates the practical connection names. An exact-value gate for contract gas snapshots, and a percentile/variance-based gate (p95/p99 vs a noise band) for matching-engine and API latency.
- gitboard: report p50/p95/p99 for any latency dashboard row, not the average. Per this page, that's closer to what users actually feel.
- CI: exclude a warm-up period from any benchmark used as a regression gate, and widen the noise threshold or use relative in-run comparison on shared runners. A fixed absolute threshold on noisy CI either lets regressions through or cries wolf until nobody trusts it.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| steady state | 안정 상태(워밍업 이후 일정해진 상태) · 초기 변동이 끝나고 값이 일정해진 구간을 말할 때. "early runs differ from steady state" |
| the noise band | 잡음 범위(측정 오차로 볼 수 있는 폭) · 실제 변화인지 오차인지 구분하는 기준. "exceeds the noise band" |
| widen the threshold | 기준(허용 폭)을 넓히다 · 잡음이 많은 환경에서 판정 기준을 느슨하게 할 때. "widen the threshold generously" |
| let ... through | ~을 걸러내지 못하고 통과시키다 · 감지해야 할 문제를 놓칠 때. "lets real performance regressions through" |
| flag as | ~로 표시하다/판정하다 · 자동화된 시스템이 특정 항목을 문제로 분류할 때. "flagging harmless changes as regressions" |
| nobody trusts the results anymore | 아무도 더 이상 결과를 신뢰하지 않게 되다 · 잘못된 판정이 반복되어 신뢰가 무너진 상태. "eventually nobody trusts the results anymore" |
| pinned to | ~에 고정되다/특정 값으로 못박히다 · 결정론적 값이라 정확히 특정할 수 있을 때. "regressions can be pinned to an exact value" |
| JIT | 즉시 컴파일(Just-In-Time compilation) · 실행 중 컴파일이 초기 실행과 안정 상태 성능을 다르게 만드는 요인. "JIT compilation, caches, branch predictors, and connection pools" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.