WAL, Group Commit, and the Cost of fsync TODO
Concept
A write-ahead log (WAL) records changes sequentially to a log file before modifying the actual data pages, guaranteeing durability and crash recovery — and as a side effect, it turns random writes into sequential ones. For a commit to be truly durable, its log record has to actually reach storage, which requires an fsync (or fdatasync) call to force the OS page cache and device cache to flush — and that call is the dominant cost in commit latency. Group commit batches the log records of several transactions that arrive within a short time window and durably writes them with a single fsync, so the number of fsyncs scales with time rather than with transaction count. The result: a slight increase in any single transaction's latency, but a large gain in overall throughput — a classic latency-for-throughput batching tradeoff. Conversely, an asynchronous-commit setting skips waiting on fsync entirely for speed, at the cost of accepting the loss of some recent commits on crash.
When a database suddenly slows down, the culprit is often fsync-per-commit rather than the query plan — and conversely, a setting that looks fast might actually have quietly traded away durability.
Code & Formula
# WAL·그룹 커밋·fsync 비용 — 변경을 로그에 먼저 순차 기록하고, 여러 트랜잭션을 모아 fsync 한 번으로 묶어 내구화한다.
# 트랜잭션마다 fsync하는 방식과, 짧은 창 안의 여러 트랜잭션을 묶어 fsync 한 번으로 처리하는 그룹 커밋을 비교한다.
class WAL:
def __init__(self):
self.log = []
self.fsync_calls = 0
def append(self, record):
self.log.append(record) # 순차 기록 (아직 장치까지 내구화되지는 않음)
def flush(self):
self.fsync_calls += 1 # fsync: 실제 장치까지 강제로 내려보내는, 비용이 큰 호출
txns = [f"txn-{i}: UPDATE balance SET ..." for i in range(12)]
# 방식 1: 트랜잭션마다 즉시 fsync
wal1 = WAL()
for t in txns:
wal1.append(t)
wal1.flush()
print(f"개별 커밋: fsync 호출 {wal1.fsync_calls}회 (트랜잭션 수만큼)")
# 방식 2: 그룹 커밋 — 4개씩 모아 fsync 한 번
wal2 = WAL()
GROUP_SIZE = 4
for i in range(0, len(txns), GROUP_SIZE):
for t in txns[i:i + GROUP_SIZE]:
wal2.append(t)
wal2.flush() # 그룹 전체를 한 번의 fsync로 내구화
print(f"그룹 커밋(그룹 크기 {GROUP_SIZE}): fsync 호출 {wal2.fsync_calls}회")
def replay(wal):
return list(wal.log) # 크래시 복구: 로그를 처음부터 재생해 마지막 커밋 상태를 되살린다
print(f"\n복구 재생 결과 (마지막 3건): {replay(wal2)[-3:]}")
docs/code/algorithms/algorithms-70.py
Exercise
Run the same insert workload against local Postgres with synchronous_commit on and off, compare TPS and p99 latency, and also measure the numbers when transactions are batched together.
Practical Connection
When an indexer writes thousands of events per block, batching commits at the block level instead of committing per event is exactly the same throughput trick, for exactly the same reason.
Where it lands in Jayverse
- Devnet/indexer: batch DB commits per block (or per N blocks), not per event, for any Verex or Rabbit indexer, and make the sync/async commit tradeoff an explicit config choice rather than a silent default.
- Auditor: if the indexer ever runs with async commit, document that a crash can lose the last unflushed block's events, and define the replay-from-chain-state recovery procedure.
- gitboard: track indexer commit p99 latency as a dashboard metric, since group-commit tuning is where indexing throughput problems actually live, not the query plan.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| the culprit | 원인, 범인 · 문제를 일으킨 실제 원인을 지목할 때. "the culprit is often fsync-per-commit" |
| at the cost of | ~을 대가로, ~을 희생하며 · 이득을 얻는 대신 잃는 것을 말할 때. "at the cost of accepting the loss of some recent commits" |
| trade away | (가치 있는 것을) 맞바꿔 잃다/포기하다 · 속도를 위해 안전성 등을 포기할 때. "quietly traded away durability" |
| scales with | ~에 비례해 늘어나다 · 어떤 변수가 커질수록 함께 커지는 관계를 말할 때. "the number of fsyncs scales with time" |
| dominant cost | 가장 큰 비중을 차지하는 비용 · 전체 비용 중 압도적으로 큰 부분을 가리킬 때. "the dominant cost in commit latency" |
| turn X into Y | X를 Y로 바꿔놓다 · 어떤 기법이 성질 자체를 바꿀 때. "it turns random writes into sequential ones" |
| a latency-for-throughput tradeoff | 지연을 대가로 처리량을 얻는 트레이드오프 · 성능 최적화에서 흔한 맞교환 구조를 가리킬 때. "a classic latency-for-throughput batching tradeoff" |
| WAL | 선행 기록 로그(Write-Ahead Log) · 실제 데이터 반영 전에 변경사항을 순차 기록해 내구성을 보장하는 로그. "A write-ahead log (WAL) records changes sequentially" |
| OS | 운영체제(Operating System) · 페이지 캐시·디바이스 캐시 플러시를 담당하는 시스템 계층. "force the OS page cache and device cache to flush" |
| TPS | 초당 트랜잭션 수(Transactions Per Second) · 커밋 처리량을 측정하는 지표. "compare TPS and p99 latency" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.