Designing an Indexing Pipeline — Replayability TODO
Concept
An indexing pipeline reads a chain's blocks, logs, and traces and transforms them into a queryable form; replayability is the property that running the whole thing again from the raw source data, at any time, reaches the exact same result. Achieving this requires the transformation logic to be a deterministic pure function — inputs that change on re-run, like the current time, randomness, or external API responses, must never enter the transformation. Progress should be represented as a cursor, such as (block number, log index), and writes must be idempotent, so that restarting after an interruption picks back up without duplication or gaps. Chains undergo reorgs, so results for unfinalized ranges must be stored together with their block hash so they can be rolled back and reprocessed; only ranges past finality are treated as immutable. In the end, replayability is a design that buys an operational freedom: the ability to throw away the data and rebuild it whenever the schema or the logic changes.
Indexers get their bugs fixed and their schemas changed often; without replayability, you end up hand-patching historical data — a special kind of hell. Skip reorg handling and wrong data sits there silently.
Code & Formula
# 인덱싱 파이프라인 설계 — 재생 가능성(replayability) — 동일 입력을 두 번 인덱싱해도 같은 결과가 나오고, reorg 시 롤백·재처리된다.
# 커서(블록번호, 로그인덱스)로 진행 상태를 추적하고 쓰기를 멱등하게 만들어, 중단 후 재시작·재구성이 안전하도록 한다.
def index_blocks(blocks, table=None):
table = table if table is not None else {}
for b in blocks:
for log in b["logs"]:
key = (b["number"], log["index"]) # 커서 = (블록번호, 로그인덱스) → 자연스러운 멱등 키
table[key] = {"block_hash": b["hash"], "value": log["value"]}
return table
def reorg_rollback(table, from_block):
# 확정되지 않은 구간을 되돌린다: from_block 이상인 항목을 모두 제거
stale_keys = [k for k in table if k[0] >= from_block]
for k in stale_keys:
del table[k]
return len(stale_keys)
chain_v1 = [
{"number": 0, "hash": "0xa0", "logs": [{"index": 0, "value": 100}]},
{"number": 1, "hash": "0xa1", "logs": [{"index": 0, "value": 200}]},
{"number": 2, "hash": "0xa2", "logs": [{"index": 0, "value": 300}]},
]
table_a = index_blocks(chain_v1)
table_b = index_blocks(chain_v1) # 같은 구간을 처음부터 다시 인덱싱
print("두 번 인덱싱한 결과가 동일한가:", table_a == table_b)
# 블록 2가 reorg로 다른 해시·값으로 교체되었다고 가정
removed = reorg_rollback(table_a, from_block=2)
chain_v2_block2 = {"number": 2, "hash": "0xb2-reorged", "logs": [{"index": 0, "value": 999}]}
index_blocks([chain_v2_block2], table=table_a)
print(f"reorg 롤백: 블록 2 이상 {removed}개 항목 제거 후 재처리")
print("reorg 이후 블록 2 상태:", table_a[(2, 0)])
print("블록 0,1은 그대로 유지:", table_a[(0, 0)], table_a[(1, 0)])
docs/code/algorithms/algorithms-75.py
Exercise
Build a test that indexes an arbitrary block range twice and checks the final tables are byte-for-byte identical, then swap in different hashes for the last few blocks and verify that reorg rollback actually works.
Practical Connection
If Verex indexes order, fill, and settlement events to show users their positions and P&L, without replayability, every time the calculation logic gets fixed, past balances are left wrong.
Where it lands in Jayverse
- Verex: add a byte-for-byte replay test to CI. Beyond just having replayable indexing, CI should index a fixed block range twice and diff the resulting tables, catching any non-deterministic transform before it reaches production P&L numbers.
- gitboard: store its own metrics with a reorg-safe cursor. If gitboard reads on-chain events for dashboards, its ingestion needs the same (block number, log index) cursor and unfinalized-range rollback Verex's indexer needs, or a reorg silently corrupts a dashboard number nobody double-checks.
- Devnet: expose a reliable finality signal for any indexer built on top. Every service indexing Devnet needs to know which ranges are still reorg-able, so Devnet's node setup should make block finality status a first-class, queryable fact.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| hand-patch | 손으로 하나하나 고치다 · 자동화 없이 수작업으로 데이터를 수정할 때. "you end up hand-patching historical data" |
| a special kind of hell | 특히나 고통스러운 상황 · 특정 작업이 유독 괴로움을 강조할 때. "a special kind of hell" |
| sit there silently | 조용히 방치된 채로 있다 · 오류가 드러나지 않고 남아있을 때. "wrong data sits there silently" |
| roll back | 되돌리다, 이전 상태로 복구하다 · 재구성이나 취소가 필요할 때. "so they can be rolled back and reprocessed" |
| byte-for-byte | 한 바이트도 다르지 않게, 완전히 동일하게 · 완벽히 일치하는지 검증할 때. "the final tables are byte-for-byte identical" |
| buy (freedom) | 대가를 치르고 확보하다 · 설계로 얻어내는 이점을 말할 때. "replayability is a design that buys an operational freedom" |
| left wrong | 잘못된 채로 방치되다 · 수정되지 않고 오류 상태로 남을 때. "past balances are left wrong" |
| P&L | 손익(Profit and Loss) · 사용자의 포지션과 손익을 계산해 보여주는 지표, 재구성 시 정확해야 함. "to show users their positions and P&L, without replayability" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.