Why
Verex's settlement pipeline (oracle lookup → settlement → payout) is literally a Saga, and its RPC/indexer calls want a circuit breaker as a baseline.
How it works
Circuit Breaker (inter-service calls): trip the circuit and fail fast once failures cross a threshold, then probe recovery half-open after a cooldown. Saga (data consistency): resolve a distributed transaction as a chain of local transactions plus compensating transactions — eventual consistency without 2PC.
Related code
"""Circuit Breaker + Saga: two microservice patterns in one toy demo.
Circuit Breaker: an inter-service call site that trips OPEN after N failures,
fails fast while open, then probes recovery via a HALF_OPEN trial after a
cooldown. Saga: a settlement pipeline of local steps, each with a compensating
rollback, undone in reverse order the moment one step fails.
"""
import time
class CircuitBreaker:
def __init__(self, fail_threshold=3, cooldown=0.05):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.failures = 0
self.state = "CLOSED"
self.opened_at = None
def call(self, fn):
if self.state == "OPEN":
if time.monotonic() - self.opened_at >= self.cooldown:
self.state = "HALF_OPEN"
else:
raise RuntimeError("circuit open: failing fast")
try:
result = fn()
except Exception:
self.failures += 1
if self.state == "HALF_OPEN" or self.failures >= self.fail_threshold:
self.state, self.opened_at = "OPEN", time.monotonic()
raise
else:
self.failures, self.state = 0, "CLOSED"
return result
def flaky_rpc(calls=[0]):
calls[0] += 1
if calls[0] <= 3:
raise ConnectionError("oracle RPC timeout")
return "oracle_price=42"
breaker = CircuitBreaker(fail_threshold=2, cooldown=0.02)
for i in range(5):
try:
print(f"attempt {i}: {breaker.call(flaky_rpc)} (state={breaker.state})")
except Exception as e:
print(f"attempt {i}: failed ({e}) (state={breaker.state})")
time.sleep(0.03)
# --- Saga: oracle lookup -> settlement -> payout, with compensations ---
ledger = []
def oracle_lookup():
ledger.append("oracle_locked")
def undo_oracle_lookup():
ledger.remove("oracle_locked")
def settle():
ledger.append("settled")
def undo_settle():
ledger.remove("settled")
def payout():
raise RuntimeError("payout provider unavailable")
steps = [(oracle_lookup, undo_oracle_lookup), (settle, undo_settle), (payout, None)]
completed = []
try:
for step, compensate in steps:
step()
completed.append(compensate)
print("saga completed:", ledger)
except Exception as e:
print(f"saga failed at step: {e} -- rolling back")
for compensate in reversed(completed):
if compensate:
compensate()
print("ledger after rollback:", ledger)
docs/code/pocs/circuit-breaker-saga.py
Where it lands in Jayverse
- Verex: write the settlement pipeline as an explicit Saga with a compensating transaction for each step. Oracle lookup, settlement and payout each need a defined rollback action, not just a sequence that assumes every step succeeds.
- Rabbit/Devnet: wrap every RPC and indexer call behind a circuit breaker with a stated half-open probe interval. Fail fast once a threshold trips and probe recovery on a schedule, rather than retrying indefinitely against a degraded RPC endpoint.
- gitboard: surface each service's breaker state — closed, open, half-open — as a visible metric. A tripped circuit is operational state worth showing on the dashboard, not something only discoverable in logs after the fact.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| fail fast | 장애를 조기에 감지해 빠르게 실패 처리하다 · "fail fast on inter-service calls" |
| probe recovery | 회복됐는지 시험적으로 확인해보다 · "probe recovery half-open" |
| half-open | (회로차단기의) 반개방 상태 · "probe recovery half-open after a cooldown" |
| compensating transaction | 앞선 작업을 되돌리는 보상 트랜잭션 · "local-transaction chains plus compensations" |
| eventual consistency | 즉시는 아니지만 결국 맞춰지는 일관성 · "eventual consistency without 2PC" |
| 2PC | 2단계 커밋(Two-Phase Commit) · 분산 트랜잭션을 원자적으로 확정하는 전통적 합의 프로토콜, Saga가 대신하는 대상. "eventual consistency without 2PC" |