Distributed Tracing and Sampling Strategy TODO
Concept
Distributed tracing is an observability technique that groups the path a single request takes across multiple services into a unit called a trace, recording each unit of work as a span with a parent-child relationship. For spans from different processes to be grouped into the same trace, the trace ID and parent span ID must be propagated through request headers, and this propagation convention has to be standardized for heterogeneous systems to link up. Storing every request is too costly, so sampling is necessary: head-based sampling makes a probabilistic decision at the start of a trace and propagates that decision downstream so the trace doesn't get fragmented. Tail-based sampling waits until the trace finishes and looks at the whole thing before deciding, so it can selectively keep error or slow requests — but it requires buffering until completion, which costs memory and structural complexity. Either way, the sampling rate has to be recorded alongside the data so aggregate metrics can be reconstructed without bias.
Failures tend to happen in the tail rather than the average, and pure random sampling alone means the slow and failed request traces you actually need for post-mortem analysis end up missing.
Code & Formula
# 분산 트레이싱과 샘플링 전략 — trace_id 전파로 스팬을 하나의 트레이스로 묶고,
# 헤드 기반 샘플링(시작 시 결정)과 테일 기반 샘플링(완료 후 에러/느린 트레이스만 선별)을 비교한다.
import random
import uuid
random.seed(3)
class Span:
def __init__(self, trace_id, name, duration_ms, error=False, parent=None):
self.trace_id = trace_id
self.span_id = uuid.uuid4().hex[:8]
self.parent = parent.span_id if parent else None
self.name = name
self.duration_ms = duration_ms
self.error = error
def make_trace(slow=False, error=False):
trace_id = uuid.uuid4().hex[:8]
root = Span(trace_id, "api.handle", random.uniform(5, 15))
child = Span(trace_id, "match.execute", random.uniform(3, 8), parent=root)
tail = Span(trace_id, "db.settle", 200 if slow else random.uniform(2, 6),
error=error, parent=child)
return [root, child, tail]
def head_sample_decision(trace_id, rate=0.1):
"""트레이스 시작 시점에 확률적으로 결정하고, 이 결정을 모든 자식 스팬에 전파한다."""
return (int(trace_id, 16) % 1000) < rate * 1000
def tail_sample_decision(spans, latency_threshold_ms=100):
"""트레이스가 끝난 뒤 전체를 보고, 에러거나 느리면 남긴다."""
total = sum(s.duration_ms for s in spans)
has_error = any(s.error for s in spans)
return has_error or total > latency_threshold_ms
traces = (
[make_trace() for _ in range(20)]
+ [make_trace(slow=True) for _ in range(2)]
+ [make_trace(error=True) for _ in range(2)]
)
head_kept = sum(1 for t in traces if head_sample_decision(t[0].trace_id))
tail_kept_important = sum(
1 for t in traces
if tail_sample_decision(t) and (any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
)
important_total = sum(1 for t in traces if any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
head_kept_important = sum(
1 for t in traces
if head_sample_decision(t[0].trace_id) and (any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
)
print(f"전체 트레이스: {len(traces)}, 그중 중요한(에러/느림) 트레이스: {important_total}")
print(f"헤드 샘플링(10%)으로 보존된 트레이스: {head_kept}, 그중 중요한 것: {head_kept_important}")
print(f"테일 샘플링으로 보존된 중요한 트레이스: {tail_kept_important} / {important_total}")
print("-> 테일 샘플링은 저장량을 줄이면서도 중요한 트레이스를 놓치지 않는다.")
docs/code/algorithms/algorithms-48.py
Exercise
Set up two or more services, propagate trace context with OpenTelemetry, and check directly — with the head sampling rate turned down — whether a given trace cuts off partway through.
Practical Connection
The flow of a user order from the API through the matching engine to transaction submission and receipt confirmation has long, asynchronous spans because of chain confirmation latency, so recording the transaction hash as a span attribute and preserving failed traces via tail sampling is critical for incident analysis.
Where it lands in Jayverse
- Verex: use tail-based sampling keyed on transaction status and latency, not random sampling. Keep 100% of traces where a chain-confirmation-bound order fails or runs long, so post-mortems never lose exactly the traces they need.
- gitboard: record the sampling rate alongside trace metrics on the dashboard. Per this PoC's note, aggregate latency numbers need the sampling rate attached or they're silently biased.
- CI: add a trace-context propagation check across the API-to-matching-engine-to-settlement path. A broken trace/parent-span header should fail CI, not surface as a fragmented trace during an incident.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| propagate | (신호·값을) 전파시키다 · "must be propagated through request headers" |
| fragmented | 조각나다, 중간에 끊기다 · "the trace doesn't get fragmented" |
| buffering | 완료될 때까지 임시로 저장해 두는 것 · "it requires buffering until completion" |
| post-mortem | 사후 분석, 사후 부검 · "traces you actually need for post-mortem analysis" |
| cut off partway through | 도중에 끊기다 · "whether a given trace cuts off partway through" |
| reconstruct without bias | 편향 없이 재구성하다 · "aggregate metrics can be reconstructed without bias" |
| asynchronous | 비동기적인 · "long, asynchronous spans because of chain confirmation latency" |
| OpenTelemetry | 오픈텔레메트리(OpenTelemetry) · 분산 추적·관측성을 위한 오픈소스 계측 표준, trace context 전파에 사용. "propagate trace context with OpenTelemetry" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.