Observing Production with eBPF TODO
Concept
eBPF is a sandboxed VM that runs safely inside the kernel. When a user-written program is loaded into the kernel, a verifier statically checks termination and memory-access safety; only programs that pass are JIT-compiled and attached to kprobes, uprobes, tracepoints, perf events, network hooks, and the like. The program and userspace exchange data through shared structures called maps (hash maps, arrays, ring buffers, etc.). The key benefit is being able to observe a running system's internal events without writing a new kernel module or restarting/recompiling the application. Higher-level tools like bpftrace and BCC wrap this whole process down to a one-line script.
It's nearly the only way to trace a latency spike or a specific syscall bottleneck in production that won't reproduce elsewhere, without a code change or redeploy, and it reaches layers that application logs never touch.
Code & Formula
# eBPF로 프로덕션 관측 — 커널 프로브가 이벤트를 map에 쌓고, 유저스페이스가 읽어 히스토그램을 낸다.
# 실제 커널 훅 대신 "probe가 이벤트를 map에 기록한다"는 구조만 순수 파이썬으로 흉내낸다.
import random
from collections import defaultdict
random.seed(1)
class EbpfMap:
"""kprobe/tracepoint가 기록하는 공유 map(dict)을 흉내"""
def __init__(self):
self.hist = defaultdict(int) # latency bucket(us, 로그 스케일) -> count
def record(self, latency_us):
bucket = 1
while bucket * 2 <= latency_us:
bucket *= 2
self.hist[bucket] += 1
def bpftrace_like_probe(events, ebpf_map):
"""실제로는 커널이 syscall 진입/종료 시각차를 계산해 넣어주는 부분을 시뮬레이션"""
for latency_us in events:
ebpf_map.record(latency_us)
# 유휴 상태: 대부분 짧은 지연
idle_events = [max(1, int(random.gauss(50, 15))) for _ in range(200)]
# 부하 상태: 디스크 I/O 경합으로 꼬리가 길어짐
loaded_events = [max(1, int(random.gauss(50, 15))) for _ in range(200)]
loaded_events += [int(random.gauss(4000, 800)) for _ in range(20)] # I/O 경합 스파이크
idle_map = EbpfMap()
loaded_map = EbpfMap()
bpftrace_like_probe(idle_events, idle_map)
bpftrace_like_probe(loaded_events, loaded_map)
def print_hist(name, m):
print(f"--- {name} (syscall latency, us, log2 bucket) ---")
for bucket in sorted(m.hist):
print(f" <= {bucket:5d}us : {'#' * m.hist[bucket]} ({m.hist[bucket]})")
print_hist("idle", idle_map)
print_hist("loaded", loaded_map)
print("\n부하 상태에서 4000us대 버킷이 새로 등장 -> 재배포 없이 I/O 경합 구간을 특정.")
docs/code/algorithms/algorithms-47.py
Exercise
Use a one-line bpftrace script to pull a histogram of disk I/O latency or syscall call counts for a specific process, and compare it under load versus idle.
Practical Connection
When sync stalls on a blockchain node, or tail latency in Verex's backend API, come from the filesystem or network stack rather than application code, this lets you narrow down the cause with evidence instead of guessing.
Where it lands in Jayverse
- Verex: reach for a one-line bpftrace script before adding app instrumentation. When backend API tail latency shows up in gitboard but not in application logs, run a one-line bpftrace script against the Cloud Run/GKE host first, before adding more instrumentation code.
- Devnet: use eBPF to isolate a stalling Anvil node's bottleneck. When the hosted Anvil node stalls syncing from Sepolia, use eBPF to tell filesystem or network-stack bottlenecks apart from RPC-layer ones, without redeploying the node.
- gitboard: point the runbook at eBPF for spikes that won't reproduce. Add eBPF as the named escalation step in gitboard's runbook for a latency spike that won't reproduce elsewhere, no code change or redeploy needed to capture it.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| sandboxed | 샌드박스 처리된, 격리된 · 커널 안에서 안전하게 격리되어 실행되는 환경을 가리킬 때. "a sandboxed VM that runs safely inside the kernel" |
| statically check | 정적으로 검사하다 · 실행 전에 코드를 분석해 안전성을 검증할 때. "a verifier statically checks termination and memory-access safety" |
| attach to | ~에 연결되다, 붙다 · 프로그램이 특정 커널 지점(훅)에 걸릴 때. "attached to kprobes, uprobes, tracepoints, perf events" |
| wrap ... down to | ~로 간단히 축약해주다 · 복잡한 과정을 한 줄 스크립트 수준으로 압축할 때. "wrap this whole process down to a one-line script" |
| narrow down | 범위를 좁혀가다 · 원인을 후보군에서 하나씩 줄여나갈 때. "this lets you narrow down the cause with evidence" |
| reaches layers that ... never touch | ~가 닿지 못하는 층까지 도달하다 · 애플리케이션 로그로는 볼 수 없는 곳까지 관찰할 때. "it reaches layers that application logs never touch" |
| tail latency | 테일 레이턴시(상위 퍼센타일의 느린 응답) · 대부분은 빠르지만 일부 요청이 느릴 때 쓰는 성능 용어. "tail latency in Verex's backend API" |
| eBPF | 확장 버클리 패킷 필터(extended Berkeley Packet Filter) · 커널 안에서 안전하게 실행되는 관찰·트레이싱 기술, 이 카드의 주제. "eBPF is a sandboxed VM that runs safely inside the kernel" |
| bpftrace | eBPF를 한 줄 스크립트로 쓸 수 있게 해주는 고수준 트레이싱 도구 · 프로덕션에서 지연시간을 즉석에서 관찰할 때 쓰는 실제 도구. "Use a one-line bpftrace script to pull a histogram of disk I/O latency" |
| BCC | BPF 컴파일러 컬렉션(BPF Compiler Collection) · eBPF 프로그램 작성을 감싸주는 고수준 툴킷. "Higher-level tools like bpftrace and BCC wrap this whole process" |
| JIT | 즉시 컴파일(Just-In-Time compilation) · 검증을 통과한 eBPF 프로그램이 커널 훅에 붙기 전 거치는 컴파일 단계. "only programs that pass are JIT-compiled and attached to kprobes" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.