Transformer compute structure, KV cache, and inference serving (continuous batching, PagedAttention), plus quantization/distillation/LoRA trade-offs TODO
Concept
Transformer inference splits into a prefill stage, which processes the whole prompt at once, and a decode stage, which generates tokens one at a time — the two stages bottleneck on different things. During decode, the key/value vectors of past tokens are kept in a KV cache so they don't need to be recomputed every step; this cache grows with sequence length and the number of concurrent requests, so GPU memory itself becomes the concurrency limit. PagedAttention manages the KV cache in fixed-size pages instead of one large contiguous block, eliminating fragmentation and over-reservation so more requests can be served concurrently. Continuous batching doesn't fix the batch composition per request — at every token-generation step it drops finished requests and admits new ones, cutting GPU idle time. On the model side, quantization lowers weights and activations to lower precision, distillation trains a small model to mimic a large model's outputs, and LoRA freezes the original weights and trains/swaps only small low-rank matrices — each has a different trade-off across quality, memory, and serving flexibility.
LLM serving cost and latency are shaped far more by KV cache management and batching strategy than by model choice, so without understanding this structure, the only lever you have left is buying more GPUs.
Code & Formula
# 트랜스포머 계산 구조·KV 캐시 — 단일 헤드 self-attention을 numpy로 구현하고,
# 자기회귀 decode 시 K/V를 매 스텝 재계산하지 않고 캐시에 append 만 하는 것을 시연.
import numpy as np
np.random.seed(0)
d_model, d_k = 8, 4
Wq = np.random.randn(d_model, d_k) * 0.1
Wk = np.random.randn(d_model, d_k) * 0.1
Wv = np.random.randn(d_model, d_k) * 0.1
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def attention(q, K, V):
scores = (q @ K.T) / np.sqrt(d_k) # (1, seq_len)
weights = softmax(scores)
return weights @ V, weights # (1, d_k), (1, seq_len)
# prefill: 초기 프롬프트 3토큰을 한 번에 처리하며 K/V 캐시를 채운다
tokens = np.random.randn(3, d_model) * 0.1
K_cache = tokens @ Wk
V_cache = tokens @ Wv
print("prefill 후 KV 캐시 길이:", len(K_cache))
# decode: 새 토큰마다 Q만 새로 계산하고, K/V는 "재계산 없이" 캐시에 append 만 한다
for step in range(3):
new_token = np.random.randn(1, d_model) * 0.1
q = new_token @ Wq
k_new, v_new = new_token @ Wk, new_token @ Wv
K_cache = np.vstack([K_cache, k_new]) # O(1) append, 과거 K 재계산 없음
V_cache = np.vstack([V_cache, v_new])
out, weights = attention(q, K_cache, V_cache)
print(f"decode step {step}: KV 캐시 길이={len(K_cache)}, "
f"attention weights={np.round(weights[0], 3).tolist()}")
print("\n캐시가 없다면 매 decode 스텝마다 전체 시퀀스의 K/V를 O(n) 재계산해야 한다.")
docs/code/algorithms/algorithms-97.py
Exercise
Serving the same model, ramp up concurrent request count and prompt length, measure throughput and p95 latency, and find the point where the estimated KV cache memory hits its limit and requests start queuing.
Practical Connection
The direct blockchain connection is weak, but as a low-latency service that caches and batches per-request state, this calls for the same mindset as batching and memory-budget design in an order-book engine or an indexer.
Where it lands in Jayverse
- Verex: admit/drop CLOB matching requests the way continuous batching admits/drops decode requests. At each matching cycle, drop settled orders and admit new ones instead of holding a fixed batch composition, to cut idle time under load the same way inference serving does.
- Devnet/Rabbit: if Rabbit's agent reasoning is ever self-hosted, size GPU memory against concurrent session KV cache, not model size alone. Concurrency, not parameter count, is what runs out first — budget Devnet's infra plan around that.
- Number: check jayverse-number's indexer for the same fragmentation PagedAttention fixes. Fixed-size pages instead of one contiguous block is a general fix for growing per-request state — worth a look at how the indexer allocates memory for open queries.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| bottleneck on | ~에서 병목이 걸리다 · 두 단계가 서로 다른 자원 제약에 막힐 때. "the two stages bottleneck on different things" |
| grow with | ~에 비례해서 커지다 · 캐시 크기가 시퀀스 길이에 따라 늘어날 때. "this cache grows with sequence length" |
| cut ... idle time | ~의 유휴 시간을 줄이다 · GPU가 노는 시간을 줄여 효율을 높일 때. "cutting GPU idle time" |
| mimic | 흉내 내다, 모방하다 · 작은 모델이 큰 모델의 출력을 따라 하도록 학습할 때. "trains a small model to mimic a large model's outputs" |
| trade-off across | ~ 사이의 트레이드오프(상충 관계) · 여러 기준(품질, 메모리 등) 사이에서 하나를 얻으면 다른 걸 잃을 때. "a different trade-off across quality, memory, and serving flexibility" |
| the only lever you have left | 남은 유일한 수단 · 다른 최적화 여지가 없어 마지막으로 쓸 수 있는 방법. "the only lever you have left is buying more GPUs" |
| shaped far more by ... than by | ~보다 ~에 의해 훨씬 더 좌우되는 · 어떤 결과의 주된 원인이 통념과 다를 때. "shaped far more by KV cache management" |
| KV cache | 키/값 캐시(Key-Value cache) · 디코드 단계에서 이전 토큰들의 키·값 벡터를 저장해 매 스텝 재계산을 피하는 캐시, 크기가 GPU 메모리 한계를 좌우함. "the key/value vectors of past tokens are kept in a KV cache" |
| PagedAttention | 페이지 단위로 KV 캐시를 관리하는 기법 · 하나의 연속된 블록 대신 고정 크기 페이지로 캐시를 나눠 메모리 단편화·과할당을 없애는 서빙 기법. "PagedAttention manages the KV cache in fixed-size pages" |
| LoRA | 저랭크 적응(Low-Rank Adaptation) · 원래 가중치는 고정하고 작은 저랭크 행렬만 학습·교체해 파인튜닝 비용을 줄이는 기법. "LoRA freezes the original weights and trains/swaps only small low-rank matrices" |
| continuous batching | 연속 배칭 · 매 토큰 생성 스텝마다 끝난 요청을 빼고 새 요청을 받아들여 GPU 유휴시간을 줄이는 서빙 전략. "Continuous batching doesn't fix the batch composition per request" |
| p95 latency | 95번째 백분위 지연시간 · 전체 요청 중 최악에 가까운 5%를 제외한 지연시간 기준, 서빙 성능을 재는 표준 지표. "measure throughput and p95 latency" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.