RAG design — retriever quality metrics and evaluation harnesses (golden sets, property tests, regression) TODO
Concept
RAG retrieves relevant documents from an external knowledge store for a given query, then generates an answer grounded in that evidence; quality has to be evaluated separately for the retrieval stage and the generation stage. Retrieval quality is measured with metrics like recall@k (whether the correct document lands in the top k), MRR (the rank of the first correct hit), and nDCG (which weights results by rank). Generation quality is split into groundedness — whether the answer is actually supported by the retrieved evidence — and relevance to the query; an unsupported claim is a failure of the generation stage, not the retrieval stage. An evaluation harness is built from a golden dataset of fixed queries with expected evidence, property tests that check invariants like "don't answer if there's no evidence," and regression tests that rerun previously passing cases every time chunking, embeddings, or the prompt changes. Without this harness, you can only judge the effect of a parameter change by feel, which makes it impossible to tell improvement from regression.
RAG systems tend to degrade quietly — without a regression harness, you only find out that changing one chunking size broke a whole class of queries after user complaints have piled up.
Code & Formula
# RAG 설계·리트리버 품질 지표 — 토이 문서 집합에서 코사인 유사도 리트리버로
# top-k 검색을 수행하고 recall@k 로 검색 품질을 측정하는 최소 예시.
import math, re
from collections import Counter
docs = [
"raft leader election uses randomized timeouts",
"b+tree index supports range queries efficiently",
"lsm tree favors write throughput over read amplification",
"vector database uses hnsw for approximate nearest neighbor search",
"merkle tree enables logarithmic membership proofs",
]
def to_vec(text):
words = re.findall(r"[a-z]+", text.lower())
return Counter(words)
def cosine(a: Counter, b: Counter) -> float:
keys = set(a) | set(b)
dot = sum(a[k] * b[k] for k in keys)
na = math.sqrt(sum(v * v for v in a.values()))
nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
doc_vecs = [to_vec(d) for d in docs]
def retrieve(query, k=3):
qv = to_vec(query)
scored = [(cosine(qv, dv), i) for i, dv in enumerate(doc_vecs)]
scored.sort(reverse=True)
return scored[:k]
query = "how does approximate nearest neighbor search work"
top = retrieve(query, k=3)
print(f"질의: {query!r}")
for score, i in top:
print(f" top: score={score:.3f} doc[{i}]={docs[i]!r}")
# recall@k 평가: 이 질의의 정답 문서는 doc[3] (hnsw ANN) 이라고 가정
gold_idx = 3
eval_set = [("how does approximate nearest neighbor search work", 3),
("what helps range queries on sorted keys", 1),
("how are membership proofs made small", 4)]
def recall_at_k(eval_set, k):
hits = 0
for q, gold in eval_set:
retrieved_ids = [i for _, i in retrieve(q, k)]
hits += gold in retrieved_ids
return hits / len(eval_set)
print(f"\nrecall@1 = {recall_at_k(eval_set, 1):.2f}")
print(f"recall@3 = {recall_at_k(eval_set, 3):.2f}")
docs/code/algorithms/algorithms-98.py
Exercise
Build a golden set of thirty queries with expected evidence documents from your own corpus, measure recall@5 across two or three different chunk sizes, and record the results in a table.
Practical Connection
This applies directly to building an internal query tool over codebase docs or market-rules docs, and the habit of setting up metrics and regression tests first is the same engineering discipline used to manage performance regressions in a matching engine or settlement logic.
Where it lands in Jayverse
- Number: harness before search, not after. If Number's research site adds semantic search over its own readings, build the golden-set-plus-recall@5 harness first, since a search feature that degrades silently is worse than no search.
- Auditor: groundedness is the methodology check, formalized. "Don't answer without evidence" is exactly what the Auditor row already wants from a citation-checking tool over methodology documents — treat it as a property test, not a style guideline.
- CI: name the regression suite explicitly. Beyond the general discipline, add a specific golden set of matching-engine or settlement-logic scenarios with expected outcomes, rerun on every change to that code path, mirroring the RAG regression harness structure.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| grounded in | ~에 근거를 둔 · 답변이 실제 증거에 기반할 때. "an answer grounded in that evidence" |
| by feel | 감으로, 직감으로 · 정량적 근거 없이 판단할 때. "only judge the effect of a parameter change by feel" |
| degrade quietly | 조용히, 티 안 나게 성능이 떨어지다 · 문제가 서서히 나빠지는데 알아채지 못할 때. "RAG systems tend to degrade quietly" |
| pile up | 쌓이다, 누적되다 · 불만이나 문제가 점점 쌓여갈 때. "until user complaints have piled up" |
| golden set | 정답이 정해진 기준 데이터셋 · 평가 기준으로 삼는 검증된 데이터 모음. "a golden dataset of fixed queries with expected evidence" |
| regression (test) | 회귀 테스트 · 이전엔 통과했던 케이스가 계속 통과하는지 재확인하는 절차. "regression tests that rerun previously passing cases" |
| invariant | 불변 조건, 항상 성립해야 하는 규칙 · 시스템이 절대 어겨선 안 되는 규칙. "property tests that check invariants like" |
| MRR | 평균 역순위(Mean Reciprocal Rank) · 첫 번째 정답 문서가 몇 번째 순위에 나오는지로 평가하는 지표. "MRR (the rank of the first correct hit)" |
| nDCG | 정규화 누적 이득(Normalized Discounted Cumulative Gain) · 순위에 가중치를 두어 검색 품질을 평가하는 지표. "nDCG (which weights results by rank)" |
| recall@k | 상위 k개 재현율(recall at k) · 정답 문서가 상위 k개 결과 안에 포함되는 비율. "measured with metrics like recall@k (whether the correct" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.