Entropy, Information, and Coding TODO
Concept
The information content of an event is defined as the log of the reciprocal of its probability, and entropy is the expected value of that quantity — the average uncertainty of a distribution. When the log base is 2, the unit is bits, and for a fixed number of outcomes, entropy is maximized by the uniform distribution. The source coding theorem says the average length of a lossless code can never be shorter than the entropy, and Huffman or arithmetic coding can approach that limit arbitrarily closely. Relative entropy (KL divergence) is the extra cost paid for encoding under an incorrect assumed distribution, and mutual information is how much one variable's uncertainty is reduced by knowing another. This maximum-entropy view is the standard by which key and randomness strength is measured, in bits.
The theoretical limits of compression, the actual entropy of seeds and passwords, and judging the information content of logs or features all hinge on this — overestimating entropy leads to using randomness that feels secure but isn't.
Code & Formula
# 엔트로피·정보·코딩 — 사건 하나의 정보량 -log2(p), 그리고 분포 전체의 섀넌 엔트로피.
import math
def entropy(probs: list[float]) -> float:
return -sum(p * math.log2(p) for p in probs if p > 0)
fair_coin = [0.5, 0.5]
biased_coin = [0.9, 0.1]
fair_die = [1 / 6] * 6
for name, dist in [("공정한 동전", fair_coin), ("치우친 동전(0.9/0.1)", biased_coin), ("주사위", fair_die)]:
h = entropy(dist)
print(f"{name:20} H = {h:.4f} bits (최대 = log2({len(dist)}) = {math.log2(len(dist)):.4f})")
# 치우친 분포일수록 "다음 결과가 뭘지 이미 어느 정도 안다" → 엔트로피(불확실성)가 낮다.
Exercise
Compute the entropy from character frequencies in a text file, compare it to its gzip compression ratio, and directly calculate the entropy in bits carried by a 12-word mnemonic.
Practical Connection
The security of private keys and seeds is ultimately defined by their entropy in bits, and the fact that LMSR's cost function takes a log-sum-exp form comes from the same mathematical root — entropy and exponential families.
Where it lands in Jayverse
- Wallet: compute the actual entropy of any seed or passkey material it generates. Beyond trusting "128-bit" or "12-word" labels, Wallet should verify the real entropy in bits of whatever randomness source it uses, since overestimating entropy is exactly the failure mode this page warns about.
- Verex: treat LMSR parameter choice as an entropy-maximization problem. The market maker's cost function is already log-sum-exp; use the maximum-entropy view to justify parameter defaults rather than picking them by trial and error.
- Devnet: audit whatever randomness source seeds test accounts or session keys. A devnet's convenience randomness is a common place for weak entropy to hide; measure it in bits before assuming it is fine because it is "only a testnet."
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| reciprocal of | ~의 역수 · 어떤 수를 1로 나눈 값을 가리킬 때. "the log of the reciprocal of its probability" |
| arbitrarily closely | 한없이 가깝게, 원하는 만큼 가깝게 · 극한에 얼마든지 근접시킬 수 있을 때. "approach that limit arbitrarily closely" |
| the extra cost paid for | ~에 대해 추가로 치르는 대가 · 잘못된 가정 때문에 발생하는 손실을 말할 때. "the extra cost paid for encoding under an incorrect assumed distribution" |
| hinge on | ~에 달려 있다, 좌우되다 · 여러 문제가 결국 한 가지 개념에 근거할 때. "all hinge on this" |
| overestimate | 과대평가하다 · 실제보다 크게 잘못 판단할 때. "overestimating entropy leads to using randomness" |
| feels secure but isn't | 안전해 보이지만 실제로는 아닌 · 겉보기와 실제가 다를 때 쓰는 경고성 표현. "randomness that feels secure but isn't" |
| KL divergence | KL 발산(Kullback-Leibler divergence) · 잘못된 분포를 가정했을 때 추가로 드는 인코딩 비용을 재는 상대 엔트로피. "Relative entropy (KL divergence) is the extra cost" |
| mnemonic | 니모닉(연상 기억) 구문 · 개인키 복구용 단어 나열(시드 구문), 12단어짜리의 엔트로피를 계산하는 대상. "the entropy in bits carried by a 12-word mnemonic" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/math-50-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.