Why
A dense Transformer ties compute cost to total parameter count: every added parameter is activated for every token, so scaling the model scales latency and FLOPs in lockstep. MoE breaks that coupling. It lets a model hold far more total knowledge — more experts, each specialised on some slice of syntax or token pattern — while keeping the per-token compute close to that of a much smaller dense model, because only a handful of experts run on any given token. The failure this prevents is a purchasing or deployment mistake: judging a model's requirements by one parameter count when two different counts govern two different resources. VRAM has to hold every expert whether or not it is used this token; latency and FLOPs only care about the experts actually selected. Conflating the two is how a "13B-class" model turns out to need 47B worth of memory.
How it works
Dense FFN vs. sparse experts
In a standard decoder block, self-attention is followed by a feed-forward network (FFN) that every token passes through in full — compute is proportional to total model size. MoE replaces that single FFN with several parallel FFNs, the experts, each a complete but independent feed-forward network. A gating mechanism, the router, decides at each layer which experts a given token actually visits, so most of the network's weights sit idle for most tokens.
The router: logits, softmax, top-K
The router is a linear layer: it projects the token representation X through a weight matrix W to produce logits H(x) = X · W. A softmax turns these logits into a probability distribution G(x) over experts. In top-K routing, only the K highest-scoring experts keep a non-zero weight; every other expert's logit is masked to −∞ before the softmax, which drives its probability to zero. The layer's output is the weighted sum of the outputs of just the selected experts: Y = Σ G(x)_i · Expert_i(X) for i in the top-K set. A detail worth keeping: experts do not specialise into human-legible domains like "the medical expert" or "the code expert." Empirically they specialise in syntactic and token-level patterns — punctuation, verb forms, numeric tokens — not subject matter.
Keeping experts busy: load balancing
Left alone, training MoE models produces a "rich get richer" failure: an expert that gets a slightly better random start attracts more tokens from the router early on, gets more gradient updates, becomes more attractive to the router, and the rest starve. The fix is an auxiliary load-balancing loss added to the training objective, computed from the coefficient of variation (standard deviation over mean) of routing probabilities across experts — a high coefficient of variation means routing is uneven, and the loss penalises that. It's a governance mechanism baked into training, not a hyperparameter you tune once and forget.
Expert capacity and token dropping
Because GPU memory allocation has to be fixed ahead of time, each expert gets a hard capacity limit: C = (tokens / experts) × capacity factor. If more tokens route to an expert than its capacity allows in a batch, the overflow tokens either fall through to a secondary expert or bypass the FFN entirely via a residual connection — they get dropped from expert computation, not from the sequence. This is a real, if usually small, source of representational noise, and it is one reason routing entropy is worth monitoring during training.
Sparse vs. active parameters
This is the number that matters for deployment. "Sparse parameters" (or total parameters) is everything loaded into memory — every expert, used or not; for Mixtral 8x7B that's roughly 47B. "Active parameters" is what actually runs per token — for Mixtral, about 13B, since only 2 of 8 experts fire per layer. VRAM sizing is governed by the sparse count; inference latency and FLOPs are governed by the active count. The same architecture also extends past language: Soft-MoE applies the same idea to Vision Transformers, routing image patches to specialised encoders instead of discrete tokens.
Where it lands in Jayverse
- alice / local AI: choose models by two numbers, not one. When picking a local model for the Mac (the #112/#116 thread), check both sparse (total, drives whether it fits in unified memory) and active (drives generation speed) parameter counts before downloading — a model advertised as fast because of a low active count can still be too large to load.
- Auditor: an auxiliary loss is an encoded rule. The load-balancing loss is training-time enforcement of a fairness invariant across experts, the same shape as #67's "an invariant is a stop" — a rule that fires automatically rather than one a reviewer has to remember to check. Worth naming as a pattern the Auditor row should recognise even outside MoE: penalty terms that keep a system from drifting into an unbalanced state.
- Knowledge Notes (Theory cross-link): softmax and coefficient of variation. Both are reusable primitives — softmax turns any score vector into a probability distribution (also used in attention), and coefficient of variation is a general dispersion metric for "how uneven is this allocation," useful anywhere load or resource distribution needs a single number.
- Eng: a three-sentence interview answer. MoE replaces one big feed-forward layer with many small expert FFNs and a router that activates only the top-K per token; a router computes
softmax(X·W)and masks the rest to −∞; the payoff is that a model can hold, say, 47B parameters in memory while only computing with about 13B per token, which is why Mixtral 8x7B is described as "a 47B model that runs like a 13B one."
Verified and unverified
Verified on 2026-09-21: Maarten Grootendorst co-authored Hands-On Large Language Models (O'Reilly, 2024) and writes a visual-guide blog series on ML topics. Mixtral 8x7B (Mistral AI, released December 2023) is a real sparse MoE model, documented at 46.7B total parameters and 12.9B active parameters per token (2 of 8 experts per layer) — close to the video's rounded "47B / 13B." The Switch Transformer (Fedus, Zoph, Shazeer, 2021) is the paper that introduced the capacity-factor formulation and an auxiliary load-balancing loss in this form; Shazeer et al. (2017) introduced sparsely-gated Mixture-of-Experts layers for neural networks, the lineage this design descends from. DeepSeek-V3 (671B total parameters, 37B active) is a current, real example of the same sparse/active split at much larger scale. GPT-4 being an MoE model is a persistent rumor in the field, not a confirmed architectural fact, and should be treated as such rather than cited as a MoE example.
Taken from the summary and not independently checked: the video's own framing and pacing, the exact claim that "experts specialise in syntactic rather than semantic patterns" as stated (this matches published MoE interpretability findings in general but was not re-verified against this specific video's sources), and the video's discussion of Soft-MoE for vision beyond the general fact that Soft-MoE exists as a published architecture (Puigcerver et al., 2023).
Sources: YouTube — Maarten Grootendorst, "A Visual Guide to Mixture of Experts (MoE) in LLMs" · Fedus, Zoph, Shazeer, "Switch Transformers," 2021 · Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer," 2017 · Mistral AI, Mixtral 8x7B model card · related items: Tech #112 (MLX local AI), Tech #116 (local AI weights and quantization), Tech #63 (governed workflows: route, then act), Tech #96 (Homa — SRPT and priority queues, routing decisions by size), #67 (an invariant is a stop).
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| Mixture of Experts (MoE) | 여러 전문화된 서브 네트워크와 라우터로 구성된 아키텍처 · 이 항목 전체의 주제. "how MoE replaces a Transformer's dense feed-forward layer" |
| dense (model) | 밀집(모든 파라미터가 매 토큰에 활성화되는 모델) · sparse와 대비되는 기준 아키텍처. "A dense Transformer ties compute cost to total parameter count" |
| feed-forward network (FFN) | 피드포워드 네트워크(트랜스포머 블록에서 어텐션 다음에 오는 층) · MoE가 대체하는 대상. "a feed-forward network (FFN) that every token passes through" |
| router / gating (mechanism) | 라우터 / 게이팅(토큰마다 어느 전문가를 쓸지 정하는 장치) · MoE의 핵심 구성요소. "A gating mechanism, the router, decides" |
| logits | 로짓(softmax 이전의 원점수) · 분류·라우팅에서 공통으로 쓰는 용어. "producing logits H(x) = X · W" |
| softmax | 점수 벡터를 확률 분포로 바꾸는 함수 · 라우팅과 어텐션 모두에 쓰이는 재사용 가능한 도구. "A softmax turns these logits into a probability distribution" |
| top-K routing | 상위 K개만 선택하는 라우팅 방식 · 나머지는 마스킹되어 계산에서 빠짐. "In top-K routing, only the K highest-scoring experts keep a non-zero weight" |
| masked to −∞ | 음의 무한대로 마스킹(softmax 전에 로짓을 −∞로 만들어 확률을 0으로 만드는 방법) · top-K 구현의 표준 트릭. "every other expert's logit is masked to −∞" |
| load-balancing loss / auxiliary loss | 로드 밸런싱 손실 / 보조 손실(전문가 쏠림을 막는 학습 시 벌점 항) · Auditor 비유의 핵심. "an auxiliary load-balancing loss added to the training objective" |
| coefficient of variation | 변동 계수(표준편차/평균, 고르지 않음을 재는 지표) · 로드 밸런싱 손실 계산의 기반. "computed from the coefficient of variation... of routing probabilities" |
| rich get richer | 부익부(초기에 앞선 쪽이 계속 더 유리해지는 현상) · 로드 밸런싱이 막으려는 실패 모드를 부르는 관용구. "a 'rich get richer' failure" |
| expert capacity | 전문가 용량(한 배치에서 전문가가 받을 수 있는 토큰 수의 상한) · GPU 메모리를 고정 배정하기 위한 제약. "each expert gets a hard capacity limit" |
| capacity factor | 용량 계수(용량 공식의 배수 항) · Switch Transformer가 도입한 튜닝 파라미터. "C = (tokens / experts) × capacity factor" |
| token dropping | 토큰 드롭(용량 초과 시 토큰이 전문가 계산에서 빠지는 것) · 용량 제한의 직접적 결과. "overflow tokens either fall through... or bypass the FFN" |
| residual bypass | 잔차 우회(FFN을 건너뛰고 잔차 연결만 통과시키는 경로) · 드롭된 토큰의 대안 경로. "bypass the FFN entirely via a residual connection" |
| sparse parameters | sparse 파라미터(모델에 로드된 전체 파라미터 수, 쓰이든 안 쓰이든) · VRAM 요구량을 정하는 숫자. "'Sparse parameters'... is everything loaded into memory" |
| active parameters | active 파라미터(토큰 하나당 실제로 실행되는 파라미터 수) · 지연 시간과 FLOPs를 정하는 숫자. "'Active parameters' is what actually runs per token" |
| VRAM | 그래픽 메모리(모델 가중치를 올려두는 GPU 메모리) · sparse 파라미터 수가 정하는 제약. "VRAM sizing is governed by the sparse count" |
| FLOPs | 초당 부동소수점 연산 수(연산량의 표준 단위) · active 파라미터 수가 정하는 비용. "inference latency and FLOPs are governed by the active count" |
| routing entropy | 라우팅 엔트로피(라우터가 전문가들에 얼마나 고르게 분산해서 배정하는지 재는 지표) · 학습 중 모니터링 대상. "one reason routing entropy is worth monitoring" |
| Soft-MoE | 소프트 MoE(비전 트랜스포머용 MoE 변형, 이미지 패치를 라우팅) · 텍스트 밖으로의 확장 사례. "Soft-MoE applies the same idea to Vision Transformers" |