Attention — the mechanism, and why context costs quadratically
Self-attention lets every token weigh every other, which is the transformer's power and its cost — compute grows with the square of context length, the reason long context is expensive.
Not yet scoped.
Why
The PoC visualizes attention on a short sequence and the O(n^2) scaling, connecting the mechanism to why context windows and caching matter downstream.
How it works
Not yet built.
Where it lands in Jayverse
Auditor/Number: budget context length instead of "just pass more." Any LLM-based feature (a research assistant, an auditor summarizer) needs an explicit context-length budget and caching strategy documented, since attention cost grows quadratically with context.
Number: prefer retrieval/chunking over long-context dumps. When building research tooling around Number's readings, chunk and retrieve relevant history rather than passing full history into one long-context call, given the O(n^2) cost this page names.
Key expressions
Words and phrases from this page worth keeping, with the Korean meaning and the sentence they come from.
Expression
뜻 · 쓰이는 자리
weigh (every token weigh every other)
(모든 토큰이 서로) 비중을 두고 고려하다 · self-attention의 핵심 동작을 설명할 때. "every token weigh every other"
grow with the square of
~의 제곱에 비례해 커지다 · 연산량이 문맥 길이의 제곱으로 늘어난다는 뜻. "compute grows with the square of context length"