Why
Getting local inference to work is really four separate decisions, not one "can my machine run it" lookup, and treating them separately is what makes the numbers below usable. Skip quantization and a 70B model needs roughly 140GB just for weights at 16-bit; pick a reasonable quantization and the same model fits under 40GB with modest quality loss. Skip the memory check and the process either refuses to load or pages to disk, no matter how good the engine is. Skip the engine layer entirely and there's nothing to load the weights or run the math at all. A leaderboard score answers "how smart," not "will this run here" — the four levers (weights, quantization, engine, memory) are what actually answer the second question, and they can be tuned independently of each other.
How it works
Weights and parameters set both intelligence and cost
A local model is, physically, a file: a few gigabytes to a few hundred, holding tens to hundreds of billions of learned floating-point parameters (02:37, 02:48). The parameter count in a model's name is a direct proxy for both capability and cost — roughly, parameter count times bytes-per-parameter sets file size and memory footprint, so doubling the parameter count roughly doubles both (03:23). That is why "how smart a model can I run" and "how big a model can I run" are the same question asked two ways.
Quantization and GGUF compress the file, not the idea
Quantization lowers each weight's numeric precision — 16-bit down to 8-bit (Q8) or 4-bit (Q4) — trading a small, largely invisible quality loss for a much smaller footprint (03:50, 04:03); Q4 roughly quarters the 16-bit size. GGUF is the file format llama.cpp defined to package a quantized model together with its metadata (tokenizer, architecture) into one portable file, and it has become the default format for locally distributed models (04:21).
The inference engine does the actual work
The engine loads GGUF weights into memory and, for each new token, runs a forward pass — the matrix multiplications between the input and the weight matrices that produce the next token's probability distribution (04:55, 05:06, 05:12). llama.cpp is the reference implementation and underlies nearly every local tool, so "which engine" and "which app" usually turn out to be the same question wearing different UI.
Memory is the gate; bandwidth is the speed
The first hard constraint is whether the weights, plus a working buffer, fit in memory at all (05:28, 06:22). On PC or Linux this means GPU VRAM specifically — the model has to fit in the card's own memory for GPU-speed inference (05:39). On a Mac it means the unified memory pool, since Apple Silicon's CPU and GPU share one physical memory bank rather than the GPU getting its own smaller VRAM allotment (05:54). As a rule of thumb from the video: 8GB fits a 3-4B model, 16GB fits 7-8B, and 32GB fits 14-30B, which the video calls the sweet spot where perceived intelligence jumps the most per gigabyte spent (06:27, 06:43, 09:09).
Once a model fits, capacity and speed pull in opposite directions across the two platforms. Apple Silicon's unified memory scales past 128GB, enough to load a 70B+ model a 24GB card never could, but memory bandwidth is the bottleneck, so tokens/second run comparatively slow (07:36, 07:41, 08:39). An Nvidia RTX 4090 tops out at 24GB VRAM — too little for the largest models — but its bandwidth is far higher, so on any model that fits both machines the 4090 runs it roughly 2-3x faster, on the order of 100-200 tokens/s in the video's numbers (08:14, 08:28). None of this is fixed once and forgotten: a longer context window adds a KV cache on top of the weights, and that cache grows with context length independent of model size, so a long-running conversation can push a model that fit at first out of memory later (07:05, 14:13).
Four ways to actually run one
LM Studio is the GUI path: search a model, pick a quantization level (it flags which ones your hardware can realistically handle), one-click download, chat locally (11:13, 11:18, 12:12, 13:02); it also runs a local HTTP server so your own code can hit it as a development endpoint (14:57). Ollama is the CLI path — ollama pull <model> then ollama run <model> (11:23, 15:52, 17:19, 17:28) — with a background daemon serving an OpenAI-compatible REST API on port 11434 by default (18:19, 18:51). Docker Model Runner, a Docker Desktop feature, treats a model like an image and a container: pull it, run it, and reference it as a service dependency in a Dockerfile or compose file (11:34, 19:02, 19:25, 20:03, 20:09). And the code path — libraries such as llama-cpp-python — loads weights directly and generates tokens with no app in between: minimum overhead, full control over sampling and prompt handling (11:45, 21:05, 21:10, 23:45).
Where it lands in Jayverse
- Dark Horse: which local model fits jay's Mac to query the alice corpus. Size it with the memory-then-bandwidth rule above — check available unified memory against the 8/16/32GB tiers, pick the largest quantization that clears it, then check whether the resulting tokens/s is tolerable for interactive queries — and write the result as a one-page decision table, not a single pick.
- Rabbit: key material and mandates stay behind a local model, full stop. Session keys and EIP-7702/7715 mandates should only ever be summarized or classified by a model running on jay's own hardware (Ollama or llama.cpp, not a hosted API); "no data leaves the machine" (01:48) is the actual requirement here, not a nice-to-have.
- CI: Ollama in Docker as a PR-summary service, Docker Model Runner for the compose file. A small quantized model serving on port 11434 inside the CI container can generate PR summaries or flag odd commit messages without an external API call; Docker Model Runner is the natural way to declare that model as a service dependency in the compose file instead of scripting a manual
ollama pull. - Theory: quantization as lossy compression, decode as bandwidth-bound. Q4/Q8 quantization is a concrete example for the lossy-compression trade-off entries in Theory, and the RTX-4090-versus-Apple-Silicon gap is a clean case of memory bandwidth, not compute, setting decode speed — the same shape as other bandwidth-bound systems already noted there.
- Eng: interview prep — VRAM vs unified memory in one breath. "VRAM is memory dedicated to the GPU and physically separate from system RAM; unified memory is one physical pool the CPU and GPU both address, which is why a Mac can load a bigger model than its GPU alone would suggest, at the cost of bandwidth" is worth being able to say without notes.
Verified and unverified
Verified on 2026-09-19: Tech With Tim is a real YouTube channel and this video exists at the given link; GGUF is llama.cpp's own model file format; llama.cpp underlies both Ollama and LM Studio; Ollama serves an OpenAI-compatible REST API on port 11434 by default; Docker Model Runner is a real Docker Desktop feature; Apple Silicon uses a unified memory architecture shared between CPU and GPU; the RTX 4090 has 24GB of VRAM; and KV-cache memory usage grows with context length — all documented, general behavior. Taken from the summary and not independently checked: the specific memory-to-model-size guide (8GB→3-4B, 16GB→7-8B, 32GB→14-30B), the tokens/s figures (100-200 tokens/s on the 4090, the 2-3x gap over Apple Silicon), and all timestamps.
Sources: YouTube — Tech With Tim, "Local AI Explained: How to Run AI Models on Your Computer" · related items: mlx-local-agentic-ai-on-mac (Apple-specific, agentic), Life 1304 (Andrew Ng: keep sensitive data local), Tech #97, Tech #62 (agentic engineering writes the boundaries).
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| weights | 가중치(학습으로 얻은 모델 파라미터 값) · 로컬 모델 파일의 실체. "you download a model's weight file" |
| parameter (8B, 70B) | 파라미터(모델을 이루는 숫자 하나하나, B=billion) · 모델 크기·능력의 표기 단위. "a bigger parameter count (8B, 70B)" |
| quantization | 양자화(가중치의 수치 정밀도를 낮춰 압축하는 것) · 로컬 모델 배포의 핵심 기법. "trading a small, largely invisible quality loss" |
| Q4 / Q8 | 4비트/8비트 양자화 표기 · 파일 크기·품질의 트레이드오프 레벨. "Q4 roughly quarters the 16-bit size" |
| GGUF | llama.cpp가 정의한 로컬 모델 파일 포맷 이름(특정 확장 약어라기보다 llama.cpp 생태계의 고유 포맷명) · 로컬 배포 모델의 기본 컨테이너. "GGUF is the standard packaging format" |
| inference engine | 추론 엔진(가중치를 적재해 토큰을 생성하는 프로그램) · llama.cpp가 대표 사례. "loads the weights into memory and runs the matrix math" |
| forward pass | 순전파(입력을 모델에 통과시켜 출력을 얻는 한 번의 연산) · 추론 한 스텝을 가리키는 표준 용어. "runs a forward pass" |
| VRAM | Video RAM(GPU 전용 메모리, 시스템 RAM과 분리) · PC/Linux 로컬 추론의 1차 제약. "GPU VRAM specifically" |
| unified memory | 통합 메모리(CPU와 GPU가 공유하는 하나의 물리 메모리 풀) · Apple Silicon의 구조, 큰 모델 적재에 유리. "share one physical memory bank" |
| GPU | Graphics Processing Unit(그래픽 처리 장치) · 로컬 추론의 주 연산 하드웨어. "run inference on your own CPU or GPU" |
| CPU | Central Processing Unit(중앙 처리 장치) · GPU와 함께 추론 하드웨어를 이룸. "your own CPU or GPU" |
| bandwidth | 대역폭(단위 시간당 메모리에서 읽어올 수 있는 데이터량) · tokens/s 속도를 정하는 실제 병목. "memory bandwidth is the bottleneck" |
| KV cache | Key-Value 캐시(이전 토큰들의 키·값을 저장해 재계산을 피하는 메모리) · 컨텍스트가 길어질수록 커지는 추가 메모리. "a KV cache on top of the weights" |
| tokens/s | 초당 생성 토큰 수 · 추론 속도를 재는 표준 단위. "100-200 tokens/s in the video's numbers" |
| REST API | Representational State Transfer API(웹 표준 방식의 요청-응답 인터페이스) · Ollama가 노출하는 인터페이스 형태. "an OpenAI-compatible REST API" |
| HTTP server | HyperText Transfer Protocol 서버 · LM Studio가 로컬로 띄우는 개발용 엔드포인트. "runs a local HTTP server" |
| CLI | Command Line Interface(명령줄 인터페이스) · Ollama를 쓰는 방식(ollama pull, ollama run). "Ollama is the CLI path" |
| GUI | Graphical User Interface(그래픽 사용자 인터페이스) · LM Studio를 쓰는 방식. "LM Studio is the GUI path" |
| sweet spot | 최적점(투입 대비 효과가 가장 좋은 지점) · 32GB 구간을 부르는 말. "the sweet spot where perceived intelligence jumps the most" |
| rule of thumb | 경험칙(엄밀한 공식이 아닌 대략적인 지침) · 메모리-모델크기 가이드에 붙이는 라벨. "As a rule of thumb from the video" |