Why
The pattern in the talk is that every attempt to help the model by building it a narrower, more specialized interface made the agent worse, and the one attempt to give it a wider, more general interface — a plain file system and a shell — made it better. That is backwards from how most agent tooling gets designed, where the instinct is to hand the model fewer, purpose-built tools so it "can't go wrong." The talk's explanation is pretraining: a base model has seen orders of magnitude more file exploration and shell usage than it has seen any bespoke tool schema, so the file system is the interface the model is already fluent in, and fluency is what turns into flexible, self-correcting behavior instead of a fixed script. The practical consequence for anyone building an internal agent is to spend the specialization budget on the data (what's in the files, how it's organized) rather than on the tool surface (how many custom functions the agent gets to call).
How it works
The bottleneck: a data team doing everyone else's SQL
The starting problem wasn't a model capability gap, it was an organizational one: every metric question from marketing or sales routed through the data team, who stopped their own work to write SQL and build one-off reports (02:45, 03:25). The goal for D0 was to let non-technical staff ask metric questions directly and get correct, governed answers without a human in the loop for every query.
Four rewrites of D0
- Mega prompt (03:36, 03:50). The Snowflake schema was dumped into the system prompt and the model generated SQL that a human copy-pasted and ran. This proved the question was answerable at all, but had no constraints and no safety net — nothing stopped a bad query from running (04:07).
- Chained multi-agent (04:57, 05:03, 05:21). The task was split across a pipeline — a planning agent, a query-writing agent, a SQL-execution agent, a reporting agent — each with its own isolated tools (schema search, YAML config reads). Only a summary of each stage passed to the next, so context was lost between stages, and when a later stage hit an error there was no way to go back and retry an earlier one; the pipeline could only fail forward (05:52, 06:14).
- Single mega agent with internal state (05:58, 06:24, 06:36). One agent replaced the pipeline, tracking its own planning, exploration, execution and reporting state and looping — reflecting on its own output — for up to 100 steps. This recovered the flexibility the chain lost, but the internal eval success rate was only around 30%, and the agent was fragile on any question it hadn't effectively seen the shape of before (06:52, 07:17).
- File-system agent (07:33, 07:42). Instead of any of the above, the semantic layer went into files in a local sandbox, and the agent got the same four generic tools a shell session gets:
list_dir,read_file,write_file,bash(08:13, 08:53). Because file navigation and shell use are exactly what pretraining makes a model best at, the agent started exploring files and running ad hoc queries on its own — behavior nobody programmed — and the eval score doubled (08:25, 09:36).
Skills: memoized context, not a new model
At production query volume, thousands of daily questions cluster into a much smaller number of repeated aggregation and lookup patterns (10:17, 10:30). Vercel runs a background batch process that mines those repeated patterns into roughly 100 "skill" files (10:39) — written procedures the agent can read before it starts exploring from scratch. Referencing a matching skill under skills/ measurably improves accuracy over starting cold (10:56, 11:13), and this pattern became the basis for Vercel's open-source skill marketplace, skills.sh (11:19).
Eve: agent structure as a file-system convention
Vercel generalized the file-system-agent pattern into a framework, Eve (eve.dev), on the explicit analogy that Next.js abstracted deployment infrastructure behind a file-system routing convention, so agent structure can be a file-system convention too (12:05, 12:47). An Eve project declares skills/ (domain knowledge and procedures), tools/ (custom tools where they're actually needed) and channels/ (Slack, a web interface) (12:31, 14:02). It's open source and self-hostable against Postgres, OpenAI and Docker adapters (13:19, 15:07); deployed on Vercel, it runs on Vercel Workflows for durability and state, Vercel Sandbox for isolated execution, and Vercel Connect for short-lived OIDC tokens, with per-step, per-tool-call and per-cost observability included by default (13:36, 14:50).
The talk's closing point is pragmatic: Vercel evaluated several off-the-shelf data-analysis agent products and none of them worked without the company's own data connections and domain context wired in (15:13, 15:35); a custom agent built around a file-system structure and accumulated skills was the shorter path to something that actually used that context, not a fallback after buying tools failed (15:58, 16:43).
Where it lands in Jayverse
- Number: this is the exact use case. An internal agent over research readings and indicators is D0 with different data — dump the semantic layer (what each reading/indicator table means, how they join) into files under a
skills/-style directory and let the agent read them instead of hand-building query tools. - Verex: same shape, tighter blast radius. A data agent over market and trade tables for ops questions should follow the same file-system pattern, but keep the tools read-only and run it in a sandbox — Verex's tables can move money, D0's could not.
- alice: the repo already is the dumped semantic layer. It's the file system an agent like this would read; the gap the talk doesn't have and alice does is an index or lint step, which is what the Obsidian item
obsidian-three-levels-llm-wikinames. - Auditor: the observability claim is the same one Tech #102 makes about MLflow. "Every step, tool call and cost" is a monitoring floor, not a nice-to-have — the Auditor row should record it the same way for any file-system agent Jayverse builds.
- Eng: a ready interview answer. "How would you build an internal data agent" maps directly onto the four stages here — mega prompt, chained agents, single stateful agent, file-system agent — as a story about where specialization helps and where it doesn't.
Verified and unverified
Verified on 2026-09-19: Vercel makes Next.js, whose file-system routing convention is the explicit analogy the talk uses; Vercel has shipped an agent-skills ecosystem at skills.sh and runtime primitives named Workflows and Sandbox; Claude Code is a file-system-and-shell agent harness of the kind described; "semantic layer" is standard data-warehouse terminology for metric and join definitions layered over raw tables. Taken from the talk's summary and not independently checked: the speaker's exact title ("Chief of Software"), the internal codename "D0," the roughly 30% and 2x eval figures, the roughly 100-skill count, and Eve's name, URL and directory layout exactly as shipped. Sources: YouTube — Andrew Qu, "How We Solved Agent Building", AI Engineer conference · skills.sh · eve.dev (as named in the talk) · related items: Tech #62 (agentic engineering writes the boundaries), Tech #97, Tech #102 (MLflow observability), the harness item harness-engineering-shift-left, the Obsidian item obsidian-three-levels-llm-wiki.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| mega prompt | 메가 프롬프트(스키마 전체 등을 통째로 시스템 프롬프트에 넣는 방식) · 1단계 아키텍처를 가리키는 이름. "Mega prompt (03:36, 03:50)" |
| chained multi-agent | 체인형 다중 에이전트(역할을 나눠 순서대로 실행하는 파이프라인) · 2단계 아키텍처. "Chained multi-agent (04:57, 05:03, 05:21)" |
| semantic layer | 시맨틱 레이어(원본 테이블 위에 지표·조인 정의를 얹은 층) · 데이터 웨어하우스 표준 용어, 파일로 덤프되는 대상. "dump the warehouse's semantic layer into a sandboxed file system" |
| sandbox | 샌드박스(격리된 실행 환경) · 에이전트가 파일을 안전하게 조작하는 공간. "a local sandbox" |
| file-system agent | 파일 시스템 에이전트(전용 도구 대신 파일 탐색·셸로 동작하는 에이전트) · 4단계, 돌파구가 된 아키텍처. "File-system agent (07:33, 07:42)" |
| emergent behavior | 창발 행동(설계자가 명시적으로 프로그래밍하지 않았는데 나타나는 행동) · 파일 시스템 에이전트가 스스로 탐색·쿼리를 시작한 현상. "behavior nobody programmed" |
| eval | 평가(evaluation, 모델·에이전트 성능을 채점하는 벤치마크) · 각 아키텍처를 비교하는 기준. "the internal eval success rate was only around 30%" |
| SQL | Structured Query Language(구조화 질의 언어, 데이터베이스 조회 언어) · 데이터팀이 대신 짜 주던 바로 그것. "writing SQL and dashboards" |
| YAML | YAML Ain't Markup Language(설정 파일에 흔히 쓰는 데이터 직렬화 형식) · 체인형 에이전트의 격리 도구 중 하나. "YAML config reads" |
| OIDC | OpenID Connect(신원 인증을 위한 개방형 프로토콜) · Vercel Connect가 발급하는 단기 토큰의 표준. "short-lived OIDC tokens" |
| skill file | 스킬 파일(반복 패턴을 절차로 적어 둔 문서) · 백지에서 시작하지 않게 해 주는 기억된 맥락. "roughly 100 'skill' files" |
| skills marketplace | 스킬 마켓플레이스(스킬 파일을 공유·재사용하는 오픈소스 저장소) · skills.sh를 가리키는 표현. "Vercel's open-source skill marketplace, skills.sh" |
| declarative convention | 선언적 규약(동작을 절차 코드가 아니라 구조/이름으로 정하는 방식) · Next.js 라우팅과 Eve의 skills/·tools/·channels/ 구조에 쓰인 원리. "a file-system convention too" |
| self-hostable | 셀프 호스팅 가능한(자체 인프라에 직접 설치해 운영할 수 있는) · Eve가 오픈소스로 제공하는 배포 옵션. "self-hostable against Postgres, OpenAI and Docker adapters" |
| observability | 관측 가능성(시스템 내부 동작을 스텝 단위로 볼 수 있는 능력) · Auditor 항목과 직접 연결되는 개념. "per-step, per-tool-call and per-cost observability" |
| blank slate | 백지 상태(사전 지식 없이 처음부터 시작하는 상태) · 스킬이 없을 때 에이전트가 놓이는 상태. "starting cold" / "백지에서 탐색을 시작" |
| pretraining data | 사전 훈련 데이터(모델이 학습에 쓴 원본 자료) · 파일 시스템·셸 사용에 모델이 유창한 이유. "pretraining makes a model best at" |
| read-only tool | 읽기 전용 도구(데이터를 바꾸지 못하고 조회만 하는 도구) · Verex처럼 돈이 걸린 데이터에 적용해야 할 안전장치. "keep the tools read-only" |
| shortest path (to something) | 지름길, 가장 짧은 경로 · 기성품보다 직접 구축이 나은 이유를 표현. "the shorter path to something that actually used that context" |
| fail forward | 앞으로만 실패하다(오류가 나도 이전 단계로 돌아가지 못하고 진행만 되는 상태) · 체인형 에이전트의 한계를 설명. "the pipeline could only fail forward" |