Why
A sequential pipeline puts a hard floor on latency: it cannot start producing a response until it has fully stopped listening, so the wait is the sum of three stages, not the slowest one. Faster STT or TTS models shrink that floor but never remove it, because the structure — listen fully, then think, then speak — is still three gated steps. Holding one open stream instead removes the gate, and it also unlocks something a pipeline cannot do at all by construction: accepting new input while a response is still being produced. There is no seam in a pipeline where "the user just started talking again" can land; in an open stream it is just another event. The same discipline — a queue that separates what you're sending from what you're receiving, so one slow leg of I/O never blocks the other — applies to any agent consuming a live event feed, not only audio.
How it works
The pipeline's silence, and the phone-call alternative
The old shape is STT → LLM → TTS, each stage blocking on the previous one's full output before starting (01:16). The video reframes the target as a persistent bidirectional connection rather than three request/response hops: audio keeps flowing both ways over a single channel, and the system has to support barge-in — the user cutting the agent off mid-sentence — as a first-class case, not an edge case (01:34).
The wiring: browser, WebSocket, ADK, Gemini Live
The browser owns the microphone, the speaker, and audio playback, and keeps one always-open WebSocket to the backend (02:02). The backend's ADK layer sits in the middle and relays the bidirectional stream between the client and the Gemini Live API without blocking on either side, using queues and a runner (02:11). Gemini Live is the part that does real-time audio understanding: it consumes the incoming audio directly and streams back both audio responses and structured events as they're ready, not after the whole utterance finishes (02:20).
ADK's three objects: Agent, Runner, Session
- Agent is a configuration object: model, persona/instructions, and capabilities exposed as tools (02:48). Adding a new capability is just adding a Python function to the agent's tool list (03:04).
- Runner executes the agent and owns the call's lifecycle from open to close (03:29). It normalizes every interaction — a spoken chunk, a tool call, an interruption — into an
Eventobject the app consumes uniformly (03:37). - Session is where conversation state lives (03:46). For real-time voice this matters concretely: a remote session store adds network I/O on the state-read path, and for a channel where hundreds of milliseconds matter, that's the wrong trade — in-memory session storage is what keeps the round trip fast (03:46, 08:09).
The LiveRequestQueue: two ways in, one stream out
Sending and receiving are decoupled through a queue, which the video compares to a conveyor-belt sushi restaurant: producers and consumers work off the same belt without waiting on each other directly (04:46). One async task keeps pushing browser audio chunks onto the LiveRequestQueue as they arrive (upstream); a separate task keeps streaming model events back out to the browser (downstream) (05:03). Feeding the queue has two distinct calls for two distinct kinds of input: send_realtime is for an open-ended continuous stream like the microphone, and it has to keep sending audio through silence too, because Gemini's own voice-activity detection (VAD) needs that continuous signal to recognize where a sentence actually ends (05:27). send_content is for a single, already-complete piece of data where the user explicitly marked the end of input — typed text, an attached image (06:06).
Events: play, caption, call, or stop
The runner hands back one event stream, and the app dispatches on event type: audio chunks go straight to playback, transcript events go to the caption UI, and function-call events get executed immediately rather than queued (06:54). If the user starts talking over the agent, Gemini detects it and the Live API sends down an Interrupted event; the backend's job is to stop playback immediately on receiving it, not to let the current audio finish (07:04).
Where it lands in Jayverse
- Rabbit: an
Interruptedevent has to reach the mandate layer, not just the speaker. If Rabbit ever ships a voice front end for an agent that moves money, "the user cut the agent off" is a stop signal with financial consequences — it should map to canceling the in-flight mandate, not just muting audio. - Verex: not an order-entry channel, but the event model still applies to fills. Voice input is the wrong UI for placing an order, but the pattern of "push structured events as they happen, dispatch by type, no waiting for a full response" is exactly what a live fill-notification feed needs.
- Auditor: an
Interruptedevent is a fact worth logging. If a voice agent ever fronts something the Auditor watches, the interruption itself — when it happened, what the agent was saying, what it was told to stop — belongs in the audit trail, same as any other checked event. - Game: this is the wiring for a talking NPC. A street-level NPC that can hold a real back-and-forth (not scripted lines) needs exactly this shape: one open connection, VAD-aware continuous input, and barge-in so the player can interrupt.
- Eng: a ready-made example for explaining latency budgets in an interview. "Why does a naive voice pipeline feel laggy, and how do you fix it structurally rather than by swapping models" is a concrete, well-scoped answer built from this video.
Verified and unverified
Verified on 2026-09-19: Google's Agent Development Kit (ADK) is a real open-source Python framework built around Agent, Runner, and Session concepts, with a LiveRequestQueue for bidirectional streaming to the Gemini Live API; the Gemini Live API is Google's real low-latency bidirectional audio API with built-in voice activity detection and interruption events; WebSocket is the standard browser transport for this kind of persistent connection. Taken from the video summary and not independently checked: the exact timestamps, the specific claim that STT→LLM→TTS pipelines produce "multi-second" silence, and the precise names/behavior of send_realtime and send_content as described (the general two-input-mode shape is consistent with how ADK's live API is documented, but exact call semantics were not independently verified here). Sources: YouTube — Build a real-time voice AI agent with Google ADK and Gemini Live API · related items: Tech #62 (agentic engineering writes the boundaries), the Homa item (homa-receiver-driven-transport) on tail latency and receiver-driven flow control, which pairs with this item's event-driven, receiver-reacts-immediately design.
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| STT | Speech-to-Text(음성을 텍스트로 변환) · 옛 파이프라인의 첫 단계. "speech-to-text, then an LLM call, then text-to-speech" |
| LLM | Large Language Model(대형 언어 모델) · 파이프라인의 가운데 단계, 추론을 맡는 부분. "then an LLM call" |
| TTS | Text-to-Speech(텍스트를 음성으로 변환) · 옛 파이프라인의 마지막 단계. "text-to-speech" |
| ADK | Agent Development Kit(구글의 에이전트 개발 키트) · Agent·Runner·Session 개념을 제공하는 오픈소스 파이썬 프레임워크. "Google's Agent Development Kit (ADK) supplies the plumbing" |
| VAD | Voice Activity Detection(음성 활동 감지) · 침묵 속에서 문장의 끝을 인식하는 기능, send_realtime이 계속 오디오를 보내야 하는 이유. "Gemini's own voice-activity detection (VAD)" |
| WebSocket | 웹소켓(브라우저와 서버 사이 상시 양방향 연결 프로토콜) · 음성 스트림을 나르는 표준 전송 수단. "keeps one always-open WebSocket to the backend" |
| barge-in | 말 끼어들기(상대가 말하는 도중에 끼어드는 것) · 전화 통화형 설계가 지원해야 하는 핵심 기능. "the user cutting the agent off mid-sentence" |
| turn-taking | 발화 순서 교대(대화에서 말할 차례를 주고받는 것) · 파이프라인이 아니라 열린 스트림이 필요한 이유. "a live turn-taking loop" |
| in-memory (session) | 인메모리(디스크·네트워크 없이 메모리에 상태를 두는 방식) · 원격 세션 저장소의 네트워크 I/O를 피해 지연을 줄이는 선택. "in-memory session storage is what keeps the round trip fast" |
| LiveRequestQueue | ADK의 실시간 요청 큐(송신·수신을 분리하는 큐 객체) · 업스트림과 다운스트림을 잇는 핵심 구조. "one async task keeps pushing browser audio chunks onto the LiveRequestQueue" |
| send_realtime | 연속 스트림 입력 함수(끝이 없는 데이터, 예: 마이크용) · 침묵 구간도 계속 보내야 VAD가 작동. "send_realtime is for an open-ended continuous stream" |
| send_content | 완성된 단일 입력 함수(사용자가 전송을 명시적으로 끝낸 데이터용) · 타이핑 텍스트·이미지 등에 사용. "send_content is for a single, already-complete piece of data" |
| upstream / downstream | 업스트림(보내는 방향) / 다운스트림(받는 방향) · 큐로 분리된 두 비동기 작업을 가리키는 말. "one async task... (upstream); a separate task... (downstream)" |
| non-blocking | 논블로킹(한쪽 작업이 다른 쪽을 막지 않는 방식) · ADK 층이 스트림을 중계하는 방식. "relays the bidirectional stream... without blocking on either side" |
| Interrupted event | 인터럽트 이벤트(사용자가 끼어들었을 때 모델이 내려보내는 신호) · 백엔드가 즉시 재생을 멈춰야 하는 신호. "the Live API sends down an Interrupted event" |
| persona | 페르소나(에이전트의 성격·지침 설정) · Agent 객체를 구성하는 세 요소 중 하나. "model, persona/instructions, and capabilities" |
| hard floor (on latency) | 지연의 바닥값(더 줄일 수 없는 최소 대기 시간) · 순차 구조 자체가 만드는 한계를 설명하는 표현. "A sequential pipeline puts a hard floor on latency" |
| gated step | 문지기 단계(앞 단계가 끝나야만 다음이 시작되는 구조) · 파이프라인의 근본적 결함을 가리키는 말. "three gated steps" |
| dispatch (on event type) | 타입별로 분기 처리하다 · 이벤트 스트림을 재생·자막·함수 호출로 나눠 처리하는 방식. "the app dispatches on event type" |