# Walkie Talkie

- **Event:** [OpenAI Voice Hack Night](https://cerebralvalley.ai/e/openai-voice-hack-night)
- **When:** Wed, May 27 at 3:00 – 9:00 PM (PDT)
- **Where:** 1515 3rd Street, San Francisco, CA
- **Team:** [Lily Zhang](https://cerebralvalley.ai/u/lilyzhng)
- **Demo video:** https://lily-walkie-talkie.vercel.app/ https://www.youtube.com/watch?v=6jWTQ3CaYrg
- **Gallery:** https://cerebralvalley.ai/e/openai-voice-hack-night/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/openai-voice-hack-night/hackathon/gallery/13

**Walkie-Talkie** is a real-time thinking partner orchestrating a dual model setup of `gpt-realtime-2` and Codex. It pairs a live voice interaction model with parallel background agents that handle tool calling, code execution, and generative UI, so the conversation never stops while work happens behind the scenes.


### Why users need this

Today, using an AI agent means watching it work. You prompt, you wait, you read, you prompt again. Voice doesn't fix this if it's still turn-based. What people actually want is to think out loud while things get done. Walkie-Talkie lets you talk through a problem, ask for a chart, pivot to a different question, request a code change, all in one continuous conversation, while each task runs in the background and results surface as they complete.


### The architecture

When Thinking Machines released their interaction model, I recognized the same architecture I'd been building independently. Their thesis: for interactivity to scale with intelligence, it must be part of the model itself. I don't fully agree. Model plus harness becomes a powerful, collaborative agent. These are two means to the same end. Interactivity is a type of user experience, and as long as users get that experience, it doesn't matter whether the system uses a full-duplex model or cascaded scaffolding. Walkie-Talkie proves this with OpenAI's realtime voice stack. `gpt-realtime-2` handles the hard interaction problems (sub-230ms responses, pause detection, mid-sentence pivots, simultaneous listening and speaking). Background agents handle the heavy work.


### Parallel agents and context management

The background is not a fixed set of tool functions. Each background agent is a coding agent with shell access in a persistent sandbox, the same pattern that makes CLI-based agents like Codex so powerful. Instead of predefining every tool (generate_chart, run_query, etc.), the agent gets intent from the voice model and figures out *how* autonomously. Multiple agents run simultaneously, each handling a different task. The front-end coordinator dispatches tasks, tracks which agents are working, and manages the context bridge between the voice conversation and each background agent. When an agent finishes (a chart, a file, a generative UI artifact), the result routes back to the voice model, which weaves it into the conversation naturally. This makes the system an agent orchestration layer with voice as the interface.


### Why this is the correct architecture

I analyzed the five capabilities needed for natural voice interaction: speaking during user speech, pause-vs-endpoint detection, real-time semantic processing, micro-responses, and simultaneous input/output. These capabilities conflict with the demands of heavy tool execution. Separating them isn't a shortcut. It's the correct design approach.


### Links

- **Using:** gpt-realtime-2 (OpenAI Realtime API), Codex (CLI/SDK)
- **Research:** https://lilyzhng.github.io/posts/interaction-model/
- **Demo:** https://lily-walkie-talkie.vercel.app/
- **GitHub:** https://github.com/lilyzhng/walkie-talkie

---

Markdown version of https://cerebralvalley.ai/e/openai-voice-hack-night/hackathon/gallery/13. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt