# Look to listen

- **Event:** [Gemini 4 NYC Hackathon](https://cerebralvalley.ai/e/gemini-4-nyc)
- **When:** Sat, Nov 21 at 9:00 AM – 10:00 PM (EST)
- **Where:** New York, NY
- **Team:** [Saran Teja Mallela](https://cerebralvalley.ai/u/nerdboss-stm)
- **GitHub:** https://github.com/Nerdboss-stm/look-to-listen
- **Demo video:** https://youtu.be/19ll8xKArOE
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/18

Look to Listen: point your phone at a person in a noisy room and hear only their voice, with live captions.

Problem: over 430 million people live with disabling hearing loss (WHO). Hearing aids amplify the whole room, so in a crowd the one voice you want is buried.

How it works: an iPhone camera feeds Chrome, where MediaPipe tracks faces. You point at someone, they consent, and every 0.5 s the browser sends 0.5 s of audio plus that person’s lip crops. On an RTX 4090, the open AV_MossFormer2_TSE_16K model (Alibaba ClearVoice) uses the lips to pull that one voice out of a rolling 3 s window. That takes 130 ms per hop, and the result plays in your AirPods about 1.1 s behind real time. Gemini 3.5 Transcribe Live captions two lanes side by side: what the microphone hears, and what the person you are looking at says. Gemini 4 Neon attributes room captions to faces. Gemini 4 Argon answers questions about the conversation. Gemini Robotics ER 2 lets the person you are looking at command a robot. A Managed Agent with hooks enforces consent and blocks regressions.

What we built: ClearVoice ships as an offline tool that processes a finished video file. We made it live, with a binary streaming protocol, rolling buffers that lock lip frames to audio, crossfaded output, automatic failover to Gemini 4 Hydrogen when the GPU is unreachable, and automatic recovery when it returns.

Measured: GPU 130 ms per 0.5 s hop. Voice about 1.1 s behind. A Gemini-only engine had 0% leakage but 46% word error and ran 8.5 s behind, which is why the separator exists. In a crowd test with 7 faces, separation gained about 6 dB (SI-SDR) when the crowd was as loud as the speaker, and broke when the crowd was 10 dB louder. Honest range: 1 to 2 m.

Privacy: nobody is captioned without consent, and the GPU server never stores audio.

## More from Gemini 4 NYC Hackathon

- [Elodie](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/15)
- [Amac and Murray](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/16)
- [Discovery Clock](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/17)
- [Casita Lista](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/19)
- [AREA-Q](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/20)
- [Werewolf League](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/21)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/18. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt
