Look to listen
Gemini 4 NYC HackathonLook to Listen: point your phone at a person in a noisy room and hear only their voice, with live captions. Problem: over 430 million people live with disabling hearing loss (WHO). Hearing aids amplify the whole room, so in a crowd the one voice you want is buried. How it works: an iPhone camera feeds Chrome, where MediaPipe tracks faces. You point at someone, they consent, and every 0.5 s the browser sends 0.5 s of audio plus that person’s lip crops. On an RTX 4090, the open AV_MossFormer2_TSE_16K model (Alibaba ClearVoice) uses the lips to pull that one voice out of a rolling 3 s window. That takes 130 ms per hop, and the result plays in your AirPods about 1.1 s behind real time. Gemini 3.5 Transcribe Live captions two lanes side by side: what the microphone hears, and what the person you are looking at says. Gemini 4 Neon attributes room captions to faces. Gemini 4 Argon answers questions about the conversation. Gemini Robotics ER 2 lets the person you are looking at command a robot. A Managed Agent with hooks enforces consent and blocks regressions. What we built: ClearVoice ships as an offline tool that processes a finished video file. We made it live, with a binary streaming protocol, rolling buffers that lock lip frames to audio, crossfaded output, automatic failover to Gemini 4 Hydrogen when the GPU is unreachable, and automatic recovery when it returns. Measured: GPU 130 ms per 0.5 s hop. Voice about 1.1 s behind. A Gemini-only engine had 0% leakage but 46% word error and ran 8.5 s behind, which is why the separator exists. In a crowd test with 7 faces, separation gained about 6 dB (SI-SDR) when the crowd was as loud as the speaker, and broke when the crowd was 10 dB louder. Honest range: 1 to 2 m. Privacy: nobody is captioned without consent, and the GPU server never stores audio.





