Skip to Main Content

Look to Listen: point your phone at a person in a noisy room and hear only their voice, with live captions. Problem: over 430 million people live with disabling hearing loss (WHO). Hearing aids amplify the whole room, so in a crowd the one voice you want is buried. How it works: an iPhone camera feeds Chrome, where MediaPipe tracks faces. You point at someone, they consent, and every 0.5 s the browser sends 0.5 s of audio plus that person’s lip crops. On an RTX 4090, the open AV_MossFormer2_TSE_16K model (Alibaba ClearVoice) uses the lips to pull that one voice out of a rolling 3 s window. That takes 130 ms per hop, and the result plays in your AirPods about 1.1 s behind real time. Gemini 3.5 Transcribe Live captions two lanes side by side: what the microphone hears, and what the person you are looking at says. Gemini 4 Neon attributes room captions to faces. Gemini 4 Argon answers questions about the conversation. Gemini Robotics ER 2 lets the person you are looking at command a robot. A Managed Agent with hooks enforces consent and blocks regressions. What we built: ClearVoice ships as an offline tool that processes a finished video file. We made it live, with a binary streaming protocol, rolling buffers that lock lip frames to audio, crossfaded output, automatic failover to Gemini 4 Hydrogen when the GPU is unreachable, and automatic recovery when it returns. Measured: GPU 130 ms per 0.5 s hop. Voice about 1.1 s behind. A Gemini-only engine had 0% leakage but 46% word error and ran 8.5 s behind, which is why the separator exists. In a crowd test with 7 faces, separation gained about 6 dB (SI-SDR) when the crowd was as loud as the speaker, and broke when the crowd was 10 dB louder. Honest range: 1 to 2 m. Privacy: nobody is captioned without consent, and the GPU server never stores audio.

Team

nerdboss-stm

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview