Hands up

Over 500,000 Americans use ASL as their primary language. When a deaf patient walks into a clinic, communication often breaks down — interpreters are rarely available on-site, scheduling can delay care, and certified medical interpreters can cost roughly $100–150 per hour. If even those 500,000 individuals have only 5–10 healthcare appointments annually, that translates to an estimated 2.5–5 million patient visits each year where ASL interpretation or accessible communication may be needed. In many cases, interpreter scheduling delays care and adds operational costs for providers, while patients lose the ability to communicate independently and are forced to rely on written notes or family members. Voice of the Voiceless is a real-time, browser-based ASL translator designed for clinic reception desks. A patient signs to a camera; the system instantly recognizes the gesture, speaks it aloud, and logs the interaction as a structured conversation transcript — empowering patients to communicate directly while reducing dependence on interpreter availability. How it works: a dual-engine pipeline runs simultaneously. Gemini Flash processes short video clips for high-accuracy sign classification, while MediaPipe Hands tracks hand landmarks frame-by-frame and matches gestures locally using Dynamic Time Warping (DTW) for low-latency offline inference. A smart auto-trigger detects intentional signing and filters out background movement, keeping interactions natural and seamless. The interface presents a two-sided conversation log: patient signs appear on one side and the clinician’s spoken responses on the other. Conversations can be exported as CSV files for medical documentation and record keeping. Stack: React · TypeScript · Gemini Flash · MediaPipe · Web Speech API · Express Ready for Google Cloud Run deployment, making it easy for clinics to scale from a single reception desk to organization-wide usage. No specialist hardware. No app installation. Just a browser and a camera — helping empower patients and remove communication barriers in healthcare settings.

Team

Placement

Participant