Skip to Main Content

Ghost Hands

Built at Meta Global AI Developer Hackathon · Oct 26, 2026 · Remote

Ghost Hands — Demo video

ISL Bridge is a two-way Indian Sign Language conversation on one phone. Sign to it and it speaks; speak to it and a real signer answers inside your camera view. It learns your way of signing in 5 repetitions. The problem India has over 6 million ISL users and only a few hundred certified interpreters. At a hospital, a Deaf patient usually falls back on written notes, but many Deaf adults were schooled orally and read little Hindi or English. ISL also varies by region, so a single fixed model fails many real signers. What it does - Sign → speech: the patient signs naturally (no pauses between signs) and lowers their hands. The phone speaks a fluent English or Hindi sentence, usually 0.5–1.5 s later. - Speech → sign: the doctor holds the mic and talks. Live captions appear, and the reply plays as real human signers cut out of sign videos, standing inside the patient's camera view, with picture cards for words like "X-ray" or "after food". - Learns your signing: a guided flow (a ghost shows each sign, you copy it five times) adapts the app to your regional variant, or adds new signs such as your name or your medicine. - For the visit: a bilingual visit summary for the doctor, SOS on the EMERGENCY sign, and Ghost Coach so families can learn ISL. Results (held-out data) 202 everyday and hospital signs. 79.2% top-1 on 624 held-out clips; 69.6% on signers never seen in training. One personal example per sign took unseen signers from 3.8% to 75.0%. Conversation context raised meaning preserved from 64.0% to 74.6%. Private by design: the video never leaves the phone; only body keypoints are sent. About $0.001 per sentence. It runs in the browser, with nothing to install. Honest scope: it recognises 202 signs, not all of ISL, and it supports interpreters rather than replacing them.

Team