Hastaak

Hastaak is a two-way Indian Sign Language ⇄ speech assistant for face-to-face conversations between a Deaf person and a hearing person. The Deaf person signs in front of a webcam. MediaPipe tracks their hands, body and face, and SignNet, a small sequence classifier that runs on the device, recognizes each sign from a 47-sign vocabulary trained on AI4Bharat’s INCLUDE dataset. When they finish a sentence, Meta’s Muse Spark turns the signs into a natural sentence, and Meta’s MMS-TTS voices run locally to speak it in the listener’s language (11 Indian languages, including English). In the other direction, the hearing person speaks. Muse Voice Transcribe captions their words, which are mapped to ISL signs and shown by a 2D avatar replaying real signers’ recorded motion, or by the original sign videos. Recognition stays on the laptop for privacy and cost, and only text and the hearing person’s voice go to Meta’s models. Raised eyebrows mark yes/no questions, and SAM 3.1 can pick out the signer when several people are in view.