LocalDub

I'm Muhammad Asim, a student developer from Pakistan, and I built LocalDub on my own during the hackathon. It takes a Hindi or Urdu video and dubs it into English: every character speaks in a voice cloned from their own original lines, the jokes are rewritten so they still land in English, each line is fitted to the time the actor actually speaks, and the original music and effects stay underneath. Why: where I live, people make great comedy, cooking, teaching and lecture videos in Urdu, Punjabi, Saraiki and Hindi, and almost none of it travels beyond people who already speak the language. Studios can pay for dubbing; a creator with a phone can't. The AI dubbing tools I tried translate jokes word for word, give everyone a stock voice, and keep talking after the actor's mouth stops, and Hinglish (Hindi and English mixed in one sentence) breaks them. How: Demucs separates the voices from the music. Omnilingual ASR transcribes each line. Muse Glimmer looks at a frame for every line to label who is on screen, then reads the whole script and writes English that keeps the joke, with a short version and an emotion tag per line. Each line is voiced with a clone of that speaker's own original line, fitted into its slot and mixed back over the original soundtrack. Then LocalDub listens to its own dub: it checks timing, loudness and pitch against the original line, re-transcribes the English with Omnilingual ASR to see what it really says, and sends the bad lines back to Glimmer for a rewrite. Constraints: Meta's hosted Model API isn't available in Pakistan, and I have no budget or card for paid compute. So all of it runs on Meta's open weights, offline, on free Kaggle CPU machines. It's slow (Glimmer runs at about 1 token per second there), but it works with zero spend. Results, on a 5-minute All India Bakchod sketch (CC BY 3.0, dialogue translated and re-voiced), after four rounds of dub, listen, fix (v1 to v5): 33 lines are dubbed (I split the opening phone call so each side has its own voice) and 26 short chunks keep the original audio (14 were already English, 12 are laughs and interjections I chose not to dub rather than dub badly). All 33 dubbed lines fit their slot (v1: 42 of 44), junk one-word dubs went from 8 to 0, voice-pitch mismatches over 4 semitones from 11 to 2, and the mean loudness gap to the original line from 1.65 to 0.61 dB. The listen check (Omnilingual ASR re-transcribing my English) gives a word error rate of 0.151. To be honest, v4 scored better (0.123), but in the last round I fixed lines that scored well and meant the wrong thing, including the sketch's main joke, and I chose correct over a better number. Every number and every rewrite is in the repo under results/. Update, Nov 2: I added a character stage (Whisper large-v3 word timestamps, speaker embeddings, face tracks) so each voice follows the character on screen. Mean voice-pitch gap per speaking turn fell from 2.17 to 1.01 semitones and turns over 4 semitones from 16 to 2 (v6 to v7). My frame-by-frame audio/video check also found that more talking faces had no dub audio (high flags 7 to 14); retiming in v8 brought that to 9, and I'm still fixing the rest. Characters, numbers and the v8 video are in the repo and the demo-v2 release. What's next: Urdu, Punjabi, Saraiki and Pashto, and the reverse direction too, English lectures and tutorials dubbed into Urdu and Hindi so students here can learn in the language they think in. After that: lip-sync with face tracking (SAM), a simple web app where a creator reviews the script line by line, and an open TTS so it all runs on one machine. A single GPU would take a 5-minute video from hours to minutes; that's the main thing holding it back.