# Tofu

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Ashmita Goyal](https://cerebralvalley.ai/u/Ashmita_1206), [Sandes Savarimuthu](https://cerebralvalley.ai/u/Sandes)
- **GitHub:** https://github.com/Landry-16/live_tutorial_coach
- **Demo video:** https://www.youtube.com/watch?v=5Yi12iO9gjI
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3695

Video tutorials only go one way: they never see your hands. You pause, rewind, squint, and find the mistake three steps too late, when it can no longer be undone.

Live Tutorial Coach turns any hands-on video tutorial into a live coach. Upload an mp4, prop up your phone, and follow along hands-free.

What it does
- Splits the video into steps. Muse Spark watches the whole tutorial, reads its narration (Muse Voice Transcribe) and returns validated steps: instruction, expected result, common mistakes, a reference picture, and whether the step can be undone.
- Coaches one step at a time. It reads each step aloud, plays only that part of the video, then waits for you.
- Watches your work. Every 4 seconds, 3 camera frames are compared with the step's reference picture. About 4 s and $0.003 per check.
- Points at the problem. On a mistake it warns you out loud and SAM 3.1 boxes the exact object on your camera view.
- Stops you before the point of no return. Before a step that can't be undone, saying "next" is not enough: the camera must have seen the current step finished. Otherwise the coach says why and boxes what is unfinished. In the demo, the user tries to power on a game console before the new fan is seated, and the coach stops them.
- Hands-free. Muse Voice Transcribe streams your voice in real time: short phrases are commands (next, done, repeat, back), longer ones are questions answered with the current step as context.
- Built not to nag. It speaks only when two checks agree, waits 15 s between warnings, treats hands in the way as normal work, and moves on by itself once the work matches the step.

It works on any tutorial, no setup or training: tested on a console repair, a recipe, first aid and origami.

How we built it
- Node 22, TypeScript, Hono, WebSockets; React 19 and Vite, built for a phone.
- Muse Spark 1.3: video understanding, structured JSON, live verdicts, answers.
- SAM 3.1: boxes the problem on the camera frame, trying up to 3 names from Muse Spark.
- Muse Voice Transcribe 1.0: file transcription in prep, realtime streaming live.
- A state machine decides when to speak, covered by 18 automated tests.

Challenges
Strictness: a coach that warns about normal hand movement gets switched off, one that misses a real mistake is useless. We tuned it on real footage until correct work stayed quiet while wrong folds, skipped steps and unfinished steps before a point of no return got caught.

What's next
We designed it for Meta AI glasses from day one; the phone is the stand-in. Glasses already carry what the coach needs: a camera that sees what your hands see, open-ear audio for its voice, and a microphone, so your hands never leave the work. The same loop then reaches every hands-on trade: mechanics, electrical work, lab protocols, and robotics, where it can check a robot's work or teach it by watching people.

Demo note: to film without a second person, the phone camera replays the tutorial's own footage; every model call is live.

## More from Meta Global AI Developer Hackathon

- [VALE](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3692)
- [Agela AI](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3693)
- [GroundTest](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3694)
- [Valentin Englert - lwnzhn.com](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3696)
- [Diligent 619](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3697)
- [Winning Startup](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3698)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3695. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt