Tofu

Video tutorials only go one way: they never see your hands. You pause, rewind, squint, and find the mistake three steps too late, when it can no longer be undone. Live Tutorial Coach turns any hands-on video tutorial into a live coach. Upload an mp4, prop up your phone, and follow along hands-free. What it does - Splits the video into steps. Muse Spark watches the whole tutorial, reads its narration (Muse Voice Transcribe) and returns validated steps: instruction, expected result, common mistakes, a reference picture, and whether the step can be undone. - Coaches one step at a time. It reads each step aloud, plays only that part of the video, then waits for you. - Watches your work. Every 4 seconds, 3 camera frames are compared with the step's reference picture. About 4 s and $0.003 per check. - Points at the problem. On a mistake it warns you out loud and SAM 3.1 boxes the exact object on your camera view. - Stops you before the point of no return. Before a step that can't be undone, saying "next" is not enough: the camera must have seen the current step finished. Otherwise the coach says why and boxes what is unfinished. In the demo, the user tries to power on a game console before the new fan is seated, and the coach stops them. - Hands-free. Muse Voice Transcribe streams your voice in real time: short phrases are commands (next, done, repeat, back), longer ones are questions answered with the current step as context. - Built not to nag. It speaks only when two checks agree, waits 15 s between warnings, treats hands in the way as normal work, and moves on by itself once the work matches the step. It works on any tutorial, no setup or training: tested on a console repair, a recipe, first aid and origami. How we built it - Node 22, TypeScript, Hono, WebSockets; React 19 and Vite, built for a phone. - Muse Spark 1.3: video understanding, structured JSON, live verdicts, answers. - SAM 3.1: boxes the problem on the camera frame, trying up to 3 names from Muse Spark. - Muse Voice Transcribe 1.0: file transcription in prep, realtime streaming live. - A state machine decides when to speak, covered by 18 automated tests. Challenges Strictness: a coach that warns about normal hand movement gets switched off, one that misses a real mistake is useless. We tuned it on real footage until correct work stayed quiet while wrong folds, skipped steps and unfinished steps before a point of no return got caught. What's next We designed it for Meta AI glasses from day one; the phone is the stand-in. Glasses already carry what the coach needs: a camera that sees what your hands see, open-ear audio for its voice, and a microphone, so your hands never leave the work. The same loop then reaches every hands-on trade: mechanics, electrical work, lab protocols, and robotics, where it can check a robot's work or teach it by watching people. Demo note: to film without a second person, the phone camera replays the tutorial's own footage; every model call is live.