ShopFloor Copilot

In small factories and repair shops, the right way to do a job usually lives in one senior person's hands. New and contract workers watch it once, maybe twice, and then they are on their own. Nobody has time to stand next to them and check every step, so mistakes only show up later, at quality check or with the customer. When the senior person leaves, the know-how leaves too. ShopFloor Copilot puts that expert in every worker's ear. It has three parts. Teach: the expert records the job once on a phone and talks through it while working. Muse Voice Transcribe turns the narration into text, and Muse Spark watches the video and writes a step-by-step procedure. Every step has to end in something a camera can see, like "top cap removed" rather than "loosen screw", because a loosened screw looks exactly like an untouched one. SAM 3.1 marks the parts for each step. The supervisor can edit any step before it goes live. Check: a new worker wears the phone on their head, hands free, and just says "done" after each step. A Muse Spark agent first describes what the hands did without being told what to expect, then a strict judge compares that against the procedure and has to call a tool: pass the step, or flag it as skipped, wrong order, wrong part, missing or unclear. If it's a flag, the worker hears a short correction out loud in Kannada, Hindi or English and tries again. SAM 3.1 tracks the hands in the clip so the supervisor gets visual evidence for every decision. Report: every verdict is saved with its clip and reason. The supervisor sees first-try pass rate, the step people fail most, and the type of mistakes, and Muse Spark suggests how to rewrite the procedure so the same mistake stops happening. I tested it on 8 real recordings of an SSD enclosure teardown, including clips where the work was half done or the hands blocked the view. Result: 12 out of 12 judgements correct, 0 missed mistakes, 0 false alarms, about 18 seconds per check. Getting there took five evaluation runs (86%, 71%, 86%, 100%), and the biggest lesson became a rule in Teach mode: only check results the camera can actually see. Nothing in the app is specific to that product. It works for any job you can film, from a valve repack to a motor service. Honest limits: so far it has been tested on one product, one camera position and one person, and a check takes about 18 seconds, so it suits step-by-step assembly better than fast line work. Live app: https://shopfloor-copilot.onrender.com (free server, so the first load can take about a minute) Built from scratch during the hackathon, solo.