# USTAD - **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon) - **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST) - **Where:** Online - **Team:** [PRABINDER SINGH](https://cerebralvalley.ai/u/prabindersinghh) - **GitHub:** https://github.com/prabindersinghh/ustad - **Demo video:** https://youtu.be/kBOjStcG4lQ - **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery - **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3224 USTAD : Ustad is an AI co-pilot for factory workers, built for Meta glasses and running on a phone today. An expert shows a job once. Ustad then guides every new worker through it by voice, and checks each step on camera. Try it in one tap (judges): https://prabindersinghh--ustad-web.us-east.modal.run/#access=USTAD-T5YHER Or open the link and enter this code: USTAD-T5YHER THE PROBLEM How a factory job is really done often lives in one expert's head. That person can't stand beside every new worker, and paper can't see the work. So new workers learn by trial and error, and one wrong step can ruin a part, stop a line or hurt someone. WHY IT MATTERS - 1.9M US manufacturing jobs could go unfilled by 2033, of up to 3.8M needed (Deloitte and The Manufacturing Institute, 2024). - Unplanned downtime costs the world's 500 largest companies about $1.4T a year, 11% of revenue (Siemens, 2024). A Vanson Bourne survey traced 23% of manufacturing downtime to human error. - 395M workers a year suffer non-fatal work injuries, across all sectors (ILO, 2019 data). - We're starting with India: 21M people work in 267,000 registered factories (ASI 2024-25), and only 4.2% of people aged 15-59 have formal vocational training (PLFS 2025). Setup is one recording, with no training data per machine. About $0.09 to record a job, about $0.12 to guide a 4-step job. REAL TRIAL RESULTS, two factories in Ludhiana: PI Cottex Pvt Ltd (textile), and MCPL (Modern Colours Private Limited). 97% of correct steps confirmed. 95% of planted errors caught. New operators 88% faster than the current method. Job cards 79% faster for senior technicians. HOW IT WORKS Capture: narration streams over WebSocket to realtime speech to text, with keyframes when the view changes. Muse Spark writes structured steps: instruction, success criteria, hazard, check type and target phrase. SAM 3.1 then tests every phrase on the expert's own frame; a miss gets 3 alternatives, tested in parallel. Guide: a frame about once a second is grounded by SAM. Pipelined background judges let a step verify itself, or the worker says "done". Muse Spark sees the frame, the expert's photo, the boxes and the criteria, and must make exactly one tool call: verify_step, flag_error or request_better_view. The server executes it, so the camera's answer changes what happens next. Safety in code, not in prompts: lockout and hazard steps are never credited from a later view, stricter steps need two passes, out-of-range readings are overridden to flag_error, and a hand near a tracked hazard triggers a spoken warning. "Can I skip the lockout?" escalates to a supervisor. Also built: Hindi with numbers checked in code, evidence packs, supervisor-approved improvements, offline recovery, and a Muse Glimmer audit of doubtful steps. An RL data pipeline built 1,046 labelled examples from one clip; training hasn't run yet. 4 to 5 s from "done" to a decision. Sign-in and spending limits on the public link. Not tested on glasses yet. ## More from Meta Global AI Developer Hackathon - [10xNG](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3221) - [Praxion Labs](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3222) - [Desert Eagle](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3223) - [PetroMind](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3225) - [Harvest Outlook](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3226) - [Independent](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3227) --- Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3224. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt