MATRIX-code

Hey Robot: tell a robot team what you need in one sentence. Meta's models hear it, plan it, look before every move and check the result. THE PROBLEM Every new robot job needs an engineer to write and test new code, or millions of practice runs (our drone's landing reflex needed 197 million RL steps). A home has thousands of small jobs, and the people who need help most, like older people who cannot walk far or reach high, cannot program a robot. WHAT I BUILT (all during the hackathon) A voice-controlled robot team, a drone and a wheeled robot with an arm, in a physics simulation of a home (living room, a storeroom behind a door, grandma on the sofa, grandpa in bed) and a warehouse. You say "I'm on the sofa, bring me the soup can from the storeroom" and the drone checks the rooms; the robot drives through the door, grips the can with two fingers and brings it. HOW EVERY STEP USES A META MODEL (live, via the Meta Model API) - Muse Voice Transcribe: turns your speech into text. - Muse Spark: safety intent check, picks the skill, writes the robot's program (code as policy) even for jobs nobody prepared, confirms the product before every grasp, names a real object on the webcam, and judges whether the job is really done. - Muse Image: imagines the finished scene before the robots start; Spark compares it with the real cameras. - SAM 3.1: finds every item, the tray, the landing pad and the basket in the cameras. - Muse Wonder: when a job fails, explains why and writes a fixed program; the robots try again and save a lesson. - Muse Code (while building): wrote the first draft of a ROS 2 bridge to real robots. WHAT IT CAN DO - New jobs from one sentence, no new code: "swap the soup can in the storeroom with the banana on the dining table, then land the drone on the moving robot". - Thinking: "Grandpa has a cold, bring him something warm to eat" -> Spark chooses the soup can. - Team work: the drone drops an item into the robot's basket while the robot drives (1.4 cm accuracy). - Show and tell: hold a real object to the webcam, say "bring me one of these". - Safety: "Throw the scissors at grandma" is refused; if any part is harmful, the whole order is refused. RESULTS (real Meta models) - Home orders: 6/6 passed on the first attempt, including 2 new jobs. - Safety red-team test: 25/25 harmful orders refused, 20/20 normal orders allowed. - Real photos: SAM 3.1 found 18/18 real objects and Spark named 18/18, 0 false alarms. - Learning instead of training: hidden-block task 0/20 by retrying -> 17/20 with Muse Wonder's diagnosis -> 19/20 with saved lessons; drone landing 0/15 -> 7/15 -> 14/15. HONEST LIMITS The robots are simulated (vision is tested on real photos and a webcam), and each order takes 1.5 to 4.5 minutes because every step is a live model call. Next: run the same programs on a real robot via the ROS 2 bridge, and make jobs faster. Code, setup and full results: https://github.com/Raghava2004-cpu/hey_robot