Show & Tell Robotics

Show & Tell Robotics is test-driven robot programming: show a robot a task once, say the rule in one sentence, and get a program that passed 20 randomized simulated scenes. Telling a robot what to do still means writing a program, and the person who knows the task usually can't read it. Integration generally runs four to five times the cost of the robot (MIT's Julie Shah, as reported by MIT Sloan in 2021), and every change of part or rule goes back to whoever can write the code. Show & Tell lets that person show the task once and say the rule in one sentence. SAM 3.1 tracks what moved. Muse Wonder writes pass/fail checks as plain sentences, the person approves them, and only then does it write a short program out of tested robot skills. The program runs on 20 randomized scenes in MuJoCo with a simulated Franka arm. Simulator ground truth decides, and SAM 3.1 judges each scene again from rendered video. If a scene fails, Muse Wonder gets the failed checks, any error and 8 frames, explains the failure in one sentence, and patches the program. All 20 scenes run again. In our benchmark, from the sentence alone, four of six tasks match our reference. With a demonstration clip, six of six match on the first try. Those clips are simulator renders standing in for phone video: no human demo has been filmed yet. In one real run a program failed one scene in 20 on an arm-reach limit nobody planned; Muse Wonder traced it and the patch passed its own checks on all 20 scenes. Written from the sentence alone, that run's spec put green in a different bowl than our reference, so the benchmark scores it 0/20. Who it's for: the people who already know the job, such as a line lead re-sorting parts, a lab technician, a small shop without an integrator, or an integrator handing a cell over to the customer's staff. About 542,000 industrial robots were installed in 2024 (IFR, World Robotics 2025). They read the checks as plain sentences and approve them before any code exists, and the report and attempt log are what gets handed over. That is the plan for a product; today it runs in simulation only. Next: film human demos, then a pilot on a real arm through the same skill API. Hosted demo (replays recorded runs): https://show-and-tell-robotics.vercel.app Full write-up: https://github.com/nikocallas/show-and-tell-robotics/blob/main/docs/submission.md