Grasp

Robots learn from demonstrations, and demonstrations are expensive: teleoperation needs a robot, an expert, and hours. Grasp replaces that with a phone video and one printed marker. You film a task once, narrating as you go. Muse Spark names the objects from a few frames, with no class list and no training. SAM 3.1 tracks each object and your hand through the whole clip, keeping one identity per object through occlusion. The marker maps pixels to centimetres on the table, and the tracks become events: what was grasped, when, where it went, what rode along with it. Muse Spark then reads the events, the first and last frame, and your narration (transcribed by Muse Voice Transcribe) and writes down the goal in a small structured vocabulary: put all pencils in the cup, move the wallet next to the pencil, swap, stack, arrange. If two readings are close, it asks a question instead of guessing. Because Grasp holds a goal rather than a motion, it can act in scenes it never saw. A planner turns the goal into pick-and-place steps for whatever layout is in front of it, and a simulated Franka Panda executes them. On ten tasks taught by phone, Grasp completed the goal in 100 of 100 shuffled scenes (with extra objects and distractors), while a robot replaying the demonstrated motions succeeded in 1. You can also edit the goal in plain words (“put all the objects in the cup”) and it re-plans, telling you what won’t fit. Every verified run is exported as a LeRobot episode: 110 episodes from ten videos. Grasp is a prototype that uses kinematic grasping in simulation, table-plane positions, five goal types, success judged against simulator state. All of it is stated in the README, alongside the related research it builds on. Everything was built between October 27 and November 4, with Meta’s models doing the perception, the transcription, and the reasoning.