understudy

What’s the biggest problem with training robot arms? Data. There isn’t nearly enough of it, and most of what exists comes from teleoperation: a person steering the arms through the task by hand, over and over. We built a way to generate training data for bimanual robot arms from a single video filmed on your phone, using SAM and Muse Spark. No task description. No code. The pipeline is a full loop, from upload to a trained policy. Muse Spark watches the video and works out what each hand does and which objects matter. SAM segments both hands and every object. Muse then rebuilds the table from the video inside the simulator, plans the task and drives each arm, checking every step through the cameras. From there, the same loop moves from the simulator onto the real arms, where Muse operates them the same way. It works close loop so it adapts to what it sees. Then Muse shuffles the task: it moves objects around and tilts things, and turns every variation into new training data. That data trains a small policy on a Mac. Within hours, you go from a video on your phone to a robot that performs the same task on its own, offline.