WorldGym

WorldGym turns a robot video into a working physics simulation of the same task. Turning a real robot demonstration into a simulation typically takes days of manual work: reconstructing the scene, sourcing or modelling objects, calibrating the camera, and scripting robot motion. WorldGym automates this from a single video upload. Gemini 4 Argon analyzes the full video in one pass, identifying the robot, objects, and action sequences. It verifies robot identity against rendered models, retrieves 3D assets through NVIDIA SimReady, reconstructs the scene, and extracts each arm's actions: what it holds, where it moves, and how many strokes it performs. In parallel, 16 Gemini Robotics-ER agents annotate grippers and objects frame by frame. Camera calibration uses the robot's own silhouette, even for handheld footage. WorldGym constructs the scene in MuJoCo and iteratively refines it across three generations. Argon compares the original video against four synchronized simulation views, identifies discrepancies with timestamped critiques, and patches scene geometry and robot motion. A locked evaluator accepts changes only when they improve the score. On a real handheld video of two robotic arms wiping a bar, simulated wipe coverage improved from 43% in Gen 1 to 84% in Gen 3.

Team