Reality Engine

Reality Engine is a hackathon prototype built at the Gemini 4 Argon SF Hackathon (Track 2, "More Dimensions"). It turns a handheld video of a real space into a measured 3D replica you can edit and simulate. OpenCV structure-from-motion, with no AI involved, recovers the camera position for each stable frame, a coloured 3D point cloud, the floor plane and the scale. Gemini 4 Argon then watches the whole video in one context and runs one agent per zone in parallel to name and place objects against those measurements. A render-and-critique loop then compares the real frames with the replica and corrects the objects. The result opens in a three.js CAD editor in the browser (gizmos, snapping, measuring, scale calibration, fitting boxes to the point cloud) and exports to glTF and MuJoCo. In this session it was extended for humanoid robot training: the reconstructed Birmingham Library scene becomes a MuJoCo world with a Unitree G1 humanoid. Obstacles come from the point cloud, and the waypoints follow the filmed walk. A Gymnasium environment with domain randomization and a PPO training script are included.

Team