foresight-worlds
Gemini 4 NYC HackathonForesight shows a robot 3 seconds of real video and predicts the next 5, by simulating physics instead of generating pixels. The problem: video models make things look right, not move right. Given the same 3-second clip of a tennis ball rolling across a table, Gemini Omni made the ball vanish and a new one roll out of the pipe, and Genie 4 rolled it back the wrong way. A robot can't plan with that. How it works: 1. Gemini 4 Hydrogen watches the clip (agentic video, structured output) and identifies each object, its real size, the surfaces it moves on, and the type of motion. 2. We track the object and convert pixels to metres using its known size. 3. A 2D physics engine is fitted to the motion we observed. 4. Gemini Decision 1 checks the fit against the real frames and returns a confidence. Below 0.7, Gemini 4 Carbon diagnoses what went wrong and proposes a different motion model, and we refit. 5. The physics runs 5 seconds forward. The output is a video, a trajectory in metres, the fitted physics (speed, friction, bounce), and a confidence. Live demo: record 8 seconds on your phone. The app sees only the first 3, streams every step as it happens, then reveals Reality, Foresight and Gemini Omni side by side. In our demo run, Foresight lands 7.3 cm from the real ball; Omni lands 36.6 cm away. Results on 15 real Physics-IQ clips, scored with the official metric code in one clean run: Foresight 59.0, Genie 4 42.2, Gemini Omni 27.5, more than twice Omni's score. We report the full picture: a straight-line baseline built on our same Gemini perception scores 75.2, because our physics still breaks on collisions and falls, and every failure is documented in the repo. Where the fit holds, Foresight is more precise than that baseline on 8 of 14 clips. Why it matters: world-model builders get a physics-checked reference to test video models against. Robotics teams get real speed, friction and bounce from a phone clip, ready for their simulator. And a robot gets a prediction with a confidence it can act on.





