Skip to Main Content

foresight-worlds

Gemini 4 NYC Hackathon

Foresight shows a robot 3 seconds of real video and predicts the next 5, by simulating physics instead of generating pixels. The problem: video models make things look right, not move right. Given the same 3-second clip of a tennis ball rolling across a table, Gemini Omni made the ball vanish and a new one roll out of the pipe, and Genie 4 rolled it back the wrong way. A robot can't plan with that. How it works: 1. Gemini 4 Hydrogen watches the clip (agentic video, structured output) and identifies each object, its real size, the surfaces it moves on, and the type of motion. 2. We track the object and convert pixels to metres using its known size. 3. A 2D physics engine is fitted to the motion we observed. 4. Gemini Decision 1 checks the fit against the real frames and returns a confidence. Below 0.7, Gemini 4 Carbon diagnoses what went wrong and proposes a different motion model, and we refit. 5. The physics runs 5 seconds forward. The output is a video, a trajectory in metres, the fitted physics (speed, friction, bounce), and a confidence. Live demo: record 8 seconds on your phone. The app sees only the first 3, streams every step as it happens, then reveals Reality, Foresight and Gemini Omni side by side. In our demo run, Foresight lands 7.3 cm from the real ball; Omni lands 36.6 cm away. Results on 15 real Physics-IQ clips, scored with the official metric code in one clean run: Foresight 59.0, Genie 4 42.2, Gemini Omni 27.5, more than twice Omni's score. We report the full picture: a straight-line baseline built on our same Gemini perception scores 75.2, because our physics still breaks on collisions and falls, and every failure is documented in the repo. Where the fit holds, Foresight is more precise than that baseline on 8 of 14 clips. Why it matters: world-model builders get a physics-checked reference to test video models against. Robotics teams get real speed, friction and bounce from a phone clip, ready for their simulator. And a robot gets a prediction with a confidence it can act on.

Team

amaan784aditp

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview