Latent Futures

Problem: To act safely in the physical world, an AI system has to anticipate what will happen and what would change if something were different. Most video models predict future pixels, which is expensive and doesn't directly tell you which objects will collide or what happens if one is removed. Language models can talk about a scene, but they have no physics model to check their answers against. Project: ObjectJEPA is an object-centric world model that forecasts the future in latent space, object by object. SAM 3 finds the objects in a video, a frozen V-JEPA 2 encodes them, and a predictor I trained during the hackathon rolls them forward. A learned head then reads collisions from the forecast. Muse Spark answers questions by calling my tools: it can remove an object, re-run the forecast, or predict what comes after the video ends. Its answers therefore come from a simulation it ran, not from a guess. Results: I tested it on held-out videos from CLEVRER, a synthetic physics benchmark. It answers 97% of "what happens next?" questions correctly, against 93% for constant-velocity physics. Running from raw video with SAM 3 masks, the full Muse Spark agent also gets 97% of those questions right, plus 81% of "what if" answer options. On "what if", the counterfactual forecast adds only about a point over assuming nothing changes, and improving that is my next step. This is a step toward planning for robotics and other physical systems. So far it has only been tested on CLEVRER's simulated scenes, not real-world video. Code: https://github.com/idrissa-gif/ObjectJEPA Weights: https://huggingface.co/Idrissa242/ObjectJEPA