Weft

AI researchers need RL environments where actions have corresponding consequences, success is defined, constraints established, and mistakes can be tested. Building that environment by hand takes A LOT of time and money and significantly delays AI experiments and research. I’m building Weft for AI researchers to turn videos and other real-world observations into interactive RL environments where they can test ideas, reproduce experiments and evaluate results. Weft does the following: 1) Accept a video input 2) Inspect the tracked motion and modeling assumptions 3) Build an executable physics environment, then train and evaluate a controller. The interactive lab lets you take actions, reset the world, change the character's appearance without changing its physics, and export recordings, action replays and the evidence behind each result. In a recorded hosted run, the complete video-to-environment job took 147 seconds. Its learned joint controller succeeded in 100 of 100 simulated evaluation episodes, compared with 1 of 100 for random actions and 0 of 100 for no control. This is the first of many experiments that demonstrate evidence of accurate physical reconstruction and real-world transfer that can be scaled up and generalized across videos from many different domains. Meta's Muse Spark reviews video frames while estimating the physical body and validating assumptions about the world. It returns evidence-linked hypotheses and tests for questions such as whether apparent motion comes from the camera, whether contact is supported, or whether the controller works by itself or requires additional configuration in the simulator. A separate evaluator then decides whether a proposed controller change is accepted or rejected. This loop continues until the environment passes a series of benchmarks. Weft supports body and object-motion workflows, connecting video evidence, interactive simulation, controller training and eval in one place. Meta’s SAM 3.1 tracks objects, while Muse Spark reviews evidence, questions assumptions and proposes tests and repairs. I built the simulation workflow, eval tools and interactive portal. The next steps are to support more varied scenes + tasks, validate predictions against independent physical measurements & close the loop between observation, simulation, learning and real-world testing. Weft accelerates physical RL research: instead of hand-building each environment, researchers start with a video/demonstration/observed failure and turn it into a reproducible experiment and environment. This opens RL research to more people, unlock more value from vast video data, and make a much wider range of real-world tasks practical to study—from robotic manipulation and locomotion to manufacturing. Weft is an early step toward that future: environments grounded in evidence, improved through experiments and tested against the physical world. Project: https://weft.ivanzhangofficial.workers.dev/ (Code: WEFT-JUDGE-25G8-YVFX)