Object Dynamics VLA

From seeing the scene to moving the arm—with Muse and SAM. Object Dynamics VLA connects Meta's multimodal perception models to continuous robot control. Built for robotics developers, it combines frozen, four-bit-quantized Muse Glimmer visual-language features, SAM 3.1 object tracking and robot state. We trained a conditioning adapter and flow-matching action expert using the SmolVLA/LeRobot architecture. The policy observes the scene, generates an arm action, executes it and observes again. Our milestone: the Muse/SAM-conditioned policy completed milk-in-basket placement in LIBERO simulation after 166 actions. The frozen checkpoint succeeded in one of three sampling-seed trials from the same demonstrated training initialization. Repeatability and generalization remain open. Our contribution is the trainable perception-to-action integration, supported by aligned training data, quantized feature handling, tracking checks, resumable training and recorded execution evidence. Muse and SAM remain frozen; the robot-conditioning modules and action expert are trained. We also completed integration checks across ten grocery-placement variants, establishing infrastructure for broader training.