ActuateOS
Gemini 4 NYC HackathonActuateOS turns a video of a person assembling something into a working 3D robotic setup and tests whether a robot can actually pull it off. Watching a human do a task on camera is the easy part. Figuring out how a machine can repeat it safely is hard a video won't tell you if a robotic arm can reach a part, if pieces will collide, or if a step requires delicate human touch. ActuateOS bridges that gap automatically: It watches and learns: Using Gemini 4 Hydrogen, it breaks an assembly video down into clear steps, noting what happened and what information is still missing. It designs the workspace: Using Gemini 4 Carbon, it lays out a 3D robotic workcell in OpenIndustries and builds a step-by-step task plan. It puts the plan to the test: It runs the steps inside a real physics simulator (MuJoCo). Right now, our virtual robot arm handles standard pick-and-place moves and if a step is too complex (like tight screwing or snapping parts together), ActuateOS flags it honestly instead of faking success.





