BrickRobotProject (Kunwoo Chang)

Just Bricks: Letting Meta AI Figure Out How a Robot Moves Inspiration Most of us built things from LEGO as kids, from simple shapes up to robots. That led to a question: if an AI sees only the bricks, can it work out how a robot moves, and turn that into code that drives the real thing? 1. What it does Starting from a LEGO model file (BrickLink Studio → LDraw), the pipeline: a. Infers the structure: links, joints, rotation axes, gear trains and coupled joints. A person splits the model into links and marks the motor paths; geometry tools and Meta Muse Spark do the rest. b. Derives kinematics: forward and inverse kinematics, checked in MuJoCo and shown in a web 3D view. c. Controls the real robot: a 6-axis arm on two LEGO EV3 bricks, connected over local MQTT with homing, limits and emergency stop. You set a target in a web console and the physical arm follows. d. Trains a VLA: the inferred kinematics generate pointing demonstrations automatically; SmolVLA is trained on Modal, then fine-tuned with real webcam images and wired into the console. e. Generalizes: the same pipeline, extended to sliding joints, analyzes a different robot, a 3-axis gantry (T-Bot), and drives it in simulation. 2. How we built it * Meta Muse Spark selects joints from tool-generated candidates and self-checks gear ratios and coupling. * Meta Muse Code was used to build the whole pipeline: parser, geometry tools, IK, simulator, web console, brick firmware and VLA training scripts. * Python, MuJoCo, three.js, Pybricks MicroPython on EV3, Mosquitto MQTT, LeRobot/SmolVLA, Modal (A100) for training. 3. Results * IK in simulation: 59 of 60 targets reached; median position error 5.3 mm against measured ground truth. * Real robot: web console target → physical arm motion. * VLA in simulation: 43–46% success at pointing to the named block. * VLA with webcam: 0/15 → 15/15 after fine-tuning with 15 real photo sets, in the same lighting. Under new lighting it dropped to 3/14. * Generalization: all three sliding joints of T-Bot found, with the original robot’s results unchanged (regression-tested). 4. Challenges * Real hardware: motor directions, homing and port dropouts. * Removing model-specific hints from the prompt showed the agent found 3–4 of 6 joints on its own. We report both numbers. * An evaluation bug (one action instead of a full chunk) briefly made the VLA look broken until we traced it. * Sim-to-real: a small set of real images fits one lighting setup but does not generalize. 5. What we learned Honest evaluation matters more than headline numbers. Structured tools plus an LLM agent can recover robot kinematics from bricks. Closing the sim-to-real gap for VLAs needs varied real data. 6. What’s next Automatic link segmentation, more varied real data for the VLA, more robot types, and a STEM learning tool where students build a robot and see its logic extracted. Longer term, applying the same approach to industrial robot models. 7. Credits Robot design: 6DOF Robotic Arm by MR JOS (Rebrickable MOC-76666), modified by the author. Used with permission. Gantry Robot “T-Bot” by glaysche (Rebrickable MOC-85202), used with permission. Built with Meta Muse Spark, Muse Code, SmolVLA/LeRobot, MuJoCo, Modal and LDraw.