ChronoRig

ChronoRig is an autonomous video-to-physics 3D rigging engine and verifiable agent that transforms standard monocular videos of moving mechanisms into verified, interactive Three.js/Rapier simulations and articulated Blender 4.x (bpy) physics rigs. (Built 100% from scratch during the Meta Global AI Developer Hackathon). THE PROBLEM Modern 3D AI pipelines—such as photogrammetry, NeRFs, and 3D Gaussian Splatting—treat the world as frozen, static statues. When given a video of an articulated physical object (a robotic arm, desk lamp, pendulum, or mechanical linkage), they fail to recover underlying kinematic joints, hinge pivots, or physical constraints. Engineers, game developers, and robotics researchers are forced to manually rig meshes and tune joint anchors by hand. OUR SOLUTION & ARCHITECTURE ChronoRig bridges video perception and deterministic 3D physics through a 4-stage closed-loop pipeline: 1. Spatiotemporal Part Perception (Meta SAM 3.1): ChronoRig probes uploaded MP4/WebM containers and issues single-concept streaming requests to Meta's SAM 3.1 API (sam-3.1) to isolate N distinct moving mechanical parts with pixel-accurate binary segmentation masks across frames. 2. Deterministic Kinematic Math Engine (OpenCV + NumPy): Rather than asking an LLM to hallucinate 3D coordinates from raw pixels, ChronoRig extracts simplified polygon contours, computes 2D centroids and principal axis orientations via Principal Component Analysis (PCA) with 180-degree axial angle unwrapping (unwrap(2*theta)/2), and solves parent-relative joint constraints. It classifies joints into Fixed, Prismatic Sliders (via SVD singular-value collinearity ratios), or Revolute Hinges (via algebraic Kasa least-squares circle fitting in the parent link's reference frame). 3. Closed-Loop Verifiable Agent (Muse Spark Open 1026 on Baseten): To guarantee physical fidelity, ChronoRig's verifier rasterizes forward-kinematic part polygons at keyframe timestamps and computes exact pixel Intersection-over-Union (IoU) against SAM 3.1 ground-truth masks. If mean IoU falls below our 0.85 threshold, meta-models/muse-spark-open-1026 diagnoses the per-frame residual and patches the 3D joint anchor coordinates until convergence. 4. Interactive Next.js 15 Studio & Blender 4.x Export: A 3-column studio built with React Three Fiber and @react-three/rapier lets users toggle between Observed Pose Replay Mode, Live WebAssembly Physics Mode, and Binary Silhouette Mode, or export an AST-sandboxed Blender 4.x Python (bpy) script complete with rigid-body masses, HINGE/SLIDER constraints, and baked timeline keyframes.