Let There Be Sims
Gemini 4 NYC Hackathon# What Problem It Solves Building and debugging real-world robotic manipulation policies is painfully slow and fragile. When a robot arm drops an object, slips, or clips an obstacle on a real table—or when an engineer wants to teach a new pick-and-place task—fixing it today requires manually authoring MuJoCo/Isaac XML scenes, tuning physics parameters by hand, writing reward functions, collecting hundreds of teleoperated demonstrations, and hoping the fix doesn't regress on slightly shifted lighting, friction, or object poses. # What LetThereBeSims Does LetThereBeSims is a multimodal Real-to-Sim-to-Real Robotics Failure-to-Policy & Training Workbench powered by the Gemini 4 Interactions API. It turns raw smartphone or multi-camera video directly into a verified 3D physics simulation, a repaired robot policy, a regression CI/CD test, and ready-to-train RL/imitation packages: Multi-Stream & QR Mobile Intake: Upload 1 or 2 synchronized camera angles (Overhead + Wrist Cam), or scan a live QR code on a smartphone to tap any object on a table and record a 6-second hand demonstration with live DeviceOrientation gravity vectors and microphone audio. 4-Stage Gemini 4 Closed-Loop Pipeline: Stage 1 — Carbon (Spatial Reconstruction & Retargeting): Calibrates metric tabletop geometry, obstacles, target pose, and spoken goals from video/audio—and retargets human hand paths directly onto an SO-ARM101 (or Franka Panda). Stage 2 — Hydrogen (20-Env Domain-Randomized Swarm): Stress-tests the controller across 20 randomized environments (varying friction, mass, and pose noise) and clusters slip/collision failure modes. Stage 3 — Neon (xhigh Adversarial Policy Repair): Iteratively patches grasp approach, grip force, lift clearance, and impedance gains until the robot completes the full pick-lift-transport-place task across 20/20 environments. Stage 4 — Long Decode (Deployment & Training Export): Compiles deployment-ready artifacts and full training packages. Interactive 3D Stage, CI/CD & Robot Training Studio: Live Three.js Dual-View 3D Stage (Overhead + Wrist Cam synchronized) with interactive physics nudging, 20-Env Swarm telemetry, 2D Spatial Map, and Policy Diff. Failure-to-Regression Suite CI/CD: Save any real-world failure as a permanent regression case and export a GitHub Actions CI workflow so future policies never repeat the same mistake. Train & Export Studio: One-click .zip export of a Gymnasium + MuJoCo MJX RL Training Package (env.py, mjx_env.py, train_rl.py, scene.mjcf.xml) and a Synthetic LeRobot v2.1 Episode Dataset for training ACT, Diffusion Policy, or SmolVLA—plus a Cloud Swarm connector for external FastAPI/Modal GPU MuJoCo workers.





