Skip to Main Content

Let There Be Sims

Gemini 4 NYC Hackathon

# What Problem It Solves Building and debugging real-world robotic manipulation policies is painfully slow and fragile. When a robot arm drops an object, slips, or clips an obstacle on a real table—or when an engineer wants to teach a new pick-and-place task—fixing it today requires manually authoring MuJoCo/Isaac XML scenes, tuning physics parameters by hand, writing reward functions, collecting hundreds of teleoperated demonstrations, and hoping the fix doesn't regress on slightly shifted lighting, friction, or object poses. # What LetThereBeSims Does LetThereBeSims is a multimodal Real-to-Sim-to-Real Robotics Failure-to-Policy & Training Workbench powered by the Gemini 4 Interactions API. It turns raw smartphone or multi-camera video directly into a verified 3D physics simulation, a repaired robot policy, a regression CI/CD test, and ready-to-train RL/imitation packages: Multi-Stream & QR Mobile Intake: Upload 1 or 2 synchronized camera angles (Overhead + Wrist Cam), or scan a live QR code on a smartphone to tap any object on a table and record a 6-second hand demonstration with live DeviceOrientation gravity vectors and microphone audio. 4-Stage Gemini 4 Closed-Loop Pipeline: Stage 1 — Carbon (Spatial Reconstruction & Retargeting): Calibrates metric tabletop geometry, obstacles, target pose, and spoken goals from video/audio—and retargets human hand paths directly onto an SO-ARM101 (or Franka Panda). Stage 2 — Hydrogen (20-Env Domain-Randomized Swarm): Stress-tests the controller across 20 randomized environments (varying friction, mass, and pose noise) and clusters slip/collision failure modes. Stage 3 — Neon (xhigh Adversarial Policy Repair): Iteratively patches grasp approach, grip force, lift clearance, and impedance gains until the robot completes the full pick-lift-transport-place task across 20/20 environments. Stage 4 — Long Decode (Deployment & Training Export): Compiles deployment-ready artifacts and full training packages. Interactive 3D Stage, CI/CD & Robot Training Studio: Live Three.js Dual-View 3D Stage (Overhead + Wrist Cam synchronized) with interactive physics nudging, 20-Env Swarm telemetry, 2D Spatial Map, and Policy Diff. Failure-to-Regression Suite CI/CD: Save any real-world failure as a permanent regression case and export a GitHub Actions CI workflow so future policies never repeat the same mistake. Train & Export Studio: One-click .zip export of a Gymnasium + MuJoCo MJX RL Training Package (env.py, mjx_env.py, train_rl.py, scene.mjcf.xml) and a Synthetic LeRobot v2.1 Episode Dataset for training ACT, Diffusion Policy, or SmolVLA—plus a Cloud Swarm connector for external FastAPI/Modal GPU MuJoCo workers.

Team

Nailymch

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview