# Let There Be Sims

- **Event:** [Gemini 4 NYC Hackathon](https://cerebralvalley.ai/e/gemini-4-nyc)
- **When:** Sat, Nov 21 at 9:00 AM – 10:00 PM (EST)
- **Where:** New York, NY
- **Team:** [Liyan Cheung](https://cerebralvalley.ai/u/Nailymch)
- **GitHub:** https://github.com/zglyn/Let-There-Be-Sims
- **Demo video:** https://www.youtube.com/watch?v=WHXyltqdjA8
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/49

# What Problem It Solves
Building and debugging real-world robotic manipulation policies is painfully slow and fragile. When a robot arm drops an object, slips, or clips an obstacle on a real table—or when an engineer wants to teach a new pick-and-place task—fixing it today requires manually authoring MuJoCo/Isaac XML scenes, tuning physics parameters by hand, writing reward functions, collecting hundreds of teleoperated demonstrations, and hoping the fix doesn't regress on slightly shifted lighting, friction, or object poses.
# What LetThereBeSims Does
LetThereBeSims is a multimodal Real-to-Sim-to-Real Robotics Failure-to-Policy & Training Workbench powered by the Gemini 4 Interactions API. It turns raw smartphone or multi-camera video directly into a verified 3D physics simulation, a repaired robot policy, a regression CI/CD test, and ready-to-train RL/imitation packages:
Multi-Stream & QR Mobile Intake: Upload 1 or 2 synchronized camera angles (Overhead + Wrist Cam), or scan a live QR code on a smartphone to tap any object on a table and record a 6-second hand demonstration with live DeviceOrientation gravity vectors and microphone audio.
4-Stage Gemini 4 Closed-Loop Pipeline:
Stage 1 — Carbon (Spatial Reconstruction & Retargeting): Calibrates metric tabletop geometry, obstacles, target pose, and spoken goals from video/audio—and retargets human hand paths directly onto an SO-ARM101 (or Franka Panda).
Stage 2 — Hydrogen (20-Env Domain-Randomized Swarm): Stress-tests the controller across 20 randomized environments (varying friction, mass, and pose noise) and clusters slip/collision failure modes.
Stage 3 — Neon (xhigh Adversarial Policy Repair): Iteratively patches grasp approach, grip force, lift clearance, and impedance gains until the robot completes the full pick-lift-transport-place task across 20/20 environments.
Stage 4 — Long Decode (Deployment & Training Export): Compiles deployment-ready artifacts and full training packages.
Interactive 3D Stage, CI/CD & Robot Training Studio:
Live Three.js Dual-View 3D Stage (Overhead + Wrist Cam synchronized) with interactive physics nudging, 20-Env Swarm telemetry, 2D Spatial Map, and Policy Diff.
Failure-to-Regression Suite CI/CD: Save any real-world failure as a permanent regression case and export a GitHub Actions CI workflow so future policies never repeat the same mistake.
Train & Export Studio: One-click .zip export of a Gymnasium + MuJoCo MJX RL Training Package (env.py, mjx_env.py, train_rl.py, scene.mjcf.xml) and a Synthetic LeRobot v2.1 Episode Dataset for training ACT, Diffusion Policy, or SmolVLA—plus a Cloud Swarm connector for external FastAPI/Modal GPU MuJoCo workers.

## More from Gemini 4 NYC Hackathon

- [Glint](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/46)
- [Mozart](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/47)
- [Overheard](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/48)
- [gridlocked](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/50)
- [GridSafe (67Hz)](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/51)
- [googol](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/52)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/49. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt
