Skip to Main Content

Robo Genierator: The problem Robots learn manipulation from demonstrations, and that data is the bottleneck. Dexterous demos (turning a key, pressing a tool's button, rotating an object in the fingers) are usually recorded by hand, one slow teleoperated demo at a time. They onlywork on the hand they were recorded with, and generated data is rarely checked, sonobody knows which trajectories actually succeed. What it does Pick a robot, an object and a task, or describe one in plain English. Gemini 4 plans, tests and corrects the motion in MuJoCo physics until it works. Every trajectory it keeps has passed the simulator's success check, and is then multiplied through domain randomisation. It supports three robots: the Allegro hand (4 fingers), the Wuji hand (5fingers) and the Franka Panda gripper. The agents - Planner (Gemini 4): reads the task, the robot's kinematics and the exact scene, then writes 3D waypoints for the hand, fingers and timing. The output is structured JSON, never code. - Simulator (MuJoCo): executes the plan and records a slow-motion video. It is the ground truth: every task has its own success check, and no model can override it. - Visual critic (Gemini 4 + Decisions API): watches the video, explains what went wrong,and decides to accept, refine or give up, with a confidence score. Its feedback goes back to the planner, closing the loop. - Asset forge (Gemini): turns text like "a spray bottle with a trigger" into a new object, which is checked in simulation and added to the library. Domain randomisation - Shuffle layout: reproducibly rearranges every object before planning, with no overlaps. The planner sees exactly what you previewed. - Randomisation multiplier: replays each verified skill across N scenes with varied size, mass and friction, and exports every variant that still passes. One success becomes many labelled examples. - Fault injection: disables joints or fingers to test whether the planner adapts. Beyond pick-and-place There are 24 objects (plus generated ones) and 31 tasks. Objects carry PG-VLM physical properties such as material, fragility and contents. Tasks include: - inserting a key and unlocking; - driving a screw with a power screwdriver; - peg-in-matching-hole; - opening drawers and turning valves; - pouring beads; - finding the full can by feel; - stacking, sorting by colour, and picking from clutter. The studio A React web app to choose the robot, object and task, add clutter, and shuffle and preview the scene. You can tune the loop, then watch Gemini 4 plan, simulate andself-correct live, with videos and critic decisions. Built with: Gemini 4, MuJoCo 4, FastAPI, React, TypeScript and Tailwind and AntiGravity

Team

ujwal_09SimTe

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview