# Robo Genies

- **Event:** [Gemini 4 NYC Hackathon](https://cerebralvalley.ai/e/gemini-4-nyc)
- **When:** Sat, Nov 21 at 9:00 AM – 10:00 PM (EST)
- **Where:** New York, NY
- **Team:** [Ujwal Konanur Vinay](https://cerebralvalley.ai/u/ujwal_09), [Simon Tebeck](https://cerebralvalley.ai/u/SimTe)
- **GitHub:** https://github.com/SimTe8/robo-genierator
- **Demo video:** https://youtu.be/Xi3C3St0W8U
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/116

Robo Genierator:


  The problem

  Robots learn manipulation from demonstrations, and that data is the bottleneck. Dexterous demos (turning a key, pressing a tool's button, rotating an object in the fingers) are usually recorded by hand, one slow teleoperated demo at a time. They onlywork on the hand they were recorded with, and generated data is rarely checked, sonobody knows which trajectories actually succeed.

  What it does

  Pick a robot, an object and a task, or describe one in plain English. Gemini 4 plans, tests and corrects the motion in MuJoCo physics until it works. Every trajectory it keeps has passed the simulator's success check, and is then multiplied through domain randomisation. It supports three robots: the Allegro hand (4 fingers), the Wuji hand (5fingers) and the Franka Panda gripper.

  The agents

  - Planner (Gemini 4): reads the task, the robot's kinematics and the exact scene, then writes 3D waypoints for the hand, fingers and timing. The output is structured JSON, never code.
  - Simulator (MuJoCo): executes the plan and records a slow-motion video. It is the ground truth: every task has its own success check, and no model can override it.
  - Visual critic (Gemini 4 + Decisions API): watches the video, explains what went wrong,and decides to accept, refine or give up, with a confidence score. Its feedback goes back to the planner, closing the loop.
  - Asset forge (Gemini): turns text like "a spray bottle with a trigger" into a new object, which is checked in simulation and added to the library.

  Domain randomisation

  - Shuffle layout: reproducibly rearranges every object before planning, with no overlaps. The planner sees exactly what you previewed.
  - Randomisation multiplier: replays each verified skill across N scenes with varied size, mass and friction, and exports every variant that still passes. One success becomes many labelled examples.
  - Fault injection: disables joints or fingers to test whether the planner adapts.

  Beyond pick-and-place

  There are 24 objects (plus generated ones) and 31 tasks. Objects carry PG-VLM physical
  properties such as material, fragility and contents. Tasks include:
  - inserting a key and unlocking;
  - driving a screw with a power screwdriver;
  - peg-in-matching-hole;
  - opening drawers and turning valves;
  - pouring beads;
  - finding the full can by feel;
  - stacking, sorting by colour, and picking from clutter.

  The studio

  A React web app to choose the robot, object and task, add clutter, and shuffle and preview the scene. You can tune the loop, then watch Gemini 4 plan, simulate andself-correct live, with videos and critic decisions.

  Built with: Gemini 4, MuJoCo 4, FastAPI, React, TypeScript and Tailwind and AntiGravity

## More from Gemini 4 NYC Hackathon

- [clipthat](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/113)
- [Iana Grytsenko](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/114)
- [SafeStep](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/115)
- [Poker Face](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/117)
- [Lang Learn](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/118)
- [Sparky Idea](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/119)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/116. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt
