Skip to Main Content

GetUp: turn any phone video into humanoid robot motion Teaching a humanoid a new task usually takes motion-capture suits, calibrated cameras or hours of teleoperation. GetUp needs one casual phone video, filmed from any angle with no calibration. Film someone picking up a bottle, moving a chair or walking across a room, and a simulated Unitree G1 with three-fingered Dex3 hands does the same thing in MuJoCo. It's rendered from the same camera viewpoint and in the same room. Gemini does the understanding in two closed loops: Environment. Gemini watches the video and rebuilds the scene: tables and counters, the object's type, size and starting pose, the camera's position and field of view, and the person's starting posture. We render a still from the estimated camera and put it next to the real first frame. Gemini compares the two and corrects the scene until it matches, and then the scene is locked. Motion. Gemini writes timed keyframes for the pelvis, trunk, legs and arms, plus when each hand grasps and releases and where the object ends up. Our replay smooths the keyframes, keeps the feet on the floor, uses 7-DoF arm IK to put each palm on the object, curls the fingers until they touch it, and checks for collisions ("right hand inside table 4 cm, 4.2–5.7 s"). Gemini then watches a time-synced side-by-side video of the human and the robot, reads the telemetry, scores how closely they match and revises the motion. If a round gets worse, the loop goes back to the best version so far. On our benchmark of raw phone videos, the best runs reached 8.5/10 on both the bottle and chair tasks, with grasp and placement errors under 2 cm. A web capture site walks people through a library of tasks, from calibration poses to walking, jumping and handling objects. A worker runs each upload in the room the person filmed in (hackathon room, kitchen or warehouse) and posts back the robot video, the side-by-side and the score. Next: physics-based RL policies with domain randomization, so the motions can move from simulation to a real robot.

Team

JBurrelliva585

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview