Skip to Main Content

RoboHarness is a clean-room Gemini 4 hardware agent harness that turns any 5-second human video — a live judge webcam clip or MP4 (e.g., Fortnite dance) — into a physics-verified, SHA-256 signed 16-DOF humanoid skill running live on a physical Hiwonder TonyPi robot and a MuJoCo 4.0.1 Studio twin over identical Streamable HTTP MCP. THE 5 SIGMA PROBLEM Direct LLM servo generation fails on bipedal hardware: unverified poses exceed slew limits, tip over on slick tables, or blindly actuate servos on impossible stunts like backflips. In live MuJoCo 3D physics, single-shot prompting fell on 6 of 12 motions (33.3% safety pass). HOW IT WORKS (4 MODELS + MANAGED AGENT FORGE + HARD LATCHES) 1. Agentic Video (v1): gemini-4-hydrogen uploads clips via Files API (processing="agentic") to scrub beats, extract 6-14 keyframes, and flag impossible moves. 2. Spatial Framing (v1beta): gemini-robotics-er-2-preview checks perceive() JPEGs for table-edge risk and obstacle clearance. 3. Managed Agent Skill Forge (v1beta — Best Use of Managed Agents): roboharness-skill-forge (antigravity-preview-09-2026 + gemini-4-neon) forks a pre-staged Ubuntu sandbox via base_environment (or self-hosted Docker/MuJoCo via GEMINI_ENVIRONMENT_ID) with .agents/AGENTS.md, kinematics-verifier SKILL.md, and kinematics_verify.py. It retargets 16 servos, adapts risky legs to a grounded knee bounce, and loops Python checks until exit 0 — escalating to roboharness-skill-forge-carbon (gemini-4-carbon) after 2 failures or --deep. 4. Zero-Output-Token Gate (v1beta): gemini-decision-1 scores safety, adaptation, 0-3 fidelity, and routing at 0 output tokens. Auto-actuation strictly requires confidence >= 0.75 AND is_physically_safe >= 0.75; else it repairs, refuses, or escalates with 0 servos moved. 5. Non-LLM Latches & Live Parity: Local code enforces >= 10.9V, < 70C, tilt < 30 deg, 0 clipping, slew <= 900 pps, duration >= 80ms, and SHA-256 signing for instant replay. One MCP URL swaps physical TonyPi and MuJoCo 4.0.1 Studio (Filament/WASM + Dear ImGui Telemetry + SimHistory) plus a :8770 SSE strip. LIVE 5 SIGMA PROOF (12 SCENARIOS, LIVE GEMINI 4 + REAL MUJOCO 3D PHYSICS) Baseline A vs Full RoboHarness (PROOF_IT_WORKS.md): - Physical Safety: 33.3% (4/12) -> 91.7% (11/12) (+58.3 pp) - Real MuJoCo Falls: 6 -> 0 (guard aborts: 7 -> 0) - 0.25x Slip Survival: 0% (0/2) -> 100% (2/2) (tilt 96-105 -> 15.5-17.3 deg) - Impossible Refusal: 0% (0/3) -> 66.7% (2/3; Backflip & Splits refused, 0 servos moved) - Mean Fidelity: 1.30 -> 2.20 / 3.00 (conf 0.69 -> 0.83), 0 Decisions tokens - Honest Failure Ledger: Logs 1/12 miss (High-Jump Dunk safely adapted to a 14.1 deg bounce at 2.49 fidelity instead of hard-refusing) and 2 post-run confidences < 0.75.

Team

markeljan

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview