RoboHarness
Gemini 4 NYC HackathonRoboHarness is a clean-room Gemini 4 hardware agent harness that turns any 5-second human video — a live judge webcam clip or MP4 (e.g., Fortnite dance) — into a physics-verified, SHA-256 signed 16-DOF humanoid skill running live on a physical Hiwonder TonyPi robot and a MuJoCo 4.0.1 Studio twin over identical Streamable HTTP MCP. THE 5 SIGMA PROBLEM Direct LLM servo generation fails on bipedal hardware: unverified poses exceed slew limits, tip over on slick tables, or blindly actuate servos on impossible stunts like backflips. In live MuJoCo 3D physics, single-shot prompting fell on 6 of 12 motions (33.3% safety pass). HOW IT WORKS (4 MODELS + MANAGED AGENT FORGE + HARD LATCHES) 1. Agentic Video (v1): gemini-4-hydrogen uploads clips via Files API (processing="agentic") to scrub beats, extract 6-14 keyframes, and flag impossible moves. 2. Spatial Framing (v1beta): gemini-robotics-er-2-preview checks perceive() JPEGs for table-edge risk and obstacle clearance. 3. Managed Agent Skill Forge (v1beta — Best Use of Managed Agents): roboharness-skill-forge (antigravity-preview-09-2026 + gemini-4-neon) forks a pre-staged Ubuntu sandbox via base_environment (or self-hosted Docker/MuJoCo via GEMINI_ENVIRONMENT_ID) with .agents/AGENTS.md, kinematics-verifier SKILL.md, and kinematics_verify.py. It retargets 16 servos, adapts risky legs to a grounded knee bounce, and loops Python checks until exit 0 — escalating to roboharness-skill-forge-carbon (gemini-4-carbon) after 2 failures or --deep. 4. Zero-Output-Token Gate (v1beta): gemini-decision-1 scores safety, adaptation, 0-3 fidelity, and routing at 0 output tokens. Auto-actuation strictly requires confidence >= 0.75 AND is_physically_safe >= 0.75; else it repairs, refuses, or escalates with 0 servos moved. 5. Non-LLM Latches & Live Parity: Local code enforces >= 10.9V, < 70C, tilt < 30 deg, 0 clipping, slew <= 900 pps, duration >= 80ms, and SHA-256 signing for instant replay. One MCP URL swaps physical TonyPi and MuJoCo 4.0.1 Studio (Filament/WASM + Dear ImGui Telemetry + SimHistory) plus a :8770 SSE strip. LIVE 5 SIGMA PROOF (12 SCENARIOS, LIVE GEMINI 4 + REAL MUJOCO 3D PHYSICS) Baseline A vs Full RoboHarness (PROOF_IT_WORKS.md): - Physical Safety: 33.3% (4/12) -> 91.7% (11/12) (+58.3 pp) - Real MuJoCo Falls: 6 -> 0 (guard aborts: 7 -> 0) - 0.25x Slip Survival: 0% (0/2) -> 100% (2/2) (tilt 96-105 -> 15.5-17.3 deg) - Impossible Refusal: 0% (0/3) -> 66.7% (2/3; Backflip & Splits refused, 0 servos moved) - Mean Fidelity: 1.30 -> 2.20 / 3.00 (conf 0.69 -> 0.83), 0 Decisions tokens - Honest Failure Ledger: Logs 1/12 miss (High-Jump Dunk safely adapted to a 14.1 deg bounce at 2.49 fidelity instead of hard-refusing) and 2 post-run confidences < 0.75.





