# RoboHarness

- **Event:** [Gemini 4 NYC Hackathon](https://cerebralvalley.ai/e/gemini-4-nyc)
- **When:** Sat, Nov 21 at 9:00 AM – 10:00 PM (EST)
- **Where:** New York, NY
- **Team:** [Markeljan Sokoli](https://cerebralvalley.ai/u/markeljan)
- **GitHub:** https://github.com/markeljan/roboharness
- **Demo video:** https://www.youtube.com/watch?v=t268LBaSmYw
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/95

RoboHarness is a clean-room Gemini 4 hardware agent harness that turns any 5-second human video — a live judge webcam clip or MP4 (e.g., Fortnite dance) — into a physics-verified, SHA-256 signed 16-DOF humanoid skill running live on a physical Hiwonder TonyPi robot and a MuJoCo 4.0.1 Studio twin over identical Streamable HTTP MCP.

THE 5 SIGMA PROBLEM
Direct LLM servo generation fails on bipedal hardware: unverified poses exceed slew limits, tip over on slick tables, or blindly actuate servos on impossible stunts like backflips. In live MuJoCo 3D physics, single-shot prompting fell on 6 of 12 motions (33.3% safety pass).

HOW IT WORKS (4 MODELS + MANAGED AGENT FORGE + HARD LATCHES)
1. Agentic Video (v1): gemini-4-hydrogen uploads clips via Files API (processing="agentic") to scrub beats, extract 6-14 keyframes, and flag impossible moves.
2. Spatial Framing (v1beta): gemini-robotics-er-2-preview checks perceive() JPEGs for table-edge risk and obstacle clearance.
3. Managed Agent Skill Forge (v1beta — Best Use of Managed Agents): roboharness-skill-forge (antigravity-preview-09-2026 + gemini-4-neon) forks a pre-staged Ubuntu sandbox via base_environment (or self-hosted Docker/MuJoCo via GEMINI_ENVIRONMENT_ID) with .agents/AGENTS.md, kinematics-verifier SKILL.md, and kinematics_verify.py. It retargets 16 servos, adapts risky legs to a grounded knee bounce, and loops Python checks until exit 0 — escalating to roboharness-skill-forge-carbon (gemini-4-carbon) after 2 failures or --deep.
4. Zero-Output-Token Gate (v1beta): gemini-decision-1 scores safety, adaptation, 0-3 fidelity, and routing at 0 output tokens. Auto-actuation strictly requires confidence >= 0.75 AND is_physically_safe >= 0.75; else it repairs, refuses, or escalates with 0 servos moved.
5. Non-LLM Latches & Live Parity: Local code enforces >= 10.9V, < 70C, tilt < 30 deg, 0 clipping, slew <= 900 pps, duration >= 80ms, and SHA-256 signing for instant replay. One MCP URL swaps physical TonyPi and MuJoCo 4.0.1 Studio (Filament/WASM + Dear ImGui Telemetry + SimHistory) plus a :8770 SSE strip.

LIVE 5 SIGMA PROOF (12 SCENARIOS, LIVE GEMINI 4 + REAL MUJOCO 3D PHYSICS)
Baseline A vs Full RoboHarness (PROOF_IT_WORKS.md):
- Physical Safety: 33.3% (4/12) -> 91.7% (11/12) (+58.3 pp)
- Real MuJoCo Falls: 6 -> 0 (guard aborts: 7 -> 0)
- 0.25x Slip Survival: 0% (0/2) -> 100% (2/2) (tilt 96-105 -> 15.5-17.3 deg)
- Impossible Refusal: 0% (0/3) -> 66.7% (2/3; Backflip & Splits refused, 0 servos moved)
- Mean Fidelity: 1.30 -> 2.20 / 3.00 (conf 0.69 -> 0.83), 0 Decisions tokens
- Honest Failure Ledger: Logs 1/12 miss (High-Jump Dunk safely adapted to a 14.1 deg bounce at 2.49 fidelity instead of hard-refusing) and 2 post-run confidences < 0.75.

## More from Gemini 4 NYC Hackathon

- [Shopexa](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/92)
- [Systemind](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/93)
- [DripVerse](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/94)
- [Sticky](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/96)
- [snag AI](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/97)
- [Visual Learning Studio](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/98)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/95. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt
