MusePilot

MusePilot is an open-source, Zero-Coordinate, Visually-Verified, Self-Healing Multimodal GUI Agent built 100% from scratch during the Meta Global AI Developer Hackathon (Oct 27 – Nov 4, 2026). THE PROBLEM: Traditional computer-use and mobile GUI agents fail in production for three structural reasons: 1. Coordinate Hallucination & Retina Drift: Vision-Language Models asked to guess raw (x, y) pixel coordinates drift by 15–30px on Retina 2.0x desktop displays and high-DPI flagship smartphones, clicking the wrong button or adjacent destructive controls. 2. DOM Blindness: Playwright, Selenium, and accessibility-tree agents fail completely on WebGL/HTML5 Canvas apps (like TradingView Pro charts), creative tools, and cross-app native Android interfaces where 0 DOM nodes exist. 3. Open-Loop Fragility: Traditional agents fire clicks blindly without visual verification, cascading into failure whenever an unexpected modal or permission dialog appears. THE MUSEPILOT SOLUTION: MusePilot enforces a strict mathematical "Zero-Coordinate Invariant" via Pydantic schema validation (`StepNode`): our LLM planner is forbidden from ever emitting raw X/Y coordinates. Instead, MusePilot orchestrates a closed-loop 4-model Meta AI pipeline: 1. Ground (Meta SAM 3.1): Segments raw screen frames into pixel-accurate masks and renders high-contrast Set-of-Mark (SoM) overlays with numbered badges ([01], [02], ...) and geometric centroids normalized across OS DPI scales. 2. Plan (Meta Muse Spark 1.3): Reasons over the SoM-annotated screen and user intent (text or voice via Meta Muse Voice Transcribe 1.0) to output a DAG referencing only discrete `target_mask_id` integers. 3. Actuate (Pluggable Drivers & MCP Server): Translates mask centroids into 1:1 OS inputs across Desktop (`pynput`), Physical Android over Wireless ADB (tested live on Samsung Galaxy S25 Ultra), and an HTML5 Canvas Sandbox, plus exposes 6 tools via a Model Context Protocol (`musepilot.mcp`) server. 4. Verify & Self-Heal (Meta Muse Spark Open `meta-models/muse-spark-open-1026`): Audits pre/post ROI pixel deltas (`>=1.5%`) and visual state transitions (`confidence >= 0.80`). When an unexpected blocker modal intercepts execution, MusePilot invalidates its visual cache, re-segments via SAM 3.1, dismisses the blocker, and completes the workflow with 100% verification pass rate and 0.0px coordinate drift.