# MusePilot

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Subhajit Mahata](https://cerebralvalley.ai/u/subhajitkp007)
- **GitHub:** https://github.com/subhajitkp007/MusePilot
- **Demo video:** https://www.youtube.com/watch?v=BmQLaIPJIjk
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4398

MusePilot is an open-source, Zero-Coordinate, Visually-Verified, Self-Healing Multimodal GUI Agent built 100% from scratch during the Meta Global AI Developer Hackathon (Oct 27 – Nov 4, 2026).

THE PROBLEM:
Traditional computer-use and mobile GUI agents fail in production for three structural reasons:
1. Coordinate Hallucination & Retina Drift: Vision-Language Models asked to guess raw (x, y) pixel coordinates drift by 15–30px on Retina 2.0x desktop displays and high-DPI flagship smartphones, clicking the wrong button or adjacent destructive controls.
2. DOM Blindness: Playwright, Selenium, and accessibility-tree agents fail completely on WebGL/HTML5 Canvas apps (like TradingView Pro charts), creative tools, and cross-app native Android interfaces where 0 DOM nodes exist.
3. Open-Loop Fragility: Traditional agents fire clicks blindly without visual verification, cascading into failure whenever an unexpected modal or permission dialog appears.

THE MUSEPILOT SOLUTION:
MusePilot enforces a strict mathematical "Zero-Coordinate Invariant" via Pydantic schema validation (`StepNode`): our LLM planner is forbidden from ever emitting raw X/Y coordinates. Instead, MusePilot orchestrates a closed-loop 4-model Meta AI pipeline:
1. Ground (Meta SAM 3.1): Segments raw screen frames into pixel-accurate masks and renders high-contrast Set-of-Mark (SoM) overlays with numbered badges ([01], [02], ...) and geometric centroids normalized across OS DPI scales.
2. Plan (Meta Muse Spark 1.3): Reasons over the SoM-annotated screen and user intent (text or voice via Meta Muse Voice Transcribe 1.0) to output a DAG referencing only discrete `target_mask_id` integers.
3. Actuate (Pluggable Drivers & MCP Server): Translates mask centroids into 1:1 OS inputs across Desktop (`pynput`), Physical Android over Wireless ADB (tested live on Samsung Galaxy S25 Ultra), and an HTML5 Canvas Sandbox, plus exposes 6 tools via a Model Context Protocol (`musepilot.mcp`) server.
4. Verify & Self-Heal (Meta Muse Spark Open `meta-models/muse-spark-open-1026`): Audits pre/post ROI pixel deltas (`>=1.5%`) and visual state transitions (`confidence >= 0.80`). When an unexpected blocker modal intercepts execution, MusePilot invalidates its visual cache, re-segments via SAM 3.1, dismisses the blocker, and completes the workflow with 100% verification pass rate and 0.0px coordinate drift.

## More from Meta Global AI Developer Hackathon

- [Reel-Studio](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4395)
- [CASER](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4396)
- [JorgeosorioDev](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4397)
- [PES Solutions](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4399)
- [CircularOS](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4400)
- [Curato](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4401)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4398. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt