# PushPilot — From Sight to Action

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Jinhong Yang](https://cerebralvalley.ai/u/sunupnet)
- **GitHub:** https://github.com/JoMinsu-KU/PushPilot
- **Demo video:** https://youtu.be/LWaUo4j7uh4
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1875

PushPilot brings Meta MUSE's multimodal intelligence into physical robot tasks, connecting what a user wants with what actually happens in the real world.

A robot can complete a command without completing the task. An object may slip, rotate unexpectedly, or disappear behind the tool. Controller feedback alone cannot establish whether the user's goal was achieved. PushPilot applies MUSE's visual understanding and reasoning to this gap between motion completion and task success.

Our current prototype uses a collaborative robot to push a model car across a printed tabletop city toward three taught destinations. MUSE interprets the user's goal and road graph, reviews a paper traffic signal while the robot is stopped, and assesses observed progress and arrival. A tablet workspace brings together the live camera, AR route, 3D route visualization, execution state, and AI judgments so the operator can inspect the plan and follow the outcome.

Our physical demonstration captures a complete action-review interval: push the car → withdraw the tool → obtain a new MUSE judgment → begin the next action. This shows model interpretation becoming part of a physical workflow.

We also evaluated model choice against our own task context. In a paired 39-case offline replay, MUSE achieved 55.7% lower estimated API cost and 20.1% shorter P95 HTTP completion time than our GPT baseline, with both models passing every structured-output contract check. These are results from that replay under its recorded settings.

Our initial target users are operators running robot demonstrations and pilot tasks. We aim to extend the shared Observe → Plan → Approve → Act → Verify structure to object relocation, workspace organization, and task inspection. The current deployment is a configured tabletop site trial; broader deployment is a future objective.

## More from Meta Global AI Developer Hackathon

- [Rocky the Raccoon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1872)
- [SafeRoute](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1873)
- [Ade Yusuf Rahardian](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1874)
- [LogicCraft](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1876)
- [Nunchi](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1877)
- [Llama Guard Sentinel](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1878)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1875. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt