# Project Prometheus

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Mingyu Jeoung](https://cerebralvalley.ai/u/Mingoomato)
- **GitHub:** https://github.com/INDIETRACK-TEAM-Prometheus/project-prometheus
- **Demo video:** https://youtu.be/XFKspe-5Shk
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4739

Smart glasses can see what is in front of you, but not what you are paying attention to, where you left things, or what you are about to do. Prometheus closes that gap. It follows the wearer's eye gaze through egocentric glasses video, segments the exact object they look at, remembers where every object is in 3D, and predicts what they may do next.

How it works, in real time on recorded Project Aria footage:

- Gaze-grounded segmentation: the wearer's gaze point becomes a SAM 3.1 point prompt, so the mask is the object they are actually looking at, not every object in view. A local RT-DETRv2 detector boxes the rest of the scene and stands in when SAM is rate limited.
- Open-vocabulary naming: SAM and detector crops are tiled into one image and Muse Spark names them ("Scrabble board", "laundry basket") instead of a fixed class list. People and hands are kept out of the memory.
- 3D object memory: each object's depth (Depth Anything V2, rescaled to metres with SLAM points) and the camera pose turn it into a world point. Repeated sightings merge, and a three.js view shows the objects in the room's point cloud along the wearer's path.
- Place memory and recall: when the wearer moves to a new recording, Muse Spark compares keyframes with the places it remembers. On a match, the objects it saw there come back into the 3D memory as recalled, before they are seen again.
- Next-action prediction: every few seconds Muse Spark receives the frame with the gaze and object tags drawn on it, the object list with distances, and what it remembers from earlier visits, and answers with up to three short predictions of what the wearer may do next.

The perception loops (detection, depth, SAM at the gaze, naming, prediction) run independently, so a slow call never blocks the others. The demo plays four recordings from one home in Meta's Aria Everyday Activities dataset, so place recall works across separate visits to the same rooms.

Prometheus points toward assistive glasses that know what you are looking for before you ask: reminding you where you put your keys, noticing an unfinished task, or helping people with memory or attention difficulties. Everything was built during the hackathon; the repository is MIT licensed with architecture diagrams and full attributions.

## More from Meta Global AI Developer Hackathon

- [George Sung](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4736)
- [TEARSHEET M](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4737)
- [Three Dimensions](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4738)
- [QQS Interactive (solo)](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4740)
- [Tenet](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4741)
- [Honest Orb](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4742)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4739. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt