# Crumbs

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Leonard Lin](https://cerebralvalley.ai/u/randomfoo)
- **GitHub:** https://github.com/lhl/crumbs
- **Demo video:** https://youtu.be/m8X9hnQrheM
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3064

Crumbs turns the photos and location history you already have into a searchable, cited record of where you went and what you ate running entirely on your own hardware.

The problem: I have tens of thousands of photos in my iPhone photo library, a large share of them food. The default photo app cannot answer "where was the eel place I went the day I visited that garden in Kyoto?", and I did not want to upload my photo library and years of location history to a third party.

What is built (full app, all during the hackathon):

 - Import: EXIF time/GPS, HEIC to JPEG analysis copies, SHA-256 dedupe; location history from GeoJSON exports becomes dwell "stays"; deterministic grouping into visits and trips.
 - Understanding: Muse Glimmer 30B (Q4_K_M, llama.cpp) classifies every photo and reads menus, signs, receipts and chopstick sleeves for text, dishes and place-name candidates. DINOv3 embeddings catch near-duplicates and drive similarity search.
 - Places: OpenStreetMap candidates cached locally, ranked check-in > signage text > GPS proximity; GeoNames reverse geocoding
 offline.
- Ask: a Glimmer tool-calling agent plans lookups over the local index. For "the eel place the day I went to Murin-an garden" it finds the garden in photo text, builds that day's timeline of moments, picks the meal, then re-reads the sign photo at full resolution before naming the restaurant. It cites the visits and photos it used, and any citation no tool returned is removed.
- Provenance: every place, dish and date is marked observed (EXIF/GPS), inferred (a model read it), lookup (map data), or confirmed (you).
 - Surfaces: a local API, a phone-friendly map/globe PWA on fully local PMTiles (zero external requests in the map view), and a Muse
 bridge that lets a Muse agent query the archive over a private tailnet with a read-only, privacy-tiered token.

Measured on my own photos: 0.976 food F1 on 61 hand-verified food photos; all 9,669 photos classified in about 12 minutes across
 four GPUs; the Ask agent answered 8 of 8 test questions with zero unsupported citations (median 21.5 s). Glimmer fits in about 20 GB,
 so the whole pipeline runs on one 128 GB mini-workstation.

The shipped pipeline has no API dependency; it runs from local weights. The only network calls are optional OpenStreetMap Overpass place lookups at development time, cached locally afterwards.

Who else benefits: anyone with a large camera roll has the same gap - a food recommendation to give a friend, the name of the place
 from last spring, what to order again. Crumbs answers those from evidence, and because inference is local, the answer never costs you your privacy.

Food is first; hotel stays and trip stories are next.

## More from Meta Global AI Developer Hackathon

- [socialStory](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3061)
- [Elton Kadiu](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3062)
- [FibreSight](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3063)
- [Pavlo Durunda](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3065)
- [BlueberrySoda](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3066)
- [Skill Issue](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3067)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3064. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt