# DuckyHelper

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Yiming Beckmann](https://cerebralvalley.ai/u/Espresso)
- **GitHub:** https://github.com/yimingbeckmann/duckyhelper-open
- **Demo video:** https://x.com/Ming_LLM/status/2118088204243063051
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3083

DuckyHelper is a tutor that lives on your computer. Hold fn and ask a question out loud: it sees the screen you are asking about, explains in speech, and draws directly over the relevant words, chart, or diagram. Follow-up questions stay in context, so learning happens where the material already is.

For this hackathon, we built DuckyHelper Open, a Meta-powered edition of the existing DuckyHelper product. The starting codebase supplied the native Mac overlay, screen capture/OCR, and drawing runtime. The hackathon contribution is the Meta Model API integration: Muse Spark 1.3 for the screen-aware tutoring and streamed drawing program, Muse Voice Transcribe for push-to-talk questions, SAM 3.1 for grounding marks on objects in the screenshot, and a native local proxy that keeps the learner's API key in macOS Keychain. The open edition removes the need for a DuckyHelper account or a hosted DuckyHelper inference backend.

The engineering goes beyond returning chat text. Spark emits an executable teaching program that coordinates speech and drawing; the app runs it as it streams. SAM refines the model's rough visual targets to the objects being discussed, and the learner can point and ask again. The result is a tutor beside your work, with explanations attached to what you are looking at.

Submitted to Startups / Build for Your World, and the Most Viral Project bonus. Product: https://duckyhelper.com

## More from Meta Global AI Developer Hackathon

- [ShortForge](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3080)
- [Lannister](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3081)
- [MAFA](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3082)
- [el taller de la gatitatech](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3084)
- [VaultLine](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3085)
- [Replace Consulting](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3086)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/3083. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt