Imageek

**Jolly** (developed under the codename Nubi) is a voice-first mixed reality AI companion running on Meta Quest 3. Unlike traditional AI assistants, Jolly shares your view of the physical world. It can recognize objects around you, have contextual conversations about what you're looking at, highlight real objects using AI-powered segmentation, and interact with you through movement and expressive actions. **The problem** Traditional AI assistants can explain how to use something, but they cannot easily show you which real-world object they're talking about. Whether you're learning to use an unfamiliar device, exploring your surroundings, or need hands-free assistance, switching between a physical task and a phone screen interrupts the experience. Jolly brings AI assistance directly into your environment. **What we've built** Our working Meta Quest 3 prototype demonstrates: - Natural voice conversations with an expressive 3D companion. - Visual understanding of real-world objects through the headset camera. - Contextual questions and answers about the user's surroundings. - Object segmentation and visual highlighting. - Character actions, movement, dancing, and celebration. - Spatial interaction within the user's physical environment. **How it works** Jolly combines Meta's AI models with mixed reality: - **Muse Spark 1.3** handles multimodal reasoning, contextual understanding, and tool-driven actions. - **SAM 3.1** enables segmentation of real-world objects. - **Muse Voice Transcribe 1.0** provides streaming speech recognition. - **Wit.ai** powers spoken responses. - **Unity 6, OpenXR, and Meta Quest APIs** bring Jolly into the user's room through passthrough, hand tracking, and spatial interactions. **Our vision** We believe the next generation of AI companions will move beyond screens and become part of our physical environment. Meta Quest 3 is our working prototype. Our longer-term goal is to bring this experience to more lightweight devices, including AI glasses. The project is open source under the MIT license.