# KHOA

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Victor Hang](https://cerebralvalley.ai/u/banhcanh)
- **GitHub:** https://github.com/Banh-Canh/nixbook-meta
- **Demo video:** https://youtu.be/mI9ZiEvd1l4
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/120

Meta Global AI Developer Hackathon · submission

# DAI, the desktop you talk to

DAI is a Linux desktop with an AI agent built in, called Muse. Muse hears you, sees your windows, notifications and calendar, and acts across your apps, running on Meta's Muse models. It is built on nixbook, an open-source NixOS configuration, forked for the hackathon.

- Muse Spark 1.3
- Muse Wonder 1.0
- Muse Voice Transcribe
- MMS-TTS
- NixOS · niri · Quickshell

## Inspiration

Most assistants live in a browser tab or a phone app, while the actual work happens on the desktop: in the windows, notifications, calls and terminals around it. We wanted an assistant that is part of the desktop itself. It should know what is on screen, act in the apps you already use, and answer fast enough to feel like part of the system rather than a chat you switch to.

## What it does

- Talk to itA speech bubble on the desktop takes typed or spoken requests. In Full AI mode the microphone stays open, a small mascot reacts and speaks the answers aloud, and every sentence is taken as an order; saying "not you" stops it when you were talking to someone else.
- Act across appsMuse moves and closes windows, launches apps, plays and pauses music and Jellyfin shows, reads and sends Discord messages, works inside browser pages, adds calendar events, tasks and notes, answers notifications, changes the theme, and reads or runs commands in its own terminal tab.
- Kiroshi scanOne key gives a briefing of what is on screen: the items that need attention, ranked by urgency, each with one-click options such as replying, adding a reminder or drafting an answer.
- Meeting notesFor any call, in any app: live captions of both sides, translation of the other language, running notes, then a summary with decisions, action items with owners and a follow-up draft.
- Smart clipboardCopy a tracking number, an address, a date, an error or a paragraph in another language, and the right action appears as a chip for a few seconds. Passwords and codes are never offered.
- Next upMuse learns your habits in plain code and offers the likely next step as one chip, such as opening Jira after the stand-up ends.
- On its ownStanding orders ("from now on, when I join a meeting, mute the music") become routines. Long requests run as background tasks or are split into parts worked on in parallel, shown as small worker figures in the bar. A journal lists what Muse did, with Undo.
- It learnsMuse keeps a memory of how to do things on this desktop, your preferences, and the corrections you give with a thumbs down, and uses them in later requests.

## How it uses Meta's models

Each job goes to the model that suits it, chosen in Settings under "Accuracy and models".

| Model | What it does in Muse |
| --- | --- |
| Muse Spark 1.3muse-spark-1.3 | The agent behind the speech bubble and the side panel: reading the context, choosing tools and answering. Also the quick one-shots (explain, translate, rewrite, suggested replies), the check that a step worked, and the question back when a request is ambiguous. |
| Muse Wonder 1.0muse-wonder-1.0 | The heavier decisions: whether a notification is worth interrupting you for, planning a request into parallel parts, writing a meeting's final record, and turning a standing order into a routine. |
| Muse Voice Transcriberealtime ASR over WebSocket | Push-to-talk, Full AI mode's continuous microphone, and live captions of both sides of a call. |
| MMS-TTSon this computer | Meta's text-to-speech voices, run locally, read the answers aloud in English or French, sentence by sentence as they stream. |

## How a spoken order runs

1. Muse Voice Transcribe turns speech into text as you talk, and ends the turn when you pause.
2. Plain commands such as "pause the music" or "volume 30" are handled in code, with no model call at all.
3. Anything else goes to Muse Spark with a short context: the time, the open windows, what you did lately, and what Muse remembers about the request.
4. Tools start as soon as each call has streamed, and Muse gives its answer together with its last step, so a simple order takes one model call.
5. The answer is shown in the bubble and read aloud by MMS-TTS while it is still streaming.

## What it's built on

DAI is a fork of nixbook (github.com/didactiklabs/nixbook), an open-source NixOS configuration we already used every day, made at the start of the hackathon on 25 October 2026. Its desktop shell, nixbook-shell, itself began as a fork of pctrade/end4-pC, which comes from end-4's illogical-impulse.

- **Already there before the hackathon:** the NixOS configuration for five machines; nixbook-shell (the bar, widgets, notifications, themes, settings) on Quickshell and niri; nixbook-desktop-mcp with 58 desktop tools for agents, the privacy mask, the desktop memory, routines, and an initiative that spoke first. The agent behind them was Claude Code.
- **Built for the hackathon:** everything on Meta's models. That covers nixbook-meta-agent, our own agent loop that replaced Claude Code; the one Messages API client and the context broker; Muse Voice Transcribe streaming and the MMS-TTS voice; the speech bubble with instant commands; Full AI mode and its mascot; the Kiroshi scan; AI in the screenshot app; the meeting companion; the smart clipboard; Next up; standing orders, the trust ladder, the journal and background tasks split into parallel parts; the visible cursor; deciding what is said to Muse; reaching Zen pages, Electron apps, Discord, Jellyfin, Seerr and the terminal; stand-in names for the privacy mask; and the latency work.

## How we built it

- **nixbook-shell**, nixbook's desktop shell, written in QML on Quickshell and running on the niri compositor, which we extended with Muse's bubble, mascot, scan, captions and chips. The whole system is declared in NixOS and Home Manager, so it rebuilds identically on another machine.
- **nixbook-meta-agent**, our own agent loop on Meta's Messages API in standard-library Python: streamed tool calls, MCP servers, sessions, a bubblewrap sandbox for shell commands, a clarifying question for ambiguous requests, and a check after acting.
- **nixbook-desktop-mcp**, nixbook's MCP server for desktop agents, grown from 58 to about 70 tools: niri windows and workspaces, an exact virtual pointer, accessibility trees, and now a browser extension for Zen, a bridge into Electron apps such as Discord, and kitty's remote control.
- **A context broker** that builds each request's context within a budget, so a bubble request went from about 37k to about 11k tokens.
- **Plain code first.** Simple commands, the smart clipboard's classifier, the next-step predictions and notification triage run without a model, and the models handle what code can't.

## Privacy and safety

- Names, emails and other personal details are replaced with placeholders or stable stand-in names before anything reaches the model provider. Everything shown on screen keeps the real names.
- Every request sent to a model is listed in Settings, under what it sent.
- Desktop control starts paused and has a kill switch on Mod+Shift+Escape. Every action is written to an audit log.
- Steps that send, pay or delete always ask first. Muse can learn to stop asking for harmless steps you keep allowing, but never for those.
- No agent can widen its own permissions, and password managers, banking sites and password prompts are off limits.

## Challenges we ran into

- **Speed.** Meta's models always reason before they answer. We measured one tool step at 0.9–1.5 s at low effort and 1.9–3.1 s with no effort set, and about 0.7 s of every call is network and prompt processing. So we cut the number of calls per order instead: plain commands in code, the open windows in the context, tools ready before the first call, and the answer given with the last step.
- **Hearing the whole room.** An always-open microphone also hears you talking to other people. We built rules and a model check for whether a sentence was meant for Muse, then found that constantly saying its name was tiring, so every sentence now counts and "not you" undoes a mistake.
- **Talkative agents.** The model narrated every step and sometimes wrote notes to itself. Muse now shows only its first line and its answer, and nothing written in between.
- **Real apps.** Desktop apps rarely expose an API, so we reached them through the compositor, accessibility, a browser extension and a bridge into Electron apps.

## Accomplishments we're proud of

- It runs as a real, everyday desktop, not a demo page.
- One assistant covers voice, the screen, notifications, calls, the clipboard, apps and the terminal.
- Privacy is built into every path to the model, not added to one of them.
- Each part has its own test suite, run on every change.

## What we learned

- Fewer model calls matter more than less thinking per call.
- Plain code should go first, with the model for what code can't do.
- An assistant that acts needs visible controls: a cursor that shows where it clicks, a journal with Undo, and a kill switch.

## What's next

- More apps reached through bridges, and more requests handled in a single call.
- Muse Wonder for longer background tasks that run while you are away.
- Packaging the shell on its own, so it can be used outside this NixOS setup.

## Try it

Mod+CSpeech bubble

Mod+TTalk to Muse

Mod+Shift+TFull AI mode

Mod+KKiroshi scan

Mod+Alt+NMeeting notes

Mod+Alt+VSmart clipboard

Mod+Shift+ReturnTake the next step

Mod+Shift+EscapePause Muse

DAI is a fork of nixbook (github.com/didactiklabs/nixbook), made on 25 October 2026. Source: github.com/Banh-Canh/nixbook-meta

The diff:
https://github.com/Banh-Canh/nixbook-meta/compare/597b2280...main


The starting point of the hackaton:
https://github.com/Banh-Canh/nixbook-meta/commit/d1e3bf9c082dfa70d12fcb01eea2b3ae8ad158b6

The feature, featured in the video and here are from the fork only. We needed meta model capabilities (especially TTS and STT) to make it happen!

https://www.youtube.com/playlist?list=PLOSvIHVRLTJ8

For me, it's an idea I wanted to realize before the hackaton, and the hackaton gave me an opportunity to experiment it. Thank you for reading !

## More from Meta Global AI Developer Hackathon

- [Cleanroom](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/117)
- [Peridot Legion](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/118)
- [Fermac AI Systems](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/119)
- [NEXUS](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/121)
- [Hero dz](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/122)
- [Agent Airlock](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/123)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/120. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt