# PocketLLM

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Mingoo Jung](https://cerebralvalley.ai/u/Looming41)
- **GitHub:** https://github.com/Looming41/pocketllm
- **Demo video:** https://youtu.be/_dJ76HwzCwM
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2526

PocketLLM runs large Mixture-of-Experts models (30B to 100B+) fully offline on laptops that don't have the RAM for them, with the weights on one or more USB drives.

Problem: organisations that must not leak data (defence, public sector, regulated industry) can't send prompts to a cloud LLM, and the laptops they actually have can't hold a 30B+ model in memory. With stock llama.cpp, a model bigger than RAM is read page fault by page fault and decode collapses to well under 1 token/s.

What I built:
- Expert streaming engine (ggml-pocket): a ~500-line patch to llama.cpp's CPU MoE kernel. It reads only the experts the router picked, as soon as it picks them, with concurrent large reads. Output is token-identical with the engine on and off (checked in CI).
- Multi-drive striping: expert tensors spread over several drives and read in parallel.
- Planner and measured tuner: before downloading, it says whether a model fits, where each part lives (GPU / RAM / streamed), and an upper bound on speed; llama-bench then measures the real speed.
- Air-gap mode: cloud features disabled by policy, signed manifest verified before anything runs, per-session key for the local model, metadata-only audit log.
- Cluster mode: pools memory of several machines over llama.cpp RPC.
- One-command Mac installer and a dashboard that shows plugged-in drives, per-drive read speed and a model picker.

Results (Linux CI lab, Qwen3-30B-A3B, 18.6 GB, 6 GiB RAM, drives throttled to 300 MB/s): stock llama.cpp 0.08 tok/s. PocketLLM default (Q3_K experts) 1.40 tok/s on 1 drive (x17.5) and 1.96 on 2 drives, with mean KLD 0.065 vs the original. Settings that lose more than KLD 0.15 are not offered by default.

Honest scope: a USB drive adds capacity and read bandwidth, not compute. Most numbers come from an emulated low-RAM machine with throttled drives; measurements on real laptops with real USB drives, and on Llama 4 Scout, are still to do.

## More from Meta Global AI Developer Hackathon

- [Heptapod](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2523)
- [Fugees](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2524)
- [Scynchro API](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2525)
- [VibeFlow Inspector AI](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2527)
- [Gtools](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2528)
- [BasilMinTyme](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2529)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2526. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt