# FleetLLM

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Michael Li](https://cerebralvalley.ai/u/mli5)
- **GitHub:** https://github.com/Michael-L-i-1/FleetLLM
- **Demo video:** https://drive.google.com/file/d/1U3qnD25SCXQY88FvoNssHdX1CDDyR5jB/view?usp=sharing
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5156

I built FleetLLM, a harness and inference engine together. Previously you would have to configure both to have agents work locally. Now they are one single application, which is especially useful in multi-turn, agentic tasks.

We live in a local compute constrained world, where the best models we run are much smaller than what the true frontier is. However, we can squeeze so much more out of what we have!

For my demonstration, I did 3 different tasks. One is that I connected this agent harness to a Raspberry to run Muse Glimmer (a 30B model) GPQA Diamond. It scored 65% by running 16 concurrent agents over the span of a day, reaching the intelligence of gpt-o1 at the time.

I also had the Raspberry Pi with Muse Glimmer to run and build a website, which it was able to generate something high quality by coordinating multiple agents to build different sections of a website.

As a final task, I did the same website building task on my mac on 4 bit quantized Muse Glimmer, and with FleetLLM, it was able to coordinate 8 agents at 100+ tokens/s to build the website in a couple seconds. That's faster inference than normal frontier lab servers.

It's crazy to see how much token throughput we can get by optimizing our harness with agentic workflows and the underlying inference processes. As we were able to produce something amazing using a 30B parameter model, Muse Glimmer, on a Raspberry Pi which is not build for such intensive compute tasks, but we made it work!

I hope to keep pushing this frontier of development, squeezing as much out of the hardware we can while achieving results that we would have called frontier not so long ago.

## More from Meta Global AI Developer Hackathon

- [LAK](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5153)
- [dicta](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5154)
- [Magneye](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5155)
- [AtMoS](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5157)
- [Stevens](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5158)
- [Chromuse](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5159)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/5156. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt