# Muse for Inference

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Apurva Mishra](https://cerebralvalley.ai/u/mav_at_columbia)
- **GitHub:** https://github.com/mav3ri3k/modular
- **Demo video:** https://youtube.com/shorts/MMdXvKNWsSU?feature=share
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4825

The main goal of the project was to run Bonsai 2 family of models at SOTA performance on my S26 which a latest Android Flagship phone.
We achieved that!

Intermediate performance writeup: https://apurva-mishra.com/posts/8/
This intermediate writeup does not reflect the current peak performance of 120tok/s. It was done because Chris Lattner had appreciated the project and we wanted to capitalise on that opportunity.

With the latest updates, on S26, it now runs (tg128 benchmark):
1. Bonsai 2 2B at 120.84 tok/s which is 2.33x of llama.cpp baseline.
2. LFM3 1.7b at 46 tok/s which is 8% faster than llama.cpp baseline.

In preparation for the hackathon I worked on adding an android target to the modular compiler stack. 
https://x.com/mav3ri3k/status/2113730606215397523?a=0oWCiuzOQDeSPAqwKBc&s=20

This was done because earlier the plan was to use my own compiler stack, but since modular open sourced their compiler just days before the hackathon, I thought that this would be the perfect opportunity to use a more mature compiler stack and push agents on.

During the hackathon, the main job was about delivering SOTA model inference performance on my S26 with the latest Qualcom Snapdragon 8 Elite Gen 5. The target set was 120tok/s since that was the reference speed I had for the model on Iphone 18 Pro (https://x.com/adrgrondin/status/2110852239690473712?a=Q6uOLHRdMHbxs6LgiHk&s=20)

During the hackathon, we went from Bonsai 2 2B initially running at 40tok/sec (below the llama.cpp baseline) to 120tok/s.

## More from Meta Global AI Developer Hackathon

- [Muse Team](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4822)
- [DepaPranata](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4823)
- [opc](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4824)
- [LeoForge](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4826)
- [Björn D. Spark](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4827)
- [Icarus Studio 3D](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4828)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/4825. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt