Muse for Inference

The main goal of the project was to run Bonsai 2 family of models at SOTA performance on my S26 which a latest Android Flagship phone. We achieved that! Intermediate performance writeup: https://apurva-mishra.com/posts/8/ This intermediate writeup does not reflect the current peak performance of 120tok/s. It was done because Chris Lattner had appreciated the project and we wanted to capitalise on that opportunity. With the latest updates, on S26, it now runs (tg128 benchmark): 1. Bonsai 2 2B at 120.84 tok/s which is 2.33x of llama.cpp baseline. 2. LFM3 1.7b at 46 tok/s which is 8% faster than llama.cpp baseline. In preparation for the hackathon I worked on adding an android target to the modular compiler stack. https://x.com/mav3ri3k/status/2113730606215397523?a=0oWCiuzOQDeSPAqwKBc&s=20 This was done because earlier the plan was to use my own compiler stack, but since modular open sourced their compiler just days before the hackathon, I thought that this would be the perfect opportunity to use a more mature compiler stack and push agents on. During the hackathon, the main job was about delivering SOTA model inference performance on my S26 with the latest Qualcom Snapdragon 8 Elite Gen 5. The target set was 120tok/s since that was the reference speed I had for the model on Iphone 18 Pro (https://x.com/adrgrondin/status/2110852239690473712?a=Q6uOLHRdMHbxs6LgiHk&s=20) During the hackathon, we went from Bonsai 2 2B initially running at 40tok/sec (below the llama.cpp baseline) to 120tok/s.