Hyperpod

HyperPod is a distributed LLM inference system that allows multiple consumer machines to work together to run language models. The project focuses on two challenges: using the available GPU resources across different machines and keeping inference running when one of those machines goes offline. It uses llama.cpp RPC to distribute model execution across machines, with a Go-based controller managing workers, health checks and inference requests. Rust-based worker agents handle communication with the controller, while a dashboard provides visibility into the system. The main focus is fault tolerance. When a worker becomes unavailable, HyperPod detects the failure, rebuilds the inference pipeline with the remaining available workers and retries the request. The goal is to make better use of existing consumer hardware instead of depending on a single high-end machine to run LLMs.