GPU Sitter

GPUSitter is an on call engineer that responds to failures in GPU datacenter clusters. \ We built tools for the agent to inspect NVIDIA sensor data, and power, temperature, and utilization data. From that it reasons about the root issue and informs about resolution. Self improvement: After its finished reasoning, with its expert knowledge, it updates its own small ML prediction model for failures. If it scores better than the existing ML model replaces it. PS: The we data (schema) used it real LLM training cluster logs from the AcmeTrace paper. So you can actually use our setup for GPU operations :)