# Capy

- **Event:** [The Recursive Agents Hackathon](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon)
- **When:** Sat, Oct 17 at 9:00 AM – 10:00 PM (PDT)
- **Where:** San Francisco, CA
- **Team:** [Nelson Lai](https://cerebralvalley.ai/u/chineseman)
- **GitHub:** https://github.com/chinesepowered/hack-crusoer
- **Demo video:** https://www.youtube.com/watch?v=RyOno2s7pEo
- **Gallery:** https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/86

Capy is a team of AI capybaras that babysits multi-node GPU training jobs on Crusoe while you sleep. When a job slows down, Capy finds the cause, proves it, fixes it, and only says "done" after it has measured the recovery. Every agent runs on Crusoe Managed Inference: three Nemotron models, plus Qwen 3.8 as an independent skeptic. It wakes you only to approve a disruptive fix, or when it can't prove a cause.

Problem
A synchronous training job runs at the speed of its slowest GPU. The expensive failures are gray, not hard: one throttling GPU, a flapping InfiniBand link or a bad config change slows all 512 GPUs by 10–30% while every health check stays green. That's about 128 GPU-hours lost every hour until someone notices in the morning. And an AI agent pointed at the problem will happily say "done" when it isn't.

Solution
Capy treats "done" as something you measure, not something a model says. Before anyone investigates, Detective Capy (Nemotron 3.5 Super) writes the contract: throughput back to ≥ 97% of baseline for 200 measured steps, a root cause that explains ≥ 90% of the slowdown, and an independent skeptic who agrees. Four Field Capys (Nemotron 3 Nano) sweep GPUs, network, storage and recent changes in parallel, and the detective works the case with read-only tools.

Every diagnosis has to survive two checks. First, a deterministic proof gate removes the claimed cause from per-rank step timings and recomputes the critical path. Then Inspector Doubt, running Qwen 3.8 so its blind spots differ, tries to falsify the cause and the fix. Safe fixes run right away. Disruptive ones send you one ping with the proof and the GPU-hour cost attached. The case closes only after 200 real steps verify the recovery, and any failed check sends Capy back to investigate.

On unseen test incidents, Capy matched a strong single agent's 97% fix rate with zero false "done" claims and zero wasted drains or restarts, and its own checks caught 7 incomplete fixes before anyone could call them done. Archivist Capy (Nemotron 3.5 Lightning) turns every verified fix into a runbook for the team's shared brain.

AutoClusters replaces dead GPUs. Capy finds the slow ones, and proves it.

## More from The Recursive Agents Hackathon

- [ARC](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/83)
- [CASCADE](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/84)
- [Zen](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/85)
- [The last harness](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/87)
- [FindingNEMO](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/88)
- [Alchemist](https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/89)

---

Markdown version of https://cerebralvalley.ai/e/crusoe-recursive-agents-hackathon/hackathon/gallery/86. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt