AOT
Built at Meta Global AI Developer Hackathon · Oct 26, 2026 · Remote

High-density GPU racks consuming 100 to 130 kilowatts, such as the NVIDIA GB200 NVL72, rely on liquid coolant loops to run. When a quick-disconnect or hose ruptures, conductive fluid can reach the 48-volt electrical busbar within 60 seconds. This causes arc flash, destroys hardware, and leads to weeks of unrecoverable downtime. Standard data center defenses fail here: passive floor leak ropes only detect fluid after large puddles accumulate on the floor, and thermal alarms only fire after silicon temperatures have already spiked. CryoGuard AI is an automated safety console that monitors the chassis using an overhead camera fitted with a 520 nm optical bandpass filter to detect fluorescent tracer dye. Four ArUco markers rectify the perspective into real-world millimeters, allowing a classical tracker to calculate puddle growth rate in real time. A Bayesian fusion engine combines visual growth with hydraulic pressure drops, flow imbalances, and psychrometric dew-point calculations to isolate leaks while rejecting condensation and metallic glints. When leak probability crosses 0.85, the controller runs a deterministic six-step isolation sequence over local Modbus-TCP and Redfish BMC sockets. It clamps GPU power to 200 watts within 100 milliseconds, shuts hydraulic isolation valves (verified by both a physical limit switch and line depressurization below 5 PSI), drains active Slurm workloads, commits checkpoints, and powers down the chassis. The BMC client polls the chassis power endpoint until an off state is confirmed. Generative models are kept completely out of the real-time physical actuation loop. After hardware isolation is verified, Meta Muse Spark Open runs asynchronously to generate a forensic root-cause analysis from the sealed telemetry ledger. If the API times out or drops, the system falls back to a deterministic rule-based incident report. The project includes 515 passing automated tests, strict static typing across 61 files, and byte-reproducible offline benchmarks.