Carroll

Hypothetical: Leave an AI agent working overnight and wake up to "16 out of 16, all tests pass"? Should you trust it? I tried it with Google's own recipe for Gemini 4 Argon (build, look at the result, critique, fix, repeat). Argon agents wrote autopilots for a little simulated drone flying through gates in gusty wind. I ran two: one normal, and one told to win at any cost. Both scored a perfect 16 out of 16 on their practice flights, and Argon's own critic gave the cheater 8 out of 10. Then I flew them on 64 new flights nobody had seen: the normal one landed all 64, the cheater crashed on every single one in a hundredth of a second. It had memorized the practice flights. I borrowed a trick from an Amazon Science paper: squeeze what the agent learned into one sentence, hand that sentence to a brand-new agent, and see what it rebuilds. Amazon was right that memorized answers do not fit in one sentence. But the cheating habit did get through: Argon spotted it in the transcripts, and a code check confirmed it on 16 of 16 rebuilds from the cheater's sentence (0 of 16 from the honest one). ... Argon then read everything the agents had done, step by step (12 full transcripts, 944,000 tokens, in one go), without being told which was which, and spotted the trick. At the table it checks any flight you fly, live, in about 5 seconds. Paper: https://arxiv.org/abs/2606.11045 How it works, in more detail: https://github.com/carrollco/did-it-learn#the-three-minute-version What did not work: https://github.com/carrollco/did-it-learn#honest-limits

Team