Self-Taught Trainer
The Harness Engineering & Model Wrangling HackathonSelf-Taught Trainer is a self-improving harness for an LLM that plays Pokémon Showdown (Gen 1 OU). The player, GPT-5.4 mini, is never retrained. Instead, a coach AI studies its lost games and rewrites the harness around it: its rules, its instructions, and small Python tools it writes itself, which run in a sandbox. Every change is an A/B test: the new version and the current best play fresh games side by side, the top idea is re-tested head to head to rule out luck, and it is kept only if it wins by 5+ points. Starting from one generic sentence with no Pokémon knowledge in the prompt, it taught itself Gen 1 quirks from its own losses (freeze is permanent, only one status at a time, a Hyper Beam that knocks out skips its recharge turn). Win rate against our training bots went from 52% to 71% (Run 1), and up to 80% in Run 2. Every harness version, battle, coach decision and lesson is a document in MongoDB Atlas. Lessons are auto-embedded with Voyage and recalled by vector search before each coach decision. We also tested it honestly on the PokéAgent Challenge ladder (NeurIPS 2025) against opponents it never trained on. The gains didn't carry over yet (13% to 14%), which taught us the key lesson: a self-improving agent is only as good as its sparring partners. The dashboard shows the real Showdown replay next to the player's reasoning and the coach-written tool hints behind each move.
