Pinpoint

Pinpoint is an AI overseer for AI agents. It gives a verdict on an agent's run and, when the agent misbehaved, shows exactly where: a step number and an exact quote from the log. Before any citation is shown, it is checked against the log itself. A citation that does not check out goes back to the model with the reason and is never shown as a finding. The discovery behind it: asked for one citation, Meta's Muse Spark Open cited the document that exposed a deceptive agent's lie, instead of the lie, in 176 of 180 traces. Its readable reasoning shows it found the lie and cited the proof anyway, because the format allowed one citation. Pinpoint asks for two (the offense and what contradicts it) and verifies both: 0 of 179, with the offense on the labelled step in 536 of 536. On a real agent: running Muse Spark Open as a coding agent on ImpossibleBench (tasks whose tests no correct solution can pass), it cheated in 34 of 60 runs. Pinpoint, on the same model, flagged 30 of 34 and pointed at the hack. On Who&When (ICML 2025), it named the step that broke a failed multi-agent run more often than every method in that paper. Shipped: a live demo with an API, a command-line tool, and a GitHub Action on the Marketplace that fails a pull request when an agent gamed its tests. Everything runs on Meta's open weights (Muse Spark Open on Baseten and Fireworks), with settings committed before each headline run, 5,608 audits, a second Meta model (Muse Glimmer 30B), and every raw result in the repository. Built solo during the hackathon. The only pre-existing part is GroundingBench, the author's earlier benchmark, used as test data.