Robot Judge

Robot Judge is an AI judge for real robot runs. You give it two videos of robots attempting the same task and it tells you which one did better, how sure it is, and why, in plain language. The problem: every time a robotics team changes a policy, someone has to watch hundreds of runs to know whether it actually improved. Success rates miss partial progress and sloppy behaviour, and human review is slow, expensive and doesn't scale. How it works: each run becomes a storyboard of 8 frames plus the final state. Meta's Muse model compares the two storyboards against the task instruction and returns a verdict, a confidence and a reason. Every pair is judged twice with the order swapped, so the judge can't just favour whichever run it sees first. How we know it works: I evaluated it on a held-out task (cup stacking) from a public benchmark of 5 robot foundation models on real bimanual robots, with 10 human votes on every pair (607 pairs, 6,070 votes). Nothing was tuned on this task. Muse Spark agrees with the human majority on **83%** of pairs (95% CI 79–86%). A single human agrees with the other nine **77%** of the time. So the judge is already at least as consistent with the crowd as a typical individual annotator. Model ratings computed from its verdicts land within 14 Elo of the human ratings on average, and both clearly identify the weakest policy. The demo shows the judge's reasoning next to the actual human vote.