Portia

Portia is a judge for AI answers about long videos. Evaluation teams compare model answers about hours of footage, and the answers often differ in one detail (a count, a colour, a caption that was never there). Checking one by hand means scrubbing through the footage, and agreement between models proves little. In our tests, one in five verdicts that three agents agreed on was wrong. Portia takes a question and two answers. Three Muse Glimmer 30B agents investigate on their own, turn by turn, with video tools we built: they search about a hundred hours of footage, read the transcripts and zoom into frames. Muse Spark 1.3 re-checks their evidence without seeing their votes, and rules. Every verdict comes with the moments it cites and an evidence clip, plus a trust level measured from how the verdict was reached. Teams can audit a whole evaluation run as a spreadsheet or through the API, and upload their own videos; the live app has Blender's open film Tears of Steel, transcribed by Muse Voice Transcribe. On 880 development runs with known answers, a majority vote of three agents is right 74% of the time, and Portia with Spark 1.3 checking every verdict is right 82.0%, at about $0.004 a case. A pre-registered test on 316 held-out contests, never used to tune anything, confirmed the Muse Glimmer jury (81.0% against 75.3% for the vote) and the trust levels ("safe to accept" was 88.6% right on held-out data and 88.7% in development). On the same held-out contests, Spark 1.3 checking every verdict was right 81.6%, about the same as Muse Glimmer alone. Of the 40 unanimous verdicts that were wrong, it fixed 16, and it broke 15 that were right. Before the hackathon there was only our video database (videos, questions and answer keys from an earlier benchmark) and its search index. We built everything else during the event, and meta_hackathon/NEW_WORK.md in the repository lists each piece in the order we built it. The live app is at https://portia.shayekh.tech. Anyone can browse and replay every recorded investigation, and the access code pomfret-9094 unlocks new questions, uploads and batch audits.