Ariana

The goal is a virtual participant for brain research: an AI agent that does an fMRI task like a specific person, explains every move in words, and whose internal state lines up with that person’s brain activity. Studying the brain during real behavior is slow and expensive, since every new question means new scanner time; a virtual participant lets researchers test ideas on it first and scan second. Mario Mind does this on Super Mario Bros., using CNeuroMod’s dataset of people playing the game in the scanner with every button press recorded. The agent is built on Meta’s Muse Glimmer and taught in three steps. First, its perception learns how the game world changes with each action, so it is not a passive observer. As it learns from how actions update world using a world model that takes into consideration actions. Second, it learns to reason about what it sees and then makes a choice. Third, it practices with a teacher, a judge that tries to tell its moves from the person’s, until its play is indistinguishable from one participant’s, without losing its reasoning. Results, on that participant and on brain scans the model never saw: the judge’s accuracy at telling the agent’s moves from the person’s, at the same game moments, falls from 84% to chance (50%); adding the agent’s internal state and the running context of what it writes to a video model’s view of the game predicts brain activity better in 721 of 1,000 brain regions (most in salience, attention and somatomotor networks), and the gain disappears when the agent’s thoughts are shifted out of sync with the game; and the agent’s sense of danger lines up with the brain’s own danger regions. The demo shows the agent playing, what it says, and the brain activity its mind implies, live.