House Rules

House Rules learns the rules of a board game by watching a video of people playing it. It works in three steps. SAM 3.1 finds the board and tracks every piece frame by frame, and House Rules turns that into a move log: who moved, from where, to where. Muse Voice Transcribe writes down what the players say, with timestamps, so a spoken rule counts as evidence just like a move. Muse Wonder then writes the rules as a small program, and a checker replays every observed move against it. Every move that does not fit goes back to Wonder with the board position and what was said at that moment, until the program explains the game. On a YouTube video of an uncle teaching a kid the Turkish game Çin daması, the first program let pieces move backward. The uncle had said you can't, except inside the opponent's corner, so Wonder rewrote that rule. The final rules explain 80 of 81 moves. The one left over came after the game was already won, and House Rules says so instead of inventing a rule for it. It then matches the rules against known games, names the game as Ugolki, states the family's twist in one sentence, and turns the result into a playable game with an AI opponent and a printable rulebook. The same checker also works in reverse. On an Italian draughts video that breaks the rules on purpose, commenters found three mistakes. House Rules found seven, each pinned to its frame and checked by hand. On 29 real Connect Four games from a public research dataset, it caught all 196 illegal moves I planted. What makes this hard is that nothing is labeled: raw phone video, hands over the board, speech in another language. It holds up because every rule has to survive every observed move.