Skip to Main Content

**StageState — A Time-Travel AI Stage Director** What if you could step into a movie, change something that happened three scenes ago, and watch the entire story evolve around your decision? StageState is an interactive AI storytelling system that turns generated video into a persistent, editable world. Instead of generating disconnected clips, it maintains a shared world state across characters, objects, scenes, and camera views. Users become the director. They can speak with emotion, perform gestures in front of a webcam, or hold up a real-world object to introduce it into the story. Gemini interprets these multimodal inputs and translates them into changes in the virtual stage. The key feature is **time-travel storytelling**. Users can revisit an earlier scene, change an event, and create an alternative timeline. StageState preserves everything that happened before the edit, reasons about which later events are affected, and regenerates the relevant scenes while enforcing continuity rules such as object permanence, character positions, and prop ownership. To make the experience tangible, we built a real-time 2.5D stage in Three.js, viewed simultaneously through two cameras. The stage provides an interactive preview in under a second, while Omni Flash generates cinematic video with audio. A physical object shown to the webcam can also be transformed into a transparent, stylized stage prop that persists across subsequent scenes. Behind the scenes, Gemini 4 Neon directs the story, Gemini 4 Hydrogen interprets multimodal commands and repairs inconsistencies, and the Gemini Decisions API evaluates generated results. The system selects between candidate video takes, detects continuity problems, and automatically attempts corrections. We also tested the system rather than relying solely on a polished demo. Our offline evaluations achieved 26/28 passing cross-camera consistency checks, 7/8 passing branch-continuity checks, and complete preservation of unchanged scene prefixes in tested rewrites. Our goal is to make storytelling feel less like prompting a video generator and more like directing a living world—one where the past can change, but the world still remembers.

Team

yijingjiaChiriSakuraBennyLi

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview