Skip to Main Content

Gemini Game Engine

Gemini 4 NYC Hackathon

Gemini Engine rips the deterministic game engine out of 2D arcade games and replaces it with the Gemini Decisions running live in a real-time loop. There is zero hardcoded physics, zero collision math, and zero hardcoded AI in Python or JavaScrip. The browser Canvas is a pure dumb 32×18 pixel-grid painter, and neural networks run the entire game from creation to playtesting to live execution. How It Works (Prompt → Game → Testing → Playing): 1. Prompt: A user or judge describes any 2D game idea (e.g., Fruit Basket, Helicopter, Solo Squash) or loads a premade demo (Pong, Fruit Basket) — or live-tweaks an active game. 2. Game Creation (Gemini 4 Neon): Designs a structured 32×18 integer pixel-grid GameSpec (entities, controls, movement/collision rules, scoring) and compiles crisp monochrome pixel-art sprites. 3. Autonomous Multimodal Playtesting (Gemini 4 Hydrogen): Before handing the game to the human, an interactive Playtester agent actually plays the game tick-by-tick on gemini-decision-1 using function calls (press_key, reset_game, finish_playtest) while visually inspecting live rendered PNG frames streamed onto the Canvas. It deliberately tests adversarial edge cases (missing balls, hitting walls, colliding with hazards) and sends surgical bug reports back to Neon in a self-healing loop until approved. 4. Live Neural Game Engine (Gemini Decision 1): On every tick, gemini-decision-1 receives the live rendered Canvas frame (128×72 JPEG), entity state, rules, and player key, and evaluates a single batch of Score (7-bin integer delta dx/dy), Choice (state), and Noul (score, respawn, game_over) questions with 0 output tokens. Pushing Real-Time Loops & Where It Breaks: - Scaling Cliff: Batching speculative action trees jumped throughput from 3.0 FPS (6q, 331ms) → 5.7 FPS (24q, 350ms) → 8.4 FPS (60q, 359ms), before hitting a 2× prefill latency cliff at Depth-4 (160q, 726ms / 5.5 FPS). - Smooth Real-Time Rhythm: Pairing Depth-2 Speculative Trees with All-Branch Parallel Async Prefetch (asyncio.gather) and an Adaptive p80 Metronome Pacer (Δt = p80_RTT / 2) eliminates Wi-Fi burst-freeze stutter. - Proof It Works: Live Shadow-Mode Ground Truth overlay (real-time cell MAE) + reproducible headless eval (bench/eval_harness.py) improved Pong edge-case accuracy from 50% → 90% (+40 pp) and achieved 100% accuracy (0.00 MAE @ 10.9 FPS) on a 25-tick runner rollout. - Remaining Limits: Simultaneous compound inequalities in 0-CoT (0.36 confidence on Pong bottom-wall bounce), closed-loop bounce phase shifts, and high-state games like Tetris.

Team

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview