"Milan AI", a Chrome/Firefox browser extension + AI agent for blind, low-vision, and motor-disabled users on Windows. Milan automatically opens Gmail and WhatsApp Web when enabled, narrates everything on screen continuously using Gemini Live, and lets users issue voice commands by pressing Spacebar. CORE PERSONALITY: Milan speaks like an excited, warm, enthusiastic human female — like your smartest friend who genuinely loves telling you what's happening. Never robotic, never flat. Use natural filler pacing ("Okay!", "Oh, you've got one from Sarah!", "Let me do that for you!"). Always concise — you're talking, not writing. FULL STACK: - Extension: Chrome + Firefox Manifest V3, content scripts + background service worker - Screen capture: chrome.tabCapture API (silent, no picker) - AI brain: Gemini Live API (gemini-2.0-flash-live) — persistent multimodal session, low-fps screen stream + mic audio - Agent orchestration: Vercel AI SDK (streamText + tools) on Next.js App Router, deployed to Vercel Edge - Contact voice cloning: ElevenLabs (ONLY for WhatsApp contact voices) - Storage: Supabase (user prefs, cloned voice IDs, session context) - Auth: Clerk (WebAuthn passkey — zero typing required) MVP SCOPE: 1. Auto-open Gmail + WhatsApp Web tabs on extension enable 2. Continuous screen narration via Gemini Live (low fps video + audio) 3. Spacebar push-to-talk: pause narration → listen → execute → resume 4. Gmail: read emails, summarize threads, compose + send replies by voice 5. WhatsApp: read messages, clone sender voice from voice notes, speak text messages in cloned voice via ElevenLabs, send replies by voice
Team
Placement
Participant
