# Trying XOXO

- **Event:** [Zero to Agent: Vercel x Deepmind Hackathon NYC](https://cerebralvalley.ai/e/zero-to-agent-nyc)
- **When:** Sat, Mar 21 at 9:00 AM – 10:00 PM (EDT)
- **Where:** 220 W 26th St, New York, NY
- **Team:** [Milan Chandiramani](https://cerebralvalley.ai/u/mgkc09)
- **GitHub:** https://github.com/mkchand02/milan_ai
- **Demo video:** https://drive.google.com/file/d/1Q5VQXy9DMeDCrCfLnFYHT_pxVWfBUVi_/view?usp=sharing
- **Gallery:** https://cerebralvalley.ai/e/zero-to-agent-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/zero-to-agent-nyc/hackathon/gallery/52

"Milan AI", a Chrome/Firefox browser extension + AI agent 
for blind, low-vision, and motor-disabled users on Windows.

Milan automatically opens Gmail and WhatsApp Web when enabled, narrates 
everything on screen continuously using Gemini Live, and lets users issue 
voice commands by pressing Spacebar.

CORE PERSONALITY:
Milan speaks like an excited, warm, enthusiastic human female — like your 
smartest friend who genuinely loves telling you what's happening. Never 
robotic, never flat. Use natural filler pacing ("Okay!", "Oh, you've got 
one from Sarah!", "Let me do that for you!"). Always concise — you're 
talking, not writing.

FULL STACK:
- Extension: Chrome + Firefox Manifest V3, content scripts + background 
  service worker
- Screen capture: chrome.tabCapture API (silent, no picker)
- AI brain: Gemini Live API (gemini-2.0-flash-live) — persistent 
  multimodal session, low-fps screen stream + mic audio
- Agent orchestration: Vercel AI SDK (streamText + tools) on Next.js 
  App Router, deployed to Vercel Edge
- Contact voice cloning: ElevenLabs (ONLY for WhatsApp contact voices)
- Storage: Supabase (user prefs, cloned voice IDs, session context)
- Auth: Clerk (WebAuthn passkey — zero typing required)

MVP SCOPE:
1. Auto-open Gmail + WhatsApp Web tabs on extension enable
2. Continuous screen narration via Gemini Live (low fps video + audio)
3. Spacebar push-to-talk: pause narration → listen → execute → resume
4. Gmail: read emails, summarize threads, compose + send replies by voice
5. WhatsApp: read messages, clone sender voice from voice notes, speak 
   text messages in cloned voice via ElevenLabs, send replies by voice

---

Markdown version of https://cerebralvalley.ai/e/zero-to-agent-nyc/hackathon/gallery/52. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt