EchoSight

EchoSight: Real-Time Spatial Audio & Vision Co-Pilot for the Visually Impaired Over 285 million visually impaired individuals face daily barriers to independent navigation. While existing assistive apps provide slow, verbose scene descriptions (e.g., "a street with people and trees"), they lack real-time spatial depth, hazard prioritization, and directional awareness. EchoSight transforms live smartphone or smart-glass video into actionable spatial intelligence. Using Meta's SAM 3.1 (Segment Anything Model), the system performs zero-shot segmentation to isolate micro-hazards (such as sidewalk curbs, descending stairs, and approaching pedestrians) and personal essentials (keys, prescription bottles). These segmented masks and spatial vectors (distance, elevation, and clock bearing) are processed by Meta Muse multimodal reasoning to calculate risk heuristics and synthesize concise, imperative guidance. Alerts are delivered through 3D binaural spatial audio (e.g., "Sidewalk curb at 12 o'clock in 1.8 meters; handrail 90cm to your right"), so users hear sound cues in the exact physical direction of the obstacle. EchoSight eliminates cognitive overload and provides safe, low-latency navigational independence.