VibroBraille

Problem Statement 1: Real-Time Multimodal Interaction Globally, over 285 million visually impaired individuals face significant accessibility barriers. Traditional refreshable Braille displays can cost up to $2,000, while conventional Text-to-Speech solutions often compromise privacy by broadcasting sensitive information aloud. VibroBraille addresses this challenge by transforming how Braille is delivered. Instead of relying on expensive spatial pin-based hardware, our system uses Temporal Signal Synthesis for Braille Transmission (TSSTB) to encode Braille as precisely timed vibration sequences on any standard smartphone. This software-defined approach removes the hardware cost barrier while preserving privacy through tactile communication. For this hackathon, we integrated Gemini 3.5 Flash Live to transform VibroBraille from a standalone text-to-vibration engine into a real-time multimodal accessibility platform capable of continuously interpreting live camera feeds, spoken conversations, and environmental context, then converting them into low-latency haptic feedback. Our solution directly satisfies the Real-Time Multimodal Interaction track by moving beyond traditional turn-based chat interfaces into continuous, context-aware interaction. Key Features Continuous Multimodal Perception Using Gemini Live, VibroBraille simultaneously processes: * Live camera input * Microphone audio * Multilingual conversations * Environmental context This enables proactive scene understanding without requiring users to repeatedly issue commands. Real-Time Conversational Interaction Users can interrupt Gemini Live at any point, change instructions naturally, and immediately receive updated tactile feedback without restarting the interaction. Private Haptic Navigation Instead of announcing surroundings through speakers, the system converts environmental events—such as obstacles, staircases, benches, or pathways—into discrete vibration patterns, enabling private and intuitive navigation. System Architecture Our architecture consists of a three-stage low-latency pipeline designed for reliable performance across both flagship and budget Android devices. 1. Perception & Semantic Processing Gemini 3.5 Flash Live serves as the multimodal reasoning engine, continuously processing: * Camera streams * Audio streams * Conversational context A lightweight Node.js backend filters and summarizes this information into compact JSON packets optimized for haptic transmission. 2. Native Haptic Delivery A custom Native Kotlin Haptic Bridge, integrated into our Flutter application, bypasses standard vibration limitations to provide fine-grained actuator control. Using persistent WebSocket connections, the system achieves an end-to-end latency floor of approximately 18 ms. 3. Temporal Signal Encoding Each Braille character is encoded into six temporal slots of 100 ms each. Carefully designed orthogonal pulse patterns provide clear tactile separation, minimizing sensory overlap while matching the sensitivity range of the human Pacinian corpuscles (approximately 40–800 Hz). Hackathon Contributions During the hackathon, we built the complete real-time interaction pipeline, including: * Gemini Live WebSocket integration for continuous multimodal perception * Real-time conversion of video and speech into tactile pulse sequences * Native Kotlin low-level haptic actuator bridge * Lightweight JSON packet protocol for low-bandwidth haptic streaming * Optimized WebSocket communication for ultra-low latency * Interactive Braille learning interface with gamified progression and words-per-minute tracking Previous Work vs. Hackathon Work Before this hackathon, our work focused on developing and validating the mathematical foundations of Temporal Signal Synthesis for Braille. The prototype was evaluated with educators and students at Rashtriya Andh Vidyalaya, where it received positive feedback for significantly reducing cognitive load compared to conventional Braille systems. However, that implementation was limited to isolated text-to-vibration synthesis. For this hackathon, we transformed that research into a fully interactive accessibility platform by integrating Gemini Live. The resulting system continuously interprets live visual scenes, spoken language, and conversational context, producing real-time tactile feedback that enables private, context-aware interaction with the surrounding environment. Presentation https://drive.google.com/file/d/1EV1BhaPUq4QAz-5IQx1OA3KJ5GzvquFp/view?usp=sharing Mathematical Research Dossier https://drive.google.com/file/d/1nxvhB2u5pScFDuwAtdeTl0dpus-ocnhV/view?usp=sharing Rashtriya Andh Vidyalaya Validation & Feedback https://drive.google.com/file/d/1Tn-lsPI9K9QDm1vgy8kCCRkUQuwYQW8a/view?usp=sharing

Team

Placement

Participant