ISee Team

Meta products are the core of iSee. Each one does a specific job in the product. SAM 3.1 (perception). SAM 3.1 runs on a Modal A100 GPU in Europe behind a real-time WebSocket with an automatic HTTP fallback. For each frame, it takes up to 22 text concepts and returns objects, boxes and masks. Those results drive the distances, the obstacle announcements, the steering toward the widest free space, and crossing mode, where SAM finds the pedestrian light and the cars and iSee reads the light's color on the device. SAM's fixed objects (poles, signs, walls) even tell iSee how the camera is moving, with no extra sensor. PyTorch, which started at Meta, runs the model. Muse Spark (understanding). Muse Spark (meta-models/muse-spark-open-1026, hosted on Baseten) is the default vision LLM. It receives the image and the question in a single chat/completions call, describes the scene, answers spoken questions, and writes the short sentences of the live narration. For live narration, we turn its reasoning off (enable_thinking=false) so the whole 200-token budget goes to the spoken sentence. Muse Voice Transcribe (listening). muse-voice-transcribe-1.0 on the Meta AI API is the default speech-to-text engine. We use the batch endpoint (/asr/transcribe) and the realtime WebSocket (/asr/realtime), which listens while the finger is held down. For a blind user, voice is the main way to talk to the app, so this model sits on the main path. Muse Image (landmarks). muse-image-1.0 on the Meta AI API (/images/edits) turns the photo of a filmed object into a black-on-yellow pictogram. The pictogram is stored right away, and iSee recognizes that landmark on live video, on the device and with no network. Muse Image and Muse Voice Transcribe share the same base URL and API key, which kept the integration simple. Ray-Ban Meta glasses and the Wearables Device Access Toolkit (wearing it). iSee integrates mwdat-core, mwdat-camera, mwdat-inputs and mwdat-motion. Registration and permissions go through the Meta AI app. The toolkit then gives us the live H.265 video stream, photo capture, the glasses microphone, the capture button, and head orientation from the Motion module when the pair provides it. Those glasses frames go through the same SAM, distance and voice pipeline as the phone camera, so the guidance built for the phone also runs on the glasses. We tested the integration on a real Ray-Ban Meta pair. We also used the Mock Device Kit (mwdat-mockdevice) in debug builds to build and test the glasses flow without hardware, alongside a test bench built into the app. It let us keep working on glasses features without a pair on hand, which made a real difference during a hackathon.