OmniSpatial

OmniSpatial is a real-time, zero-shot physical AI copilot designed to guide precision hands-on tasks, such as electronics assembly, PCB inspection, and hardware maintenance, through an interactive augmented reality heads-up display (HUD). Human physical work requires manual dexterity and spatial awareness. Yet existing computer vision systems are brittle: traditional detectors (like YOLO) require thousands of labeled training images for every unique circuit board, while conversational LLMs lack spatial intuition and cannot verify physical alignment in real time. OmniSpatial solves this by uniting Meta's newest foundation models into an embodied, real-time spatial pipeline: 1. Zero-Shot Spatial Perception (Meta SAM 3.1): Using promptable concept segmentation, SAM 3.1 detects micro-components, IC sockets, pin headers, and hand tools on-the-fly directly from video frames without fine-tuning. Contours are refined using Douglas-Peucker polygon simplification and Chaikin corner-cutting algorithms to eliminate camera jitter. 2. Sub-Millimeter AR Trajectory Guidance: OmniSpatial calculates Euclidean distance vectors, centroid bearings, and alignment tolerances between detected parts, projecting dynamic directional chevrons, target crosshairs, and a vernier tolerance gauge (countdown from 80mm down to 0.4mm) onto the live video feed. 3. Perspective Blueprint Projection (Meta Muse Image 1.0): Synthesizes perspective-accurate visual reference diagrams and projects a holographic ghost blueprint directly over the workbench so operators can physically match components to the blueprint. 4. Asynchronous Physical Reasoning (Meta Muse Spark 1.3): Operators press Spacebar to verify step completion. Muse Spark performs multimodal chain-of-thought physical inspection, analyzing pin orientation, polarity alignment, and solder bridges before triggering an audio confirmation chime and advancing the assembly stepper. 5. Real Hardware Anomaly Detection: When tested with live smartphone video of a physical development board, Muse Spark read the physical silkscreen pinout (A0..VIN, D0..3V) and caught an anomaly in real time, flagging that the board was an ESP8266 NodeMCU rather than the specified ESP32. Architecture: Built with FastAPI, WebSockets, Python 3.12, OpenCV, and a zero-framework HTML5 Canvas HUD. OmniSpatial features a dual-loop streaming engine that decouples 60 FPS visual tracking from 6-second cloud LLM reasoning, maintaining sub-millimeter responsiveness with zero queue lag.