SpectraPatch

SpectraPatch is a hands-free CLI developer agent that detects, patches, and visually verifies temporal mobile and web UI regressions (e.g., animated keyboard occlusions, bottom-sheet bounce collisions, and z-index overlap) that static DOM checkers and accessibility trees falsely mark as passing. Problem & Impact (30%): Static test runners inspect widget trees that still report "exists = true" when a critical action button is visually covered by an animated keyboard. Developers spend hours debugging layout shifts across screen ratios. SpectraPatch converts a spoken bug report and a simulator/browser capture into a verified, pull-request-ready code fix with zero manual geometry markup. Meta AI Multi-Model Architecture (25%): 1. Muse Voice Transcribe: Ingests spoken tester audio via /v1/asr/transcribe in ENDPOINTING mode to extract exact turn-level boundaries (0ms -> 4220ms). 2. Muse Spark 1.3: Extracts normalized colliding UI noun-phrases ("checkout button" vs "keyboard") with strict JSON schema constraints. 3. SAM 3.1 as a Temporal Layout Assertion Engine: Uploads screen recordings once via the Files API and opens parallel streaming sessions per concept across 60 frames (720p normalized). Decodes RLE masks and bounding boxes via meta_sam_parser to compute frame-by-frame pixel collision ratios: Overlap = Area(Mask_A ∩ Mask_B) / Area(Mask_A). 4. Muse Code Sub-Agent (muse exec): Dispatches an isolated git worktree with high reasoning effort to inspect collision keyframes, apply layout fixes (:focus-within safe-area styling and Flutter resizeToAvoidBottomInset), and re-render. 5. Closed-Loop Verification: Headless Chrome re-renders the patched worktree layout; SAM 3.1 re-evaluates the stream to mathematically prove 0.0% overlap before marking the patch verified. Technical Depth & Provenance (20% & 25%): - Provenance: Built 100% from scratch during the Meta Global AI Developer Hackathon (Oct 26 – Nov 4, 2026). - Live Telemetry: Fully operational against production Meta Model API endpoints with live ASR timestamps, real SAM 3.1 video grounding, and worktree git isolation. - Offline Deterministic Replay: Features a standalone offline verification harness (--demo) backed by 24/24 passing pytest unit/e2e tests so judges can verify the full loop without API credits.