PerritoTalk

PerritoTalk — A live, Spanish-first dog watcher on Muse Spark that describes only what it can see and hear, and claims a feeling only when the science backs it. LINKS - Live app: https://perritos.edevcore.com - Repository: https://github.com/yamidnozu/perrito-talk - Demo video (2:56): https://github.com/yamidnozu/perrito-talk/releases/download/v1.0-hackathon/demo_final.mp4 DESCRIPTION PerritoTalk is a mobile web app for dog owners. Its interface is Spanish first, with English one tap away. Live watcher: point the phone at your dog and the watching happens on the device. MediaPipe EfficientDet-Lite2 finds the dog (Lite0 fallback on slow devices); MediaPipe YAMNet listens for barks, howls and whimpers. About a second after the dog is detected, an instant PRELIMINAR card appears. It comes from the device only, uses no AI and attributes no emotion. Muse Spark reading: when the dog's state changes, the watcher sends one 10 s clip (4 s before the change, 6 s after) to Muse Spark 1.3. Spark returns strict-JSON observable descriptors (posture, vocalization, gaze, context). Our code then checks them against an evidence base of 45 cited studies. The reading has three layers: what was observed; possible interpretations, with the certainty stated in words and the source cited; and what we still don't know. When no observed signal matches a rule, the app says so ("sin señal con respaldo científico suficiente") and does not invent a feeling. Deep analysis (uploaded clips): Muse Spark describes the clip 3 times in parallel and a descriptor counts only if 2 of 3 agree; SAM 3.1 locates the dog; Muse Voice Transcribe separates the owner's speech from dog sounds and uses it as context. Also: camera mode for batches of home-camera clips, a private on-device diary, Muse Image share cards, and a vet banner for urgent physical signs. Every result is an interpretation, not a diagnosis. PROBLEM Owners misread their dogs every day: a howl as noise, stillness as calm, a stare as stubbornness. Most consumer "dog translators" print a confident emotion with a percentage that no study supports. In Latin America behaviorists are scarce and most such apps are English-only. We wanted a second pair of eyes that is careful, cheap, Spanish-first, and honest when it does not know. HOW WE USED META'S AI TOOLS - Muse Spark 1.3 (Responses API) is the core: video with embedded audio, images or WAV in; observable descriptors out via strict json_schema at reasoning effort low. Called only on a trigger, one request in flight, 30 s timeout, one retry. Strategy chosen by measuring 12 strategies. - Deep analysis: 3 parallel Muse Spark calls with a per-field 2-of-3 vote in code. No judge model. - SAM 3.1: locates the dog in uploaded/deep-analysis clips. - Muse Voice Transcribe: diarized human speech used as context, in parallel with SAM before Spark. - Muse Image 1.0: optional illustrated share card, with canvas fallback. - Meta's Muse coding agent on the Meta Model API built our latest iteration: the instant on-device preliminary card, a slimmer live payload and Server-Timing instrumentation. - Tried and dropped, because we measured it: multi-agent pipeline with a judge, effort medium, frames without audio, realtime ASR stream. RESULTS (real numbers) - Benchmark: 811 Muse Spark calls (10 real clips x 12 strategies x 3 repetitions), 0 errors, about $6.28 total. A smoke test, not an accuracy claim. - Video + effort low + YAMNet (live winner): 6.0/10 hits, 0 over-claims, 0 false vet alerts, p50 8.7 s (p95 13.3 s), about $0.014 per new clip. - 3x self-consistency vote (deep analysis): 6.0/10, 0 over-claims, 0 false vet alerts, p50 9.4 s, about $0.04. - Effort medium: 4.3/10, 1.0 over-claims, 1.0 false vet alerts, p50 12.5 s. - 4 frames, no audio: 4.0/10, p50 16.3 s. - Multi-agent + judge: 3.7/10, 1.0 over-claims, 1.0 false vet alerts, p50 20.1 s, about $0.05. - Live audit on production: 46 runs with fake camera fed real clips; 0 non-200 responses, 0 page/console errors. First live reading: vocal type correct 15/20 (up from 12/20), observable behavior 11/20, state rubric met 14/20. Upload path: vocal 23/30, state 21/30, behavior 17/30. - Demo recording: dog detected, PRELIMINAR card about 1.2 s later, Muse Spark reading ready in 12.8 s. TECH Vanilla JS PWA; MediaPipe Tasks (EfficientDet-Lite2/Lite0, YAMNet) on device; MediaRecorder pre-roll clips and a local acoustic heuristic. Backend: Cloudflare Worker (/api/analyze, deep analysis, cards); media is not stored on the server. Meta Model API: Muse Spark 1.3, SAM 3.1, Muse Voice Transcribe, Muse Image 1.0. Evidence engine: evidence.json with 45 citations (the model describes, code decides, certainty in words). 187 automated tests, reproducible benchmark harness and live audit harness. License: MIT. CHALLENGES - Over-interpretation: the more the model reasoned, the more it read into a closed eye (leading to a pain flag). Effort medium and the multi-agent judge both produced false vet alerts. We split the work: the model only describes, code decides, 2-of-3 voting removes stray observations. - Audio: Meta documents Muse Spark 1.3 audio as not fully supported, so YAMNet hints and a local acoustic heuristic run alongside it. - Latency: a Muse reading takes 10-17 s; the instant PRELIMINAR card fills the wait without claiming anything. - Detection: small, dark or security-camera dogs are often labeled cow or horse; we accept that animal set and use Lite2 with a low-light boost. - Honest limits: whimpers are hard (whine clip right in 1/2 live runs and 0/3 upload runs); play without a play bow has no evidence rule, so the app describes it instead of labeling it. WHAT'S NEXT RTSP and low-cost home-camera input; validation with real owners and veterinarians; better whimper detection; more evidence rules only where peer-reviewed support exists; a native wrapper for background watching.