# PerritoTalk

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Yamid Noguera](https://cerebralvalley.ai/u/yamidnozu)
- **GitHub:** https://github.com/yamidnozu/perrito-talk
- **Demo video:** https://github.com/yamidnozu/perrito-talk/releases/download/v1.0-hackathon/demo_final.mp4
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1228

PerritoTalk — A live, Spanish-first dog watcher on Muse Spark that describes only what it can see and hear, and claims a feeling only when the science backs it.

LINKS
- Live app: https://perritos.edevcore.com
- Repository: https://github.com/yamidnozu/perrito-talk
- Demo video (2:56): https://github.com/yamidnozu/perrito-talk/releases/download/v1.0-hackathon/demo_final.mp4

DESCRIPTION
PerritoTalk is a mobile web app for dog owners. Its interface is Spanish first, with English one tap away.
Live watcher: point the phone at your dog and the watching happens on the device. MediaPipe EfficientDet-Lite2 finds the dog (Lite0 fallback on slow devices); MediaPipe YAMNet listens for barks, howls and whimpers. About a second after the dog is detected, an instant PRELIMINAR card appears. It comes from the device only, uses no AI and attributes no emotion.
Muse Spark reading: when the dog's state changes, the watcher sends one 10 s clip (4 s before the change, 6 s after) to Muse Spark 1.3. Spark returns strict-JSON observable descriptors (posture, vocalization, gaze, context). Our code then checks them against an evidence base of 45 cited studies. The reading has three layers: what was observed; possible interpretations, with the certainty stated in words and the source cited; and what we still don't know. When no observed signal matches a rule, the app says so ("sin señal con respaldo científico suficiente") and does not invent a feeling.
Deep analysis (uploaded clips): Muse Spark describes the clip 3 times in parallel and a descriptor counts only if 2 of 3 agree; SAM 3.1 locates the dog; Muse Voice Transcribe separates the owner's speech from dog sounds and uses it as context.
Also: camera mode for batches of home-camera clips, a private on-device diary, Muse Image share cards, and a vet banner for urgent physical signs. Every result is an interpretation, not a diagnosis.

PROBLEM
Owners misread their dogs every day: a howl as noise, stillness as calm, a stare as stubbornness. Most consumer "dog translators" print a confident emotion with a percentage that no study supports. In Latin America behaviorists are scarce and most such apps are English-only. We wanted a second pair of eyes that is careful, cheap, Spanish-first, and honest when it does not know.

HOW WE USED META'S AI TOOLS
- Muse Spark 1.3 (Responses API) is the core: video with embedded audio, images or WAV in; observable descriptors out via strict json_schema at reasoning effort low. Called only on a trigger, one request in flight, 30 s timeout, one retry. Strategy chosen by measuring 12 strategies.
- Deep analysis: 3 parallel Muse Spark calls with a per-field 2-of-3 vote in code. No judge model.
- SAM 3.1: locates the dog in uploaded/deep-analysis clips.
- Muse Voice Transcribe: diarized human speech used as context, in parallel with SAM before Spark.
- Muse Image 1.0: optional illustrated share card, with canvas fallback.
- Meta's Muse coding agent on the Meta Model API built our latest iteration: the instant on-device preliminary card, a slimmer live payload and Server-Timing instrumentation.
- Tried and dropped, because we measured it: multi-agent pipeline with a judge, effort medium, frames without audio, realtime ASR stream.

RESULTS (real numbers)
- Benchmark: 811 Muse Spark calls (10 real clips x 12 strategies x 3 repetitions), 0 errors, about $6.28 total. A smoke test, not an accuracy claim.
  - Video + effort low + YAMNet (live winner): 6.0/10 hits, 0 over-claims, 0 false vet alerts, p50 8.7 s (p95 13.3 s), about $0.014 per new clip.
  - 3x self-consistency vote (deep analysis): 6.0/10, 0 over-claims, 0 false vet alerts, p50 9.4 s, about $0.04.
  - Effort medium: 4.3/10, 1.0 over-claims, 1.0 false vet alerts, p50 12.5 s.
  - 4 frames, no audio: 4.0/10, p50 16.3 s.
  - Multi-agent + judge: 3.7/10, 1.0 over-claims, 1.0 false vet alerts, p50 20.1 s, about $0.05.
- Live audit on production: 46 runs with fake camera fed real clips; 0 non-200 responses, 0 page/console errors. First live reading: vocal type correct 15/20 (up from 12/20), observable behavior 11/20, state rubric met 14/20. Upload path: vocal 23/30, state 21/30, behavior 17/30.
- Demo recording: dog detected, PRELIMINAR card about 1.2 s later, Muse Spark reading ready in 12.8 s.

TECH
Vanilla JS PWA; MediaPipe Tasks (EfficientDet-Lite2/Lite0, YAMNet) on device; MediaRecorder pre-roll clips and a local acoustic heuristic. Backend: Cloudflare Worker (/api/analyze, deep analysis, cards); media is not stored on the server. Meta Model API: Muse Spark 1.3, SAM 3.1, Muse Voice Transcribe, Muse Image 1.0. Evidence engine: evidence.json with 45 citations (the model describes, code decides, certainty in words). 187 automated tests, reproducible benchmark harness and live audit harness. License: MIT.

CHALLENGES
- Over-interpretation: the more the model reasoned, the more it read into a closed eye (leading to a pain flag). Effort medium and the multi-agent judge both produced false vet alerts. We split the work: the model only describes, code decides, 2-of-3 voting removes stray observations.
- Audio: Meta documents Muse Spark 1.3 audio as not fully supported, so YAMNet hints and a local acoustic heuristic run alongside it.
- Latency: a Muse reading takes 10-17 s; the instant PRELIMINAR card fills the wait without claiming anything.
- Detection: small, dark or security-camera dogs are often labeled cow or horse; we accept that animal set and use Lite2 with a low-light boost.
- Honest limits: whimpers are hard (whine clip right in 1/2 live runs and 0/3 upload runs); play without a play bow has no evidence rule, so the app describes it instead of labeling it.

WHAT'S NEXT
RTSP and low-cost home-camera input; validation with real owners and veterinarians; better whimper detection; more evidence rules only where peer-reviewed support exists; a native wrapper for background watching.

## More from Meta Global AI Developer Hackathon

- [Volshi](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1225)
- [Aleksandr Nariadchikov](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1226)
- [Voxarah Capture](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1227)
- [Biscuit - Kindred AI](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1229)
- [Anamnesis](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1230)
- [Walk Me There](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1231)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/1228. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt