KIBO

Kiosks replaced the cashier. Nobody replaced the help. KIBO (Kiosk Intelligence Bridge & Operator) is that help, on your own phone, for people who get stuck at self-service kiosks: older adults, people with low vision, travelers who can't read the screen. In a Korea Consumer Agency observation (2020, n=5), all five first-time users aged 70+ failed to finish a fast-food kiosk order. Tap the mic once and say what you want in your own words. The phone sends it when you stop talking, and after KIBO answers it listens again, so a conversation needs no more taps. Muse Voice Transcribe turns the speech into text, Muse Spark 1.3 turns the text into a structured intent, and our server plans the exact taps with deterministic code, checks every tap against the kiosk's own state machine and plays them on the kiosk's screens with a visible pointer. Paying, consenting and scanning documents always stay with you. We built three kiosks during the event (our own replicas, fictional brands): - Bridge Burger: "Double cheeseburger, large meal, zero cola, no pickles, for here. That's all." becomes 13 screens and 14 taps, ending on the review screen with Pay left to you. - Bridge Air check-in: "My mom and me, to Tokyo, seats together, one checked bag" drives the airline choice, seat map and bags. KIBO stops for the dangerous-goods declaration, the passport scans and the final confirmation. - Bridge Rail ticket machine (not in the video). The phone speaks English, Korean, Japanese, Vietnamese, Chinese and Spanish; the kiosks show only Korean and English. For kiosks that can't connect, guide mode: photograph the screen, Muse Spark finds the button, SAM 3.1 outlines it, and KIBO tells you where to tap. It is still an experiment: on 10 kiosk photos it had never seen, it outlined the right button 5 times and named the button in all 10. How we checked it: on a held-out set we never used while building, 39 of 40 everyday requests were exactly right in all three runs and 100% of unsafe requests (10 requests x 3 runs) were blocked; 1,000 random action sequences never reached payment. We also checked what KIBO says: replies were checked by code against the kiosks' own data and read blind by a separate model. After we gave Spark each kiosk's facts, replies judged right went from 65% to 95% (fast food) and 43% to 98% (airport) on the development sets. Limits: spoken turns are slower than we wanted (p50 5.8 s from sending the recording to KIBO's reply; we missed our p95 under 4 s target), changing items already in the cart isn't supported yet, and a real kiosk would have to accept KIBO's taps the way our replicas do. Try it without a phone: https://kibo-muse.vercel.app ("Touch vs KIBO", then type a sentence into the phone panel). All code was written during the hackathon (Oct 26 - Nov 4, 2026, US Eastern) in a new repository.