# Point, Say, Press

- **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon)
- **When:** Oct 26, 12:00 PM – Nov 4, 9:00 PM (EST)
- **Where:** Online
- **Team:** [Jeonggil Bae](https://cerebralvalley.ai/u/ollekil)
- **GitHub:** https://github.com/jejuolledev/point-say-press
- **Demo video:** https://youtu.be/qZ089EwfhtI
- **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/725

Point, Say, Press is a phone co-pilot for touchscreen kiosks. Point your phone at the kiosk and say what you want. The screen dims except the one button to press next, outlined in the shape of that button. It never presses anything itself, and when its readings do not agree it points at nothing.

Kiosks ask people to find one button among dozens, against a timer, while pop-ups try to sell them something. In a Seoul Digital Foundation survey of 5,000 residents, published in May 2022, only 45.8% of those aged 55+ had ever used a kiosk, and the top reason among non-users was not knowing how or finding it hard (33.8%) (Newsis, 2022-05-16). There is nothing to integrate and no app to install: a web page runs on the phone, so a QR sticker on a kiosk could open it. It is built for older adults (18+, per Meta's terms). It has no regular users yet.

What it does today: takes a spoken Korean order with live captions; plans the next press on a screen it has not seen before; names a pop-up you did not ask for as an extra offer and points at the button that declines it; reads the cart back before payment and shows a mismatch in red.

How it was measured. We set aside 80 public web images of 54 kiosk interfaces it had never been tuned on (hospitals, trains, cinemas, parking, cafes, burger chains), mostly frames of public tutorial videos, mostly Korean. Labels were written by one Claude agent and reviewed image by image by a second; neither saw model output or the rules being tried. They were hashed before the new rules went into the code, and trials up to then used only the separate tuning images. We froze the code, then ran it once. No person re-checked the labels. Both old and new code ran on this same set:

| 80 unseen images, 62 with a right button | Before | Now |
|---|---|---|
| Right button (centre of the outline inside the labelled button) | 68% | 87% (54 of 62; 95% interval 77% to 93%). 84% (52 of 62) if answers past the page's 45 s limit do not count. Pair by pair at that limit: 13 gained, 3 lost, exact McNemar p = 0.02 |
| Wrong button, on all images | 10% (8 images) | 2.5% (2 images) |
| Right when it pointed | 84% | 96% of 56 |
| Did not point | 38% | 30% (24: 17 correct, 7 missed a right button) |
| Typical wait per step | 8.4 s | 8.4 s (slow end 14.7 s) |

These are single steps: each image is given the right order and the right steps so far. An order takes several steps, so the score for a whole order on a real kiosk would be lower, and that has not been measured. Without the 19 images where a tutorial finger is already on the answer: 93% right of 43. Images with no right button (18, such as a request this kiosk cannot do): it pointed at nothing on 94%. Both remaining wrong images were also wrong before; none is new. Upsell screens with a right button: 5 of 5, up from 1 of 5, though the upsell warning itself fell from 5 of 6 to 4 of 6. "Right" is about where the centre lands: on 10 of the 54 the overlap with the labelled button is below 0.5.

Three smaller checks after the freeze, no rule changed: Phone-like damage on 30 of the images (blur, darker, 4-degree turn, heavy JPEG): right 24 became 12 of 26, wrong 0 became 2, not pointed 6 became 16. It is a software stand-in, not a phone. Speech: 20 of 30 orders heard right across three synthetic Korean voices, quiet and with chatter mixed in at 10 dB; the standard voice got 10 of 10, the two older-sounding voices 10 of 20. That matters, since older adults are the target. Older-sounding synthetic voices are not older people, the check used the file endpoint rather than realtime, and the keyword list is short and coffee-only. A misheard order shows on screen and can be typed instead.

On our replica kiosk, with real models: 12 of 12 full orders ended with the right cart, up from 8 of 10 (5 different orders, typed, pressed by a script, half with English on screen, no camera, typically 51 s an order; one order hit the kiosk's time-out screen and recovered; 4 of 4 more after the last cart change). The cookie order that failed before went 4 of 4; we built this kiosk and wrote the rule after seeing that failure, so it is a regression check, not a score.

What changed: looking at every miss on the tuning images, all 8 wrong answers and 9 of 13 missed buttons had the same cause: Spark named the right button, and its box landed in the wrong place. Now Spark reads each screen twice at once, on two differently drawn coordinate rulers. The two answers must land on the same place. If not and one reading is sure of something, a third look decides (9% of images); with no majority the app says "show the screen again" and points at nothing. A wrong outline got rarer and the no-outline rate did not rise. The price is time: extra calls, and a slow tail. Details and failed trials: `docs/submission/05_why_wrong.md`.

What these numbers do not show: nobody has used it at a real kiosk, and no person has tried it. Tutorial footage is steadier and better lit than a phone in a hand. Several images share a chain or kiosk brand, so the effective sample is smaller than the count. Hashes and times are self-reported: the public history starts at publication, so it does not prove the order. The first labelling agent re-checked only 10 of its 55 frame-filter sheets (`docs/labeling_criteria.md`). Its wait is still long for someone standing at a kiosk, so the page shows progress: the screen it found at 1.9 s (on the 37 images where one was found), a dashed candidate marked "do not press yet" at 6.1 s, then the green answer. All runs, labels and the hash are in `docs/submission/measured/`.

## More from Meta Global AI Developer Hackathon

- [Field from Above](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/722)
- [EdgeCases](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/723)
- [RAJAKUMARAN](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/724)
- [Agentic Verification Layer](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/726)
- [ProofPath](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/727)
- [Farhan](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/728)

---

Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/725. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt