# SureStep

- **Event:** [Gemini 4 NYC Hackathon](https://cerebralvalley.ai/e/gemini-4-nyc)
- **When:** Sat, Nov 21 at 9:00 AM – 10:00 PM (EST)
- **Where:** New York, NY
- **Team:** [Manseerat Kaur](https://cerebralvalley.ai/u/Seerat)
- **GitHub:** https://github.com/seerat16/misstep-gemini4
- **Demo video:** https://drive.google.com/drive/folders/1IrHFIMtNbTr3njV6l5o7tpBwCI3zdauQ?usp=sharing
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/136

Misstep watches someone carry out a written procedure on camera (a recorded video or a live overhead webcam) and reports which step went wrong, how sure it is, and when it isn't sure.

The Problem: Wherever human hands follow written procedures, medication preparation, clinical checklists, wet-lab protocols, manufacturing assembly, or food safety, things fail one skipped, out-of-order, or miscounted step at a time. Frontier vision-language models are good at summarizing a video, but single-call prompting struggles to spot fine-grained procedural mistakes in egocentric/overhead video and gives uncalibrated guesses without telling you when to abstain.

How Misstep Solves It: Given any written procedure (pasted text, PDF, or photo of a card) and a video or live webcam stream, Misstep compiles the procedure into a machine-checkable step graph with visual checks, quantities, and ordering constraints, localizes when each step happens, evaluates calibrated per-question probabilities (noul, choice, score), and escalates only uncertain or flagged steps to high-resolution adjudication. In live MedPrep mode, it monitors a 6-frame rolling webcam buffer every 2 seconds, speaks real-time alerts seconds after a skipped or out-of-order step, and verifies final pill-organizer compartment counts at ultra-high resolution.

We evaluate Misstep against single-call Gemini 4 Hydrogen (B1) and Neon (B2) baselines on held-out recipes from CaptainCook4D (40 test recordings / 6 held-out recipes, disjoint from 16 dev recordings), a blind live field evaluation with strangers at the hackathon, and an ffmpeg degradation stress ladd

## More from Gemini 4 NYC Hackathon

- [Ethyneon](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/133)
- [Guru Check](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/134)
- [NULL](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/135)
- [Vouch](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/137)
- [GeneSleuth](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/138)
- [Forkcast](https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/139)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-nyc/hackathon/gallery/136. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt
