SureStep
Gemini 4 NYC HackathonMisstep watches someone carry out a written procedure on camera (a recorded video or a live overhead webcam) and reports which step went wrong, how sure it is, and when it isn't sure. The Problem: Wherever human hands follow written procedures, medication preparation, clinical checklists, wet-lab protocols, manufacturing assembly, or food safety, things fail one skipped, out-of-order, or miscounted step at a time. Frontier vision-language models are good at summarizing a video, but single-call prompting struggles to spot fine-grained procedural mistakes in egocentric/overhead video and gives uncalibrated guesses without telling you when to abstain. How Misstep Solves It: Given any written procedure (pasted text, PDF, or photo of a card) and a video or live webcam stream, Misstep compiles the procedure into a machine-checkable step graph with visual checks, quantities, and ordering constraints, localizes when each step happens, evaluates calibrated per-question probabilities (noul, choice, score), and escalates only uncertain or flagged steps to high-resolution adjudication. In live MedPrep mode, it monitors a 6-frame rolling webcam buffer every 2 seconds, speaks real-time alerts seconds after a skipped or out-of-order step, and verifies final pill-organizer compartment counts at ultra-high resolution. We evaluate Misstep against single-call Gemini 4 Hydrogen (B1) and Neon (B2) baselines on held-out recipes from CaptainCook4D (40 test recordings / 6 held-out recipes, disjoint from 16 dev recordings), a blind live field evaluation with strangers at the hackathon, and an ffmpeg degradation stress ladd





