Skip to Main Content

Misstep watches someone carry out a written procedure on camera (a recorded video or a live overhead webcam) and reports which step went wrong, how sure it is, and when it isn't sure. The Problem: Wherever human hands follow written procedures, medication preparation, clinical checklists, wet-lab protocols, manufacturing assembly, or food safety, things fail one skipped, out-of-order, or miscounted step at a time. Frontier vision-language models are good at summarizing a video, but single-call prompting struggles to spot fine-grained procedural mistakes in egocentric/overhead video and gives uncalibrated guesses without telling you when to abstain. How Misstep Solves It: Given any written procedure (pasted text, PDF, or photo of a card) and a video or live webcam stream, Misstep compiles the procedure into a machine-checkable step graph with visual checks, quantities, and ordering constraints, localizes when each step happens, evaluates calibrated per-question probabilities (noul, choice, score), and escalates only uncertain or flagged steps to high-resolution adjudication. In live MedPrep mode, it monitors a 6-frame rolling webcam buffer every 2 seconds, speaks real-time alerts seconds after a skipped or out-of-order step, and verifies final pill-organizer compartment counts at ultra-high resolution. We evaluate Misstep against single-call Gemini 4 Hydrogen (B1) and Neon (B2) baselines on held-out recipes from CaptainCook4D (40 test recordings / 6 held-out recipes, disjoint from 16 dev recordings), a blind live field evaluation with strangers at the hackathon, and an ffmpeg degradation stress ladd

Team

Seerat

More from Gemini 4 NYC Hackathon

We design an agentic system to automatically create + manage intelligent AI opponents/NPCs in video games. The suite of agents analyzes user gameplay…

Adaptive NPCs project preview

Manually routing PCBs is time-consuming and cost-inefficient. Current best algorithms (freerouting, PCBRouter) score very low on PCBWorld's hardest PCB…

routr project preview

Heliocam is a camera app where Gemini controls the camera. It watches what you're pointing at, sets the ISO, shutter, focus, white balance and zoom for you…

heliocam project preview

Repairs and assembly fail because manuals are lost, vague, missing, or nonexistent, so people break things or throw them out. KLIK turns a photo of any object…

Klik project preview

Iron Pulse turns your laptop into a 3D boxing ring. Fight an adaptive AI rival in real time using just your webcam. Throw real punches and lean to dodge…

Iron Pulse Arena project preview

Text your raw footage to an iMessage number and get a finished, trend-informed Reel back in the same thread.

Sticky project preview