BlinkFlow

BlinkFlow is a debugging tool for people who run a robot in a simulator. Its capture and measurement stage works on ordinary window pixels, with no simulator API; the diagnosis reads the robot's own source and logs, and the proof needs a deterministic replay adapter for the robot (two exist: the Microduck biped and a 3-joint arm, both in MuJoCo). It keeps a rolling 60-second buffer of the simulator window. When the robot does something wrong, you hold a key (or tap Ray-Ban Meta glasses) and say what you saw: "a couple of seconds ago the robot hit the purple box." BlinkFlow aligns the sentence to the moment, sends the clip to SAM 3.1 to track the robot and the object frame by frame and find the first contact, and has Muse Wonder read the key frames, the robot's log and its source code to name the cause, with citations that BlinkFlow checks against the source and flags when one points at nothing. It asks "Fix it?", lets Muse Code patch the robot's code with every tool call and changed path checked against the robot's folder (a filter, not OS isolation), then replays the identical scene in a deterministic twin and measures again on pixels. The fix is kept only if the incident is gone, the robot does not fall, it still travels, and every other scene stays clean, each tracked against its own obstacle; a scene whose obstacle is never seen gets no verdict and blocks the fix. Nothing about the content of the change is inspected. Otherwise it rolls back to the git checkpoint and tries again (up to three times), then hands back to the human. "Keep the fix?" is the last word, and it is yours. Who it is for: people who build or train robots and are not robotics engineers — hobbyists on open hardware like the Microduck biped, students, small teams. In my own experience, turning "its leg brushed the box" into a line number means an evening of logs and code. Measured, with every number traceable in results/; the public examples re-check offline from the saved model replies (the 12 ablation manifests additionally need the local raw run folders): - Public run (1 Nov, replayed from the saved voice clip): from the spoken sentence to a verified fix in 368.0 s and $0.95 (30 of 52 SAM replies paid), fixed on the first attempt. One-line fix (side memory 0.0 → 3.0). Contacts 1 → 0. Control 4/4 clean on each scene's own obstacle. The run in the video (31 Oct): 386.6 s, $0.38 with most SAM replies served from cache. - Live simulator: 5/5 runs hit the box before the fix, 0/5 after (same scene, five repeats). - Bug suite with BlinkFlow's own code frozen (Oct 28, before the current gate and control rule): 11 planted bugs, 8 produced a visible incident, 8/8 repaired and passed the incident and control checks; Wonder's cause named a planted line in 4 of 8, the kept patch touched one in 7 of 8; 7 of the 8 mutations repeat a first-pass mutation, one is new. - Ablation (12 runs, manifests in results/ablation/): the same coding agent given only the sentence — precise (39–597 s of agent time) or as vague as "something went wrong with the robot" (105–772 s) — found the same one-line fix on both bugs tried. So on these bugs the cause was findable from the code alone. What BlinkFlow added: it said what happened and when ("touched the purple box at 10.90 s", "fell off the yellow pit at 10.05 s") from the vague sentence, and its coding turns took 42–44 s on the first attempt. Four of the agent's 18 turns failed the path audit and two returned no change; replay and controls accepted all 12 retained candidates — this experiment did not measure the pixel check rejecting a wrong fix. - A second robot (examples/arm/, examples/arm-bug2/, examples/arm-voice/, results/arm.md): the same loop on a hand-written 3-joint MuJoCo arm (43-line scene, 45-line controller), three published runs under one rule: contact and the two control scenes are measured from pixels; reaching the target (within 20 mm) is read from the simulator's own end-effector position. (1) Planted clearance = 0.0: SAM 3.1 found first contact within 25 ms of the physics; Muse Wonder's cause cited the clearance line, grounded; Muse Code changed one line on the first attempt (0.0 → 0.15 m; earlier runs chose 0.08 and 0.12, all pass). (2) A sign error in the controller's lift (- clearance instead of +): fixed with one logic line in brain.py on the first attempt. (3) The clearance bug driven by my real spoken sentence through Muse Voice Transcribe: same fix, same proof. All three: contacts 1 → 0, target reached, 2/2 controls clean, re-derivable offline. In runs 2 and 3 Wonder named the right line but the diagnosis is flagged ungrounded and kept as such (run 2: categories cited instead of frames/ticks; run 3: a real tick but no key frame and two references to nothing); the fixes still passed. $1.86 for everything ever spent on the arm, calibration and replaced runs included. Limits: SAM did not find the arm from text (0 of 180 frames), so the arm is initialised with a visual box marked once on the first frame (Meta's documented box prompt), identical before and after; the voice run used a replayed key-press time; ideal planar simulation. - Replaying recorded inputs without a simulator (results/sil.md): the brain's own sensor recordings (here from the simulator, not a physical robot) are replayed through the brain before and after a fix, using the brain's pre-existing recording replay. On 4 of the 8 suite bugs the fixed brain brakes 2.85–8.2 s before the recorded impact while the buggy one keeps walking; on the other 4 and on purple-box the result is inconclusive. Open-loop decision evidence only, not an outcome proof. Pre-existing and declared: the Microduck robot, its walking policy and the MuJoCo simulator from Pollen Robotics / Hugging Face; the robot's brain microduck-brain (Apache-2.0), my own earlier project, which BlinkFlow works on as a clean copy; the SAM 3.1 masks are decoded with Meta's official meta-sam-parser package, installed as a dependency. BlinkFlow's implementation was built after kickoff (first commit 26 Oct, 16:51 UTC); the arm and the recording replay were added on 1 Nov, built with Codex from written briefs, as the DEVLOG says; evidence bundles include declared pre-existing brain snapshots. Limits: simulation only (that is exactly what makes the before/after replay exact); two robots, one simulator, no other simulator tested; "verified" means the tested scenes in the twin, not correctness in general. Where this could go: put on Ray-Ban Meta glasses, look at a real robot arm, say what went wrong; BlinkFlow writes the fix, verifies it in simulation first, and asks before it touches the real robot. That is future work; today the glasses are the microphone and the tap. Repo: https://github.com/samoxis/blinkflow · Public run report (no install, no key): https://samoxis.github.io/blinkflow/report · Arm run: https://samoxis.github.io/blinkflow/arm/ · Video: https://youtu.be/6EDwRU-f-xI