dekant

AgentSeg is an agent that labels video. You tell it what to label in plain language, like "label the red car in every frame", and it does the whole job: it picks a good frame, draws a box, checks the mask, tracks the object through the video, reviews the frames that look off, fixes what's wrong and exports YOLO segmentation labels. Labeling is still the slow part of building a vision model, especially for robotics teams. A robot has to recognize the specific objects in its own environment, so there's rarely a ready dataset. Someone has to mask hundreds or thousands of frames, and most teams still do it by hand. SAM 3 helps a lot, but on its own it isn't enough. Given only a name, it can miss an unusual object entirely. Given a good box, it can drift onto another object partway through the video and never come back, and it gives no sign that anything went wrong. So we put an agent on top of SAM 3. Muse Code, running Muse Spark, drives our command-line tool (the same one a person can use) and follows a skill file that explains how to label. The backend keeps the agent's keyframes as the source of truth and flags frames where the mask vanished, jumped, broke apart or stopped looking like the keyframe. The agent looks at those frames and fixes only what's wrong. Everything the agent does shows up live in our labeling UI: its cursor, its boxes and clicks, the frames it looks at, and a short line about what it's doing. A person can take over with the normal manual tools at any time, and every run can be replayed. The agent does much better than plain SAM 3. On a 1000-frame car chase from the LVOS benchmark, where the target car disappears behind buildings and goes through a tunnel, box IoU against ground truth was: - SAM 3 with a text prompt: 0.29 - SAM 3 from the agent's first box, no review: 0.51 - AgentSeg: 0.94 Plain SAM 3 loses the car in the tunnel and never finds it again. The agent catches the mistake and stays on the car to the last frame.