ActionOS - Zakaria Kaddari

ActionOS — Program What You See ActionOS is an agentic visual programming platform that enables anyone to perform advanced video understanding, event reasoning, and automated video operations simply by describing what they want in natural language. As a computer science researcher working with computer vision and multimodal AI, I've experienced firsthand a fundamental limitation of today's visual AI systems: recognizing objects and tracking their movements is relatively accessible, but understanding complex events, reasoning about sequences of actions, and turning those insights into reliable, executable operations still requires significant engineering. For example, finding when one car overtakes another isn't just about detecting two vehicles. It requires identifying their roles, tracking how their spatial relationships evolve, recognizing a meaningful transition from behind to alongside to ahead, and verifying that these conditions occur in the correct temporal order. More sophisticated tasks introduce additional challenges: detecting multi-step activities, reasoning about interactions between objects and people, evaluating temporal constraints, distinguishing incomplete observations from actual failures, and triggering actions based on combinations of conditions. Traditionally, these tasks require specialized computer vision pipelines, custom temporal logic, and extensive post-processing. ActionOS removes this complexity through two complementary capabilities: First, it introduces natural-language visual programming. Users describe the events, sequences, conditions, and desired operations they want to perform. ActionOS compiles these instructions into structured, inspectable, and versioned visual programs that can be reviewed, modified, and executed without writing code. Second, ActionOS provides an advanced video reasoning and execution engine. It combines semantic video understanding, object segmentation and tracking, temporal event evaluation, and evidence-based reasoning to detect meaningful events and track how they unfold over time. Rather than merely labeling video content, it evaluates complex relationships, ordered actions, and conditional rules against source-backed observations. The results can trigger executable operations such as extracting event-specific video clips, generating timestamped evidence reports, and supporting further analysis. Users can also revise their visual programs in natural language, introducing new conditions and reevaluating existing evidence. For example, a user can ask: "Find when the red car overtakes the yellow car," then refine the rule to: "Only count overtakes where the red car remains ahead for at least two seconds." ActionOS is designed to distinguish confirmed observations from uncertain ones, preserving traceability to source timestamps and visual evidence rather than relying solely on opaque AI conclusions. Our vision is to make advanced video intelligence programmable. By bringing together natural-language programming, complex event reasoning, and executable visual operations, ActionOS enables researchers, developers, educators, and professionals to analyze and interact with visual data in ways that previously required substantial custom engineering. We want to move beyond AI that simply sees and describes the world, toward AI that can reason about observable events and execute user-defined visual workflows.