yantraOS

Most computer-use agents rely heavily on a repetitive screenshot loop: capture the screen, send it to a vision model, predict pixel coordinates, perform an action, wait a few seconds, and repeat. This approach creates three major problems: Slow execution: Each action requires another screenshot and model inference. Even simple workflows can take several minutes. High token consumption: Sending large screenshots repeatedly increases vision token usage and makes basic workflows expensive. Unreliable interaction: Agents often cannot tell whether an application has finished loading or a menu has opened. This leads to missed clicks, incorrect focus, and workflows getting stuck. We believe computer use needs a fundamentally better approach. The Solution: Extra's Dual-Plane Architecture Extra is an open-source, model-agnostic computer-use engine and desktop companion app for Windows, macOS, and Linux. It exposes native computer interaction capabilities through the Model Context Protocol (MCP). Instead of depending entirely on screenshots and arbitrary waiting periods, Extra combines two complementary approaches: 1. Deterministic UI Tree & Semantic Grounding Extra uses native operating system accessibility interfaces, including Windows UI Automation (UIA), Linux AT-SPI2, and macOS accessibility APIs, to inspect interactive elements directly. It converts the available interface information into a compact semantic representation, helping AI models identify buttons, text fields, menus, and other controls without repeatedly guessing pixel coordinates. 2. SOUL: Real-Time Visual Settle Detection Traditional agents often use fixed sleep timers after every action. Extra's SOUL engine monitors visual changes locally and detects when the screen has stabilized, allowing the next action to execute without unnecessary waiting. Our visual settle detection operates in under 15ms in supported scenarios, reducing delays caused by arbitrary sleep timers. 3. Direct Action Feedback & Execution Batching Extra returns updated UI information directly in tool execution responses, reducing the need for additional screenshot and observation cycles. It also supports batching predictable actions, such as launching an application, focusing a window, and entering text, through native OS input mechanisms like Win32 keyboard events and Linux uinput. Together, these capabilities make computer interaction more efficient, predictable, and accessible to different AI models. Quantitative Impact: 85%+ fewer tokens: Reduces typical UI context payloads from approximately 1,600 tokens to around 200 tokens. Faster interaction: Visual settle detection under 15ms compared with traditional fixed waits of 3–5 seconds. Local-first architecture: Runs on the user's own machine without requiring a cloud VM or continuous screen streaming. Production testing: 2,154 automated tests passing across Windows, macOS, and Ubuntu, with Python 3.11–3.13 support.