Rygtus

KARYA Node is a Windows-native, local-first autonomous agent for document-heavy government/office workflows. It perceives the screen, drives desktop and browser apps, processes documents, writes code, trains computer-vision models, validates databases, and escalates ambiguous cases to humans — with a full audit trail and a policy guard on every action. # ⚡ KARYA Node — Problem Statement & System Specification ## 1. Executive Summary **KARYA Node** (from the Sanskrit *Karya* meaning duty or work) is a Windows-native, local-first autonomous operations agent designed for document-heavy workflows in Indian government offices, public sectors, banks, and small businesses. Unlike conventional automation tools or generic LLM chatbots, KARYA Node runs the core **Sense → Decide → Act → Check → Recover** loop on-device to operate legacy Windows applications, spreadsheets, and web browsers. It protects sensitive data by keeping it local, functions seamlessly during network blackouts, automatically verifies its own work, and escalates ambiguous decisions to human officers. --- ## 2. Context & Background (The Bharat Office Challenge) Public administration and commercial operations in developing regions, especially in India (Tier-2/Tier-3 districts and rural talukas), are governed by highly repetitive, manual, and document-heavy processes. These environments present unique operational challenges: ### 2.1 Low and Unstable Network Connectivity Offices frequently lose internet access. Stalling operations due to lack of network is unacceptable. Cloud-only agents or software-as-a-service (SaaS) products that fail when the internet is disrupted cannot be deployed. ### 2.2 Legacy Desktop Applications A significant portion of national record-keeping is locked behind 10-to-20-year-old desktop software (built on Win32, VB6, Java Applets, or custom forms) that lack APIs. Traditional Robotic Process Automation (RPA) tools are highly brittle, breaking whenever the application UI changes by even a single pixel. ### 2.3 Sensitive Personal and Citizen Data Citizen applications contain sensitive data, including Aadhaar numbers, bank account details, and land registry records. Under local sovereignty and data privacy laws, sending this information to foreign cloud servers for LLM inference is highly restricted or illegal. ### 2.4 Lack of Tech-Savvy Operators The operators of these legacy systems are not developers or IT professionals. They require intuitive interfaces—such as voice commands or simple natural language instructions—rather than complex configuration consoles. --- ## 3. The Core Problem Statement > **"How can we build an autonomous, resilient, and secure desktop operations assistant that operates legacy systems without APIs, respects local data sovereignty, handles network drops, and ensures 100% data entry accuracy without hallucinating or taking unauthorized actions?"** To solve this, we must replace two failing paradigms: 1. **The Brittle RPA Script:** Traditional macro-recorders fail when UI layouts shift, validation errors occur, or applications crash. 2. **The Passive Chatbot:** Standard LLMs can discuss a spreadsheet or draft an email but cannot verify the results, navigate the screen, run terminal commands, or submit entries. --- ## 4. Key Design Objectives KARYA Node solves the problem using six core principles: ``` ┌────────────────────────────────────────────────────────┐ │ KARYA │ │ Sense → Decide → Act → Check → Recover │ └────────────────────────────────────────────────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ 👁️ Vision 🔒 Policy ✅ Verifier 👤 Human Perception Guard (Independent) Escalation ``` ### 4.1 Local-First Reasoning The primary brain is **Gemma 3** running locally on-device. Sensitive information stays inside the office. A cloud path via the **Gemini API** is utilized only as a secondary fallback for complex multi-modal analysis. ### 4.2 Screen Perception (Computer Vision) Rather than relying on static coordinate clicks, the agent utilizes a visual perception loop. It takes screenshot captures, feeds them to a vision-language model, understands the current active application/state, and computes the exact keyboard/mouse actions required. ### 4.3 Safe by Design (Policy Guard) A local guardrail module enforces safety by intercepting every action. It blocks destructive commands (`rm -rf`, disk formatting), restricts folder paths, and enforces explicit human approval for any transaction that has real-world consequences (e.g., submitting an application online, making database changes). ### 4.4 Self-Correction & Verification An independent verifier module double-checks every entry by comparing the source document against the output database or screen state. If an error is detected, the agent attempts to fix it. If the application crashes, the system uses durable checkpoints to resume from the last clean state without duplicating operations. ### 4.5 Human-in-the-Loop Escalation The agent does not guess. When faced with handwritten documents with low confidence, conflicting details (e.g., mismatched name and ID), or policy violations, the task is paused. It is escalated to a human queue with a clear explanation and supporting evidence. --- ## 5. System Architecture & Modular Agent Ecosystem KARYA Node is divided into 15 interconnected modules, allowing a single operator or IT administrator to toggle features as needed. ``` ┌────────────────────────────────────────┐ │ React Glassmorphism UI (Port 5173) │ └───────────────────┬────────────────────┘ │ (REST / WebSockets) ▼ ┌────────────────────────────────────────┐ │ FastAPI Orchestrator (Port 8765) │ └──────────┬───────────────────┬─────────┘ │ │ ┌───────────────────┘ └───────────────────┐ ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────┐ │ Agent Core │ │ Modular Engines │ │ ─────────────────────── │ │ ─────────────────────── │ │ • LangGraph Runtime │ │ • Desktop Vision Agent │ │ • Gemma Local / Gemini │ │ • Playwright Browser │ │ • Policy Guard │ │ • Code Agent (Git/CLI) │ │ • Independent Verifier │ │ • Server Manager │ │ • Audit Logger │ │ • StratifyLabs (CV) │ │ • Offline Sync Queue │ │ • ValETL (DB Validation)│ │ • SQLite Database │ │ • Creators Hub (Video) │ └─────────────────────────┘ └─────────────────────────┘ ``` ### 5.1 Work Queue Manager (`manager.py`) Maintains the state machine for all operational cases (Pending, Processing, Completed, Failed, Human Review, Waiting). It persists state to SQLite, allowing it to survive system reboots or power outages. ### 5.2 Document Extractor & Validator - **Extractor (`extractor.py`)**: Uses local PDF parsing and Tesseract OCR to convert scans, images, spreadsheets, and Word documents into structured JSON. - **Validator (`validator.py`)**: Checks for complete fields, formats (e.g., Aadhaar and phone verification), and flags duplicates or database conflicts. ### 5.3 Desktop Vision Agent - **Screen Perception (`screen_perception.py`)**: Captures screenshots and uses Gemini Vision to detect active windows, buttons, input fields, and potential error dialogs. - **Windows app controller (`controller.py`)**: Executes actions via `pywinauto` or PyAutoGUI, automating inputs into legacy applications with automatic crash recovery. ### 5.4 Playwright Browser Agent (`automation.py`) Performs web operations in a sandboxed Chromium session. It records full video sessions of web actions, allowing managers to replay web submissions for security audits. ### 5.5 Coding Agent (`coding_agent/agent.py`) An autonomous software engineer that writes Python/JS scripts, runs terminal commands, executes tests, and commits code to local Git repositories. It automates repetitive coding chores and handles local bug fixes. ### 5.6 StratifyLabs Integration (`stratify_bridge.py`) An automated bridge to the StratifyLabs vision training platform. The agent translates natural language instructions (e.g., "Train a ResNet model on my vehicle detection dataset") into configuration parameters, starts the PyTorch/FastAPI server, monitors metrics, and runs inference. ### 5.7 ValETL Integration (`bridge.py`) A database validation bridge. It handles database schema discovery, translates natural language questions into secure PostgreSQL queries, and validates data migrations (comparing source database values against target database values). ### 5.8 Creators Hub (`video_generator.py`) Powered by **Gemini Omni Flash** via the Interactions API. It supports stateful, multi-turn conversational video editing, enabling non-technical users to generate, modify, and swap visual elements in videos using simple natural language feedback. --- ## 6. End-to-End Validation Suite (The Demo Case) To prove KARYA Node meets true enterprise requirements, it includes a simulation testbed: 1. **Synthetic Data**: Generates a set of 50 disaster-relief case documents containing a mix of complete files, blurry images, incomplete forms, conflicting IDs, and duplicate entries. 2. **Legacy App Simulation**: Runs a custom-built desktop application representing typical government data entry screens. 3. **Test Flow**: - The operator instructs the agent: *"Process today's relief cases, enter valid ones, flag conflicts, and alert me to exceptions."* - KARYA extracts the cases, enters them into the simulated app, and recovers from deliberate app crashes using LangGraph checkpoints. - It queues web checks during simulated network drops and syncs them once online. - It flags mismatched records for human review. - The independent verifier checks the final DB state and generates a compliance audit report.