# Owly - **Event:** [Meta Global AI Developer Hackathon](https://cerebralvalley.ai/e/meta-ai-developer-hackathon) - **When:** Oct 26, 12:00 PM โ€“ Nov 4, 9:00 PM (EST) - **Where:** Online - **Team:** [Ahmad Ridlo Fadlli Robbi](https://cerebralvalley.ai/u/arfabi) - **GitHub:** https://github.com/arfabi/owly - **Demo video:** https://youtu.be/OIoH8C0ULLY - **Gallery:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery - **Page:** https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2201 **Owly.AI โ€“ Your AI Browser Companion** ## ๐Ÿ”ด The Problem Every year, millions of people around the world apply for life-changing opportunities: university scholarships, immigration visas, civil service registrations, public grants, and government welfare programs. These portals โ€” built for administrative compliance, not human usability โ€” consistently fail the very people they are designed to serve. The scale of the problem is documented. Immigration processing consultancies and visa legal networks widely estimate that **15โ€“30% of initial application setbacks** involve purely administrative or clerical errors โ€” incorrect form fields, mismatched passport details, missing document specifications, or non-compliant file formats โ€” that are entirely preventable. Across Schengen member states, overall rejection rates fluctuate between **10% and 20%** globally, with documentation errors cited as a consistent driver of non-substantive refusals. UK Visas and Immigration (UKVI) regularly identifies mismatched personal data, incorrect visa category selection, and poorly structured evidence as major sources of avoidable first-time rejections. Academic research on government benefit programs further demonstrates a direct correlation between **form complexity and administrative error rates**: as documentation requirements and conditional field logic increase, procedural denial rates rise sharply โ€” disproportionately affecting eligible applicants who simply could not navigate the system. In all cases, the applicant pays the non-refundable processing fee again and restarts from scratch. Applicants face a crushing combination of barriers: - **Ambiguous bureaucratic jargon** that differs across portals, languages, and jurisdictions, leaving applicants confused about what is actually being asked. - **Hidden disqualification traps** buried in footnotes, asterisked clauses, and conditional dependencies โ€” rules that are technically published but practically invisible to the average person. - **Strict, unforgiving formatting requirements** for fields like passport numbers, national IDs, file types, and word counts that result in silent rejections with no explanation. - **Physical and cognitive accessibility barriers** that make dense, information-heavy portals actively hostile to users with low vision, dyslexia, motor impairments, or non-native language fluency. - **Document anxiety** โ€” the fear of uploading sensitive personal identifiers (passports, tax IDs, national registration numbers) to unknown government or institutional servers with no transparency on data retention. The personal cost is severe. Applicants are disqualified not because they lack qualification, but because the system was never designed with them in mind. Processing fees are lost. Opportunities close. Dreams are deferred over a single mistyped digit or a misunderstood clause. --- ## ๐Ÿ’ก The Solution: Owly.AI Owly.AI is a privacy-first Chrome Extension (Manifest V3) and compound autonomous browser agent that accompanies users through every phase of a public application โ€” from understanding the form to submitting it with full confidence. It is built on a single architectural philosophy: > **"Models propose via Meta Muse Spark; deterministic validators dispose."** Owly does not replace the user. It does not submit on their behalf. It sits beside them as an active, intelligent companion that reads what they read, explains what they don't understand, guides them field by field, validates their answers before submission, and communicates with them in their own language and voice. Owly addresses six interconnected problem layers through six integrated features: ### 1. Forms Guide & Analyze Form Owly captures a compressed WebP screenshot of the live viewport and sends it to Meta Muse Spark's Compound Multi-Agent System. Three agents run in parallel: one analyzes the visual layout (institutional logos, deadline banners, visual authority markers), one retrieves official guidelines from an internal RAG database, and one synthesizes a complete structured roadmap with field strategies and disqualification traps. The result is a set of interactive numbered badges (1, 2, 3...) projected directly onto the live form inputs in the user's browser โ€” a step-by-step visual roadmap they can follow in sequence. ### 2. Field-by-Field AI Coach Clicking any badge opens a 4-part guidance card: what the field means in plain language, exactly how to fill it, a valid concrete example, and the most common mistake that causes rejection. Rendered in under 600ms using Meta Muse Spark with low reasoning effort, the coach provides context-aware, field-specific guidance that generic chatbots cannot deliver. ### 3. Explain This (Visual Snip) Users can drag to select and crop any confusing element on the page โ€” a legal clause, an institutional stamp, a table of requirements, or a dense footnote. Owly sends the cropped image to Meta Muse Spark's multimodal vision and reasoning pipeline, which returns a five-part structured explanation: the identified element name, a plain-language meaning, its legal implication, physical or digital authenticity markers to verify a genuine document, and a concrete recommended action. ### 4. Chat & Voice Companion Owly features a full dual-mode voice pipeline. Push-to-Talk (PTT) sends audio via chunked HTTP for rapid walkie-talkie queries. Live Talk maintains a persistent WebSocket for hands-free, continuous dialogue โ€” designed for applicants who hold physical documents with both hands. Both modes use `muse-voice-transcribe-1.0` for accurate speech recognition on complex jargon across 14 languages, streamed through a background Chrome AudioWorkletNode to eliminate UI thread lag. Transcribed intent feeds directly into Meta Muse Spark's 14-tool autonomous ReAct loop, which can highlight fields, navigate the page, and control accessibility features in real time. ### 5. 1-Click Autofill & Private Vault Users upload a government ID, passport, or certificate once. Meta Muse Spark's vision OCR extracts structured data entirely in server memory โ€” no data written to disk, no data logged. The extracted data is immediately encrypted in the user's browser RAM using WebCrypto AES-256-GCM with a PBKDF2-derived key (100,000 iterations). With one click, any form field on any website is filled automatically from the vault, with full undo support. ### 6. Pre-Submit Form Audit Before submission, Owly runs a hybrid 4-layer audit: a deterministic client engine checks required attributes, international format compliance (ICAO 9303 passports, global national IDs, E.164 phone numbers, file size and type limits, minimum word counts), and Levenshtein-distance typo detection against vault-verified data โ€” all in under 5ms. Concurrently, Meta Muse Spark with high reasoning effort analyzes unstructured page footnotes and cross-field logic contradictions that no rule engine can read. Results surface as a traffic-light card (green / amber / red) with the exact field highlighted and one-click corrective actions. --- ## โš™๏ธ Sophistication of Meta AI Tooling Owly.AI is not a wrapper around a single API call. It is a compound autonomous system engineered around Meta's native model capabilities in ways that would be impossible to replicate with any other stack: Multimodal Vision (image_url โ†’ <|patch|> tokens) Agent 1 reads live 1024px WebP screenshots to extract institutional authority, visual deadlines, and form context that DOM parsers structurally cannot access. Native ASR (muse-voice-transcribe-1.0) Streams 16 kHz Mono PCM audio via Chrome's dedicated AudioWorkletNode (zero main-thread lag). PTT via HTTP chunks; Live Talk via persistent WebSocket. Superior to Chrome Web Speech API on jargon, accents, and 14-language robustness. Autonomous Function Calling (ReAct Loop) Muse Spark drives 14 deterministic browser tools in a continuous verification loop โ€” find_and_highlight_element, set_accessibility_feature, capture_current_screen โ€” executing spoken user intents as live DOM mutations. Dynamic Reasoning Effort (low / medium / high / xhigh) Adaptively tuned per task: low (<600ms) for field coaching, medium for visual OCR and snip analysis, high for cross-field audit contradiction detection, xhigh for multi-clause essay coherence evaluation. Structured JSON Output Enforcement Outputs are schema-validated JSON aligned 1:1 with live DOM field IDs, preventing autofill targeting errors and hallucination-driven form corruption. The system runs 9 specialized agents: a 3-agent Core Triad (Perception, Regulation Researcher, Decision & Synthesis) and 6 on-demand agents (Field Coach, Pre-Submit Auditor, Show Me L3/L4, Visual Snip Explainer, A11y Intent Router). --- ## ๐Ÿ“ˆ Impact **Immediate User Impact:** - Removes the expertise barrier from complex public applications. A first-generation university applicant and an experienced immigration lawyer now have access to the same quality of form guidance. - Prevents irreversible disqualifications caused by formatting errors, overlooked conditions, and document mismatches. - Provides accessibility-first design for users with low vision, dyslexia, motor impairments, or language barriers โ€” supporting 14 languages with WCAG 2.2 AAA-compliant tools. **Data Privacy Impact:** - Eliminates the need to trust opaque third-party platforms with sensitive personal documents. All PII is encrypted client-side before any external call and is never stored, logged, or used for model training. **Systemic Impact:** - Challenges the assumption that public-sector digital portals must remain exclusionary by design. Demonstrates that an AI layer can make existing bureaucratic infrastructure universally accessible without requiring any change to the portals themselves. --- ## โœ… Outcome In one week of development, Owly.AI delivered: - A fully functional Chrome MV3 extension with 9 AI-powered features deployable on any public web portal. - 20/20 end-to-end test cases passing with 816ms average live inference latency and <5ms cache response. - Zero-knowledge document vault with AES-256-GCM encryption verified against standard cryptographic benchmarks. - Complete 14-language internationalization across all UI strings and AI output formatting. - An open-source codebase with 40+ incremental commits, full setup documentation, and `.env.example` for immediate reproducibility. --- ## ๐ŸŒ Vision & End Goal Owly.AI's long-term vision is to become the universal access layer between people and the complex digital systems that govern their lives. Today, Owly works on scholarship and visa portals. Tomorrow, it can work on tax filing systems, healthcare registration platforms, court document submission portals, and welfare application services โ€” anywhere that an ordinary person faces an extraordinary administrative burden. The end goal is not to automate bureaucracy. It is to ensure that no person โ€” regardless of their education, language, age, ability, or access to legal counsel โ€” is structurally excluded from the systems that exist to serve them. > **Every person deserves a companion who understands the system, speaks their language, and stands beside them when the stakes are highest.** That is Owly. ## More from Meta Global AI Developer Hackathon - [Nexby](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2198) - [Ailism](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2199) - [CoreAudit](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2200) - [JunSuzukiJapan](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2202) - [Charie-AI-Data-Analyst](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2203) - [SENSI](https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2204) --- Markdown version of https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2201. Site index for agents: https://cerebralvalley.ai/llms.txt ยท full text: https://cerebralvalley.ai/llms-full.txt