# Tactile Browser

- **Event:** [Gemini 4 Argon SF Hackathon](https://cerebralvalley.ai/e/gemini-4-argon-sf)
- **When:** Sun, Oct 25 at 9:00 AM – 10:00 PM (PDT)
- **Where:** San Francisco, CA
- **Team:** [Nasif Zaman](https://cerebralvalley.ai/u/znasif)
- **GitHub:** https://github.com/Znasif/monarch-chromium
- **Demo video:** https://youtu.be/Cc2gcv7Sehg?is=-1k6yyPLmusBt2D_
- **Gallery:** https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery
- **Page:** https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/56

---

**Monarch Tactile Browser** is a universal web, book, and tactile image browser built for the **HumanWare Monarch**—a refreshable 10-line Braille and **96×40 multi-line tactile graphic pin array** device running Android 13.

### The Problem It Solves
While standard screen readers allow blind users to read web text line-by-line, the visual web—scientific diagrams, textbook illustrations, scanned archival books (`archive.org`, Google Books), multi-panel webcomics (`xkcd`), and interactive charts—remains virtually inaccessible. Worse, naive attempts to convert web images into tactile graphics by thresholding or pixel-dithering onto a 96×40 pin grid produce dense, unreadable "walls of raised pins" where text labels collide with shapes and fine spatial structure is lost.

### Our Solution
Monarch Tactile Browser gives blind users what a modern browser gives sighted users: **seamless multi-line Braille reading of web pages combined with interactive, hierarchical tactile exploration of any visual graphic on the web—with zero hardcoded site rules (`No Code Names a Site`).**

1. **Universal Text-View & Scanned-Page DOM Bridge (`bridge/`)**:
   - Operates strictly on standard web signals (`<img>`, `<svg>`, `<canvas>`, `<figure>`, `[role="img"]`).
   - Promotes meaningful images and diagrams into keyboard- and screen-reader-focusable buttons (`Explore tactile image: <label>`) reachable via KeySoft's native **`B`** quick-navigation key, while filtering out tiny decorative icons and merging decorative drop-caps into prose.
   - Intercepts live `CanvasRenderingContext2D` drawing operations and inline `<svg>` vector DOM trees so exact geometry is captured before falling back to pixels.
   - Extracts embedded dialogue and printed text on scanned book pages and webcomics so users can read the text natively in multi-line Braille and explore cropped illustrations on the tactile pin array.

2. **Two-Tier Tactile Compiler (`tactile-core/` & `tier2/`)**:
   - **Tier 1 (Instant, Offline / Exam-Mode Ready, < 35 ms)**: Directly compiles `<svg>` and `<canvas>` vector primitives (`path`, `rect`, `circle`, `ellipse`, `line`, `polygon`, transforms) into crisp 1-pin outlines on the 96×35 tactile canvas (`SvgCompiler`), with a morphological Zhang-Suen centerline skeletonization fallback (`RasterContourCompiler`) that keeps raised-pin fill density strictly **≤ 25%** across all zoom levels (`1x..8x`).
   - **Tier 2 (`gemini-4-argon` Semantic & Structural Decomposer)**: Sends a 768 px copy of any raster photograph, diagram, scanned book figure, or comic strip to **Gemini 4 Argon** via the Interactions API and receives a hierarchical **Structured Tactile Graphic**—a tree of semantic components (`panel`, `figure`, `object`, `text`, `arrow`) with bounding boxes, normalized 1-pin centerline polyline strokes, spoken descriptions, and reading-order `embeddedText`.
   - **Clutter-Free Tactile Rendering (`StrokeRenderer`)**: Renders shapes as 1-pin centerlines with zero interior texture fill (putting color semantics into labels), suppresses inline lettering pins at `1x` overview so text remains a clean touch-to-read region without crowding adjacent strokes, guarantees a blank pin column between comic/diagram panels, and automatically prunes lowest-priority strokes first to maintain **≤ 25% pin fill** at every zoom level. Results are cached by image SHA-256 so no image is decomposed twice.

3. **Generalized `Data Navigator` Spatial & Hierarchical Graph + Hardware Viewer (`android-app/`)**:
   - Adapts Frank Elavsky's open-source `Data Navigator` paradigm from charts to arbitrary visual scenes (`ComponentGraph` + `HitGrid` with O(1) `IntArray(3840)` 1-pin halo hit-testing at 40 Hz).
   - **Touch-to-Read (40 Hz First-Finger Tracking)**: Touching any part of the 96×35 tactile canvas speaks the component's label and renders it in user-profile Braille on the bottom 4-row **Braille Caption Bar** (rows 36–39, separated by blank row 35) without mutating the physical pins under the user's fingers.
   - **Hierarchical Drill-Down (`Enter` / Action Double-Tap) & Hardware Controls**: Pressing `Enter` (Dot 8), `Space`, or double-pressing the Monarch Action button with a finger on a component (e.g., *Panel 3* of a comic or the *Left Ventricle* of a heart diagram) isolates and auto-fits that single component across the full 96×35 array and exposes its child sub-components. Full hardware integration via `res/raw/commands.xml` binds the Monarch's physical Zoom ovals (`+`/`-`), Left D-Pad (previous/next component), Right D-Pad (viewport pan/zoom), and Pan keys.

---

## More from Gemini 4 Argon SF Hackathon

- [TechDataCenters](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/53)
- [Shadow CVE](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/54)
- [PortOS](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/55)
- [athens](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/57)
- [Team Blitz Learning](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/58)
- [Boundary Line](https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/59)

---

Markdown version of https://cerebralvalley.ai/e/gemini-4-argon-sf/hackathon/gallery/56. Site index for agents: https://cerebralvalley.ai/llms.txt · full text: https://cerebralvalley.ai/llms-full.txt