#346 fix(capture): fall back to OCR when browser a11y tree is chrome-only

[Anonymous]

Originally created by: adityaharishch

Summary

• Chrome's a11y tree returns non-empty text even before page content is materialised — the tab strip (AXTab/AXButton nodes) contributes ~695 chars of tab titles alone
• try_walk_a11y had no thinness check, so it returned an Accessibility frame from chrome-only text and the OCR fallback never ran → actual page content was never captured
• Ports the content-density heuristic from screenpipe-capture::paired_capture::a11y_content_is_thin (the screenpipe CLI path that had this working): classifies each node's role as chrome vs content, falls back to OCR when content-role chars < 30% of total (or < 100 total chars)

Root cause

tray/src-tauri/src/capture/screenpipe.rs:try_walk_a11y only checked for an empty text_content to decide whether to OCR. Chrome's tab strip is non-empty (always present), so it always "won" over OCR — page content was never captured from day one on the screenpipe-fork tray path.

The screenpipe CLI (paired_capture.rs) already had a11y_content_is_thin solving exactly this; the tray's simplified engine was missing it.

Test plan

• [ ] Open Chrome to any page with visible text content
• [ ] Verify capture_frames rows now have accessibility_text / full_text with actual page text, not just tab titles
• [ ] Verify incognito windows still produce no frames (the Skipped path is untouched)
• [ ] Verify non-browser apps (VS Code, Slack) still use the a11y path (their nodes are mostly AXStaticText/AXGroup → ratio > 0.3)

[Anonymous]

Originally posted by: coderabbitai[bot]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 260ff6e1-7373-4abb-a4dc-32e3f5b2d1e1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:
- [ ] 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

- [ ] Create PR with unit tests - [ ] Commit unit tests in branch `fix/chrome-ocr-fallback`

---

_{Comment @coderabbitai help to get the list of available commands.}

[Anonymous]

Originally posted by: adityaharishch

Thanks for the thorough review. All findings addressed in commit 469621cc.

Finding [#1] — clock freeze (Critical): block_last_ts and block_last_frame_id now advance on every coding-agent skip when a block is open. A burst of skipped frames no longer freezes the gap-detector clock, so no spurious system_sleep gap fires when real editor frames resume.

Finding [#2] — unconditional thin check (High): a11y_content_is_thin is now gated on is_browser(&app_lower) inside try_walk_a11y. Permission dialogs, preferences panes, and any non-browser app with legitimately sparse a11y text are no longer pushed to OCR.

Finding [#3] — missing canvas-app / canvas-URL / meeting-app branches (High): Added is_canvas_app() with an early-return FallBackToOcr branch in try_walk_a11y for GPU-rendered design tools (Figma, Miro, Sketch, Canva, Framer, Penpot, Origami, Principle). These apps' content is absent from the AX tree; OCR is the only useful path. Meeting apps are left on the default a11y path — video frames aren't OCR-able, and a11y gives toolbar/chat text which is meaningful.

Finding [#4] — unanchored .contains() (High): Replaced session_lower.contains(name) with first_word.starts_with(name). "notcodex" or a process that happens to mention "claude" in a path argument no longer matches.

Finding [#5] — is_version_label false positives (High): Removed is_version_label entirely. Semver strings like "2.1.193" won't be detected, but that's the right tradeoff — the heuristic also matched Maven artifact strings, build-tool output, and anything with three dot-separated numbers.

Finding [#6] — empty-nodes guard (Medium): a11y_content_is_thin now returns false early when snap.nodes is empty. If the node budget was exhausted and we have text but no role metadata, we keep the a11y text rather than discarding it.

Finding [#7] — em-dash separator (Medium): is_coding_agent_terminal now strips both "Terminal - " (hyphen) and "Terminal \u{2014} " (em-dash) prefixes. Tests cover both forms.

Finding [#8] — code duplication: Acknowledged. The is_browser in screenpipe.rs and session_builder.rs stay separate for now (different layers, different call sites, different import graphs). Factoring into meridian-core is a clean-up deferred for a later pass.

Finding [#9] — hardcoded screenpipe-fork path (Medium): dev-start.sh now uses ${SCREENPIPE_FORK_PATH:-} — the --watch flag is only appended when the var is set and the directory exists.

Finding [#10] — dead "q" in CODING_AGENT_NAMES (Low): Removed from the array. The exact first-word == "q" check below it is the live gate.

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed