#324 feat(tray): in-process capture — boundary + screenpipe-screen engine (Gap-2 Bucket 2, slices 1-2)

[Anonymous]

Originally created by: Akarsh-Hegde

Starts Bucket 2 (in-process capture) — the cutover that makes the drag-to-Applications DMG a complete product (one "Meridian" Screen-Recording TCC entry, no external screenpipe). Plan: Obsidian Decisions/Bucket 2 implementation plan - in-process capture.md. Spike verdict: GO.

All behind the off-by-default capture feature — default builds, CI, and the current DMG are unchanged and pull none of the capture stack.

Prerequisite done

Created Meridiona/screenpipe-fork (private) — a clean snapshot of mediar-ai/screenpipe @ last-MIT 892199f74 + a provenance NOTICE (MIT LICENSE preserved; nothing ≥ the commercial relicense 81e412ff5).

Slice 1 — boundary (capture/mod.rs)

• CapturedFrame mirrors the columns meridian's ETL reads (app/window/url/timestamp + text + text_source). Text-only — no pixels/video (privacy; no ffmpeg binary needed).
• CaptureEngine trait + mpsc FrameTx → backend (screenpipe-screen now, scap later) and consumer (logger now, meridian.db writer in slice 4) are independently swappable.

Slice 2 — engine (capture/screenpipe.rs)

• ScreenpipeEngine via the fork: list_monitors → capture_monitor_image → perform_ocr_apple, in-process, emitting OCR text.
• Calls CGRequestScreenCaptureAccess() at init (the lib only preflights + returns empty on denial — never prompts; the key spike finding).
• Runs on isolated tokio tasks — a capture panic ends only that task, never the tray (we gave up the screenpipe daemon's process isolation).
• Monitor-level OCR only; window metadata + a11y-tree text + ui_events = slice 3, meridian.db write = slice 4. Consumer logs frames for now.

Wiring

• anyhow + optional screenpipe-screen git-dep pinned to the fork rev; capture = ["dep:screenpipe-screen"].
• .cargo/config.toml: net.git-fetch-with-cli so cargo authenticates to the private fork via the user's git creds. CI capture builds will need a deploy token (default/feature-off builds use the lockfile, don't fetch/build the fork).

Verified

• default cargo clippy clean — fork not built, no repo access needed ✓
• cargo build/clippy --features capture clean — fork + native frameworks compile ✓
• full pre-push suite green.

Not yet (next slices)

3: window/app/url metadata + a11y-tree text + ui_events (Accessibility pane) · 4: meridian.db capture table + repoint the reader · 5: wizard grants as Meridian · 6: retire stock screenpipe from the DMG · 7: dev parallel-run parity check.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
• Added in-process screen and input capture with database persistence
• Implemented current task progress tracking and display in tray
• Introduced DMG auto-update support for macOS with version checking
• Expanded setup wizard with system specs detection and model selection
• Added task-specific tooltip display in menu bar

• Implemented background MLX runtime upgrades

• Refactors

• Restructured ETL to consolidate database operations
• Reorganized health checks to use in-process capture tables

• Redesigned tray UI with live timer, category tiles, and task details

• Bug Fixes

• Added database busy timeout handling
• Cleaned up legacy background service remnants

[Anonymous]

Originally posted by: Akarsh-Hegde

🤖 Code review — in-process capture + screenpipe cutover (overnight, high-recall)

Reviewed the full PR diff (3b/4a/3c/4b) via 8 finder angles + dedup/verify. CLAUDE.md conventions clean (headers, no unwrap, migrations all-new, none >500 lines). Findings ranked by severity.

✅ Fixed in this review

• SQLITE_BUSY on the daemon ETL — setup_db opened meridian.db with no busy_timeout while the tray now writes the same DB; daemon writes would fail immediately under contention. Fixed in 75b6633 (5s, matching the tray's open_existing).

🔴 Critical — gates merge to main / shipping

• The capture feature is OFF by default and no release build enables it (tray/src-tauri/Cargo.toml, tauri build passes no --features). The daemon now reads only the capture tables, so a packaged release would capture nothing — a silent no-op pipeline, with screenpipe already removed as a fallback. Do not merge/ship until the capture-enabled build is the shipping default (DMG path). The stock-screenpipe early-dev install is incompatible with this branch.

🟠 Real bugs to fix before the cutover ships

• commands/setup.rs::check_screen_recording still detects the grant via pgrep screenpipe + ~/.screenpipe/db.sqlite (not in this diff, but the cutover invalidates it). On an in-process install there's no screenpipe process/DB, so the first-run wizard reports Screen Recording as ungranted even when granted to Meridian. Switch to CGPreflightScreenCaptureAccess() on the tray (slice 5).
• health/capture.rs::capture_coverage ghost-detection is miscalibrated for in-process capture. It assumes ≥1 frame per focus event (true for screenpipe's push-based per-focus frames), but the new engine samples on a fixed 2s cadence decoupled from focus. An app focused repeatedly for <2s (quick alt-tabs) accumulates focus events with 0 frames → false CRITICAL "ghosted app". Recalibrate for time-driven sampling.

🟡 Accepted degradations (already documented, flagging for the record)

• All gaps now classify system_sleep; idle_frame_count always 0 — insert_capture_frame never writes capture_trigger (NULL until in-process idle detection lands), so count_frames_in_window's idle branch is unreachable. Tests stay green only because they seed capture_trigger directly. (Audio is likewise dropped — get_audio_snippets stubbed empty.)

🟡 Observability regressions from the health simplification (follow-up)

• No ground-truth Accessibility-grant probe remains — the screenpipe-log/a11y-helper probes were removed; only the statistical DB-yield proxy is left, which fires late. Consider an in-process AXIsProcessTrusted check.
• No tray/capture process-liveness check — capture-down is detected only by the 600s frame_freshness proxy (10-min blind window). The old screenpipe_service() pgrep was immediate.
• wal_size deleted, not repointed — meridian.db is now the high-write sink but nothing watches its WAL for stalled checkpoints.

⚪ Minor / cleanup

• capture_coverage compares RFC3339 …Z timestamps against datetime('now',…) (space-separated, no Z) → the rolling window skews to midnight of the cutoff day (low-risk; both subqueries skew identically).
• ui-event consumer does one INSERT per event on a high-frequency path (clicks/keys) — consider batching per drain to reduce write-lock contention.
• Vestigial/dead (intentional, noted in commits): Config.screenpipe_db/SCREENPIPE_DB unused by the daemon; the whole audio plumbing (AudioSnippet, join arm, merge, JSON column) is now a dead path fed by an empty stub; db::screenpipe module name kept; duplicated share/share_pct + insert_capture_frame/insert_capture_ui_event tails; capture_once_ocr re-enumerates monitors every 2s tick.
• Migration 048 (cursor reset to 0) is safe by ordering (daemon applies migrations before first run_etl) — noted, not a bug.

Generated overnight by Claude Code. The 🔴 + 🟠 items are the ones to resolve before this branch becomes the shipping default.

[Anonymous]

Originally posted by: Akarsh-Hegde

✅ Manual end-to-end verification (2026-06-22)

Ran a --features capture tray + the daemon against one shared throwaway MERIDIAN_DB, exercising 4 real apps (VS Code, DBeaver, Arc, Obsidian) over ~5 min. The cutover is functionally validated end-to-end and multi-app — including the link I flagged as unverified in the earlier review comment (capture frames actually flowing through the daemon ETL into app_sessions).

E2E capture → ETL → sessions — PASS.

• capture_frames: 153 frames across Code/DBeaver/Obsidian/Arc, all text_source=accessibility (a11y capture; OCR fallback not exercised — every app exposed an a11y tree), clean ~2s cadence.
• capture_ui_events: click×53, text×21, window_focus×18, app_switch×12, clipboard×11.
• Daemon ETL consumed the capture tables and produced per-app app_sessions + active_session. 5 etl_runs, all status=success.

Concurrent-write / busy_timeout (75b6633) — PASS. 5 ETL runs while the tray streamed writes to the same DB → zero database is locked / SQLITE_BUSY, no failed runs. The 5s busy_timeout on the daemon pool holds under sustained tray+daemon contention.

App-switch fidelity — PASS. app_switch events tracked every focus change in order. Finder/UserNotificationCenter appear as app-switch events with no frames (focus <2s) — the input recorder catches transitions the 2s frame cadence misses; the two paths complement each other as designed.

Gaps — PASS (expected). gaps empty in this short run; idle detection unwired, so anything appearing classifies system_sleep per the accepted v1 degradation.

Finding — VS Code session fragmentation: confirmed BY DESIGN, not a bug. A continuous VS Code stretch split into ~19 sessions because vscode_project() (src/etl/runner.rs:260) re-splits on the integrated-terminal tab title (+ Claude-Code spinner glyphs ✳/⠂). Clean contrast: DBeaver captured as a single 28-frame/55s block (not is_vscode_like). Product decision (Akarsh): this granularity is intentional — Meridian is a developer-efficiency tool and wants fine-grained context from dev IDEs/tools, so per-project/per-tool splitting inside an editor is a feature. Only possible future tweak is a noise-only one (strip the animated spinner glyph so the same terminal tab doesn't split frame-to-frame) — deferred, low priority.

Net: the two blockers from the prior comment still stand before this becomes the shipping default (capture feature off by default; check_screen_recording still pgreps screenpipe). The pipeline itself is verified working.

[Anonymous]

Originally posted by: Akarsh-Hegde

✅ Both shipping blockers resolved (2026-06-22)

The two blockers from the verification comment above are fixed and pushed (75b6633..3119f74, full pre-push suite green):

1. capture feature is now default-on — 3119f74 a01ad01
tray/src-tauri/Cargo.toml now sets default = ["capture"]. Every release path (tauri build, npm run build, npm run tauri build) and a bare cargo build in tray/src-tauri builds capture-on — there's no per-call --features flag to forget, which is the exact footgun that created this blocker. --no-default-features still builds a lean capture-free tray.

• Verified: cargo tree shows screenpipe-screen in the default dep tree and absent with --no-default-features; tray clippy clean.
• ubuntu CI is unaffected — root cargo clippy/test select the meridian package only and never compile this macOS-only crate (cidre/screenpipe-screen).

2. check_screen_recording no longer pgreps screenpipe — 3119f74
Replaced the pgrep screenpipe + ~/.screenpipe/db.sqlite proxy with CGPreflightScreenCaptureAccess() — a pure, prompt-free read of the tray's own Screen Recording grant (the tray is the capturing process post-cutover). Implemented as a raw extern "C" FFI mirroring the existing AXIsProcessTrusted pattern in the same file → no new dependency.

Scope note (deliberately deferred to slice-5): this fixes the check. The wizard's Screen-Recording grant action (the button that surfaces the system prompt) may still assume screenpipe spawns to trigger the dialog; if so it needs CGRequestScreenCaptureAccess(). Left out to keep this scope bounded.

Also in this push: b845647 drops the vestigial disk (screenpipe) health check (capture data lives in meridian.db under ~/.meridian, already covered by disk (meridian); the old check pointed at the retired ~/.screenpipe). Removed rather than relabelled disk (capture) — a relabel would either mispoint at the dead path or duplicate the meridian check.

[Anonymous]

Originally posted by: Akarsh-Hegde

Setup-wizard fixes + macOS install/signing hardening — fresh-account / Air test (2026-06-22)

End-to-end DMG test on a clean user surfaced a batch of wizard + macOS-permission issues; all fixed (commits 34edc5b..5573294).

Wizard fixes
| Fix | Commit | Summary |
|---|---|---|
| 1 — manifest 404 | 34edc5b | build:staging bakes the runtime-staging manifest URL; rerun-if-env-changed guard |
| 2 — Input Monitoring card | 33a8404 | third Permissions card + check_input_monitoring (IOHIDCheckAccess) |
| 3 — eager spec-aware model download | 3c9756b | /prefetch_model + /prefetch_status, wizard progress, card copy |
| A — onboarding notice | 491479f | gate mlx.down on ~/.meridian/onboarded; drop dev fix-text |
| B — Input Monitoring grant | 1a6dfee | request_input_monitoring (IOHIDRequestAccess) so the app registers in the pane |
| C-a — prefetch hardening | 100876a | check HTTP status; degrade silently on a runtime missing the endpoints |
| C-b — runtime republish | 8942252 | bump services → 1.60.0; tag runtime-staging-v1.60.0 (published) ships the endpoints |

macOS permission / install hardening (from the repeated-Screen-Recording-prompt debugging)
| Change | Commit | Summary |
|---|---|---|
| preflight permissions | b3beecb | check-before-prompt for Screen Recording / Accessibility / Input Monitoring |
| stale-daemon footgun | 3d3634b | build/build:staging rebuild the daemon (build:daemon) before bundling |
| installer hygiene | 0c36a55 | backend_install purges a leftover pre-cutover com.meridiona.screenpipe agent/binary on install |
| stable dev signing | 5573294 | scripts/dev-signing.sh + APPLE_SIGNING_IDENTITY wiring → constant self-signed identity so TCC grants persist across rebuilds (no more cdhash churn / re-prompt loop) |

Root cause of the prompt loop: not a code bug — TCC grant churn. Ad-hoc rebuilds change the cdhash; macOS anchors permission grants to the signing identity, so each build looked ungranted → re-prompt + dead duplicate TCC records. Fix = stable signing (Lever above) + clearing stale records. Verified on two machines: in-process capture_frames flows (Studio 414 / Air 2331, live) with no re-prompt.

Follow-up (b) — background runtime auto-upgrade (2026-06-23)

Change	Commit	Summary
background runtime auto-upgrade	e267939	tray poll loop checks for a newer published runtime (~6 h cadence + tick 0) and swaps it in in the background — installed machines pick up runtime republishes without an app update or a manual rm -rf ~/.meridian/runtime

Design — stage-and-swap, not stop-then-download. download_runtime split into download_and_stage (download + SHA-256 verify + extract → runtime.incoming/) and commit_staged_runtime (atomic rename runtime→runtime.old, runtime.incoming→runtime, stamp version, rollback on failure). The download/extract run with the old server still serving, so the offline window is just the instant rename + restart — the same blip any supervise restart already produces. A 60–120 s stop-then-download window would instead guarantee the daemon's "Classifier offline" notice (it probes /health every ~60 s with no debounce); a brief transient toast during the restart is still possible but self-clears next tick. No daemon changes, no cross-process marker. auto_upgrade_runtime never auto-provisions (first ~156 MB pull stays the wizard's job); the poll loop pauses supervise_mlx while an upgrade runs; the in-flight flag is Drop-guard + 600 s-timeout protected so a stalled download can't latch it off; after restart it holds in-flight until the new server answers /health (bounded ~60 s) so a slow-binding server isn't killed as "wedged". Wizard path + live live_pull_from_runtime_staging test behavior-unchanged (now exercises stage→commit; green against runtime-staging 1.60.0).

Known follow-ups (not in this PR): scripts/install-from-bundle.sh is still the old screenpipe-based npm installer (separate cutover cleanup); shipping still needs Developer ID + notarization.

[Anonymous]

Originally posted by: coderabbitai[bot]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 07da3759-03eb-4584-89e5-f66ec379de03

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:
- [ ] 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

- [ ] Create PR with unit tests - [ ] Commit unit tests in branch `feat/in-process-capture`

---

_{Comment @coderabbitai help to get the list of available commands.}

[Anonymous]

Originally posted by: Akarsh-Hegde

✅ Review findings addressed — 7732a82

Worked through the CodeRabbit changes-requested set (34 inline) + the overnight review's 🟠 items. All fixes in one commit; full pre-push suite green (fmt · ui build · clippy · ui tests · security audit · cargo test 323 passing). Each finding was verified against current HEAD first — a couple were already fixed by later pushes and skipped.

🦀 Capture / data integrity (writer-side)

• meridian-core/capture.rs — normalise text_source to the ocr|accessibility contract before persisting (binds the normalised value, not raw input); enforce the capture_ui_events contract at the write boundary — text_content nulled for every non-clipboard type (privacy), unknown event_type dropped with a warn. Added tests for both.
• Migrations 046/047 CHECK constraints — SKIPPED (intentional). Editing an already-applied migration trips the sqlx-checksum crash-loop (cf. the 031-renumber incident) and violates the "never modify a migration" rule — those tables are live on test machines. The tray is the only writer, so the writer-side enforcement above is a genuine substitute for the SQL CHECKs, not a downgrade. A table-rebuild migration would be heavy for marginal value.
• Deviation from suggestion (capture.rs event_type): warn-and-drop rather than bail! — this module's contract is "a schema lag never crashes the capture loop" (it already swallows missing-table → Ok); a hard error would break that idiom.

🩺 Health (src/health/capture.rs)

• Blank-rate now uses COALESCE(NULLIF(full_text,''), NULLIF(accessibility_text,'')) so an empty full_text can't mask non-empty a11y text.
• Distinguish empty capture_frames (→ permission prerequisite hint) from unreadable (→ schema/migration fault, no permission remedy) via a new FramesState.
• capture_coverage ghost-detection (overnight 🟠) — DEFERRED, with reason. The miscalibration (focus-event count vs the new time-driven 2s cadence → false CRITICAL on rapid <2s alt-tabs) is real, but the discriminating fix is gating on focus duration, which needs live tuning data to set thresholds. It's a heuristic, not in the changes-requested set, and fixing it blind risks a worse calibration. Tracking as follow-up.

🛠️ Tray stability (mlx_server / poll / update / refresh / lib)

• Critical (poll runtime swap): the outer MLX_UPGRADE_TIMEOUT could cancel auto_upgrade_runtime mid-commit_staged_runtime (after stop_server/the rename), leaving no live runtime. Reworked so only the network download phase is time-bounded (inside mlx_server, via RUNTIME_DOWNLOAD_TIMEOUT) and the stop→swap→restart runs to completion. The whole path stays bounded (download cap + bounded health-wait), so the in-flight flag can't latch.
• download_runtime (wizard first-run) now bounded by the same timeout.
• commit_staged_runtime surfaces rollback failure (machine left runtime-less → loud distinct error) and warns on version-stamp write failure (else repeated reinstalls).
• install_update single-flight guarded across all UI surfaces (INSTALLING + drop guard), not just the tray-menu path.
• Poison-safe state.lock() in the refresh paths + the tray-enter emit (warn-and-skip / default payload instead of panic); top_title scans all titles rather than only the first.

🧩 Setup wizard (commands/setup.rs + ui/app/setup)

• cfg(target_os = "macos")-gate the CoreGraphics FFI in check/request_screen_recording (mirrors the IOKit pattern → non-mac builds compile).
• RAM now /1024³ so a 16 GiB Mac reads as 16, not 17.
• set_model_preference validates the id is a plausible HF repo (org/name, [A-Za-z0-9._-]) + trims. Deviation: format-check rather than an exact catalog allowlist — the supported-model list lives in the wizard UI and duplicating it in Rust would drift. Verified all 3 catalog ids pass.
• UI: listener-cleanup race guard (cancelled flag); stop the flow on set_model_preference / mark_setup_complete write failure instead of swallowing (prevents "ready/done" lying); rail can no longer jump past a required step (gated on each prior step's canNext); Mark tint works for var(--accent) via color-mix; accurate privacy copy (pixels/video never stored vs the old "frames never stored"; "stays local unless you connect tools").

🎨 Misc

• Sidebar npm-update banner now shows only when the DMG updater is genuinely unsupported (upd == null || state === 'unsupported', after the check settles) — not on a packaged DMG's uptodate/error. (Deviation: keyed on unsupported/null, not !== 'available'.)
• tooltip.css box-sizing: border-box reset (mirrors style.css) so the 300px window body + 16px padding don't overflow to 332px.
• dev-signing.sh cleanup trap for the temp key material + dropped -A (codesign-only key access).
• Python (server.py): prefetch idempotency now scoped to the requested model_id (a done prefetch for model A no longer blocks model B); extra={} structured log fields in the prefetch/log paths (+ run_task_linker_mlx.py).
• tray/package.json build:daemon uses --locked.

Skipped (with reasons)

Finding	Why skipped
Migrations 046/047 SQL CHECKs	sqlx-checksum crash-loop on already-applied DBs + "never modify a migration"; enforced writer-side instead
package-updater.sh fail-fast on missing .sig	The skip is documented-intentional (artifacts-off builds; the build already fails earlier if signing was expected) — forcing exit 1 would break legitimate non-updater releases
style.css stylelint value-keyword-case	No stylelint is configured anywhere in the repo; lowercasing proper font names (SFMono-Regular, Helvetica) would be a regression — the correct fix (if stylelint is ever added) is quoting, not lowercasing
capture_coverage recalibration	Deferred — needs live duration-tuning data (see Health section above)

🤖 Generated with Claude Code

[Anonymous]

Ticket changed by: Akarsh-Hegde

• status: open --> closed