#545 feat(observability): centralised error observability for packaged installs

[Anonymous]

Originally created by: Akarsh-Hegde

What & why

Gives packaged Meridian installs a path to auto-ship redacted, error-only telemetry to a central, team-visible OpenObserve (via an OTel Collector gateway), so we can debug production without asking every user to hand-export a diagnostics bundle. Reuses the existing store-and-forward spool (pending → shipper → sent/quarantine, retention, retry) — the genuinely new parts are a central endpoint, distributable auth, and on-device redaction, not a policy flip.

Local capture and meridian logs stay full-fidelity throughout — redaction produces a separate stripped copy for egress only. The whole thing fails safe: with no endpoint/token/DSN baked in, all shipping is inert and only local capture runs.

Architecture — two shipping modes (src/observability/otlp_target.rs)

	Dev / Bare checkout	Canonical (packaged) install
Ships when	otlp_enabled + OO creds	error_reporting_enabled (opt-out, on by default)
Target	the engineer's own local OO	central gateway (release-baked endpoint)
Auth	Basic base64(email:pass)	Bearer <write-only token> (release-baked)
Redaction	none (full fidelity)	error-only + allowlist

The central endpoint + token are release-injected option_env! constants (Dayflow-style), inert in source builds so a dev build can never ship to prod by accident. ship_one now takes a scheme-aware Authorization header (Basic or Bearer).

The privacy boundary — src/telemetry_spool/redact.rs

Runs on the ship leg only, immediately before egress. Two fail-closed rules:

1. Error-only — WARN+ log records and ERROR-status spans; everything else dropped (this is also where the high-volume INFO/SPAN records carrying OCR / a11y / LLM I/O get dropped wholesale). A tight NOISE_SUBSTRINGS denylist additionally drops known-benign WARN repeaters that would flood the backend.
2. Allowlist by value-type + key — numeric/bool always kept (can't hold content); strings only if the key is on SAFE_STRING_KEYS; bytes/array/kvlist dropped. Free-text fields (*.message, stacktraces, log body, span status.message) additionally get URL/email/token scrubbing + a length clamp. Undecodable payloads are never shipped raw (quarantined).

Empirical finding that shaped this: probing the real spool showed 0 ERROR-status spans / 18,785; the error signal is ~10% of log records (WARN+), correlated by trace_id. So dashboards/gateway target error logs, not traces. The trace-redaction path stays (correct, cheap, future-proof) but ships near-empty today.

Consent — opt-out, on by default

Matches the reference apps (screenpipe, Dayflow): error_reporting_enabled defaults true. Surfaced as a disclosure note + off-switch in Settings → Advanced → Observability, plus a first-run note on the setup wizard's Completion screen ("Meridian sends error reports to help fix bugs (change in Settings)"). Deliberately does not claim the reports are anonymised — they are redacted, not anonymous (that's a later phase). The MERIDIAN_TELEMETRY_DISABLED hard kill still stops all capture.

Gateway ops stack (ops/central-observability/)

Deployable turnkey stack for the central instance, separate from the per-developer local OO (scripts/install-openobserve-daemon.sh):

• docker-compose.yml — OpenObserve + OTel Collector (contrib) + Caddy (auto-TLS).
• otel-collector-config.yaml — bearer-token auth on the OTLP receiver (validates the rotatable write-only ingest token), basic-auth export to OO (root creds live only here).
• Caddyfile + .env.example — TLS reverse proxy + config template. Ops guidance is in file comments (no standalone README, per repo convention).

Tray coverage (Phase 2)

• 2A — meridian::observability::init("meridian-tray") is now unconditional (was a dev-only otel feature), so the tray's logged errors reach the spool → central pipeline. The tray was otherwise dark in release, and it's the process the user actually clicks. No new deps (OTel machinery already linked via the always-present meridian dep).
• 2B — tray/src-tauri/src/crash.rs adds tauri-plugin-sentry 0.5 + re-exported sentry-rust-minidump for the crash class OTLP can't see (segfault, panic-before-init, OOM), with fingerprint grouping. Gated identically to central shipping (consent + kill-switch + baked DSN, gate at init not by muting). before_send reuses the same scrub_text as OTLP, and the sentry dep omits the tracing feature so logged errors aren't double-reported. DSN release-injected (MERIDIAN_SENTRY_DSN), inert without it.

Release injection (.github/workflows/release-build.yml)

Both build steps (macOS + Windows) bake in, from repo variables:

• MERIDIAN_CENTRAL_OTLP_ENDPOINT (daemon, option_env!)
• MERIDIAN_SENTRY_DSN (tray, option_env!)

The write token (MERIDIAN_CENTRAL_TELEMETRY_TOKEN, a secret) is intentionally not yet wired — added at activation time.

Verification

• cargo test — 38 telemetry tests incl. redaction (allowlist / error-only / free-text scrub / clamp / noise-drop / status.message).
• fmt + clippy clean across the daemon; the tray isn't in pre-push (only the meridian package is), so it was validated directly: cargo check + clippy + fmt -p meridian-tray under the real capture feature.
• Redaction allowlist reviewed against real spool content.

Deferred activation (infra-gated — separate follow-up)

1. Deploy ops/central-observability/ on a Linux VM (creds + INGEST_TOKEN + domains + DNS + docker compose up -d).
2. Repo variable MERIDIAN_CENTRAL_OTLP_ENDPOINT = https://<gateway>/v1/traces.
3. Repo secret MERIDIAN_CENTRAL_TELEMETRY_TOKEN (= INGEST_TOKEN) and wire it into the two build steps.
4. Create a Sentry project → repo variable MERIDIAN_SENTRY_DSN.
5. Cut a release; confirm redacted error logs in OO + a test crash in Sentry.

Known follow-ups

• Real anonymisation of shipped error text (v1 is redacted, not anonymous).
• Sentry release-health sessions (sentry::start_session(); 0.42 dropped the auto_session_tracking option).
• Optional server-side redaction second pass at the gateway (on-device allowlist is the sole net today, by design).

🤖 Generated with Claude Code

[Anonymous]

Originally posted by: coderabbitai[bot]

[!IMPORTANT]

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 52dc40a9-cb0a-40ff-b67f-bfb034820e74

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:
- [ ] 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

- [ ] Create PR with unit tests - [ ] Commit unit tests in branch `feat/central-error-observability`

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

_{Comment @coderabbitai help to get the list of available commands.}