#325 feat(etl): noise filter — clean session_text at ingest time

[Anonymous]

Originally created by: adityaharishch

Summary

• Adds src/etl/text_filter.rs — landmark detection, log-noise classification, alpha-ratio quality gate, and chrome pre-pass (lines in ≥4 frames that are not landmarks are persistent UI chrome)
• Wires the filter into build_session_text() in text_merge.rs so all noise is removed before writing to meridian.db — the classifier always receives clean text, no separate cleaning step needed
• Updates integration tests in tests/etl_session_text.rs to use ≥15-char realistic content strings (required by the new MIN_LINE_LEN gate)

Design

Landmark bypass — URLs, shell prompts ($, %, ❯), error/warning keywords, code signatures (fn/def/class/impl), SQL keywords, git branch refs, issue refs (#123), and commit hashes (7–40 hex) always survive, bypassing all quality checks. Safe failure mode: false positives in landmark detection keep noise (acceptable); false negatives would drop signal (unacceptable).

Chrome pre-pass — reuses a single HashSet<u64> (hashes) across all frames instead of allocating per frame; the frequency HashMap is freed after build_session_text returns (~400KB peak for 1000-frame blocks).

Zero-allocation hot path — ascii_icontains() replaces to_ascii_lowercase() throughout is_landmark and is_log_noise; alphabetic_ratio has an is_ascii() fast path (~3-4× faster for ASCII OCR content).

Rust compact tracing — INFO module::path: format (no timestamp prefix, from cargo watch / direct daemon output) is now filtered alongside the timestamped 2026-… INFO form; the compact check is outside the len > 20 guard.

Verification

Checked on 30+ live sessions post-deploy:

• 79–90% raw→stored char reduction (screenpipe frame totals vs app_sessions.session_text)
• Zero Rust compact tracing, JSON log blobs, uvicorn INFO: lines, or MLX memory lines in stored text
• Meaningful content preserved: shell commands, code diffs, cursor lines, prose conversations

Test plan

• [ ] cargo test passes (all ETL unit + integration tests)
• [ ] cargo clippy -- -D warnings clean
• [ ] Check a new Terminal session in app_sessions.session_text — Rust log lines absent, shell commands present
• [ ] Check a new Code session — code diffs and error messages present

[Anonymous]

Originally posted by: Akarsh-Hegde

Review follow-up — status of the 4 comments (checked against dd6f9a2)

Three of four are genuinely fixed; the fourth's fix is botched and is currently breaking the build.

#	Finding	Status
1	Ticket keys / short filenames dropped by MIN_LINE_LEN	✅ Fixed — contains_ticket_key + contains_code_filename in is_landmark (verified)
3	Chrome exclusion too broad (#/>, unanchored SQL)	✅ Fixed — tighter is_chrome_exempt now used by build_chrome_set (verified)
2	Chrome leak via additive merge	✅ Now fixed — close_block re-fetches the full session (active.min_frame_id..max_frame_id) and rebuilds session_text from scratch; open block stays additive (None), close rebuilds (Some). Both call sites updated, scoping correct.
4	text_filter.rs over the 500-line cap	⚠️ Broken — split into text_filter/mod.rs + tests.rs, but the old src/etl/text_filter.rs (892 lines) was never deleted

❌ Blocker: Rust CI is failing

The incomplete split in [#4] leaves two definitions of the same module, so cargo fmt --check (the first Rust step) dies before anything else compiles:

Error: failed to resolve mod `text_filter`: file for module found at both
  src/etl/text_filter.rs  and  src/etl/text_filter/mod.rs

Because the crate no longer builds, the correct fixes for #1/#2/#3 are also unshippable until this is resolved.

Fix: remove the stale file —

git rm src/etl/text_filter.rs
git commit -m "fix(etl): drop stale text_filter.rs after module split"
git push

Minor (non-blocking)

• The characterization test merge_path_chrome_leaks_when_first_batch_below_threshold still asserts the old leaky behaviour. It passes because it calls merge_session_texts directly, but the close path no longer takes it — worth re-pointing the test at the rebuild-on-close.
• text_filter/mod.rs is exactly 500 lines; the rule is "under 500," so it's technically still 1 over.

Bottom line: not ready to merge. Needs the git rm above to get CI green, then an approving review (reviewDecision: REVIEW_REQUIRED). Once CI is green the code-level changes for all four comments look correct.

[Anonymous]

Originally posted by: adityaharishch

Fixed in 4717e9a:

• Stale text_filter.rs: git rm'd — the split is now clean (text_filter/mod.rs + text_filter/tests.rs, no duplicate module definition, build passes).
• mod.rs at 500 lines: consolidated the 4-line comment above build_chrome_set's filter to 2 lines — file is now 498 lines.
• Stale chrome-leak characterisation test: removed merge_path_chrome_leaks_when_first_batch_below_threshold from text_merge.rs — it was asserting the old leaky behaviour which no longer exists (close path now rebuilds from scratch). CI should be green.

[Anonymous]

Ticket changed by: adityaharishch

• status: open --> closed

[Anonymous]

Originally posted by: adityaharishch

🎉 This PR is included in version 1.63.0 🎉

The release is available on:

• GitHub release
• v1.63.0
• npm package (@latest dist-tag)

Your semantic-release bot 📦🚀