#2 feat: add Compare Mode — blind A/B validation layer for Mneme profiles

[Anonymous]

Originally created by: TheoV823

Summary

• comparison_results table — new schema + migration (migrate_add_comparison_results.sql), with cross-column CHECK constraint enforcing preferred_mode IS NULL on ties/skips
• app/models/comparison.py — insert_comparison, get_comparisons_for_user, compute_win_rate (win rate = mneme wins / decisive comparisons, ties/skips excluded from denominator)
• app/runner/compare.py — run_comparison: calls Claude API twice (default + Mneme system prompts), randomizes which output is Option A/B for blind comparison
• flask compare — CLI command: runs both modes, displays blind A/B, captures a/b/tie/skip preference, persists result
• flask compare-stats — CLI command: shows cumulative win rate for a user
• 21 new tests (8 model, 4 runner, 9 CLI) — 99 total passing

Architecture

flask compare --user-id <id> --prompt "..."
      │
      ├─ call_claude(default system prompt)  ─┐
      ├─ call_claude(mneme system prompt)    ─┤
      │                                       │ randomized A/B
      └─ display blind comparison ────────────┘
             │
             user picks A/B/tie/skip
             │
             insert into comparison_results

Test Plan

• [x] 99/99 tests passing
• [x] python -m pytest tests/ -q to verify
• [x] Schema cross-column constraint verified: tie/skip → preferred_mode NULL; a/b → preferred_mode NOT NULL
• [x] A/B randomization covered by mocked random.choice tests
• [x] CLI reprompt loop covered (invalid input → reprompts until valid)

Migration

For any database created before this feature:

sqlite3 mneme.db < migrate_add_comparison_results.sql

Note

This branch is based on feature/signal-profile-refactor (PR [#1]), not main. It should be merged after PR [#1] lands.

---

🤖 Generated with Claude Code

[Anonymous]

Ticket changed by: TheoV823

• status: open --> closed