#455 feat(llm-lab): dev-only multi-provider LLM comparison harness

[Anonymous]

Originally created by: Akarsh-Hegde

What

A dev-only multi-provider comparison harness ("LLM Lab"): replay any prose LLM stage from stored meridian.db inputs across N provider/model variants and compare the outcomes side by side - including rendering each model's simulated day on the real dashboard timeline.

Stacked on feat/llm-provider-connectivity-test; retarget to pre-main once that merges.

Processes covered

• Hour report - replays a distilled past hour; rendered through the pipeline's own parse/clamp/assemble.
• Day-task fold (one hour) - each variant's placements are applied to the prior state and the resulting day renders as the REAL DayTaskColumn timeline.
• Full-day fold - every stored hour report of a day, chained per variant, each model evolving its OWN day from empty; final timelines compare side by side.
• Worklog draft - the match/propose/update call for a day-task card.

Design

• A variant = provider[:model] (so two models of one provider compare exactly like two providers; params_json reserved for future dimensions). The runner calls resolver::backend_for() directly - one level below llm::complete() - so a variant's rate-limit/failure is recorded as that variant's outcome, never silently substituted by the on-device fallback. backend_for remains the only match on LlmProvider.
• Results live in new llm_experiments / llm_experiment_results tables (migration 061, etl_runs ledger pattern). Production tables are read-only inputs - the fold replays run the pure parse_placements -> apply_placements -> to_rows pipeline on in-memory state.
• Create/exec split: the tray gets the experiment id fast and spawns meridian llm-experiment exec detached; the UI polls per-variant progress. Killed runs are resumable.
• Behavior-identical extractions expose each stage's exact PromptRequest builder (compose_report_input, report_request, workstream_request, generate_request, hour_bounds), so a replay is byte-identical to the pipeline's own call.

Dev gate (this must not exist for users)

• UI: the toolbar Lab button + modal render only when get_app_info().channel === 'dev' (cfg!(debug_assertions)).
• Tray commands: refused at runtime in release builds (dev_only()), even against a hand-crafted invoke.
• The meridian llm-experiment CLI stays ungated deliberately (field debugging; only writes the experiment tables).

Known caveats (documented in module docs + UI copy)

• Hour-fold replays use the day's CURRENT prior task state (fold-time state isn't archived) - identical input for all variants, which is what comparison needs.
• Only distilled/reported hours replay; CLI backends report 0 tokens (UI shows "-").
• Also adds a MERIDIAN_BIN env override to the tray's CLI resolver so a tauri dev tray can spawn a workspace build without restaging ~/.meridian/bin (which would migrate the shared DB ahead of the installed daemon).

Testing

• 551 lib tests + integration tests, clippy -D warnings on both crates, UI build + 194 UI tests, 3 new seeded reader tests.
• Real E2E against a DB copy: hour report (claude 6.9s proper report vs local 2B unparseable prose - honestly surfaced), hour fold, worklog draft, and an 11-hour full-day fold (local, 80s) that visibly reproduced the 2B's task-fragmentation failure - exactly the quality difference the Lab exists to show.
• Verified in the running dev app (composer -> run -> live polling -> timeline columns; past runs reopen).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ctcf2YbLSiWTT5q47RZxNg

[Anonymous]

Originally posted by: coderabbitai[bot]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a738ddf1-469d-4ae1-9f35-8c202f62862b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:
- [ ] 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

- [ ] Create PR with unit tests - [ ] Commit unit tests in branch `feat/llm-lab`

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

_{Comment @coderabbitai help to get the list of available commands.}

[Anonymous]

Originally posted by: adityaharishch

Review — PR [#455] (LLM Lab)

Reviewed via gh pr diff 455 (33 files, +3282/-243). This is a large, self-contained dev-tooling addition, so I focused verification on the parts most likely to hide a real bug or a leak into production behavior: the dev-gating, the "no silent fallback" guarantee, the new DB ledger's resumability, and whether any existing component/route got a behavior change rather than a pure addition.

What I verified directly (not just from the description)

• Dev-gate is real and doubled. UI: the Lab button (Toolbar.tsx) and the modal render (MeridianTimelineShell.tsx) both gate on channel === 'dev' from get_app_info(). Independently, all three tray commands (run_llm_experiment, get_llm_experiments, get_llm_experiment in commands/llm_lab.rs) call dev_only() → cfg!(debug_assertions), so a hand-crafted invoke against a release build is refused regardless of what the UI shows. The llm-experiment CLI itself is deliberately left ungated (stated intent, and correct: it only ever writes the new llm_experiments* tables, nothing production).
• "No silent fallback" claim holds. runner.rs's variant_backend() calls resolver::backend_for() directly — one level below llm::complete() — so a variant's RateLimited/Failed becomes that variant's recorded outcome (rate_limited/failed in llm_experiment_results.status) instead of being masked by complete()'s on-device retry/fallback. Confirmed backend_for is still the only match site on LlmProvider (nothing new pattern-matches the enum elsewhere).
• Full-clear/FK-style correctness class of bug doesn't recur here, but similarly-shaped resumability logic does check out: store::load_pending returns rows with status IN ('pending','running'), and the a_running_row_is_resumable test proves a running row from a killed process gets picked up again on the next exec — no experiment can get stuck with a variant neither pending nor terminal.
• Migration 061 schema: both tables are CREATE TABLE IF NOT EXISTS (idempotent), llm_experiment_results PKs on (experiment_id, variant_idx) matching every write path's binding order, and the reader's tables_exist() probe means a pre-061 DB (the tray opens meridian.db without running migrations) degrades to empty/None rather than erroring — verified this is actually tested (llm_experiments_degrade_to_empty_on_a_pre_061_db), not just asserted in the module doc.
• generate_request/workstream_request/report_request extraction is behavior-preserving. src/pm_worklog/generate.rs's generate() now calls the extracted generate_request() and uses its result exactly where the inline code used to be — the token budget constant (GENERATE_MAX_TOKENS) just widened from private to pub(crate), nothing about the real worklog-generate path changed. Same pattern for the worklog_pipeline module visibility bump (mod → pub(crate) mod) — purely a visibility change so the Lab can reuse the request builders, not a behavior change.
• DayTaskColumn's new tasks? prop is additive and opt-in. When omitted (every existing caller), the component fetches/polls exactly as before — verified the useEffect still has its original fetch/poll body gated behind if (tasks) { ...; return }, so nothing about the real dashboard timeline changes for non-Lab callers. This is the one component the Lab reuses live (feeding a simulated day straight into it per the PR's design goal), and it's the right kind of reuse — no forked copy of the timeline for the Lab.
• Subprocess safety. Both run_meridian/spawn_meridian_detached (new shared cli_exec.rs, itself a good dedup of the copy-pasted pattern in statuses.rs/worklog_generate.rs) use tokio::process::Command with argv (args: &[&str] / &[String]), never a shell string — so body.process/body.variants/user-influenced strings reaching the subprocess can't cause command injection even though they're not otherwise sanitized.
• MERIDIAN_BIN env override (install.rs) isn't itself gated to dev builds, but it mirrors the exact precedence pattern meridian_db_path() already uses for MERIDIAN_DB (process env wins, logged at info!) — consistent with an existing, already-accepted convention in this file, not a new class of exposure.

Minor / non-blocking observations

• runner::run()'s day-fold branch (run_day_fold_variant) computes elapsed/in_tokens/out_tokens as running totals across every hour in the chain and only writes them once, at the end (success or the failure branch) — correct, but worth double-checking against the UI's fmtElapsed: for an 11-hour chain (the PR mentions an 80s real run) that's a sum across all hours, which is what "time to build day X under model Y" should mean — looks intentional, not an oversight.
• ResultsGrid.tsx's fold-payload parse (JSON.parse(result.output_rendered)) is wrapped in try/catch with a safe fallback to raw-text rendering if it's not the expected shape — good defensive parsing for something that's ultimately model-adjacent-but-Rust-produced JSON, not user input.
• The PR is currently showing CONFLICTING mergeable state against its base (feat/llm-provider-connectivity-test) per gh pr view — not something to review in the diff, just flagging that a rebase/merge will be needed before this can land, independent of code review.
• Nothing found that writes to pm_worklog_hours, day_tasks, day_task_worklogs, or pm_tasks from anywhere under llm_experiment/ — the "experiments never touch production tables" claim holds on inspection, not just by module-doc assertion.

Summary

No correctness issues found. This is a well-isolated, defense-in-depth-gated dev tool: the parts that touch real pipeline code are pure extractions (request builders, fold pipeline reuse) with no behavior change to the production path, the new tables are additive and degrade gracefully on old DBs, and the one live-UI-component reuse (DayTaskColumn) is opt-in via an optional prop. The three "why this can't leak to users" claims (UI gate, command gate, CLI-only-writes-new-tables) were each checked against the actual code rather than taken from the description, and all hold.

[Anonymous]

Ticket changed by: Akarsh-Hegde

• status: open --> closed