Menu

#400 feat(mlx): retry MLX runtime download with exponential backoff

closed
nobody
None
2026-07-07
2026-07-07
Anonymous
No

Originally created by: Akarsh-Hegde

Summary

  • Classify each download_and_stage failure as retryable or permanent using a small DownloadError enum.
  • Wrap the download in a 5-attempt loop with 3s → 12s → 30s → 75s backoff (~2m total worst case (4 waits between 5 attempts)), inside the existing RUNTIME_DOWNLOAD_TIMEOUT so the overall budget stays bounded.
  • Between attempts the wizard sees "attempt N/5 — retrying in Ns…" via the existing mlx-download-progress event.
  • Background auto_upgrade_runtime uses the same retry helper.

Retryable vs permanent

Failure Verdict
Network error, HTTP 5xx / 429 retryable
Disk write error, stream error retryable
Checksum mismatch (truncated stream) retryable
Wrong arch, unsupported macOS permanent
HTTP 4xx (except 429) permanent
tar missing / extract failed permanent

Why

The wizard's runtime download failed on the first network hiccup — one dropped connection, one 502, one truncated stream and the user had to hit Retry. The background auto-upgrade did the same but silently gave up. Neither is honest about how flaky first-attempt downloads actually are.

Test plan

  • [x] cargo check --workspace + cargo clippy --workspace -- -D warnings clean.
  • [x] cargo test --lib on tray/src-tauri/ — 42 tests pass including two new ones (http_retryable_classification, retry_backoff_covers_all_gaps).
  • [ ] Manual: block egress to the runtime URL, run the wizard's download — user sees the "attempt N/5 retrying in Ns…" message and the final "(after 5 attempts)" error.
  • [ ] Manual: unblock mid-retry — download succeeds and wizard proceeds.
  • [ ] Manual: intentionally serve a corrupted tarball once, then a good one — first attempt fails on checksum, second succeeds.

🤖 Generated with Claude Code

Related

Tickets: #416

Discussion

  • Anonymous

    Anonymous - 2026-07-07

    Originally posted by: coderabbitai[bot]

    [!IMPORTANT]

    Review skipped

    Auto reviews are disabled on base/target branches other than the default branch.

    Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.


    ⚙️ Run configuration

    Configuration used: Organization UI

    Review profile: ASSERTIVE

    Plan: Pro Plus

    Run ID: f5c31d2c-584a-4fff-aa5c-df38953bec5c

    You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

    Use the checkbox below for a quick retry:
    - [ ] 🔍 Trigger review

    ✨ Finishing Touches
    🧪 Generate unit tests (beta) - [ ] Create PR with unit tests - [ ] Commit unit tests in branch `feat/runtime-download-retry`

    Comment @coderabbitai help to get the list of available commands.

     
  • Anonymous

    Anonymous - 2026-07-07

    Originally posted by: adityaharishch

    Reviewed the diff (tray/src-tauri/src/mlx_server.rs, +199/-39). The retry/backoff structure, retryable-vs-permanent classification, and graceful degradation (staging dir wiped, overall RUNTIME_DOWNLOAD_TIMEOUT still wraps the whole retry loop) are sound overall. A few concrete issues to fix before merging:

    Correctness

    • mlx_server.rs:559-565 (DOWNLOAD_RETRY_BACKOFF) — the array has 5 entries (2s, 8s, 20s, 45s, 90s) but only 4 gaps exist between 5 attempts (attempt == DOWNLOAD_MAX_ATTEMPTS short-circuits before the 5th sleep). The loop only ever indexes backoff[0..=3], so the 90s entry is dead code and the real worst-case cumulative wait is 75s (2+8+20+45), not the "~2m45s" claimed in the PR description and in the doc comment above the const ("attempts 1→2, 2→3, 3→4, 4→5" is 4 transitions but 5 values are listed). Either trim the array to 4 entries or bump DOWNLOAD_MAX_ATTEMPTS to 6 so the 90s value is actually used — whichever matches the intended budget.
    • mlx_server.rs:1266 (retry_backoff_covers_all_gaps test) — asserts DOWNLOAD_RETRY_BACKOFF.len() >= DOWNLOAD_MAX_ATTEMPTS - 1, which is too loose to catch the off-by-one above (5 >= 4 passes even though the 5th entry is unreachable). Should assert exact equality (== DOWNLOAD_MAX_ATTEMPTS - 1) so a future attempt/backoff mismatch fails the test.

    Design / robustness (non-blocking, worth a note)

    • No jitter on the fixed backoff schedule (mlx_server.rs:559-565). auto_upgrade_runtime runs on many machines against the same manifest/CDN host; a shared outage (5xx blip) means every client retries on the exact same 2s/8s/20s/45s cadence — a small thundering-herd risk. Not critical at current scale, but worth a + rand jitter if this runs fleet-wide.
    • Checksum mismatch is always classified transient (mlx_server.rs:508-515), on the assumption it's a truncated stream. If the server-side artifact itself is corrupted (not just a network truncation), this burns up to 5 full multi-hundred-MB re-downloads before giving up. Probably fine given the stated reasoning, but flagging since it's the one "retryable" case that isn't obviously network-transient.

    Style

    • New/changed fns keep returning Result<_, String> rather than anyhow::Result + .context(...), which conflicts with CLAUDE.md's Rust conventions — but this matches the file's pre-existing pattern throughout (not introduced by this PR), so not a blocker, just noting for awareness if this file is ever migrated to anyhow.

    No file-header, unwrap(), tracing-interpolation, or clippy 7-arg violations found — those all look clean.

    Overall: solid feature, but please fix the backoff/attempts mismatch (and tighten the test) before merging — right now the shipped behavior doesn't match what the PR description promises.

     
  • Anonymous

    Anonymous - 2026-07-07

    Originally posted by: Akarsh-Hegde

    Thanks — the backoff/attempts off-by-one is fixed in 18d9363d.

    Correctness (fixed)

    • Trimmed DOWNLOAD_RETRY_BACKOFF to exactly 4 entries (3s → 12s → 30s → 75s, ~2m total) so there's one wait per inter-attempt gap and no unreachable trailing value. You were right — the old 90s entry was dead and the true worst-case was 75s. Doc comment rewritten to the real schedule.
    • Tightened retry_backoff_covers_all_gaps from >= to assert_eq!(DOWNLOAD_RETRY_BACKOFF.len(), DOWNLOAD_MAX_ATTEMPTS - 1), so a future attempt/backoff drift (or a re-introduced trailing entry) fails the test instead of passing.

    Non-blocking, acknowledged:

    • Jitter: fair point for the fleet-wide auto_upgrade_runtime path (every machine on a shared ~6h cadence hitting the same CDN). I left it out of this fix to avoid pulling a randomness dep into a targeted correctness change; tracking it as a follow-up (cheap to add jitter derived from the machine's uid/hostname hash, no rand needed).
    • Checksum always transient: agreed it's the one non-network-obvious retryable case. Kept as-is per the truncated-stream reasoning, but noted — if we ever see a genuinely corrupt server-side artifact in the wild, capping checksum retries at 1–2 (vs the full 5) is the right escalation.
    • Result<_, String> vs anyhow: matches the file's existing pattern; leaving consistent rather than half-migrating one function.

    Re the PR description's "~2m45s" — now corrected to ~2m in the code; I'll fix the PR body too.

     
  • Anonymous

    Anonymous - 2026-07-07

    Ticket changed by: Akarsh-Hegde

    • status: open --> closed
     

Log in to post a comment.