#350 fix(prefetch): disable Xet + retry-with-resume to stop 4h download hangs

[Anonymous]

Originally created by: Akarsh-Hegde

Problem

A first-run model prefetch wedged at 57% for 4+ hours on a healthy 16 Mbps link (the bar froze, bytes never resumed).

Root cause: hf_xet does its transfer in Rust with no per-read timeout and can't be interrupted from Python. When a connection to the Xet CAS endpoint goes half-open (stops sending bytes but never errors — e.g. a firewall/proxy/regional block on *.xethub.hf.co), snapshot_download blocks the download thread forever: prefetch_state stays downloading and the wizard bar sits at the last byte count indefinitely.

Fix (services/agents/routes/prefetch.py, _download_spec)

• Disable Xet for the prefetch by flipping huggingface_hub's module constant (HF_HUB_DISABLE_XET) — not the env var, which is parsed once at import. This forces the classic LFS path, which caps every read at HF_HUB_DOWNLOAD_TIMEOUT (10s) and resumes from the .incomplete partial via a Range request, so a stall raises and self-heals instead of hanging.
• Bounded retry-with-resume — 5 attempts, 2/4/8/16s backoff (MERIDIAN_PREFETCH_MAX_ATTEMPTS overridable). Each retry resumes from disk, so transient drops converge without re-pulling.

Why dropping Xet is fine here

A/B on the same model showed Xet-on == Xet-off == network ceiling (no benefit on a cold first-run pull of distinct weights — chunk-dedup buys little), while parallel range requests did not beat a single stream (link-bound, not per-connection-throttled). So this trades an unused fast-path for a bounded, resumable one.

Bonus: the classic path grows the on-disk blob linearly, so /prefetch_status's byte-delta speed reads smoothly instead of sitting at 0 then bursting as hf_xet flushed reconstructed chunks (fixes the frozen-looking bar).

Verification

Full 3-model prefetch (~2.46 GB) completed in ~11 min at the link ceiling with smooth progress and no hang. A mid-download stall (dropped to ~1 MB/s for ~90s) recovered on its own instead of wedging — exactly the timeout+resume behavior intended.

Notes

• Python-only change; no Rust touched. Pre-push suite (fmt/clippy/test/audit) passed.
• This commit was originally pushed onto the already-merged fix/prefetch-speed-state-lint (PR [#348], closed) — re-homed here on a fresh branch off pre-main so it can actually land.

🤖 Generated with Claude Code

[Anonymous]

Originally posted by: coderabbitai[bot]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 44f59871-4fd7-4bc1-9add-ae1cf994b5fa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:
- [ ] 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

- [ ] Create PR with unit tests - [ ] Commit unit tests in branch `fix/prefetch-xet-hang`

---

_{Comment @coderabbitai help to get the list of available commands.}

[Anonymous]

Originally posted by: Akarsh-Hegde

Fixes pushed (4f4b944c)

Resolved both findings from the review above:

✅ 1. Startup-crash on bad env value — fixed

Replaced the bare int(os.environ.get(...)) with a defensive helper:

def _env_positive_int(name: str, default: int) -> int:
    raw = os.environ.get(name)
    if raw is None:
        return default
    try:
        return max(1, int(raw))
    except ValueError:
        log.warning("server: ignoring non-integer env override; using default",
                    extra={"env_var": name, "raw": raw, "default": default})
        return default

_PREFETCH_MAX_ATTEMPTS = _env_positive_int("MERIDIAN_PREFETCH_MAX_ATTEMPTS", 5)

Verified: MERIDIAN_PREFETCH_MAX_ATTEMPTS=oops now logs a warning and falls back to 5 instead of raising at import; a valid =3 is still respected.

✅ 2. assert-guarded raise — fixed

Dropped last_exc and the assert entirely; the loop re-raises the active exception with a bare raise on the final attempt:

    for attempt in range(1, _PREFETCH_MAX_ATTEMPTS + 1):
        try:
            snapshot_download(spec.model_id, allow_patterns=spec.allow_patterns)
            return
        except Exception as exc:  # noqa: BLE001
            log.warning(..., extra={"attempt": attempt, ...})
            if attempt >= _PREFETCH_MAX_ATTEMPTS:
                raise  # exhausted — surface to _run_prefetch
            time.sleep(min(2**attempt, 30))

Now -O-safe (no assert) and the exhaustion path is unconditional.

Not changed (intentional, per review notes [#3] & [#4])

• Broad except Exception retry — kept for resilience to backend-specific stall exceptions; backoff caps waste at ~30s.
• Process-global Xet disable not restored — deliberate so later lazy loads inherit the bounded path.

No behavioural change to the happy path or the hang fix. Pre-commit/pre-push checks pass.

[Anonymous]

Ticket changed by: Akarsh-Hegde

• status: open --> closed