Originally created by: Thanatos9404
Summary
We checked every published claim about llmslim wrap claude and the real-session replay: 381 claims across the README, CHANGELOG, docs/PROXY.md, the site's Claude Code page and changelog, and the benchmark README. Each was recomputed from the raw runs by two independent reviewers, one recomputing and one trying to refute. This PR fixes what they found.
Factual errors
- Lint task: exactly 240 warnings, not "about 260".
- JSON task: the agent's command returned the 74 open issues (the fixture holds 90).
- Real-session replay priced Claude Opus 5.5 cache reads at 0.1x input; Opus 5.5 reads at 0.05x. Re-run at each model's list rates on the same requests (
benchmarks/claude_code_cost/replay_bill.py, cut off at the original run time):
| Measure |
Was |
Now |
| Cache reads |
75.6% |
60.9% |
| Cache writes |
18.7% |
30.1% |
| Output |
5.6% |
9.0% |
| Compressible-tool ceiling |
13.8% |
13.4% |
| All-tool ceiling |
19.2% |
18.7% |
| Re-reading beyond 150k |
41% |
33% |
| 0.11.1's replayed saving |
4.3% |
4.1% |
- Sample size: "444 sessions" is 32 sessions plus the 412 subagent runs they started.
- API thinking check: the log was sent in six requests, not five, and six requests (not seven) had thinking in their history.
Framing fixed
- Primary result first: results lead with the primary measurement (both arms in default permission mode). Pooled is +5.3%, not the range "-5% to +5%"; test runs are about -4% and within noise.
- Untested claims narrowed:
- interactive sessions' permission mode through the proxy was never tested;
- "JSON is left alone" applies only when the whole output is one JSON document;
- 35% is a ceiling on what's kept, not what's kept (median 21%);
- the HTTP 400 is the default only for accounts created on or after 31 Aug 2026;
- the 7,100-token overhead is without MCP servers;
- files printed through the shell count as shell output.
The benchmark README keeps its pre-registered protocol text as written and adds a dated Corrections section. It also discloses that the runs inherited DISABLE_MICROCOMPACT=1 from the launching shell, in both arms. That's irrelevant at these context sizes, and a probe confirmed the runs used default effort.
Checks
ruff check and ruff format --check are clean.- Site data changes are strings only; the website build runs in CI.
Originally posted by: vercel[bot]
The latest updates on your projects. Learn more about Vercel for GitHub.
Ticket changed by: Thanatos9404