#76 Release v0.11.1: wrap fixes found by a billed-cost benchmark

[Anonymous]

Originally created by: Thanatos9404

Summary

We measured what llmslim wrap claude does to a real Claude Code bill. The benchmark counts billed dollars, with cache reads and writes priced separately. It used Claude Code 2.1.296 on Claude Haiku 5.5, 8 tasks, and 572 runs with each run's result committed (benchmarks/claude_code_cost/). It found three problems in 0.11.0, all fixed here.

• Tool search: with a custom base URL, Claude Code turns tool search off and sends every tool definition, about 7,100 more tokens per request. wrap now sets ENABLE_TOOL_SEARCH=true unless the user set it. Same prompt: 26,885 tokens through the 0.11.0 proxy, 19,716 with the fix, 19,712 without the proxy.
• Permission mode: through a proxy, claude -p starts in auto mode instead of default, so commands outside --allowedTools went to a classifier. wrap now starts print-mode runs with --permission-mode default unless the user passes a mode or sets permissions.defaultMode. Interactive sessions are unchanged.
• JSON: shortened JSON made one task cost about twice as much (12 turns instead of 4). JSON is now left alone by default in the proxy and in LLMSlimToolOutputMiddleware, and --json / compress_json=True opts back in.
• Docs and site: README, docs/PROXY.md, the LangChain guide, the CHANGELOG and the site's Claude Code page now report the measured cost, wins and losses. That replaces the token-only claims. They also document the Claude Code limitation that a custom base URL skips an organization's server-managed settings.
• Packaging: version 0.11.1 in pyproject, __init__, CITATION, server.json and the plugin manifest.

Measured (full protocols and results in benchmarks/claude_code_cost/README.md)

	Change in billed cost with llmslim
Failing CI run, about 12 KB of log	-26% to -32%
Lint run, 3 errors among about 260 warnings	-17% to -21%
Test runs	-2% to -15% (agents already trim test output)
Pooled over 8 tasks, 0.11.0 settings	-5% to +5%

A replay over 444 real Claude Code sessions (48,911 requests) puts the ceiling for any tool-output compressor at 13.8% of the bill.

Release candidate check (measurement 4, written before the runs)

The wheel was built from this branch and run with the permission mode not pinned. All four conditions held:

• 30/30 llmslim runs in default permission mode;
• JSON regression gone: +0.9%, not significant;
• CI log -29.7%;
• equal pass rates.

Tests

• New: test_print_mode_runs_keep_the_default_permission_mode, test_wrap_keeps_the_users_tool_search_choice, test_json_is_left_alone_by_default, test_text_that_only_starts_like_json_is_still_shortened, and middleware test_json_results_are_left_alone_unless_asked.
• The full suite passes locally, and ruff check and format are clean. The LangChain middleware tests pass in a LangChain 1.4.6 environment.

[Anonymous]

Originally posted by: vercel[bot]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
llmslim	Ready	Preview	<relative-time datetime="2026-10-17T16:03:57.201Z">Oct 17, 2026 4:03pm UTC</relative-time>

[Anonymous]

Ticket changed by: Thanatos9404

• status: open --> closed