LLMSlim v0.7.1 — Cache-Aware Context Runtime

LLMSlim now plans for context reuse across agent turns while leaving model inference and actual prompt/KV storage with the provider or host.

Highlights

• Opt-in cache policy and tenant-scoped metadata track stable prefixes, append-only history, invalidation reasons, and estimated reuse.
• Cache-aware planning considers the cost of rewriting a reusable prefix while preserving existing budget, quality, and authorization checks.
• Execution-free request builders cover OpenAI Responses, Anthropic Messages, Gemini cached resources, and vLLM. A local Transformers session validates DynamicCache and StaticCache continuation.
• The Context Inspector shows cache diagnostics without calling a provider. Its server installs the hash-pinned v0.7.1 wheel. The v0.7.0 wheel remains available in the repository.

Verification

The offline suite passed with 90.28% coverage; Ruff, MyPy, the website build, clean wheel/sdist installs, and a local browser flow passed. The multi-turn benchmark reports structural reuse and serialized request bytes. A tiny CPU model showed a 35.7% lower median time for a cached one-token continuation after prefill; this is not a provider or production-model latency claim.

No live provider cache hit rate, TTFT, billed saving, or paid API test is claimed. Sarvam prompt-cache support remains unverified.

See caching guidance, the benchmark, and the engineering report.