LMCache Files

Big Contributions coming soon: - Hetero TP by @novahow - More CacheBlend support by @junl666

What's Changed

• [Core] Refactor the storage manager to support using CPU & GPU backends at the same time by @ApostaC in https://github.com/LMCache/LMCache/pull/1578
• Fix a hardcoded device assignment when init FlashInferSparseBackend by @xleoken in https://github.com/LMCache/LMCache/pull/1571
• Add sphinx-multiversion for tracing docs versions by @kobe0938 in https://github.com/LMCache/LMCache/pull/1599
• [Doc] Fold the navigation bar for KV Cache management by @kobe0938 in https://github.com/LMCache/LMCache/pull/1598
• Bump step-security/harden-runner from 2.12.2 to 2.13.1 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1597
• Bump pypa/gh-action-pypi-publish from 1.12.4 to 1.13.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1541
• Bump actions/setup-python from 5.6.0 to 6.0.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1540
• Bump actions/stale from 9.1.0 to 10.0.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1539
• Refactor lmcache_get_config to lmcache_get_or_create_config by @maobaolong in https://github.com/LMCache/LMCache/pull/1586
• [Bugfix] Fix gds loading bytes comparison error by @kebe7jun in https://github.com/LMCache/LMCache/pull/1605
• [feat] add continuous usage context by @royyhuang in https://github.com/LMCache/LMCache/pull/1612
• Docs update for newcomers. by @KuntaiDu in https://github.com/LMCache/LMCache/pull/1595
• [Refactor] Cleanup PD code and separate transfer code by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1579
• Fix root doc endpoint refer to latest release snapshot(0.3.6) instead of latest commit issue by @kobe0938 in https://github.com/LMCache/LMCache/pull/1625
• Add .github/workflows/build_doc.yml to auto build by @kobe0938 in https://github.com/LMCache/LMCache/pull/1626
• [Core] Implement nixl storage backend batched_get_non_blocking by @tshmilnvidia in https://github.com/LMCache/LMCache/pull/1559
• Fix two bugs in async loading with MLA and TP > 1 by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1619
• [Misc] Hotfix: missing dependency: cpuinfo by @ApostaC in https://github.com/LMCache/LMCache/pull/1627
• Update vLLM options for MLA test by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1588
• [Doc] Fix blending related docs by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1624
• support tmp path in FSConnector by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1603
• [Test] Add a test for fs_connector by @maobaolong in https://github.com/LMCache/LMCache/pull/1594
• [Chore]: Remove Python 3.9 as the minimum Python version by @hickeyma in https://github.com/LMCache/LMCache/pull/1618
• [Core] Fix NixlStorageBackend import by @tshmilnvidia in https://github.com/LMCache/LMCache/pull/1631
• [Bugfix] Fix formatting and testing issues in latest branch by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1634
• [feat]: working async s3 by @sammshen in https://github.com/LMCache/LMCache/pull/1614
• [Docs] Fix the issue that left sidebar is unable to unfold by @KuntaiDu in https://github.com/LMCache/LMCache/pull/1637
• Improve installation section to make it less confusing by @kobe0938 in https://github.com/LMCache/LMCache/pull/1641
• [CD]: Lightweight Dockerfile by @sammshen in https://github.com/LMCache/LMCache/pull/1515
• Set Qwen/Qwen3-8B as default model; Remove all hf tokens; Remove config file in minimal example by @kobe0938 in https://github.com/LMCache/LMCache/pull/1643
• Replace default python package manager with uv to ensure fast & smooth experience + add vllm in minimal example to streamline user experience by @kobe0938 in https://github.com/LMCache/LMCache/pull/1642
• [Bugfix] Fix unit from s to ms in disagg_proxy_server.py by @aztecher in https://github.com/LMCache/LMCache/pull/1652
• [Improve] Add timeout for lookup client by @maobaolong in https://github.com/LMCache/LMCache/pull/1615
• [Doc] Fix broken link to mooncake store by @aztecher in https://github.com/LMCache/LMCache/pull/1651
• Add MRU policy to caching_policies doc by @xleoken in https://github.com/LMCache/LMCache/pull/1607
• Refactoring the cache policy factory by @xleoken in https://github.com/LMCache/LMCache/pull/1602
• [Core] P2P Backend by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1610
• [ROCM] [BUGFIX] : add missing hip files by @TaeSooRhee-moreh in https://github.com/LMCache/LMCache/pull/1654
• Fixed the socket resource leakage issue of LMCacheLookupClient by @xleoken in https://github.com/LMCache/LMCache/pull/1613
• [Improve] Support pass through the lmcache config from vllm extra config by @maobaolong in https://github.com/LMCache/LMCache/pull/1568
• [feat][internal_api] Add inference info internal api by @maobaolong in https://github.com/LMCache/LMCache/pull/1653
• [Test]: Enable unit tests to run on non-CUDA environment by @hickeyma in https://github.com/LMCache/LMCache/pull/1575
• [Backend][BugFix]: Refactor StorageBackendInterface constructor to enable dynamic loading by @hickeyma in https://github.com/LMCache/LMCache/pull/1636
• Improve the log message clarity when retrieve tokens by @xleoken in https://github.com/LMCache/LMCache/pull/1623
• [CI/CD] Fix GDS test failure by adding use_direct_io = True by @ApostaC in https://github.com/LMCache/LMCache/pull/1659
• Remove redundant connection check in remote_backend by @xleoken in https://github.com/LMCache/LMCache/pull/1667
• [bugfix] fix broadcast and to_gpu error by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1675
• [Improve][LocalCpu]Remove stream from local_cpu_backend.py by @maobaolong in https://github.com/LMCache/LMCache/pull/1673
• Get a default value if old vllm doesn't have priority by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1678
• [BugFix][Testing]: Enable unit tests for GDS backend by @hickeyma in https://github.com/LMCache/LMCache/pull/1648
• Initialize stack_frames to None to prevent NameError. by @xleoken in https://github.com/LMCache/LMCache/pull/1649
• Remove redundancy. Also old version of vllm do not support size local. by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1669
• [patch]: add eagle decode layer by @sammshen in https://github.com/LMCache/LMCache/pull/1668
• [patch]: async stress on cpu by @sammshen in https://github.com/LMCache/LMCache/pull/1665
• Add metrics remote_put_task_num by @maobaolong in https://github.com/LMCache/LMCache/pull/1620
• [core] support set limit of cache hit by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1666
• [bugfix] fix memory leak in fs connector by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1656
• [CD]: add flash infer by @sammshen in https://github.com/LMCache/LMCache/pull/1699
• [Docs] Layerwise Docs by @sammshen in https://github.com/LMCache/LMCache/pull/1697
• [Benchmark] configure user stacking at high qps in multi-round-qa by @yuezhu1 in https://github.com/LMCache/LMCache/pull/1609

New Contributors

• @royyhuang made their first contribution in https://github.com/LMCache/LMCache/pull/1612
• @TaeSooRhee-moreh made their first contribution in https://github.com/LMCache/LMCache/pull/1654
• @yuezhu1 made their first contribution in https://github.com/LMCache/LMCache/pull/1609

Full Changelog: https://github.com/LMCache/LMCache/compare/v0.3.6...v0.3.7