LMCache Files

The torch version is bumped to 2.8.0 in sync with vllm v0.10.2: https://github.com/vllm-project/vllm/releases/tag/v0.10.2

What's Changed

• [Perf] Series of optimizations in vllm adapter by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1463
• [patch]: s3 mem leak by @sammshen in https://github.com/LMCache/LMCache/pull/1461
• [Misc] Replace deprecated pkg_resources with importlib.metadata by @yankay in https://github.com/LMCache/LMCache/pull/1421
• 🔥Fix: the gds filename convert back failure by @panpan0000 in https://github.com/LMCache/LMCache/pull/1328
• [PD][Bugfix] Fix the communication between prefiller and proxy in tensor parallel case by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1472
• [Core] Sglang Layerwise Integration by @Oasis-Git in https://github.com/LMCache/LMCache/pull/1431
• [Docs]: Update central configuration documentation by @sammshen in https://github.com/LMCache/LMCache/pull/1458
• [bugfix] Fix crash caused by raised runtime error due to inconsistent number of hit tokens across tp ranks by @Hadigan in https://github.com/LMCache/LMCache/pull/1426
• Fix device ordinal overflow bug in retrieve broadcast for TP > 8 by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1480
• [InternalApiServer] Support get version and commit_id from internal_api_server by @maobaolong in https://github.com/LMCache/LMCache/pull/1475
• use enum auto instead of number literal by @mengbingrock in https://github.com/LMCache/LMCache/pull/1390
• [CI/Build][Bugfix] Fix pre-exit in clean script by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1469
• [Bugfix] Fix mem leak in S3 connector by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1495
• [Refactor]: Duplicate code in config handling by @hickeyma in https://github.com/LMCache/LMCache/pull/1492
• [Docs]: Initial Versioining Matrix by @sammshen in https://github.com/LMCache/LMCache/pull/1489
• [Misc][Core] add extra config force_store_wait to make sure store does not skip any requests by @ApostaC in https://github.com/LMCache/LMCache/pull/1499
• [Bug][PD] Avoid doing int16 conversion on hash by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1446
• [Bugfix][PD] Fix nixl initialization problem by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1494
• [bugfix] Fix missing parent_allocator while init TensorMemoryObj for save_only_first_rank by @maobaolong in https://github.com/LMCache/LMCache/pull/1493
• [CI/Build] Add /opt/venv/bin to PATH in lmcache/vllm-openai Build and Release images by @Zerohertz in https://github.com/LMCache/LMCache/pull/1498
• [Bugfix] layerwise store: assert token_ids is list[int], not tensor by @mengbingrock in https://github.com/LMCache/LMCache/pull/1517
• [Bugfix] Fix lookup server registration bug [#1189] by @chickeyton in https://github.com/LMCache/LMCache/pull/1237
• [Feat]: Mock Connector by @sammshen in https://github.com/LMCache/LMCache/pull/1500
• [infinistore] Remove infinistore dependency by @maobaolong in https://github.com/LMCache/LMCache/pull/1524
• [CI/Build] Set timeout for unit tests to 30 minutes by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1526
• [InternalApiServer] downgrade access log level of internal api server and support configure by @maobaolong in https://github.com/LMCache/LMCache/pull/1505
• [Core] NIXL storage backend misc fixes by @tshmilnvidia in https://github.com/LMCache/LMCache/pull/1476
• [DOCS] Add document for plugin framework by @maobaolong in https://github.com/LMCache/LMCache/pull/1488
• Async KV loading by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1513
• [DOC][Bugfix] fix cuda device export in doc by @mengbingrock in https://github.com/LMCache/LMCache/pull/1508
• [Doc] updated cache controller API document by @ziruiliu in https://github.com/LMCache/LMCache/pull/1527
• Fix a format issue of plugin_framework.rst by @maobaolong in https://github.com/LMCache/LMCache/pull/1532
• Fix a small typo issue within the log of LocalCPUBackend by @maobaolong in https://github.com/LMCache/LMCache/pull/1531
• [Misc] add dummy decorator when nvtx is not available by @hijeffwu in https://github.com/LMCache/LMCache/pull/1502
• [CI] pytest benchmark for engine lookup store and retrieve operations by @ApostaC in https://github.com/LMCache/LMCache/pull/1484
• [Improvement][PluginFramework] Add default interpreter from plugin script by @maobaolong in https://github.com/LMCache/LMCache/pull/1504
• [Doc] Update Slack by @kobe0938 in https://github.com/LMCache/LMCache/pull/1535
• [Metrics] add vllm hit tokens metrics by @zerofishnoodles in https://github.com/LMCache/LMCache/pull/1538
• [Benchmark] Add completions and visualization to long doc qa by @sammshen in https://github.com/LMCache/LMCache/pull/1519
• [Misc] Add NIXL storage backend unit tests by @tshmilnvidia in https://github.com/LMCache/LMCache/pull/1490
• [Bugfix] Fix ImportError in unit tests by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1536
• [PD] Support /chat/completion endpoint in pd by @kobe0938 in https://github.com/LMCache/LMCache/pull/1511
• [CI]: Import for integration test after extra Long Doc QA pandas + matplotlib dependencies by @sammshen in https://github.com/LMCache/LMCache/pull/1556
• [Benchmark]: Benchmarking Recommendation + Documentation by @sammshen in https://github.com/LMCache/LMCache/pull/1534
• [Misc] Fix mooncake doc by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1554
• [MISC] Print version info while lmcache started by @maobaolong in https://github.com/LMCache/LMCache/pull/1548
• [bugfix] fix not hit in LocalCpuBackend by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1545
• [Improve] Reuse the config generated to reduce get config from file again by @maobaolong in https://github.com/LMCache/LMCache/pull/1566
• [Refactor] Small refactor to simplify code relevant to remote connector by @maobaolong in https://github.com/LMCache/LMCache/pull/1565
• [PD][CI/Build] Add comprehensive test interface for pd and minor fixes by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1496
• [optimize] reduce str concat by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1549
• [fix] fix to workaround for dp !=1 by @maobaolong in https://github.com/LMCache/LMCache/pull/1479
• [doc] add monitoring session by @panpan0000 in https://github.com/LMCache/LMCache/pull/1437
• [Core] Add unit tests for LMCacheEngineBuilder.destroy() method by @ikaadil in https://github.com/LMCache/LMCache/pull/1530
• [Diagg_Proxy] to support multi-host and tensor-parallel by @panpan0000 in https://github.com/LMCache/LMCache/pull/1543
• Add a comprehensive test for MLA models by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/1533
• Priority based storing -- only store kv cache for high priority requests by @FerdinandZhong in https://github.com/LMCache/LMCache/pull/1368
• [Core] Update mm_hashes to the mm_feature format for compatibility with vLLM by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1582
• [Feat]: Add Async Remote Load (Mock + Redis) by @sammshen in https://github.com/LMCache/LMCache/pull/1546
• [Misc] Add MRU cache policy to v1 by @yunjiangster in https://github.com/LMCache/LMCache/pull/1560
• fix mm_features with vllm v0.10.1 by @novahow in https://github.com/LMCache/LMCache/pull/1587
• [Bugfix] Fix blending bugs introduced by recent updates by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1585
• [feat] Add a request level skip_save parameter by @maobaolong in https://github.com/LMCache/LMCache/pull/1574
• Support override config by env by @maobaolong in https://github.com/LMCache/LMCache/pull/1572
• [Docker] Allow specifying vLLM version when building docker by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1581
• [Core] Add backward compatibility with vllm by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1583

New Contributors

• @Hadigan made their first contribution in https://github.com/LMCache/LMCache/pull/1426
• @Zerohertz made their first contribution in https://github.com/LMCache/LMCache/pull/1498
• @chickeyton made their first contribution in https://github.com/LMCache/LMCache/pull/1237
• @hijeffwu made their first contribution in https://github.com/LMCache/LMCache/pull/1502
• @ikaadil made their first contribution in https://github.com/LMCache/LMCache/pull/1530
• @FerdinandZhong made their first contribution in https://github.com/LMCache/LMCache/pull/1368
• @yunjiangster made their first contribution in https://github.com/LMCache/LMCache/pull/1560

Full Changelog: https://github.com/LMCache/LMCache/compare/v0.3.5...v0.3.6