LMCache Files

v0.3.3 sees some performance improvements from the last release.

What's Changed

• [DOC] [ROCm]: Update LMCache installation procedure for ROCm by @vllmellm in https://github.com/LMCache/LMCache/pull/1037
• [CD]: Clean up unnecessary manylinux base file by @sammshen in https://github.com/LMCache/LMCache/pull/1043
• [Refactor]: Remove unnecessary build package dependency by @hickeyma in https://github.com/LMCache/LMCache/pull/1059
• [CI] Speedup by canceling previous commit run by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1067
• [Refactor] Open up interface for storing hashes by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1040
• [Bugfix] wrong key check in kv_controller.py by @wxsms in https://github.com/LMCache/LMCache/pull/1053
• [CI] Fix step dependency on unit test by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1069
• [Core] XPYD support by @YaoJiayi in https://github.com/LMCache/LMCache/pull/895
• Update modelconfig.json by @JasmondL in https://github.com/LMCache/LMCache/pull/1048
• [Bugfix] Make nixl import lazy by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1080
• [CI/Build] Optimization for multiple buildkite agents by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1074
• [bugfix] adapt old vllm version by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1072
• [Doc] fix config file name consistency in KV cache sharing example by @yankay in https://github.com/LMCache/LMCache/pull/1083
• [CI/Build][Doc] Remove deprecated pipelined_backend configuration option from testcase, documentation and examples by @yankay in https://github.com/LMCache/LMCache/pull/1085
• [CI/Build] Refactor integration test exit and cleanup by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1090
• [bug]: unpin unretrieved by @sammshen in https://github.com/LMCache/LMCache/pull/1092
• extra indentation from 1092 by @sammshen in https://github.com/LMCache/LMCache/pull/1093
• [Bugfix] Lookup compatibility with tp by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1101
• Bump ossf/scorecard-action from 2.4.1 to 2.4.2 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/995
• Bump step-security/harden-runner from 2.12.1 to 2.12.2 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/994
• [ci]: Refactor docs workflow by @hickeyma in https://github.com/LMCache/LMCache/pull/1098
• [security]: Tighten runner security by specifying endpoints by @hickeyma in https://github.com/LMCache/LMCache/pull/1097
• [fix]: Enable container access for score card and PyPI workflow by @hickeyma in https://github.com/LMCache/LMCache/pull/1103
• Introduce a remote monitor thread to monitor remote and support fallback to blackhole by @maobaolong in https://github.com/LMCache/LMCache/pull/764
• [Misc] Added unit tests for local cpu and disk backend by @weicaivi in https://github.com/LMCache/LMCache/pull/1041
• [Fix][P2P] Set retrieved token mask for P2P mode by @zejunchen-zejun in https://github.com/LMCache/LMCache/pull/1105
• [Example]: reversed conditional in 1p1d by @sammshen in https://github.com/LMCache/LMCache/pull/1076
• [Add] fix not implemented error in gds by @ApostaC in https://github.com/LMCache/LMCache/pull/1115
• [Bugfix] Metadata file path missing suffix in GdsBackend and WekaGdsB… by @YurianStormrage in https://github.com/LMCache/LMCache/pull/1026
• [Core] Make PD and offloading compatible by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1112
• [Doc] Updating community meeting time by @YuhanLiu11 in https://github.com/LMCache/LMCache/pull/1124
• [Doc] Fix documentation error by @YuhanLiu11 in https://github.com/LMCache/LMCache/pull/1125
• [bugfix] fix AttributeError in LMCacheConnectorMetadata by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1120
• Revert "[ci]: Refactor docs workflow" by @hickeyma in https://github.com/LMCache/LMCache/pull/1129
• [bugfix] adapt old vllm version by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1119
• Add structure for pluggable hash libraries by @hickeyma in https://github.com/LMCache/LMCache/pull/1089
• [Bugfix] Assertion Error for CPU offloading in PD by @vladnosiv in https://github.com/LMCache/LMCache/pull/1132
• [CI/Build] Check end to end results by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1126
• [Bugfix] Lazy init for kv caches pointer by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1135
• [Bugfix] Fix unit test by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1137
• [fix]: zeroing out separator token positions by @sammshen in https://github.com/LMCache/LMCache/pull/1108
• [Connector] Support config multiply base dir for fs remote connector by @maobaolong in https://github.com/LMCache/LMCache/pull/1058
• [Doc] Fix documentation error by @qaz-t in https://github.com/LMCache/LMCache/pull/1148
• [ADAPTER|vLLM] fix: Handle the prefix cache hit >= num_external_hit_tokens case avoid storing invoked by @maobaolong in https://github.com/LMCache/LMCache/pull/1096
• [CD]: nightly build patch by @sammshen in https://github.com/LMCache/LMCache/pull/1164
• [Bugfix] Fix torch mem alloc by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1163
• [Bugfix] Cross-process cache sharing by @junl666 in https://github.com/LMCache/LMCache/pull/1140
• [Refactor] refactor disk and prefetch code paths by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1172
• [Bugfix] Concurrent LMCache instances on the same machine conflict due to a hardcoded IPC socket path, causing a race condition. by @Foreverythin in https://github.com/LMCache/LMCache/pull/1166
• [chore]: Change code headers to SPDX License headers by @hickeyma in https://github.com/LMCache/LMCache/pull/1145
• [Bugfix] Decode in PD never makes a cache hit and causes OOM by @vladnosiv in https://github.com/LMCache/LMCache/pull/1133
• [CI]: isolate versioning fix by @sammshen in https://github.com/LMCache/LMCache/pull/1199
• [CI]: Refactor/Fix CI by @sammshen in https://github.com/LMCache/LMCache/pull/1198
• [CI]: follow up fix gpu contention by @sammshen in https://github.com/LMCache/LMCache/pull/1201
• [CI]: final fix to find-free-gpu.sh to avoid CI race conditions by @sammshen in https://github.com/LMCache/LMCache/pull/1202
• Introduce create_lookup_server_only_on_worker_0 extra config by @maobaolong in https://github.com/LMCache/LMCache/pull/1144
• [Enhancement] Add move and compress controller APIs by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1206
• [opt]: evict suffixes before prefixes by @sammshen in https://github.com/LMCache/LMCache/pull/1154
• [Enhancement] Update more controller apis by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1217
• [bugfix] fix save unfull chunk in vllm_v1_adapter by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1170

New Contributors

• @wxsms made their first contribution in https://github.com/LMCache/LMCache/pull/1053
• @JasmondL made their first contribution in https://github.com/LMCache/LMCache/pull/1048
• @yankay made their first contribution in https://github.com/LMCache/LMCache/pull/1083
• @weicaivi made their first contribution in https://github.com/LMCache/LMCache/pull/1041
• @zejunchen-zejun made their first contribution in https://github.com/LMCache/LMCache/pull/1105
• @YurianStormrage made their first contribution in https://github.com/LMCache/LMCache/pull/1026
• @YuhanLiu11 made their first contribution in https://github.com/LMCache/LMCache/pull/1124
• @vladnosiv made their first contribution in https://github.com/LMCache/LMCache/pull/1132
• @qaz-t made their first contribution in https://github.com/LMCache/LMCache/pull/1148
• @junl666 made their first contribution in https://github.com/LMCache/LMCache/pull/1140
• @Foreverythin made their first contribution in https://github.com/LMCache/LMCache/pull/1166

Full Changelog: https://github.com/LMCache/LMCache/compare/v0.3.2...v0.3.3