LMCache Files

What's Changed

• [Doc] fix typo of LMCACHE_LOCAL_CPU env var by @andyzhangx in https://github.com/LMCache/LMCache/pull/1220
• [Doc] fix broken links in CPU offloading example doc by @andyzhangx in https://github.com/LMCache/LMCache/pull/1221
• [Core] Fix decoder memory leaks and chunked prefill compatability in PD by @vladnosiv in https://github.com/LMCache/LMCache/pull/1162
• [Misc] Fix misleading logs in local disk backend by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1225
• [Bugfix] Use more stable mem alloc with cpp CUDA API. by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1233
• Allow skipping cache saving in decode phase by @yoo-kumaneko in https://github.com/LMCache/LMCache/pull/973
• [optimize] support not save chunk meta in remote backend by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1107
• [Core] Make PD and disk offloading compatible by @vladnosiv in https://github.com/LMCache/LMCache/pull/1134
• Update jinja2 requirement from <3.1 to <3.2 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1236
• [docker] remove unused patch files by @panpan0000 in https://github.com/LMCache/LMCache/pull/1235
• [Benchmark] Add long doc qa benchmark by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1248
• [Bugfix] remove duplicate loop code by @ZhongsJie in https://github.com/LMCache/LMCache/pull/1245
• [Bugfix] Get cache size from memory when removing a cache without accessing disk by @YichuanSun in https://github.com/LMCache/LMCache/pull/1197
• [Bugfix]: dont pin on second lookup by @sammshen in https://github.com/LMCache/LMCache/pull/1254
• [Feat] Enable using CXX 11 ABI when compiling LMCache by @ApostaC in https://github.com/LMCache/LMCache/pull/1250
• Update pytest requirement from <8.2,>=7.0 to >=7.0,<8.5 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1240
• [Misc] Update long doc qa for gpt-oss by @kobe0938 in https://github.com/LMCache/LMCache/pull/1255
• [Doc] Update PD doc by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1263
• [Fix] Fix the CI failure by using cxx11 abi by default by @ApostaC in https://github.com/LMCache/LMCache/pull/1262
• [Bugfix] Fix synchronization bug in layerwise pipelining by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1267
• [CI/Build] Add comprehensive test pipeline by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1249
• [Fix]: Pin torch version again by @sammshen in https://github.com/LMCache/LMCache/pull/1274
• Refactor config to make it easy to extend by @maobaolong in https://github.com/LMCache/LMCache/pull/1266
• [feature] support batched_get_blocking in remote backend by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1027
• [Docs] Multimodality by @sammshen in https://github.com/LMCache/LMCache/pull/1226
• [Bugfix] Close file descriptor when using use_odirect feature by @kyet in https://github.com/LMCache/LMCache/pull/1264
• [fix]: race condition on multi-lookup then multi-forward-passes on the same request by @sammshen in https://github.com/LMCache/LMCache/pull/1268
• [Doc] Update Pd doc (XPYD part) by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1272
• [Refactor, Enhancement] Add more cache policies and separate them from storage backends by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1278
• [feature] add exists sync interface by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/828
• [Docs] Add docs for new features by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1287
• [Misc]: move init_lmcache_engine() to v1 adapter by @sammshen in https://github.com/LMCache/LMCache/pull/1283
• [fix]: Re-enable PR 1098 by @hickeyma in https://github.com/LMCache/LMCache/pull/1247
• [Connector] mooncake: add multi-layer storage by @stmatengss in https://github.com/LMCache/LMCache/pull/1271
• [chore] Update MAINTAINERS.md to keep it up to date by @ApostaC in https://github.com/LMCache/LMCache/pull/1289
• [bugfix] doc improvement: use the same model name in quick start example share kv cache by @mengbingrock in https://github.com/LMCache/LMCache/pull/1286
• Update pytest-html requirement from <4.0,>=3.2 to >=3.2,<5.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1230
• Bump step-security/harden-runner from 2.12.1 to 2.13.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1232
• Bump docker/login-action from 3.4.0 to 3.5.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1231
• [Add] new prometheus metrics to calculate hit rate during lookup by @ApostaC in https://github.com/LMCache/LMCache/pull/1261
• [Benchmark + Perf Improvement] Improve cache blending perf + add a benchmark by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1301
• [metrics] add local cpu evict metrics by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1210
• [doc] Fix wrong mailto format by @xleoken in https://github.com/LMCache/LMCache/pull/1307
• [Core] move os.remove from submit_put_task thread to worker thread by @RangerCD in https://github.com/LMCache/LMCache/pull/1242
• Use dedicated read-write functions and eliminate redundant codes in local disk backend by @llc-kc in https://github.com/LMCache/LMCache/pull/1308
• [RemoteBackend] Improve the auditconnector to help insight traffic of remote by @maobaolong in https://github.com/LMCache/LMCache/pull/1204
• [fix]: check remote put task by @sammshen in https://github.com/LMCache/LMCache/pull/1113
• Fix the config logic of create_lookup_server_only_on_worker_0 by @maobaolong in https://github.com/LMCache/LMCache/pull/1256
• [Controller] Introduce health check by controller by @maobaolong in https://github.com/LMCache/LMCache/pull/1219
• [CacheEngine] Support separate cache key by user through lmcache tags by @xshwu-ai in https://github.com/LMCache/LMCache/pull/1200
• [VLLM|CacheEngine]support only save on rank 0 for DS MLA by @maobaolong in https://github.com/LMCache/LMCache/pull/1128
• [Fix] Fix compress failed by @fourierr in https://github.com/LMCache/LMCache/pull/1333
• [Add] add decompress support by @fourierr in https://github.com/LMCache/LMCache/pull/1335
• [Misc] Update contributor doc by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1348
• [metrics] add memory related metrics by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1334
• Support exclude specific cmds from audit_connector by @zhengliyuan1987 in https://github.com/LMCache/LMCache/pull/1338
• [Docs]: Link the onboarding issue by @sammshen in https://github.com/LMCache/LMCache/pull/1347
• Lmcache internal api server for metrics export by @maobaolong in https://github.com/LMCache/LMCache/pull/1318
• [Misc] Fix error handling and documentation errors by @tanruixiang in https://github.com/LMCache/LMCache/pull/1336
• [bugfix] adapt old vllm version by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1332
• [bugfix] fix local cache usage error by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1211
• [CI/Build] Cleanup thoroughly nightly by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1360
• [Enhancement, Bugfix] Improve lookup in TP by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1366
• [Bugfix] Fix layerwise tensor unalignment by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1369
• Support Get or set log level via internal api server by @maobaolong in https://github.com/LMCache/LMCache/pull/1359
• [Fix] EngineCore shutdown when some .pt files was lost by @fourierr in https://github.com/LMCache/LMCache/pull/1314
• [Enhancement] Adding S3 connector by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1374
• Workaround for issue [#1346] by @sdimitro in https://github.com/LMCache/LMCache/pull/1356
• [CD]: Dynamic Torch Versioning by @sammshen in https://github.com/LMCache/LMCache/pull/1363
• [Fix] add awscrt to common requirements by @walterbm in https://github.com/LMCache/LMCache/pull/1383
• Support addon num_mtp_layers from num_nextn_predict_layers by @maobaolong in https://github.com/LMCache/LMCache/pull/1320
• Add metrics to LocalCPUBackend by lambda by @maobaolong in https://github.com/LMCache/LMCache/pull/1344
• Introduce audit_backend to record audit log of backend by @maobaolong in https://github.com/LMCache/LMCache/pull/1205
• Support show thread info within cache engine internal api server by @maobaolong in https://github.com/LMCache/LMCache/pull/1358
• Misc Upstreaming by @sdimitro in https://github.com/LMCache/LMCache/pull/1299
• [Bugfix][Enhancement] Improve multi-round-qa.py and data_preprocessing.py scripts by @kyet in https://github.com/LMCache/LMCache/pull/1321
• [Core] Fix hash mismatch between LMCache and vLLM by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1380
• [Bugfix][PD] Fix config setting by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1391
• perf(mooncakestore connector): optimize get/put with zero-copy operations by @xiaguan in https://github.com/LMCache/LMCache/pull/1269
• [Benchmark]: Add a TTFT estimator/drawer by @sammshen in https://github.com/LMCache/LMCache/pull/1389
• Improve internal api server to support multiple lmcache server by @maobaolong in https://github.com/LMCache/LMCache/pull/1387
• Introduce run scripts api to cache engine api server by @maobaolong in https://github.com/LMCache/LMCache/pull/1385
• Bump actions/checkout from 4.2.2 to 5.0.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1312
• [CI]: Add build support for Nvidia B200 GPU by @hickeyma in https://github.com/LMCache/LMCache/pull/1396
• [Bugfix] Fix layerwise eviction by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1398
• Support get config and metadata api of LMCache internal api server by @maobaolong in https://github.com/LMCache/LMCache/pull/1382
• [Misc] Improve S3 a bit by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1402
• [refact] use request_configs replace tags by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1377
• [bugfix][#1357] Unified the default value if use mla by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/1394
• Bump actions/download-artifact from 4.3.0 to 5.0.0 by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1309
• Bump pre-commit from 4.2.0 to 4.3.0 in the minor-update group by @dependabot[bot] in https://github.com/LMCache/LMCache/pull/1386
• [Enhancement] Add NUMA-aware Memory Placement by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1409
• [CI/Build] Refactor comprehensive test config settings by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1400
• [bugfix] Fix unclose element in md file by @xleoken in https://github.com/LMCache/LMCache/pull/1392
• [Misc] Allow passing nixl backends when creating nixl_agent in nixl_connector_v3 by @jinqinn in https://github.com/LMCache/LMCache/pull/1343
• [CI/Build] Calculate performance number for cpu and disk backend by @Shaoting-Feng in https://github.com/LMCache/LMCache/pull/1410
• [feat] Introduce a plugin framework by @maobaolong in https://github.com/LMCache/LMCache/pull/1401
• [InternalApiServer]Support start internal api_server by socket path by @maobaolong in https://github.com/LMCache/LMCache/pull/1407
• [Enhancement] Add sparse attention for kv blending by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1414
• [bugfix] Fix unresolved attribute reference 'tokens' for class 'ClearWorkerMsg' by @xleoken in https://github.com/LMCache/LMCache/pull/1418
• [Bugfix] Fix double semaphore acquire in s3 connector by @YaoJiayi in https://github.com/LMCache/LMCache/pull/1427
• [Docker]: Add build support for Nvidia B200 GPU to container image by @hickeyma in https://github.com/LMCache/LMCache/pull/1408

New Contributors

• @andyzhangx made their first contribution in https://github.com/LMCache/LMCache/pull/1220
• @ZhongsJie made their first contribution in https://github.com/LMCache/LMCache/pull/1245
• @YichuanSun made their first contribution in https://github.com/LMCache/LMCache/pull/1197
• @kyet made their first contribution in https://github.com/LMCache/LMCache/pull/1264
• @RangerCD made their first contribution in https://github.com/LMCache/LMCache/pull/1242
• @xshwu-ai made their first contribution in https://github.com/LMCache/LMCache/pull/1200
• @fourierr made their first contribution in https://github.com/LMCache/LMCache/pull/1333
• @zhengliyuan1987 made their first contribution in https://github.com/LMCache/LMCache/pull/1338
• @tanruixiang made their first contribution in https://github.com/LMCache/LMCache/pull/1336
• @walterbm made their first contribution in https://github.com/LMCache/LMCache/pull/1383
• @jinqinn made their first contribution in https://github.com/LMCache/LMCache/pull/1343

Full Changelog: https://github.com/LMCache/LMCache/compare/v0.3.3...v0.3.4