LMCache Files

What's Changed

• [MP] fix: add thread safety to Session for concurrent TP worker access by @maobaolong in https://github.com/LMCache/LMCache/pull/2807
• [CLI] Implement initial framework of LMCache CLI by @KuntaiDu in https://github.com/LMCache/LMCache/pull/2775
• [MP][Observability][1/3] EventBus core infrastructure + OpenTelemetry dependency by @royyhuang in https://github.com/LMCache/LMCache/pull/2792
• [MP]: Support delay start heartbeat thread to avoid unhealthy while start vllm for a huge module warmup. by @maobaolong in https://github.com/LMCache/LMCache/pull/2798
• fix: add None check before stream synchronization by @hlin99 in https://github.com/LMCache/LMCache/pull/2810
• [Core] Add VRAM_SEG support for NIXL OBJ plugin by @jgoldsch12 in https://github.com/LMCache/LMCache/pull/2640
• [CI]: create fallback for flaky nightly index by @sammshen in https://github.com/LMCache/LMCache/pull/2809
• [CI]: add full tag selectively by @sammshen in https://github.com/LMCache/LMCache/pull/2820
• fix: replace global lock with per-device transfer_lock to prevent deadlock by @maobaolong in https://github.com/LMCache/LMCache/pull/2816
• Refactor KV cache shape/dtype extraction for robustness by @hlin99 in https://github.com/LMCache/LMCache/pull/2537
• Support non-contiguous alloc in MemoryAllocator by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/2767
• [MP][Observability][2/3] Migrate L1 + SM to EventBus + OTel, remove old Prometheus pipeline by @royyhuang in https://github.com/LMCache/LMCache/pull/2794
• [MP][Bugfix] fixing race condition for zmq output notifier by @ApostaC in https://github.com/LMCache/LMCache/pull/2808
• [ci]: agent reviewer prompt engineering by @sammshen in https://github.com/LMCache/LMCache/pull/2800
• [refactor]: clean up the messy LMCacheManager by @sammshen in https://github.com/LMCache/LMCache/pull/2683
• [Platform]: Add Intel Gaudi (HPU) Support by @hlin99 in https://github.com/LMCache/LMCache/pull/2822
• [CLI] Implement lmcache describe kvcache subcommand by @royyhuang in https://github.com/LMCache/LMCache/pull/2825
• [MP][Feat] Query lookup-phase status for MP mode by @ApostaC in https://github.com/LMCache/LMCache/pull/2818
• Add Device-DAX (/dev/dax) storage backend for KV cache (follow-up to [#2714]) by @jayhpark530 in https://github.com/LMCache/LMCache/pull/2788
• [Temp CI Patch]: torch version for UT by @sammshen in https://github.com/LMCache/LMCache/pull/2856
• [CI] Add GitHub Action to auto-sync torch version with vLLM by @deng451e in https://github.com/LMCache/LMCache/pull/2796
• [MP][Feat] support worker-affinity in the MQ thread pool by @ApostaC in https://github.com/LMCache/LMCache/pull/2842
• Introduce native fs connector by @maobaolong in https://github.com/LMCache/LMCache/pull/2779
• [CLI] Implement lmcache ping subcommand by @Oasis-Git in https://github.com/LMCache/LMCache/pull/2859
• [MP] Fault Tolerance CI by @Oasis-Git in https://github.com/LMCache/LMCache/pull/2764
• feat: improve ValkeyConnector with cluster mode, TLS, and GLIDE optimizations by @omerrubi-amzn in https://github.com/LMCache/LMCache/pull/2790
• fix: auto-generate lmcache_instance_id when value is None by @can-sun in https://github.com/LMCache/LMCache/pull/2732
• [CI]: use job-level path filtering so skipped tests pass required checks by @royyhuang in https://github.com/LMCache/LMCache/pull/2855
• [MP] Print inference request id to help identify which vllm request the current log belongs to by @maobaolong in https://github.com/LMCache/LMCache/pull/2812
• [HW: XPU] Enable Layerwise XPU Connector by @slokesha in https://github.com/LMCache/LMCache/pull/2611
• [CLI] lmcache query engine subcommand by @deng451e in https://github.com/LMCache/LMCache/pull/2846
• [CLI]: Server command by @sammshen in https://github.com/LMCache/LMCache/pull/2836
• [LMCache CLI] Design and implementation of lmcache kvcache by @KuntaiDu in https://github.com/LMCache/LMCache/pull/2827
• [Bugfix]: Fix pin count balancing in PD Disaggregation mode by @lisiG9 in https://github.com/LMCache/LMCache/pull/2786
• [Core] [GDS] Improve GDS backend error handling and retry logic by @oferki in https://github.com/LMCache/LMCache/pull/2675
• [CLI][Doc] Edit the doc for LMCache CLI by @KuntaiDu in https://github.com/LMCache/LMCache/pull/2870
• Add hipFile support for AIS (AMD Infinity Storage) storage by @glimchb in https://github.com/LMCache/LMCache/pull/2799
• [CI]: Fix the LMCache random throughput being higher than native vllm by @sammshen in https://github.com/LMCache/LMCache/pull/2864
• [3/N][Feat]Persist metadata on device and fix raw-device benchmark setup by @DongDongJu in https://github.com/LMCache/LMCache/pull/2614
• [Core]: Support HND KV Format by @sammshen in https://github.com/LMCache/LMCache/pull/2826
• [Chore][Docs] Fix mp docs for store policy: skip_l1 by @ApostaC in https://github.com/LMCache/LMCache/pull/2869
• [MP][Core] Block id based kernel for MP mode by @ApostaC in https://github.com/LMCache/LMCache/pull/2838
• [CLI] update cli lmcache query engine by @deng451e in https://github.com/LMCache/LMCache/pull/2871
• [MP] Improve the stability for controllers and improve log clarity by @ApostaC in https://github.com/LMCache/LMCache/pull/2883
• [Chore][Docs] Stale MP CLI and Flags by @sammshen in https://github.com/LMCache/LMCache/pull/2882
• [Fix][Operator] Add privileged mode and nvidia runtime for GPU visibility by @royyhuang in https://github.com/LMCache/LMCache/pull/2749
• [Chore][CI]: chmod +x scripts in k3 test entrypoints by @sammshen in https://github.com/LMCache/LMCache/pull/2886
• feat(gds): add multipath KV-cache offloading support by @glimchb in https://github.com/LMCache/LMCache/pull/2817
• fix: add missing lock protection for LRU cache policy by @SYaoJun in https://github.com/LMCache/LMCache/pull/2860
• [MP][Observability][3/3] Migrate MP server telemetry to EventBus, unify config by @royyhuang in https://github.com/LMCache/LMCache/pull/2806
• [doc] update installation compatibility doc by @deng451e in https://github.com/LMCache/LMCache/pull/2868
• [Build] add SM120 for wheel build by @deng451e in https://github.com/LMCache/LMCache/pull/2873
• [1/2] L2 CI: End to End Performance by @Oasis-Git in https://github.com/LMCache/LMCache/pull/2884
• [fix] add missing request type in blend server by @deng451e in https://github.com/LMCache/LMCache/pull/2894
• type: Add missing return type annotations to storage backend methods by @SYaoJun in https://github.com/LMCache/LMCache/pull/2829
• [CLI] Implementation of lmcache bench engine by @ApostaC in https://github.com/LMCache/LMCache/pull/2889
• feat(gds): enable parallel I/O thread pool for all cuFile filesystems by @glimchb in https://github.com/LMCache/LMCache/pull/2802
• [DSA] support DSA in Mooncake connector by @chunxiaozheng in https://github.com/LMCache/LMCache/pull/2897
• [Core] Add L2 eviction in mp mode by @YaoJiayi in https://github.com/LMCache/LMCache/pull/2824
• [Bugfix] fix the invalid image path by @SYaoJun in https://github.com/LMCache/LMCache/pull/2899
• [Chore][CI] Split k3 multiprocess tests into parallel pipeline steps by @sammshen in https://github.com/LMCache/LMCache/pull/2914
• Support l2 adapter check and improve basic_check tool by @maobaolong in https://github.com/LMCache/LMCache/pull/2895
• [Chore][CI/Docs]: Switch all the documentation and CI over to lmache cli by @sammshen in https://github.com/LMCache/LMCache/pull/2917
• [CI] Add CI test for CB by @deng451e in https://github.com/LMCache/LMCache/pull/2900
• [2/2] L2 CI: Telemetry Test by @Oasis-Git in https://github.com/LMCache/LMCache/pull/2913
• [Core] Add eviction for CB by @YaoJiayi in https://github.com/LMCache/LMCache/pull/2893
• Refactor: Generalize utils.py for all devices by lifting the CUDA limitation by @hlin99 in https://github.com/LMCache/LMCache/pull/2848
• Add argument --prefetch-max-in-flight to fix hardcode by @maobaolong in https://github.com/LMCache/LMCache/pull/2789
• [MP] Refactor l2 plugin framework to support dynamic load third-party native l2 connector by @maobaolong in https://github.com/LMCache/LMCache/pull/2851
• fix: relax worker port count assertion by @can-sun in https://github.com/LMCache/LMCache/pull/2867
• [Bugfix]: patch save_decode_cache by @sammshen in https://github.com/LMCache/LMCache/pull/2929
• vllm block event by @Oasis-Git in https://github.com/LMCache/LMCache/pull/2930
• [Feat]: Add eviction to L2 Native Backend by @sammshen in https://github.com/LMCache/LMCache/pull/2939
• [Connector] Maru: zero-copy KV cache sharing via CXL shared memory by @jooho-XCENA in https://github.com/LMCache/LMCache/pull/2705
• [MP] Fix UT after merge [#2851] by @maobaolong in https://github.com/LMCache/LMCache/pull/2931
• [Bugfix]: fix get_num_heads for MLA format by @sammshen in https://github.com/LMCache/LMCache/pull/2941
• [MP] Introduce l2 mooncake adapter by @maobaolong in https://github.com/LMCache/LMCache/pull/2911
• [CLI]Add long-doc-permutator CLI bench workload by @deng451e in https://github.com/LMCache/LMCache/pull/2937
• feat(gds): add gds_path_sharding config for multi-path strategy by @glimchb in https://github.com/LMCache/LMCache/pull/2922
• [Security][Remote Connector]: Add env var auth config for RESP by @sammshen in https://github.com/LMCache/LMCache/pull/2949
• Refactor: Align pd_buffer_size to chunk size in PD backend by @hlin99 in https://github.com/LMCache/LMCache/pull/2694
• [Chore] Add CODEOWNERS for automated PR review assignments by @sammshen in https://github.com/LMCache/LMCache/pull/2950
• [Chore][CI]: Change dst for K3 nightly comprehensive results by @sammshen in https://github.com/LMCache/LMCache/pull/2958

New Contributors

• @jgoldsch12 made their first contribution in https://github.com/LMCache/LMCache/pull/2640
• @jayhpark530 made their first contribution in https://github.com/LMCache/LMCache/pull/2788
• @omerrubi-amzn made their first contribution in https://github.com/LMCache/LMCache/pull/2790
• @can-sun made their first contribution in https://github.com/LMCache/LMCache/pull/2732
• @slokesha made their first contribution in https://github.com/LMCache/LMCache/pull/2611
• @glimchb made their first contribution in https://github.com/LMCache/LMCache/pull/2799
• @SYaoJun made their first contribution in https://github.com/LMCache/LMCache/pull/2860
• @jooho-XCENA made their first contribution in https://github.com/LMCache/LMCache/pull/2705

Full Changelog: https://github.com/LMCache/LMCache/compare/v0.4.2...v0.4.3