LMDeploy Files

What's Changed

🚀 Features

• Support Interns2 mobius by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4816
• feat: support GLM-5.2 by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4737
• Intern-S2-Mobius meta-MoE support, MoE gate v2, CP attention fixes by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4835
• Add TurboMind ViT support for InternVL and Qwen VL models by @irexyc in https://github.com/InternLM/lmdeploy/pull/4719
• feat: add Hy3 support, MTP, and FP8 optimizations by @yidingcheng0206 in https://github.com/InternLM/lmdeploy/pull/4815

💥 Improvements

• refactor: report cache usage directly by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4798
• SM90 native BF16/FP8 GEMM kernels, fused-SiLU quantization, and linear test harness by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4795
• refactor: split api server endpoints by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4797
• refactor(pytorch): derive CUDA step metadata from selected operators by @grimoire in https://github.com/InternLM/lmdeploy/pull/4805
• optimize and modularize SSM prefix caching by @grimoire in https://github.com/InternLM/lmdeploy/pull/4788
• perf(guided-decoding): optimize with async D2H copy and xgrammar v0.2.1 by @windreamer in https://github.com/InternLM/lmdeploy/pull/4605
• refactor(serve): split chat_completions endpoint into a package by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4840
• feat(chat-completions): add usage.completion_tokens_details by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4842
• feat(pytorch): add optimized Gluon blocked FP8 GEMM for Hopper by @grimoire in https://github.com/InternLM/lmdeploy/pull/4830
• perf(pytorch): add opt-in torch.compile for decode CUDA graphs by @grimoire in https://github.com/InternLM/lmdeploy/pull/4808
• Ssm prefix cache non aligned by @grimoire in https://github.com/InternLM/lmdeploy/pull/4799
• perf: optimize GLM-5.2 serving by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4827
• refactor: separate request preprocessing from generation by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4856

🐞 Bug fixes

• [Bugfix] Fix PyTorch H2D input lifetime across CUDA streams by @grimoire in https://github.com/InternLM/lmdeploy/pull/4792
• fix(serve): reject empty/falsy prompt input in format_prompts and AsyncEngine.generate by @SuperMarioYL in https://github.com/InternLM/lmdeploy/pull/4803
• Fix ray mp duplicate output by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4833
• fix(turbomind): dispatch cuMemcpyBatchAsync by CUDA runtime version by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4838
• fix(disagg): use JSON instead of pickle for P2P ZMQ requests (#4804) by @Anai-Guo in https://github.com/InternLM/lmdeploy/pull/4812
• fix(serve): emit signatures for Anthropic thinking blocks by @matrix72c in https://github.com/InternLM/lmdeploy/pull/4851
• Fix int4 KV cache quantization range when the packed head width is not a power of two by @truong-v in https://github.com/InternLM/lmdeploy/pull/4850
• fix: harden serving request validation by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4872
• fix: fix allgather/allgather2d for cuda-ipc when byte_width is not multiple of uint by @irexyc in https://github.com/InternLM/lmdeploy/pull/4873

📚 Documentations

• docs: update recent model support by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4855
• docs,tests: cover Qwen3.8 preserve_thinking support by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4869

🌐 Other

• docs: remove non-existent --enable-metrics flag from metrics/spec_decoding guides by @latent-9 in https://github.com/InternLM/lmdeploy/pull/4809
• Upgrade to cu130 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4753
• TEST: update turbomind qwen3.5 config by @littlegy in https://github.com/InternLM/lmdeploy/pull/4778
• chore: use python3.12 for docformatter pre-commit hook by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4839
• ci: support CUDA 13.0 Docker builds and publishing by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4817
• TEST: update deepseekv4-flash config by @littlegy in https://github.com/InternLM/lmdeploy/pull/4836
• [ci] Adjust evaluation gate benchmark datasets to reduce runtime and extend coverage by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4834
• [ci] remove old models and refactor interface testcase by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4806
• bump version to v0.16.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4847

New Contributors

• @latent-9 made their first contribution in https://github.com/InternLM/lmdeploy/pull/4809
• @SuperMarioYL made their first contribution in https://github.com/InternLM/lmdeploy/pull/4803
• @yidingcheng0206 made their first contribution in https://github.com/InternLM/lmdeploy/pull/4815
• @matrix72c made their first contribution in https://github.com/InternLM/lmdeploy/pull/4851
• @truong-v made their first contribution in https://github.com/InternLM/lmdeploy/pull/4850

Full Changelog: https://github.com/InternLM/lmdeploy/compare/v0.15.0...v0.16.0