LMDeploy Files

What's Changed

🚀 Features

• FP8 kv cache quantization by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4563
• Support Qwen3 Omni by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4411
• support qwen3.5(vit) inference in turbomind backend by @irexyc in https://github.com/InternLM/lmdeploy/pull/4602
• Add OpenAI Responses-compatible endpoint by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4582
• Add /get_ppl endpoint by @irexyc in https://github.com/InternLM/lmdeploy/pull/4679

💥 Improvements

• Update turbomind modeling infrastructure by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4557
• refactor(turbomind): consolidate CUDA error handling and add manual stacktracing by @lzhangzz in https://github.com/InternLM/lmdeploy/pull/4565
• Add Qwen3.5 Moe lite awq by @43758726 in https://github.com/InternLM/lmdeploy/pull/4561
• [Improve]: Drain queues when sleep engine by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4577
• Extend chat completions by introducing token-in/out and returning routed experts by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4593
• Follow openai's spec to add "AllowedToolChoice" and report 400 when parsing request failed by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4585
• Improve health endpoint by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4615
• Remove state init by @grimoire in https://github.com/InternLM/lmdeploy/pull/4604
• Include spec stats in metrics by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4625
• Add raw chat completion logprob output by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4637
• fix(pytorch): offload guided decoding CPU ops to thread pool to prevent event loop blocking by @windreamer in https://github.com/InternLM/lmdeploy/pull/4590
• update gated delta rule state layout by @grimoire in https://github.com/InternLM/lmdeploy/pull/4636
• Improve kernel dispatch for dp>1 by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4653
• Extend v1/messages by introducing token-in/out and returning routed experts by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4642
• Fuse gdr preprocess by @grimoire in https://github.com/InternLM/lmdeploy/pull/4656
• refactor: simplify multimodal preprocessing expansion by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4663
• feat: configure cudagraph capture batch sizes by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4573
• Refactor prefix caching for pytorch engine by @grimoire in https://github.com/InternLM/lmdeploy/pull/4618
• Pading one more block for fa3 prefill by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4674
• Add usage.prompt_tokens_details.cached_tokens for prefix caching by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4670
• Optimize XML tool parsers with incremental streaming and fast-path buffering by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4664

🐞 Bug fixes

• fix the anthropic adapter by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4578
• Fix Structured Output for GPT-OSS Models by @windreamer in https://github.com/InternLM/lmdeploy/pull/4386
• Allow W8A8Linear to accept dtype during initialization instead of hard code by @43758726 in https://github.com/InternLM/lmdeploy/pull/4586
• fix: compact split multimodal tensors by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4583
• Fix legacy VLM preprocessors for normalized image data by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4584
• fix dockerfile which missing common.txt by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4608
• fix: enable FA3 for SM80+ GPUs and fix CUDA version comparison by @windreamer in https://github.com/InternLM/lmdeploy/pull/4591
• flatten_kv_cache zero padding by @grimoire in https://github.com/InternLM/lmdeploy/pull/4613
• align streaming usage chunks with OpenAI spec by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4616
• fix(vl): reduce multimodal feature memory use by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4603
• fix memleak when input contain large image data by @grimoire in https://github.com/InternLM/lmdeploy/pull/4610
• fix(turbomind): map Intern-S1 HF checkpoint keys by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4617
• fix(serve): emit all stream_chunk deltas to fix concurrent tool-call streaming by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4622
• fix cp inference by @irexyc in https://github.com/InternLM/lmdeploy/pull/4619
• refactor(serve): avoid per-request tokenizer work in parsers by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4633
• Bring MixtralForCausalLM back to Turbomind by @43758726 in https://github.com/InternLM/lmdeploy/pull/4623
• fix model loading on windows by @irexyc in https://github.com/InternLM/lmdeploy/pull/4626
• Fix mtp cudagraph when no warmup in RL by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4641
• fix: remove hard CUDA_PATH assert on Windows, search DLL paths from multiple sources by @windreamer in https://github.com/InternLM/lmdeploy/pull/4628
• Fix unit test by removing latest-transformers-unsupported models by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4649
• Fix qwen3.5 mtp by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4652
• fix gdr kernel for tilelang>=0.1.9 by @grimoire in https://github.com/InternLM/lmdeploy/pull/4660
• [Fix]: Revert the reuse of cudagraph buffer for mtp by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4661
• Fix client-disconnect session leaks in PyTorch MP engine by @grimoire in https://github.com/InternLM/lmdeploy/pull/4655
• fix cancel stopped seq by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4654
• feat: support num_experts_per_tok=10 in turbomind backend by @irexyc in https://github.com/InternLM/lmdeploy/pull/4665
• fix batched seqs with different stop words by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4671
• Move warmup inside wakeup by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4667
• Fix dequant_mixed by @irexyc in https://github.com/InternLM/lmdeploy/pull/4657
• Improve engine health monitoring by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4645
• fix qwen3.5 27b gdr preprocess by @grimoire in https://github.com/InternLM/lmdeploy/pull/4676
• Fix dequant mixed for qwen3.5 quantized model made by vllm/llm-compressor by @irexyc in https://github.com/InternLM/lmdeploy/pull/4675
• [Bugfix] Fix double-counted max_q_seqlen in decode delta kv_seqlens by @waynehacking8 in https://github.com/InternLM/lmdeploy/pull/4685
• Fix scheduler for ssm by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4691
• fix(serve): avoid parallel tool-call argument leakage in XML parsers by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4692
• fix prefix caching by @grimoire in https://github.com/InternLM/lmdeploy/pull/4700

📚 Documentations

• docs: update multimodal model support docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4643

🌐 Other

• chore: gate request logs behind request level by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4581
• miss rdkit for intern-s models by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4587
• extract common deps into requirements/common.txt by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4595
• Remove staled cli arg in vlmevalkit docs by @CUHKSZzxy in https://github.com/InternLM/lmdeploy/pull/4598
• log reponse for debugging by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4592
• cancel in-progress runs when PR is updated or merged by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4609
• TEST: update qwen3.5 397b test by @littlegy in https://github.com/InternLM/lmdeploy/pull/4607
• TEST: update video test by @littlegy in https://github.com/InternLM/lmdeploy/pull/4606
• Validate final chat response structure by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4621
• Support dp for qwen35 mtp by @RunningLeon in https://github.com/InternLM/lmdeploy/pull/4611
• [ci] refactor testcoverage config by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4630
• TEST: update ascend and mtp test config by @littlegy in https://github.com/InternLM/lmdeploy/pull/4659
• TEST: update FP8 processing logic and remove duplicate MTP tests by @littlegy in https://github.com/InternLM/lmdeploy/pull/4668
• freeze tilelang version by @grimoire in https://github.com/InternLM/lmdeploy/pull/4669
• fix windows ci by @irexyc in https://github.com/InternLM/lmdeploy/pull/4672
• [ci] add mtp test config in pr_test by @zhulinJulia24 in https://github.com/InternLM/lmdeploy/pull/4651
• support disaggregated weight update by @irexyc in https://github.com/InternLM/lmdeploy/pull/4638
• bump version to v0.14.0 by @lvhan028 in https://github.com/InternLM/lmdeploy/pull/4689

New Contributors

• @waynehacking8 made their first contribution in https://github.com/InternLM/lmdeploy/pull/4685

Full Changelog: https://github.com/InternLM/lmdeploy/compare/v0.13.0...v0.14.0