SGLang Files

Highlights

• Day 0 Support for DeepSeek-V3.2 with Sparse Attention: https://lmsys.org/blog/2025-09-29-deepseek-V32/
• Deterministic inference on multiple attention backends: https://lmsys.org/blog/2025-09-22-sglang-deterministic/
• Integration of FlashAttention 4 prefill kernels
• Enhancing support of Qwen3-Next with MTP, DP, optimized kernels and multiple hardware platforms
• Support models including Qwen3-VL series, dots.vlm1, Ling-V2, Apertus, SOLAR

What's Changed

• [Auto Sync] Update server_args.py (20250912) by @merrymercy in https://github.com/sgl-project/sglang/pull/10347
• [CPU][doc] add torch.compile param in example commands by @ZailiWang in https://github.com/sgl-project/sglang/pull/10349
• [router][ci] Add gpu utilization analyze with nvml by @key4ng in https://github.com/sgl-project/sglang/pull/10345
• [NVIDIA] [3/N] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked by @wenscarl in https://github.com/sgl-project/sglang/pull/9199
• fix: flashinfer_cutlass_moe: Use max of global expert scales instead of local for input scale by @trevor-m in https://github.com/sgl-project/sglang/pull/10296
• model: support Apertus by @EduardDurech in https://github.com/sgl-project/sglang/pull/9774
• fix dual stream bug by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10352
• [router] Basic OAI Response api by @key4ng in https://github.com/sgl-project/sglang/pull/10346
• Implement Standalone gRPC Server for SGLang Python Scheduler by @CatherineSue in https://github.com/sgl-project/sglang/pull/10283
• support memory_pool_host page first direct layout by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/10031
• fix the break in FlashInferFusedMoE by @chenqianfzh in https://github.com/sgl-project/sglang/pull/10356
• fix: resolve transfer_kv_all_layer_direct_lf_pf import error by @zhyncs in https://github.com/sgl-project/sglang/pull/10360
• Support LingV2 model by @strgrb in https://github.com/sgl-project/sglang/pull/10359
• Fix Bailing MoE model bugs by @yuan-luo in https://github.com/sgl-project/sglang/pull/10362
• Revert add mainprocess's proctitle by @whybeyoung in https://github.com/sgl-project/sglang/pull/10351
• model: support dots.vlm1 model by @yonghenglh6 in https://github.com/sgl-project/sglang/pull/8778
• Support loading weights from remote instance by @amysaq2023 in https://github.com/sgl-project/sglang/pull/8215
• add qwen3-next ut by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10355
• Fix chunked prefix cache for nvfp4 by @wenscarl in https://github.com/sgl-project/sglang/pull/10180
• Fix FA4 import cause moe_fused_gate output be illegal memory by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10368
• Fix global input scale incompatible with CuTe DSL moe by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10370
• [router] Add Rerank Routing Logic in Regular Router by @fangjian601 in https://github.com/sgl-project/sglang/pull/10219
• [router] enable sccache in ci and local build by @slin1237 in https://github.com/sgl-project/sglang/pull/10099
• fix: add fast path for function call by @yizhang2077 in https://github.com/sgl-project/sglang/pull/9023
• [Auto Sync] Update base_grammar_backend.py, llguidance_back... (20250911) by @merrymercy in https://github.com/sgl-project/sglang/pull/10333
• fix: resolve gb200 image link by @zhyncs in https://github.com/sgl-project/sglang/pull/10343
• fix: exclude protobuf generated code by @zhyncs in https://github.com/sgl-project/sglang/pull/10388
• [bug] fix ci syntax by @slin1237 in https://github.com/sgl-project/sglang/pull/10390
• Fix GPU fault issue when run dsv3 with dp mode and enable torch-compile by @kkHuang-amd in https://github.com/sgl-project/sglang/pull/10361
• feat: add deepseek v3 fp4 ut by @zhyncs in https://github.com/sgl-project/sglang/pull/10391
• Add sentencepiece to project dependencies by @mmangkad in https://github.com/sgl-project/sglang/pull/10386
• [router] allow one router to support different model families and serving mode by @slin1237 in https://github.com/sgl-project/sglang/pull/10244
• [router] Add get and cancel method for response api by @key4ng in https://github.com/sgl-project/sglang/pull/10387
• Benchmark: Support API_KEY without 'bearer' by @Muqi1029 in https://github.com/sgl-project/sglang/pull/10380
• Support Qwen3-Next on Ascend NPU by @iforgetmyname in https://github.com/sgl-project/sglang/pull/10379
• [HiCache] fix mooncake config in different tp size by @stmatengss in https://github.com/sgl-project/sglang/pull/10377
• [HiCache] doc: update deployment in readme by @stmatengss in https://github.com/sgl-project/sglang/pull/10332
• [router] add not implemented functions for multi model trait by @slin1237 in https://github.com/sgl-project/sglang/pull/10394
• [Auto Sync] Update xgrammar_backend.py (20250913) by @merrymercy in https://github.com/sgl-project/sglang/pull/10395
• fix probs name which without temp scaling name by @narutolhy in https://github.com/sgl-project/sglang/pull/9984
• Fix the style of sgl kernel by @merrymercy in https://github.com/sgl-project/sglang/pull/10398
• fix: tool parse in large streaming chunk beginning with normal content by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10397
• [Fix] Init mamba related memory pools with torch.zeros by @byjiang1996 in https://github.com/sgl-project/sglang/pull/10400
• support qwen3_next blackwell by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10403
• [Fix] Support qwen3-next MTP+DP by @byjiang1996 in https://github.com/sgl-project/sglang/pull/10392
• Update ROCm docker image to add sgl-router support by @kkHuang-amd in https://github.com/sgl-project/sglang/pull/10406
• [Performance] Dynamic Batch Tokenizer by @sundar24295s in https://github.com/sgl-project/sglang/pull/9382
• [Generative Score API] Scoring(Prefill-only) optimizations. by @sundar24295s in https://github.com/sgl-project/sglang/pull/9748
• Remove repeatedly lists adding in init_incremental_detokenization by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10412
• [Hack] Add pd-disaggregation decode polling interval by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10411
• fix duplicated logger in eager_utils by @lj970926 in https://github.com/sgl-project/sglang/pull/10410
• Fix cutlass moe accuracy drop caused by attention UB from DP padding mode by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10414
• Add self.capture_aux_hidden_states For GLM-4.5V by @zRzRzRzRzRzRzR in https://github.com/sgl-project/sglang/pull/10228
• Add h200 fused moe config for Qwen3-Next by @Ximingwang-09 in https://github.com/sgl-project/sglang/pull/10404
• Auto determine sgl kernel version in blackwell CI by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10318
• Fix the global scale fix does not support EPLB and improve enabling condition by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10369
• Let sgl-kernel changes be tested on srt by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10313
• [2/2] Speed up prefill mla attention concat by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10157
• Support offloading in fp8 by @fzyzcjy in https://github.com/sgl-project/sglang/pull/9948
• Support global scale in addition to per expert scale for cutedsl moe by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10270
• Support profile args in Engine API by @fzyzcjy in https://github.com/sgl-project/sglang/pull/6539
• Fix sgl-kernel + srt CI by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10419
• [PD metrics] Fix some uncompleted PD related metrics by @acelyc111 in https://github.com/sgl-project/sglang/pull/8627
• Typo: in --enable-custom-logit-processor: agree with cli arg by @thalahors in https://github.com/sgl-project/sglang/pull/10076
• [Feature] Sglang Tracing: Fine-Grained Tracking for Request Latency - Part 1 by @sufeng-buaa in https://github.com/sgl-project/sglang/pull/9962
• fix: use latest flashinfer by @zhyncs in https://github.com/sgl-project/sglang/pull/10428
• fix: enable cu124 and cu128 build on main push by @zhyncs in https://github.com/sgl-project/sglang/pull/10431
• [Fix] MoE: fix w8a8_fp8 MoE and add tests to cover this code path by @ch-wan in https://github.com/sgl-project/sglang/pull/10429
• Add split tile size for Triton attention by @ispobock in https://github.com/sgl-project/sglang/pull/10425
• Fix correction bias undefined behavior for nvfp4 models by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10426
• feat: add dsv3 fp4 cutlass moe etp ut by @zhyncs in https://github.com/sgl-project/sglang/pull/10433
• router: Add Embedding routing logic by @tao12345666333 in https://github.com/sgl-project/sglang/pull/10129
• Revert "Fix FA4 import cause moe_fused_gate output be illegal memory" by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10432
• [4/N]DP refactor: support watching mode get_load and shortest queue strategy by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10201
• automatically label pr for ci by @merrymercy in https://github.com/sgl-project/sglang/pull/10435
• Refactor TopK to ensure readability and extensibility by @ch-wan in https://github.com/sgl-project/sglang/pull/9338
• Tiny fix wrong naming by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10437
• Fix label pr for ci by @merrymercy in https://github.com/sgl-project/sglang/pull/10441
• metrics: support customer labels specified in request header by @acelyc111 in https://github.com/sgl-project/sglang/pull/10143
• [docs / oneliner] update mmmu docs instruction by @vincentzed in https://github.com/sgl-project/sglang/pull/9768
• Add reasoning examples for GPT-OSS in Markdown examples by @vincentzed in https://github.com/sgl-project/sglang/pull/9626
• Fix label PR by @merrymercy in https://github.com/sgl-project/sglang/pull/10445
• Update permissions in label-pr.yml by @merrymercy in https://github.com/sgl-project/sglang/pull/10450
• [CI] Fix token key in label-pr.yml workflow by @merrymercy in https://github.com/sgl-project/sglang/pull/10452
• fix: fix max_new_tokens uninitialized error by @mickqian in https://github.com/sgl-project/sglang/pull/9343
• [router] fix service discovery and mcp ut by @slin1237 in https://github.com/sgl-project/sglang/pull/10449
• fix(server_args): Skip chunked_prefill_size validation when disaggregation mode is decode by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/10358
• [router] add dependency for router by @ooapex in https://github.com/sgl-project/sglang/pull/10401
• [router] fix logger ordering git ctx by @CatherineSue in https://github.com/sgl-project/sglang/pull/10457
• Update GITHUB_TOKEN secret for documentation push by @merrymercy in https://github.com/sgl-project/sglang/pull/10458
• [HotFix]: Hot fix import path in 3fs_bench_client.py by @hzh0425 in https://github.com/sgl-project/sglang/pull/10463
• Add rtx5880 moe triton by @Jimmy-L99 in https://github.com/sgl-project/sglang/pull/10439
• Run tests based on labels by @merrymercy in https://github.com/sgl-project/sglang/pull/10456
• Fix trtllm_moe wrong correction bias by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10440
• feat: support pip install sglang by @zhyncs in https://github.com/sgl-project/sglang/pull/10465
• chore: bump v0.5.3rc0 by @zhyncs in https://github.com/sgl-project/sglang/pull/10468
• [router] minor code clean up in server startup by @CatherineSue in https://github.com/sgl-project/sglang/pull/10470
• [bugfix] fix typo by @1195343015 in https://github.com/sgl-project/sglang/pull/10471
• [PD metrics] Add latency Histogram metrics of each stage for generate requests by @acelyc111 in https://github.com/sgl-project/sglang/pull/8710
• [CI] Fix runner for sgl-kernel by @csahithi in https://github.com/sgl-project/sglang/pull/9887
• fix(internvl): fix accuracy issue of normalization by @KEVINTUAN12 in https://github.com/sgl-project/sglang/pull/10375
• fix: gpt-oss streaming dropping normal content when tools are provided but not used by @jonaslsaa in https://github.com/sgl-project/sglang/pull/9657
• model: support solar by @ppraneth in https://github.com/sgl-project/sglang/pull/8189
• fix: resolve sgl-kernel ut by @zhyncs in https://github.com/sgl-project/sglang/pull/10476
• [1/2] Speed up trtllm_mla attention backend (>10% e2e) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10473
• Fix --dataset-path in bench_one_batch_server by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10475
• [Env] minimal version for organizing envs by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10479
• chore: bump v0.3.10 sgl-kernel by @zhyncs in https://github.com/sgl-project/sglang/pull/10478
• [router] multi model registration fix by @CatherineSue in https://github.com/sgl-project/sglang/pull/10481
• [2/4] Introduce Chunked-SGMV kernels and corresponding LoRA backend for improved performance by @lifuhuang in https://github.com/sgl-project/sglang/pull/10286
• [Auto Sync] Update registry.py (20250915) by @merrymercy in https://github.com/sgl-project/sglang/pull/10484
• [router] fix worker registration in multi model mode by @CatherineSue in https://github.com/sgl-project/sglang/pull/10486
• fix crash of DeepSeek-V3 update_weights_from_disk by @scut-cbq in https://github.com/sgl-project/sglang/pull/8863
• Temporay work-around for rocm 7.0.0 alpha with enabling data-parallel issue by @kkHuang-amd in https://github.com/sgl-project/sglang/pull/10434
• [Hicache] Evaluate Per-Round Metrics in Multiturn Bench by @ykwd in https://github.com/sgl-project/sglang/pull/10203
• [ModelOpt] Respect kv_cache_quant_algo in ModelOpt checkpoints by @brayden-hai in https://github.com/sgl-project/sglang/pull/10336
• Add Logprobs unit test with a loose threshold by @PrinsYin in https://github.com/sgl-project/sglang/pull/10230
• [router] add router db connector for responses api by @slin1237 in https://github.com/sgl-project/sglang/pull/10487
• Remove wrong imports from sglang.python by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10493
• [router] fix router manager and router init in server by @CatherineSue in https://github.com/sgl-project/sglang/pull/10499
• Cache the result of is_blackwell platform check by @b8zhong in https://github.com/sgl-project/sglang/pull/10498
• feat: update support for qwen3next model by @cao1zhg in https://github.com/sgl-project/sglang/pull/10466
• Minor fix lint introduced by [#10466] by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10507
• chore: upgrade sgl-kernel 0.3.10 by @zhyncs in https://github.com/sgl-project/sglang/pull/10500
• Update CUTLASS. Refine KernelSchedule for fp8 (grouped) gemm. by @HydraQYH in https://github.com/sgl-project/sglang/pull/10491
• Fix CI when sgl-kernel is changed but srt is not changed by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10515
• Support sgl-router parallel_batch in bench_one_batch_server by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10506
• [CPU] fix CPU backend sel. issue for Llama4 by @ZailiWang in https://github.com/sgl-project/sglang/pull/10511
• adjust import setuptools_rust by @whybeyoung in https://github.com/sgl-project/sglang/pull/10524
• Fix formatting in long code blocks by @philipkiely-baseten in https://github.com/sgl-project/sglang/pull/10528
• skip vision_model for lora by @gongwei-130 in https://github.com/sgl-project/sglang/pull/10530
• [2/2] Speed up trtllm_mla attention backend by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10474
• support using fa4 on deepseek on blackwell by @cicirori in https://github.com/sgl-project/sglang/pull/9928
• [Auto Sync] Update scheduler_profiler_mixin.py, rpd_utils.p... (20250916) by @merrymercy in https://github.com/sgl-project/sglang/pull/10494
• [Auto Sync] Update activation.py, chunk_cache.py, utils.py (20250917) by @merrymercy in https://github.com/sgl-project/sglang/pull/10538
• feat: add priority based scheduling with priority based request acceptance and preemption by @harrisonlimh in https://github.com/sgl-project/sglang/pull/8746
• Fix decord dependency for aarch64 docker build by @kyleliang-nv in https://github.com/sgl-project/sglang/pull/10529
• enable prefix cache with dp by @wenscarl in https://github.com/sgl-project/sglang/pull/10459
• [bugfix]hicache bench_long_context.py run failed by @zhannngchen in https://github.com/sgl-project/sglang/pull/10523
• Remove duplicated code by @oraluben in https://github.com/sgl-project/sglang/pull/10545
• CUDA Arch Independent by @EduardDurech in https://github.com/sgl-project/sglang/pull/8813
• [bench] Fix random seed in bench_one_batch_server by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10548
• [HiCache] Add tests for hicache storage mooncake backend by @stmatengss in https://github.com/sgl-project/sglang/pull/10171
• [BugFix] Fix incorrect hidden_states_tensor in pd disaggregation + eagle by @ZeldaHuang in https://github.com/sgl-project/sglang/pull/9976
• fix: update dsv3 fp4 ut by @zhyncs in https://github.com/sgl-project/sglang/pull/10584
• vlm: remove redundant d2h movement of mm feature tensors by @AlienKevin in https://github.com/sgl-project/sglang/pull/9987
• Enable trtllm mla prefix extend by @wenscarl in https://github.com/sgl-project/sglang/pull/10526
• [ROCm] Fix fp8 quantization accuracy issue. by @sogalin in https://github.com/sgl-project/sglang/pull/10558
• [HICache] introduce evict policy by @XucSh in https://github.com/sgl-project/sglang/pull/10190
• aiter v0.1.5.post2 by @HaiShaw in https://github.com/sgl-project/sglang/pull/10563
• [PD] Improve disaggregation common backend and refactor mooncake backend by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10273
• chore: upgrade mooncake 0.3.6 by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10596
• [improvement] add average input/output token length for hicache benchmark stats output by @zhannngchen in https://github.com/sgl-project/sglang/pull/10525
• Scale kkt after reduction by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10604
• fix deepep assert when PD disaggregation == null by @alpha-baby in https://github.com/sgl-project/sglang/pull/8274
• [RL] Add destroy process group api by @penguin-wwy in https://github.com/sgl-project/sglang/pull/9979
• Feat/add heartbeat mechanism for nixl conn by @shaharmor98 in https://github.com/sgl-project/sglang/pull/10222
• update deepep version for qwen3-next deepep moe by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10624
• support qwen3-next-fp8 deepep by @yizhang2077 in https://github.com/sgl-project/sglang/pull/10622
• Fix sgl_kernel import failure on devices other than CUDA by @ZailiWang in https://github.com/sgl-project/sglang/pull/10610
• [Performance] qwen3-next improve causal conv1d in prefill phase by @liz-badada in https://github.com/sgl-project/sglang/pull/10595
• Fix bias handling in TritonMoeQuantInfo within quantization/mxfp4.py by @yhyang201 in https://github.com/sgl-project/sglang/pull/10579
• feat: Add FlexAttention Backend for Efficient Sparse Attention by @yukiy927 in https://github.com/sgl-project/sglang/pull/9947
• Garbage collector regression in the online server by @brayden-hai in https://github.com/sgl-project/sglang/pull/10621
• [router] refactor worker to builder pattern 1/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10628
• refactor: use registry for _get_attention_backend_from_str by @zhyncs in https://github.com/sgl-project/sglang/pull/10629
• [Feature] Speculative decoding support lookahead by @a4zhangfei in https://github.com/sgl-project/sglang/pull/9873
• [Performance] Qwen3-Next: replace arange to cached query_start_loc_li… by @byjiang1996 in https://github.com/sgl-project/sglang/pull/10553
• [Performance] Qwen3-Next: speed up update_mamba_state_after_mtp_verify by 10x; e2e up to 3.54% faster by @byjiang1996 in https://github.com/sgl-project/sglang/pull/10586
• model support: Sarashina2VisionForCausalLM by @CatherineSue in https://github.com/sgl-project/sglang/pull/10632
• feat: add fused moe config for Qwen3-Next-80B-A3B-Instruct on B200 by @zixuanzhang226 in https://github.com/sgl-project/sglang/pull/10631
• chore: bump sgl-kernel 0.3.11 by @zhyncs in https://github.com/sgl-project/sglang/pull/10630
• Hicache L3 backend mooncake optimization configuration reading method by @leihuang-sketch in https://github.com/sgl-project/sglang/pull/10319
• [router] refactor worker to builder pattern 2/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10633
• [Feature]feat(get_ip): unify get_ip_xxx by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/10081
• [router] refactor worker to builder pattern 3/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10647
• [1/2][sgl-kernel] Support moe_sum_reduce cuda kernel by @yuan-luo in https://github.com/sgl-project/sglang/pull/10321
• [router] refactor worker to builder pattern 4/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10650
• Fix fast decode plan for flashinfer v0.4.0rc1 and upgrade sgl-kernel 0.3.11 by @Fridge003 in https://github.com/sgl-project/sglang/pull/10634
• [router] refactor worker to builder pattern 5/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10653
• [HiCacheStorage]support page_first_direct layout for generic set&get by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/10522
• [router] preserve order of json params using preserve_order feature by @fgebhart in https://github.com/sgl-project/sglang/pull/10661
• [router] refactor router and worker management 1/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10664
• fix: resolve sync issue by @zhyncs in https://github.com/sgl-project/sglang/pull/10668
• [Auto Sync] Update .clang-format (20250919) by @zhyncs in https://github.com/sgl-project/sglang/pull/10670
• [router] refactor router and worker management 2/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10666
• router-spec: Reorder ChatCompletionRequest and fix validation logic by @CatherineSue in https://github.com/sgl-project/sglang/pull/10675
• chore: cleanup docker image by @zhyncs in https://github.com/sgl-project/sglang/pull/10671
• limit sgl-kernel causal conv1d to cuda only by @liz-badada in https://github.com/sgl-project/sglang/pull/10648
• [Auto Sync] Update model_runner.py (20250920) by @zhyncs in https://github.com/sgl-project/sglang/pull/10679
• [router] refactor router and worker management 2.5/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10677
• [1/2] Support deterministic inference with flashinfer attention backend by @Fridge003 in https://github.com/sgl-project/sglang/pull/10645
• [Auto Sync] Update deepseek_v2.py (20250920) by @zhyncs in https://github.com/sgl-project/sglang/pull/10683
• chore: upgrade mooncake 0.3.6.post1 to fix gb200 dockerfile by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10681
• [Performance] Qwen3-Next: optimize causal_conv1d_fn triton kernel - up to 9% faster by @byjiang1996 in https://github.com/sgl-project/sglang/pull/10680
• Replace os.environ in layernorm.py by @Fridge003 in https://github.com/sgl-project/sglang/pull/10684
• fix(disagg): fix sending KV cache in case of MLA for NIXL backend by @dmitrygx in https://github.com/sgl-project/sglang/pull/10673
• fix: update run_suite by @zhyncs in https://github.com/sgl-project/sglang/pull/10685
• fix: remove awq_dequantize deps by @zhyncs in https://github.com/sgl-project/sglang/pull/10686
• [Auto Sync] Update modelopt_quant.py (20250920) by @zhyncs in https://github.com/sgl-project/sglang/pull/10688
• [Feature] Support deterministic inference with FA3 backend by @hebiao064 in https://github.com/sgl-project/sglang/pull/10651
• feat: update server args by @zhyncs in https://github.com/sgl-project/sglang/pull/10696
• Super tiny fix extra logs by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10697
• [3/4] Speed up CSGMV backend perf by 10% through dynamic chunking + kernel optimization by @lifuhuang in https://github.com/sgl-project/sglang/pull/10592
• Update release-docs.yml by @sglang-bot in https://github.com/sgl-project/sglang/pull/10706
• Refactors radix cache for extra key support by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10317
• [Router]fix: fix get_load missing api_key by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/10385
• fix: disable gpt-oss b200 ut by @zhyncs in https://github.com/sgl-project/sglang/pull/10716
• Optimize cutlass int8 gemm kernel for large M on SM89 Ada GPU by @HydraQYH in https://github.com/sgl-project/sglang/pull/10714
• [Auto Sync] Update deepseek_v2.py (20250922) by @zhyncs in https://github.com/sgl-project/sglang/pull/10717
• Support deterministic inference with triton backend (Hardware test: NV and AMD GPUs) by @yushengsu-thu in https://github.com/sgl-project/sglang/pull/10694
• [deterministic inference] Move batch invariant pkg to sglang by @hebiao064 in https://github.com/sgl-project/sglang/pull/10695
• [2/2] Support deterministic inference for temperature > 0 by @Qiaolin-Yu in https://github.com/sgl-project/sglang/pull/10678
• [Ascend] codeowner updates for ascend related files by @ping1jing2 in https://github.com/sgl-project/sglang/pull/10699
• [4/4] Introduce CachedKernel to reduce CSGMV kernel launch overheads by 60% by @lifuhuang in https://github.com/sgl-project/sglang/pull/10709
• Convert FLASHINFER_WORKSPACE_SIZE to integer by @reyoung in https://github.com/sgl-project/sglang/pull/10731
• EPLB: prefer to use physical experts in the same node by @acelyc111 in https://github.com/sgl-project/sglang/pull/9849
• fix capture_bs when speculative decoding enabled by @feng397 in https://github.com/sgl-project/sglang/pull/10730
• Fix flaky logprobs test by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10728
• Fix CI TestChunkedSGMV by @lifuhuang in https://github.com/sgl-project/sglang/pull/10737
• [Docs, minor] Fix LLM doc matrix by @adarshxs in https://github.com/sgl-project/sglang/pull/10753
• Add warnings and remove dependency for deterministic inference by @Fridge003 in https://github.com/sgl-project/sglang/pull/10724
• bugfix: Fix get_worker_urls_for_model in http/router.rs by @CatherineSue in https://github.com/sgl-project/sglang/pull/10754
• [router] refactor router and worker management 3/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10727
• [router] update ci so only execute benchmarks when labels are added by @slin1237 in https://github.com/sgl-project/sglang/pull/10757
• Fix MTP MoE weight loading with NVFP4 target model. by @LorrinWWW in https://github.com/sgl-project/sglang/pull/10758
• chore: bump sgl-kernel v0.3.12 by @zhyncs in https://github.com/sgl-project/sglang/pull/10732
• [Generative Score API] Added test_scores_api.py to github CICD to run per commit by @vedantjh2 in https://github.com/sgl-project/sglang/pull/10755
• refactor zero copy by @pansicheng in https://github.com/sgl-project/sglang/pull/10300
• Fix multimodal registry and code sync scripts by @merrymercy in https://github.com/sgl-project/sglang/pull/10759
• Enables TRT-LLM backend to be used for target_verify by @pranavm-nvidia in https://github.com/sgl-project/sglang/pull/10281
• fix: kv events with tp > 1 by @ishandhanani in https://github.com/sgl-project/sglang/pull/10541
• [Auto Sync] Update flashattention_backend.py (20250922) by @zhyncs in https://github.com/sgl-project/sglang/pull/10762
• [Feature] Add MLAProcess for DeepSeek MLA on NPU by @iforgetmyname in https://github.com/sgl-project/sglang/pull/10130
• [Ascend] optimize Qwen-vl on Ascend by @ping1jing2 in https://github.com/sgl-project/sglang/pull/10556
• [Ascend]optimize Qwen3 on Ascend by @ping1jing2 in https://github.com/sgl-project/sglang/pull/10574
• [Auto Sync] Update configurer.py (20250923) by @merrymercy in https://github.com/sgl-project/sglang/pull/10765
• [router] refactor router and worker management 4/n by @slin1237 in https://github.com/sgl-project/sglang/pull/10756
• [router] remove pd router draining channel by @slin1237 in https://github.com/sgl-project/sglang/pull/10767
• [router] fix logger type mismatch by @CatherineSue in https://github.com/sgl-project/sglang/pull/10774
• Use simulate acc len from sglang.environ by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10771
• Fix trtllm_mla slow concat kernel in MTP by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10777
• Move cached kernel to srt.utils by @lifuhuang in https://github.com/sgl-project/sglang/pull/10776
• feat: unify dockerfiles by @ishandhanani in https://github.com/sgl-project/sglang/pull/10705
• Introduce FutureMap by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10715
• chore: upgrade sgl-kernel 0.3.12 by @zhyncs in https://github.com/sgl-project/sglang/pull/10782
• followup: clean up dockerfiles and release yamls by @ishandhanani in https://github.com/sgl-project/sglang/pull/10783
• Clean up server args by @merrymercy in https://github.com/sgl-project/sglang/pull/10770
• move environ into sglang.srt to avoid break SRT auto sync. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10791
• Fix hicache mooncake backend CI by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10792
• [router] fix cache aware lock by @slin1237 in https://github.com/sgl-project/sglang/pull/10773
• [router] responses api POST and GET with local storage by @slin1237 in https://github.com/sgl-project/sglang/pull/10581
• model: support qwen3-vl series by @zju-stu-lizheng in https://github.com/sgl-project/sglang/pull/10323
• [fix][pd-disag]no need set next batch sampling info done in prefill by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/10259
• [ROCm] Update aiter to v0.1.5.post3 by @sogalin in https://github.com/sgl-project/sglang/pull/10812
• [router] use dashmap for radix tree instead of hash for multi model by @slin1237 in https://github.com/sgl-project/sglang/pull/10814
• router(grpc): Implement route for chat_cmpl endpoint by @CatherineSue in https://github.com/sgl-project/sglang/pull/10761
• fix ceval by @ZhengHSI in https://github.com/sgl-project/sglang/pull/10504
• Remove duplicate code in qwen2 model by @Lzhang-hub in https://github.com/sgl-project/sglang/pull/10540
• [router] fix axum default body limit by @CatherineSue in https://github.com/sgl-project/sglang/pull/10818
• Fix latest main ci by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10799
• add tunning files for QWEN-3-NEXT by @yiakwy-xpu-ml-framework-team in https://github.com/sgl-project/sglang/pull/10794
• [Auto Sync] Update protocol.py (20250923) by @zhyncs in https://github.com/sgl-project/sglang/pull/10820
• fix: draft model IMA by overide max_positional_embeddings by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10787
• [Auto Sync] Update elementwise.py (20250923) by @merrymercy in https://github.com/sgl-project/sglang/pull/10823
• [Auto Sync] Update simple_eval_common.py (20250923) by @merrymercy in https://github.com/sgl-project/sglang/pull/10824
• [router] Support streaming for Openai Router Response api by @key4ng in https://github.com/sgl-project/sglang/pull/10822
• [router] add auth middleware for api key auth by @CatherineSue in https://github.com/sgl-project/sglang/pull/10826
• [Auto Sync] Update load_config.py, model_config.py, configu... (20250923) by @merrymercy in https://github.com/sgl-project/sglang/pull/10825
• Revert "[fix][pd-disag]no need set next batch sampling info done in prefill" by @merrymercy in https://github.com/sgl-project/sglang/pull/10828
• Add CI timeout guidelines by @merrymercy in https://github.com/sgl-project/sglang/pull/10829
• feat: add cache_salt support to request by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10718
• fix bailing_moe with enable_dp_attention by @GuoweiWangU in https://github.com/sgl-project/sglang/pull/10860
• ci: free space on workers for build by @ishandhanani in https://github.com/sgl-project/sglang/pull/10786
• router-grpc: Support jinja chat template content format detection by @CatherineSue in https://github.com/sgl-project/sglang/pull/10832
• [router] select first healthy worker on proxied get requests by @lun-4 in https://github.com/sgl-project/sglang/pull/10827
• chore: Initial support for input config files by @kushanam in https://github.com/sgl-project/sglang/pull/10534
• router-grpc: Add tools processing and other paramters for apply_chat_template by @CatherineSue in https://github.com/sgl-project/sglang/pull/10877
• [router] consolidate health endpoints and flush cache by @slin1237 in https://github.com/sgl-project/sglang/pull/10876
• Restruct sgl-kernel benchmark by @BBuf in https://github.com/sgl-project/sglang/pull/10861
• [Bug] Fix Issue#10215 by @yuhyao in https://github.com/sgl-project/sglang/pull/10572
• [router] consolidate worker get loads by @slin1237 in https://github.com/sgl-project/sglang/pull/10880
• [router] Support Oracle DB(ATP) Data Connector by @key4ng in https://github.com/sgl-project/sglang/pull/10845
• [router] simplify tokenizer dev doc by @slin1237 in https://github.com/sgl-project/sglang/pull/10895
• [Auto Sync] Update model_config.py (20250925) by @merrymercy in https://github.com/sgl-project/sglang/pull/10885
• [ci feature] add ci monitor by @BBuf in https://github.com/sgl-project/sglang/pull/10872
• [HiCache] Cleaning the deprecated host memory state by @xiezhq-hermann in https://github.com/sgl-project/sglang/pull/10778
• integrate AIBrix KVcache by @yapple in https://github.com/sgl-project/sglang/pull/10376
• Add fuse_moe per-channel tune by @sleepcoo in https://github.com/sgl-project/sglang/pull/10915
• [router] consolidate worker load monitoring by @slin1237 in https://github.com/sgl-project/sglang/pull/10894
• router: Fix constraint proto and build_constraint in grpc router by @CatherineSue in https://github.com/sgl-project/sglang/pull/10881
• Refactor kv_cache_scheme handling for quantization by @mmangkad in https://github.com/sgl-project/sglang/pull/10132
• refactor: Move grpc/client.rs to grpc_client/sglang_scheduler.rs by @CatherineSue in https://github.com/sgl-project/sglang/pull/10924
• fix env flashinfer by @Swipe4057 in https://github.com/sgl-project/sglang/pull/10910
• [minor] Remove deprecated function get_ip by @merrymercy in https://github.com/sgl-project/sglang/pull/10883
• Rename customer label -> custom label by @merrymercy in https://github.com/sgl-project/sglang/pull/10899
• [router] change log level to warning by @slin1237 in https://github.com/sgl-project/sglang/pull/10926
• [router][refactor] Clean up protobuf fields by @CatherineSue in https://github.com/sgl-project/sglang/pull/10923
• Replace the Kimi-K2 generated tool call idx with history tool call count by @eraser00 in https://github.com/sgl-project/sglang/pull/10612
• [ci] add ci-monitor workflow by @BBuf in https://github.com/sgl-project/sglang/pull/10898
• Remove pull_request trigger from CI monitor workflow by @merrymercy in https://github.com/sgl-project/sglang/pull/10932
• router: Support parallel sampling num > 1 in grpc_server and non-stream handling by @CatherineSue in https://github.com/sgl-project/sglang/pull/10929
• Revert "Refactor kv_cache_scheme handling for quantization (#10132)" by @zhyncs in https://github.com/sgl-project/sglang/pull/10935
• Update CODEOWNERS to include JustinTong0323 in FC by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10939
• [PD-HiCache]: Support Async Offloading KVCache In Decode Side by @hzh0425 in https://github.com/sgl-project/sglang/pull/10192
• CI: Fix docker manifest build by @csahithi in https://github.com/sgl-project/sglang/pull/10936
• [router] update owners for router components by @slin1237 in https://github.com/sgl-project/sglang/pull/10927
• Fuse write kv buffer into rope for qwen3 moe & bailing moe by @yuan-luo in https://github.com/sgl-project/sglang/pull/10749
• [router] add grpc client get and set by @slin1237 in https://github.com/sgl-project/sglang/pull/10955
• [router]fix code owner syntax error by @slin1237 in https://github.com/sgl-project/sglang/pull/10956
• [router] move grpc client from router to worker and builder by @slin1237 in https://github.com/sgl-project/sglang/pull/10958
• [router] add move grpc worker management from router to worker manager by @slin1237 in https://github.com/sgl-project/sglang/pull/10960
• [router] grpc router regular mode import cleanup by @slin1237 in https://github.com/sgl-project/sglang/pull/10963
• [router] remove old/oudated/useless comments by @slin1237 in https://github.com/sgl-project/sglang/pull/10967
• [router] remove old/oudated/useless comments across code base by @slin1237 in https://github.com/sgl-project/sglang/pull/10968
• ci: fix rate-limit of huggingface with hf auth login by @mickqian in https://github.com/sgl-project/sglang/pull/10947
• Update label field comment to indicate deprecation by @merrymercy in https://github.com/sgl-project/sglang/pull/10970
• Restruct gpu_memory_settings in a unify function and relax max_cuda_graph_bs by @BBuf in https://github.com/sgl-project/sglang/pull/10372
• ci: refactor nightly test by @mickqian in https://github.com/sgl-project/sglang/pull/10495
• refactor loading weights from remote instance coding format by @amysaq2023 in https://github.com/sgl-project/sglang/pull/10941
• [router][grpc] Add helpfer functions for decoder in router.rs and fix specs by @CatherineSue in https://github.com/sgl-project/sglang/pull/10971
• Add simple docker file for B300 by @hlu1 in https://github.com/sgl-project/sglang/pull/10944
• Ci monitor support performance by @BBuf in https://github.com/sgl-project/sglang/pull/10965
• [HiCache]: Support dynamic loading backends for hicache by @hzh0425 in https://github.com/sgl-project/sglang/pull/10551
• [Bugfix][Minor][Benchmark] Fix some bugs due to PR [#10495] by @Muqi1029 in https://github.com/sgl-project/sglang/pull/10982
• [router][grpc] Support E2E non-stream chat completions by @CatherineSue in https://github.com/sgl-project/sglang/pull/10980
• fix: fp8 quantization failure of qwen 2.5 VL 7B model by @PanJason in https://github.com/sgl-project/sglang/pull/10112
• [Fix] RuntimeError: get_cfg Unsupported input_type:Float4_e2m1fn_x2 in using aiter-mxfp4-moe by @kkHuang-amd in https://github.com/sgl-project/sglang/pull/10981
• fix: make inference deterministic for large TP by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10930
• Add auth to get server info by @Muqi1029 in https://github.com/sgl-project/sglang/pull/10751
• Add support for topk metadata transferring for PD by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10616
• [PD] Extract the PP transfer layer calculate logic from Mooncake to Common backend by @ShangmingCai in https://github.com/sgl-project/sglang/pull/10565
• Use jsonschema to constrain required or specific tool choice by @TJ5 in https://github.com/sgl-project/sglang/pull/10550
• Fix profiler by @merrymercy in https://github.com/sgl-project/sglang/pull/10997
• [router][tool parser] Modify tool parser to return both normal text and tool calls (non-stream) by @CatherineSue in https://github.com/sgl-project/sglang/pull/10995
• [router] basic mcp support for openai router response api by @key4ng in https://github.com/sgl-project/sglang/pull/10978
• [router] fix chat template loading and tokenizer path by @slin1237 in https://github.com/sgl-project/sglang/pull/10999
• Fix CI failure of TypeError: RotaryEmbedding.forward_cpu() got an unexpected keyword argument 'fused_set_kv_buffer_arg' by @yanbing-j in https://github.com/sgl-project/sglang/pull/11009
• [bugfix]Add empty_context import to two_batch_overlap.py by @wejoncy in https://github.com/sgl-project/sglang/pull/10964
• prepare for sglang+verl by @lbk-sys in https://github.com/sgl-project/sglang/pull/10555
• [sgl-kernel] Optimize concat_mla_k kernel by @yuan-luo in https://github.com/sgl-project/sglang/pull/10543
• [HiCache] bug: fix mooncake store batch set v1 by @stmatengss in https://github.com/sgl-project/sglang/pull/11013
• Fix FusedSetKVBufferArg in RotaryEmbedding by @merrymercy in https://github.com/sgl-project/sglang/pull/11003
• Update GLM-4.5 Model Doc by @zRzRzRzRzRzRzR in https://github.com/sgl-project/sglang/pull/11017
• [router] migrate to rust python module for pythonic parser by @slin1237 in https://github.com/sgl-project/sglang/pull/11033
• fix: show failed models in nightly ci by @mickqian in https://github.com/sgl-project/sglang/pull/10986
• [router][tool call] Support normal content extraction before tool call (streaming) by @CatherineSue in https://github.com/sgl-project/sglang/pull/11038
• [router] add harmony tool parser base structure and interface by @slin1237 in https://github.com/sgl-project/sglang/pull/11036
• Unify SGL Kernel Releases by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/10701
• [1/2] Support FA4 for MHA Prefill in sgl-kernel by @lifuhuang in https://github.com/sgl-project/sglang/pull/10940
• fix: check if weights are already local before downloading by @mickqian in https://github.com/sgl-project/sglang/pull/11015
• [HiCacheStorage] mooncake store support page_first_direct layout by @huangtingwei9988 in https://github.com/sgl-project/sglang/pull/10591
• [speculative decoding] rename lookahead to ngram by @a4zhangfei in https://github.com/sgl-project/sglang/pull/11010
• Fix gemma 3 launch with transformers: the error: AttributeError: 'TransformersForCausalLM' object has no attribute 'tp_size' by @vincentzed in https://github.com/sgl-project/sglang/pull/9614
• Fix sgl-kernel benchmark dead code by @BBuf in https://github.com/sgl-project/sglang/pull/11022
• [router][tool call] Improve normal content extraction and error handling (non-stream) by @CatherineSue in https://github.com/sgl-project/sglang/pull/11050
• chore: upgrade cutedsl 4.2.1 by @zhyncs in https://github.com/sgl-project/sglang/pull/11054
• [Ci Monitor] Auto uploaded performance data to sglang_ci_data repo by @BBuf in https://github.com/sgl-project/sglang/pull/10976
• chore: upgrade sgl-kernel 0.3.13 by @zhyncs in https://github.com/sgl-project/sglang/pull/11056
• [router] add n to generate sampling params by @slin1237 in https://github.com/sgl-project/sglang/pull/11069
• Use more general heuristics to set the default value of --mem-fraction-static by @merrymercy in https://github.com/sgl-project/sglang/pull/10975
• [router][tool call] Separate JsonParser and LlamaParser by @CatherineSue in https://github.com/sgl-project/sglang/pull/11073
• Fix mem fraction static for nightly tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11076
• fix: fp8 mllama4 without vision modules being quantized by @mickqian in https://github.com/sgl-project/sglang/pull/10611
• Use get_pooled in process_single_choice by @CatherineSue in https://github.com/sgl-project/sglang/pull/11079
• [router][grpc] Add logprobs support to router by @CatherineSue in https://github.com/sgl-project/sglang/pull/11082
• feat(reasoning): improve enable thinking from request by @jinmingyi1998 in https://github.com/sgl-project/sglang/pull/10875
• [Profile] dump memory trace when cuda graph profile is enabled by @ch-wan in https://github.com/sgl-project/sglang/pull/11083
• Remove hybrid_linear_attn attention backend and refactor attention registry by @samuellees in https://github.com/sgl-project/sglang/pull/10816
• [model] added support for w8a8int8 used by neuralmagic/Qwen2-0.5B-Ins… by @DevashishLal-CB in https://github.com/sgl-project/sglang/pull/9642
• Enable optional FP32 compute for LM Head by @narutolhy in https://github.com/sgl-project/sglang/pull/10729
• Update CODEOWNERS for attention/ascend_backend.py by @merrymercy in https://github.com/sgl-project/sglang/pull/11092
• [router] grpc router generate endpoint support by @slin1237 in https://github.com/sgl-project/sglang/pull/11070
• [router][tool call] Full support for ToolChoice by @CatherineSue in https://github.com/sgl-project/sglang/pull/11085
• Fix spec filter batch when target extend by @ispobock in https://github.com/sgl-project/sglang/pull/10991
• [Fix] Resolve performance drop in speculative decoding aiter backend by @yichiche in https://github.com/sgl-project/sglang/pull/11087
• [Auto Sync] Update fused_moe_triton_config.py (20250930) by @merrymercy in https://github.com/sgl-project/sglang/pull/11099
• chore: bump sgl-kernel v0.3.14 by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11067
• [router][grpc-server] Fix gRPC server shutdown by @slin1237 in https://github.com/sgl-project/sglang/pull/11094
• Fix eagle radix cache by @ispobock in https://github.com/sgl-project/sglang/pull/10846
• [Eval] Add --repeat in run_eval by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11101
• [CPU] Adding Memory Capacity Acquisition Functionality by @ZailiWang in https://github.com/sgl-project/sglang/pull/11102
• Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization by @trevor-m in https://github.com/sgl-project/sglang/pull/11081
• Support Dots.ocr model by @albaNnaksqr in https://github.com/sgl-project/sglang/pull/11071
• [router][bugfix] Fix input_logprobs handling with None value and logprob_start_len = -1 by @CatherineSue in https://github.com/sgl-project/sglang/pull/11113
• Feature/make PEFT adapter module format compatibile by @ConnorLi96 in https://github.com/sgl-project/sglang/pull/11080
• fix: KimiK2Detector Improve tool call ID parsing with regex by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/10972
• [router] add mcp list and mcp call in output array by @key4ng in https://github.com/sgl-project/sglang/pull/11112
• Organize spec-related data structures by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10735
• [AMD] Add Tilelang and Fast Hadamard Transform builds to Dockerfile.rocm by @hubertlu-tw in https://github.com/sgl-project/sglang/pull/11114
• [Auto Sync] Update base_grammar_backend.py, xgrammar_backen... (20250930) by @merrymercy in https://github.com/sgl-project/sglang/pull/11115
• [Doc] Update multimodal language models documentation by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11111
• Quick Fix: fix Qwen3-VL launch failure caused by MRotaryEmbedding arg by @yhyang201 in https://github.com/sgl-project/sglang/pull/10985
• docker: x86 dev builds for hopper and blackwell by @ishandhanani in https://github.com/sgl-project/sglang/pull/11075
• Refactor AMD CI. by @saienduri in https://github.com/sgl-project/sglang/pull/11128
• feat: add fast_decode_plan from flashinfer, flashinfer to 0.4.0rc3 by @yyihuang in https://github.com/sgl-project/sglang/pull/10760
• [HiCache]bug fix: fixed blank item in host_mem_release_queue by @zhangzuo21 in https://github.com/sgl-project/sglang/pull/11005
• [Feature] Add EIC as sglang HiCache Storage backend by @mss1213 in https://github.com/sgl-project/sglang/pull/10271
• [HiCache] Configurable and Dynamic Prefetch Timeout by @ykwd in https://github.com/sgl-project/sglang/pull/10512
• [router] add pd service in grpc router for pd by @slin1237 in https://github.com/sgl-project/sglang/pull/11120
• [router] Add multi-turn tool calling loop support for MCP integration by @key4ng in https://github.com/sgl-project/sglang/pull/11143
• Fix metrics and request tracing (TimeStats) by @merrymercy in https://github.com/sgl-project/sglang/pull/11123
• Remove debug print statement from scheduler output by @merrymercy in https://github.com/sgl-project/sglang/pull/11145
• Intoduce cpu tensor as metadata to avoid blocking gpu kernel launch by @AHEADer in https://github.com/sgl-project/sglang/pull/10720
• Fix ngram spec with page size > 1 by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11135
• [ROCm] To reduce the compiling time when using torch compile. by @sogalin in https://github.com/sgl-project/sglang/pull/10559
• Fix DeepSeek chunked prefill memory issue by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11149
• Clean up parallel_state.py by @merrymercy in https://github.com/sgl-project/sglang/pull/11148
• Tiny improve dumper by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11132
• Tiny fix missing alt stream in nextn layer by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10768
• Fuse quantize and rope in trtllm_mla MTP by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10779
• Tiny detect slow ranks by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10508
• Remove unused pack .item() in paged allocator. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11156
• Support dispatch low latency by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10263
• Support single batch overlap by @fzyzcjy in https://github.com/sgl-project/sglang/pull/10422
• [router][grpc] Support tool call parser in streaming by @CatherineSue in https://github.com/sgl-project/sglang/pull/11160
• [model] Add mamba2 and Falcon-H1 support. by @ilyasch2 in https://github.com/sgl-project/sglang/pull/10988
• Clean up ascend allocator by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11152
• fix cpp JIT compilation issue of ngram speculative decoding by @b8zhong in https://github.com/sgl-project/sglang/pull/10837
• Tiny cleanup deepseek_v2.py by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11163
• Tiny fix ep_gather behavior different in CI by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11130
• Tiny remove duplicated code by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11164
• [proto] Add script to compile python protos by @CatherineSue in https://github.com/sgl-project/sglang/pull/11171
• Unify forward output datastructure by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11124
• [grpc] style fix for grpc compilation. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11175
• Remove dp balance metadata and minimul token balance. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11170
• Minor fixes for server_args, parallel_state, and test_deterministic.py by @merrymercy in https://github.com/sgl-project/sglang/pull/11159
• fix: shoudn't include CUDA_ARCH 100 and 120 for cuda12.6.1 by @gongwei-130 in https://github.com/sgl-project/sglang/pull/11176
• [router][grpc] Support streaming for v1/chat/completions by @CatherineSue in https://github.com/sgl-project/sglang/pull/11179
• Allow use of TRTLLM_MHA backend for hybrid attention on Blackwell by @DomBrown in https://github.com/sgl-project/sglang/pull/11138
• Introduce naming convention in io_struct and base sglang io classes. by @hnyls2002 in https://github.com/sgl-project/sglang/pull/10133
• [Generative Scores API] add performance tests to CICD by @vedantjh2 in https://github.com/sgl-project/sglang/pull/10830
• [1/n] Enable DCA CUDA graph capture by @b8zhong in https://github.com/sgl-project/sglang/pull/9537
• [Fix] Update to v0.1.5.post4 and refine HIP attention backend selection by @yichiche in https://github.com/sgl-project/sglang/pull/11161
• [CI]] Tee server logs to both file and stdout/stderr using PIPE by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11185
• fix: radix cache memory accounting by @skyzh in https://github.com/sgl-project/sglang/pull/10637
• Tiny add PD disaggregation + DP attention test by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11167
• [router] Steaming support for MCP Tool Calls in OpenAI Router by @key4ng in https://github.com/sgl-project/sglang/pull/11173
• [Feature] Option to save model weights to CPU when memory saver mode is enabled by @mattnappo in https://github.com/sgl-project/sglang/pull/10873
• Add --thinking-mode to run_eval by @hlu1 in https://github.com/sgl-project/sglang/pull/11189
• [hot-fix] Fix CI break which caused by adding thinking_mode in eval by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11192
• Tiny move files to utils folder by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11166
• Fix CUDA illegal memory access issues in speculative decoding by @ur4t in https://github.com/sgl-project/sglang/pull/10892
• Fix [test]: Env:SGLANG_TORCH_PROFILER_DIR for pytest. by @singhalshubham03 in https://github.com/sgl-project/sglang/pull/10780
• Optimize debug log position of PD abort request by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11090
• fix 3fs indices by @pansicheng in https://github.com/sgl-project/sglang/pull/10855
• model: support starcoder2 by @ppraneth in https://github.com/sgl-project/sglang/pull/10609
• [Test] Initialize mem_fraction_static in setUpClass to fix pytest VLM test crashes. by @vshekhawat-hlab in https://github.com/sgl-project/sglang/pull/10859
• fix xeon ci check by @DiweiSun in https://github.com/sgl-project/sglang/pull/10838
• fix qwen2 eagle3 runtime error by @jiapingW in https://github.com/sgl-project/sglang/pull/10517
• [minor] fix the lint by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11198
• [Fix] Fix the bug of the calculation of base_gpu_id (dp offset) in data_parallel_controller.py by @XSongQ in https://github.com/sgl-project/sglang/pull/10741
• [fix]missing prefix_lens_cpu init when p/d disaggregation by @HanHan009527 in https://github.com/sgl-project/sglang/pull/11196
• fix self.enable_kv_cache_events by @narutolhy in https://github.com/sgl-project/sglang/pull/11178
• [HICache]: Refactor HiCache CI by @hzh0425 in https://github.com/sgl-project/sglang/pull/11011
• fix sampling_seed handling when deterministic is enabled by @skyzh in https://github.com/sgl-project/sglang/pull/11096
• [fix]enable flashmla when using draft model P/D attention select by @HanHan009527 in https://github.com/sgl-project/sglang/pull/11012
• [router] fix get load response parsing by @slin1237 in https://github.com/sgl-project/sglang/pull/11213
• [router] add grpc router pd mode for chat and generate by @slin1237 in https://github.com/sgl-project/sglang/pull/11140
• EAGLE cache fix for HiCache by @ispobock in https://github.com/sgl-project/sglang/pull/11215
• Add --max-new-tokens CLI flag for MMMU evaluation by @yhyang201 in https://github.com/sgl-project/sglang/pull/11217
• Add DeepSeek-V3.2 Tool Call Template by @Xu-Wenqing in https://github.com/sgl-project/sglang/pull/11063
• Tiny skip_sample adjust by @hnyls2002 in https://github.com/sgl-project/sglang/pull/11225
• [Feature] Add a fast-topk to sgl-kernel for DeepSeek v3.2 by @DarkSharpness in https://github.com/sgl-project/sglang/pull/11194
• Update v1/responses to be more OpenAI-compatible. by @vincentzed in https://github.com/sgl-project/sglang/pull/9624
• chore: bump sgl-kernel v0.3.14.post1 by @FlamingoPg in https://github.com/sgl-project/sglang/pull/11137
• Update DeepGEMM repository tag to specific commit by @merrymercy in https://github.com/sgl-project/sglang/pull/11229
• [Feat] Support Torch Symm Mem AllReduce by @yuan-luo in https://github.com/sgl-project/sglang/pull/10571
• Refactor and optimize mooncake CI by @ShangmingCai in https://github.com/sgl-project/sglang/pull/11162
• [Fix AMD CI] VRAM cleanup by @sunxxuns in https://github.com/sgl-project/sglang/pull/11174
• Update transformers package version to 4.57.0 by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11222
• Remove gdrcopy check in ci_install_deepep.sh by @ch-wan in https://github.com/sgl-project/sglang/pull/11237
• Rename runner labels by @merrymercy in https://github.com/sgl-project/sglang/pull/11228
• [Auto Sync] Update io_struct.py (20251004) by @merrymercy in https://github.com/sgl-project/sglang/pull/11206
• Create two new GH workflows to automatically bump SGLang and Kernel version by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/10996
• Fix spec_utils.py by @sglang-bot in https://github.com/sgl-project/sglang/pull/11247
• ci: make find_local_hf_snapshot_dir more robust by @mickqian in https://github.com/sgl-project/sglang/pull/11248
• [quantization] Fix scale remapping for mllama4 by @BowenBao in https://github.com/sgl-project/sglang/pull/10042
• [quantization] Enable aiter mxfp4 fused_moe for Quark by @BowenBao in https://github.com/sgl-project/sglang/pull/10048
• Use cu128 for torch audio to fix some CI tests by @merrymercy in https://github.com/sgl-project/sglang/pull/11251
• Bump torch_memory_saver 0.0.9rc2 by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11252
• update sgl kernel version to 0.3.14.post1 by @merrymercy in https://github.com/sgl-project/sglang/pull/11242
• Update condition for sgl-kernel-benchmark-test by @merrymercy in https://github.com/sgl-project/sglang/pull/11254
• feat: add shortcut detection for multimodal templates in Jinja format by @JustinTong0323 in https://github.com/sgl-project/sglang/pull/11209
• Improve bot release workflow by @Kangyan-Zhou in https://github.com/sgl-project/sglang/pull/11240
• Add flashmla and fast hadamard transform to Dockerfile by @Fridge003 in https://github.com/sgl-project/sglang/pull/11235
• Support DeepSeek V3.2 Exp by @fzyzcjy in https://github.com/sgl-project/sglang/pull/11061
• chore: bump SGLang version to 0.5.3rc2 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11259
• chore: bump SGLang version to 0.5.3 by @sglang-bot in https://github.com/sgl-project/sglang/pull/11263

New Contributors

• @chenqianfzh made their first contribution in https://github.com/sgl-project/sglang/pull/10356
• @yonghenglh6 made their first contribution in https://github.com/sgl-project/sglang/pull/8778
• @amysaq2023 made their first contribution in https://github.com/sgl-project/sglang/pull/8215
• @lj970926 made their first contribution in https://github.com/sgl-project/sglang/pull/10410
• @thalahors made their first contribution in https://github.com/sgl-project/sglang/pull/10076
• @sufeng-buaa made their first contribution in https://github.com/sgl-project/sglang/pull/9962
• @tao12345666333 made their first contribution in https://github.com/sgl-project/sglang/pull/10129
• @ooapex made their first contribution in https://github.com/sgl-project/sglang/pull/10401
• @Jimmy-L99 made their first contribution in https://github.com/sgl-project/sglang/pull/10439
• @1195343015 made their first contribution in https://github.com/sgl-project/sglang/pull/10471
• @csahithi made their first contribution in https://github.com/sgl-project/sglang/pull/9887
• @scut-cbq made their first contribution in https://github.com/sgl-project/sglang/pull/8863
• @brayden-hai made their first contribution in https://github.com/sgl-project/sglang/pull/10336
• @PrinsYin made their first contribution in https://github.com/sgl-project/sglang/pull/10230
• @philipkiely-baseten made their first contribution in https://github.com/sgl-project/sglang/pull/10528
• @zhannngchen made their first contribution in https://github.com/sgl-project/sglang/pull/10523
• @alpha-baby made their first contribution in https://github.com/sgl-project/sglang/pull/8274
• @yukiy927 made their first contribution in https://github.com/sgl-project/sglang/pull/9947
• @a4zhangfei made their first contribution in https://github.com/sgl-project/sglang/pull/9873
• @leihuang-sketch made their first contribution in https://github.com/sgl-project/sglang/pull/10319
• @fgebhart made their first contribution in https://github.com/sgl-project/sglang/pull/10661
• @dmitrygx made their first contribution in https://github.com/sgl-project/sglang/pull/10673
• @sglang-bot made their first contribution in https://github.com/sgl-project/sglang/pull/10706
• @yushengsu-thu made their first contribution in https://github.com/sgl-project/sglang/pull/10694
• @reyoung made their first contribution in https://github.com/sgl-project/sglang/pull/10731
• @feng397 made their first contribution in https://github.com/sgl-project/sglang/pull/10730
• @LorrinWWW made their first contribution in https://github.com/sgl-project/sglang/pull/10758
• @vedantjh2 made their first contribution in https://github.com/sgl-project/sglang/pull/10755
• @zju-stu-lizheng made their first contribution in https://github.com/sgl-project/sglang/pull/10323
• @ZhengHSI made their first contribution in https://github.com/sgl-project/sglang/pull/10504
• @GuoweiWangU made their first contribution in https://github.com/sgl-project/sglang/pull/10860
• @lun-4 made their first contribution in https://github.com/sgl-project/sglang/pull/10827
• @eraser00 made their first contribution in https://github.com/sgl-project/sglang/pull/10612
• @TJ5 made their first contribution in https://github.com/sgl-project/sglang/pull/10550
• @wejoncy made their first contribution in https://github.com/sgl-project/sglang/pull/10964
• @lbk-sys made their first contribution in https://github.com/sgl-project/sglang/pull/10555
• @Kangyan-Zhou made their first contribution in https://github.com/sgl-project/sglang/pull/10701
• @samuellees made their first contribution in https://github.com/sgl-project/sglang/pull/10816
• @albaNnaksqr made their first contribution in https://github.com/sgl-project/sglang/pull/11071
• @ConnorLi96 made their first contribution in https://github.com/sgl-project/sglang/pull/11080
• @zhangzuo21 made their first contribution in https://github.com/sgl-project/sglang/pull/11005
• @AHEADer made their first contribution in https://github.com/sgl-project/sglang/pull/10720
• @ilyasch2 made their first contribution in https://github.com/sgl-project/sglang/pull/10988
• @DomBrown made their first contribution in https://github.com/sgl-project/sglang/pull/11138
• @skyzh made their first contribution in https://github.com/sgl-project/sglang/pull/10637
• @mattnappo made their first contribution in https://github.com/sgl-project/sglang/pull/10873
• @ur4t made their first contribution in https://github.com/sgl-project/sglang/pull/10892
• @singhalshubham03 made their first contribution in https://github.com/sgl-project/sglang/pull/10780
• @XSongQ made their first contribution in https://github.com/sgl-project/sglang/pull/10741
• @sunxxuns made their first contribution in https://github.com/sgl-project/sglang/pull/11174
• @BowenBao made their first contribution in https://github.com/sgl-project/sglang/pull/10042

Full Changelog: https://github.com/sgl-project/sglang/compare/v0.5.2...v0.5.3